Displaying data visualizations in Virtual Reality environments enables analysts to explore large amounts of information more easily, supported by different perspectives and stereoscopic 3D display. Easy-to-use interaction affords additional benefits, such as more intuitively querying or filtering the data. Many Immersive Analytics systems employ either a virtual hands metaphor, with actions such as grabbing and stretching, or virtual ray pointers, with actions assigned to controller buttons. However, the effect of this choice in immersive visualization systems is unknown. Considering that each approach has different advantages, we implemented and evaluated both as well as a third option: seamlessly integrating both modes and allowing the user to alternate between them without explicit mode switches. We compared these three conditions through a controlled user study in the spatio-temporal data domain. We did not find significant differences between hands and ray-casting in task performance, workload, or interactivity patterns. Yet, 60% of the participants preferred the mixed mode and benefited from it by choosing the best alternative for each low-level task. This mode significantly reduced completion times by 23% for the most demanding task, at the cost of a 5% decrease in overall success rates.