The vergence–accommodation conflict (VAC) in stereoscopic head-mounted displays is known to affect depth perception and interaction performance. Prior studies have examined its impact on a controller, but it remains unclear how the VAC affects gaze pointing in Virtual Reality. Here, we investigate gaze- and controller-based 3D target selection under varying depth conditions. Results show that gaze is more sensitive to depth variation, exhibiting longer movement times, higher variability along the task axis, and lower throughput compared to controller-based interaction. "Variation in Diopter" Fitts' law modeling further indicates greater depth-dependent variability for gaze input. Despite these differences, perceived usability and workload were comparable. These findings suggest that the VAC affects pointing modalities differently, which should be considered when designing gaze-enabled 3D interfaces.