One of the elements that affect 3D tracking performance in Virtual Reality (VR) systems is fluctuations in the signal, called jitter, which occurs regardless of the sensor technology used. In real-life VR systems, positional and rotational jitter can be found in all tracked objects, including the headset, controllers, or other trackers. Previous work had identified that ±0.5° rotational jitter negatively affects user performance for distal pointing. Yet, they also found that even using a second controller to reduce the “Heisenberg effect” introduced by the button press does not address the problem completely. Moreover, with jitter on the position of a virtual object, user performance significantly decreases with jitter above one fourth of the size of that virtual object. Still, users preferred to have positional jitter on a virtual target rather than rotational jitter on a VR controller. In this paper, we extended the previous literature by conducting a user study on angular jitter with controllers held with two different grip styles and targets at two different depth distances. The results revealed that user performance decreases (already) with ±0.25° jitter. Thus, we suggest that practitioners/developers who design 3D user interfaces, controllers, and interaction techniques for daily 3D VR usage should focus on reducing jitter. Decreasing jitter not only improves user performance, but also decreases frustration, which improves the user experience.