Object selection and manipulation (e.g. moving, rotating) are the most basic tasks in 3D scene construction. While most research on selection and manipulation techniques targets single objects, we examine the concept of group selection in this paper. Group selection is often given lesser importance than single object selection, yet is vital in providing users with a way to modify larger scenes with objects which are repetitive, sequential, or otherwise inherently understood as 'belonging together' by a user. We observed users manipulating objects in 3D scenes, and while doing so, they clearly expected that objects would be grouped based on their gravitational relationship. That is, all objects that are supported by some selected object will follow the motion of the selected object when manipulated. In this paper, we present a system that efficiently supports the manipulation of groups of objects via a gravitational hierarchy. As this hierarchy is derived with a collision detector, the new grouping techniques do not require semantic or user specified information to work. The results of the evaluation show that using the gravitational hierarchy improves scene rearrangement significantly compared to conventional non-hierarchical methods. Finally, we discuss lessons learned from this study and make some suggestions on how the results can be incorporated into other systems.