Synthetic imagery has often been considered unsuitable for demonstrating the performance of vision algorithms and systems. We argue that (despite many remaining difficulties) simulation and computer graphics are at a point today that make them extremely useful for evaluation and training, even for complex outdoor applications. This is particularly valuable for autonomous and robotics applications, where the lack of suitable training data and ground truth information is a severe bottleneck. Extensive testing in a synthetic environment should become an integral part of the systems development and evaluation process to reduce the possibility of failure in the real word. We describe ongoing efforts towards the development of an "Immersive Perception Simulator" and discuss some of the specific problems involved.