Chatterjee, AgneetLuo, YiranGokhale, TejasYang, YezhouBaral, Chitta2024-08-272024-08-272024-10-30Chatterjee, Agneet, Yiran Luo, Tejas Gokhale, Yezhou Yang, and Chitta Baral. “REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models.” Edited by Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol. Computer Vision – ECCV 2024, 2025, 339–57. https://doi.org/10.1007/978-3-031-73404-5_20.https://doi.org/10.1007/978-3-031-73404-5_20http://hdl.handle.net/11603/35795Computer Vision – ECCV 2024 - 18th European Conference, Milan, Italy, September 29–October 4, 2024,Text-to-Image (T2I) and multimodal large language models (MLLMs) have been adopted in solutions for several computer vision and multimodal learning tasks. However, it has been found that such vision-language models lack the ability to correctly reason over spatial relationships. To tackle this shortcoming, we develop the REVISION framework which improves spatial fidelity in vision-language models. REVISION is a 3D rendering based pipeline that generates spatially accurate synthetic images, given a textual prompt. REVISION is an extendable framework, which currently supports 100+ 3D assets, 11 spatial relationships, all with diverse camera perspectives and backgrounds. Leveraging images from REVISION as additional guidance in a training-free manner consistently improves the spatial consistency of T2I models across all spatial relationships, achieving competitive performance on the VISOR and T2I-CompBench benchmarks. We also design RevQA, a question-answering benchmark to evaluate the spatial reasoning abilities of MLLMs, and find that state-of-the-art models are not robust to complex spatial reasoning under adversarial settings. Our results and findings indicate that utilizing rendering-based frameworks is an effective approach for developing spatially-aware generative models.19 pagesen-USThis item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.Computer Science - Computer Vision and Pattern RecognitionText