Towards Integrated Multimodal Interaction: Merging Immersive 3D Worlds with Language Based Retrieval for 3D Scene Understanding

Abstract

We propose a novel multimodal interactive system for 3D scene understanding and question-answering for disaster scenarios and related tasks. Our approach creates open-domain annotations for arbitrary RGB image sequences, enabling natural language-based retrieval of 3D scenes. We incorporate an automated evaluation strategy using a vision-language model (VLM) to identify temporal differences between two scenes, significantly increasing scene understanding and same-place recognition accuracy. We demonstrate the robustness of our method on dynamic scenes, including indoor environments and real-world disasters. Finally, we test our method within human-agent collaboration by designing a novel interface for users to ask questions and retrieve visual evidence from a 3D scene rendered with 3D Gaussian Splatting (3DGS) as well as navigate through it on a desktop. Users were highly engaged with the interface, and succeeded in providing visual evidence using natural language based queries and navigation for tasks with properties that may appear in emergency responses. Both the method and task-based interface lay foundations for more resilient emergency management technologies that can adapt to rapidly changing environments.