Mind: A Context-Based Multimodal Interpretation Framework in Conversational Systems
Files
Links to Files
Collections
Author/Creator
Author/Creator ORCID
Date
Type of Work
Department
Program
Citation of Original Publication
Chai, Joyce Y., Shimei Pan, and Michelle X. Zhou. “Mind: A Context-Based Multimodal Interpretation Framework in Conversational Systems.” In Advances in Natural Multimodal Dialogue Systems, edited by Jan C. J. van Kuppevelt, Laila Dybkjær, and Niels Ole Bernsen, 265–85. Dordrecht: Springer Netherlands, 2005. https://doi.org/10.1007/1-4020-3933-6_12.
Rights
This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Subjects
Abstract
In a multimodal human-machine conversation, user inputs are often abbreviated or imprecise. Simply fusing multimodal inputs together may not be sufficient to derive a complete understanding of the inputs. Aiming to handle a wide variety of multimodal inputs, we are building a context-based multimodal interpretation framework called MIND (Multimodal Interpreter for Natural Dialog). MIND is unique in its use of a variety of contexts, such as domain context and conversation context, to enhance multimodal interpretation. In this chapter, we first describe a fine-grained semantic representation that captures salient information from user inputs and the overall conversation, and then present a context-based interpretation approach that enables MIND to reach a full understanding of user inputs, including those abbreviated or imprecise ones.