Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech

Thumbnail Image

Files

21335-Article Text-25348-1-2-20220628.pdf (667.7 KB)

Links to Files

https://ojs.aaai.org/index.php/AAAI/article/view/21335

Permanent Link

http://hdl.handle.net/11603/24077
 https://doi.org/10.1609/aaai.v36i10.21335

Collections

UMBC Computer Science and Electrical Engineering Department
UMBC Faculty Collection
UMBC Student Collection

Author/Creator

Author/Creator ORCID

https://orcid.org/0000-0001-5744-8736
 https://orcid.org/0000-0002-9900-1972

Date

2022-06-28

Type of Work

journal articles
Text

Department

Program

Citation of Original Publication

Kebe, Gaoussou Youssouf, Luke E. Richards, Edward Raff, Francis Ferraro, and Cynthia Matuszek. 2022. “Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech”. Proceedings of the AAAI Conference on Artificial Intelligence 36 (10):10884-93. https://doi.org/10.1609/aaai.v36i10.21335.

Rights

This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Attribution 4.0 International (CC BY 4.0)

Subjects

UMBC Interactive Robotics and Language Lab
UMBC Ebiquity Research Group

Abstract

Learning to understand grounded language, which connects natural language to percepts, is a critical research area. Prior work in grounded language acquisition has focused primarily on textual inputs. In this work we demonstrate the feasibility of performing grounded language acquisition on paired visual percepts and raw speech inputs. This will allow interactions in which language about novel tasks and environments is learned from end users, reducing dependence on textual inputs and potentially mitigating the effects of demographic bias found in widely available speech recognition systems. We leverage recent work in self-supervised speech representation models and show that learned representations of speech can make language grounding systems more inclusive towards specific groups while maintaining or even increasing general performance.