Transfer Learning of Grounded Language Models For Use In Robotic Systems

Simple item page
dc.contributor.advisorMatuszek, Cynthia
dc.contributor.authorJenkins, Patrick
dc.contributor.departmentComputer Science and Electrical Engineering
dc.contributor.programComputer Science
dc.date.accessioned2021-09-01T13:55:36Z
dc.date.available2021-09-01T13:55:36Z
dc.date.issued2020-01-01
dc.description.abstractGrounded language acquisition is the modeling of language as it relates to physical objects in the world. Grounded language models are useful for creating an interface between robots and humans using natural language, but are ineffective when a robot enters a novel environment due to lack of training data. I create a novel grounded language dataset by capturing multi-angle high resolution color and depth images of household objects, then collecting natural language text and speech descriptions of the objects. This dataset is used to train a model that learns associations between the descriptions and the color and depth percepts. Vision and language domains are embedded into an intermediate lower dimensional space through manifold alignment. The model consists of two simultaneously trained neural nets, one each for vision and language. Triplet loss ensures that the two spaces are closely aligned in the embedded space by attracting positive associations and repelling negative ones. First, separate models are trained using the University of Washington RGB-D and UMBC GLD datasets to get baseline results for grounded language acquisition on domestic objects. Then the baseline model trained on the UW RGB-D data is fine tuned through a second round of training on UMBC GLD. This fine tuned model performs better than the model trained only on UMBC GLD, and in less training time. These experiments represent the first steps of the ability to transfer grounded language knowledge from previously trained models on large datasets onto new models operating on robots operating in novel domains.
dc.formatapplication:pdf
dc.genretheses
dc.identifierdoi:10.13016/m2vbzz-sfnp
dc.identifier.other12267
dc.identifier.urihttp://hdl.handle.net/11603/22874
dc.languageen
dc.relation.isAvailableAtThe University of Maryland, Baltimore County (UMBC)
dc.relation.ispartofUMBC Computer Science and Electrical Engineering Department Collection
dc.relation.ispartofUMBC Theses and Dissertations Collection
dc.relation.ispartofUMBC Graduate School Collection
dc.relation.ispartofUMBC Student Collection
dc.sourceOriginal File Name: Jenkins_umbc_0434M_12267.pdf
dc.subjectComputer Vision
dc.subjectDomain Adaptation
dc.subjectGrounded Language
dc.subjectNatural Language Processing
dc.subjectRobotics
dc.subjectTransfer Learning
dc.titleTransfer Learning of Grounded Language Models For Use In Robotic Systems
dc.typeText
dcterms.accessRightsAccess limited to the UMBC community. Item may possibly be obtained via Interlibrary Loan thorugh a local library, pending author/copyright holder's permission.
dcterms.accessRightsThis item may be protected under Title 17 of the U.S. Copyright Law. It is made available by UMBC for non-commercial research and education. For permission to publish or reproduce, please see http://aok.lib.umbc.edu/specoll/repro.php or contact Special Collections at speccoll(at)umbc.edu

Files

Original bundle
Now showing 1 - 2 of 2
No Thumbnail Available
Name:
Jenkins_umbc_0434M_12267.pdf
Size:
6.08 MB
Format:
Adobe Portable Document Format
Download
Thumbnail Image
Name:
Jenkins_umbc_0434M_12267.pdf
Size:
6.08 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
Jenkins-Patrick_Open.pdf
Size:
605.85 KB
Format:
Adobe Portable Document Format
Description:
Download

Collections

UMBC Theses and Dissertations
UMBC Computer Science and Electrical Engineering Department
UMBC Graduate School
UMBC Student Collection