Use Defines Possibilities: Reasoning about Object Function to Interpret and Execute Robot Instructions
Loading...
Files
Links to Files
Permanent Link
Author/Creator ORCID
Date
2023
Type of Work
Department
Program
Citation of Original Publication
Rights
This work was written as part of one of the author's official duties as an Employee of the United States Government and is therefore a work of the United States Government. In accordance with 17 U.S.C. 105, no copyright protection is available for such works under U.S. Law.
Public Domain Mark 1.0
Public Domain Mark 1.0
Subjects
Abstract
Language models have shown great promise
in common-sense related tasks. However, it
remains unseen how they would perform in
the context of physically situated human-robot
interactions, particularly in disaster-relief scenarios. In this paper, we develop a language
model evaluation dataset with more than 800
cloze sentences, written to probe for the function of over 200 objects. The sentences are
divided into two tasks: an “easy” task where
the language model has to choose between vocabulary with different functions (Task 1), and
a “challenge” where it has to choose between
vocabulary with the same function, yet only
one vocabulary item is appropriate given real
world constraints on functionality (Task 2). DistilBERT performs with about 80% accuracy
for both tasks. To investigate how annotator
variability affected those results, we developed
a follow-on experiment where we compared
our original results with wrong answers chosen
based on embedding vector distances. Those
results showed increased precision across documents but a 15% decrease in accuracy. We conclude that language models do have a strong
knowledge basis for object reasoning, but will
require creative fine-tuning strategies in order
to be successfully deployed.