Using Web Images & Natural Language for Object Localization in a Robotics Environment

Abstract

The ability for humans to interact with robots via language would allow for more natural interactions between robots and humans. To this end, in this work I introduce a novel approach that allows robots to localize objects from an unbounded set of classes given only a description of a target object. The first part of this work is a performance analysis of current state of the art object detectors and a region proposal approach \cite{UijlingsIJCV2013} on the Autonomous Robot Indoor Dataset \cite{arid}. The second part of this work introduces a three stage natural language guided webly object localization approach and associated experiments to evaluate its performance. The first stage of the approach generates a webly dataset without any manual curation from a human description of the target object. The second stage of the approach uses the webly dataset to train a binary classifier for the target object. Finally, region proposals from selective search \cite{UijlingsIJCV2013} are input to the webly supervised binary classifier and the region proposal with the highest confidence score is returned as the prediction.