Browsing by Subject "probabilistic planning"
Now showing 1 - 1 of 1
Results Per Page
Sort Options
Item Abstract Decision Making and Concept Formation for Adaptability and Generalization(2019-01-01) Winder, John; desJardins, Marie; Matuszek, Cynthia; Computer Science and Electrical Engineering; Computer ScienceGeneralization remains a central challenge for machine learning algorithms, especially when embodied in artificially intelligent agents that learn and plan under uncertainty. By using reinforcement learning (RL) or probabilistic planning techniques, such agents may be trained successfully to excel at solving a specific, narrow task. Upon transfer to a different environment, however, where they face novelty in the form of new goals or unusual surroundings, their lack of an ability to adapt is most clearly highlighted by degraded performance. In contrast, humans possess a facility for adaptation. We create and recall concepts that enable us to interpret any anomalies we encounter. Likewise, we develop and repeat habits that help us navigate our life, allowing us to think further into the future by alleviating the burden of contemplating all the details of tasks we tackle in a common day. In this thesis, I aim to make agents more adaptable by developing new methods for reasoning abstractly. Through a process of concept formation, agents expand their understanding of entities in the world such that any anomalies may be interpreted based on their conceptual relation to what has already been learned. I develop an algorithm for concept-aware feature extraction, such that agents maintain a conceptual knowledge base that grows to accommodate new concepts. Exploring the application of this approach to two decision-making paradigms---contextual bandits and temporal difference RL---I demonstrate how explicitly reasoning about concepts makes agents adapt more readily when facing a stream of anomalous objects or upon transfer to harder tasks. For habits, I articulate how decision-making agents may assemble useful patterns of behavior into formal structures called subtasks, which aid an agent's ability to reason abstractly, over varying timescales. Subtasks, thus, facilitate creating and reusing solutions to common problems. I build upon two separate formulations of subtasks: the options framework (a standard approach to hierarchical RL) and abstract Markov decision processes (AMDPs). I develop new algorithms to investigate how abstract option models may be approximated efficiently from experience, how abstract option policies may be adapted to novel tasks, and how hierarchies of AMDPs let agents plan more flexibly and effectively at varying levels of abstraction. Finally, I combine these ideas to make a new model-based RL algorithm for planning with abstract, learned models: an agent creates AMDP subtasks bottom-up from data and learns to plan with them top-down, using the hierarchy it generated to generalize and adapt to variant tasks.