John W. Moore - US grants

BARTO Learning in animals involves complex brain processes that integrate sensory information into a coordinated set of actions: It is something that animals do very well, but man-made thinking machines (computers) do rather poorly by comparison. In order to improve computer performance, many scientists and engineers have turned to the brain for theoretical insights into the processes of learning. These insights, when applied to computer technology, have become increasingly important in applications ranging from industrial robotics to process control. Studies of animal behavior, dating from the early years of this century, have provided a rich scientific literature for evaluating theories of learning. These are normally expressed in terms of mathematical relationships between environmental events (stimuli) and actions (responses). State-of-the-art learning theories are known as "real-time computational learning models" because they can readily be translated into computer programs for application to technology. Drs. Moore and Barto are using a well-characterized associative learning paradigm (that is, learning that one event preceeds another event), classical conditioning of the nictitating membrane response in the intact rabbit. This preparation serves them as a laboratory benchmark for evaluating their recently proposed mathematical learning model, and holds much promise both for clafifying brain processes underlying learning, and for enhancing and advancing computer technology. The ultimate goal of this research is to discover how information is processed so efficiently by the brain. These investigators will incorporate this knowledge into a mathematical model, in a way that best describes the relationship between a specific behavior and the brain mechanisms underlying that behavior. This new information can then be applied directly to information acquisition by the latest generation of computer systems.

This project investigates a new approach to learning, planning, and representing knowledge at multiple levels of temporal abstraction. It develops methods by which an artificial reinforcement learning system can model and reason about persistent courses of action and perceive its environment in corresponding terms, and it develops and examines the validity of models of animal behavior related to this approach. The project's objectives are to develop the mathematical theory of the approach, to refine, extend, and conduct validation studies of related models of animal behavior, to examine the theory's relationship to control theory and artificial intelligence, and to demonstrate its effectiveness in a number of simulated learning tasks.

Most current reinforcement learning (RL) research uses a framework in which an agent has to take a sequence of actions paced by a single, fixed time step: actions take one step to complete, and their immediate consequences become available after one step (modeled as a Markov decision process, or MDP). This makes it difficult to learn and plan at different time scales. Some RL research instead uses a generalization of this framework (semi-Markov decision processes, or SMDPs) in which actions take varying amounts of time to complete, and the existing theory specifies how to model the results of these actions and how to plan with them. However, this approach is limited because temporally extended actions are treated as indivisible and unknown units. For the greatest flexibility and best performance, it is necessary to look inside temporally extended actions to examine or modify how they are comprised of lower-level actions, which is not considered in existing approaches.

This project, by contrast, will model extended courses of action as SMDP actions overlaid upon a base MDP. These courses of action, called options, can then be treated as if they were primitive actions; existing RL can be applied almost unchanged. This approach enables options to be analyzed at both the MDP and SMDP levels and introduces new issues at the interface between the levels. This approach is appealing because of its simplicity, similarity to previous approaches using primitive actions, and its solid mathematical foundation in MDP and SMDP theory. This is being developed further into a general approach to hierarchical and multi-time-scale planning and learning.