John W. Moore - US grants

BARTO Learning in animals involves complex brain processes that integrate sensory information into a coordinated set of actions: It is something that animals do very well, but man-made thinking machines (computers) do rather poorly by comparison. In order to improve computer performance, many scientists and engineers have turned to the brain for theoretical insights into the processes of learning. These insights, when applied to computer technology, have become increasingly important in applications ranging from industrial robotics to process control. Studies of animal behavior, dating from the early years of this century, have provided a rich scientific literature for evaluating theories of learning. These are normally expressed in terms of mathematical relationships between environmental events (stimuli) and actions (responses). State-of-the-art learning theories are known as "real-time computational learning models" because they can readily be translated into computer programs for application to technology. Drs. Moore and Barto are using a well-characterized associative learning paradigm (that is, learning that one event preceeds another event), classical conditioning of the nictitating membrane response in the intact rabbit. This preparation serves them as a laboratory benchmark for evaluating their recently proposed mathematical learning model, and holds much promise both for clafifying brain processes underlying learning, and for enhancing and advancing computer technology. The ultimate goal of this research is to discover how information is processed so efficiently by the brain. These investigators will incorporate this knowledge into a mathematical model, in a way that best describes the relationship between a specific behavior and the brain mechanisms underlying that behavior. This new information can then be applied directly to information acquisition by the latest generation of computer systems.

DESCRIPTION (Adapted from applicant's abstract): A central question in the neurobiology of learning and memory is the extent to which single neurons integrate real-time information from multiple sources. Predictions of Sutton and Barto's 1990 Time Derivative (TD) model of Pavlovian learning will be tested in behavioral and single-unit recording studies of the cerebellum, using the conditioned rabbit eyeblink response as a model system. Because its parametric features and neural substrates have been well characterized, classical eyeblink conditioning in humans and other species has been applied to several health related problem-areas, including aging, brain function, development, drug abuse, and toxicology. A growing body of evidence from a variety of methodologies suggests that the cerebellum is essential for learning and performance of conditioned eyeblink responses. The aim is to determine whether the firing patterns of single neurons in the cerebellum are capable of representing the timing and amplitude of classically conditioned eyeblink responses among rabbits trained in paradigms requiring real-time integration of information about the timing and likelihood of the unconditioned stimuli (US). Conditioned responses typically anticipate the timing of the US, but the peak amplitude of the response corresponds to the temporal locus of the US. Training with two conditioned stimulus (CS)-US intervals, which vary randomly from one training trial to the next, a paradigm called temporal uncertainty, produces bimodal response topographies. Because they reflect two expectations of US timing, such bimodal responses express temporal integration. Other paradigms that require this sort of processing include blocking, second-order conditioning, conditioned inhibition, and others. The research plan has three components. (1) behavioral tests of the TD model regarding conditioned response topography; (2) micro-electrode recording from single neurons of the cerebellum after animals have achieved stable (asymptotic) modes of responding; (3) development and evaluation of a cerebellar implementation of the TD model.

This project investigates a new approach to learning, planning, and representing knowledge at multiple levels of temporal abstraction. It develops methods by which an artificial reinforcement learning system can model and reason about persistent courses of action and perceive its environment in corresponding terms, and it develops and examines the validity of models of animal behavior related to this approach. The project's objectives are to develop the mathematical theory of the approach, to refine, extend, and conduct validation studies of related models of animal behavior, to examine the theory's relationship to control theory and artificial intelligence, and to demonstrate its effectiveness in a number of simulated learning tasks.

Most current reinforcement learning (RL) research uses a framework in which an agent has to take a sequence of actions paced by a single, fixed time step: actions take one step to complete, and their immediate consequences become available after one step (modeled as a Markov decision process, or MDP). This makes it difficult to learn and plan at different time scales. Some RL research instead uses a generalization of this framework (semi-Markov decision processes, or SMDPs) in which actions take varying amounts of time to complete, and the existing theory specifies how to model the results of these actions and how to plan with them. However, this approach is limited because temporally extended actions are treated as indivisible and unknown units. For the greatest flexibility and best performance, it is necessary to look inside temporally extended actions to examine or modify how they are comprised of lower-level actions, which is not considered in existing approaches.

This project, by contrast, will model extended courses of action as SMDP actions overlaid upon a base MDP. These courses of action, called options, can then be treated as if they were primitive actions; existing RL can be applied almost unchanged. This approach enables options to be analyzed at both the MDP and SMDP levels and introduces new issues at the interface between the levels. This approach is appealing because of its simplicity, similarity to previous approaches using primitive actions, and its solid mathematical foundation in MDP and SMDP theory. This is being developed further into a general approach to hierarchical and multi-time-scale planning and learning.