John E R Staddon - US grants

NONTECHNICAL SUMMARY PI: J. E. R. Staddon, IBN 94-20646, Reinforcement Mechanisms One of the most basic, enduring goals of behavioral science is to understand how reward and punishment affect behavior and what these effects tell us about the basic mechanisms of learning. In this research project, Dr. Staddon will investigate three aspects of this problem: the effects of reward on choice, the processes by which patterns of reward affect behavior, and the processes that produce novel or creative behavior. The studies of choice will address the fact that choices are affected by both recent and remote events as well as by correlations between responses and rewards. Experiments will be carried out to disentangle these three variables and their effects on choice responses. Organisms also react immediately and proportionately to sudden changes in the time interval between rewards. These fast-acting processes allow organisms to track the intervals between successive rewards and react immediately to changes in the temporal patterning of rewards. A series of experiments will examine the dynamics of these timing effects. An act must occur before it can be modified by reward or punishment. This raises the question, How do novel behaviors occur? A third series of experiments will use the vocal behavior of budgerigars to study is process. Previous research has shown that budgerigars can easily learn to emit specific vocal names in the presence of different visual stimuli and will then spontaneously emit the name of a to-be-chosen stimulus during a choice task even if this is not required by the experimental procedures. By selectively rewarding rare vocalizations, it has been possible to expand this repertoire. A series of experiments will be carried out to explore the role of this novel vocal behavior in the learning of complex tasks. This research should help us understand the processes that produce new behaviors and may further our un derstanding of the role of speech in cognition and learning.

This award supports Professor John E. R. Staddon of Duke University to collaborate in behavioral research with Professor Juan D. Delius of the Department of Social Sciences of the University of Konstanz, Federal Republic of Germany. They are interested in the "transitive inference" paradigm, a form of serial learning task in which subjects are given information about the relative values of stimuli presented in pairs, and then asked to infer the relation between novel pairs. In their research, they plan to investigate the mechanisms used by several different species to solve this task. They are considering a number of theories ("associative linkage", "rule stack", and "inference engine") that could account for successful solution of the serial learning problem. In addition, Dr. Staddon has developed a novel theory ("value transfer") that seems capable of describing many of the experimental results in Dr. Delius' laboratory, as well as others in the literature. They will collaborate in a set of new experiments designed to compare the four theories, then focus further experimental effort on the details of the most successful theory. Dr. Staddon has an outstanding reputation for formulating and experimentally testing formal models that explain complex adaptive behaviors of animals in terms of simple rules. His contributions to the collaborative research will be theoretical expertise, experimental design and some experimen- tal work. Dr. Delius is internationally recognized for his studies of comparative cognition. His laboratory and research team are excellently equipped to train pigeons in complex tasks in ecologically valid environments. Most of the experimental work will be carried out there. Recent research by these investigators and others suggests that animals are able to solve tasks traditionally held to require deductive logic. These results raise important questions about the phylogenetic distribution of reasoning and its underlying mechanisms. Drs. Staddon and Delius do not assume that success in serial learning tasks is positive proof of an ability to reason logically using cognitive representations. Instead, they propose to examine four different mechanisms that could account for success in the transitive inference task without recourse to symbolic reasoning.

All animals show some form of learning, but the range, sublety and pervasiveness of human learning is the main thing that sets human beings apart from other species. How does the nervous system work to produce learning? There are two ways to approach this problem: bottom up, and top down. The bottom-up approach begins with the properties of individual neurons and synapses and aims to understand complex learning through exhaustive analysis of simple neural circuits and simple learning processes. This approach is being applied very successfully to elementary learning in several invertebrate species, but it is still a long way from the complexities of learning in higher animals. A complementary alternative is the top down approach, which underlies the work here proposed. The top-down method is to begin with careful behavioral analysis of apparently complex tasks that can nevertheless be reduced to simple performance rules. Theoretical analysis can often suggest formal real-time models that behave in the ways described by a given performance rule. A dynamic model of this sort that survives rigorous behavioral testing is likely to reflect enduring and measurable properties of the underlying neural machinery. We have found a very simple class of performance rules, the value-transfer hypothesis, that may underlie animals' ability to solve a "reasoning" task, transitive inference. We have also proposed a very simple dynamic model for the assignment-of-credit problem in operant conditioning, that is, the process by which a response is selected by consequential reinforcement. And most recently we have discovered a simple way to produce sequence learning in a recurrent "neural" network. The proposal describes additional experimental tests of the value-transfer hypothesis, and theoretical explorations of learning systems built out of the assignment- of-credit model and different forms of neural network. Our immediate objective is to arrive at a dynamic model for the transitive-inference task and for a range of other discrimination-learning tasks. Our ultimate objective is to use these models as guides to understanding the role of the nervous system in learning.

All animals show some form of learning, but the range, subtlety and pervasiveness of human learning is the main thing that sets human beings apart from other species. There are two ways to approach the learning problem: bottom up, and top down. The top-down method is to begin with careful behavioral analysis of apparently complex tasks that can nevertheless be reduced to simple performance rules. Theoretical analysis can often suggest formal real-time models that behave in the ways described by a given performance rule. A dynamic model of this sort that survives rigorous behavioral testing is likely to reflect enduring and measurable properties of the underlying neural machinery. This is an application for fellowship (RSA) support for the Principal Investigator. This PI's research comprises two linked projects that use the top-down approach to understand the mechanisms of learning. One project, Models for Learning, Memory and Inference is primarily theoretical and collaborative; the other project, Reinforcement Mechanisms is primarily experimental. The theoretical project has two parts: the study of general properties of operant learning, such as contingency and contiguity, serial order and simple associativity; and complex tasks, such as nonverbal inference. We have developed real-time models for the assignment-of-credit problem in operant learning for a simple kind of nonverbal inference and for different kinds of serial order in behavior. The experimental work uses animal subjects to study the role of dynamic memory processes in time discrimination and choice. A key feature of the research strategy is continual interplay between experiment and real-time models of the processes under study. Our immediate objective is to arrive at dynamic models for specific experimental situations. We then attempt to extend and combine these models to embrace as wide as possible a range of related tasks. Our ultimate objective is to use real-time models as guides to understanding the role of the nervous system in learning. The most intractable and disturbing human behavior disorders involve the systems for memory and learning. The proposed work is intended to increase our basic understanding of how these systems work, hence aid in the search for means to eliminate or alleviate memory and learning disorders.

All animals show some form of learning, but the range, subtlety and pervasiveness of human learning is the main thing that sets human beings apart from other species. There are two ways to approach the learning problem: bottom up, and top down. The top-down method is to begin with careful behavioral analysis of apparently complex tasks that can nevertheless be reduced to simple performance rules. Theoretical analysis can often suggest formal real-time models that behave in the ways described by a given performance rule. A dynamic model of this sort that survives rigorous behavioral testing is likely to reflect enduring and measurable properties of the underlying neural machinery. This is an application for fellowship (RSA) support for the Principal Investigator. This PI's research comprises two linked projects that use the top-down approach to understand the mechanisms of learning. One project, Models for Learning, Memory and Inference is primarily theoretical and collaborative; the other project, Reinforcement Mechanisms is primarily experimental. The theoretical project has two parts: the study of general properties of operant learning, such as contingency and contiguity, serial order and simple associativity; and complex tasks, such as nonverbal inference. We have developed real-time models for the assignment-of-credit problem in operant learning for a simple kind of nonverbal inference and for different kinds of serial order in behavior. The experimental work uses animal subjects to study the role of dynamic memory processes in time discrimination and choice. A key feature of the research strategy is continual interplay between experiment and real-time models of the processes under study. Our immediate objective is to arrive at dynamic models for specific experimental situations. We then attempt to extend and combine these models to embrace as wide as possible a range of related tasks. Our ultimate objective is to use real-time models as guides to understanding the role of the nervous system in learning. The most intractable and disturbing human behavior disorders involve the systems for memory and learning. The proposed work is intended to increase our basic understanding of how these systems work, hence aid in the search for means to eliminate or alleviate memory and learning disorders.

DESCRIPTION (Adapted from applicant's abstract): The most intractable and disturbing human behavior disorders involve the systems for memory and learning. The proposed work is intended to increase the basic understanding of how learning systems work, hence aid the search for ways to cure these afflictions. This proposal is to continue top-down theoretical work on basic learning dynamics. The method is to explore very simple dynamic models that account for a wide range of both steady-state and transient behavioral data. Currently non-associative processes are emphasized, as a necessary preliminary to understanding associative learning. Work is proposed in three areas: rate-sensitive effects, specifically habituation and feeding regulation; recurrent choice, a much-studied area that embraces most operant behavior; and, diffusion models for generalization and spatial orientation. Work in all three areas is based on elementary dynamic models that have already successfully simulated a substantial range of experimental results. The habituation work aims to understand the dynamics of habituation in simple systems. Are the apparent similarities between habituation in different species only at the level of the empirical phenomenon, or do they extend to dynamics? If the habituation process is similar in different species, what common properties of these different nervous systems underlie this similarity? If there are differences between the dynamic models needed to account for habituation in different species, what are the corresponding neural differences? Are habituation-like processes involved in feeding dynamics and in "higher" learning (operant and classical conditioning) and, if so, how? Is rate-sensitivity involved in recurrent choice behavior. Many experimental results suggest that it is, but no existing models take account of it. How can assignment-of-credit (response selection) be incorporated into models for choice? The work on diffusion arose from study of the dynamics of stimulus generalization, a basic property of all associative learning. A simple diffusion process developed for generalization dynamics can also solve a number of classical "spatial-insight" problems. It will be seen how far this simple idea can be extended to spatial and temporal learning. The ultimate aim is to combine these elementary processes into increasingly comprehensive, associative models whose properties can be compared to the structure and function of the neural systems that bring about learning.

DESCRIPTION: (Applicant's Abstract) This is a collaborative proposal to extend and refine a simple dynamical model of adaptive and maladaptive motivated behavior (feeding, drug addiction). There are strong linkages at several levels between feeding regulation, habituation, and food-reinforced operant learning. There are also suggestive resemblances between the regulatory features of feeding and drug addiction. We have shown that very simple multiple-time-scale (MTS) models can simulate the major dynamic properties of feeding regulation, simple habituation and some aspects of operant choice behavior. We propose to extend and test a family of MTS models with data sets from all three areas. Our aim is to use insights gained from the study of feeding regulation and habituation to expand our understanding of the dynamics of addiction. We contend that a thorough under-standing of dynamics at the level of behavioral data is an essential foundation for advances in our knowledge of the neurophysiological and neuropharmacological bases of addiction and motivated behavior generally. This proposal is relevant to following basic behavioral sciences topics under the RFA: learning and memory, animal learning and behavior, and the motivational and learning processes underlying craving.

This is a proposal for renewal of a Senior Scientist Award. Both theoretical and experimental research is proposed on several topics in learning and motivation: (a) Experimental and theoretical studies of recurrent choice; (b) complex learning, transitive inference in animals and the emergence of novel behavior and associations; (c) sequence learning and models for complex discrimination; (d) time discrimination; experimental and theoretical studies of transient and steady-state phenomena; (e) multiple-time scale models for behavior; habituation and reinforcement effects; (g) feeding dynamics; theoretical research of dynamic mechanisms for feeding and foraging; (h) motivational dynamics, multiple time scale processes and drug addiction. The experimental work uses animal subjects to study the role of dynamic processes in learning and motivated behavior. Our research strategy emphasises continual interplay between experimental and real-time models. Our first objective is refine models for specific experimental situations; we then attempt to extend and combine these models to embrace as wide as possible a range of related tasks. Our ultimate objective is to use real-time models as guides to understanding the role of the nervous system in leaning. The most intractable and disturbing human behavior disorders involve the systems for memory and learning. The proposed work is intended to increase our basic understanding of how these systems work , hence aid in the search for means to eliminate or alleviate memory and learning disorders.

DESCRIPTION (provided by applicant): All behavior involves choice and many social and health problems arise from bad or self-destructive choices. Experimental research on free-operant choice behavior over the past 40 years is based on the idea that response rate or choice probability is determined by the value of outcomes. The simplest case is concurrent schedules where the organism is confronted with a free choice between two or more response alternatives and choices are reinforced directly with primary reinforcement. When the reinforcement schedules are variable-interval (VI), the usual result is the so-called matching law: relative response rates match relative reinforcement rates. A more complex case is when the choices are reinforced not with food but by the appearance of a second stimulus in the presence of which food is delivered according to another schedule (concurrent chain schedules). These experiments are usually thought of as studies of conditioned or secondary reinforcement - and results do not fit the matching law. Several alternatives have been proposed, but all of them use response rate in the choice link as dependent variable, and some measure of 2nd - link value as independent variable. Almost none of these experiments or theories has addressed the problem of fixed- (as opposed to variable-) interval choice links nor do they deal with dynamics or explain behavior in non-choice chain experiments. Yet there must be common principles underlying these varied performances. The research for which support is requested in this proposal began with a longstanding puzzle on simple chain schedules: unstable behavior, even the complete cessation of responding, in early links of a multi-link chain, even when individual links are relatively short. This effect seemed to us to be explainable, in general if not in detail, by a process we had studied for many years called linear waiting: that organisms on interval schedules will wait before responding for a time proportional to the just-experienced time to reinforcement (TTR). This led us to conjecture that perhaps behavior on concurrent chain schedules is mainly controlled not by the value of outcomes, i.e., by conditioned reinforcement, buy simply by TTR, as we had found for a simple interval and chain schedules. A reviewer suggested that this idea is unlikely to apply to simple concurrent VI-VI schedules, where the TTR may be very short and waiting time (WT) is comparably short. But in this revised proposal we present striking new data showing unequivocally that relative waiting time on concurrent chain schedules, and even on simple concurrent VI-VI schedules, is highly correlated with measures of preference. Moreover, WT seems to behave in a similar way on all interval schedules and thus offers the possibility of a truly general approach to choice behavior that incorporates simple as well as concurrent and fixed- as well as variable-interval schedules. We also present new data on a version of specialized chain schedule known as the time-lift (T-L) procedure, which has played a key role in promoting a representational view of interval timing as a linear-scaled process. The T- L procedure has been criticized recently on both theoretical and experimental grounds. Our new experiment shows that the TTR idea seems to apply here also and the T-L experiment indeed can not (as originally claimed) prove that "subjective time" is or is not linear. We propose a total of 12 experiments to explore the further implications of these findings for the dynamics of behavior on simple and concurrent interval schedules of reinforcement. We believe that these studies may lead to an entirely new unified approach to choice.