Peter R. Killeen - US grants

It is proposed to extend Incentive Theory to account for long-term behavioral contrast and extinction, and to clarify the role of attention in the organism's allocation of behavior to competing activities. The various effects of training regimens on resistance to extinction will be reviewed, and quantitative theory that will account for phenomena such as the partial reinforcement extinction effect, the overtraining extinction effect, and the effects of interpolated schedules of reinforcement will be derived. In all cases the magnitude of the incentives involved and the species of organism studied are likely to play a critical role, and a theoretical explanation of why that is the case will be suggested. The fundamental mechanisms in incentive theory that will be developed are the accumulation of arousal to signals of reinforcement, and the renormalization of that arousal based on the animal's internal clock, whose speed is adjusted on the basis of overall frequency and value of incentives in real time. The same mechanisms should help us to understand both incentive and behavioral contrast, although they are unlikely to be able to account for the types of contrast that are manifest early in training. Notions of behavioral competition, perceptual contrast, and adventitious reinforcement will be evaluated for those effects. The theory invokes attentional processes in its predictions of the acquisition of autoshaping and in selection of alternate patches; current theories of association learning will be studied for their usefulness in extending Incentive Theory; conversely, the theory may shed light on empirical data by generalizing our notions of the maximum associative strength of stimuli to include intertrial interval and scheduling parameters as determinants of that maximum. Incentive Theory provides a very different approach to learning than traditional reinforcement theory, and thus should inform alternate conceptualizations of training programs for regular and special populations, as well as form a theoretical basis for our understanding of the tuning of the organism to the rhythmicity of its environment, either to its benefit or its detriment.

The long-term objective is to develop and test a new theory concerning the control of behaviors that are regularly elicited by reinforcing events in that environment. The elicitation process may be described as a poisson-emitter type of pacemaker-accumulator system, one whose rate constant is proportional to the rate of reinforcement in the environment. This conceptualization, or variants of it (e.g., normal approximations, or recurrent processes with error in the accumulator as well as the pacemaker), permits specific predictions to be made in numerous conditioning situations. The assumption of proportionality between the rate of the internal clock (pacemaker) and rate of reinforcement is a rad- ical one. It has implications for mental health issues in that it entails a fundamental reorientation of our notions about how reinforcement and punishment controls behavior. For instance, the delay of reinforcement gradient will not be a constant, but will become steeper in richer environments. We will evaluate the theory both against extant data, and against numerous proposed experi- ments. These address the issues of the mediating role of adjunctive behavior in the estimation of time, of the variation in the speed of the pacemaker with the richness of the environment, of the context over which the speed of the pacemaker is normalized, and of alternate mechanisms that may mediate the fine-scale judgments of time. The experiments will typically be psychophysical conditioning experiments with non-human subjects (rats and pigeons), and will serve both to test specific predictions, and to evaluate tentative extensions of the theory to behavioral and incentive contrast, rate laws, and choice of delayed reinforcers.

IBN-9408022 Killeen Abstract It is indisputable that rewards and punishments change behavior, but less certain is just what behavior they affect; just how is "credit" (or blame) allocated to the stream of prior behavior? Dr. Killeen posits that in order to be reinforceable, a behavior (response) must be in short-term memory (STM) along with the incentive that rewards it. Because this memory fades over time, more than just the most recent response is reinforceable; the effects of a reward impinge on the previous stream of behavior, with decreasing efficacy as the response-memories age. Dr. Killeen's research will define the shape and extent of this response STM by a "resonance" technique in which the correct characterization of the memory window is signaled by a maximum in learning rate. There are many implications of this theory and its perspective: It changes our understanding of how reinforcement schedules affect behavior, provides a new model for temporal effects in classical conditioning, reopens questions of chunking and automatization, clarifies how incentives may at the same time reinforce a response and erase memory of that response, and provides a new approach to the classic problem of response units. The results of the research should help to significantly improve training techniques; provide independent indices of memory, motivation, and response facility; and reduce the plethora of scheduling techniques to a concise and predictive mathematical system.

This ADAMHA RSA will provide the PI an opportunity to develop a model of the reinforcement process into a general theory of performance based on principles of arousal, temporal constraint, and coupling. It will give the PI the time necessary to acquire the theoretical tools required to develop the core model, which treats reinforcement as the coupling of these responses and incentives that fall within the window of an organism's short-term memory . That hypothesis is validated by tuning experiments in which the experimenter's definition of the response is varied until it is maximally effective in controlling behavior, thus providing a template of the animal's definition. Under this RSA the theory will be developed in a number of interlinked ways: 1) Validate the model of arousal in terms of extent literature on conditioning, extinction, and the partial- reinforcement extinction effect. Develop it for conditions of varying motivation, such as those occurring in closed economies and contexts where warm-up and satiation play significant roles. 2) Generate explicit performance rules relating arousal level to response rate, and to response latency and probability in discrete trials. 3) Derive mathematical models for contingency in the context of: 3a) Concurrent responses; 3b) Classical conditioning; 3c) Heterogeneous response classes; and 3d) Sequences of incentives of arbitrary number and spacing. 4) Replicate the tuning experiments with visual and auditory stimuli, to provide a bridge to the ubiquitous literature on short-term memory for stimuli. It will also permit him to 5) Continue to develop and begin to integrate theories of timing with this more general theory; 6) Extend the theory to forthcoming incentives, so as to provide a rationale for the steep and crossing discount functions that underlie all models of self control; 7) Embed the theory in a larger context, akin to mechanics, in which stimuli, responses, incentives, and time are represented in a multidimensional state-space. The award will make it possible for the candidate's skills to develop in pace with empirical results and to drive further research. It is consistent with the mission of ADAMHA as it promises a more accurate picture of how incentives interact with responses to generate habits. The leading causes of death and disability are behavioral; existing theory is near-impotent in helping us to consistently avoid rich foods, drugs, alcohol, and violence; it is not much better in promoting medical compliance, seat-belt usage, exercise, and safe sex. The proposed research provides novel and effective reconceptualizations of the role of memory in the reinforcement process, and of their interaction with motivation. It can help us learn how to more effectively couple healthful behaviors to their consequences. The PI's approach to theory development and deployment evolved in collaboration with Dr. Hestenes; their plan to review the psychologists' way of knowing through modelling will provide a powerful template for students; way of coming to know their discipline. The value of this RSA accrues both to the PI in terms of the new quantitative skills he will acquire, and to the field in the new interpretation of memory and performance he offers.

DESCRIPTION (provided by applicant): These experiments and correlated theoretical development will measure changes in the probability of responding over thousands of trials using continual conditioning and extinction paradigms (autoshaping and negative automaintenance). These paradigms pit Pavlovian conditioning of approach against instrumental contingencies that discourage approach. Trial durations and the first and second moments of the intertrial interval and probability of reinforcement will be varied. The relative likelihood of existing models of conditioning and extinction will be compared in light of the data. These differ from traditional experiments in that the degrees of freedom in the data will vastly outnumber the degrees of freedom in the models. Other experiments will vary attention to the CS and background using reinforcement contingencies, to bridge a theory of schedule control to associative conditioning. In other experiments the context will be localized and presented in trace and delay paradigms to test a model of blocking-by and protecting-of context. A general theory of the control of target responses by concurrent and forthcoming incentives will be developed to unify the findings. This proposal will: (a) Provide a new test bed for models of associative conditioning; (b) develop analytic techniques to extract orders of magnitude more information from select experiments; (c) provide techniques for analysis of nonlinear dynamic conditioning paradigms, including spectral analyses and Hearst exponents; (d) address why the context can be more powerful than discrete CSs; and (e) develop a theory for the attraction to signs, and the handing-off of this attraction to their significates. Health implications: If people naturally behaved well, health promotion would be mundane. People do not behave well because they find the attraction to food, drugs, risk and aggression more powerful than attraction to the benefits of abstinence. This proposal is predicated on a theory of approach to attractors and signs of them. It provides new measurement tools for the vacillation in motivations over epochs of abstinence and relapse. It embeds the theory of conditioning in a real-time, closed-loop context, and develops requisite analytic tools. The pigeon's attraction to a response that has proven self-defeating a thousand times is perhaps not different in kind than an addict's craving and relapse. By providing statistically powerful tests of competing models of underlying behavioral machinery, we may get closer to understanding how to mitigate such fatal obsessions.

Abstract
When we judge the magnitude of a stimulus-the brightness of a light, heaviness of a weight, or duration of a sound-we are mapping that aspect onto another dimension, typically a numerical one. Even when comparing two stimuli directly, that comparison may be mediated by a neural representation akin to counting. A counter is a neural device that adds sensory action potentials, or, in the case of time, pacemaker pulses. Inaccuracies reveal the structure of the counter: Proportional error in sensory discriminations - Weber's law or scalar timing - is the most general laws of psychophysics; All models of counters must yield it. Minuscule deviations from it and from linear timing further strongly constrain candidates. Models of failure modes of counters will be developed and tested. Examples are: Failing to set a higher bank (e.g., 10's) when a lower one resets (the Y2K glitch); failing to reset; conditioning of weights in other than a power series. These will be compared to data collected on the discriminative and productive abilities of rats and pigeons, in particular: (1) mean performances in discriminating and producing temporal intervals; (2) Weber functions; (3) count distributions and hitting-times. Alternate weighting schemes for the accumulator that arise from conditioning will be studied. These schematic models will be integrated with realistic neural networks. The likelihood of this theory will compared with those of other current timing theories given these data. Results will be applied to information transmission and signal detection theories.