Richard M. Shiffrin - US grants

This research is designed to elucidate the fundamental mechanisms of human information processing and retrieval. The empirical research is supplemented by quantitative models, generally stated in the form of computer simulations. The research is directed toward a wide variety of areas in the general domain of memory, retrieval, and forgetting. We shall explore: 1) The cue-dependent nature of retrieval, with particular emphasis upon the ways in which cues are combined when probing long-term memory; 2) The limitations upon capacity when retrieving, with emphasis upon the role of attention to cues; 3) The process of search in recall and the effects upon search of strategies; 4) The ways in which memory models can predict both accuracy data and reaction time data simultaneously; 5) Whether a single model can explain the results from recognition studies, category learning studies, and list-discrimination studies; 6) How recall and recognition of sentences can be related to recall and recognition of the words and elements making up those sentences; 7) Whether a retrieval model can explain the traditional findings in learning of items and lists; 8) Whether retrieval models can explain the traditional findings of "interference" encountered in learning of multiple lists. A special effort will be made in the proposed project to tie all of these areas together, both empirically and theoretically. We shall use as a starting point the SAM model that has already proved successful in predicting a great deal of data in the domains of recall and recognition, but shall also explore the relationships of the SAM model to other models. Our goal is the development of a comprehensive yet simple and consistent quantitative theory for the basic mechanisms of human information processing, learning, memory, retrieval and forgetting.

DESCRIPTION (ADAPTED FROM THE APPLICANT'S ABSTRACT): This proposal explores several fundamental issues concerning memory storage and retrieval. The primary goal is the development of a theory capable of quantitative predictions of the major phenomena in the field and the results from proposed studies in three main research areas: 1) The representation of information in memory. Is memory storage composite, as in many neural net and connectionist models, or separate, as traditionally assumed in the investigator's theory known as SAM (or Search of Associate Memory)? How are different levels of information represented together, such as features, words and sentences? 2) The role of context in storage and retrieval. When does it change during storage, and how is it chosen at test? To what degree does context change underlie forgetting? 3) The time course of retrieval and response times. What factors affect retrieval time? What model explains the time to respond? How are response times related in recognition, cued recall, and free recall? How can accuracy and response time data be modeled together? The investigator hopes to create a general model capable of predicting such phenomena.

The Cognitive and Information Sciences (CIS) program offers a model undergraduate curriculum designed to expose students to the study of intelligent systems. It is intended as a model for the development of similar programs at other institutions as they begin to move into this emerging field. The core of the newly proposed major concerns formal theories of mind and information that integrate computer science, psychology, philosophy, neuroscience, and mathematics. The curriculum gives students training in the basic skills of this new field (writing, mathematics, experimentation, and computer programming), but also fosters the development of expertise in particular sub-fields such as human cognition, learning and instruction, logic, and computation. Newly designed courses will expose students to both basic scientific theories and to applications of these theories to education, system design, and database retrieval. The program will be assessed internally by pre/post exams within courses and exit interviews upon graduation, and externally by quarterly site visits by expert consultants and a comprehensive third-year review by consultants from Indiana University's School of Education. The tangible products of this proposal will include general information on the CIS degree and its requirements and materials for specific courses in the form of syllabi, lecture notes, slides, bibliographies, assignments, and exams. Most important will be the course software and laboratory software developed for all four of the core CIS courses, and several others; this courseware will emphasize interactive, hands on learning through doing, and will be in a form easily usable by other programs that are beginning or being planned in this area. These products will be disseminated through our existing technical report series and newsletter, through electronic media, web sites, and contacts at other universities, and will be provided upon request of other institutions starting such programs.

[unreadable] DESCRIPTION (provided by applicant): The relation of perception to memory is a fundamental characteristic of human cognition, each affecting and determining the other. This relation is explored empirically with a task known as priming: The primes are visually presented words or objects, or aurally presented words. In many situations the primes are irrelevant to the main task, yet nonetheless facilitate responding. In short-term priming, the primes are presented just prior to a trial, and in long-term priming many minutes prior, often in another task. The main task can be word naming, deciding whether the stimulus is a word, or matching of a test item to one or more choices presented after the trial. The ways in which the primes affect performance provide a great deal of information about the interaction of perception and memory, and about the way our knowledge is formed and modified. Models of perception, memory, and decision, have been and are being developed and tested for short-term priming (ROUSE) and long-term priming (REMI). Particular issues being explored are spatial and temporal confusions and the way the system discounts evidence for features to prevent the harmful effects of these confusions. The present project involves empirical research on a variety of these topics, coupled with theory development, to explain the results and to predict new findings. Particular emphasis in the present project is given to analysis and test of stages of perceptual processing, empirical and theoretical development of models that predict both accuracy and response time together, perceptual inference as a decision making principle, the ways that new information is added to our knowledge, and the links between episodic memory and the permanent knowledge store. The developments are aimed to contribute toward a relatively complete theory of perception, memory, and decision. [unreadable] [unreadable]

DESCRIPTION (provided by applicant): The project attempts to produce a theory of storage in and retrieval from human memory, using a Bayesian-based modeling approach (termed REM), rooted in principles of optimized processes operating in a cognitive environment constrained by capacity limitations. The approach will involve modeling of data collected from a wide variety of tasks emphasizing different domains of cognition; in each testing of recognition and recall using free response and signal-to-respond will be the normative experimental method. The domains include: 1) Episodic memory (recognition, associative recognition, cued recall, free recall), emphasizing the role of context in storage and retrieval. 2) Knowledge retrieval. 3) Implicit memory and long-term priming. These latter two domains will include studies of naming, perceptual identification, lexical decision, production of associates, animacy judgments, stem- and fragment-completion. 3) Short-term priming (negative priming). 4) Perceptual benefits due to priming (both long- and short-term). 5)Visual search (to establish one key capacity limitation and its nature). To develop the theory, we will explore response time and accuracy (and their relation) as measures of underlying processing mechanisms. We shall also develop means to replace the currently abstract features of the REM modeling framework with 'real' features obtained through new forms of data analysis.

DESCRIPTION (provided by applicant): The relation of perception to memory is a fundamental characteristic of human cognition, each affecting and determining the other. This relation is explored empirically with a task known as short-term priming: Words (termed 'primes') are presented just prior to a trial, but are irrelevant to the main task. Then a target word is presented briefly and masked, and followed by two choice words, one of which had been the target. Performance is strongly affected when one or more primes are related to one or more of the choice words. Our research has been able to separate these priming effects into those due to 'preference' and those due to 'perception': Preference effects are indicated by the fact that the direction of performance change is determined by the way the primes are processed. The findings have recently been fit by a quantitative model called ROUSE that assumes: 1) Features of the primes are sometimes confused with those from the target flash. 2) The decision system matches the detected features to the choice word features. 3) The evidence from matching and mismatching features is discounted almost optimally when a choice word feature had also been in a prime word. This model fits many studies remarkably well, and suggests the priming task is a good model system for further investigations of the relation of memory to perception. The present project explores many aspects of this relation, including decision strategies and their determining factors, the nature of source confusion for features, the ways in which priming improves perception, the relation of accuracy to response time, and the relation to other attention and perception tasks. The data will serve as a base for further development of the ROUSE model for priming, and eventually a more complete model of perception,

Collaborative Research: Modeling Perception and Memory: Studies in Priming

David E. Huber, University of California-San Diego
Richard M. Shiffrin, Indiana University

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).

It is said that "seeing is believing", and we take it for granted that vision operates efficiently and accurately. This suggests that vision is easy. However, failed attempts at producing computer vision demonstrate exactly the opposite--vision is perhaps the most difficult operation performed by the brain, requiring one third of the neocortex. The NSF-funded research project being conducted by David Huber at the University of California, San Diego and Richard Shiffrin at Indiana University focuses on an important question in visual perception: How is it that we can keep separate what we are currently viewing from that which came immediately before? In truth, vision is constantly "blurring" together information over time, such as when viewing the smooth motion at the cinema that is produced by a sequence of still images shown in rapid succession. However, while reading, our eyes constantly move from one word to the next, and yet unlike a movie, we see each word separately and do not confuse it with the previous words. To accomplish this, the brain must have a trick for deciding when the previous image should be combined with the next image and when each should be kept separate. Huber and Shiffrin hypothesize that the process of identifying each word or each movie image causes it to be suppressed so as to reduce inappropriate blending with the next word or image. In the case of a movie, the images appear too briefly, and the blending produces apparent movement. In the case of reading, our eyes dwell on each word exactly the right amount of time to fully identify and suppress each word so as to reduce confusion with the next word. Huber and Shiffrin investigate this ability to separate visual images in a variety of tasks, including reading, face identification, and rapid detection of change, to name just a few examples. If their hypothesis is correct, manipulating the timing of stimuli should produce analogous behavioral effects in all of these situations. Beyond laboratory studies, this hypothesis may also improve computer vision systems in situations requiring rapid identification. For instance, computer controlled cameras at the airport might be used to identify faces of suspects, but this requires separating one face from another when there is a crowd of faces moving quickly past the camera. The results of this research may also be relevant to disorders such as autism, schizophrenia, and dyslexia, which often involve a component of distorted or abnormal perception. For instance, one account of dyslexia suggests that reading difficulties arise from an inappropriate blending of letters and words. Understanding the manner in which the brain separates visual information over time may help with the diagnosis, interpretation, and treatment of these perceptual deficits.

The human perceptual system receives a constant stream of continually changing information. For example, the eyes move several times each second, providing different views of different objects or words. This project investigates the dynamic process of separating in time and space information pertaining to previous sources (e.g., a previously viewed word) from information pertaining to the current source (e.g., the currently viewed word). Behavioral studies will address the process of discounting that serves to reduce perceptual separation errors due to source confusion. This discounting process can be understood at multiple levels of description and the proposed experiments test complimentary and related mathematical models at the causal and neural levels of analysis. Two causal models use Bayesian statistical techniques and focus on optimizing perception in a noisy world perceived with a limited capacity processing system; discounting is implemented as "explaining away" between competing sources. The neural model implements discounting through habituation that arises with the transient depletion of synaptic resources. In combination, these models demonstrate why perceptual discounting exists and the particular manner in which it is implemented. A wide variety of experimental paradigms involve the rapid presentation of visual objects and the proposed studies use these models to investigate whether perceptual source confusion and discounting may provide a unified account of these phenomena. Besides visual short-term priming with words, the proposed studies examine the popular perceptual and cognitive paradigms of repetition blindness, flanker effects, the attentional blink, negative priming, semantic satiation, and affective priming. All of these paradigms involve presenting a picture, word, or symbol on a computer screen followed by a second presentation that is either identical, positively related, or negatively related to the first presentation. An important goal of this endeavor is to provide a unified account of these perceptual phenomena that are currently considered in isolation by researchers.

A conference titled "Drawing Causal Inference from Big Data" will be held March 26 and 27, 2015, in the National Academy of Sciences auditorium in Washington DC. The purpose of this conference is to present state-of-the-art approaches to the problem, and to bring together leading experts, both the featured speakers and other experts, who will generate progress through their interactions. In many respects the subject of this conference is in its infancy because the many methods that have been developed and used for causal inference in small data do not scale up, because Big Data is often collected in the field in uncontrolled fashion, and because of the sheer size of the data that, contrary to popular belief, make it more rather than less difficult to identify causal effects. The problems in dealing with Big Data are in good part rooted in the limitations of human cognition, so ongoing efforts are aimed at the development of computational algorithms. However it is likely that computational techniques are best viewed as augmenting rather than replacing human insight: Current algorithms can find complex patterns and associations but most are not aimed to discover causal explanations. The conference also addresses the appropriate way to define causality in large data collected from chaotic and noisy systems, and the way to find causes that lie outside the measured variables. For example a correlation observed in a health survey based on genetic mapping might be due to an unmeasured environmental factor such as poverty.

The subject of the conference is of vital and current interest to every field of study, business, and government agencies. Our society has developed methods of collecting and storing enormous amounts of data, and is increasingly doing so. The data can arrive from controlled experiments, but most often comes from relatively uncontrolled field observations, such as those from social networks, human medical and genetic measurements, and patterns of purchases. The amount of data has far outstripped our ability to discern what important patterns are in the data, and most important, what explains those patterns. In a typical large database there are huge number of variables that can be measured, and virtually uncountable numbers of correlations between different subgroups of those variables. There are enormous potential benefits to science, business, government, and society if the critical patterns in Big Data can not only be ascertained but explained. Explanation is the goal of this conference, represented by the phrase, "drawing causal inference." The most pressing questions we face are causal in nature. In health we might observe that a particular treatment is associated with a decrease of cancer deaths, but need to know if the treatment is the cause of the decrease. In education we might observe that students held back in early grades tend to drop out of high school, but need to know if the treatment causes that result.

This is a grant for student (undergrad, grad) and postdoc support for travel and attendance at the Sackler Colloquium May 2-3 2019 at the Beckman Center in Irvine CA. The title is: "The Brain Produces Mind by Modeling". Speakers include a large variety of world leaders in computational and behavioral modeling of the brain-mind connection. The colloquium will bring together various threads and approaches in neuroscience, cognitive science, and psychological science by focusing on computational modeling that attempts to bridge the gaps between these fields. It aims to present a coherent view of the brain as a model builder, using the model to maximize survival and utility in a complex world.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.