Kenneth A. Norman - US grants

Dual-process theories or recognition posit that participants can recognize a stimulus as having been studied earlier because 1) the stimulus seems familiar, or 2) they recollect specific details pertaining to that stimulus; recent neuropsychological evidence suggests that familiarity and recollection are subserved by distinct brain structures (rhinal neocortex and the hippocampus, respectively). The goal of the proposed research is to: 1) use neural network models to explain how neocortex implements familiarity, and how the hippocampus implements recollection; and 2) test these models predictions regarding recognition performance in normal subjects and subjects with focal hippocampal damage. The first specific aim is to establish, using simulations and experiments, key qualitative differences between neocortical and hippocampal contributions to recognition (e.g., I will explore the idea that increasing the similarity of lures to studied items impairs neocortically-driven recognition performance more that hippocampally-driven recognition performance). The second specific aim is to integrate the neocortical and hippocampal models, and use the combined model to: 1) precisely account for recognition data when both processes are contributing to recognition; and 2) explore the relationship between familiarity and recollection (e.g., are they independent). Extant formal models of recognition do not make contact with the underlying neurobiology as such, the proposed research constitutes a major step forward in recognition memory modeling. More practically, this research will improve our understanding of preserved memory capacities in brain-damaged patients; this, in turn, should benefit rehabilitation of these patients.

DESCRIPTION (provided by applicant): The goal of this research is to develop and test a mechanistically explicit theory of how the brain gives rise to episodic memory: our ability to recall previously experienced events, and to recognize events as having been experienced previously. Researchers have known for years that the hippocampus is the key neural substrate of recall; more recently, several studies have found that -- after focal hippocampal damage -- medial temporal lobe cortex (MTLC) can support some degree of spared recognition performance on its own. To explore how the contributions of hippocampus and MTLC differ, the research proposed here uses neural network models of these structures to simulate patterns of memory performance from normal and brain-damaged subjects. The first specific aim is to test the model's predictions regarding when item and associative recognition performance will be affected by knocking out the hippocampal contribution; "knockout" will be operationalized by using patients with focal hippocampal damage, and also normal subjects who are forced to respond quickly. Other experiments will test model predictions regarding how context change and interference manipulations affect recognition in the two systems. The second specific aim is to run simulations using a combined cortico-hippocampal model to address how the two processes conjointly determine recognition performance (e.g., in paradigms that place the two processes in opposition), and to run experiments in amnesic patients and controls to test these predictions. Extant formal models of recognition memory do not make contact with the underlying neurobiology; as such, this research constitutes a major step forward in recognition memory modeling. This link to neurobiology provides extra constraints that can be leveraged to gain new insights into fundamental puzzles in the memory literature. Regarding health benefits: The model's ability to address lesion data will bolster our understanding of what kinds of learning are spared after different kinds of brain damage, which (in turn) will help doctors develop more effective rehabilitation regimes. Furthermore, as researchers start to develop therapies that change the underlying parameters of learning in the brain, we will need some way of assessing how these changes scale up and affect behavioral memory performance; the research proposed here is ideally positioned to meet this growing need.

DESCRIPTION (provided by applicant): The overarching goal of this research is to understand neural learning rules: What are the circumstances that give rise to strengthening and weakening of memories? Our specific goal is to evaluate the hypothesis that learning is competition-dependent. According to this hypothesis, whenever neural representations compete to be active, the winning representations are strengthened, the losing representations are weakened, and learning is modulated by the margin of victory (i.e., closer competition yields more learning). The present research builds on previously funded work where biologically-based computational models were used to explore the neural mechanisms of competition- dependent learning. These simulations demonstrated that competition-dependent learning could be implemented in the brain by leveraging oscillations in the strength of neural inhibition; they also demonstrated that competition- dependent learning could account for detailed patterns of human memory data. The first specific aim of this research is to refine our computational model of competition-dependent learning. In particular, the simulations will focus on paradigms where there is close competition between the sought-after memory and other memories. In situations of this sort, the competition-dependent learning hypothesis predicts that learning effects will be highly volatile - small differences in memory excitation can affect the outcome of the competition, which (in turn) will affect which memories are strengthened and which are weakened. The computational model will be extended to include a more realistic form of inhibition, a more biophysically detailed hippocampal model, and a cognitive control system that allows the model to recall the sought-after memory even when it is weaker than competitors. These changes should improve our ability to predict behavioral forgetting effects and neural activity patterns in these high-volatility situations. The second specific aim of this research is to develop and utilize new experimental methods for testing hypotheses about competition-dependent learning. Testing the competition-dependent learning hypothesis is difficult because (according to the model) learning effects depend on the precise level of excitation of the competing memories. To address this problem, the proposed studies will use highly sensitive pattern classifier algorithms, applied to brain imaging data, to track the extent to which memories compete on a trial-by-trial basis. This neural readout of the competing memories can be used to test the model's predictions about how competition drives learning. The proposed studies will test the model's predictions about memory weakening in a perceptual priming paradigm, a short-term memory paradigm, and a long-term paired-associate learning paradigm. With regard to mental health: This research will improve our understanding of the circumstances that trigger strengthening and weakening of memories, and it will improve our ability to detect (based on neural activity) when strengthening or weakening will occur. These developments will help us to devise better methods for strengthening desirable memory associations and also methods for weakening maladaptive memories (e.g., when treating phobias or post-traumatic stress disorder).

The PIs will develop new machine learning algorithms to explore how meaning is represented in the brain and how meaning representations shape human memory. Current neuroscientific theories of memory posit that forming a memory for a particular event involves associating the details of that event with the person's current mental context, i.e., everything else that she is thinking about at the time. When trying to remember the event, the person can access stored details by reinstating the mental context that was present when the memory was formed. This fits with the intuition that forgotten details (e.g., the location of misplaced house keys) can be retrieved by mentally "re-tracing steps", i.e., trying to reinstate the mindset that was present at the time of the original event. With these theories in mind, the goal of this work is to develop machine learning algorithms that make it possible to track, based on fMRI brain data and behavioral memory data, the process of "mentally re-tracing steps"---the proposed algorithms will be able to decode the state of a person's mental context as she forms memories and (later) as she searches for these memories.

The proposed work uses two fundamental ideas about memory and meaning: The first idea is that mental context is shaped by the meanings of recently encountered stimuli. The second idea is that semantic relationships between concepts in the brain mirror statistical relationships between words in naturally occurring language. The developed algorithms will bring together data from three sources---behavioral data from subjects performing memory recall tasks, fMRI neuroimaging data collected while subjects performed these tasks, and large collections of documents---to discover a latent meaning space that can simultaneously describe all three types of information. Each point in this space describes a mental context. Thus the core of the proposed work is to develop latent variable models and algorithms that can infer from data how the mental context moves through meaning space as a person stores and searches for memories.

The proposed work will lead to fundamental advances in machine learning (new algorithms for inferring hidden variables based on multiple, heterogeneous data types) and neuroscience (more refined theories of how memory search is accomplished in the brain). Furthermore, this work will catalyze the development of new technologies for diagnosing and remediating memory problems, by making it possible to track how the contextual reinstatement process is going awry in people experiencing memory retrieval failure.

The PIs and Co-PIs of grants supported through the NSF-NIH-BMBF Collaborative Research in Computational Neuroscience (CRCNS) program meet annually. This will be the seventh meeting of CRCNS investigators, and the first involving US-German projects supported by NSF/NIH and BMBF. The meeting brings together a broad spectrum of computational neuroscience researchers supported by the program, and includes poster presentations, talks and plenary lectures. The meeting is scheduled for October 9-11, 2011 and will be held at Princeton University.

This Major Research Instrumentation award permits Dr. Jonathan Cohen and four co-investigators to purchase a high-performance computing instrumentation (3,584 cores; 2TB/core; 100TB flash storage) to be used by faculty, postdocs, graduate students and undergraduates within the Princeton Neuroscience Institute (PNI). The instrumentation will allow the analysis of human brain imaging data at a speed and scale not previously possible.

The collaborating researchers are cognitive neuroscientists and computer scientists at Princeton with complementary expertise in human brain imaging and large scale computing. Two primary research objectives are proposed, building on recent progress in applying multivariate pattern analysis (MVPA) methods from machine learning to detect neural signals that correspond to internal mental states, such as perceptions, memories and intentions that are otherwise not accessible to direct observation. To date, use of MVPA has been restricted to the "offline" analyses" after data have been fully collected. However, a growing and powerful use of brain imaging is to give participants feedback about their brain states in real time, allowing them to use this information to better control brain function (e.g., providing feedback about pain areas as a way of learning to control chronic pain). Such real-time feedback methods could be greatly enhanced by adding MVPA. However, this has been computationally intractable until now. Objective 1 addresses this challenge, by inserting a high performance computing system into the brain scanning pipeline. This will be tested in an experiment that uses MVPA to detect patterns of brain activity associated with sustained attention, allowing us to provide real-time brain-based feedback to improve attentional abilities (with potential educational and health benefits).

Objective 2 focuses on another major advance in brain imaging, in which correlations between areas of activity are analyzed, rather than areas of activity in isolation of one another. Such correlations - often referred to as "functional connectivity" - are likely to reveal more about how the brain actually functions, by providing critical information about the interactions between areas. At present, virtually all approaches to functional connectivity focus on the correlations among a limited set of brain areas of interest. However, a more powerful approach would be to examine the correlation of every area with all others. This requires computing the whole-brain correlation matrix. The analysis of such high dimensional data would be further enhanced by applying MVPA to patterns of correlation. However, doing this further increases computational demands. Applying this approach to a routine brain imaging dataset, using currently available instrumentation, would take 880 years to complete. The work under Objective 2 addresses this challenge, by coupling massively parallel computing with sophisticated software optimizations. Doing so can bring previously intractable problems into the range of practicality. These methods will be tested in an experiment that seeks to identify neural representations of intentions, and their influence on brain mechanisms responsible for executing these intentions.

This proposal is for the renewal of a predoctoral and postdoctoral Quantitative Neuroscience Training Program (QNTP) at Princeton University. Neuroscience research is becoming increasingly quantitative. Formal theoretical techniques are essential for understanding how complex, large-scale interactions between neurons give rise to thought and behavior, and advanced quantitative methods of data analysis are necessary for addressing the increasingly large, multidimensional data sets generated by modern brain imaging techniques (e.g., multiunit recording, fMRI). These methods are also necessary for future progress to be made in understanding, diagnosing, treating and, ultimately, curing brain disturbances that give rise to psychiatric disorders. Unfortunately, the mathematical and computational skills required to address these needs are not a focus of standard neuroscience curricula. Princeton's QNTP is designed to address this need, by providing the next generation of neuroscientists with the necessary mathematical and computational skills for measuring, analyzing, and modeling brain function. The establishment of the QNTP sparked several developments at Princeton, that (in turn) have accelerated the pace at which the goals of the QNTP are being met. By bringing Princeton's neuroscientists together with faculty in Physics, Mathematics, Computer Science and Engineering, the QNTP helped to spur the formation of the Princeton Neuroscience Institute (PNI) in 2005. The QNTP also helped to inspire the formation (in 2008) of PNI's free-standing PhD Program in Neuroscience, which strongly emphasizes classroom and laboratory training in basic quantitative and computational methods during its first two years. These developments have made it possible for us to refocus the QNTP from its original purpose (providing a foundation in quantitative neuroscience for trainees who are starting out in this area) to providing advanced training in quantitative neuroscience. Specifically, we will take the most quantitatively-focused subset of our predoctoral and postdoctoral trainees and provide them with the additional tools and training that they need to excel in computational neuroscience research. This training will be accomplished via advanced quantitative and computational neuroscience elective courses that were developed for the QNTP and are taught by leaders in the field, as well as participation in research seminars, journal clubs, and career development activities that are designed to deepen the trainees' knowledge and bolster community among the trainees. PNI faculty have made seminal contributions to quantitative neuroscience, ranging from information- theoretic analyses of neuronal spiking and dynamical systems analysis of decision-making to multivariate decoding of human neuroimaging data. The QNTP has been specifically formulated to bring predoctoral and postdoctoral trainees into contact with this expertise and, through this, to catalyze their transformation into full- fledged computational neuroscientists. As with the prior funding period, we are requesting support for four predoctoral trainees and four postdoctoral trainees (with 2 year appointments for each trainee).

Identifying the cognitive, computational and neural mechanisms responsible for determining why some memories survive when others fade is one of the many grand challenges facing researchers of the human mind and brain. It is widely understood that sleep plays a critical role in long-term remembering, yet what exactly happens during sleep to affect the persistence of memories remains largely unknown. This project brings together a team of researchers who will integrate multiple independent lines of work in cognitive neuroscience, cognitive psychology, and computer science in order to investigate the precise mechanisms undergone by recently-formed memory representations as a person sleeps and how these mechanisms determine which memories survive and which fade. The proposed integration of cutting-edge neural data analysis methods for EEG and neuroimaging data, basic human memory theory, and neural network modeling make possible the ability to non-invasively track individual memories in the human brain as they compete with each other and are modified during sleep. The potential advances from this work could impact education, training situations, and public health by facilitating the development of new strategies for ensuring that important memories survive after initial learning.

Research suggests that memories compete for neural space such that reactivating one particular memory can exert "collateral damage" on other related memories. In other words, accessing one memory can come at the expense of later being able to access other nearby memories in the network space. The proposed studies test the hypothesis that importance shapes neural dynamics during sleep by selectively boosting memory reactivation; this boost ensures that important memories out-compete related memories during sleep, resulting in strengthening of important memories and weakening of less-important memories. To test this hypothesis, competition between memories will be elicited during sleep by playing sound cues, each of which was linked (during wake) to two different picture-location memories. Multiple interlocking approaches will track how memory competition during sleep shapes a memory's persistence versus fading. Neural network models will be used to generate predictions about how reward responses during encoding shape competitive dynamics during sleep, and how these competitive dynamics determine the eventual fates of competing memories. Predictions will be tested by using fMRI to measure neural activity associated with reward processing during encoding, EEG to measure brain activity during sleep, and pattern classifiers to decode memory activation from the sleep EEG data. Observations of competitive dynamics during sleep will then be related to later memory performance and to multivariate fMRI measures of memory change. The project has the potential to provide, for the first time, a comprehensive look "under the hood" at the life of a memory as it is acquired, processed during sleep, and eventually recalled. Pivotal knowledge will be gained about how variance in reward processing at encoding influences sleep replay dynamics, and about how sleep replay dynamics affect subsequent memory performance and the structure of neural representations.

PROJECT SUMMARY Our overarching goal is to understanding how stored memories change as a function of experience. The pro- posed work builds on prior research showing a U-shaped relationship between memory activation and learn- ing, whereby strong activation leads to synaptic strengthening, moderate activation leads to synaptic weaken- ing, and no activation leads to no change in synaptic strength. The present grant focuses on the implications of this U-shaped relationship for representational change: Learning is not just about making memories stronger or weaker?it can also decrease neural overlap between memories (differentiation) or increase neural overlap (integration). These neural changes can have profound effects on memory retrieval: Decreased overlap can reduce interference, at the cost of preventing generalization. Our specific goal is to construct and test a com- putational model of representational change and how it is shaped by competitive neural dynamics. When im- plemented in neural networks that are capable of self-organizing internal representations, our theory makes clear, novel predictions about when differentiation and integration will occur: Differentiation of memories A and B will occur when (i) B is moderately activated while processing A, causing weakening of connections between B and A, and (ii) B is reactivated later, allowing it to acquire new features that do not overlap with A; by con- trast, integration will occur if B is strongly activated during A, causing strengthening of connections between B and A. Aim 1 will use neural network simulations to address vexing puzzles in the literature and to generate novel empirical predictions. Aim 2 will test these predictions using behavioral and fMRI experiments focused on learning of new associations in the hippocampus, with a particular emphasis on testing the model's predic- tions about how competitive dynamics relate to representational change. Aim 3 will test the model's predictions regarding cortical plasticity, using a novel sketching task that induces competition between representations of familiar objects. Representational change will be assessed behaviorally in terms of how sketches and object recognition change over learning and neurally using fMRI of visual cortex; a deep neural network model of the ventral stream will be used to measure changes in the features of sketches. In summary: The proposed studies use multiple innovative approaches (fMRI pattern analysis, neural network modeling, free-form object sketching, and computer vision) to address the fundamental question of when experience causes neural repre- sentations to differentiate or integrate, thereby advancing our basic understanding of neuroplasticity. Improving our understanding of neural differentiation could have transformative implications for treating cognitive deficits in a wide range of clinical conditions, including stroke, dyslexia, and dementia. In all of these conditions, cogni- tive deficits can arise from insufficient separation of representations. This research may lead to better ways of re-differentiating these representations and?through this?ameliorating the associated cognitive deficits.

Project Summary Present and prior information converge often in everyday life. For example, during language comprehension, each syllable achieves its meaning in the context of a word, and each word in the context of a sentence. Despite the clear importance of such integration of past and present, most studies of memory use simple stimuli that are isolated in time. The long-term goal of this laboratory is to understand how the brain uses past information, gathered over seconds to hours, to make sense of a stream of incoming information. Previous work from the laboratory shows that many areas of cortex can accumulate information over time and use it for online processing. Furthermore, this research showed that early sensory areas use past information gathered over milliseconds, and this timescale increases to minutes in higher-order brain areas. These findings suggest that memories needed for online stimulus processing are topographically distributed in a hierarchy across the cortex based on their temporal properties. The overall objective of this application is to investigate the functional role of cortical areas at the top of the processing hierarchy; in particular, we will investigate to what extent these cortical areas have an intrinsic ability to accumulate information over minutes, and to what extent these long-timescale properties emerge from interactions with the hippocampus. This contribution is significant because the proposed research will provide new insights into a central function of the brain: the ability to accumulate information over minutes and use it to process an incoming information stream. This approach is innovative because it uses new experimental paradigms, both fMRI and ECoG methods, both neurotypical and brain lesioned amnesic patients, and includes development of novel analysis methods for brain responses to complex natural stimuli (stories and movies). The work proposed in this application will advance knowledge of how the brain combines information across minutes and will produce new approaches to the study of how memory is dynamically used during online stimulus processing.

This project will break new ground in the study of memory by partnering with competitors in the USA Memory Championship. These competitors are not savants, but instead are well-practiced in the use of mnemonic techniques and, as a result, exhibit enhanced powers of memory on a range of real-world tasks, such as memorizing the items on a shopping list. All of these techniques rely on the practitioner structuring prior knowledge in very specific ways that facilitate the incorporation of new information. By scanning the brains of these trained memorists with functional magnetic resonance imaging (fMRI) and comparing their brain activity to participants who are learning these mnemonic systems for the first time, the researchers will identify principles for optimal scaffolding: How can prior knowledge be structured and used to most effectively support new learning? Identifying these principles will improve our fundamental understanding of real world-memory and will also lay the foundation for future educational interventions based on these principles. This project is funded by Integrative Strategies for Understanding Neural and Cognitive Systems (NCS), a multidisciplinary program jointly supported by the Directorates for Computer and Information Science and Engineering (CISE), Education and Human Resources (EHR), Engineering (ENG), and Social, Behavioral, and Economic Sciences (SBE).

The goal of the project is to extend theories of memory to address how people can optimally use cognitive maps (structured prior knowledge) to support new learning. Reinforcement learning algorithms will be applied to computational models of memory to make predictions about which strategies will result in the best performance, factoring in biological constraints on the human memory system. Model predictions about optimal memory strategies will be tested using fMRI data from memory experts who have spent years optimizing their ability to bind arbitrary information to an internal cognitive map (a ?memory palace?), and who therefore serve as a unique comparison group for optimized memory models; these subjects will be compared to a sample of young adult subjects who are being trained to use these memorization techniques. New neuroimaging approaches developed by the researchers will allow them to map the brain patterns corresponding to each room of the memory palace and the patterns corresponding to each individual memory, and then track the activation of these patterns as subjects recall memories using mental walks through their palace. Results of these analyses will be used to test detailed model predictions about how memory training will alter the structure and use of subjects? cognitive maps, and how these changes relate to memory performance. As a final test of the models, the researchers will use neural measurements of individual subjects? cognitive maps to predict which specific items they will recall. By examining how prior knowledge is deployed to support learning in experts and novices at a much finer resolution than was previously possible, this work will provide the foundation for understanding why wide variations in memory performance exist across individuals and how memory can be improved, paving the way for targeted interventions to improve memory performance.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.