Nicholas B. Turk-Browne - US grants

DESCRIPTION (provided by applicant): The overarching goal of this research is to characterize how perception and memory interact, in terms of both the learning mechanisms that help transform visual experience into memory, and the intentional mechanisms that regulate this transformation. The specific goal of this proposal is to test the hypothesis that incidental learning about statistical regularities in vision (visual statistical learning) is limited to selectively attend visual information, and that this behavioral interaction arises because of how selective attention modulates neural interactions between human visual and memory systems. We propose a two-stage framework in which selective attention to a high-level visual feature/category increases neural interactions between regions of occipital cortex that represent low-level features and the region of inferior temporal cortex (IT) that represents the attended feature/category, and in turn between this IT region and medial temporal lobe (MTL) sub regions involved in visual learning and memory. In addition to assessing how feature-based selective attention influences learning at a behavioral level, we will use functional magnetic resonance imaging to assess how attention influences evoked neural responses in task-relevant brain regions, as well as neural interactions between these regions in the background of ongoing tasks. We will develop an innovative approach for studying neural interactions in which evoked responses and global noise sources are scrubbed from the data and regional correlations are assessed in the residuals. This background connectivity approach provides a new way to study how intentional goals affect perception and learning. Aim 1 examines the first stage of our framework, testing: how selective attention modulates background connectivity between IT and occipital cortex, where in retinotopic visual cortex this modulation occurs, and how these changes are controlled by frontal and parietal cortex. Aim 2 examines the second stage of our framework, first establishing the role of the MTL in visual statistical learning, and then testing: how selective attention modulates interactions between IT and the MTL, where in cortical and hippocampal sub regions of the MTL this modulation occurs, and how these changes facilitate incidental learning about statistical regularities and later retrieval of this knowledge. In sum, we relate behavioral interactions between selective attention and learning to neural interactions between the mechanisms that represent visual features and those that learn about their relations. This proposal addresses several key issues in the field, including: how attention modulates the MTL, how feature-based attention is controlled, whether different neural mechanisms support rapid versus long-term visual learning, how tasks and goals are represented, and how attention and memory retrieval are related. This research will improve our understanding of how humans learn from visual experience, and how visual processing is in turn influenced by learning. These advances will shed light on the plasticity that occurs during development and during the recovery and rehabilitation of visual function following eye disease, injury, or brain damage.

This Major Research Instrumentation award permits Dr. Jonathan Cohen and four co-investigators to purchase a high-performance computing instrumentation (3,584 cores; 2TB/core; 100TB flash storage) to be used by faculty, postdocs, graduate students and undergraduates within the Princeton Neuroscience Institute (PNI). The instrumentation will allow the analysis of human brain imaging data at a speed and scale not previously possible.

The collaborating researchers are cognitive neuroscientists and computer scientists at Princeton with complementary expertise in human brain imaging and large scale computing. Two primary research objectives are proposed, building on recent progress in applying multivariate pattern analysis (MVPA) methods from machine learning to detect neural signals that correspond to internal mental states, such as perceptions, memories and intentions that are otherwise not accessible to direct observation. To date, use of MVPA has been restricted to the "offline" analyses" after data have been fully collected. However, a growing and powerful use of brain imaging is to give participants feedback about their brain states in real time, allowing them to use this information to better control brain function (e.g., providing feedback about pain areas as a way of learning to control chronic pain). Such real-time feedback methods could be greatly enhanced by adding MVPA. However, this has been computationally intractable until now. Objective 1 addresses this challenge, by inserting a high performance computing system into the brain scanning pipeline. This will be tested in an experiment that uses MVPA to detect patterns of brain activity associated with sustained attention, allowing us to provide real-time brain-based feedback to improve attentional abilities (with potential educational and health benefits).

Objective 2 focuses on another major advance in brain imaging, in which correlations between areas of activity are analyzed, rather than areas of activity in isolation of one another. Such correlations - often referred to as "functional connectivity" - are likely to reveal more about how the brain actually functions, by providing critical information about the interactions between areas. At present, virtually all approaches to functional connectivity focus on the correlations among a limited set of brain areas of interest. However, a more powerful approach would be to examine the correlation of every area with all others. This requires computing the whole-brain correlation matrix. The analysis of such high dimensional data would be further enhanced by applying MVPA to patterns of correlation. However, doing this further increases computational demands. Applying this approach to a routine brain imaging dataset, using currently available instrumentation, would take 880 years to complete. The work under Objective 2 addresses this challenge, by coupling massively parallel computing with sophisticated software optimizations. Doing so can bring previously intractable problems into the range of practicality. These methods will be tested in an experiment that seeks to identify neural representations of intentions, and their influence on brain mechanisms responsible for executing these intentions.

Scientific researchers on university campuses create, analyze, visualize, and share large and diverse datasets from experimental devices like brain scanners, particle colliders, and genome sequencers. However, these "big data" applications place strain on traditional campus networks, due to rapidly increasing volumes of data, the need for either predictably low latency (to adapt experiments in real time) or high throughput (to transfer large data sets between locations), and sophisticated access-control policies (to protect the privacy of human subjects). To enable the next wave of scientific advances, university campuses must find effective ways to meet these challenging demands, at reasonable cost. The emerging technology of Software-Defined Networking (SDN) lowers the barrier to innovation in network management, and can substantially reduce cost through (i) inexpensive commodity network switches, (ii) greater automation of network configuration, and (iii) novel network-management applications that optimize bandwidth usage. Yet, existing innovation in SDN focuses primarily on the needs of commercial cloud providers, rather than the unique requirements of university campuses and scientific researchers. Princeton University is creating a software-defined campus network that can enable the next generation of data-driven scientific research. The initiative brings together big-data science researchers, computer scientists who are experts in SDN, and the campus Office of Information Technology. Princeton is deploying an open-source SDN platform for monitoring and configuring the network, conducting trials of new ways to support big-data applications, and bridging with the larger community, on and off campus, to support the sharing of scientific data, SDN software, and operational experiences.

PROJECT SUMMARY Our overarching goal is to understanding how stored memories change as a function of experience. The pro- posed work builds on prior research showing a U-shaped relationship between memory activation and learn- ing, whereby strong activation leads to synaptic strengthening, moderate activation leads to synaptic weaken- ing, and no activation leads to no change in synaptic strength. The present grant focuses on the implications of this U-shaped relationship for representational change: Learning is not just about making memories stronger or weaker?it can also decrease neural overlap between memories (differentiation) or increase neural overlap (integration). These neural changes can have profound effects on memory retrieval: Decreased overlap can reduce interference, at the cost of preventing generalization. Our specific goal is to construct and test a com- putational model of representational change and how it is shaped by competitive neural dynamics. When im- plemented in neural networks that are capable of self-organizing internal representations, our theory makes clear, novel predictions about when differentiation and integration will occur: Differentiation of memories A and B will occur when (i) B is moderately activated while processing A, causing weakening of connections between B and A, and (ii) B is reactivated later, allowing it to acquire new features that do not overlap with A; by con- trast, integration will occur if B is strongly activated during A, causing strengthening of connections between B and A. Aim 1 will use neural network simulations to address vexing puzzles in the literature and to generate novel empirical predictions. Aim 2 will test these predictions using behavioral and fMRI experiments focused on learning of new associations in the hippocampus, with a particular emphasis on testing the model's predic- tions about how competitive dynamics relate to representational change. Aim 3 will test the model's predictions regarding cortical plasticity, using a novel sketching task that induces competition between representations of familiar objects. Representational change will be assessed behaviorally in terms of how sketches and object recognition change over learning and neurally using fMRI of visual cortex; a deep neural network model of the ventral stream will be used to measure changes in the features of sketches. In summary: The proposed studies use multiple innovative approaches (fMRI pattern analysis, neural network modeling, free-form object sketching, and computer vision) to address the fundamental question of when experience causes neural repre- sentations to differentiate or integrate, thereby advancing our basic understanding of neuroplasticity. Improving our understanding of neural differentiation could have transformative implications for treating cognitive deficits in a wide range of clinical conditions, including stroke, dyslexia, and dementia. In all of these conditions, cogni- tive deficits can arise from insufficient separation of representations. This research may lead to better ways of re-differentiating these representations and?through this?ameliorating the associated cognitive deficits.

This project will build a transformative bridge between data science and neuroscience. These two young fields are driving cutting-edge progress in the technology, education, and healthcare sectors, but their shared foundations and deep synergies have yet to be exploited in an integrated way - a new discipline of "data neuroscience." This integration will benefit both fields: Neuroscience is producing massive amounts of data at all levels, from synapses and cells to networks and behavior. Data science is needed to make sense of these data, both in terms of developing sophisticated analysis techniques and devising formal, mathematically rigorous theories. At the same time, models in data science involving AI and machine learning can draw insights from neuroscience, as the brain is a prodigious learner and the ultimate benchmark for intelligent behavior. Beyond fundamental scientific gains in both fields, the project will produce additional outcomes, including: new collaborations between universities, accessible workshops, graduate training, integration of undergraduate curricula in data science and neuroscience, research opportunities for undergraduates that help prepare them for the STEM workforce, academic-industry partnerships, and enhanced high-performance computing infrastructure.

The overarching theme of this project is to develop a two-way channel between data science and neuroscience. In one direction, the project will investigate how computational principles from data science can be leveraged to advance theory and make sense of empirical findings at different levels of neuroscience, from cellular measurements in fruit flies to whole-brain functional imaging in humans. In the reverse direction, the project will view the processes and mechanisms of vision and cognition underlying these findings as a source for new statistical and mathematical frameworks for data analysis. Research will focus on four related objectives: (1) Distributed processing: reconciling work on communication constraints and parallelization in machine learning with the cellular neuroscience of motion perception to develop models of distributed estimation; (2) Data representation: examining how our understanding of the different ways that the brain stores information can inform statistically and computationally efficient learning algorithms in the framework of exponential family embeddings and variational inference; (3) Attentional filtering: incorporating the cognitive concept of selective attention into machine learning as a low-dimensional trace through a high-dimensional input space, with the resulting models used to reconstruct human subjective experience from brain imaging data; (4) Memory capacity: leveraging cognitive studies and natural memory architectures to inform approaches for reducing/sharing memory in artificial learning algorithms. The inherently cross-disciplinary nature of the project will provide novel theoretical and methodological perspectives on both data science and neuroscience, with the goal of enabling rapid, foundational discoveries that will accelerate future research in these fields.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Project Summary Research on perceptual learning (PL) has been dominated by studies that seek to isolate and improve individual visual processes. However, an important translational outcome of PL research is to address the needs of patients with vision loss, who seek to improve performance on daily tasks such as reading, navigation, and face recognition. These more ecological cases of behavioral change and cortical plasticity, which are inherently complex and integrative, have revealed significant gaps in a more holistic understanding of how multiple visual processes and their associated brain systems jointly contribute to durable and generalizable PL. To address these gaps, here we study simulated and natural central vision loss. We focus on macular degeneration (MD), one of the most common causes of vision loss (projected to affect 248 million people worldwide by 2040), which results from damage to photoreceptors in the macula that disrupts central vision. Such central vision loss is a superb lens through which study to how ecologically relevant changes in the use of vision relate to changing brain activity and connectivity because it represents a massive alteration in visual experience requiring reliance on peripheral vision for daily tasks. With the use of eye-trackers and gaze-contingent displays that induce central scotomas, central vision loss can be simulated in normally seeing individuals, who then develop peripheral looking patterns that resemble compensatory vision strategies seen in MD patients. Ideal use of peripheral vision requires improvement in multiple vision domains, three of the most important being: early visual processing (e.g., visual sensitivity), mid-level visual processing (e.g., spatial integration), and attention and eye-movements. To date, no study has systematically investigated these three domains of PL and their neural underpinnings. The proposed research plan rests on rigorous prior work showing that PL influences multiple brain structures and functions related to these three domains. We propose a novel approach of systematically measuring how different training regimes related to the three domains influence a broad range of psychophysical and ecological behaviors (Aim 1), how these changes arise from plasticity in brain structure and function (Aim 2), and how PL after simulated central vision loss compares to PL in MD (Aim 3). This work is significant and innovative as it will be the first integrated study of PL characterizing multiple trainable factors and their impact on diverse behavioral outcomes and on cutting-edge assessments of neural representations and dynamics. It is also the first study to directly compare PL in MD patients with PL in a controlled model system of central visual field loss with simulated scotomas, which if validated will allow the use of this model system to interrogate MD in larger samples of healthy individuals. We will also share a unique dataset that will help the field to understand behavioral and neural plasticity after central vision loss and individual differences in responsiveness to training. Finally, this work will illuminate basic mechanisms of brain plasticity after sensory loss that may generalize to other forms of rehabilitation after peripheral or central damage.