Mary M. Hayhoe - US grants

The proposed experiments attempt to develop an explicit functional description of the processes underlying light and dark adaptation and also color appearance. An attempt will be made to relate the visual phenomena to underlying physiological and anatomical mechanisms, but emphasis is given to an explicit description of the variables controlling visual sensitivity as a first step in relating these phenomena to their physiological underpinnings. The following specific issues are addressed: (1) what are the precise mechanisms by which lateral inhibitory processes control sensitivity? (2) Can the after effects of bright light exposure and the effects of continuously exposed weaker lights be described in terms of one variable, or are a number of processes involved? (3) What is the role of lateral neural connections in hue constancy?

DESCRIPTION: (Investigator's Abstract) Any description of the processes which preserve the color and brightness of objects independently of the illumination must begin with a consideration of light adaptation. The aim of this proposal is to identify these adaptational transformations and show how they affect the brightness and color of lights. The chromatic and brightness induction paradigms will be used to analyze the transition to a steady state of adaptation following a change in background color of intensity. The underlying mechanisms will be classified into either multiplicative or substractive operations. These mechanisms have different effects on color appearance and have been useful in analyzing the processes controlling sensitivity. Both the spatial and temporal properties of these mechanisms will be investigated, in response to both large and small changes such as might occur in normal viewing of complex scenes. The experiments will reveal the properties of the adaptational mechanisms underlying brightness and color constancy, and will show to what extent they involve common mechanisms of adaptation and to what extent separate, parallel mechanisms, with different properties, operate in determining brightness and color.

The visual cortex of the brain is an area where there now is evidence for mechanisms of plasticity and learning, even at the earliest stages of processing, and even in the adult brain. It is appropriate to consider new approaches to the study of vision in this context. This meeting brings together researchers whose work focuses on the plastic nature and dynamic representation of vision. Presentations and discussions cover physiological, anatomical, behavioral and computational perspectives. Sessions cover the topics of cellular and anatomical foundations of learning and memory; adult cortical plasticity; perceptual learning; short-term memory and attention in active observers; and visual short-term memory. The conference will have an impact on understanding how the visual system interacts with areas traditionally associated with memory, and so have an impact across neuroscience and psychology, in addition to visual science.

skeletal system; mental disorders; cognition; nervous system; computers; bioengineering /biomedical engineering; biomedical resource; informatics; psychology; behavioral /social science research tag;

skeletal system; mental disorders; cognition; nervous system; computers; bioengineering /biomedical engineering; biomedical resource; informatics; psychology; behavioral /social science research tag;

skeletal system; mental disorders; cognition; nervous system; computers; bioengineering /biomedical engineering; biomedical resource; informatics; psychology; behavioral /social science research tag;

skeletal system; mental disorders; cognition; nervous system; computers; bioengineering /biomedical engineering; biomedical resource; informatics; psychology; behavioral /social science research tag;

This project involves both computational and experimental aspects. Visual search is a primary function of visual behavior. Little is known about the underlying mechanisms, however, particularly in understanding how the visual search process is manifest in eye movements, especially in a natural task. The experiments examine the eye movements made during search for a target object in a naturalistic scene. Our experiments reveal unexpected but very regular patterns of fixations. Gaze was typically directed to the center of the image before being shifted to the target, as shown in Figure ??. In this situation, `serial' search does not involve sequential fixation of individual objects, and is much more easily explained in terms of signal/noise models and a coarse-to-fine spatial analysis. In a separate theoretical effort we have been able to predict the pattern of fixations in human subjects using a simple model based on biologically plausible computations such as those performed in the primary visual cortex. A target for search can be simply described by the response of oriented spatial filters at different spatial scales (as might be carried out by simple cells in Vi). Rao and Ballard have shown that such a code discriminates well between different regions in a visual display, and can be used to reliably relocate the region in a visual search task (Figure ??). A simple filter template like this could be used for the target selection process, and used by the observer to mark locations to be held in memory to guide subsequent saccades. The linking of the human search experiments and computational modeling in this tight fashion is one of the major scientific achievements of the Resource. We will continue this work in the context of studies with patients with localized lesions to further strengthen the modeling and link it more closely with the underlying brain mechanisms.

Our major focus is the development of a driving simulator using a virtual display, with eye and head position monitoring. We chose driving as a test paradigm to investigate perceptual and cognitive function in a natural environment. Driving provides a good environment to study short-term cognitive information processing since crucial unprocessed information leads to an obvious behavioral outcome in driver errors. It also allows investigation of natural behavior in a situation that is still constrained enough to draw theoretically rigorous inferences. The relation of attention deficits to driver errors has been well established. Such deficits are far more correlated with accident records than visual deficits. One of the problems with classical paradigms is arranging for `unattended' stimuli to influence performance without covert attention shifts. This problem is almost always approached by using very brief presentations, which leaves open the question of how attention is deployed under natural conditions. In our driving simulator we can observe an extended behavioral sequence while maintaining tight experimental control. Covert attention shifts can be controlled by manipulating the demands of the primary driving task. Recent Progress. The car was mounted on a six degree-of-freedom motion platform which provides vestibular input.

This project developed a system that allows manipulation of complex displays in real time, contingent on the eye position of an observer. The goal of these experiments is to examine the nature of the representations that guide visual behavior during normal tasks, involving movements of the eyes and coordination of the eyes and hand. These experiments extend the analysis of visual operations to processes operating over a longer time scale than is usually considered, with an unprecented degree of stimulus control. This now allows the analysis of visual representations moment by moment. The other innovative aspect of the technique is that by measuring the consequences of the visual manipulation on task performance (e.g., by measuring fixation duration), we have a much more sensitive indicator of the experimental manipulation than is normally available. The experimental methodology has been developed over the last two years, and our results in the current grant period reveal that vision may be much more task-dependent than previously thought. Thus different fixations on the same visual stimulus serve a different purpose. The results also indicated that the visual information that is retained across successive fixations depends on moment-by-moment task demands and is used to minimize the amount of working memory. Recent Progress. The software for virtual displays of the Baufix parts has been installed. This was made available to us through our collaboration with Sagerer at the University of Bielefeld.

Two related illusions of motion reveal links between human perceptual representations and the motor system. In the `sliding rods illusion' a vertical and a horizontal line overlap to form a cross, and each line moves along a separate counterclockwise circular path in antiphase, without changing orientation. The intersection of the lines moves clockwise, but it is wrongly perceived as rotating counterclockwise. The implication is that these intersections are not parsed as objects, and therefore their motion path is not extracted, but instead the motion of the terminators (tips) is assigned to the vintersection. In the `sliding rings illusion,' two rings overlapped in a figure-8 are rotated about the center of the figure-8. Small texture cues de-termine whether observers see the figure as breaking into two separate rings that slide over each other or as a rigid rotating figure-8. For both illusions, the eyes can readily track the intersections when the figure is perceived as rigid but not when it is perceived as non-rigid. Therefore pursuit eye movements are compelled to rely upon perceptual interpretation of objects.

This project involves both computational and experimental aspects. Visual search is a primary function of visual behavior. Little is known about the underlying mechanisms, however, particularly in understanding how the visual search process is manifest in eye movements, especially in a natural task. The experiments examine the eye movements made during search for a target object in a naturalistic scene. Our experiments reveal unexpected but very regular patterns of fixations. Gaze was typically directed to the center of the image before being shifted to the target, as shown in Figure ??. In this situation, `serial' search does not involve sequential fixation of individual objects, and is much more easily explained in terms of signal/noise models and a coarse-to-fine spatial analysis. In a separate theoretical effort we have been able to predict the pattern of fixations in human subjects using a simple model based on biologically plausible computations such as those performed in the primary visual cortex. A target for search can be simply described by the response of oriented spatial filters at different spatial scales (as might be carried out by simple cells in Vi). Rao and Ballard have shown that such a code discriminates well between different regions in a visual display, and can be used to reliably relocate the region in a visual search task (Figure ??). A simple filter template like this could be used for the target selection process, and used by the observer to mark locations to be held in memory to guide subsequent saccades. The linking of the human search experiments and computational modeling in this tight fashion is one of the major scientific achievements of the Resource. We will continue this work in the context of studies with patients with localized lesions to further strengthen the modeling and link it more closely with the underlying brain mechanisms.

One of the fundamental difficulties in understanding the neural basis of perception/cognition is, understanding the computational or informational significance of neural activity. This topic is of central importance to understanding brain function at all levels. The enormous complexity of the brain and the behavior it generates demands the development of sophisticated theories of neural coding and communication on a large scale. While this is a largely intractable problem, a variety of recent experimental and computational studies make this a timely topic for a symposium. We propose to hold a Symposium at the Center for Visual Science in June, 2000, on the topic of neural coding. This will be the 22nd in our Symposium series, beginning in 1964. The goal is to have speakers using a variety of approaches: optical imaging, in vitro slice preparations, anatomy, single and multi-cell recordings, and psychophysics, and to consider the information encoded in events at individual synapses, in single neurons, small scale circuits and the overall flow of information through the brain in a simple sensory-motor act. In the tradition of past CVS Symposia, the goal is to bring recent developments in this fundamentally important topic to a broader audience than that captured by more specialized meetings. We also wish to bring together speakers from a wider variety of areas than usual in order to promote interactions between groups of investigators from diverse areas. This, too, is in the tradition of CVS Symposia.

DESCRIPTION (provided by applicant): The proposed research will attempt to provide a unified theory of task control of gaze and walking trajectories as humans move through natural environments. Until recently this goal would have been intractable, but a number of recent research results have illuminated the connection between simple sensory- motor decisions and behavioral goals. In particular, reinforcement-learning algorithms use reward signals to predict optimal behavior, and the central role of reward is well established in neurophysiological studies. Nonetheless it is unclear how these mechanisms determine natural visually guided behavior. Since natural gaze behavior is tightly linked to behavioral goals, reinforcement learning has the potential for understanding how behaviorally relevant targets are selected. We will develop a theoretical framework based on reinforcement learning for understanding sensory-motor decisions when humans move through natural environments. We first use Inverse Reinforcement Learning methods to estimate the internal reward associated with different behavioral goals when subjects navigate through obstacles and targets in a virtual environment, and then use the estimated reward values to predict the specific fixation sequences made while performing the task. We will test whether reward-weighted uncertainty determines gaze changes, predict gaze allocation in novel environments, and test how reward and uncertainty combine. A critical feature of the approach taken here is the decomposition of complex behavior into a set of sub-tasks. This approach has the potential for making complex behavior theoretically tractable and we will test this assumption. We will attempt to identify and quantify the potential sources of uncertainty such as sensory encoding, decay in spatial working memory, and uncertainty stemming from the observer's own motion in the environment Prior knowledge of an environment allows more efficient allocation of attention to novel or unstable regions. We will attempt to model the development of memory representations as a reduction in uncertainty, and evaluate how prior knowledge changes attentional allocation in uncertain environments. The work represents a major advance by developing a theoretical context for understanding selection of gaze targets in a moving observer. To date, formal theoretical approaches to decision making have addressed highly simplified scenarios. Because we are investigating natural vision there are very direct implications for both clinical and human factors situations involving multi-tasking. Eye movements are diagnostic of a variety of neural disorders and the exploration of normal gaze patterns in natural tasks provides essential data for comparison with disease states.

Project Summary/Abstract The Center for Perceptual Systems (CPS) is an interdisciplinary program at the University of Texas that provides a focal point for research and training in sensory systems. Continued growth in the Neurosciences at UT, as well as in CPS, particularly in the area of vision research, has lead to the development of a distinctive group in the Center that has a broad interdisciplinary research program in vision, representing psychophysical, neurophysiological, imaging, and computational approaches. Additionally, we have distinctive expertise and resources for the investigation of vision in the context of natural environments, and the development of computational tools for neural modeling and for modeling visually guided behavior. Because of both our broad expertise and our special strengths in these areas we are well positioned to train new scientists who will be at the forefront of vision research in the future. There are three important components to our training focus. First, we take advantage of the highly interdisciplinary and collaborative structure of CPS to provide broad cross- disciplinary training, which we consider essential for students of vision and visual performance. Second, we believe that training in computational methods is an essential component of research in vision and we take advantage of our particular strengths in this area. Third, we take advantage of our strengths in the area of natural systems analysis to provide a unique training opportunity in an area that is becoming increasingly important for a broad understanding of vision. These components lie at the core of our program, which includes basic courses, specialized seminars, training in advanced methodologies, attendance at the CPS colloquium series and the CPS symposium on Natural Environments, Tasks, and Intelligence, ethics training, and the development of professional skills.

Computer systems are increasingly coming to be relied upon to augment or replace human operators in controlling mechanical devices in contexts such as transportation systems, chemical plants, and medical devices, where safety and correctness are critical. A central problem is how to verify that such partially automated or fully autonomous cyber-physical systems (CPS) are worthy of our trust. One promising approach involves synthesis of the computer implementation codes from formal specifications, by software tools. This project contributes to this "correct-by-construction" approach, by developing scalable, automated methods for the synthesis of control protocols with provable correctness guarantees, based on insights from models of human behavior. It targets: (i) the gap between the capabilities of today's hardly autonomous, unmanned systems and the levels of capability at which they can make an impact on our use of monetary, labor, and time resources; and (ii) the lack of computational, automated, scalable tools suitable for the specification, synthesis and verification of such autonomous systems.

The research is based on study of modular reinforcement learning-based models of human behavior derived through experiments designed to elicit information on how humans control complex interactive systems in dynamic environments, including automobile driving. Architectural insights and stochastic models from this study are incorporated with a specification language based on linear temporal logic, to guide the synthesis of adaptive autonomous controllers. Motion planning and other dynamic decision-making are by algorithms based on computational engines that represent the underlying physics, with provision for run-time adaptation to account for changing operational and environmental conditions. Tools implementing this methodology are validated through experimentation in a virtual testing facility in the context of autonomous driving in urban environments and multi-vehicle autonomous navigation of micro-air vehicles in dynamic environments. Education and outreach activities include involvement of undergraduate and graduate students in the research, integration of the research into courses, demonstrations for K-12 students, and recruitment of research participants from under-represented demographic groups. Data, code, and teaching materials developed by the project are disseminated publicly on the Web.

Computational and quantitative approaches to the study of the brain and nervous system have been a part of neuroscience since its beginnings as a field. However, recent advances in the ability to collect large data sets, to create and run very large simulations, and in the development of new data analysis approaches may soon lead to an even more central role for computational thinking in this field. The goal of this meeting is to promote discussions and collaboration between experimental and computational neuroscientists. The meeting will highlight strategies for increasing the participation of historically under-represented groups in computational neuroscience.

The Principal Investigators and Co-Principal Investigators of grants supported through the NSF-NIH-ANR-BMBF-BSF-NICT-AEI-ISCIII Collaborative Research in Computational Neuroscience (CRCNS) program meet annually to report on projects; to discuss scientific, educational, and program-related issues; and to develop a cohesive investigator community representing many different approaches to computational neuroscience. This 15th meeting of CRCNS investigators brings together a broad spectrum of computational neuroscience researchers supported by the program, and includes plenary lectures, oral and poster presentations, and panel discussions. The meeting is scheduled for September 2-4, 2019 and is hosted by the University of Texas, Austin.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.