Miguel Eckstein - US grants

Evaluation based on observer studies is time consuming and costly and optimization is just not viable. There is therefore a strong rationale for image quality measures with the predictive accuracy of observer studies but that are time and cost efficient. We build on our previous work on computer model observers to extend the application of model observers from basic detection tasks where the lesion is invariant (detection/signal- known exactly on D-SKE tasks) to more realist tasks where the lesion appearing in the image can be of different types and sizes (detection- classification/signal known statistically or DC-SKS tasks). Our goal is to develop a computer model observer that can be used to predict the effect of image compression in these more realistic detection/classification tasks with lesion variability. This model could be used for optimization of compression algorithm parameters with respect to task performance in these more realistic tasks. To achieve this goal we propose five specific aims: 1) To develop computer model observers for tasks where the observers have to detect and classify a lesion and where the lesions have different sizes and/or orientations. 2) To perform psychophysical measurements of five different state of the art image compression algorithms (some of them which are being considered as the JPEG 2000 international standard) in the DC-SKS task and compare it to our previous results with D-SKE tasks. 3) To compare the newly proposed DC-SKS model observers with respect to their ability to predict the observer task performance measured in specific aim 2. 4) To use the DC-SKS model observer with highest predictive power to perform automated optimization of compression algorithms using stimulated annealing techniques. 5) To perform psychophysical comparisons of the effects of the compression algorithms on tasks with the default compression parameters, DCS-SKS model optimized parameters, and the more traditional D-SKE model optimized parameters. If successful, we will extend the use of model observers to more realistic tasks. The impact of this research will be improved computer based metrics for cost and time efficient evaluation of medical image quality as well as more rapid and cost effective communication and storage of digital coronary angiograms.

Visual search is a process that humans engage in on a daily basis. Looking for a friend in a crowd, a car in a parking lot, or a tumor in an x-ray mammogram are all examples of visual search. Visual search involves two processes, (1) scrutinizing the visual scene by making eye movements to orient the high-resolution fovea to the regions of interest and (2) attending to different points in the visual scene independent of the eye position to select relevant visual information. For over three decades, investigators have studied visual attention and offered different theories about its function and its processing limitations during search. This research will test a number of existing and newly proposed theories using novel mathematical/computer implementations of models of visual attention. The results will allow for a better understanding of the function of visual attention during visual search. The research will also provide a new theoretical and quantitative framework to study the types of attentional disorders present in patients with schizophrenia, Alzheimer's disease, attentional deficit/hyperactive disorders, and hemi-neglect. The education plan will incorporate the use of computer and mathematical models in the study of visual attention in undergraduate and graduate education. In addition, the education plan will include implementation of a website that will allow students and researchers worldwide Internet access to use and test computer models of visual attention.

Neural plasticity and perceptual learning are fundamental in the developmental stages of vision, in attaining expertise in specialized perceptual tasks, and in recovery from brain injuries and low- vision disorders. One important process in perceptual learning is the improvement in humans' ability to use task-relevant (signal) information. Although there have been advances in the understanding of the dynamics and algorithms mediating how humans optimize the selection of task relevant visual information, little is known about how eye movement patterns vary with practice and their impact in optimizing perceptual performance. Yet, in real world environments, eye movements are a critical component of active vision as humans explore the visual scene to make perceptual judgments. Understanding perceptual learning in human daily life requires studying the mechanisms mediating the changes in the planning of eye movements with learning and their contributions to optimizing perceptual performance. We hypothesize that two new experimental paradigms with digitally designed visual stimuli, in conjunction with eye position recording, and a newly developed foveated ideal observer and Bayesian learner will help elucidate how humans learn to strategize their eye movements and the contributions of the optimized sampling of the images to improvements in perceptual learning. The proposed work will address the following questions: 1) Do humans use learned information about the statistical properties of the visual stimuli and the requirements of the task at hand to strategize their eye movements to optimize the foveal sampling of the visual scene and perceptual performance?; 2) Do humans use knowledge of the varying resolution of their foveated visual system to optimally learn to plan eye movements for a given set of visual stimuli and task?; 3) What are the contributions of learning to strategize eye movements to the overall improvements in perceptual performance in ecologically important tasks such as face recognition, object identification and visual search?; 4) How do human fixation patterns and performance benefits from strategizing eye movements compare to an optimal foveated observer and learner? The proposed work will improve our understanding of the human neural algorithms mediating the dynamics of adult perceptual learning during active vision for ecologically important tasks. The proposed experimental protocols and theoretical developments will also provide a novel, powerful and flexible framework with which other researchers can study eye movements and learning of humans undergoing visual loss recovery as well as patients with learning disabilities.

DESCRIPTION (provided by applicant): Evaluation of medical image quality based on human observer studies is time consuming and costly and optimization is just not viable. There is therefore a strong rationale for image quality measures with the predictive accuracy of observer studies but that are time and cost efficient. We build on our previous work on optimization of image compression using computer model observers to extend their application to complex classification tasks (e.g., stent deployment assessment) and image sequences of patient structured backgrounds. Our new goal is to develop a computer model observer that can be used to optimize the processing and display of dynamic x-ray coronary angiographic images acquired with the newly introduced flat panel digital detectors. To achieve this goal we propose six specific aims: 1) To develop a large set of test images combining x-ray coronary angiograms acquired with the flat panel digital detectors and simulated arterial segments and lesions; 2) To develop computer model observers for more complex classification tasks and sequences of patient structured backgrounds; 3) To perform human visual psychophysical studies evaluating different processing and display parameters including: 14 bit to 8 bit transformations, pixel binning, display frame rate and number of frames; 3) To compare the newly proposed model observers with respect to their ability to predict the observer task performance; 4) To use the model observer with highest predictive power to perform automated optimization of the processing algorithm methods; 5) To perform a psychophysical validation study comparing the optimized processing functions to other standard functions. The impact of this research is two-fold: 1) Improved computer based metrics of medical image quality for more complex classification tasks and medical image sequences; 2) Improved processing and display of digitally acquired x-ray coronary angiograms leading to increased accuracy in clinically relevant tasks.

Human visual search involves scrutinizing the visual scene by making eye movements to orient the center of the eye, the fovea, to regions of interest. The high resolution of the fovea allows for processing of the visual world with finer detail. In addition, visual search also involves attending to different points in the visual scene independent of the eye position. Visual search is also often facilitated when there are other objects (cues) that co-occur with the target of interest. When the target object appears at unexpected locations, people often have difficulty finding the target. Researchers have for decades studied how such predictive cues benefit human search performance in a variety of tasks. However, most of these studies were designed to prevent humans from making eye movements to study the mechanisms of covert visual attention in isolation. Little is known about the mechanisms by which cues and context aid search in natural behavior where people are free to move their eyes. In this proposal, the investigators seek to understand the mechanisms by which humans use cues and context to plan eye movements and covert attention to optimize visual search. They will develop mathematical models implemented on computers to mimic the processes by which the human brain mediates visual search. The research will allow for a better understanding of the mechanisms by which cues and context aid visual search and also predict the benefits in accuracy of perceptual judgments brought by the use of predictive cues in visual search.

Looking for a friend in a crowd, a car in a parking lot, and your house keys in the living room are all examples of visual searches that humans do on a daily basis. The proposed research is a necessary step to understand human visual search in the natural environment. Visual search also include life-critical tasks such as finding a tumor in a medical image or a potentially dangerous object in a baggage x-ray. This research will also have the potential to provide insights into the efficiency in life-critical search tasks. Finally, the research can potentially contribute to better assessments of the expected deficits of patients with attentional disorders (e.g., hemi-neglect) in naturalistic search behavior. The work will integrate research and educational activities at all levels (high school interns, undergraduate, and graduate) through the development of a website that enables students and researchers worldwide to enter experimental data through an internet browser and test the various computer models.

DESCRIPTION (provided by applicant): Synthetic cues (e.g., arrows and boxes) predictive of a target location speed up search times and result in increased decision accuracy. Similarly, when observers search in natural scenes, a highly visible object (e.g., house) that often co-occurs in natural environments with a sought target (e.g., chimney) will influence eye movements and facilitate search when the target appears close to the object. While the last few decades have seen single cell neurophysiology, human electrophysiology and neuroimaging lead to great advances in the understanding of the effects of attention and synthetic cues on neural activity, little is known about the underlying neural mechanisms mediating context effects during visual search in real scenes. Here, we propose to separately measure neural activity using functional magnetic resonance imaging (fMRI) while observers search for targets in real scenes and use neural decoding methods (multivariate pattern analysis) and a novel variation of population receptive field methods to: 1) Determine the brain areas (fMRI) that represent the spatial location of scene context and thus might mediate guidance of search in real scenes; 2) To evaluate whether the coding of scene context is automatic or whether it is modulated by top-down visual attention. The proposed work will improve our understanding of the neural mechanisms of scene context which arguably is one of the most important strategies used by observers to optimize visual search in natural environments. Our results will also advance our understanding of the function and role of brain regions related to attention, objects/scenes, and contextual associations for visual search. These advances might potentially help in identifying neural correlates of poor behavioral performance for patients with low-vision and attentional deficits in an ecologically important task such as visual search in real scenes.

? DESCRIPTION (provided by applicant): Medical image quality can be objectively defined in terms of diagnostic decision accuracy in clinically relevant perceptual tasks. Because of the high cost and effort involved in evaluating image quality using clinical studies, especially in early technological developments, there has been an ongoing effort to develop numerical algorithms (model observers) that can be applied to images to predict human accuracy in clinically relevant perceptual tasks. In recent years model observers have transitioned from laboratory investigations to actual tools used in technology development in the industry and for image quality evaluation by manufacturers to seek approval from the Food and Drug Administration. However, the recent increase of the use of 3D medical images (computed tomography, breast tomosynthesis, magnetic resonance) has motivated a need for the development of the next generation of model observers. A fundamental limitation of current model observers is that they disregard that the human brain processes an image with decreasing spatial resolution from the point of fixation. With 3D data-sets, radiologists rarely exhaustively fixate every region of every slice; instead, they process a significant portion of images with their retinal periphery which has drastically different visual processing. Increased computer power and recent advances in the understanding of the computational neuroscience of visual search provide the opportunity to develop the next generation model observers which potentially can more accurately characterize how radiologists scrutinize medical images, as well as their decision accuracy and errors. The current project proposes to develop the 1st model observer to emulate radiologists by processing medical images with varying spatial processing resolution across the human visual field, searching through the image with simulated eye movements, and reaching a decision through integration across fixations. The foveated search model, which makes eye movements unlike any previous model observer in medical imaging, will be the 1st model to emulate radiologists in making two distinct types of errors: search errors ( missed lesions that are not fixated) perceptual errors (missed lesion that are fixated). The decisions and eye movements of over twenty radiologists reading digital breast tomosynthesis (DBT) images will be compared to the newly proposed foveated search model and a comprehensive list of existing non-scanning and scanning model observers in what will represent the most extensive validation study to date of model observers with actual radiologists' decisions. The newly proposed model will be utilized to optimize DBT acquisition geometry and compared to use of current metrics of medical image quality. If successful, the newly proposed foveated search model will allow for more accuracy assessment of medical image quality, could be utilized to accelerate the evaluation of new technology, optimize parameters of current technology and gain a better understanding how radiologists search and reach diagnostic decisions.

Project Summary Early detection through screening mammography has decreased death rates from breast cancer. There are approximately 39 million mammogram procedures conducted each year in US. However, there are still alarmingly high error rates in radiological interpretations, with missed cancer rates ranging from 10-18 percent and false positive rates as high as 67% over a 10-year period. In order to reduce errors rates, digital breast tomosynthesis, a new 3D imaging technology intended to make cancers more visible to the radiologist, is rapidly being introduced throughout clinics in the US. However, there is no thorough understanding of the potential impact of these new 3D imaging technologies on radiological errors, and no knowledge of what eye movement strategies should be used by radiologists to minimize errors when searching through these volumes, while keeping manageable reading times. The current proposal combines expertise in medical image perception and state of the art vision science to increase the theoretical and empirical understanding of 3D search. To achieve such goal we aim: a) To understand how the types of errors detecting masses and microcalcifications are impacted by 3D search in digital breast tomosynthesis images; b) To gain an understanding of the functional impact on errors of adopting different eye movement strategies to search through 3D volumes; c) To develop a computational model of 3D search that includes foveated visual processing, scanning and drilling. The model will be used to assess the adequacy and efficiency of different eye movement strategies and to identify potential suboptimalities associated with an individual?s eye movement strategies or visual capabilities in the visual periphery. The psychophysical studies, eye tracking and computational models will be initially developed with trained non-radiologists, filtered noise and digital breast tomosynthesis phantoms. Subsequently, the findings and model will be validated with radiologists and real clinical images. If successful, the proposed studies will provide a new theoretical understanding of the types of radiological errors that occur and the functional role of search patterns on 3D search with digital breast tomosynthesis images, and provide computational tools to assess whether a radiologist?s eye movement patterns are well matched to their detection capabilities in their visual peripheral. Together, these advances can potentially help reduce errors in cancer detection. Although the proposed methodology is in the context of breast cancer and digital breast tomosynthesis, the principles investigated are potentially applicable to other areas of 3D medical images in radiology.

Face recognition is central for social interaction, and people with face recognition impairments su?er from di?culties that impact their personal and professional lives. Previous research has indicated that face recognition ability is strongly in?uenced by where on the face people look, and that this tends to be consistent for a given person. For example, some people always look between the eyes, while others look close to the tip of the nose or the mouth. It has also been shown that each person recognizes faces best when they look near the location they normally ?xate. These habits suggest that each person's face processing and eye movement systems have been mutually shaped, or tuned, to optimize recognition ability. However, little is known about how these factors contribute to di?erences in face recognition ability, how they change in adults, and how they develop in children. This project will address these fundamental questions, and could prove important in treating patients with deficits in face recognition.

The project will consist of three components: In the project?s ?rst component, the investigators will examine where adults between the ages of 18 and 50 usually look on faces and where their ?sweet spot? (optimal match between fixation and processing) lies. Preliminary results indicate that adults tend to look lower on faces as they age, indicating that the tuning of our face system changes even in adulthood. This study will examine whether this holds for a larger sample by comparing across ages and will explore the dynamics of gaze changes for individuals by measuring the same participants once and then again three years later. In the second component, the investigators will examine where people with developmental prosopagnosia (DP), a condition characterized by severe face recognition impairments, look on faces and whether they show a good match between where they look and their sweet spot. In the third component, the investigators will determine where children look on the face and how their tuning to faces changes as they age. It is expected that where children look will not change but that their face tuning will become sharper as they age. These possibilities will be tested by comparing children across di?erent ages and through longitudinal testing of the same children.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.