Shihab A. Shamma - US grants

The objective of this research is to develop a speech processing scheme based on new models of the auditory system. The results are expected to be applied in the analysis and automatic recognition of speech. This may lead to better speech processing for auditory prostheses.

In prior work, the P.I. has developed and tested a detailed quantitative model of how speech information is processed in the cochlea of mammals. He has studied how this information, after processing, is allocated with speech sounds, as recognized by humans. In this project, the P.I. will go on to apply various learning algorithms to processed human speech, so as to demonstrate the value of this processing in speech recognition applications. Innovative learning algorithms exploiting the true series nature of speed data will be explored, along with more clinical biologically-motivated learning rules.

9720334 Krishnaprasad The goal of this research project is to investigate time coding in the central nervous system, specifically the auditory system of the barn owl, the early development of such codes, the learning of associated maps, and the exploitation of such sound codes and maps in source localization and sound separation. The approach consists of electrophysiological and anatomical study, coupled with mathematical modeling of neural circuitry, the rigorous investigation of the structure and performance of relevant learning algorithms and the creation of an experimental robotic testbed. This testbed, a binaural head, is intended to be capable of orienting itself to sound sources in complex acoustic environments through pure auditory servoing, by utilizing the development of control architectures capable of learning maps of the auditory space of the robot, and drawing upon an evolving understanding of barn owl auditory system. The results of this research will provide insights into the design of novel roles for auditory sensing, interpretation and discrimination in autonomous robotic systems. This research could lead to applications in hands-free human-machine communications in acoustically cluttered environments and in monitoring complex environments such as highly automated manufacturing plants.

IBN 98-03836 Shihab Shamma, P.I. This project is a series of annual 3-week hands-on workshops held in Telluride, Colorado. Neuromorphic engineering is a discipline which reverses the usual thrust of computational neuroscience, orienting itself toward the derivation of engineered devices from an understanding of the design principles of the nervous system. In so doing, it is having a unique beneficial effect on neurobiology research. On the one hand, it represents a departure from the usual engineering principles based on linear analog devices or unabashedly digital ones (e.g. most modern computers), and hence is contributing to development of robots which perform smoothly under highly variable external conditions the way biological organisms do. On the other hand, it drives biological research because biologists typically cannot supply (and may tend to avoid) the level of quantitative detail of questioning asked about the way the nervous system is designed. The workshop involves tutorial-style lectures on general as well as practical topics in neuromorphic engineering, practical courses on VLSI, circuits, "floating gate" technology; both short and long hands-on projects (e.g building robots with neuromorphic control circuits). Funding is partially supporting travel for faculty and students for a total of around 60 persons (faculty, students, staff).

Although large parts of our brains are devoted to the processing of sound cues and sound plays an important role in the way we interface with the world, this rich channel has not been extensively exploited for displaying information. The mechanisms by which received sound waves are processed neurally to form objects with auditory properties in many perceptual dimensions, including three corresponding to the source location (range, azimuth, elevation) and three to qualities ascribed to the source (timbre, pitch and intensity), are beginning to be understood. There has been significant progress over the last decade in understanding the mechanisms by which acoustical cues arise and how the biological system performs transduction and neural processing to extract relevant features from sound, and in the way we perceive and organize objects in acoustical scenes. Our goal is to exploit this understanding, and uncover the scientific principles that govern the computerized rendering of artificial sound scenes containing multiple sound objects that are information and feature rich. We will test, use and extend this knowledge by creating auditory user interfaces for the visually impaired and the sighted. The work aims both at developing interfaces and answering fundamental questions such as: Is it possible to usefully map "X" to the auditory axes of a virtual auditory space? Here "X" could be an image (e.g., a face), a map, tabular data, uncertain data, or temporally varying data. Are there neural correlates that can guide natural mappings to acoustic cues? What limitations does our perception place on rendering hardware? How important is

DESCRIPTION (provided by applicant): It is well established that auditory experience can cause significant and continuous reorganization and receptive field plasticity in the primary auditory cortex (AI). The exact form of this plasticity depends on the details of the behavioral context, and of the spectral and temporal cues in the acoustic stimuli. Recent findings indicate further that neuronal responses in AI of awake behaving animals reflect motor, attention, and reward dimensions, rather than simply encoding the acoustic features of the stimuli. This is consistent with findings in other neural systems and supports the hypothesis that auditory cortical cells may undergo rapid, short-term, and context-dependent changes of their receptive field properties when an animal is engaged in different auditory behavioral tasks. This kind of plasticity would likely involve a selective functional reconfiguring of the underlying cortical circuitry to sculpt the most effective receptive field for accomplishing the auditory task. The proposed research explores this hypothesis in combined physiological/behavioral experiments in which spectroternporal receptive fields (STRFs) will be rapidly and comprehensively characterized simultaneous with the animal behavior. The experiments also contrast STRF plasticity in a single cell across different auditory tasks employing various acoustic signals with controlled spectral and temporal features. These unique capabilities are made possible by the development of new ripple stimuli to engage the animal behaviorally while rapidly measuring its cortical STRFs, and versatile response indicators that are sensitive barometers of how behavior affects a cell's responsiveness. The overall goal of the proposed study is to test whether it is possible, under the same reference-target discrimination paradigm, to induce rapid STRF plasticity along specific temporal or spectral dimensions by choosing the appropriate cues in the acoustic target. All experiments follow the same behavioral procedure in which reference signals are broadband, temporally and spectrally rich stimuli that also serve during physiological experiments to characterize the STRF of the cell under study. By contrast, the target provides the discrimination signal. It varies from one experiment to the other with distinctive features that can be (AIM I) purely temporal, (AIM II) purely spectral, or (AIM III) combined spectrotemporal.

[unreadable] DESCRIPTION (provided by applicant): Hearing engages, in a seemingly effortless way, complex processes and transformations collectively known as auditory scene analysis, through which the auditory system consolidates acoustic information from the environment into perceptual and cognitive experiences. The project proposed here explores a fundamental perceptual component of auditory scene analysis called auditory stream segregation. This phenomenon manifests itself in the ability of humans and animals to attend to one of many competing acoustic streams even in extreme noisy and reverberant environments - also known in the literature as the "Cocktail Party Problem". While completely intuitive and omnipresent in humans, mammals, birds, and fish, this remarkable perceptual ability remains shrouded in mystery. It has been rarely quantified in objective psychoacoustical tests or investigated in non-human species, and seldom explored in physiological experiments in humans or animals. Consequently, the few attempts at developing computational models of auditory stream segregation remain highly speculative, and lack the perceptual and physiological data to support their formulations. This in turn has considerably hindered the development of such capabilities in engineering systems such as in automatic speech recognition or the detection and tracking of target sounds in sensor networks. [unreadable] The proposed research seeks to develop a computational model of auditory scene analysis that accounts for perceptual and neuronal-findings of auditory stream segregation. The intellectual merit of this work is providing a rigorous framework for the design of new psychoacoustic and physiological experiments of streaming, and for developing effective algorithmic implementations to tackle the "cocktail party problem" in engineering applications. The proposed research project draws upon the expertise of neurobiologists, psychoacousticians, and engineers in integrating psychoacoustic, physiological and computational techniques. The broader impact of this effort is in providing versatile and tractable models of auditory stream segregation, and hence significantly facilitating the integration of such capabilities in engineering systems, such as in automatic speech recognition or the detection and tracking of target sounds in sensor networks. This project will also provide a rigorous foundation for the design and generation of new hypotheses in order to better understand the neural basis of active listening [unreadable] [unreadable]

Project Summary The goal of this research is to explain how the auditory system analyzes complex acoustic scenes, in which multiple sound ?streams? (e.g., concurrent voices) interact and compete for attention. Synergy is created by combining 1) simultaneous behavioral and neurophysiological measures from primary and secondary auditory cortex in ferrets with 2) comparable behavioral and electroencephalographic (EEG) measures in humans within 3) the theoretical and computational framework of the temporal coherence hypothesis. Specific Aim 1 involves recordings of single-unit cortical responses while ferrets segregate speech mixtures and detect the presence of a target word spoken by the target talker. The experiments will be paralleled by tests of human streaming of speech sounds that are comprised of natural mixtures of voiced (harmonic) and unvoiced (noise-like) sounds. Specific Aim 2 explores the role of coherence and attention in stream binding and segregation using stimuli with simpler spectral characteristics, e.g., pure-tones and noise sequences. The goal is to test strong predictions of the temporal coherence hypothesis with respect to the effects of temporally synchronous, alternating, or overlapping sound sequences. Specific Aim 3 extends this approach to higher-level attributes of more complex stimuli, such as pitch and timbre, which are more ecologically relevant. Here we examine the role of pitch and timbre in characterizing the continuity of a stream, and binding the elements of the stream in both ferrets and humans. This research has direct and significant health implications, because one of the most common complaints of hearing-impaired individuals (including wearers of hearing aids or cochlear implants) is that they find it difficult to separate concurrent streams of sounds, and to attend selectively to one of these streams (such as someone's voice) among other streams. A clearer understanding of the mechanisms underlying the perceptual ability to separate, and attend to, auditory streams will likely lead to a clearer understanding of the origin of these selective-listening difficulties, and it may inspire the design of more effective sound-separation algorithms for use in hearing aids, cochlear implants, and automatic speech recognition devices.

The Telluride Workshop on Neuromorphic Cognition Engineering
Neuromorphic engineers design and fabricate artificial neural systems whose detailed architecture, design, and computational principles are based on those of biological nervous systems. Over the past 12 years, this research community has focused on the understanding of low-level sensory processing and systems infrastructure; efforts are now expanding to apply this knowledge and infrastructure to addressing higher-level problems in perception, cognition, and learning.
The annual three-week intensive Workshop (held in Telluride, Colorado) consists of background lectures (from leading researchers in biological, cognitive, computational, engineering and learning sciences), practical tutorials (from state-of-the-art practitioners), hands-on projects (involving established researchers and newcomers/students), and special interest discussion groups (proposed by the workshop participants). For researchers in this community, this is the premier workshop for training students, initiating collaborations, and in-depth discussions on scientific issues.
In this workshop and through the Institute for Neuromorphic Engineering (INE), the mission is to promote interaction between senior and junior researchers; to educate new members of the community; to introduce new enabling fields and applications to the community; to promote on-going collaborative activities emerging from the Workshop, and to promote a self-sustaining research field.
Specific Goals for the period of 2007-2012: While there is no question that the Workshop has been very successful in its mission, three new challenges have been identified for the Workshop: 1) with a rapidly expanding community in both the U.S. and Europe, the Workshop experience needs to reach more people without increasing the size of the Workshop, 2) as larger and more challenging projects are tackled, more opportunities for group interactions are needed throughout the year, and 3) as more complex questions are asked at the system-level, more voices from cognitive neuroscience are needed.
To meet these new challenges, a new version of the Workshop is envisioned with: 1) an expanded theme to focus on Perception, Cognition, and Learning, 2) an expanded constituency, educational mandate and research focus to incorporate members of the NSF Science of Learning Centers (SLC), 3) to create a two-part Workshop series (to allow yearlong collaborations and deeper investigation into large scale projects), one held in the U.S. and funded by U.S. resources and the other held in Europe and supported by European resources and 4) a modified Workshop schedule to emphasize training at the beginning of the workshop to provide a needed focus on education for both beginners and experts alike. The infusion of new researchers (from the SLCs) that focus on learning at multiple scales (from synapses to classroom) will provide the needed knowledge, new collaborations, and new perspectives to move the community towards cognitive-level neuromorphic systems.
Broader impact of the Workshop to the public: The Telluride Neuromorphic Cognition Engineering Workshop will continue its tradition of public interaction. In particular, there will be a continuation of the educational program for K-12 students (based on neuromorphic/robotics design kits), undergraduate and graduate students (Workshop courses, new classes/lectures at participants? universities and REU), and to established researchers (exposure to new areas in the field). The workshop will also continue to educate the Telluride community with public lectures on the latest developments/issues in the field.
Recruitment of minorities and women to the field will be continue by organizing lectures at various Universities, particularly HBCUs (Morgan State U., MD, Lincoln U., PA, Morehouse College, Atlanta, GA, and others). By sending presenters to institutions local to their home universities, minimal funding will be required and provide the most likely connections for future collaborations. The Institute for Neuromorphic Engineering, currently housed at the University of Maryland (College Park, MD), will arrange the logistics. The lectures and other teaching materials developed at the workshop will also be made available to all interested parties and posted on the INE website.
Lastly, the workshop will continue to develop the researchers and leaders for the emerging field of biologically-inspired systems, cognitive/learning systems, robotics and implantable electronics. Various agencies and governments have recognized that smart devices (such as interactive humanoid robots) that mimic living organisms will have great academic and commercial value in future.

Project Summary Auditory experience can reshape cortical maps and transform receptive field properties of neurons in the auditory cortex of the adult animal. The exact form of this plasticity depends on the behavioral context, and the spectrotemporal features of the salient acoustic stimuli. This has been shown by combined physiological and behavioral approaches in our previous experiments in which on-line spectrotemporal receptive fields (STRFs) were rapidly and comprehensively characterized simultaneous with the animal behavior. The experiments also contrasted STRF plasticity in single cells across different auditory tasks employing various acoustic signals with controlled spectral and temporal features. These results are consistent with findings of adaptive plasticity in the motor and other sensory systems and support the hypothesis that auditory cortical cells may undergo rapid, context-dependent changes of their receptive field properties when an animal is engaged in different auditory behavioral tasks. This kind of plasticity would likely involve a selective functional reshaping of the underlying cortical circuitry to sculpt the most effective receptive field for accomplishing the current auditory task. What are the underlying mechanisms that give rise to this extraordinary functional plasticity? The goals of the proposed research are to extend our studies of task-related plasticity to higher order auditory cortical areas, and to a variety of new tasks (including temporal and spectrotemporal tasks), new behavioral paradigms (utilizing either positive or negative reinforcement) and furthermore, to investigate the possible role of top-down signals from frontal cortex in modulating adaptive plasticity in the primary auditory cortex. We propose to rigorously test the hypothesis that frontal cortical neurons encode task rules, expectancies, goals and the task-related meaning of acoustic stimuli, and further, that when an animal performs different auditory tasks, top-down influences from frontal areas contribute to the induction of rapid adaptive plasticity in auditory cortex, that reflects both the nature of the stimuli and goals of the tasks. We shall also combine our physiological experiments with microstimulation in FC and other areas to test if that modulates responses and receptive fields. Our preliminary studies from simultaneous neuronal recordings of single units and local field potentials in auditory and frontal cortex have already lead to exciting new insights, that may lead to progress in understanding the interactions within an extended neuronal network that give rise to adaptive plasticity.

This INSPIRE award is partially funded by the Science of Learning Centers Program in the Division of Behavioral, Cognitive and Social Sciences in the Directorate for Social, Behavioral and Economic Sciences; the Perception, Action, and Cognition Program in the Division of Behavioral, Cognitive and Social Sciences in the Directorate for Social, Behavioral and Economic Sciences; the Energy, Power, and Adaptive Systems Program in the Division of Electrical Communication and Cyber Systems in the Directorate of Engineering; and the Applied Mathematics and Mathematical Biology Program in the Division of Mathematical Sciences in the Directorate for Mathematical and Physical Sciences. This research project draws on knowledge from many disciplines (neuroscience, cognitive science, computational science, mathematics and engineering) to create cognitive systems capable of interpreting observed, complex human movements and actions. New design methodologies will be developed for the integration of sensory modalities (vision, audition, touch) and their support of higher cognitive function (language, reasoning). In contrast to existing approaches which tend to be assemblies of modular components each solving its task in isolation, this team takes a novel approach called Active Cognition which has the following features: 1) Instead of modeling the different perceptual processes (vision, audition, and haptics), cognition, and motor control in isolation, the modules are integrated and capabilities co-developed in the tradition of dynamical systems theory to obtain a reasoning system where "the whole is greater than the sum of its parts"; 2) instead of segregating the low level processing of signals from the processing of higher level symbolic information, they will interact in a continuous dialogue, such that high level knowledge will leverage perception; and 3) instead of separating physical embodiment from algorithmic considerations, biologically inspired real-time hardware will be developed that implements complex functions by integrating signals and symbols. The project is organized in two working groups. The first group will develop a cognitive robot that can recognize complex human activities using visual and auditory signals captured by biological-inspired hardware. The second group will study attention in humans by measuring human response to audition and vision through EEG and MEG, and subsequently implementing the findings in robots. A yearly three-week, hands-on workshop will educate students, serve as testing ground for the team's ideas, and stimulate new collaborations. This workshop will also engage the involvement of the interdisciplinary research community that has formed around the goal of building biologically inspired cognitive systems.

Success in integrating different components of a cognitive system (hardware, sensors, and software) has the potential to catalyze a new industry of biologically-inspired cognitive systems, including household and service robots, and systems for intelligent transportation and smart manufacturing. In addition, this interdisciplinary project will play a significant role in building capacity for a new emphasis area in engineering and training of cognitive systems engineers who need combined expertise in computer science, electrical engineering and cognitive neuroscience.

The motivation for this biologically-inspired approach is to design systems that perceive and act in cluttered and noisy scenes that they have never experienced. This stands in contrast with the state of the art in computational engineering systems that need to be re-trained each time they confront an unanticipated environment. The main reason is that current approaches to perception address specific problems in isolation and do not consider that the primary role of perception is to support systems with bodies in action. As a result, they are constrained to the situations for which they were trained and cannot react to changing tasks and scenes. By focusing on cognition primitives rather than specific applications, the work is expected to greatly advance the state of the art of machine perception and lead to the development of systems that can robustly and on-line adapt to new environments, react to novel situations and learn new contexts. To do so, novel theoretical formulations of perception and action and high-speed, low-power, hardware implementations with on-line learning capabilities will be studied while assimilating new insights from the neurosciences. Consequently, this work will network neuroscience, cognitive science, applied mathematics, computer science and engineering so as to lower one of the few remaining barriers that keeps interactive robots in the realm of science fiction. Beyond the scholarly contribution, the work is expected to provide know-how for the design of systems with adaptive perception in a modular fashion with reusable components. Such systems have applications in computational vision and auditory perception problems and can advance the industry of cognitive biologically-inspired robotics and assistive devices.

This proposal sets forward novel ideas in the design of intelligent perceptual systems and the development of synthetic intelligence. Just about any task which an intelligent system solves involves the interplay of four basic processes that are devoted to: (a) context, (b) attention, (c) segmentation and (d) categorization. The members of the proposed network will study these canonical cognitive primitives by combining neural modeling with neural and behavioral experiments, theoretical and computational modeling and implementation in robotics. The findings of theoretical insights will then be adapted to satisfy the demands of realistic behavior, and to develop technological solutions for applications of robust and invariant perception and action. The proposed collaborative network will consist of a small science and engineering research team to directly address the questions in robust adaptive perception and action. It will then direct personnel, and inject results and pedagogical content to a Summer Workshop that aims to include a global network of researchers.

Project 1 Abstract The focus of this project is Age-related hearing loss (AHRL, presbyacusis), a widespread condition due to age related changes along the auditory pathway. Great strides have been made elucidating the peripheral changes, but age related changes in the central nervous system remain largely unknown. The project revolves around the hypothesis that cortical plasticity can play an effective role in maintaining effective auditory function in old age, by countering diminished synaptic depression and rewired inhibitory circuits. The project describes a series of behavioral and neurophysiological/imaging experiments in aged animals, which explore the physiological and behavioral correlates of ARHL and propose experiments to test how cortical plasticity could be directed to counter it. In pursuit of these goals, behavioral, neurophysiological, and imaging approaches are combined to characterize the encoding of speech in noise in aged animals and characterize how cortical plasticity which enhances the robustness of speech perception in noise in the young is inefficient in the aged. The project then goes on to study the cellular mechanisms that fail with age, and develop therapeutic strategies through plasticity induced by engagement of cognitive functions. The project exploits many technologies to achieve its goals such as single-unit recordings, in vivo 2-photon imaging in passive and the behaving animals and in vitro circuit mapping approaches. The final aim of the project is to develop remedial interventions and adaptive training procedures to reverse some of the age-related changes. 1

The objective of this Research Coordination Network (RCN) is to advance understanding of how biological systems learn complex symbolic signals, and create artificial systems with similar capabilities. By defining a common framework to describe these signals, and their variability across space and time, the RCN will develop methods and tools applicable to a wide range of domains, including language, music, action, perception, and navigation. The RCN will build upon research in Neuromorphic Engineering and its development of bio-inspired, low-power computing platforms, sensors, and signal processing. Using these tools, the RCN will focus on high-level cognitive functions, to create complex, bio-inspired systems that learn through engagement in tasks. The network will bring together neuroscience, cognitive science, applied mathematics, computer science, and engineering, with emphasis on machine learning and artificial intelligence. Network members will participate in a yearly three-week, hands-on workshop, that will develop and test new tools and ideas, stimulate new collaborations, and educate students on unique interdisciplinary skills.

The RCN will facilitate interactions and collaborative projects among participating researchers employing a wide range of paradigms that specifically deal with three thrusts: the role of neural plasticity for learning symbolic systems; the adaptive mechanisms underlying the learning of sensory-motor tasks; and transitioning to real-world applications such as automatic speech and dynamic scene understanding, neuromorphic hardware implementations, cognitive computational algorithms, and databases acquisition. Specific examples of such diverse projects include brain process models that assess learning and expertise; algorithms, based on physiological or abstract events, that process input from neuromorphic hardware; and development of software and neuromorphic hardware for signal interpretation and action execution.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Artificial intelligence is becoming ubiquitous in modern life. To build systems under the current paradigm, large amounts of energy are required for computing and sensing. This causes environmental problems, pollution, and challenges for small-sized systems, as well as privacy issues. The field of neuromorphic science and technology offers an alternative by seeking to understand principles of biological brains and build on their basis artificial systems using low-power hardware and software solutions. While its advantages have been demonstrated, further advances are necessary and will require common computational tools and principled experimental approaches. This AccelNet project, NeuroPacNet, links international experts in neuromorphic engineering with computational neuroscientists, roboticists, control theorists, and researchers of perception from seven global networks to set the foundations for building systems that can robustly process real-world signals in time and adapt to changes. This network of networks will facilitate the development of new methods and approaches for intelligent system design and prepare the next generation of leaders in neuromorphic science and technology. As different industries adopt neuromorphic hardware, society will have access to new applications, such as in computing on cell phones, neuroprostheses, intelligent hearing aids, and smart sensory systems with predictive capabilities.

NeuroPacNet will advance computational research on modeling the integration of perception, action, and cognition. The network of network will coordinate across those research thrusts and develop new approaches grounded in theoretical neuroscience for sensorimotor control, motor learning, event-based computations, and learning in spiking neural networks. NeuroPacNet will also include robotics research in the areas of drone navigation and human activity understanding for humanoids and will address social and ethical issues in humanoid robotics. The network of networks will use innovative hardware design and mixed signals computational systems to address computation for emerging and unconventional technologies. International collaboration and knowledge exchange will include an immersive research exchange program providing scholarships to students and postdoctoral researchers, an annual workshop to discuss common issues and concerns in a stimulating environment and to engage in hands-on projects, meetings to define challenges, opportunities, and actions to accelerate progress, and competitions with two challenges to be solved by teams of researchers and students. An interactive project website will become a portal for archived webinar talks, tools, and data.

The Accelerating Research through International Network-to-Network Collaborations (AccelNet) program is designed to accelerate the process of scientific discovery and prepare the next generation of U.S. researchers for multiteam international collaborations. The AccelNet program supports strategic linkages among U.S. research networks and complementary networks abroad that will leverage research and educational resources to tackle grand scientific challenges that require significant coordinated international efforts.

Co-funding for this project is provided by the Directorate for Social, Behavioral, and Economic Sciences.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.