Aravind Joshi - US grants

Significant progress in computational neuroscience and neuroengineering requires a multifaceted interdisciplinary research program in several interrelated areas: Mathematical modeling and analysis of neural nets to better understand their collective behavior and capabilities for practical applications. Neurophysiological studies to better understand how retinal information is processed by neuronal assemblies in the striate and extrastriate cortex. Neural vision systems and their VLSI implementation for scene analysis and primitive extraction. Architectures and opto-electronic implementations of self-organizing neural nets partitioned into input/output and internal neurons for supervised and unsupervised learning with stochastic and deterministic state update rules. Higher order processing, in interconnected neural net modules utilizing, sequential and cyclic storage and recall, generalization, for multisensory data fusion and knowledge aggregation. Smart sensing and recognition from sketchy information with emphasis on object recognition including study of object representations that produce distortion invariant recognition. Highly structured associative memory and processing of spoken language. Study of optical materials and devices suitable for realizing artificial plasticity and learning specially in nets with unipolar binary neurons and ternary synaptic weights that facilitate opto-electronic implementations. The present proposal deals with studies to be carried out by a group of faculty with extensive expertise in the above areas, from the schools of Engineering and Medicine. Results of this research are expected to contribute to the development of a new generation of neuromorphic cognitive systems and to outperform more conventional approaches to signal processing. outperform more conventional approaches to signal and

This is a joint award with Dr. Vijayshanker at the University of Delaware. This team has proposed several major research tasks in natural language processing with special emphasis on several mathematical and computational aspects. The work clearly has a formal and mathematical character. The various computational structures and strategies the team has developed and will develop, need to be investigated mathematically because such investigations shed light on the descriptive and processing powers of these formalisms. These two aspects, i.e., the development of the structures and strategies, and their mathematical investigations, are very much interrelated. It is believed that natural language processing backed up by a formal framework, and mathematical investigations grounded in empirical studies are two very productive areas of research. The researcher will focus on interested in mathematical investigations only to the extent to which the results have important implications for natural language processing.

Constructing Science: Materials and Activities for kindergarten and First grade will use a collaboration of classroom teachers, science educators, university psychology and education researchers, and university scientists to develop, implement, evaluate and disseminate on a national basis model sets of instructional methods and materials for the active engagement of kindergarten and first-grade children in science learning and exploration. This process for curriculum development has been successfully piloted through a colloquium series initiated by the Citizens' Committee for Public Education in Philadelphia and facilitated by directors of PENNlincs with funding from PATHS/PRISM, ARCO and the Institute for Research in Cognitive Science, U of P. The proposed program will improve the quality of science education and at the same time ensure that more time is spent on science education in kindergarten. This project will bring developmentally appropriate science content and science processes to primary grade children in such a way (a) that scientists, educators and parents will be assured that children are truly learning important science concepts and processes, (b) that the methods have applicability to the realities of classrooms, and (c) that children's initial school experiences in science are rich and that children gain positive attitudes, motivation and vocabulary that will serve as a strong foundation for more formal science study. The methods will enhance any science curriculum available or under development.

This grant will fund a special conference session on 'world-situated language use in natural dialog', held in conjunction with the 14th Annual CUNY Conference on Human Sentence Processing. The conference, held March 15-17, 2001, at the University of Pennsylvania in Philadelphia, is the most prominent U.S. conference for the interdisciplinary study of human language understanding. On an annual basis, it brings together roughly 250 linguists, psycholinguists and computational linguists interested in detailed processing accounts of language comprehension and production.

The special session, entitled "Approaches to Studying World-Situated Language Use: Bridging the Language-as-Product and Language-as-Action Traditions," is designed as a step toward linking conversational/discourse research with the formal linguistic and mechanistic approaches typically found at the CUNY conference. Five prominent researchers working in this bridging area have been invited to give talks and participate in a panel discussion. In addition, peer-reviewed submitted talks and posters on this topic will be presented in accompanying sessions. It is hoped that by holding this symposium at the CUNY Conference, timely cross-disciplinary discussions will occur so to inspire a new generation of psycholinguistic and computational research on questions such as how natural utterances with disfluencies are processed, how information from context, gesture and linguistic input are combined in real-time processing, how interlocutors coordinate attention, and how these coordination processes impact real-time language processing commitments.

EIA-0224417
Aravind K. Joshi
Mark Liberman
University of Pennsylvania

CISE RR: Discourse Penn Trebank and Multimodal FORM: Development of Two Richly Annotated Corpora

This project, providing critical resources for research discourse modeling and conversational interaction, aims at developing new technologies and systems for information retrieval and human computer interaction. Centering on the construction of annotated corpora, two large-scale resources, one in the discourse domain and one in the dialog domain will be built:

1. Discourse Penn Treebank (DPTB) and
2. MultiFORM: Augmenting the FORM corpus with body movements, speech, and intonation.

The former project develops a large scale and reliably annotated corpus that will encode coherence relations associated with discourse connectives, including their argument structure and anaphoric links, thus exposing a clearly defined level of discourse structure and supporting the extraction of a range of inferences associated with discourse connectives. This annotation will be "on top of" the Penn Treebank (PTB) annotations as well as the predicate-argument annotations of PTB (called the Proposition Bank or Prop Bank). The latter involves a corpus of gesture-annotated videos, FORM that was designed to be extensible in order to eventually represent the entire multimodal experience of conversational interaction. This multimodal FORM , MultiFORM, will be created by adding body movement, speech and syntactic structure, and intonation. Large-scale annotated corpora have played a critical role in speech and natural language research by enabling large-scale integration of statistical knowledge (derived from the corpora) with linguistic knowledge (as represented in annotations) leading to scientific and technological advances. Representative examples constitute robust parsing and automatic extraction of relations and coreferences and their applications to information extraction, question answering, summarization, and machine translation. PTB, a resource developed a decade ago, represents an example of such a resource that impacts natural language processing worldwide. PTB deals with corpora at the sentence level warranting a new large scale and reliable discourse and dialog structure annotated corpora. Although intellectual and practical connections exist between studies of the structures of discourse and dialog, the initial requirements for resources to study these areas diverge while overlapping in conception. On the discourse side, we need for corpora that deals with the kinds of structures found in composed text such as journalistic articles. The dialog side needs to focus on interactions among people and on extemporized rather than pre-composed material.

EIA-0205456
Joshi. Aravind K
University of Pennsylvania

ITR: Language, Learning, and Modeling Biological Sequences

Recent significant advances in natural language processing such as the integration of grammatical and probabilistic machine-learning techniques have not been exploited for modeling biological sequences. These new techniques are highly relevant to the biological domain because they support the integration of sequence features at several scales, from dependencies between successive items through dependencies involving complex structures to overall sequence statistics. Hence, the major goals to be pursued are: (1) Development of new techniques for integrating grammatical and probabilistic information, in particular, integration and evaluation of grammatical, probabilistic, and approximate counting methods for fold prediction in secondary and tertiary structures of biomolecules. (2) Development and evaluation of probabilistic exponential models for gene finding, in particular genes for apicoplast-targeted proteins in eukaryotic human pathogens of the phylum `Apicomplexa'.

This research is highly interdisciplinary, involving the disciplines of computer science, biology and linguistics. It will have a significant impact on the modeling of biological sequences. It will also provide a wonderful opportunity to train new researchers to carry out this interdisciplinary research, thus contributing to science and mathematical education and human resource development.

The proposed research arose out of many discussions that took place at a landmark workshop on `Language Modeling of Biological Data' held at the University of Pennsylvania in February 2001.

Large scale corpora annotated at the sentence level have played a critical role in natural language research. They have enabled large scale integration of statistical knowledge (derived from the corpora) with linguistic knowledge leading to both technological and scientific applications, such as information extraction, question answering, summarization, and machine translation, among others. This approach is now being extended to the discourse level, thus going beyond the sentence level. Using a resource called the Penn Discourse Treebank (PDTB), a large scale corpus annotated with discourse structure along with the associated semantics, new major experimental work on discourse processing is being carried out, leading to the generation of more coherent summaries and texts, extraction of complex relations in texts, among others, as well as foundational research relevant to language technology. This work is also providing a deeper understanding of the relationship between sentence level and discourse level structures. While pursuing these goals, a variety of tools for making a productive use of the PDTB resource are also being developed. This research program is also coupled with a strong educational program involving training researchers in the PDTB methodology so that similar resources can be developed in other languages substantially divergent from English. This part of the research program has international components including collaboration with research groups in Czech Republic, India, and Finland. The international collaboration is funded by the NSF Office of International Science and Engineering.

Machine Translation, automated question answering, dialogue systems -- the many useful, emerging language technologies -- depend on recognizing patterns in text. Right now, the only patterns that can dependably be recognized are very local, no bigger than a sentence clause. Enabling patterns to be recognized across clauses in a text by identifying what links them and what the link conveys was the goal of the NSF-supported Penn Discourse TreeBank (PDTB), a nearly 1-million word text resource labelled with text-linking devices ("discourse connectives" and adjacency), the spans of text they link, and what the link conveys. In the five years since the release of the PDTB, computational linguistics researchers from around the world have used the format it pioneered,to develop similar resources for other languages and to use these resources for recognizing larger patterns in text. The current PDTB, however, lacks the full range of explicit and implicit text-linking devices in English and what they convey; the information which is badly needed by many forward-looking language technology applications. The goal of this project is to conduct research with the purpose to enrich the PDTB with these additional devices and to develop ways for authoritatively annotating other texts with similar information, but with less manual effort, as a basis for extending the range of texts whose larger, cross-clausal patterns can be recognized automatically.

This project is a response to calls (from both the language technology and computational psycholinguistics communities) for increased coverage and continuity of discourse relation annotation, both across and within the sentences of a text. To ensure a systematic annotation scheme grounded in evidence, the project starts by addressing some foundational questions about the properties of additional linguistic signals of discourse relations and how to capture these properties consistently and completely through manual annotation. From this follows systematic, evidence-grounded annotation of Entity Relations; constructions (other than discourse connectives) that reliably signal discourse relations; implicit intra-sentential discourse relations (building on PropBank annotation of the Penn TreeBank, and concurrent discourse relations (where implicit relations hold in addition to ones signaled explicitly). The project also explores the use of crowd-sourcing to support sub-tasks in discourse relation annotation that would lead to a reduction in the manual effort needed for expert annotation of other corpora, or enable large-scale experiments on aspects of human understanding of discourse relations. As with the Penn Discourse TreeBank 2.0, the enhanced corpus resulting from the project will be disseminated by the Linguistic Data Consortium (LDC), a well-established institution for world-wide distribution of language resources.