Richard Golden, Ph.D. - US grants

The goal of this project is to develop a nationally available fully automated web-based diagnostic system called ARCADE (Automated Reading Comprehension and Diagnostic Evaluation) that will be capable of assessing complex comprehension based upon student free response data. The technology used will be to combine information extraction (IE) technologies and advanced psychometric techniques in novel ways so as to provide detailed assessment and diagnostic information for use by teachers and students in the service of improving classroom instruction and learning. The ARCADE system will also contribute to a data infrastructure useable by other investigators. Specifically, computer scientists interested in the development of new information extraction technology and cognitive scientists and educators interested in the development of new theories of comprehension and assessment would be able to access the database that will be developed in the course of using ARCADE with students. Thus, in addition to improving educational effectiveness in classrooms on a national level, the ARCADE system has the potential to provide a nation-wide resource for facilitating the advancement of scientific research in both the fields of reading comprehension and information technology research.

The specific research being pursued is to develop and empirically test the core mathematical algorithms of the ARCADE system with respect to their reliability and validity for assessing reading comprehension in foundational literacy in science and literature. The empirical database will consist of free response data generated by student examinees in grade school and junior high school classroom settings in response to open-ended probe questions. The core ARCADE system employs innovative combinations of information extraction and psychometric techniques to address a critical educational need, namely, ways to assess multiple dimensions of complex comprehension. Such dimensions of comprehension are specified by a set of special semantic networks (called "knowledge digraphs") which embody meaning relations among ideas in texts and documents as well as relations to prior knowledge and inferences. A new statistical model of examinee behavior is then defined which incorporates techniques from the fields of Item Response Theory (IRT), Hidden Markov Model IE technology, and Knowledge Digraph Contribution analysis. The important innovation of this new statistical model is that multiple dimensions of comprehension in conjunction with their respective standard errors can be directly estimated from examinee free response data using Monte Carlo simulation and econometric methods Moreover, using an approach analogous to that developed in IRT , these assessments of comprehension dimensions can be mathematically proven to be reliable across a given family of equivalent testing materials.

This award will provide partial support for the 41st Annual Meeting of the Society for Mathematical Psychology to be held in Washington, D.C. on July 26-29, 2008. The Society for Mathematical Psychology conference encourages the presentation of research in which mathematical, statistical, or simulation methods play a significant role in the development of hypotheses or the interpretation of experimental results in the behavioral, neural, and cognitive sciences. In addition, accepted research papers focus on theoretical developments clearly relating to substantive issues or methodologies of obvious use in psychology, cognitive science, cognitive neuroscience, and related areas and/or experimental results which bear directly on particular mathematical or simulation models of aspects of human behavior. This year's conference themes include: (1) cognitive decision theory; (2) causal modeling; (3) computational linguistics; and (4) psychometric assessment.

Mathematical psychology brings together social scientists, statisticians, mathematicians and computer scientists to work on problems critical to behavioral scientists. Much transformational research has come from the mathematical psychology community, including mathematical models of brain function, memory, and decision-making, as well as the introduction of new methods for data analysis. While mathematical models improve our understanding of human behavior and provide formal structures for future scientific exploration, new and better methods for data analysis allow us to derive more accurate and nuanced conclusions from behavioral data. The annual meeting of the Society advances discovery and understanding. It promotes training and learning of new models and methods for analyzing behavioral data and it makes possible the broad dissemination of new findings important in all areas of psychology, as well as economics, political science, and sociology.