Jeffrey N. Rouder - US grants

This project seeks to develop a class of statistical models for the analysis of data having skewed distributions, especially data arising from hierarchical or multi-level settings. Skewed distributions are ubiquitous in the social sciences. Often, the higher-order characteristics of the distribution, such as the scale (variability) and shape, can provide important insight into substantive issues and provide for significant theoretical development. In addition to having skew, these distributions typically have variability at several levels. For example, businesses may be clustered by economic sectors and completion time data may be clustered by participant. Data in these contexts often are analyzed with linear models such as regression or ANOVA. Although these methods can account for the hierarchical nature of the data and often are well-suited to analyzing differences in means, it is difficult, if not impossible, to perform inference on higher order characteristics. The researcher team will develop a Bayesian approach to analyzing a broad class of models in which statistical inference about location, scale, and shape is both possible and practical. Bayesian statistics is adopted because it is ideally suited to hierarchical models. Bayesian analysis depends on the researcher's informed knowledge of experimental conditions -- the "prior distribution." In some cases, Bayesian analysis is relatively insensitive to this prior; however, in other cases subtle errors in prior specification can lead to erroneous inference. For these reasons, the research team will develop appropriate "noninformative priors." The project will produce software tools so that other researchers can perform Bayesian analysis on these hierarchical models.

In the social sciences, researchers have a well-developed set of statistical tools for analyzing the overall effects of manipulations on outcomes. For example, experimental psychologists study how practice (a manipulation) improves performance (an outcome). Current statistical tools are well-suited for assessing the overall (e.g., average) improvement with practice but are ill-suited for assessing whether practice affects the variability of performance or the skew in the pattern of performance (skew would occur if performance is good on many trials and poor on a few). The goal of the project is the development of statistical tools for assessing differences in variability and skew of outcome measures as well as overall effects due to manipulations. The results will lead not only to better understanding of the data but, more importantly, to better theoretical development. For example, learning theories which predict that practice affects the variability of performance can be rigorously tested. The developed statistical tools would be broad and applicable to many social science fields such as psychology, education, economics, and other social sciences.

This project will develop new methodology in Bayesian hierarchical analysis in order to provide efficient inference for a number of nonlinear models of mental processing. Specific methodological goals include: (1) developing new semiparametric methods and (2) developing and implementing model selection strategies and Bayes factors appropriate for the proposed Bayesian hierarchical models. New semiparametric methods will enable psychologists to critically examine proposed models as well as analyze data without making undue assumptions. Model selection methods and Bayes factors in particular are Bayesian analogues to hypothesis testing and are crucial tools for verifying or disproving psychological models. The new methodology will be applied in two substantive domains of experimental psychology: (1) the measurement of learning or skill acquisition curves, and (2) the assessment of conscious and unconscious influences in memory. This basic methodological research, along with the proposed applications, will provide the tools for better understanding of learning and memory.

Experimental psychology has provided profound insights into the nature of perception, learning, and memory. The current research addresses a flaw in experimental practice. As psychological theories progress, they tend to become nonlinear. Unfortunately, unmodeled variance in nonlinear settings generally distort inference, making the link from data to theory tenuous. Psychological research is characterized by several sources of variance, including those from the selection of participants, test items, and moment-to-moment fluctuations in performance. The presence of these distinct sources presents a significant challenge to nonlinear theory testing. The project will develop new statistical tools for modeling variability at several levels in specific, pertinent, nonlinear models of psychological process. The new methods will then be used to address several long-standing controversial issues in how learning occurs and how memory operates.

DESCRIPTION (provided by applicant): Theories and models in cognitive psychology tend to be nonlinear, e.g. Process Dissociation Model of Memory, but statistical methodology for assessing these theories is based on linear models. As a consequence, psychologists using nonlinear models have no suitable means of accounting for extraneous variability from the selection of items and participants. In these nonlinear contexts, unaccounted variability often leads to asymptotic bias in estimation and may lead to flawed inference in hypothesis testing. To fill this void, we propose a series of hierarchical nonlinear models to capture psychological processes of interest. Although models are custom-tailored for specific applications, the form of these models and the corresponding analytic techniques will have broad applicability across experimental psychology. Our overall strategy is to place linear models on parameters in nonlinear processes. For example, to account for item and participant variability in a signal detection analysis of memory performance, we assume that each individual and each item have separate effects on sensitivity. These items and participant effects are assumed to be random effects from parent distributions and modeled accordingly. The overall benefit is accurate estimation and vastly improved inference. In the course of specifying these models for psychological process, we necessarily make significant progress in Bayesian methodology, most notably in improving mixing and implementing Bayes Factor computations in hierarchical models with non-informative priors.

The project will investigate two distinct but related questions in psychological research. The first concerns response time (RT), the time taken to complete an experimental task. The shape of the RT distribution serves as a key marker of cognitive processing. Research will develop statistical methodology for testing whether shape is invariant or depends on covariates such as participant characteristics and experimental manipulations. The approach will be to choose appropriate distributions from a large class of three parameter distributions derived from the two-parameter exponential families introduced by Bar-Lev and Reiser augmented with a shift parameter. A unified approach will develop objective Bayesian methodology for this class of nonregular distributions. The second key research question is about the association of latent mental processes across people, items, and conditions. Understanding how these processes are related will provide insight into understanding cognition. The specific problem is to model the association of covariance matrices of two or more related bivariate distributions in a hierarchical setting. The project will develop objective priors for Bayesian analysis of these covariance matrices, generalizing recent developments in univariate signal-to-noise ratio priors.

Both phases of the project address fundamental and significant questions in cognitive research in psychology, with potential impact in related areas as well. The study of response times also is fundamental to research in developmental and social psychology as well as psychopathology. A number of theories have been developed for cognitive processing in these fields and how it is affected by experimental conditions and characteristics of the participant. Most of these theories predict shape changes in the distribution of response times. However, the fundamental question of whether or not the shape of these distributions actually changes as a response to experimental condition has not been studied. Understanding if and how response time distribution shape changes will spur new theoretical directions. In addition, useful new statistical methods will be developed applicable beyond psychology. The second problem, estimating covariance matrices of latent variables, is motivated by the study of different modes of recall in memory tasks. Participants given lists of words to study and subsequently queried on these words may respond because of an "automatic" response, based perhaps on previous familiarity, or they may respond based on actual recollection. Assessment of these two processes is complicated by the fact that some words may be simultaneously easier to recall automatically or easier to remember; similarly, people may tend to be better simultaneously at automatic response or recollection. Assessment of these relationships is delicate and challenging. The research will have impact on the psychological study of working memory and cognitive aging. In addition, the statistical models are closely related to those used in areas such as epidemiology, economics, and ecology. Thus the results of the project will have impact well beyond the psychological sciences.

This project will focus on developing Bayes factors, a methodology that is an alternative to traditional statistical testing. Bayes factors have a natural advantage over classical frequentist hypothesis testing methods in that they can assess the evidence both for and against a null hypothesis. Bayesian analysis, however, relies on a degree of prior information supplied by the investigator. This prior specification can be viewed as representation of the investigator's belief in the state of nature before collecting data, and this degree of subjectivity has been a major criticism of Bayesian analysis. The project will develop so-called "default" methods that both minimize the reliance on this subjectivity and also provide optimal statistical properties in testing both for and against the null hypothesis. The theoretical study of Bayes factors will guide the choice of default prior.

The physical sciences have made gains by demonstrating that certain relationships hold across all conditions. These kinds of relationships can be termed invariant. Some social and behavioral sciences, in contrast, traditionally discover new theories by demonstrating that experimental manipulations produce altered responses rather than by proving that the response is unchanged. Conventional statistical methodology has been developed to prove that responses are not invariant to stimuli, but these tools are ill-suited to proving invariance. In addition to the development of a Bayesian alternative to classical statistical testing, the project will develop software and make it available through web applets so that researchers can easily use the new statistical tools. It is anticipated that these new developments will make Bayes factors useful and common in a number of fields, including epidemiology, economics, psychology, wildlife, and biology.

One of the primary tools in the investigation of perception and memory is the Receiver Operating Characteristic (ROC), which is a plot of hits ('yes' responses on target trials) vs. false alarms ('yes' responses on non-target trials). Researchers commonly claim that the details of these curves provide insight into cognitive phenomena of interest. It is not clear, however, that ROCs are sufficiently rich so as to allow differentiation of processes. The basic idea to be explored is whether ROCs collected across disparate domains have the same pattern, that is, can all be explained by common processing, or have different patterns, that is, can support rich theories of processing. Based on our observations, we will develop flexible, nonparametric, model-free tests of ROC pattern equivalence. These tests will allow us to assess whether ROCs can be used to make detailed inferences about processing.

Understanding human memory has many practical consequences in education, professional training, health delivery, and even in legal settings. Memory researchers have posited many two-process models to account for their results, including 'remember-know', 'automatic-controlled', and 'unconscious-conscious'. Unfortunately, direct tests of these models have often lacked rigor, because they have used statistical tests such as ROCs in a confirmatory mode. This project will allow perception and memory researchers to understand how to use ROCs in a more principled way.

This project is a five-year longitudinal study designed to examine early foundations of formal mathematical learning. Approximately, 250 children in Missouri will be assessed twice per year from preschool through first grade. Specific competencies to be measured include number, number relations, and number operations as well as language, executive function, attention, IQ, and social behavior. Symbolic and non-symbolic quantitative skills are considered. Data collected on this project will link to the Missouri Longitudinal Study of Mathematical Development and Disability.

Competence in arithmetic and basic algebra has been shown to be strongly related to employability, wages, and on-the-job productivity. Children who begin school behind their peers in mathematical competencies tend to stay behind throughout their schooling. The goal of this project is to conduct a longitudinal study with at-risk 3 year olds to improve the field?s understanding of the development of early numeracy development in young children and its relation to school mathematics outcomes. The project hopes to shed light on how domain general executive functions, nonverbal intelligence, and verbal intelligence interact with quantitative knowledge to lead to math achievement at the end of kindergarten.

The project will develop Bayes factors for common research designs. Bayes factors provide an attractive alternative to conventional significance tests, in particular to F-tests for linear models. They have failed, however, to achieve broad acceptance for at least two reasons: They are perceived as having an undesirable dependence on the chosen prior distribution, and they are viewed as being difficult to compute. To address the first concern, Bayes factors for a class of "default" priors will be developed; this is, with priors that result in Bayes factors with desirable theoretical properties, impart a minimal degree of information, and are broadly applicable in a wide range of common designs. One natural property of a desirable prior is "consistency," the ability to support the correct model in the large sample limit. The focus is on consistency when the model dimension is relatively large compared to the sample size, as is common in many ANOVA designs. Consistency will be proved for common one-way and two-way and possibly higher order designs. With respect to the second concern, computation entails integration across perhaps many dimensions. There are several choices (including quadrature, Monte Carlo sampling, bridge sampling, Laplace approximation, or Savage-Dickey density ratio estimation), and which choice works best will vary depending on the sample size and design. Heuristics for picking a method of computation that is quick and efficient will be developed. The end result will be the development of easy-to-compute Bayes factors with excellent properties.

The physical sciences have made gains by identifying invariances -- those elements that stay constant when others change. In contrast, the social sciences have emphasized demonstrations of effects rather than of invariances. One difficulty in demonstrating invariances in noisy environments is methodological -- conventional hypothesis testing allows researchers to amass evidence against the null but never for it. Bayes factors provide an ideal solution because they can be used to assess evidence for the null or alternative, are straightforward to interpret, and provide a natural penalty for model complexity. The project's ultimate goal is that Bayes factors become a widely-adopted, everyday method in substantive researchers' methodological toolkit. To that end, the project will develop a series of software applications. Some of these will be R packages for methodologists. Others will be web applets and GUI software for substantive researchers without statistical expertise. These latter products will be very easy to use, and this ease should encourage rapid adoption. In addition, conference workshops and tutorials, including short courses, are planned in the investigators' respective disciplines.