Hua-Hua Chang - US grants

While the implementation of computerized adaptive testing (CAT) has many advantages, many issues related to CATs are not well understood. This project will study three specific issues in development and implementation of CAT: (1) compatibility between CAT and "paper and pencil" (P&P) tests, (2) test security and item pool usage, and (3) how to calibrate test items in large quantities efficiently and economically. With respect to compatibility between CAT and "paper and pencil" (P&P) tests, it has been widely reported that some students get much lower scores than they would if an alternative P&P version were given. However, examinees currently required to take Graduate Record Examination (GRE) in the United States, for instance, are not given a choice between the standard P&P version of the tests and the CAT versions. Without effective remedial measures, the credibility of CAT could be significantly undermined. This project proposes to modify the statistical procedure used for CAT item selection by incorporating some advanced analytic techniques. It is expected that the analytic and simulation results will show that weighting likelihood score may alleviate the problem of underestimation. With respect to test security and item pool usage, in current operational CATs, computers tend to select certain types of items too frequently, making item exposure rates quite uneven. This project will show that test security and the underestimation problem discussed in the research on CAT and P&P tests are closely related. It is also expected that the project will show that the alpha-stratified approach proposed by Chang and Ying in 1999 tends to improve both the underestimation and test security. With respect to calibrating test items in large quantities efficiently and economically, administration of CATs requires very large item pools. Fortunately, CAT provides great potential to large-scale calibration during on-line testing. This project will explore the development of on-line calibration in CAT.

CAT has become a popular mode of educational assessment in the United States. Examples of large scale CATs include the Graduate Record Examination (GRE), the Graduate Management Admission Test (GMAT), the National Council of State Boards of Nursing, and the Armed Services Vocational Aptitude Battery (ASVAB). Findings from this research project may speed up the process of improvement over current item selection algorithms. Because many CATs are high-stakes examinations, improving their test reliabilities will greatly benefit society.

Computerized Adaptive Testing (CAT) has become popular in high-stakes testing programs. Examples of large-scale CATs include the Graduate Record Examination, the Graduate Management Admission Test, the National Council of State Boards Nursing, and the Armed Services Vocational Aptitude Battery. An advantage of CAT over paper-and-pencil exams is that it provides more efficient estimation of abilities because it can appropriately tailor item selection to the estimated abilities of examinees. This research addresses two areas of critical importance for CAT. First, flexible models for response times will be developed to assist in controlling the duration of an exam, and also to assist in the measurement of ability in appropriate circumstances. Second, statistical methods for constraint management will be developed to ensure that an exam has sufficient information to diagnose fine-grained skills while also providing an accurate summary score.

The impact of the research will be to provide technology to better utilize response-time information and also enhance the ability of CAT to provide diagnostic information. Models for response-time distributions will be developed that make few assumptions concerning functional form and allow for dependence between response times and a latent trait that represents ability on the studied domain. Estimation techniques will be developed that may be used with data previously collected from exams administered with CAT. Algorithms for utilizing estimated response-time distributions will be constructed to better manage duration of exams and to extract information from response times to better estimate the ability the exam is designed to measure. In addition to addressing response times, the problem of managing the diagnostic information to assess mastery of fine-grained skills will be studied for exams that also aim to provide a single summary score. CAT methods for adaptively selecting items to more efficiently provide a score will be modified to simultaneously balance the coverage of skills and attributes of interest.

The PI porposes to employ computer adaptive testing (CAT) to improve: STEM learning of Physics, test performance and student retention. The PI notes that this interdisciplinary, empirical proposal combines expertise in testing/measurement, cognition, and STEM education to diagnose students? problem solving and conceptual deficits prior to taking high-stakes course exams, and then to devise interventions aimed at remedying identified deficits in order to improve students? course performance and, ultimately, their retention in STEM disciplines. Working within the domain of physics at the undergraduate level, the research team will investigate the feasibility of using computer adaptive testing (CAT) techniques to devise a cognitively diagnostic computer adaptive testing (CD-CAT) tool that accurately predicts students? future performance on tests in difficult STEM introductory undergraduate courses prior to their administration. The potential findings will have relevance for other STEM disciplines, and if successful, for broadening participation.

This research project will develop statistical models that describe the way students learn and will design efficient training methods to help them learn efficiently. The project will advance psychometric theory by developing dynamic cognitive diagnosis models that capture the skills a student has mastered in the course of his training. It will impact psychometric and educational methodology by improving the design of e-learning environments and intelligent tutoring systems, where students are trained in a large number of skills. One of the final products of this research will be publicly available software that will incorporate the methodologies to be developed in this work. The theoretical knowledge that will be gained in this project will be incorporated into the material of graduate-level courses that cover item response theory and sequential analysis. Two graduate students will play key roles in conducting this research, and undergraduate students will be involved in certain aspects of the project. The investigators will make every effort to include qualified students of underrepresented groups in these research activities.

This research will address two fundamental questions. First, how do students acquire the skills to master a series of tasks? Second, how should these tasks be selected in real time in order to help students learn efficiently? The first question will be answered with the development of complex statistical models that are grounded in the theory of cognitive diagnosis. The second question will require the development of on-line algorithms for detecting quickly that a student has mastered a skill and for selecting the best possible tasks in order to facilitate learning. These algorithms will be developed through the fusion of statistical techniques from the fields of sequential change detection and experimental design. The methodologies developed to address these two distinct research questions will be merged by having the developed learning models inform the detection and task-selection algorithms. Overall, this project will consider statistical problems at the heart of educational and instructional practice, and it will highlight the interplay among the fields of cognitive diagnosis, latent class modeling, quickest change detection, and adaptive design.