Frances Lawrenz - US grants

9714189 Lawrenz This project proposes to fill the gap in long-term enhancement evaluation by conducting a multiple method, longitudinal study of a science teacher enhancement project. This evaluation will examine both the long term effects of teacher enhancement on an existing cohort of students, and the subsequent effects of teacher enhancement on new cohorts of students. Specifically, the project will examine the persistence of a teacher enhancement project by comparing groups of students whose teachers did and did not participate in the enhancement project. Two investigations are proposed: 1) follow cohorts of 9th grade students into 12th grade to compare science achievement; 2) compare the learning environment and science achievement of subsequent cohorts of 9th grade students. These two investigations will allow us to determine the extent to which the teacher enhancement project persisted into the future. The proposed evaluation will be conducted at five high schools that participated in the teacher enhancement project. We will collect a comprehensive variety of data including classroom observations of teachers twice a year, interviews with teachers and students, interviews with principals, student written questionnaires, and both hands-on and written assessments of students' achievement of the NRC science standards. The results of this evaluation will be used to examine the extent to which NSF funded teacher enhancement can affect teacher performance and subsequently affect outcomes in science.

The proposed study will incorporate the six drivers of systemic reform and a theoretical framework of school change into an empirical model to help explain the impact of a systemic reform effort on student achievement. Quasi-experimental studies involving four states (Louisiana, Montana, Colorado, and Illinois) are planned. Schools in an SSI state with high levels of involvement (treatment) will be contrasted with schools with low levels of involvement to allow the identification of the differing relationships among the drivers and student achievement that occur with different levels of involvement with an SSI within an SSI state. Results will also be compared to schools in non-SSI state. Information will be collected from approximately 60 schools in each SSI state (30 schools affected continuously by the SSI and 30 schools not directly affected by the SSI) and 30 schools in the non-SSI state. Site visits will be made to six schools within each state. Data will be collected with a variety of methods including student, teacher, and principal questionnaires, interviews, and observations. The student achievement data within each state will be used as the outcome measure. Analyses will range from simple descriptive statistics to complex modeling procedures.

The project is developing a core data collection and reporting system for the NSF Collaboratives for Excellence in Teacher Preparation program. the development is in three phases: year one, focuses on consensus building, and device and process development and pilot testing; the second phase concentrates on field testing and evaluation of devices and processes; and the third phase concentrates on scaling up the process. The project builds on prior work with the CETPs, Horizon Research's Local Systemic Change data collection mechanisms (participially its web based design) the experience of current CETP site evaluators. The web based data collection, enables data input from many remote sites into a common data base, accessible for the development of reports containing data from all sites overall as well as for reports containing data from individual sites. The database is compatible with existing NSF databases, as appropriate and provides impact information in a standardized and easily accessible format.

This three-year continuation of assessing the ATE Program will address four major questions: 1) To what degree is the program achieving its goals? 2) Is it making an impact, reaching the individuals and groups intended? 3) How effective is it when it reaches it constituents? 4) Are there ways the program can be significantly improved?

The project will be comprised of three components. The first is the continuing collection and reporting of monitoring information for the ATE program through a database this project developed. Second is the design and implementation of four studies examining critical issues for ATE: the value added by ATE as determined by business and industry, the quality of materials development and their impact, the quality of professional development, and the nature and extent of sustainability of ATE project. The third component is dissemination of the evaluation project's findings through traditional and innovative approaches.

The project is designed to contrast the type and scope of influence of different approaches to program evaluation in order to increase the field's understanding of how evaluation results are used when evaluations are conducted for a large agency with both national and local implications. The project is grounded in two theoretical bases: 1) evaluation use, and 2) participation in evaluation. These are based on the theoretical work of Kirkhart, Greene, Gary and Mark, Patton, Burke, King and Lawrenz and Huffman. The primary theory of evaluation use is Kirkhart's three-dimensional influence of evaluation model - source of influence (process or
results), intention (intended and unintended), and time (immediate, end-of-cycle, or long-term). Participation in evaluation draws on a continuum of Lawrenz and Huffman bounded by program evaluation conducted by an entity separate from the projects within the program to one when where the projects independently determine the evaluation procedures and what data to collect.
`
The proposed research will investigate the continuum of participation and how it relates to use.

The research questions are:

1) What patterns of participation exist in large, multi-site evaluation?
2)) To what extent do different levels of project participation in program evaluations result in different patterns of evaluation use?
3) What evaluation practices are most directly related to enhancing the influence of evaluations.

Case studies will be carried out of four NSF Programs: ATE, LSC, CETP, and MSP RETAs. Within the case studies, methods will include artifact analysis, interview, surveys and site visits.

The Collaborative Evaluation Communities in Urban Schools (CEC) project will create school-based collaborative evaluation communities to achieve the following goals:

1. Improve the evaluation capacity of urban schools.
2. Develop graduate level educational leaders with the knowledge and skills to evaluate
science and mathematics programs.
3. Provide teachers of science and mathematics professional growth by improving their capacity to use evaluation data and engage in continuous improvement and data based decision-making.

The collaborative evaluation communities created by the CEC project will serve as professional growth activities for science and mathematics teachers, and as training grounds to help graduate students develop evaluation skills. By engaging schools in evaluation the team will assist teachers and graduate students in the collection and analysis of data, the examination of student assessment results, the evaluation of school programs, and the analysis of science and mathematics instructional practices. The CEC project will assist partner districts in the creation of an evaluation culture designed to support the continual evaluation of programs with the ultimate intent of improving the educational opportunities for all students.

Intellectual Merit:

The CEC project will create a total of eight collaborative evaluation communities over the three-year grant period; four collaborative evaluation communities in the Kansas City, Kansas School District and four in the St. Paul School District. The collaborative evaluation communities will be co-led by a faculty member and a lead science or mathematics teacher in the schools. The faculty will also help supervise graduate students in each community who will help lead and develop the collaborative evaluation community. The lead teacher will serve as the liaison between the university and other classroom teachers who are in the same school as the lead teacher. There are various ways that the collaborative evaluation communities can interact, and it is expected that this project will inspire new ways for the schools and universities to collaborate on evaluation activities.

The use of collaborative evaluation communities to build evaluation capacity in urban schools is a unique and important activity for the field of evaluation. The schools are in need of teachers who can use evaluation and assessment data to make decisions regarding science and mathematics education. The CEC project will advance knowledge and understanding of how to help schools engage in evaluation. The investigators are very well qualified to design and implement the CEC project. They have extensive experience in the field of science and mathematics program evaluation, and will be able to use this experience to develop graduate students and help schools engage in evaluation. In addition, the CEC project will be externally evaluated by an expert in the field of science and mathematics program evaluation.

Broader Impact:

The activities of the CEC project will advance discovery and understanding regarding the use of collaborative evaluation communities for teachers in urban schools. In addition, the CEC project will promote teaching, training, and learning for graduate students. The CEC project will broaden the participation of underrepresented groups by enhancing the evaluation capacity of teachers in urban schools who serve a diverse student population. The investigators on the CEC project also have the ability to recruit graduate students of diverse backgrounds. The results of the CEC project will be widely disseminated to the education and evaluation community. The project has great potential to benefit society by developing the evaluation capacity of urban schools, and by developing the evaluation abilities of graduate students who can go on to serve society. All in all, the CEC project will create lasting collaborative evaluation communities in urban schools in order to develop schools' capacity to engage in evaluation and to help all students achieve in science and mathematics.

This proposal seeks to conduct an evaluation of the Noyce program. There are four components to the evaluation: statistical analysis of the MIS data; analysis of project reports; case studies of the projects that build on project evaluations using a participatory evaluation model; and a review of the related literature

Intellectual Merit: Environmental biology graduate students will learn to use inquiry-based pedagogy to become effective in teacher training, in K-12 schools and in university teaching, and will gain direct experience with translating their research expertise into a K-12 learning environment. Participating graduate fellows will be more likely to connect with the broader community throughout their careers, thus strengthening US STEM education. Through a competitive process, 12 fellows will be selected annually from six U of MN graduate programs: Ecology, Evolution, and Behavior; Entomology; Natural Resources Science and Management; Plant Biology; Fisheries, Wildlife and Conservation Biology; and Water Resources Science. Each fellow will be paired with an experienced teacher from one of four Minneapolis and St. Paul schools (three fellow-teacher pairs per school). Teachers will gain increased STEM content and process knowledge, and their students will engage in real science and scientific research. Consequently, the urban students involved in the project will gain confidence in their science skills and thus be more likely to see themselves as potential scientists. Three categories of project activities are planned: summer workshops for fellows and teachers, science activities in K-8 classrooms during the academic year, and activities to supplement and support interactions between fellows and the K-12 school community. Summer provide science content training for the teachers and initial pedagogy training for fellows. Classroom and after-school science activities are designed to support and to extend national, state and district science content standards. Fellow/ student classroom and science club projects will culminate with in-school Science Fairs at each school and a project-wide Natural story Student Research Fair that promote connections between fellows and the broader school community, including parents and families. During the academic year, fellows will enroll in a seminar course led by the PIs, and receive additional training in teaching practices, grant writing and providing resources for K-12 education.


Broader Impacts: The primary goal of this project is to train 36 graduate fellows who will then work with K-12 schools throughout their careers and eventually train the next generation of graduate students to do likewise. These fellows will serve as role models in four urban schools in which students from under-represented groups predominate and share their K-12 experience with colleagues within the university and the broader scientific community. An important outcome of this project will be the development of closer ties between public schools and the STEM programs of the University of Minnesota, and an improved understanding at the university of how to create effective K-12 collaborations. After the grant period, elements of the program will be incorporated into graduate student curriculum offerings and Bell Museum programmatic efforts.

The Cognition, Measurement and Evaluation (CME) project combines the latest thinking in cognitive science about scientific reasoning and its attainment with recent advances in modern instrument development techniques. The CME project will produce a linked system of developmentally-appropriate, multiple-choice and constructed-response assessment instruments designed to measure scientific reasoning skills across age levels. These tools will be useful for researchers, classroom teachers and educators in settings for learners from 5th grade through college. The project focuses on scientific reasoning skills of control of variables and evaluating evidence. These skills will be defined in ways suitable for modern instrument development and will result in a draft instrument and model computer platform that would be ready for initial field testing. The ultimate goal is to create an assessment system that will provide information to: (1) STEM researchers who want to understand how innovative technologies, instructional approaches, and/or teaching practices impact students' scientific reasoning abilities, and (2) teachers who need to understand how students are responding to particular aspects of inquiry instruction. The institutions involved include the University of Minnesota and the University of Kansas and surrounding school districts.

Based on needs assessment, existing research and instruments and content expert advice, we will map the constructs by developmental level across the three age groups. Each construct map will include developmental levels and a qualitative description of scientific reasoning at each level. The item development process will include development of scoring systems to map responses to certain levels of the scientific reasoning constructs. Initial drafts of the instruments will be investigated through think-alouds, exit interviews, and focus group interviewing. An initial pilot test with a more substantial sample will allow for modeling of the data using IRT. We will use item fit, differential item function, and coverage maps to assess item properties. We will also use item fit and model comparisons to examine the structure of the constructs. During the model-building phase, the Rasch family models, which are the most parsimonious models, will be fitted to the pilot data. If it is discovered that Rasch family models do not fit the data, revisions to the construct, instrument, scoring model, and measurement model will be implemented.

One of the most pressing needs in the evaluation of science education programs is the need for appropriate measuring devices. Currently, the field uses a variety of assessment devices, making it almost impossible to determine national effects or to compare different approaches. Although assessment of scientific reasoning is not a new concept, recent ideas about its definition and its measurement may be transformative. The assessment devices developed by this project will be validated to ensure cultural relevancy and absence of bias and therefore will enhance the infrastructure available for research and education. Over time the use of the developed instruments will allow in-depth understanding of the impact of different programmatic approaches to the development of scientific reasoning, a critical goal of STEM education.

Complexity theory as a theoretical framework for the evaluation of the Nanoscale Information Science Education Network (NISE Net) is the focus of this evaluation research project. The researchers include evaluation scholars at the University of Minnesota and research directors at the Science Museum of Minnesota, The Museum of Science in Boston, and the Oregon Museum of Science and Industry. The intent of the project is to study how complexity theory supports evaluation capacity building within the network of informal science organizations that makes up a national community of researchers and informal science educators whose goal is to improve public awareness of, engagement in and understanding of nanoscale science, engineering and technology. Components of complexity theory that are being studied include internal diversity and redundancy, the interactions among different individuals and organizations with the network, the extent to which the control of the network is distributed, the extent of a shared goal or identity operates throughout the system established by the network, and how the system responds to change in the external environment.

This study employs a tiered case study of the way that primary actors within NISE Net are designing and implementing a new evaluation system to examine the programs that the institutions within the network are conducting. The first tier includes those individuals who are responsible for designing and implementing the resources used in the network to improve the public understanding of nanoscale science. The second tier are the individuals who use the resources developed by the network. The study utilizes a multiple case study methodology that examines individual cases that can be collected together to describe the operation of a complex network of individuals and organizations. NISE Net operates as a heterarchy whereby individuals are organized hierarchically within institutions and communicate horizontally across organizations. The case study methodology results in a framework of the evaluation efforts that will be studied iteratively using a design-based research approach to inform stakeholders and support changes in the framework as it evolves over the course of the study. Observations of the planning and implementation of evaluation efforts, document review and interviews with key stakeholders comprise the data collected and analyzed in this study.

Complex evaluation efforts such as those being conducted by NISE Net require more than simple evaluation frameworks that examine inputs, activities, outputs and outcomes. Networks by their very nature are a system of stakeholders whose interactions quickly become complex and difficult to study. Complexity theory offers the conceptual organization by which complex systems can be studied and their evaluation efforts characterized. This project informs scholars in the evaluation field about models of evaluation that support examining the quality of work being conducted by complex systems such as NISE Net. The publications intended by this study have the potential to inform the evaluation community about how complexity theory can be brought to bear to study network systems in education.

The C2C award addresses the lack of validated instruments to measure teamwork and collaboration in middle and high school students in out of school time (OST) settings by implementing a rigorous four-phase process to develop new assessments. Phase 1 focuses on defining the construct of teamwork and collaboration skills so it aligns with the research literature and is relevant to outcomes in a variety of STEM OST programs. Construct maps are developed during Phase 2 to guide item development. The instruments are piloted in Phase 3 through think-aloud interviews and survey administration with a diverse set of youth and programs. Through an iterative process, items are revised or removed based on their psychometric properties. The final phase is a national field test with a cross-section of STEM OST programs.

C2C's intellectual merit is its potential to advance understanding of how to measure teamwork and collaboration skills in STEM OST programs. There is a national call for more measures to evaluate 21st century skills. C2C's creation of instruments to measure teamwork and collaboration skills in STEM OST programs helps to address this gap.

The work of C2C addresses broader impacts and benefit society by creating tools to understand the role STEM OST programs play in readying our nation's youth for the STEM workforce. C2C will create instruments validated specifically for this diverse population, allowing programs to understand the role they play in important societal STEM workforce readiness outcomes. C2C also benefits the informal science education field by conceptualizing the construct of teamwork and collaboration within STEM OST programs and developing validated instruments to understand the impact of these programs on youth.

The National Science Foundation (NSF) Graduate Research Fellowship Program (GRFP) is a highly competitive, federal fellowship program. GRFP helps ensure the vitality and diversity of the scientific and engineering workforce in the United States by recognizing and supporting outstanding graduate students who are pursuing research-based master's and doctoral degrees in fields within NSF's mission. GRFP provides three years of support for the graduate education of individuals who have demonstrated their potential for significant achievements in science and engineering research. The award to this GRFP institution supports NSF Graduate Fellows pursuing graduate education at the institution.

? DESCRIPTION (provided by applicant): This 3-year R01 based at the University of Minnesota and Vanderbilt University will convene a national Working Group of top legal and scientific experts to analyze current U.S. federal and state law, regulation, and guidance on translational genomics, and to generate consensus recommendations on what the law should be, to optimize successful translation of genomics into clinical use. The law underlying genomics is currently unclear, poorly understood, and contested. As this technology is now poised for wide clinical integration, legal scholars need to work with genomics experts to build the required legal foundation. Work is especially needed in 4 domains, the law of: Liability, Quality of sequencing and interpretation, Privacy & Access to genomic results, and the overarching Framework clarifying when research rules vs. clinical rules apply. Aim 1 of this project is to build a searchable public online Database of state and federal statutes, regulation, case law, plus official guidance and standards, and an Annotated Bibliography of core literature-a map of what current genomics law is. The project will use established legal search methods and analysis to construct the Database, working with a professional web design firm to construct a highly functional, durable, and free public resource. We will use multiple search methods to build the Bibliography, posting it online for free access. Aim 2 is to use mixed methods to ascertain stakeholder views of the law and generate rich inputs to inform a systematic Working Group process generating consensus recommendations on what the law undergirding translational genomics should be. Led by a senior social scientist, the team will use a modified Delphi method to elicit views of Working Group experts on issue priorities. To systematically capture a wide range of stakeholder views on issues and potential solutions, the project will use a streamlined survey and interview process. We will use an online REDCap(tm) survey to identify the perspectives of NIH-funded genomics investigators, legal counsel, industry representatives, genomics clinic directors, professional society leaders, and government authorities. We will also conduct semi-structured interviews with qualitative content analysis of the transcripts to capture more fully the views of a sample of legal counsel and government authorities. Informed by these rich inputs, the Working Group will pursue a structured process of analysis and consensus building that is well-established in law and bioethics. Each of 4 Task Forces will generate consensus recommendations, reporting to the full Working Group for feedback and project coordination. We will seek feedback on our recommendations from a subset of survey and interview respondents plus other expert readers, and through a major public conference. Project products will include: the online legal Database and Annotated Bibliography, 4 Task Force papers with consensus analysis and recommendations, individual targeted articles, published empirical analyses, a videotaped public conference, a symposium issue of a peer-reviewed journal presenting project publications, online access to our work, and wide dissemination.

Project Summary / Abstract This 4-year Neuroethics R01 based at the University of Minnesota (UMN) will convene a national Working Group of top neuroethics, neurolaw, and neuroscience experts to conduct empirical research and generate evidence-based consensus recommendations for the ethical conduct of population research using highly portable, cloud-enabled MRI in new and diverse populations in field settings. NIH is supporting the development of both high-field portable MRI (3U01EB025153-02S2, PI: Garwood), and ultra- low field MRI (P41EB015896, PI: Rosen). As portable MRI develops quickly, guidance is urgently needed on unresolved ethical, legal, and social issues (ELSI). This R01 project builds on two NIH Administrative Supplements that have preliminarily identified the most pressing unresolved ELSI issues: (1) informed consent; (2) data security and privacy; (3) establishing local capacity to interpret and communicate neuroimaging data; (4) extensive reliance on cloud-based artificial intelligence (AI) for data analysis; (5) potential bias of interpretive algorithms in diverse populations; (6) return of research results and incidental (or secondary) findings to research participants; and (7) responding to participant requests for access to their data. Building on this preliminary work, Aim 1 will utilize survey research to inform a systematic Working Group (WG) process described in Aim 2. In Aim 1a, we will survey the U.S. general public, including over- sampling of rural, older adult, non-Hispanic African American, Hispanic/Latino, and economically disadvantaged respondents, to probe likely research use cases, issues they raise, potential solutions, and willingness to participate in research. In Aim 1b, we will survey expert stakeholders to elicit views on current/future research use cases and how to address the ELSI challenges. Expert stakeholders will be from 5 key groups: (1) researchers utilizing brain MRI and scientists developing new MRI technology; (2) neuroethics and legal scholars; (3) industry stakeholders; (4) leaders in regulatory agencies and standard-setting organizations; and (5) leaders in patient advocacy organizations. Aim 2 builds on Aim 1 to generate evidence-based consensus guidance on the ethical conduct of research in the field using highly portable, cloud-enabled neuroimaging. In Aim 2a, we will use a modified Delphi method to elicit initial WG views on issue priorities, research use cases, and potential recommendations, and will develop an Annotated Bibliography. In Aim 2b, the WG will pursue a structured process of analysis and consensus building that is well-established in bioethics and law, in order to identify best practices and formulate recommendations informed by the Aim 1 work. In Aim 2c, we will solicit feedback on our recommendations from expert readers and through a major public conference. Project products will include: an online Annotated Bibliography, WG consensus recommendations, individual targeted articles, published empirical analyses, a webcast public conference, a symposium issue of a peer-reviewed journal, online access to our work, and wide dissemination.