Audrey S. Rorrer, Ph.D. - US grants

This project will provide long-term support for external evaluation activities and services for the Computing and Information Sciences and Engineering Research Experiences for Undergraduates (CISE REU) community. This project will address the ongoing need among principal investigators who are fulfilling CISE REU Site program evaluation efforts by providing an evaluation toolkit that facilitates evaluation reporting while strengthening research about undergraduate research program outcomes. The primary purpose of the evaluation toolkit is to provide targeted instructional resources and tools for quality program evaluation that balance the desire for standardized assessment along with the responsibility to account for individual program contexts. Toolkit contents include instructional materials about evaluation practice, a standardized applicant management tool, and a validated modulated outcomes measure. Services associated with the toolkit are providing a Common Application for site recruiting, a Shared Applicant Pool for principal investigators to share applicants across sites, delivery of an online student survey known as the A la Carte Student Survey, and corresponding responses to REU Site principle investigators. Project expansion will continue these tools and services as well as expand to include a Faculty Career Impact Survey, Student REU Alumni Survey, and a study of site organizational characteristics. The benefits of the toolkit are in providing cost effective, sustainable evaluation tools, a community forum for program evaluation, and aggregate measurement of key program outcomes indicators for the national program. The evaluation toolkit project balances the need for individual site context evaluation with the need for broader, generalizable program impact evaluation.

Intellectual Merit
The intellectual merit of this project resides in the contributions to understanding the impact of undergraduate computing research programs for both students and faculty participants. The comprehensive program evaluation will inform collective efforts to recruit undergraduate students to doctoral programs. The program evaluation will generate knowledge about features of REU Site structure for enhancing student learning outcomes and motivating individuals to pursue advanced degrees and careers in research. The knowledge generated by this study will also provide insights about how faculty engage in REU Sites and the impact on their careers.

Broader Impacts
The broader impact of the project will be evidenced by the instruments and online tools developed by this project that will benefit any researcher involved in supporting research experiences for undergraduates. This will enable formative improvements for numerous REU programs, enrich REU experiences for undergraduate students, and augment faculty capacity for leading successful REU programs, thereby enhancing the graduate school pipeline and faculty career development. In addition, the outcomes of the proposed project will contribute to evaluation capacity building research for multi-site program assessment.

This project will provide long-term support for external evaluation activities and services for the Computing and Information Sciences and Engineering Research Experiences for Undergraduates (CISE REU) community. This project will address the ongoing need among principal investigators who are fulfilling CISE REU Site program evaluation efforts by providing an evaluation toolkit that facilitates evaluation reporting while strengthening research about undergraduate research program outcomes. The primary purpose of the evaluation Toolkit is to provide targeted resources and tools for quality program evaluation that balance the desire for standardized assessment along with the responsibility to account for individual program contexts. Toolkit contents include instructional materials about evaluation practice, a standardized applicant management tool, and a validated modulated outcomes measure. Services associated with the Toolkit are providing a Common Application for site recruiting, coordinating a Shared Applicant Pool for principal investigators to share applicants across sites, delivering an online student survey known as the A la Carte Student Survey, and developing tailored analyses to REU Site principle investigators. Project expansion will include development of an Alumni Tracking tool to help investigators evaluate long-term outcomes and a study of faculty mentoring and career impact. The benefits of the toolkit are in providing cost effective, sustainable evaluation tools, a community forum for program evaluation, and aggregate measurement of key program outcomes indicators.

The key contribution of project is in the refinement of understanding evidence-based practices in evaluating undergraduate research programs to help faculty successfully manage these programs. The project contributes to understanding the impact of undergraduate computing research programs for both students and faculty participants. The comprehensive program evaluation will inform collective efforts to recruit and mentor undergraduate students involved in cutting-edge research. The knowledge generated by this study will also provide insights about how faculty engage in REU Sites and the impact on their careers.

African-American, Latinx and students from low socio-economic households too often do not have opportunities to learn computer science and computational thinking (CS/CT), concepts that are crucial for participation in the 21st century workforce and citizenry. These limited opportunities are often the result of systemic practices in schools that impact a wide range of connected parts of the school including leadership, access to CS/CT resources for students and teachers, CS/CT learning opportunities for students and teachers, and prior experiences and encouragement to participate in CS/CT. This project will build on a long-standing Research-Practice Partnership (RPP) between NC State University's Friday Institute for Educational Innovation and a middle school in Wake County Public School System to deepen the RPP to further develop conceptual, theoretical, and applied frameworks for CS/CT. Additionally, the project will scale this RPP work to another newly-forming CS/CT focused magnet through a developing partnership with UNC Charlotte and Charlotte-Mecklenburg Schools to replicate the RPP for a new "Computer science and coding" magnet program. Since mathematics and science classes are taken by all middle school students, the project will use this academic context as the focus of the project's work. This proposal specifically targets middle schools in both Charlotte and Raleigh with relatively high African-American and Latinx populations, and free and reduced lunch percentages, to scale the existing RPP. The project goals are to provide equitable access to underrepresented students in computing; to prepare middle school students for computing curriculum in high school; to develop and test effective CS/CT modules that integrate in existing math and science curriculum; to investigate CS/CT-focused systemic barriers and supports; and to propagate RPP findings to other schools and districts.

The STEM ecosystems framework is an emerging model for framing the barriers and support structures students have in STEM learning, including CS/CT that recognizes that effective CS/CT learning is the product of the entire connected academic enterprise--with elements including school leadership, teacher and student CS/CT resources, and available CS/CT learning opportunities, along with prior experiences, encouragement, and training in CS/CT. The project will adapt the STEM ecosystem model as a scalable, generalizable approach for systematic, school-wide integration of CS/CT into required math and science courses. Within the context of two magnet middle schools seeking to integrate CS/CT across their curricula, this work will be guided by the following overarching research questions: 1. What are the barriers to developing a STEM ecosystem that supports CS/CT for every student through integration into middle school science and math courses? 2. What factors or interventions are needed to support the development of a CS/CT focused STEM ecosystem that supports everyone in a school? 3. What are the indicators of success for a CS/CT focused STEM ecosystem in a school? 4. How does the ecosystem prepare and engage all students, especially those from underrepresented student groups, for CS/CT work in high school? To address the first two research questions, data will include meeting notes for thematic analyses from RPP leadership teams, interviews with teachers and school leaders, observational and self-report survey data from teachers and school leadership. Additionally, professional development workshop data will be included to triangulate responses from interviews and surveys. To address the third research question, data will include student demographic data, participation in STEM activities, classroom observations, student engagement as measured by demographics exposed to CS/CT within and external to the curriculum, activity observations of engagement levels, self-report data of attitudes, learning and future interest in CS/CT, teacher observations of learning and engagement, activity/module ratings from students, teachers, observers, class artifacts (portfolios or projects), and End of Grades science and math performance. For Question 4, members of the project and RPP teams, with input from the Advisory Board will perform an annual review of NC high school CS course syllabi against emerging K12 CS standards. Lesson plans and artifacts of student work from Reedy Creek and Northside will be sampled and compared against current high school expectations for CS academic work.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project provides long-term support for external evaluation activities and services for the Computing and Information Sciences and Engineering (CISE) Research Experiences for Undergraduates (REU) community. This project addresses the ongoing need among principal investigators who are conducting CISE REU Site program evaluation efforts by providing an evaluation toolkit that facilitates evaluation reporting while strengthening research about undergraduate research program outcomes. The primary purpose of the CISE REU Evaluation Toolkit is to provide targeted resources and tools for quality program evaluation that balance the desire for standardized assessment along with the responsibility to account for individual program contexts. This project aims to enhance the evaluation tools and resources and to provide for sustainability and maintenance of the resources. Services associated with the Toolkit are providing a Common Application for site recruiting, coordinating a Shared Applicant Pool for principal investigators to share applicants across sites, delivering an online student survey known as the A la Carte Student Survey,and providing an Alumni Tracker tool to assist sites with tracking their students after their participation in summer research at the site. The project also provides aggregate and site-specific analyses to REU Site principal investigators. The benefits of the Toolkit are in providing cost effective, sustainable evaluation tools, a community forum for program evaluation, and aggregate measurement of key program outcomes indicators.

The key contribution of project is in the refinement of understanding evidence-based practices in evaluating undergraduate research programs to help faculty successfully manage these programs. The goals of the project include managing the various components of the Toolkit, providing analyses and summative annual reporting of the Evaluation Toolkit results, and providing timely, site-specific data packages of student results. The strength of this project comes from the strong research community of practice that has been nurtured by this project and the ongoing commitment to support and enhance the resources through this funding. The project contributes to understanding the impact of undergraduate computing research programs for both students and faculty participants. The comprehensive program evaluation will inform collective efforts to recruit and mentor undergraduate students involved in cutting-edge research. The knowledge generated by this study will also provide insights about how faculty engage in REU Sites and the impact on their careers.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This award is a one-year extension of the STARS Computing Corps Alliance for Broadening Participation in Computing (BPC). The primary research aim of extension is to investigate research questions around the impact of the STARS Computing Corps Alliance on its past and current efforts to broaden participation in computing. This longitudinal evaluation has the potential to elucidate successful components of BPC Alliances and their role as national resources on issues of diversity, equity, and inclusion in computing. Further, the results of this study could further extend knowledge about practices designed to increase persistence and support career advancement in computing for undergraduate students, graduate students, and faculty, particularly for those from underrepresented groups. The conferences, communities, and networks supported through STARS will continue to create significant institutional and human resources that could increase the reach of BPC research to a larger audience of researchers, educators, administrators, CS departments, and K-20 students.

In addition to understanding the longitudinal impact of the STARS Computing Corps on computing persistence and career advancement for underrepresented students and faculty, the work also aims to 1) formalize existing STARS efforts for propagating best-practices for BPC; 2) expand STARS online resources designed to build capacity for engaging in BPC evaluation and research; 3) develop a virtual STARS alumni community; and 4) continue to support an academic community for BPC through STARS Celebration and RESPECT.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

It is critical to address the longstanding issue of underrepresentation of women, Black, and Hispanic students in computing degree programs to provide an equitable foundation for all to participate in our society and the global economy as controllers and creators of technology, and to advance the preparation of a diverse, innovative, and competitive tech workforce. Building on the prior success of the STARS Computing Corps Alliance for Broadening Participation in Computing, the goal of the STARS Catalyst project is to: 1) increase the number of women, Black, and Hispanic students that persist in computing degree programs, and 2) advance the careers of students and faculty from groups that have been historically underrepresented in computing. Through research and evaluation around STARS Catalyst activities, this project will advance knowledge about practices designed to increase persistence and support career advancement in computing for college computing students and faculty, particularly for those from underrepresented and intersectional groups in computing. The STARS Catalyst Alliance is a collaborative effort across Temple University, North Carolina State, Kent State, Florida State University, Morgan State, and University of North Carolina Charlotte.

The STARS Computing Corps Alliance for Broadening Participation in Computing (BPC) engages computing faculty and students at colleges, universities, and community colleges in a community of practice with a shared commitment to take action to advance diversity, equity, and inclusion in computing. STARS Computing Corps conferences, communities, and networks create significant institutional and human resources that can expand BPC research to a larger audience of researchers, educators, administrators, CS departments, and K-20 students, and can dramatically increase the number of people taking action in BPC efforts. Prior results show that the STARS Computing Corps alliance increases intentions to persist in computing among STARS students and faculty, with enhanced outcomes for Black students and faculty. This project will significantly extend the STARS alliance to expand upon those impacts, by 1) including new partners that expand the reach of STARS and that emphasize participation of Black and Hispanic students and faculty, particularly from emerging Hispanic Serving Institutions and community colleges, 2) creating new program elements that test new and propagate evidence-based BPC practices within computing departments, and 3) leveraging partnerships to support identity-focused affinity groups, and 4) developing STARS Alumni groups employed in industry positions to promote transition to and retention within the tech workforce. Extensions to the STARS Leadership Corps program, STARS Launch program, and the STARS Celebration conference will serve to develop and propagate evidence-based approaches aimed at improving the teaching and learning of computing for Black and Hispanic students and build evidence of their effectiveness, and the RESPECT research conference will continue to advance peer-reviewed BPC scholarship.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project aims to serve the national interest by improving undergraduate student success in computer science courses. To do so, it will study relationships between student success and the course climate and teaching practices of their computer science courses. It will further analyze these relationships for students with intersecting identities, such as women who are first generation college students. Many STEM classrooms feature passive learning approaches and intentionally or unintentionally establish create competitive course climates that inhibit collaboration. Alternative teaching approaches that encourage collaboration and active learning are gaining traction in many undergraduate computing programs. Studies have generally indicated a positive relationship between collaborative active learning and increased student learning outcomes. However, implementation of active learning varies and the results on student learning have been inconsistent. It is possible that the inconsistencies may reflect differential impacts of competitive, passive learning on groups of students with specific intersecting identities. This project aims to explore the relationship between active learning and cooperative course climate on students with intersecting identities, including race, gender, and social class. The knowledge generated by this research has the potential to improve student success in computer science and thus broaden participation of individuals from groups that are not yet equitably represented in computing.

This project identifies course structure and sociality as two major components of course climate. Course structure includes features such as the course timeline, expectations for attendance or participation, and how grades are determined. Sociality involves forming and working in supportive communities. The project uses the constructs of course structure and sociality to capture pedagogical components that faculty select and combine in their adoption active learning approaches. Correlations will then be identified between these active learning components and student outcomes, and will be used to explore differential outcomes across intersectional student identities. This knowledge may help instructors select active learning strategies that would lead to more effective student learning, or combine elements of structure and sociality to develop new promising pedagogies in undergraduate computing education. The NSF IUSE: EHR Program supports research and development projects to improve the effectiveness of STEM education for all students. This project is in the Engaged Student Learning track, through which the IUSE program supports the creation, exploration, and implementation of promising practices and tools.   

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.