Annual Project Directors and Evaluators Meeting October 2022

Day 1- Tuesday, October 18, 2022

Opening Plenary: The Art of Intentionally Planning for Educational Innovation and Transformation

This session will help educators utilize the same strategies for effective lesson planning when attempting to transform their scholars’ educational experiences. Innovation and transformation do not occur without a thoughtful planning process with clear objectives and intended outcomes.

Presentation: PDF

Advancing STEM and CS Integration through Partnerships and Professional Development

In this panel discussion, participants will hear various perspectives on professional development, CS integration, and participant recruitment from key personnel in one early-phase project. The panel will describe innovative practices including a two-phase PD model that responded to rural schools’ COVID-era needs for flexible facilitated and self-directed learning, the development and release of microcredentials that allow educators to receive recognition for standards-based competencies in CS integration and evolving post-pandemic methods for recruiting and retaining multiple cohorts of participants. The panel will include representatives from the Advancing Rural Computer Science project’s implementation team, PD providers, recruitment team, external evaluator, and partnering Department of Education.

Presentation: PDF

Building an Engaged School Network to Boost Your Project’s Success

This interactive workshop explores BARR’s adaptations to its school networking strategies during COVID and beyond to create an engaged network of schools. Attendance grew by 300% during COVID, as BARR coordinators and administrators came to count on these times to learn and grow from each other, using BARR as the foundation and unifying force across school size and geography. As measured by AIR in the Final i3 Evaluation Report, these virtual meetings “creat[ed] space for adult connection and focus[ed] on professional development to build educator capacity to support students’ SEL and educator capacity to connect students, families, and partners.” BARR facilitators will lead you through a virtual network meeting including small-group breakout time where you can apply your learning and share best practices.

Presentation: PDF

GOBSconnect: Relationships Build Greater Opportunities for Better Success

GOBSconnect is a multi-tiered approach the district utilizes to reach out to students PK through 12th grade in the rural Boone County, Arkansas area to improve literacy and math scores, address social emotional learning, health, and wellness issues, and to build strong relationship connections with students and their parents/guardians. These approaches include after school activities, traditional summer school (K-8), credit recovery (9-12), Boot Camp PreTeaching (9-12), GOBSmobile (PK-12 ), Health & Wellness (all PK-age 18 through meals), special populations, community partnerships and digital coaching through high quality academic staffing, homeless liaison, nurse, ELL/migrant staff, counselor, school based mental health and special education. Through these multiple approaches, the district hopes to show growth in academics, reduce discipline, health, and social emotional referrals, increase attendance, and expose students to STEM/STEAM activities to promote future educational growth.

Presentation: PDF

SEL From the Inside Out: Building a Culturally Relevant Identity and a Climate Learning

During this session we will explore the “Inside Out Approach” to integrating social and emotional learning into educational settings. This approach begins with self-awareness, self-care, and with emotional well-being, helps develop students to be their best selves and to achieve their optimum personal success in the learning environment. There are thousands of research-based practices that support and develop emotional and social literacy. Educational success does not happen without connecting to emotion. As educators we need to help our learners develop a positive self-identity that recognizes their culture, identity, and community. We cannot have equity without inclusive practices that address all learner and stakeholder needs. Together we will help to build a pathway for success with SEL.

Presentation: PDF

Ensuring Effective Implementation of Innovations

What strategies and methods are needed to ensure effective use of innovative practices in our schools and classrooms? How do we ensure these strategies and methods are relevant and feasible for our different contexts? This session will dive into why effective implementation strategies are needed and share how active implementation strategies can be used to answer these questions through the use of case examples and data stories. As a result of attending the session, participants will be able to: Articulate why implementation science is needed for supporting uptake and use of innovative practices; make connections between current methods of supporting innovation practices in schools and active implementation strategies; and identify an active implementation strategy and tool to support their innovation

Presentation: PDF

Conducting Equitable Evaluations

This session will focus on helping evaluators embed an equity focus into research questions and design, with attention to measuring who benefits most and least from an intervention, including those who will be impacted in the evaluation process, sampling populations of focus, avoiding underpowered subgroup analyses, and measuring equity outcomes and their drivers. After an initial presentation, attendees will break into small discussion groups to share perspectives and experiences.

Presentation: PDF

Analysis Topics: Changes in the WWC v5.0 Standards & Procedures

This session will help evaluators continue to design and conduct rigorous evaluations consistent with changes in the recently released WWC v5.0 Standards and Procedures Handbook. The WWC v5.0 Handbook includes changes to (a) handling multiple outcome measures in the same domain; (b) acceptable approaches to handling missing data; (c) review of study-developed measures; and (d) the importance of the Study Review Protocol v1.0 for review of EIR evaluations.

Presentation: PDF

Data Visualizations for Impact and Implementation Findings

This session will help attendees improve their grant and evaluation reports by sharing best practices for displaying findings from your impact and implementation analyses. Examples will include data displays for comparing treatment and comparison group outcomes, showing how well the treatment group met fidelity of implementation thresholds, and illustrating relationships between implementation and impact. The session will also include general tips to improve the clarity of your data visualizations.

Presentation: PDF

Creating Possibilities: Considerations for Effective Partnerships between Evaluators & Project Teams

This panel, moderated by an experienced evaluation technical assistance provider, will help grantees and evaluators identify key features of partnerships that are actionable, relevant, and useful. This panel will highlight two EIR projects being conducted within large school districts; The Seminole County Public Schools Exceptional Student Education Curriculum Project and San Diego Unified School District Coding Our Future: Creating Equitable Computer Science Pathways in Urban Schools. Through consistent communication, creative opportunities for collaboration, and clearly identified roles, these teams have found a strong base from which to operate. The ESECP team will share their experience in developing a fidelity matrix and associated program data. The CoF team will share their experiences in approaching the creation of novel computer science assessments as well as engaging in student and teacher focus groups, think alouds, and surveys.

Presentation: PDF

Innovative Approaches to Understanding the Impact of Teacher Professional Development and the Academic Writing Skills of English Learners

We, the Pathway Project, will present on an innovative framework to examine specific components of writing produced by multilingual and English-only students that we believe our program may have affected. The framework was developed to code for specific and discrete parts of student writing, such as the quality of claims, the use of textual evidence, the quality of commentary, sentence fluency, and so on. Moreover, by eliciting teacher input through focus groups and surveys, an understanding of the impact of professional development was accomplished. Feedback also helped facilitators re-design different components of the program. This research presentation will engage the audience in using our framework to understand the academic writing development of English learners. The audience will also engage in practicing different professional development tools in support of student writing. Additionally, our redesign process based on teacher feedback will be discussed.

Presentation: PDF

Leveling Up Your Computer Science (or Tech) Education Offerings

With the growing demand for rigorous computer science education offerings from parents, students, policy makers and industry, meeting this demand and the emerging standards and requirements can be a challenge. In this session, tech inclusion advocate Ruthe Farmer will help decode the tech education ‘word salad’, share evidence-based resources for building your computer science education pathway, and highlight unexpected partnerships to ensure access and inclusion of underrepresented students.

Presentation: PDF

New Perspectives for Cluster RCTs under the WWC v5.0 Standards: Joiners & Attrition

To help evaluators design cluster RCTs that can produce rigorous evidence of program effects, this session will review new perspectives in the WWC v.5.0 Handbook on potential bias from changes to the study sample in cluster RCT designs. The WWC v5.0 Handbook defines scenarios when individuals entering clusters after random assignment (“joiners”) do and do not pose a risk of bias to evidence about program effects, as well as acceptable reference samples for calculating attrition.

Presentation: PDF

Measuring Students’ Social Emotional Learning

Many EIR grants address the absolute priority focused on promoting social emotional learning (SEL) skills that prepare students to be informed, thoughtful, and productive individuals. However, measuring the student SEL outcomes that are the target of these interventions pose unique challenges for meeting WWC standards. The session will focus on the range of SEL outcomes that EIR evaluators will measure, how the WWC defines SEL domains, how the WWC treats study-developed measures, and the types of SEL measures that are not reviewable by the WWC. The session will cover possible strategies for mitigating risks that SEL outcomes will not be reviewed. After an initial presentation, attendees will break into discussion groups to discuss their challenges measuring SEL outcomes.

Presentation: PDF

Challenges of Evaluation in Rural Schools (10/18)

This session will examine challenges for conducting evaluations on rural schools, including identifying a comparison group similar to the treatment sites; getting a large enough sample; and issues for randomization, such as blocking and assignment at different levels.

Presentation: PDF

ED Closing Comments – Equity in the Context of EIR

Closing comments from Dr. Sonji Jones-Manson and Dr. Bernadine Futrell, OESE Deputy Assistant Secretary for Equity and Discretionary Grants and Support Services.

Presentation: PDF

Day 2- Wednesday, October 19, 2022

INFACT: Including Neurodiversity in Foundational and Applied Computational Thinking

The panel will discuss how INFACT provides inclusive CT teaching and learning experiences for grades 3-8 through teaching and learning materials that are specifically designed to support neurodiversity in inclusive classrooms. This includes executive function scaffolds embedded within the materials and an INFACT delivery portal that enables differentiation of materials for a wide range of learners and learning environments. The team will highlight how INFACT was used during COVID to meet the needs of learners who may have otherwise fallen through the cracks. The INFACT research team will discuss the innovative research methods used to measure computational thinking, student affect, attention, as well as impact on teaching during COVID. The team will discuss various modes of assessment used in the design and efficacy research, along with noting the advantages and disadvantages of remote data collection and the necessity of staying nimble during unpredictable times.

Presentation: PDF

Leveraging MiPLAN Partnerships

ESC 18–TxCEE is implementing the Math instructor Professional Learning Autonomous Network (MiPLAN) to provide self-directed professional learning for math teachers in grades 3-8. Like other grantees, we initially faced a number of challenges in getting the work off the ground, including developing communication materials, recruiting district partners, and connecting teachers with high quality professional learning. This session will focus on lessons learned and strategies to leverage partnerships for project success.

Presentation: PDF

How Putting the Needs of Students First Can Transform a School, A District & The Communities We Serve

Come be informed about the next and the best trauma-informed social & emotional learning practices and learn how to create a system of supports to meet the needs for all students utilizing the whole child/whole community framework. Learn how to implement a trauma-informed logic model that will enable you to 1) increase social & emotional supports for students and families and 2) address issues of equity, reduce discipline disparities and manage issues related to public health and safety, and approve academic outcomes for students. Leave with a plan to transform your classroom, school, or district.

Presentation: PDF

The World Shut Down & Our Project Blossomed

March 2020, the world shutdown when our team was closing out year 2 of our EIR project. Join this interactive session to engage in dialogue with an EIR Project Director of a grant investigating whole school restorative practices in Austin ISD in Texas. Learn 1) how the project, grounded in social emotional and academic learning, set a strong foundation for Multi-Tiered Systems of Support in 11 schools, and 2) how the project launch prepared the team to support their school communities during the global health crisis.

Presentation: PDF

Creating an Equitable System: Key learnings in leading through the complexities in urban systems

The session will provide participants with a broad overview of the current state of education through the lens of equity and innovation within the context of Hartford Public Schools. Strategic priorities that frame and guide Hartford’s pre- and post-pandemic operating plan will be discussed. There will be a special focus on community-centered design processes and implementation of innovative programming. Key lessons will be shared about the district leadership moves necessary to navigate the complexities of systemic change such as school consolidation and district redesign, equity-centered resource allocation, student-centered learning, and authentic stakeholder engagement.

Presentation: PDF

Orientation to a Report Template for EIR Grantees

This session will focus on helping EIR evaluators prepare to write a report on their EIR impact evaluation. The session will provide an overview of the EIR Report Template (shared with evaluators in June 2022). The template supports EIR evaluators to include all essential data in their public reports on their EIR evaluations for (1) a WWC review and (2) for meeting EIR evaluation-related expectations.

Presentation: PDF

Challenges of Evaluation in Rural Schools (10/19)

Presentation: PDF

Innovation Through Empathy Driven Design – Developing Lead Learners

In this session, we will share our mindset and guiding philosophy along with the meaningful partnerships that have allowed us to continue to innovate. We have developed solutions that are responsive to our schools’ needs and have shifted from a focus on developing CS expertise (in elementary teachers) to developing a curriculum that removes barriers to teaching CS. We continue to develop resources and training that support a Lead Learner mindset in teachers. We promote collaborative classrooms that eliminate hierarchy as students learn from and with each other as well as the teacher. We are a team embedded in San Diego Unified School District that combines the expertise of experienced computer scientist education researchers (from UCSD), computer science teachers, language development specialists (from the California Reading and Literature Project), and classroom teachers who are skilled at classroom instruction and student relationship building.

Presentation: PDF

Leveraging Partnerships to Influence Systemic Change

In this session, we review the Teachers CHOICE project, evaluation, and district partnerships that have allowed the team to leverage available data to influence systematic change in the district such as improvements to communication and data systems. We will discuss the roles of the project team and evaluators, how the team has worked together to gather disparate data sources and align them in a user-friendly format, and how this work is being used to inform systemic change in the district. The team will also discuss challenges, how they have been overcome, and the impact this work is having on systems within the district. We will also discuss some of the innovative aspects of the data dashboards that the team has been working to develop.

Presentation: PDF

Expanding Visibility – Stakeholder Engagement and Dissemination/Marketing

In response to the annual EIR TA needs assessment, grantees identified two of their biggest project management needs as 1) creating a dissemination plan and 2) creating MOUs and partnership agreements. This session builds on prior Sustainability and Scaling workshops to help grantees identify stakeholders and understand marketing to the “ideal” partner(s). Hear about successful strategies and approaches from grantee peers and plan to attend the next Sustainability and Scaling workshop in November to continue your project work.

Presentation: PDF

Your Role in The Evidence-Building Ecosystem at ED

This session will highlight the critical role EIR grantees play in the Department of Education’s work to build rigorous evidence of “what works” to improve outcomes for all learners. IES Commissioners Albro and Soldner will describe the key features of the evidence-building ecosystem at ED, and how the work of EIR grantees contributes to the Department’s larger mission. Grantees will learn about opportunities to take advantage of Department and other resources that can strengthen their work and how collaboration with ED can further accelerate the scaling process for successful interventions.

Presentation: PDF

“Gradual Release Model”: How to Build Sustainability from the First Conversation

One of the goals of Future Forward’s recently awarded EIR Expansion grant is to develop sustained local capacity for implementing the program over the five-year partnership between Future Forward and participating sites. The development of local capacity for managing program implementation is essential both to sustaining the program beyond EIR grant funding and continuing to expand the program beyond the over 6,000 children and their families served through this grant. To achieve this goal the Future Forward national team will work with their site and school partners to deploy a Gradual Release Model (GRM) which provides:

A blueprint for understanding all that is required for independent program management
A self-assessment process to identify management capacity and plan for improvement
A process for highlighting strengths and sharing best practices with the broader FF network
An evidence-based process for identifying readiness for increased management responsibilities.

Presentation: PDF

Iterative Development of a Physics Application Toward Broadening Access to Physics Education and Instruction

A key component of the Education, Innovation and Research program is broadening access to learning opportunities for diverse students. One important strategy for broadening access is using digital tools to provide technological solutions to instructional questions. This session explores the development of the Playground Physics Program, consisting of a digital application, curriculum, and professional development activities. The Playground Physics App allows students to record videos of embodied play and learn about the physics in their movements. The Program has gone through multiple stages of development based on teacher and student feedback and advances in technology. This session will cover the teacher and student experience of using Playground Physics, the challenges of accessibility and utilization of technology and digital tools to address problems of practice, and the lessons learned from its development and how it addresses issues of equity in underserved schools.

Presentation: Coming Soon!

Measures of the Fidelity of Instructional Coaching

This panel discussion will provide insights into the collaborative process between program developers and evaluators to create methods for measuring the fidelity of instructional coaching. The panelists come from two EIR grants focused on teacher professional learning programs (Professional Learning with Impact, MyTeachingPartner). In each program, local coaches deliver coaching using a specific model. The program developers and evaluators selected measures of the fidelity of the coaching. In addition, they chose measures of the fidelity of the support provided to coaches. The panelists will discuss how the program developers and evaluators collaborated to identify specific measures. Each program developer will briefly describe their coaching model and measures and then answer questions about the tradeoffs involved in these selections.

Presentation: PDF

Out There Thinking: Rural Innovation and the Pragmatics of Place

A common misconception about innovation is that it is akin to inspiration—sudden keen insights and novel solutions. But as Isaac Asimov put it, discovery tends not to be heralded by the phrase “Eureka!” but rather by the observation, “Hm, that’s funny…” Part of Asimov’s point was that innovation results from deeply engaged practice, experimentation, and reflection, from noticing what is working as expected and what is not, and not from lightning bolts of understanding. The same is true with innovating to improve teaching and learning in rural places. In this discussion, I will describe several principles of rural innovation, including working in community, attending to local context, engaging lived experience, and iterating thoughtfully. Throughout, I will share examples of rural education innovations to bring these principles to life for participants.

Presentation: PDF

Helping Youth Thrive with Social, Emotional, and Identity Development: Why It Matters and What Works

Emerging science tells us an optimistic story about the potential of all learners and the ways that learning settings can enable student success and healthy development. It makes clear that social, emotional, and cognitive experiences are intertwined and influence how young people learn. It also underscores the critical role of identity safety in the learning process, noting how having one’s identity nurtured and valued can enable engagement, achievement, and positive school attachments. This session provides an overview of this research base—often called the science of learning and development (SoLD)—and highlights the importance of attending to these areas of practice in teaching and learning. Attendees will also hear from district and nonprofit leaders to learn about their efforts to create or redesign school settings so that they create consistent opportunities for social, emotional, and identity development that enables all students to thrive.

Presentation: PDF

Closing Plenary: Innovation and Equity in Education

Major sectors of the economy focus on continuous improvement and innovation as a matter of necessity. In contrast, we typically do not embrace R&D in the education sector — despite the primacy of education as a driver of both individual and collective outcomes. Instead, we prioritize the two extremes of the data and evidence spectrum: compliance at one end, and lengthy, expensive impact studies at the other.

The COVID-19 pandemic has exposed this gap, as well as an opportunity to do better. Millions of dollars have gone to districts to address inequities and better serve students — but efforts to address learning loss have been put in place without a plan to learn under what conditions the proposed solutions are successful. In this session, Project Evident Founder and Chief Executive Kelly Fitzsimmons will discuss how, by connecting equity and evidence and embracing an R&D approach, we can achieve stronger and more equitable outcomes for students.

Presentation: PDF