Selecting Evidence-Based Practices for Tier 4: Gathering Evidence Over Time

Tier 4 evidence-based practices (EBPs) include logic models based on high-quality research or evaluation. Tier 4 EBPs can provide schools and districts with an opportunity to implement innovative, targeted interventions that reflect local needs and have the potential to ignite meaningful changes but do not yet have a rigorous evidence base (i.e., where researchers have not yet demonstrated that these practices have had a significant positive impact on the desired outcomes). In addition, schools and districts may contribute to the evidence base for these promising practices over time through their own monitoring and evaluation efforts.

EXAMPLES OF STATE-, DISTRICT-, AND SCHOOL-IDENTIFIED EBPS THAT MEET TIER 4 CRITERIA

Culturally Responsive Positive Behavior Intervention Supports (CR-PBIS): This EBP was developed by an urban, high-poverty, linguistically diverse school struggling with behavioral challenges. It builds on the Positive Behavior Incentive System (PBIS) by integrating culturally responsive practices into the system. The CRPBIS includes evidence-based practices intended to foster the development of academic behavior and cultural competencies in all students and acknowledges that gaps between school and home behavior expectations may exist.
Critical Student Intervention: This EBP was designed by IDEA public schools in Texas to help to close performance gaps in mathematics for student subgroups (e.g., special education students). This approach was developed after multiple other strategies and approaches to intervention were unsuccessful. After engaging in case studies of schools without performance gaps and engaging stakeholders in strategic feedback sessions, the district team designed the Critical Student Intervention model, which included earlier and more integrated interventions as well as culturally responsive approaches. The Critical Student Intervention model utilizes research-based interventions within a customized delivery approach within the school day.
Massachusetts Level 4 and 5 School Turnaround Model: In Massachusetts, the 2010 Achievement Gap Act allowed the state to expand its involvement in low-performing schools; however, the Massachusetts Department of Elementary and Secondary Education (MA-DESE) did not have a clear road map from existing research for how to best approach improvement. Instead, state staff worked with local stakeholders to design and implement a customized turnaround model, focused on strong school leadership and professional collaboration, improved classroom instruction, individualized student support, and positive school climate. MA-DESE also commissioned impact studies to identify the specific practices linked to program success and inform changes over time.

Tier 4 EBPs require substantial planning and information gathering over time to evaluate the effectiveness of the intervention (i.e., to gauge both the quality and fidelity of implementation). Therefore, it is important to consider the evaluation up front when designing or selecting EBPs under Tier 4. This differs from the needs assessment process, which should be used thoughtfully to ensure these EBPs are the best fit for local needs. It is important that schools and districts use the needs assessment to ensure that the EBPs are targeting the true challenges of the school or district rather than the symptoms. One of the components of the needs assessment process is a root cause analysis; for districts considering Tier 4 EBPs, it may be helpful to revisit the root cause analysis to reflect on the need for innovative EBPs and what the most promising approaches might be. For more information, see the Needs Assessment Guidebook from the State Support Network and the resources on root cause analyses from the Center on Great Teachers and Leaders.

Creating a Logic Model for Tier 4 EBPs

A logic model maps how the resources and activities associated with selected EBPs will lead to intended outputs and outcomes over time. Staff should consider creating a logic model that outlines the theory of action or theory of change for the selected EBPs, as applied to the local program context; this will help inform the continuous evaluation efforts required under Tier 4. Figure 1 below shows an example of a logic model from the Pacific Regional Educational Laboratory (REL Pacific).

For more information on developing logic models, see the information from the REL Pacific on creating Logic Models and the Reform Support Network on creating a Theory of Action. This resource also provides an online tool that allows users to create their own logic model using a computer program.

Examining Tier 4 EBP Implementation and Results

Districts will also need to carefully examine the progress of selected Tier 4 interventions by identifying expected outcomes, tracking implementation, collecting follow-up data, and analyzing data to inform ongoing efforts. The expected outcomes, data sources and data collection and analysis processes should be determined during the creation of the logic model and included in the final logic model for the EBP implementation. Local staff can help to streamline this process by considering what data they are collecting that addresses key outcomes for the EBP or what data they might have school staff gather iteratively over time. Districts may consider the following when designing progress evaluation approaches:

Differentiate between measures of perception, behavior, and student learning: Surveys are a common evaluation measure, but they are limited in that they measure changes in perception only. Local staff may also consider using measures of changes in behavior (e.g., observation, meeting minutes) and measures of student learning (e.g., student work, quiz results).
Use a variety of short-term and long-term measures: While changes in perception may happen quickly, changes in behavior and achievement often take much longer to become apparent. Using a variety of short-term and long-term measures (including both formative and summative data) can help ensure that the EBP implementation is on track and working as planned or allow time for timely modifications.
Don’t let perfect be the enemy of the good: Meaningful change and progress may happen in fits and starts or may involve some growing pains. Local staff may benefit from focusing on using data to inform implementation over time, including refinements and improvements to implementation approaches or supports. Involving stakeholders early in the process can also help them understand the implementation “story” and bolster their support during challenges.