Evaluating State Accountability Systems Under the ESEA

The Purpose of this Tool

The purpose of Title I of the ESEA is to provide all children significant opportunity to receive a fair, equitable, and high-quality education and to close achievement gaps (§1001). State accountability systems under Title I support meeting this purpose by working to improve student achievement and school success (§1111(c)(4)). A state’s accountability system is a key lever for school improvement and improving student outcomes. Each state’s accountability system should focus on key indicators of student outcomes and achievement.

Each state’s accountability system should support its overall theory of action for improving student outcomes for all students and closing achievement gaps for groups of students who are behind, including students in schools identified for comprehensive or targeted support and improvement.

State accountability systems should be a living system; that is, an SEA should continue to evaluate whether the system is working as intended and supporting continued school improvement. While it is important to have stability in the system, to provide schools and districts a clear set of expectations and metrics, ongoing evaluation and refinement will be needed to identify strategies for continuous improvement. The purpose of this tool is to provide a process for SEA staff to engage in this type of reflection, where SEA accountability specialists and other SEA staff can:

  • Review their state’s accountability system.
  • Document areas where their state’s accountability system is and, if applicable, is not functioning as intended.
  • Identify results to inform communications with stakeholders about their state’s accountability system, including the strengths of the system to date.
  • Identify data to inform needed adjustments to their state’s accountability system.

As a result of completing this tool, a state should be able to answer the following types of questions:

  • How clear is our state’s theory of action?
  • How well does our state’s system for annually meaningfully differentiating schools communicate results to the public?
  • Are the results of our state’s system for annual meaningful differentiation of schools aligned with the goals of our state’s accountability system?
  • How do we know if our state’s system of annual meaningful differentiation has identified the “right” low-performing schools for support and improvement (i.e., schools for comprehensive or targeted support and improvement)?
  • Are the indicators in our state’s system of annual meaningful differentiation interacting as intended?
  • Should we explore any indicators in greater detail?
  • What, if any, are the unintended consequences of our accountability system?
  • Is our reporting approach helping us achieve the intended outcomes of our state’s accountability system?

States’ experiences with accountability systems over the past several decades have shown that annual meaningful differentiation and identification of low-performing schools is only the first phase in a comprehensive system designed to build local capacity through support and guidance, and that the first phase alone is insufficient to drive improvement. Reflecting on how a state’s accountability system functions after the state runs its system of annual meaningful differentiation and identifies schools can help SEA staff consider how the accountability system indicators align with state supports for low-performing schools (e.g., coaching, evaluation) as well as policy incentives.

This tool is designed to help state educational agency (SEA) staff reflect on how their state’s accountability system under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the Every Student Succeeds Act (ESSA), is functioning and (where applicable) identify strategies for continuous improvement. The tool provides SEA staff with a process to evaluate their state’s accountability system. SEA staff can complete the tool in its entirety or complete selected modules to target specific components of their plan. This tool is intended to spur SEA staff’s considerations for evaluating their accountability systems; a complete evaluation is likely to take significant time and effort beyond the reflections articulated in the tool. To read more about the purpose of the tool, click on the “The Purpose of This Tool” header above.

The Organization of this Tool

This tool includes seven modules that SEA accountability staff can use to reflect on how the state’s accountability system is functioning. The modules are shown in Figure 1 below.

Diagram showing the Progression of Modules 1-7

Module 1: Theory of Action Module 2A: State’s System of Annual Meaningful Differentiation (AMD) Module 2B: Indicator Interaction in the State’s System of Annual Meaningful Differentiation (AMD) Module 3A-3E: Indicators Module 4: Comprehensive Support and Improvement (CSI) Schools Module 5: Targeted Support and Improvement (TSI) and Additional Targeted Support and Improvement (ATSI) Schools Module 6: Reporting Module 7: State Support System for Identified Schools

$(document).ready(function (e) { $(‘img[usemap]’).rwdImageMaps(); });

$(“area”).hover(function () { var areaId = $(this).attr(“id”); var areaIdNumber = areaId.replace(“area”, “”);

$(“#image-map”).attr(‘src’, ‘/sites/img-map-assets/ssntool-evaluating-‘ + areaIdNumber + ‘.png’); }, function () { $(“#image-map”).attr(‘src’, ‘/sites/img-map-assets/ssntool-evaluating.png’); });

 

Resources

How to Use this Tool

To use this tool, the SEA needs to determine which modules to use and who will be involved (e.g., SEA staff, LEA staff, other stakeholders).

Selecting Modules to Complete

The SEA may complete this tool in its entirety or complete selected modules. It is recommended that the SEA begin use of the tool by completing Module 1: Theory of Action. When using these modules, we encourage SEA staff to complete all sections within each module.

This tool can be used by states in various stages of accountability system validation efforts. SEA staff should determine which modules of this tool to complete based on where they are in documenting and evaluating the results of their state’s accountability system.

  • For states with clear and consistent documentation of their state’s accountability system theory of action and results, this tool can provide an external validation process to help confirm design and implementation decisions.
  • For states that need to formalize their documentation of their state’s accountability system theory of action and results, this tool can provide a process for gathering or creating this documentation.
  • For states that need to confirm their understanding of their state’s accountability system theory of action and results across staff from various SEA offices (and new SEA staff, if applicable), this tool can help build a common understanding of the state’s accountability system and provide a process for documenting outcomes and impact.

Selecting SEA Staff to Participate

The SEA should determine the team or lead individual who will engage with the tool. This tool is designed to be used by SEA accountability specialists and their colleagues (e.g., state accountability directors, SEA staff overseeing school improvement).1 The SEA staff who use this tool will need a detailed understanding of state accountability systems and access to relevant data to fully engage with the tool. A team of SEA staff can come together to use this tool to guide group reflection and discussion; or, alternatively, an SEA accountability specialist may first use this tool to guide his/her own individual reflection, gather input or data as needed from colleagues or other stakeholders (e.g., LEA staff), and then share his/her results with SEA leaders to inform next steps. SEA staff (whether individually or as a group) can view this tool as a webpage or can download the tool as a pdf file.

Table 1: Suggested SEA Staff for Completing Each Module

Modules 1, 2A, and 2B Modules 3A–E, 4, and 5 Modules 6 and 7
Policy leaders: Individuals who are familiar with the policy history and intent of the state accountability system

Accountability and school support system designers: Individuals who are responsible for the development and implementation of the system and how improvement efforts are delivered

SEA researchers and data analysts: Individuals who can conduct analyses and interpret results alongside system designers to confirm design assumptions

Accountability and school support system designers: Individuals who are responsible for the development and implementation of the system and how improvement efforts are delivered

SEA researchers and data analysts: Individuals who can conduct analyses and interpret results alongside system designers to confirm design assumptions

Accountability and school support system designers: Individuals who are responsible for the development and implementation of the system and how improvement efforts are delivered

Information technology and programming specialists: Individuals who are responsible for making public reporting happen

Due to the complexity and scope of state accountability systems, using this tool will take significant time by the SEA staff, regardless of the number of modules completed (see Table 1). However, the tool also provides a significant opportunity to increase SEA staff’s capacity by providing a process to think critically about their states’ theories of action, implementation of their accountability systems, and overall system alignment. For additional technical assistance, you can engage your regional Comprehensive Center or any technical advisory committees in place in the state.

This tool is designed to be used by a team of SEA staff. The tool can be used by a team of SEA staff to consider how to strengthen or revise the state’s accountability system to better meet the state’s goals and outcomes for students and schools. SEA staff can use this tool together or an individual can use the tool on their own, gathering input or data from colleagues or local educational agency (LEA) staff. This tool provides self-reflection exercises and does not include any data sharing or documentation of results. SEA staff can use this tool to guide conversations and reflections about their state accountability system. To read more about how to use this tool, click on the “How to Use This Tool” header above.

To use the tool beginning with Module 1: Overall Theory of Action, click on the link below:

[Click here to use the tool beginning with Module 1: Overall Theory of Action]

To explore other modules in this tool, click on any of the module names in the above list or on the above graphic.

To access the downloadable PDF version of the tool, click here.

 

Thank you to Juan D’Brot from the National Center for Assessment, Kerstin LeFloch from American Institutes for Research, and David English formerly with American Institutes for Research for their support and contributions to this resource.

Suggested citation: D’Brot, J., LeFloch, K., English, D., Jacques, C. (2020). State Support Network: Evaluating State Accountability Systems. Washington, DC: American Institutes for Research.


1 The SEA staff who use this tool will need detailed understanding of state accountability systems and access to relevant data to fully engage with the tool. [Back]

Evaluating State Accountability Systems Under the ESEA