Module 2A, Section 2: Consider Stakeholder Perceptions of the State’s System of AMD Rationale

Evaluating State Accountability Systems Under ESEA

Module 2A: State’s System of Annual Meaningful Differentiation (AMD)

Section 2: Consider Stakeholder Perceptions of the State’s System of AMD Rationale

This webpage is part of the Evaluating State Accountability Systems Under ESEA tool, which is designed to help state educational agency (SEA) staff reflect on how their accountability system achieves its intended purposes and build confidence in the state’s accountability system design decisions and implementation activities. Please visit the tool landing page to learn more about this tool and how to navigate these modules.

The overall objectives and design of a state’s accountability system should be well understood. The greater the understanding, the less risk there is of accountability results being misinterpreted. After articulating or revisiting the rationale behind the state’s overall system of AMD in Section 1 of this module, SEA staff also should examine the strengths of the rationale behind the state’s system of AMD to ensure they support state accountability objectives and are technically sound. In addition, SEA staff should examine whether the public perceptions of the system promote its intended behaviors. Together, the technical soundness and public perceptions of the state’s system of AMD are likely to determine its success.

Use the reflection questions in Table 3 to consider whether the design and presentation of the state’s system of AMD and its rationale is understood (or is likely to be understood) by stakeholders. You may print this webpage and use it as a template for note taking if working with colleagues.

Table 3. Consider Stakeholder Perceptions of the Rationale Behind the State’s System of AMD Rationale

Consider Stakeholder Perceptions of the Rationale Behind the State’s System of AMD
Perception Reflection	Why is it important?	Reflection questions	Notes
Stakeholder perceptions of the state’s system of AMD	Rationales help “connect the dots” of the state’s accountability system. It is important that stakeholders and the public understand the rationale behind the state’s system of AMD, which might include the mechanisms, connections, and assumptions that inform design decisions.	Can you easily explain the rationale behind the state’s system of AMD and how it supports your policy objectives (as articulated in the state theory of action)? How do stakeholders react to the way it is designed? Do these reactions differ by audience? What policy levers (e.g., decision rules, improvement requirements, measure selection, state’s system of AMD reporting strategies) are embedded in the rationale support or could potentially compromise your policy objectives? Are these clear to the public? Is the design of the state’s system of AMD (e.g., indicator weights, performance standards for indicators, overall performance) and how it differentiates schools clearly communicated to the public?
Potential misunderstanding of the state’s system of AMD	Public perceptions are important to increase buy-in for the system. Without considering public perceptions, advocacy groups may not understand how their concerns have been addressed and stakeholders may not understand the meaning of the state’s system of AMD results.	Which policy mechanisms or levers of the state’s system of AMD are most likely to receive public attention? Why? What constituencies, stakeholders, or advocacy groups might question the rationale for the state’s system of AMD? What kinds of questions would they ask? How might public perceptions change over time?

Based on the results of your previous reflections, consider the degree to which you believe the following statements regarding (1) communication and clarity of your rationale and (2) the risk of the public misunderstanding the rationale (Table 4).

Table 4. Clarity and Risk of the Rationale Behind the State’s System of AMD

Communication and Clarity of Rationale	No Clarification Needed	Clarification May Be Needed	Additional Clarification Needed	Notes
Communication and Clarity of Rationale	We have clearly stated the rationale behind the state’s system of AMD, and the rationale reflects the overall objectives for the accountability and support system. School performance is communicated clearly and is easily accessible by the public.	We have stated the rationale behind the state’s system of AMD, but the rationale may not clearly reflect the overall objectives for the accountability and support system. School performance differentiates schools, but it may not be easily accessible by the public.	We have not stated the rationale behind the state’s system of AMD, or the rationale does not reflect the overall objectives for the accountability and support system. School performance metrics are unclear, or we do not understand how they are differentiating schools.
Risk of Misunderstanding the Rationale	Low	Moderate	High	Notes
Risk of Misunderstanding the Rationale	We have identified possible areas of the state’s system of AMD that might be misunderstood by the public. Based on this examination, we have clarified aspects of the system and created clear documentation explaining the system.	We have examined what parts of the state’s system of AMD might be misunderstood by the public but have not clarified them fully. Documentation specifically addressing areas of risk may or may not be available.	We have not examined the state’s system of AMD for areas that could be misunderstood.

For areas that need additional clarification or those that are high risk, you may need to prioritize future efforts. The potential next steps described below (Table 5) are important to consider as you review the confidence claims in the next section. If the rationale for the state’s system of AMD needs clarification or the risk for misunderstanding is high, what would you do next? For example, an undocumented rationale may increase the risk that the state’s system of AMD does not work as intended. It is important to determine whether the risk is based on a case of lack of documentation or if it is based on incomplete or less-than-ideal assumptions. These next steps are intended to help prime your thinking or prioritize areas of interest for the remainder of this module.

Table 5. Potential Next Steps Around Stakeholder Perceptions of the Rationale Behind the State’s System of AMD

Area of Exploration	Potential Next Steps	Notes
Communication and Clarity of Rationale	Clearly document the rationale behind the state’s system of AMD. Ensure it supports your overall theory of action and the policy objectives associated with your state’s accountability system. Clearly document how you expect the state’s system of AMD to communicate school performance. Make clear how thresholds of performance for overall ratings or indicator results support the state’s system of AMD’s policy objective(s). Ensure this information is presented, formatted, and available in a way that can be understood by the public and educators throughout the state.
Risk of Misunderstanding the Rationale	Clarify what aspects of the state’s system of AMD are likely to receive the most public attention. Specify whether there are particular design decisions, measures, or reports that might be controversial or difficult to understand. Refine messages to make controversial or challenging aspects of the state’s system of AMD more accessible. Anticipate the types of questions (or engage in additional listening sessions) to highlight the most important issues to address. Document how you have addressed the two bullet points above. Identify how responses from the public perceptions and state’s system of AMD rationale Next Steps can be compiled into a single set of resources.

[Click here to continue on to the final section (Section 3) of Module 2A: State’s System of AMD]