Module 2B, Section 2. Stakeholder Perceptions of Rationale for Combining Indicators
Evaluating State Accountability Systems Under ESEA
Module 2B: Indicator Interaction in the State’s System of Annual Meaningful Differentiation (AMD)
Section 2. Stakeholder Perceptions of Rationale for Combining Indicators
This webpage is part of the Evaluating State Accountability Systems Under ESEA tool, which is designed to help state educational agency (SEA) staff reflect on how the state’s accountability system achieves its intended purposes and build confidence in the state’s accountability system design decisions and implementation activities. Please visit the tool landing page to learn more about this tool and how to navigate these modules.
Each component of a state’s accountability system should be well understood. The greater the understanding, the less risk there is of indicators being misunderstood or misused (e.g., incorrect interpretations, overemphasis on importance or influence). A state can benefit from reflecting on how its indicators interact within its state’s system of AMD and whether the rationale is (or is likely to be) understood by stakeholders. Based on the responses to the questions below, a state can determine whether design assumptions and sources of evidence are clear to stakeholders or there is a need to create or organize additional documentation, increase transparency, or engage in outreach efforts with stakeholder groups. This section includes recommendations for potential next steps in stakeholder outreach efforts.
You may print this webpage and use it as a template for note-taking if working with colleagues.
Table 3. Stakeholder Perceptions of Rationale for Combining Indicators
|Stakeholder Perceptions of Rationale for Combining Indicators|
|Perception Reflection||Why is it important?||Reflection Questions||Notes|
|Stakeholder perceptions of how indicators are combined||Rationales help “connect the dots” of the state’s accountability system. It is important that stakeholders and the public understand the rationale behind your system (as articulated in Section 1 of this module), which might include the mechanisms, connections, and assumptions that inform design decisions and evidence collection.||
|Potential misunderstanding of how indicators are combined||Public perceptions are important to increase buy-in for the system. If a state does not consider public perceptions, advocacy groups may not understand how their concerns have been addressed, and stakeholders may not understand why indicators were weighted certain ways.||
Based on the results of your previous reflections, consider the degree to which you believe the following regarding (1) communication and clarity of your rationale and (2) the risk of the public misunderstanding the rationale.
Table 4. Clarity and Risk of the Indicator Interactions
|Communication and Clarity of Rationale||No Clarification Needed||Clarification May Be Needed||Additional Clarification Needed||Notes|
|We have clearly stated the rationale behind the weighting of indicators or the order of decision rules, and the rationale reflects the overall objectives for the accountability and state’s system of AMD. These expectations are based on past experience or research and are readily available to the public.||We have stated the rationale behind the weighting of indicators, or the order of decision rules is stated, but the rationale may not clearly reflect the overall objectives for the accountability and support system. These expectations may have some support from previous experience or research. Some of the rationale is available to the public.||We have not clearly stated the rationale behind the weighting of indicators, or the order of decision rules, or the rationale does not reflect the overall objectives for the accountability and support system. These expectations have not been supported with prior experience or research. Supporting materials are not available to the public.|
|Risk of Misunderstanding the Rationale||Low||Moderate||High||Notes|
|We have identified possible areas of the indicator weighting/decision rules that might be misunderstood by the public. Based on this examination, we have clarified aspects of the system and created clear documentation explaining the system.||We have examined what parts of the indicator weighting/decision rules might be misunderstood by the public but have not clarified them fully. Documentation specifically addressing areas of risk may or may not be available.||We have not examined the indicator weighting/decision rules for areas that could be misunderstood.|
For areas that need additional clarification or those that are high risk, you may need to prioritize future efforts. The potential next steps described below are important to consider as you review the confidence claims in the next section. If the rationale for combining indicators or the order of decision rules needs clarification or the risk for misunderstanding is high, what would you do next? For example, an undocumented rationale may increase the risk that indicator weights are not working as intended because of a lack of documentation, or it may be a result of incomplete or less than ideal assumptions. These considerations are intended to help prioritize next steps in supporting stakeholder perceptions of the state’s accountability system.
Table 5. Potential Next Steps Around Stakeholder Perceptions of Indicator Interaction
|Area of Exploration||Potential Next Steps|
|Communication and Clarity of Rationale||
|Misunderstanding the Rationale||
[Click here to continue to the final section (Section 3) of Module 2B: Indicator Interaction in the State’s System of AMD.]