Module 2B, Section 2. Stakeholder Perceptions of Rationale for Combining Indicators

Evaluating State Accountability Systems Under ESEA

Module 2B: Indicator Interaction in the State’s System of Annual Meaningful Differentiation (AMD)

Section 2. Stakeholder Perceptions of Rationale for Combining Indicators

This webpage is part of the Evaluating State Accountability Systems Under ESEA tool, which is designed to help state educational agency (SEA) staff reflect on how the state’s accountability system achieves its intended purposes and build confidence in the state’s accountability system design decisions and implementation activities. Please visit the tool landing page to learn more about this tool and how to navigate these modules.

Each component of a state’s accountability system should be well understood. The greater the understanding, the less risk there is of indicators being misunderstood or misused (e.g., incorrect interpretations, overemphasis on importance or influence). A state can benefit from reflecting on how its indicators interact within its state’s system of AMD and whether the rationale is (or is likely to be) understood by stakeholders. Based on the responses to the questions below, a state can determine whether design assumptions and sources of evidence are clear to stakeholders or there is a need to create or organize additional documentation, increase transparency, or engage in outreach efforts with stakeholder groups. This section includes recommendations for potential next steps in stakeholder outreach efforts.

You may print this webpage and use it as a template for note-taking if working with colleagues.

Table 3. Stakeholder Perceptions of Rationale for Combining Indicators

Stakeholder Perceptions of Rationale for Combining Indicators
Perception Reflection	Why is it important?	Reflection Questions	Notes
Stakeholder perceptions of how indicators are combined	Rationales help “connect the dots” of the state’s accountability system. It is important that stakeholders and the public understand the rationale behind your system (as articulated in Section 1 of this module), which might include the mechanisms, connections, and assumptions that inform design decisions and evidence collection.	Can you easily explain the rationale behind the weights for each indicator or the order of decision rules in the state’s system of AMD and how this rationale contributes to your policy objectives? How do stakeholders react to the way in which indicators are weighted or ordered? Do these reactions differ by audience? Are the linkages between policy decisions, indicator outputs, and expected behaviors based on state experience or research? How are these linkages presented to the public? Would the public perceive them as conceptually defensible? Are the connections between the indicator weights or decision rules clearly communicated with the public?
Potential misunderstanding of how indicators are combined	Public perceptions are important to increase buy-in for the system. If a state does not consider public perceptions, advocacy groups may not understand how their concerns have been addressed, and stakeholders may not understand why indicators were weighted certain ways.	Which indicator weights or decision rules are most likely to receive public attention? Would various constituencies understand or question why certain indicators were weighted or ordered in certain ways? Which indicators may be obscured by others? Can stakeholders quickly understand from accountability data presentations of indicators which areas of student performance are in the most need of attention?

Based on the results of your previous reflections, consider the degree to which you believe the following regarding (1) communication and clarity of your rationale and (2) the risk of the public misunderstanding the rationale.

Table 4. Clarity and Risk of the Indicator Interactions

Communication and Clarity of Rationale	No Clarification Needed	Clarification May Be Needed	Additional Clarification Needed	Notes
Communication and Clarity of Rationale	We have clearly stated the rationale behind the weighting of indicators or the order of decision rules, and the rationale reflects the overall objectives for the accountability and state’s system of AMD. These expectations are based on past experience or research and are readily available to the public.	We have stated the rationale behind the weighting of indicators, or the order of decision rules is stated, but the rationale may not clearly reflect the overall objectives for the accountability and support system. These expectations may have some support from previous experience or research. Some of the rationale is available to the public.	We have not clearly stated the rationale behind the weighting of indicators, or the order of decision rules, or the rationale does not reflect the overall objectives for the accountability and support system. These expectations have not been supported with prior experience or research. Supporting materials are not available to the public.
Risk of Misunderstanding the Rationale	Low	Moderate	High	Notes
Risk of Misunderstanding the Rationale	We have identified possible areas of the indicator weighting/decision rules that might be misunderstood by the public. Based on this examination, we have clarified aspects of the system and created clear documentation explaining the system.	We have examined what parts of the indicator weighting/decision rules might be misunderstood by the public but have not clarified them fully. Documentation specifically addressing areas of risk may or may not be available.	We have not examined the indicator weighting/decision rules for areas that could be misunderstood.

For areas that need additional clarification or those that are high risk, you may need to prioritize future efforts. The potential next steps described below are important to consider as you review the confidence claims in the next section. If the rationale for combining indicators or the order of decision rules needs clarification or the risk for misunderstanding is high, what would you do next? For example, an undocumented rationale may increase the risk that indicator weights are not working as intended because of a lack of documentation, or it may be a result of incomplete or less than ideal assumptions. These considerations are intended to help prioritize next steps in supporting stakeholder perceptions of the state’s accountability system.

Table 5. Potential Next Steps Around Stakeholder Perceptions of Indicator Interaction

Area of Exploration	Potential Next Steps
Communication and Clarity of Rationale	Clarify the rationale behind the weights of the indicators, or the ordering of decision rules, that comprise your state’s system of AMD. Ensure they support your overall theory of action. Clarify what signals each indicator is supposed to send and how they contribute to the state’s conceptualization of school performance. Document how you have addressed the two bullet points above. Ensure this information is presented, formatted, and available in a way that it can be shared with the public and educators throughout the state.
Misunderstanding the Rationale	Clarify what indicators or weights are likely to receive the most public attention. Specify whether there are particular design decisions, measures, or data associated with each indicator that might be controversial or difficult to understand. Refine messages to make controversial or challenging aspects of each indicator more accessible. Anticipate the types of questions (or engage in additional listening sessions) to highlight the most important issues to address. Document how you have addressed the two bullet points above. Identify how public perceptions and your reflections can be compiled into a single set of resources. Combine this with other documentation from other modules.

Area of Exploration

Potential Next Steps

Communication and Clarity of Rationale

Clarify the rationale behind the weights of the indicators, or the ordering of decision rules, that comprise your state’s system of AMD. Ensure they support your overall theory of action.
Clarify what signals each indicator is supposed to send and how they contribute to the state’s conceptualization of school performance.
Document how you have addressed the two bullet points above. Ensure this information is presented, formatted, and available in a way that it can be shared with the public and educators throughout the state.

Misunderstanding the Rationale

Clarify what indicators or weights are likely to receive the most public attention. Specify whether there are particular design decisions, measures, or data associated with each indicator that might be controversial or difficult to understand.
Refine messages to make controversial or challenging aspects of each indicator more accessible. Anticipate the types of questions (or engage in additional listening sessions) to highlight the most important issues to address.
Document how you have addressed the two bullet points above. Identify how public perceptions and your reflections can be compiled into a single set of resources. Combine this with other documentation from other modules.

[Click here to continue to the final section (Section 3) of Module 2B: Indicator Interaction in the State’s System of AMD.]