General Recommendations

Recommendations

Use the “State Index” form provided by USED as the template for organizing the submission.

Rationale

Clearly identify the appropriate reference and where in the reference the evidence appears. Be as specific as possible. All attachments and citations should be clearly labeled and numbered sequentially. It is time consuming and frustrating for reviewers to search through a submission for cited references.

Recommendations

In the section entitled “Evidence” provide a clear and concise description of how the state meets the requirement.

Rationale

Narrative and citations of evidence that do not directly address the element give the impression that the state is not sure its system is compliant with the requirement(s). Many states submit too much material, much of it irrelevant to the questions in the Peer Review Guidance keep the evidence submissions

Recommendations

Provide duplicates of any materials that cannot be photocopied, e.g. CD-ROM, color brochures, etc.

Rationale

A State is required to submit only one copy of all materials, but if the state wishes to ensure that reviewer’s copies are complete and formatted as originally designed, that state may provide 6 complete copies of the submission.

Recommendations

List the assessments currently in use in your State. Components currently under development for future use, such as content standards, performance standards and descriptors, and assessments, should be clearly described with proposed dates for full implementation.

Rationale

In 2005-06 many states will be implementing parts of the assessment system for the first time and may not yet have implemented science assessments. Reviewers need to be clear about which assessments are currently used to calculate AYP and which are planned for future development.

Recommendations

If the state’s program does not currently meet the requirement, say so; but give the projected plan and timeline for when the requirement will be met.

Rationale

Do not attempt to obfuscate. If it appears that the state is trying to misrepresent the existing program, reviewers may question the integrity and intent of the submission and give closer scrutiny to all aspects of the documentation provided.

Recommendations

Committees, panels and reviewers cited in evidence as being used in development processes should reflect the demographics of the student population for your state, but at the least should include representatives for students with disabilities, limited English proficient students, and racial/ethnic groups.

Rationale

The participation of representatives that reflect the diversity of the state in all aspects of development is a requirement for several sections of the submission. Documentation should include some indication of panel (committee, reviewer) qualifications.

Recommendations

Participation data should include student enrollment data for the grades tested as well as the number of students assessed.

Rationale

All students must be tested, regardless of the length of time in the school. Reviewers must be able to verify that all students are included in testing so the state should provide accurate enrollment data for the grades tested.

Recommendations

Reports and studies by independent evaluators are valuable types of evidence.

Rationale

Independent evaluations and studies add a level of credibility to findings. This is particularly relevant for studies pertaining to comparability, alignment and validity but it applies to all development processes.

Reports

Critical Element

Report participation and assessments results for each of the required subgroups for the required tested grades and grade ranges at the school, LEA, and State level.

Evidence

Required subgroup reporting for assessment reports includes: gender, race/ethnicity, English proficiency status, migrant, students with disabilities, and economically disadvantaged. For LEA and State Report Cards, comparisons of students with disabilities vs. non-disabled and economically disadvantaged vs. non-economically disadvantaged are also required. Provide sample reports for the required grades.

Critical Element

Ensure that personally identifiable information about an individual student is not reported

Evidence

Documentation on alternate achievement standards must include all five criteria.

Critical Element

Provide documentation on how the State ensures student confidentiality.

Evidence

Produce individual interpretative, descriptive, and diagnostic reports following each administration of its assessments. The reports must include the following:

• valid and reliable information regarding achievement on the assessments in relation to the State’s academic content and achievement standards;
• information for parents, teachers, and principals in a format and language that is understandable, helps them address a student’s specific academic needs, and is accompanied by interpretive guidance; and
• how the State disseminates these reports as soon as possible after the assessments.

Critical Element

Ensure the identification and inclusion of migrant and other mobile students in the assessment system in the required tested grades.

Evidence

Evidence may include examples of the individual student reports for the required grades tested, interpretive guides, scoring and reporting timelines.

Critical Element

Ensure that student-level assessment data are maintained securely to protect student confidentiality

Evidence

Evidence should include policies and procedures on access to student assessment data.

Critical Element

Produce itemized score analyses so that parents, teachers, and principals can interpret and address the specific academic needs of students.

The itemized score analyses should include results for each of its academic content standards and each of the sub-domains/strands within these standards, to the extent that these sub-scores are based on enough items or score points to be meaningful

*Note:

• If different assessments are used the state must demonstrate how
  the results are aggregated to yield valid and reliable scores.
• Provide evidence on how the state monitors the quality of reports.
• School and student report cards should provide comparison data for
  student performance (e.g., how the student compares with his
  classmates, his district and the state in reading).

Inclusion

Critical Element

Participation data must be included and show that all students in the required grades/grade ranges are included in the assessments.

Evidence

Documentation must be provided that shows the participation rate is calculated as a proportion of the students enrolled and the students assessed in total and in each subgroup.

Critical Element

Provide guidelines the State has for including all students with disabilities in the regular assessment system.

Evidence

Documentation must be provided to show how the State is making efforts to include students with significant cognitive disabilities in the regular assessment and that if alternate achievement
standards are adopted for these students, guidelines and training are provided for the identification and assessment of these students.

Critical Element

If alternate achievement standards have been adopted/approved for students with the most significant cognitive disabilities, provide the following:

• guidelines for IEP teams to use in identifying which students
  should be assessed with alternate achievement standards;
• how the State ensures that parents are informed that their
  student will be assessed on alternate achievement standards;
• documentation that, to the extent possible, students with the
  most significant cognitive abilities are included in the regular
  assessments;
• documentation on how the State has developed, disseminated
  information on, and promoted use of appropriate accommodations to
  increase the numbers of these students participating in the regular
  assessment system; and
• procedures to ensure that personnel are trained in
  administration of all assessments including making use of
  accommodations.

Evidence

Documentation on alternate achievement standards must include all five criteria.

Critical Element

Provide the following:

• Whether they have provided assessments, to the extent practicable, in the language and form most likely to yield accurate and reliable information on what these students know and can do.
• Whether the State requires participation of all LEP students regardless of how long they have been enrolled in U. S. schools.
• Whether the State has adopted policies that LEP students be assessed on reading/English language arts standards in English if they have been enrolled in US schools for three consecutive years or more.

Evidence

For the 2003-04 assessments, students in their first year of enrollment in U. S. schools may be exempt from the State’s reading/language arts assessment.

Provide policies, regulations, guidelines related to inclusion of LEP students including information such as allowed accommodations.

Critical Element

Ensure the identification and inclusion of migrant and other mobile students in the assessment system in the required tested grades.

Evidence

Evidence should include policies, guidelines, and practices used by the State to include mobile students.

*Note:

• Provide participation data for students with disabilities and LEP
  students (in addition to participation rates for all students).
• Indicate how the needs of students with disabilities and LEP
  students were considered as part of the test development process.
• Indicate how the assessments are linguistically-accessible for LEP
  students.
• Describe how the range of assessment options for students with
  disabilities and LEP students yields scores that are valid and
  reliable for these groups.
• Describe how the state monitors the application and use of its
  inclusion policies.

Alignment

Critical Element

The State must document their coherent approach to ensuring alignment between each of its assessments and the content and achievement standards the assessment is designed to measure.

Evidence

Provide documentation of the process used including stakeholder involvement, ongoing quality control reviews, how the alignment is maintained over time, reports of alignment studies and how the results were addressed.

Critical Element

Provide evidence that the assessments and standards are aligned comprehensively including range, degree of cognitive complexity, level of difficulty, and depth.

Evidence

Provide evidence such as assessment plans, assessments blueprints, and item/task specifications. Include information that shows which standards are assessed and which are not along with the weights. Include any reports of independent alignment studies and studies of cognitive complexity of the assessments.

Critical Element

Assessments and standards must be aligned in terms of both content (knowledge) and process (how to do it).

Evidence

Provide evidence that processes and skills required in the content standards are assessed as well as the content.

Critical Element

Assessments and standards must be aligned in terms of degree and pattern of emphasis.

Evidence

Include information on how the assessments reflect the weights of the content standards and their sub-domains.

Critical Element

Assessments must yield scores that reflect the full range of the state’s academic achievement standards.

Evidence

Provide evidence that the state’s assessment system provides a sufficient number of items to assess students at all levels of achievement–for example, basic, proficient, advanced.

Critical Element

Assessment results must be expressed in terms of achievement
standards, not just scale scores or percentiles.

Evidence

In order for members of the school community to see the alignment between the standards and assessments, reports must include information on how the students perform relative to the achievement standards. Provide evidence such as sample student score reports.

Critical Element

Provide evidence such as sample student score reports.

Provide evidence on how it maintains and/or improves alignment of assessments and standards over time.

Evidence

Documentation should be provided on planned alignment reviews, external studies of alignment, and how assessments are modified if content standards are revised.

*Note:

• Item difficulty is not a sufficient indicator of cognitive complexity.
• Evidence should include sample items which are illustrative of cognitive difficulty. Documentation and sample items should reflect the use of multiple measures and items that require cognitively complex response processes.
• If there is a reliance on multiple-choice items, the state is responsible for providing evidence that the items span the ranges of both difficulty and cognitive complexity specified in its standards.

Technical Quality

Seven categories of validity are noted in the Guidance. To address the first category of validity, provide written documentation, such as State Board policies or state legislative code that defines the purpose of the state assessment system. For the remaining six categories provide written documentation from the studies done that provide evidence of validity. If deficiencies exist provide the plan and timeline to address them.

Critical Element

A full range of reliability must be documented.

Evidence

Three categories of reliability are noted in the Guidance. Provide written documentation of the studies that support the reliability of the assessments with the State’s own student population. Also provide documentation of the precision of cut scores and the consistency of student classification. If any deficiencies exist (such as in establishing cut scores or in classification consistency) provide the plan and timeline to address them.

Critical Element

The assessment system must be fair and accessible to all students, including students with disabilities and students with limited English proficiency.

Evidence

The State assessment system must be designed to be valid and accessible for use by the widest possible range of students. Provide evidence that there is an appropriate variety and number of accommodations to meet the needs of students with disabilities and students with limitedEnglish proficiency; that individuals with experience and expertise in working with
SWD and LEP students were included in test development processes; that educators are trained on the appropriate selection and use of accommodations and alternate assessments.

Statewide Assessment

Critical Element

The State assessment system may include different types of
assessments (e.g., CRT, Augmented NRT, and various types of Alternate Assessments).

Evidence

Complete the Overview chart provided in the Guidance. Clearly indicate what type (s) of assessments are used for each required grade and subject area (s).

Critical Element

If different assessments are used (e.g., CRT, ANRT, native language assessments, simplified English versions), document a rational overall design and coherence, as well as alignment with the academic content and achievement standards.

Evidence

Provide the rationale and blueprint for the design. Also provide documentation of the processes and reports from studies that clearly indicate coherence, alignment, and comparability.

Critical Element

Multiple measures that assess higher-order thinking skills and understanding of challenging
content.

Evidence

Provide test blueprints, item specifications, statistical evidence, descriptions of the processes used to judge the cognitive level of the test or item content. The intent of the evidence is to ensure content coverage in terms of depth, breath and cognitive load. It is also useful to provide samples of the measures being used.

Critical Element

The State assessment system may employ a matrix design. If a matrix design is used, the
multiple forms within a content area and grade must:

• all be aligned with the content and achievement standards;
• be equivalent in terms of content coverage, difficulty and quality; and
• yield comparable results.

Evidence

Provide technical manuals, state reports and/or reports from independent evaluators on the quality, equivalence, and comparability of the forms.

Critical Element

The State assessment system may include alternate assessments. Student achievement may be measured against grade level standards or against alternate achievement standards.

Evidence

Provide:

• written documentation describing the processes used to develop the alternate assessment(s) and the associated achievement standards;
• documentation of the criteria that local IEP teams should use to determine which students are eligible to participate in the alternate assessment;
• sample score reports from the alternate assessment or other materials used to communicate the meaning of results to parents.

Critical Element

Local assessments, if used as part of the State assessment system, must:

• be aligned with the state academic and achievement standards;
• be equivalent to one another in terms of coverage, difficulty and quality;
• yield comparable results for all subgroups;
• yield results that can be aggregated with other local assessments
  and with the state assessments; and
• provide unbiased, rational and consistent determinations for AYP.

Evidence

Provide documentation of the processes used by the State to ensure quality and comparability,
including reports done by independent evaluators as well as follow-up plans to address any identified deficiencies.

*Note If the state assessment system is comprised of more than one assessment, the state must provide reports from comparability studies for the various assessments to document that all components are aligned with the same academic content and achievement standards*

Academic Achievement

Critical Element

Reading/language arts and Mathematics: Achievement standards for each of grades 3-8 and 10-12 grade range.

Evidence

Formal adoption/approval of challenging academic achievement standards for all public schools and students in the state is required by 2005-2006. Provide evidence such as state statutes, regulations, State Board minutes or if approved by Chief State School Officer, written documentation of formal approval. The evidence requirement also applies to any revision of cut scores and levels. The evidence must specifically state that the standards apply to all students, unless alternate achievement standards have been developed for students with the most severe cognitive disabilities. In this case, include the formal adoption/approval of the alternate achievement standards.

Critical Element

Science: Achievement standards for each of the grade spans 3-5, 6-9, and 10-12.

Evidence

The standards must apply to all public schools and students in the state. As with reading/language arts and mathematics, achievement descriptors for science are due by 2005 – 2006. However, cut scores for science are not due until 2007-08. Provide evidence such as state statutes, regulations, or State Board minutes. The evidence requirement also applies to any revision of cut scores and levels. The evidence must specifically state that the standards apply to all students, unless alternate achievement standards have been developed for students with the most severe cognitive disabilities. If these are under development, describe the development process and timelines.

Critical Element

Reading/language arts, Mathematics and Science: Alternate achievement standards for students with the most severe cognitive disabilities (if alternate achievement standards have not been developed then the alternate assessment must be based on grade level achievement standards).

Evidence

Alternate achievement standards must be:

• aligned with the State’s academic content standards,
• promote access to the general curriculum and
• reflect professional judgment of the highest achievement standards possible.

Provide evidence such as how the alternate standards were defined and how they are linked to grade level content.

Critical Element

The Academic and Alternate achievement standards must include:

• at least 3 levels (2 indicating high achievement and 1 indicating basic achievement);
• descriptions that clearly define the competencies associated with each level;
• cut scores that differentiate between the levels; and
• rationale and procedures used to determine the levels.

Evidence

The submission must include all four criteria. As evidence, provide official summary reports and documentation of the process (including how impact data was considered).

Critical Element

In addition, for alternate achievement standards the state must document that it has:

• implemented guidelines for IEP teams to apply in deciding when an individual student should be assessed on the basis of alternate achievement standards;
• ensured that parents are informed when a child’s achievement will be based on alternate achievement standards and any possible consequences imposed by the LEA or State;
• reported separately the number and percent of those students with disabilities assessed against alternate achievement standards, those assessed on an alternate assessment against grade-level standards, and those included in the regular assessment (including those administered with appropriate accommodations);
• documented that students with the most significant cognitive disabilities are included in the general curriculum to the extent possible;
• taken steps to promote use of appropriate accommodations for students tested against grade-level standards; and
• provided information for teachers and other staff regarding appropriate test administration practices, including use of accommodations for students with the most significant cognitive disabilities.

Evidence

The submission must include all six criteria. As evidence, provide official summary reports, documentation of the processes (including the involvement of parents in decision making), guidelines for the use of the alternate assessment, training materials and sample test reports.

Critical Element

The academic and alternate achievement standards must be aligned with the content standards.

Evidence

The achievement standards should fully reflect the content standards for each required grade and describe the content based expectations each achievement level represents. Provide the descriptions of the process used to develop the levels, descriptions and cut scores and how the state assures itself that alignment is present.

Critical Element

The development of achievement standards must involve diverse stakeholders.

Evidence

The development of achievement standards must involve a broad range of stakeholders that reflect the diversity and needs of students in the state. Provide descriptions of the group compositions; minutes or summaries from public hearings and/or public reviews; and reports which indicate the degree of involvement. Special attention should be made to include individuals knowledgeable of and concerned about the various categories of special needs students.

Content Standards

Critical Element

Reading/language arts and Mathematics: Content standards for each of grades 3-8 and 10-12 grade range or content standards for grade ranges with specific content expectations for each grade.

Evidence

Formal adoption/approval of challenging academic content standards for all public schools and students in the state is required. Provide evidence such as state statutes, regulations, State Board minutes or if approved by Chief State School Officer, written documentation of formal approval.

If the content standards have been adopted/approved in the past and
revisions have been made to the standards, provide evidence of the formal adoption/approval of the revised content standards or a timeline for the approval and implementation.

Critical Element

Science: Content standards for grade ranges 3-5, 6-9, and 10-12

Evidence

Formal adoption/approval of challenging academic content standards for all public schools and students in the state is required by 2005-06. Provide evidence such as state statutes, regulations, State Board minutes or if approved by Chief State School Officer, written documentation of formal approval. If they have not been approved yet include your plan and timeline for development.

Critical Element

Academic content standards must be challenging with rigorous content and encourage the teaching of advanced skills.

Evidence

Evidence may include aspects of your development process which address higher order thinking skills or studies conducted by an external group. If recommendations have been made for changes to the academic content standards to make them more rigorous, provide documentation on how you addressed these recommendations and how the academic content standards have been changed.

Critical Element

Involvement of education stakeholders in the development of its academic content standards.

Evidence

Stakeholders include educators, parents, community members, higher education representatives, and representatives for student with disabilities and students with limited English proficiency.

*Note: Academic content standards are the basis for the development of performance descriptors, assessments, and achievement standards. Clearly identify any continued development/revisions of the academic content standards and how the revision has impacted the development/revision of the assessment system.*

Requirements at a Glance

Fulfilling the NCLB requirements for a state’s standards and assessment system can be challenging. In preparing the submission states must first decide if their program meets the requirements. Then the appropriate documentation must be attached to substantiate their response. Most state departments house large volumes of reports, memos, policies and regulations. Although extensive amounts of evidence may be submitted, in reviewing submissions we have found that states often overlook, omit or fail to adequately address essential aspects of critical elements. In this part of the Guide for each section of the requirements we cite the critical elements and appropriate evidence. As needed, we also provide reminders under the heading of “Notes”.

Conclusion

It is our hope that this Guide will facilitate the process, provide information that will enhance the submission and thus expedite the review process.

Return to Standards, Assessment and Accountability