Wyoming Assessment Letter

May 10, 2006

The Honorable Jim McBride
Superintendent
Wyoming Department of Education
Hathaway Building
2300 Capitol Avenue
Cheyenne, WY 82002-0050

Dear Superintendent McBride:

Thank you for submitting Wyoming’s assessment materials for review under the standards and assessment requirements of the Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). We appreciate the efforts required to prepare for the peer review and hope that the process provides useful feedback that will support Wyoming’s efforts to monitor student progress toward challenging standards.

External peer reviewers and U.S. Department of Education (ED) staff evaluated Wyoming’s submission and found, based on the evidence received, that it did not meet all the statutory and regulatory requirements of Section 1111(b)(3) of the ESEA. I know that my staff has discussed the results of this review with your staff. However, I want to take this opportunity to enumerate the evidence that Wyoming must provide in order to have a fully compliant standards and assessment system under NCLB. That evidence is listed on the last pages of this letter.

For this initial review, very few details were provided regarding the technical quality, alignment, reliability, and validity of the Wyoming assessment system, making it almost impossible to determine whether the system meets professionally accepted testing standards and NCLB requirements. I understand you have submitted additional evidence for a second round of peer review that is currently underway. Following the review, you will receive a letter detailing any additional evidence needed, if any, to demonstrate that Wyoming’s system meets the standards and assessment requirements.

Enclosed with this letter are detailed comments from the peer review team that evaluated Wyoming’s assessment materials in February. The peer reviewers, experts in the areas of standards and assessment, review and discuss a State’s submission of evidence and prepare a consensus report. I hope you will find the reviewers’ comments and suggestions helpful. I also want to remind you of our offer to provide you further technical assistance at your request.

We look forward to working with Wyoming to support a high-quality assessment system. If you would like to discuss this further or request technical assistance, please do not hesitate to call Sue Rigney (202-260-0931) or Patrick Rooney (202-205-8831) of my staff.

Sincerely,

Henry L. Johnson

Enclosure

cc: Governor Dave Freudenthal
Annette Bohling

Summary of Additional Evidence that Wyoming Must Submit to Meet ESEA Requirements for the Wyoming Assessment System

1.0 – CONTENT STANDARDS

1. Provide results from an external evaluation that documents that the language arts, mathematics, and science content standards are challenging and contain rigorous and coherent content (this might be included as part of an overall alignment study).
2. Provide documentation that diverse stakeholders were involved in development of the content standards. Specifically, provide evidence of the consideration of the applicability to all students, including students with disabilities and limited English proficient students.

ACADEMIC ACHIEVEMENT STANDARDS

1. Provide documentation of the standard setting process employed, qualifications of the panelists, and final recommendations for the Proficiency Assessments for Wyoming Students (PAWS) and the PAWS-Alt assessments.
2. Provide evidence of adoption of the achievement standards for the PAWS and the PAWS-Alt assessments by the Board.
3. Provide a plan and timeline for completion and adoption of the final academic achievement descriptors and cut scores for the science assessment.
4. Produce evidence that the standard-setting panelists reflect knowledge of students with disabilities and limited English proficient students.

FULL ASSESSMENT SYSTEM

1. Provide evidence that the operational PAWS and PAWS-Alt assessments have been administered to all students in grades 3-8 and 11.
2. Provide documentation regarding the determination of student scores.

TECHNICAL QUALITY

1. Provide, at a minimum, basic validity and reliability information (i.e., consequential validity, item interrelationships and structural consistency, criterion validity and standards setting procedures) beginning with the 2004-05 pilot.
2. Due to the nature of recent and planned changes to the PAWS assessment program, provide technical manuals for 2004-05, 2005-06, and 2006-07 assessments. This practice should continue annually until the assessment system stabilizes.
3. Provide complete reliability analyses for the 2005-06 PAWS and PAWS-Alt assessments (note: the 2004-05 Wyoming Comprehensive Assessment System (WyCAS) technical manual chapter on reliability was comprehensive, consistent with the peer review guidelines, and would serve as a good model for such evidence).
4. Provide detailed standards-setting plans and procedures. Provide a rationale for the adequacy of the proposed 2-phase process in determining AYP for the PAWS and PAWS-Alt, including:
   1. Detailed information on the materials and process used, panelist instructions, and results of the panel’s recommendations;
   2. Specific information regarding the processes for reconciling results from the 2006 Blue Ribbon Panel and the Summer 2005 standards-setting participants; and
   3. Detailed description of Wyoming’s scaling and equating design.
5. Provide quality control procedures for monitoring accommodations, administration, scoring, and reporting. This should include plans and procedures for communicating the quality control procedures to the field and the contractor.
6. Provide specific information regarding vertical and horizontal equating needed to ensure consistent meaning of scores across grades and years.
7. Provide evidence that the accommodations permitted support valid use of the assessment results.
8. Provide evidence that the State has contracted for production of a complete technical manual that will include complete data for each PAWS assessment, including the PAWS-Alt, timeline for delivery, and delivery of the complete manual as quickly as possible.

ALIGNMENT

1. Clear evidence that its assessments are aligned to content standards and that they represent the depth and breadth of the Wyoming Content Standards.
2. An overall rationale that connects the content standards, blueprints, and assessments as well as documentation that Wyoming educators and parents have access to this information.
3. Documentation of a more-detailed test blueprint and item specifications for the field test and operational test, including the rationale for item selection for the Spring 2006 assessments.
4. Evidence that the State will require the contractor conducting the external alignment study to provide specific recommendations regarding the quality of alignment.
5. Documentation describing Wyoming’s ongoing process for addressing deficiencies found by any independent alignment studies.
6. Provide results from the independent alignment study for the PAWS and PAWS-Alt assessments.

INCLUSION

1. Produce participation rates for all student groups based on the number enrolled at the time of testing.
2. Provide information regarding accommodations training.
3. Produce participation guidelines and administration procedures for the PAWS-ALT assessment.

REPORTS

1. Provide sample reports for the PAWS and PAWS-Alt assessments at the student level that present results in terms of the achievement standards, including explanatory materials provided to parents.
2. Provide sample reports for the PAWS and PAWS-Alt assessments at the State, district, and school levels that include all required disaggregation.
3. Conduct sample item analysis and provide the results of these analyses.
4. Produce evidence that the State has notified local educational agencies (LEAs) of the procedures to be used for distributing student reports to parents in a timely manner.

Return to state-by-state listing