Wyoming Assessment Letter

July 13, 2007

The Honorable Jim McBride
Superintendent
Wyoming Department of Education
Hathaway Building
2300 Capitol Avenue
Cheyenne, Wyoming 82002-0050

Dear Superintendent McBride:

I am writing regarding our review of Wyoming’s standards and assessment system under the Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). We appreciate the efforts required to prepare for the peer review and hope that the process provides useful feedback that will support Wyoming’s efforts to monitor student progress toward challenging standards.

External peer reviewers and Department staff evaluated Wyoming’s fourth peer review submission and found, based on the evidence received, that it still does not meet all the statutory and regulatory requirements of section 1111(b)(1) and (3) of the ESEA. I know that my staff has discussed the results of this review with your staff and I appreciate you taking the time to meet with me during my visit to Cheyenne on June 28. I want to take this opportunity to enumerate the evidence that Wyoming must provide in order to have a fully compliant system. Specifically, we have concerns with the technical quality and alignment of the Proficiency Assessments for Wyoming Students (PAWS) to Wyoming’s grade-level academic content standards and academic achievement standards as well as concerns with the technical quality and alignment of Wyoming’s alternate assessment based on alternate academic achievement standards (the Alternate Proficiency Assessments for Wyoming Students, or PAWS-Alt) to grade-level content standards. We appreciate the effort your staff has made to conclude the peer review process but we are concerned about the lack of comprehensive and coherent approach to documenting validity of the assessment system. The complete list of evidence needed to address these concerns is enclosed with this letter.

I know that Wyoming submitted a draft timeline for how and when Wyoming will satisfy the remaining requirements for the PAWS and additional evidence for peer review on July 3 and that additional evidence will be submitted in the coming month. Sue Rigney of my staff will review and respond under separate cover to Wyoming’s draft timeline. Please note that all evidence regarding the PAWS must be submitted no later than August 27, 2007 so that it may be peer reviewed during the Department’s review from September 17-21. Therefore, we are not assigning an approval status to Wyoming’s system at this time. Because that system is not fully approved, a condition was placed on your fiscal year 2007 Title I, Part A grant award.

If Wyoming is unable to resolve the remaining issues with the PAWS, we will take appropriate enforcement actions as outlined in the Department’s May 10, 2007, fact sheet, including the possibility of a Compliance Agreement under Section 457 of the General Education Provisions Act. For your convenience, I am enclosing a copy of that fact sheet, which is also available on the Department’s website (http://www.ed.govhttps://www2.ed.gov/admins/lead/account/statesystems.html).

We look forward to working with Wyoming to support a high-quality standards and assessment system. If you would like to discuss this further, please do not hesitate to call Sue Rigney (202-260-0931) or Patrick Rooney (202-205-8831) of my staff.

Sincerely,

Kerri L. Briggs, Ph.D.

Enclosures

cc: Governor Dave Freudenthal
Joe Simpson
Lesley Wangberg

SUMMARY OF ADDITIONAL EVIDENCE THAT WYOMING MUST SUBMIT TO MEET ESEA REQUIREMENTS FOR THE WYOMING ASSESSMENT SYSTEM

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

1. Confirmation that the PAWS-Alt is administered as intended, e.g. number and percent of non-scoreable components.
2. The scoring rubrics for all three PAWS-Alt components: the Data Collection Form (DCF), the Teacher Observation of Academic Skills (TOAS), and the Student Performance Events (SPE). The DCF rubric was provided but the scoring criteria for TOAS and SPE components were not.
3. Training materials for the accurate administration, scoring, and interpretation of results from the PAWS-Alt.
4. A description of the standards-setting process for the PAWS-Alt that includes documentation of the procedures employed, including:
   1. The standards-setting model;
   2. The materials used to train the panelists;
   3. The rationale and procedures for selecting student work samples;
   4. A sample student profile as used by the panelists; and
   5. The numerical information used to summarize the contents of the student profiles used for standard setting.
   6. An explanation of how variation in student profiles was considered in setting performance standards and drafting the final performance descriptors.

4.0 – TECHNICAL QUALITY

1. Evidence of technical quality for PAWS to address the reviewers’ concerns regarding:
   1. Procedures for the development of test forms appropriate for the design and intended purpose (reading, writing, and mathematics);
   2. Stability of results from multiple forms within year (split-half option) and across time (reading, writing, and mathematics);
   3. Consistency between test administration practices and the standards-setting process (reading, writing, and mathematics);
   4. Clear blueprints, item specifications, and test development procedures for 2006-07 (reading and writing);
   5. Implementation of a writing scale that combines with reading scale scores to form a combined English/language arts score for accountability; and
   6. Implementation of sound scaling and equating procedures across test forms and years.
2. Implementation of sound scaling and equating procedures across test forms and years.
3. Evidence of technical quality for PAWS-Alt to address the reviewers’ concerns regarding:
   1. Clear explanations of design and scoring;
   2. Documentation of scoring reliability (consistency and accuracy);
   3. Justification of score use given the threat to validity represented by the inter-component correlations that show higher correlations between content areas than between components within content areas;
   4. The low percentage of items related to assessment targets; and
   5. Data discrepancies within and between successive versions of the Technical Reports.
   6. Complete validity information for the operational PAWS and reliability and validity information for the PAWS-Alt assessments, including data supporting concurrent and consequential validity.
   7. Data confirming that the accommodations permitted on the PAWS assessment support valid use of the assessment results.
   8. A plan and commitment for ongoing analysis of the relation between test administration patterns (split-half option), test results and the implications for interpretation, and intended consequences.

5.0 – ALIGNMENT

1. The systematic procedures that will be used to address the alignment gaps documented in the ELY alignment study and the concerns about alignment stated in the technical manual.
2. Test blueprints or other documents that show alignment of the PAWS with the content standards rather than “skills.”
3. Clarification regarding the design and administration of the operational PAWS-Alt sufficient to determine whether the assessment content is aligned with grade-level content standards.

6.0 – INCLUSION

1. Official grade-by grade enrollment data (either for spring 2006 or for the date of enrollment that is used by Wyoming to calculate the participation rate for adequate yearly progress determinations) for all students. This should include official State assessment reports for the assessments administered in 2006-07 that display the number of students with disabilities tested on PAWS and PAWS-Alt.

Return to state-by-state listing