July 13, 2007

The Honorable Jim McBride
Superintendent
Wyoming Department of Education
Hathaway Building
2300 Capitol Avenue
Cheyenne, Wyoming 82002-0050

Dear Superintendent McBride:

I am writing regarding our review of Wyoming’s standards and assessment system under the Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). We appreciate the efforts required to prepare for the peer review and hope that the process provides useful feedback that will support Wyoming’s efforts to monitor student progress toward challenging standards.

External peer reviewers and Department staff evaluated Wyoming’s fourth peer review submission and found, based on the evidence received, that it still does not meet all the statutory and regulatory requirements of section 1111(b)(1) and (3) of the ESEA. I know that my staff has discussed the results of this review with your staff and I appreciate you taking the time to meet with me during my visit to Cheyenne on June 28. I want to take this opportunity to enumerate the evidence that Wyoming must provide in order to have a fully compliant system. Specifically, we have concerns with the technical quality and alignment of the Proficiency Assessments for Wyoming Students (PAWS) to Wyoming’s grade-level academic content standards and academic achievement standards as well as concerns with the technical quality and alignment of Wyoming’s alternate assessment based on alternate academic achievement standards (the Alternate Proficiency Assessments for Wyoming Students, or PAWS-Alt) to grade-level content standards. We appreciate the effort your staff has made to conclude the peer review process but we are concerned about the lack of comprehensive and coherent approach to documenting validity of the assessment system. The complete list of evidence needed to address these concerns is enclosed with this letter.

I know that Wyoming submitted a draft timeline for how and when Wyoming will satisfy the remaining requirements for the PAWS and additional evidence for peer review on July 3 and that additional evidence will be submitted in the coming month. Sue Rigney of my staff will review and respond under separate cover to Wyoming’s draft timeline. Please note that all evidence regarding the PAWS must be submitted no later than August 27, 2007 so that it may be peer reviewed during the Department’s review from September 17-21. Therefore, we are not assigning an approval status to Wyoming’s system at this time. Because that system is not fully approved, a condition was placed on your fiscal year 2007 Title I, Part A grant award.

If Wyoming is unable to resolve the remaining issues with the PAWS, we will take appropriate enforcement actions as outlined in the Department’s May 10, 2007, fact sheet, including the possibility of a Compliance Agreement under Section 457 of the General Education Provisions Act. For your convenience, I am enclosing a copy of that fact sheet, which is also available on the Department’s website (http://www.ed.govhttps://www2.ed.gov/admins/lead/account/statesystems.html).

We look forward to working with Wyoming to support a high-quality standards and assessment system. If you would like to discuss this further, please do not hesitate to call Sue Rigney (202-260-0931) or Patrick Rooney (202-205-8831) of my staff.

Sincerely,

Kerri L. Briggs, Ph.D.

Enclosures

cc: Governor Dave Freudenthal
Joe Simpson
Lesley Wangberg

SUMMARY OF ADDITIONAL EVIDENCE THAT WYOMING MUST SUBMIT TO MEET ESEA REQUIREMENTS FOR THE WYOMING ASSESSMENT SYSTEM

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

1. Confirmation that the PAWS-Alt is administered as intended, e.g. number and percent of non-scoreable components.
2. The scoring rubrics for all three PAWS-Alt components: the Data Collection Form (DCF), the Teacher Observation of Academic Skills (TOAS), and the Student Performance Events (SPE). The DCF rubric was provided but the scoring criteria for TOAS and SPE components were not.
3. Training materials for the accurate administration, scoring, and interpretation of results from the PAWS-Alt.
4. A description of the standards-setting process for the PAWS-Alt that includes documentation of the procedures employed, including:
   1. The standards-setting model;
   2. The materials used to train the panelists;
   3. The rationale and procedures for selecting student work samples;
   4. A sample student profile as used by the panelists; and
   5. The numerical information used to summarize the contents of the student profiles used for standard setting.
   6. An explanation of how variation in student profiles was considered in setting performance standards and drafting the final performance descriptors.

4.0 – TECHNICAL QUALITY

1. Evidence of technical quality for PAWS to address the reviewers’ concerns regarding:
   1. Procedures for the development of test forms appropriate for the design and intended purpose (reading, writing, and mathematics);
   2. Stability of results from multiple forms within year (split-half option) and across time (reading, writing, and mathematics);
   3. Consistency between test administration practices and the standards-setting process (reading, writing, and mathematics);
   4. Clear blueprints, item specifications, and test development procedures for 2006-07 (reading and writing);
   5. Implementation of a writing scale that combines with reading scale scores to form a combined English/language arts score for accountability; and
   6. Implementation of sound scaling and equating procedures across test forms and years.
2. Implementation of sound scaling and equating procedures across test forms and years.
3. Evidence of technical quality for PAWS-Alt to address the reviewers’ concerns regarding:
   1. Clear explanations of design and scoring;
   2. Documentation of scoring reliability (consistency and accuracy);
   3. Justification of score use given the threat to validity represented by the inter-component correlations that show higher correlations between content areas than between components within content areas;
   4. The low percentage of items related to assessment targets; and
   5. Data discrepancies within and between successive versions of the Technical Reports.
   6. Complete validity information for the operational PAWS and reliability and validity information for the PAWS-Alt assessments, including data supporting concurrent and consequential validity.
   7. Data confirming that the accommodations permitted on the PAWS assessment support valid use of the assessment results.
   8. A plan and commitment for ongoing analysis of the relation between test administration patterns (split-half option), test results and the implications for interpretation, and intended consequences.

5.0 – ALIGNMENT

1. The systematic procedures that will be used to address the alignment gaps documented in the ELY alignment study and the concerns about alignment stated in the technical manual.
2. Test blueprints or other documents that show alignment of the PAWS with the content standards rather than “skills.”
3. Clarification regarding the design and administration of the operational PAWS-Alt sufficient to determine whether the assessment content is aligned with grade-level content standards.

6.0 – INCLUSION

1. Official grade-by grade enrollment data (either for spring 2006 or for the date of enrollment that is used by Wyoming to calculate the participation rate for adequate yearly progress determinations) for all students. This should include official State assessment reports for the assessments administered in 2006-07 that display the number of students with disabilities tested on PAWS and PAWS-Alt.

Return to state-by-state listing

April 26, 2007

The Honorable Jim McBride
Superintendent
Wyoming Department of Education
Hathaway Building
2300 Capitol Avenue
Cheyenne, Wyoming 82002-0050

Dear Superintendent McBride:

Thank you for your participation in the U.S. Department of Education’s (Department) standards and assessment peer review process under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB).

In the letter to you on June 22, 2006, the Department enumerated the fundamental components as well as a number of additional technical issues that had not met the standards and assessment requirements of the ESEA. Specifically, the Department could not approve Wyoming’s standards and assessment system due to outstanding concerns with the alternate assessment based on alternate academic achievement standards (the Alternate Proficiency Assessments for Wyoming Students, or PAWS-Alt) and the process for standards setting for the new assessments based upon the field trial and operational assessments.

In January 2007, the peer reviewers and Department staff evaluated additional evidence submitted by Wyoming. Based on the evidence received, we have determined that Wyoming’s system still does not meet all of the statutory and regulatory requirements of Section 1111(b)(1) and (3) of the ESEA. I appreciate you taking the time last month to meet with Catherine Freeman and Sue Rigney to discuss the results of that peer review and I hope it helped you in preparing for the next peer review. I know that Wyoming submitted additional evidence that will be reviewed by the peers next week; I encourage you to bring in all available evidence for review.

Because the peer review did not resolve all outstanding issues, the status of Wyoming’s standards and assessment system remains Approval Pending. The enclosure provides a detailed list of the evidence Wyoming must still submit to meet the requirements for a fully approved standards and assessment system. Please note that Wyoming must address all outstanding issues in order to have a fully compliant standards and assessment system under the ESEA by the end of the 2006-07 school year. Wyoming continues to be under Mandatory Oversight, as authorized under 34 C.F.R. §80.12, and the condition on your Title I, Part A grant award will continue.

I appreciate the steps Wyoming has taken toward meeting the requirements of the ESEA, and I know you are anxious to receive full approval of your standards and assessment system. We are committed to helping you get there and remain available to provide technical assistance regarding issues that you identify. Toward that end, let me reiterate my earlier offer of technical assistance. We will schedule an additional peer review when you have evidence available to further evaluate your system. If you have any questions or would like to discuss this further, please do not hesitate to contact Sue Rigney (Sue.Rigney@ed.gov) or Patrick Rooney (Patrick.Rooney@ed.gov) of my staff.

Sincerely,

Kerri L. Briggs, Ph.D.
Acting Assistant Secretary

Enclosure

cc: Governor Dave Freudenthal
Joe Simpson
Lesley Wangberg

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

1. Clarification of the design elements of the Alternate Proficiency Assessments for Wyoming Students (PAWS-Alt) and evidence that it generates appropriate and accurate information about student performance.
2. Directions for teachers to administer the assessment and confirmation that the directions are implemented as intended.
3. The scoring rubrics for all PAWS-Alt components, as well as the training provided to ensure reliable scoring.
4. A description of the standards setting process for the PAWS-Alt that includes the procedures employed, the qualifications of the panelists, and an explanation of how data from the three main components is integrated into a performance standard.

4.0 – TECHNICAL QUALITY

1. A completed technical manual that includes detailed information regarding the Proficiency Assessments for Wyoming Students (PAWS) item and test development; procedures for combining results based on multiple choice and constructed response items; procedures for the combination of reading and writing results to produce a language arts proficiency score; standards setting, scaling, scoring, and equating procedures; and validity and reliability information.

5.0 – ALIGNMENT

1. The systematic procedures that will be used to address the alignment gaps documented in the ELY alignment study.
2. Test blueprints or other documents that show alignment of the PAWS with the content standards (as opposed to alignment with “skills”).
3. Clarification regarding the design and administration of the operational PAWS-Alt sufficient to determine whether the assessment content is aligned with grade-level content standards.

6.0 – INCLUSION

1. Participation data for PAWS and PAWS-Alt showing that all students enrolled are included in the State assessments. This is not the same as AYP participation rates and AYP reports will not meet this requirement.

7.0 – REPORTS

1. Sample student reports for the PAWS and PAWS-Alt with the explanatory materials provided to parents and educators.
2. Sample item analysis consistent with the description in the Department’s Standards and Assessment Guidance, dated March 2003.

Return to state-by-state listing

June 22, 2006

The Honorable Jim McBride
State Superintendent of Public Instruction
Wyoming Department of Education
2300 Capitol Ave, 2nd Floor
Hathaway Building
Cheyenne, Wyoming 82002-0050

Dear Superintendent McBride:

Thank you for your participation in the U.S. Department of Education’s (Department) standards and assessment peer review process under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). I appreciate the efforts required to prepare for the peer review. As you know, with the implementation of NCLB’s accountability provisions, each school, district, and State is held accountable for making adequate yearly progress (AYP) towards having all students proficient by 2013-14. An assessment system that produces valid and reliable results is fundamental to a State’s accountability system.

I am writing to follow up on the letter that was sent to you on May 10, 2006. In that letter we presented the results of the peer review of the Wyoming standards and assessment system and detailed the additional evidence necessary for Wyoming to meet the statutory and regulatory requirements of Section 1111(b)(1) and (3) of the ESEA. A second peer review in May 2006 was not sufficient to resolve all of the issues. The need for the remaining outstanding evidence, as listed in the enclosure to this letter, remains.

As you will recall, the Department laid out new approval categories in the letter to the Chief State School Officers on April 24, 2006. These categories better reflect where States collectively are in the process of meeting the statutory standards and assessment requirements and where each State individually stands. Based on these new categories, the current status of the Wyoming standards and assessment system is Approval Pending. This status indicates that Wyoming’s standards and assessment system administered in the 2005-06 school year has two or more fundamental components that are missing or that do not meet the statutory and regulatory requirements, in addition to other outstanding issues that can be addressed more immediately. These deficiencies must be resolved in a timely manner so that the standards and assessment system administered next year meets all requirements. The Department believes that Wyoming can address the outstanding issues by the next administration of its assessment system, that is, by the end of the 2006-07 school year.

Wyoming’s system has a number of fundamental components that warrant the designation of Approval Pending. Specifically, the Department cannot approve Wyoming’s standards and assessment system due to outstanding concerns with the alternate assessment based on alternate achievement standards (the Alternate Proficiency Assessments for Wyoming Students, or PAWS-Alt) and the process for standard setting for the new assessments based upon the field trial and operational assessments. Please refer to the enclosure for a detailed list of the evidence Wyoming must submit to meet the requirements for an approved standards and assessment system.

Accordingly, Wyoming is placed under Mandatory Oversight, pursuant to 34 C.F.R. §80.12. Under this status, there will be specific conditions placed on Wyoming’s fiscal year 2006 Title I, Part A grant award. Wyoming must provide, not later than 25 business days from receipt of this letter, a plan and detailed timeline for how it will meet the remaining requirements to come into full compliance by the end of the 2006-07 school year. Beginning in September 2006, Wyoming must also provide bi-monthly reports on its progress implementing the plan. If, at any time, Wyoming does not meet the timeline set forth in its plan, the Department will initiate proceedings, pursuant to Section 1111(g)(2) of the ESEA, to withhold 15 percent of Wyoming’s fiscal year 2006 Title I, Part A administrative funds, which will then revert to local educational agencies in Wyoming.

I know you are anxious to receive full approval of your standards and assessment system and we are committed to helping you get there. Toward that end, let me reiterate my earlier offer of technical assistance. We remain available to assist you however necessary to ensure you administer a fully approved standards and assessment system. We will schedule an additional peer review when you have evidence available to further evaluate your system. If you have any questions or would like to discuss this further, please do not hesitate to call Patrick Rooney (Patrick.Rooney@ed.gov) or Sue Rigney (Sue.Rigney@ed.gov) of my staff.

Sincerely,

Henry L. Johnson

Enclosure

cc: Governor Dave Freudenthal
Annette Bohling

Summary of Additional Evidence that Wyoming Must Submit to Meet ESEA Requirements for the Wyoming Assessment System

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

1. Final comprehensive reports from the Proficiency Assessments for Wyoming Students (PAWS) and the PAWS-Alt standard setting, including the qualifications of the panelists indicating experience with students with disabilities and students with limited English proficiency.
2. Evidence of State Board adoption of the achievement standards for the PAWS and the PAWS-Alt assessments.
3. A plan and timeline for the completion and adoption of the final academic achievement descriptors and cut scores for science assessments.

4.0 – TECHNICAL QUALITY

1. A completed technical manual that includes detailed information regarding item and test development, procedures for combining results based on multiple choice and constructed response items as well as the combination of reading and writing results to produce a language arts proficiency score, standard setting, scaling, scoring and equating procedures and validity and reliability information.
2. Reliability and validity information for the operational PAWS and PAWS-Alt assessments, including data supporting concurrent and consequential validity.
3. Data confirming that the accommodations permitted on the PAWS assessment support valid use of the assessment results.
4. Comparability evidence showing that tests administered both on-line and by paper and pencil are comparable in order to support standards set on the basis of paper/pencil administration (field trial) but applied to the online administration (operational).
5. Specific information regarding the processes for reconciling results from the 2006 Blue Ribbon Panel and the summer 2005 standard-setting participants.

5.0 – ALIGNMENT

1. The systematic procedures that will be used to address the alignment gaps documented in the Ely alignment study.
2. Final test construction blueprints for the PAWS assessment as administered in 2005-06 and future years.
3. Documentation showing that the operational PAWS-Alt assessment is aligned with grade-level content standards.

6.0 – INCLUSION

1. Participation data for the PAWS and PAWS-Alt assessments showing that all students enrolled are included in the State assessments. Please note that this is not the same as AYP participation rates; AYP reports will not meet this requirement.
2. Additional documentation for the PAWS-Alt assessment, including guidelines for eligibility for the PAWS-Alt assessment, evidence that the guidelines were disseminated to the field in a timely manner, and a complete description of the test administered in 2005-06 with an explanation of how pilot results were or will be incorporated in the final operational test.

7.0 – REPORTS

1. Sample student reports for the PAWS and PAWS-Alt with the explanatory materials provided to parents and educators.
2. Sample school, district, and State reports that include data for the migrant population.
3. Sample item analysis.
4. Evidence that the State has notified LEAs of the procedures to be used for distributing student reports to parents in a timely manner.

Return to state-by-state listing

May 10, 2006

The Honorable Jim McBride
Superintendent
Wyoming Department of Education
Hathaway Building
2300 Capitol Avenue
Cheyenne, WY 82002-0050

Dear Superintendent McBride:

Thank you for submitting Wyoming’s assessment materials for review under the standards and assessment requirements of the Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). We appreciate the efforts required to prepare for the peer review and hope that the process provides useful feedback that will support Wyoming’s efforts to monitor student progress toward challenging standards.

External peer reviewers and U.S. Department of Education (ED) staff evaluated Wyoming’s submission and found, based on the evidence received, that it did not meet all the statutory and regulatory requirements of Section 1111(b)(3) of the ESEA. I know that my staff has discussed the results of this review with your staff. However, I want to take this opportunity to enumerate the evidence that Wyoming must provide in order to have a fully compliant standards and assessment system under NCLB. That evidence is listed on the last pages of this letter.

For this initial review, very few details were provided regarding the technical quality, alignment, reliability, and validity of the Wyoming assessment system, making it almost impossible to determine whether the system meets professionally accepted testing standards and NCLB requirements. I understand you have submitted additional evidence for a second round of peer review that is currently underway. Following the review, you will receive a letter detailing any additional evidence needed, if any, to demonstrate that Wyoming’s system meets the standards and assessment requirements.

Enclosed with this letter are detailed comments from the peer review team that evaluated Wyoming’s assessment materials in February. The peer reviewers, experts in the areas of standards and assessment, review and discuss a State’s submission of evidence and prepare a consensus report. I hope you will find the reviewers’ comments and suggestions helpful. I also want to remind you of our offer to provide you further technical assistance at your request.

We look forward to working with Wyoming to support a high-quality assessment system. If you would like to discuss this further or request technical assistance, please do not hesitate to call Sue Rigney (202-260-0931) or Patrick Rooney (202-205-8831) of my staff.

Sincerely,

Henry L. Johnson

Enclosure

cc: Governor Dave Freudenthal
Annette Bohling

Summary of Additional Evidence that Wyoming Must Submit to Meet ESEA Requirements for the Wyoming Assessment System

1.0 – CONTENT STANDARDS

1. Provide results from an external evaluation that documents that the language arts, mathematics, and science content standards are challenging and contain rigorous and coherent content (this might be included as part of an overall alignment study).
2. Provide documentation that diverse stakeholders were involved in development of the content standards. Specifically, provide evidence of the consideration of the applicability to all students, including students with disabilities and limited English proficient students.

ACADEMIC ACHIEVEMENT STANDARDS

1. Provide documentation of the standard setting process employed, qualifications of the panelists, and final recommendations for the Proficiency Assessments for Wyoming Students (PAWS) and the PAWS-Alt assessments.
2. Provide evidence of adoption of the achievement standards for the PAWS and the PAWS-Alt assessments by the Board.
3. Provide a plan and timeline for completion and adoption of the final academic achievement descriptors and cut scores for the science assessment.
4. Produce evidence that the standard-setting panelists reflect knowledge of students with disabilities and limited English proficient students.

FULL ASSESSMENT SYSTEM

1. Provide evidence that the operational PAWS and PAWS-Alt assessments have been administered to all students in grades 3-8 and 11.
2. Provide documentation regarding the determination of student scores.

TECHNICAL QUALITY

1. Provide, at a minimum, basic validity and reliability information (i.e., consequential validity, item interrelationships and structural consistency, criterion validity and standards setting procedures) beginning with the 2004-05 pilot.
2. Due to the nature of recent and planned changes to the PAWS assessment program, provide technical manuals for 2004-05, 2005-06, and 2006-07 assessments. This practice should continue annually until the assessment system stabilizes.
3. Provide complete reliability analyses for the 2005-06 PAWS and PAWS-Alt assessments (note: the 2004-05 Wyoming Comprehensive Assessment System (WyCAS) technical manual chapter on reliability was comprehensive, consistent with the peer review guidelines, and would serve as a good model for such evidence).
4. Provide detailed standards-setting plans and procedures. Provide a rationale for the adequacy of the proposed 2-phase process in determining AYP for the PAWS and PAWS-Alt, including:
   1. Detailed information on the materials and process used, panelist instructions, and results of the panel’s recommendations;
   2. Specific information regarding the processes for reconciling results from the 2006 Blue Ribbon Panel and the Summer 2005 standards-setting participants; and
   3. Detailed description of Wyoming’s scaling and equating design.
5. Provide quality control procedures for monitoring accommodations, administration, scoring, and reporting. This should include plans and procedures for communicating the quality control procedures to the field and the contractor.
6. Provide specific information regarding vertical and horizontal equating needed to ensure consistent meaning of scores across grades and years.
7. Provide evidence that the accommodations permitted support valid use of the assessment results.
8. Provide evidence that the State has contracted for production of a complete technical manual that will include complete data for each PAWS assessment, including the PAWS-Alt, timeline for delivery, and delivery of the complete manual as quickly as possible.

ALIGNMENT

1. Clear evidence that its assessments are aligned to content standards and that they represent the depth and breadth of the Wyoming Content Standards.
2. An overall rationale that connects the content standards, blueprints, and assessments as well as documentation that Wyoming educators and parents have access to this information.
3. Documentation of a more-detailed test blueprint and item specifications for the field test and operational test, including the rationale for item selection for the Spring 2006 assessments.
4. Evidence that the State will require the contractor conducting the external alignment study to provide specific recommendations regarding the quality of alignment.
5. Documentation describing Wyoming’s ongoing process for addressing deficiencies found by any independent alignment studies.
6. Provide results from the independent alignment study for the PAWS and PAWS-Alt assessments.

INCLUSION

1. Produce participation rates for all student groups based on the number enrolled at the time of testing.
2. Provide information regarding accommodations training.
3. Produce participation guidelines and administration procedures for the PAWS-ALT assessment.

REPORTS

1. Provide sample reports for the PAWS and PAWS-Alt assessments at the student level that present results in terms of the achievement standards, including explanatory materials provided to parents.
2. Provide sample reports for the PAWS and PAWS-Alt assessments at the State, district, and school levels that include all required disaggregation.
3. Conduct sample item analysis and provide the results of these analyses.
4. Produce evidence that the State has notified local educational agencies (LEAs) of the procedures to be used for distributing student reports to parents in a timely manner.

Return to state-by-state listing