July 23, 2007 – Hawaii Assessment Letter

July 23, 2007

The Honorable Patricia Hamamoto
Superintendent of Education
Hawaii Department of Education
1390 Miller Street, #307
Honolulu, Hawaii 96813

Dear Superintendent Hamamoto:

I am writing regarding our review of Hawaii’s standards and assessment system under the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). We appreciate the efforts required to prepare for the peer review and hope that the process provides useful feedback that will support Hawaii’s efforts to monitor student progress toward challenging standards.

External peer reviewers and Department staff evaluated additional evidence of the Hawaii standards and assessment system on April 30-May 2, 2007, and found, based on the evidence received, that it still does not meet all the statutory and regulatory requirements of section 1111(b)(1) and (3) of the ESEA. I know that my staff has discussed the results of this review with your staff, most recently when Catherine Freeman and Zollie Stevenson visited Honolulu on July 13. I appreciate your taking the time to meet with them.

I want to take this opportunity to enumerate the evidence that Hawaii must provide in order to have a fully compliant system. Specifically, we have concerns with the technical quality and alignment of the regular assessment, the Hawaii State Assessment (HSA), to Hawaii’s grade-level academic content standards; the technical quality and alignment of the Hawaiian native language assessment (the Hawaii Aligned Portfolio Assessment, or HAPA); and the academic achievement standards, technical quality, and alignment of Hawaii’s alternate assessment based on alternate academic achievement standards (the Hawaii State Alternate Assessment, or HSAA) to grade-level academic content standards. The complete list of evidence needed to address these concerns is enclosed with this letter.

I know that Hawaii is currently completing several studies and technical documents regarding the HSA, which was administered for the first time in spring 2007 and that this additional evidence as well as additional evidence for the HAPA will be submitted in the coming month. Please note that all evidence regarding the HSA and HAPA must be submitted no later than August 27, 2007, so that it may be peer reviewed during the Department’s review from September 17-21. I also know that Hawaii is considering its options regarding the HSAA in order to meet all ESEA requirements. Until we have the results of the peer review and information and a timeline for revising the HSAA, we are not assigning an approval status to Hawaii’s system. When an approval status is assigned, the Department will take appropriate enforcement actions as outlined in the May 10, 2007, fact sheet, including the possibility of a Compliance Agreement under Section 457 of the General Education Provisions Act. For your convenience, I am enclosing a copy of that fact sheet, which is also available on the Department’s website (http://www.ed.govhttps://www2.ed.gov/admins/lead/account/statesystems.html). Because Hawaii’s system is not fully approved, a condition was placed on your fiscal year 2007 Title I, Part A grant award.

We look forward to working with Hawaii to support a high-quality standards and assessment system. If you would like to discuss this further, please do not hesitate to call Valeria Ford (202-205-2213) or Patrick Rooney (202-205-8831) of my staff.

Sincerely,

Kerri L. Briggs, Ph.D.

Enclosures

cc: Governor Linda Lingle
Robert McClelland

SUMMARY OF ADDITIONAL EVIDENCE THAT HAWAII MUST SUBMIT TO MEET ESEA REQUIREMENTS FOR THE HAWAII STANDARDS AND ASSESSMENT SYSTEM

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

Hawaii State Assessment (HSA):

1. Grade-level representation for the reading and mathematics panels that reviewed the performance level descriptors (PLDs).
2. Clarification of the composition of the participants on the Standards-Setting Committee in grades 6-8 and 10. Hawaii must ensure that all grade levels (and student groups) are adequately represented on standards setting panels in the future.
3. Evidence of the adoption of the achievement standards for the HSA by the State Board of Education.
4. Results of the April 23 review of science PLDs for establishing the final approved PLDs.

Hawaii State Alternate Assessment (HSAA):

1. Evidence of approved/adopted alternate academic achievement standards for students with the most significant cognitive disabilities in reading/language arts and mathematics for each of grades 3 through 8 and high school.
2. Evidence that the alternate academic achievement standards include, for each content area:
   1. At least three levels of achievement, including two levels of high achievement (e.g., proficient and advanced) that determine how well students are mastering a State’s academic content standards and a third level of achievement (e.g., basic) to provide information about the progress of lower-achieving students toward mastering the proficient and advanced levels of achievement;
   2. Descriptions of the competencies associated with each achievement level; and
   3. Assessment scores (“cut scores”) that differentiate among the achievement levels.
3. Evidence that the State Board of Education or other authority has adopted alternate academic achievement standards.
4. Documentation that the State has reported separately the number and percentage of students with disabilities assessed using an alternate assessment based on alternate academic achievement standards, an alternate assessment based on grade-level academic achievement standards, and the general HSA assessment with and without accommodations.
5. Evidence that the State has documented the involvement of diverse stakeholders in the development of its alternate academic achievement standards.

Hawaii Aligned Portfolio Assessment (HAPA):

1. Documentation of the standards-setting process employed, qualifications of the panelists, and final recommendations for the achievement standards for the HAPA.
2. Evidence of adoption of the achievement standards for the HAPA by the State Board of Education.

4.0 – TECHNICAL QUALITY

HSA:

1. Specific plans for additional studies concerning reliability and validity when the operational data become available after June 2007, along with dates for completion and prioritization.
2. Complete reliability analyses for the 2006-07 administration of the HSA.
3. Technical manuals for the 2006-07 administration of the HSA.
4. Clarification on the scoring procedures for the short-answer and extended-response items and their impact on reliability.
5. Procedures for the accommodation that allows for the explanation of directions using simplified vocabulary.
6. Justification for the accommodation of “having all reading passages, mathematic problems, and related test questions read aloud to IDEA eligible or Section 504 students who meet the criteria for eligibility.” Provide evidence that this procedure does not impact the validity of the reading scores and instruction.
7. Validity studies and/or descriptive statistics on the accommodations used in all assessments, including analyses comparing accommodated and non-accommodated scores.

HSAA:

1. Evidence that the State has documented validity (in addition to the alignment of the HSAA with the academic content standards) as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999).
2. Evidence that the State has provided documentation of the standards-setting process, including a description of the selection of judges, methodology employed, and final results.
3. Evidence that the State has considered the issue of reliability, as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999).
4. Evidence that the State has ensured that its alternate assessment system is fair and accessible to eligible students, including students with limited English proficiency.
5. Evidence that the State has taken steps, such as bias review of items, to ensure fairness in the development of the alternate assessment.
6. When different test forms or formats are used for the alternate assessment, evidence that the State has ensured that the meaning and interpretation of results are consistent.
7. Evidence that the State has established:
   1. Clear criteria for the administration, scoring, analysis, and reporting components of its alternate assessment; and
   2. A system for monitoring and improving the on-going quality of its alternate assessment.

HAPA:

1. Information regarding basic validity and reliability (i.e., consequential validity, item interrelationships and structural consistency, criterion validity and standards setting procedures) for the HAPA.
2. Technical manuals for the 2005-06 and 2006-07 administrations of the HAPA.
3. Complete reliability analyses for the HAPA.

5.0 – ALIGNMENT

HSA:

1. A clear plan for addressing the alignment gaps in mathematics and reading identified in the alignment study within a reasonable timeline. The plan should propose solutions for both mathematics and reading for each of the cells in the alignment studies resulting in “Weak” or “No” alignment.
2. The rationale, analyses, and processes used to determine proficiency for students at the strand levels used in the family report for reading and mathematics.

HSAA:

1. Evidence that the State has taken steps to ensure alignment between its alternate assessments and the State’s academic content and alternate achievement standards.
2. Evidence that the State has developed on-going procedures to maintain and improve alignment between the alternate assessment(s) and academic content and alternate academic achievement standards over time, particularly if gaps have been noted.

HAPA:

1. Documentation of the alignment of the HAPA to academic content standards and plans to address gaps or weaknesses identified in previous alignment studies.

6.0 – INCLUSION

HSA:

1. Report the spring 2007 participation rates of students in Hawaii’s public schools on each assessment by subject. Totals should be disaggregated by gender, ethnicity, special education student status, limited English proficiency status, economically disadvantaged status, and migrant status.
2. Clarification regarding how “dismissed” students, students who are excused due to parent request, and out-of-state students are included in the State reporting system.

HSAA:

1. Evidence that the State has implemented alternate assessments for students whose disabilities do not permit them to participate in the regular assessment even with accommodations.
2. Evidence of guidelines and training that the State has in place to ensure that all students with disabilities taking the alternate assessment are included appropriately in the State assessment system.
3. Evidence that the State has developed clear guidelines for Individualized Education Program (IEP) teams to apply in determining which assessment is most appropriate for the student.
4. Regarding the alternate academic achievement standards:
   1. a. Evidence that the State has developed clear guidelines for IEP teams to apply in determining when a child’s cognitive disability justifies assessment based on alternate academic achievement standards; and
   2. Evidence of the steps the State has taken to help regular and special education teachers and other appropriate staff know how to administer the alternate assessment(s), including making use of accommodations, for students with the most significant cognitive disabilities.

7.0 – REPORTING

HSA:

1. Complete chart of disaggregated reports, populated with actual results, from the spring 2007 administration of the HSA.

HSAA:

1. Evidence that the State’s reporting system facilitates appropriate, credible, and defensible interpretation and use of the alternate assessment data.
2. Evidence that the State has provided for the production of individual interpretive, descriptive, and (non-clinical) diagnostic reports that indicate relative strengths and instructional needs, including:
   1. Evidence that these individual student reports express results in terms of the State’s academic achievement standards rather than numerical values such as scale scores or percentiles.
   2. Evidence that these individual student reports provide information for parents, teachers, and principals to help them understand and address a student’s specific academic needs. This information must be displayed in a format and language that is understandable to parents, teachers, and principals, for example, through use of descriptors that describe what students know and can do at different performance levels. The reports must be accompanied by interpretive guidance for these audiences; and
   3. Evidence that the State ensures that these individual student reports will be delivered to parents, teachers, and principals as soon as possible after the assessment is administered.

Return to state-by-state listing