October 30, 2007 – Hawaii Assessment Letter

October 30, 2007

The Honorable Patricia Hamamoto
Superintendent of Education
Hawaii Department of Education
1390 Miller Street, #307
Honolulu, Hawaii 96813

Dear Superintendent Hamamoto:

Thank you for your participation in the U.S. Department of Education’s (Department) standards and assessment peer review process under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). We appreciate the efforts required to prepare for the latest peer review, which occurred in September 2007.

In a letter to you on July 23, 2007, the Department enumerated several fundamental components of the Hawaii standards and assessment system that had not met ESEA requirements. In subsequent conversations, you made clear that Hawaii will develop a new alternate assessment based on alternate academic achievement standards for students with the most significant cognitive disabilities. In the September 2007 peer review, therefore, peers examined evidence pertaining to Hawaii’s general reading/language arts and mathematics assessments for students in grades 3 through 8 and high school, the Hawaii State Assessment (HSA), and Hawaii’s native language assessment, the Hawaii Aligned Portfolio Assessment (HAPA), for students in grades 3 and 4 in the Hawaiian Language Immersion Program.

External peer reviewers and Department staff evaluated Hawaii’s most recent submission and concluded that, while the HSA and HAPA have met most of the ESEA requirements, we continue to have concerns regarding the technical quality, alignment, and inclusion of all students in the HSA and HAPA. Enclosed is a detailed list of the evidence Hawaii must still submit to meet the requirements for a fully approved standards and assessment system. Because the latest peer review did not resolve all outstanding issues, the status of Hawaii’s standards and assessment system remains Approval Pending. Under this status, the condition on Hawaii’s fiscal year 2007 Title I, Part A grant award will continue.

Given Hawaii’s current plans to develop a new alternate assessment based on alternate academic achievement standards and the time it will take Hawaii to develop this new assessment, Hawaii must enter into a Compliance Agreement with the Department, as authorized by Section 457 of the General Education Provisions Act. The purpose of the agreement is to enable a grantee to remain eligible to receive funding while coming into full compliance with applicable requirements as soon as feasible but within three years. The Department and Hawaii will need to agree on the components of the agreement, including a detailed plan and specific timeline for how Hawaii will accomplish the steps necessary to bring its system into compliance. For example, Hawaii may need to contract with outside experts or technical assistance providers knowledgeable in the areas of non-compliance. Before entering into the agreement, the Department must hold a hearing to explore why full compliance with the Title I standards and assessment requirements is not feasible until a future date. The State, affected students and their parents, and other interested parties may participate in the hearing. The Department must publish findings of noncompliance and the substance of the agreement in the Federal Register.

I appreciate the draft outline of Hawaii’s plan that Hawaii submitted on September 28, 2007. When finalized, the plan will include all of the steps necessary to develop a fully compliant alternate assessment based on alternate academic achievement standards as well as the remaining components of the HSA and HAPA, as listed in the enclosure to this letter. The action plan must also include a budget for each year the agreement is in place that demonstrates that Hawaii has committed sufficient resources to correct the areas of non-compliance. The budget must reflect that Hawaii will use a reasonable portion of its Title I, Part A administrative funds, in addition to State funds and funds it receives under section 6111 of the ESEA, toward improving its assessment system. The Title I, Part A administrative funds that are used must supplement, not supplant, the State funds dedicated for this purpose.

As announced in my September 15, 2007, letter to all Chief State School Officers, we have developed a Web-based tracking tool that will allow States to track their progress toward having a fully approved standards and assessment system. The Department is piloting this project with any State entering into a compliance agreement to help the State track the steps necessary for the completion of the signed agreement. Because the tool is password-protected to ensure privacy, my staff will follow up with you regarding the details of Hawaii’s involvement in this pilot project.

I appreciate the steps Hawaii has taken toward meeting the requirements of the ESEA, and I know you are anxious to receive full approval of your standards and assessment system. I have asked my staff to periodically visit with you and your staff throughout the term of the compliance agreement to discuss the progress that is being made to bring your standards and assessment system into compliance with the ESEA requirements and any other concerns or issues that may arise. If you have any questions or would like to discuss this further, please do not hesitate to contact Valeria Ford (Valeria.Ford@ed.gov) or Patrick Rooney (Patrick.Rooney@ed.gov) of my staff.

Sincerely,

Kerri L. Briggs, Ph.D.

Enclosure

cc: Governor Linda Lingle
Cara Tanimura
Kent Hinton
Sarah Gronna

SUMMARY OF ADDITIONAL EVIDENCE THAT HAWAII MUST SUBMIT TO MEET ESEA REQUIREMENTS FOR THE HAWAII STANDARDS AND ASSESSMENT SYSTEM

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

Hawaii State Assessment (HSA):

1. The subject area representation for panels that reviewed the performance level descriptors.
2. A plan and process for future selection of panels to ensure representation of all relevant stakeholder groups.

Hawaii State Alternate Assessment (HSAA):

1. Evidence of approved/adopted alternate academic achievement standards for students with the most significant cognitive disabilities in reading/language arts and mathematics for each of grades 3 through 8 and high school.
2. Evidence that the alternate academic achievement standards include, for each content area:
   1. At least three levels of achievement, including two levels of high achievement (e.g., proficient and advanced) that determine how well students are mastering a State’s academic content standards and a third level of achievement (e.g., basic) to provide information about the progress of lower-achieving students toward mastering the proficient and advanced levels of achievement;
   2. Descriptions of the competencies associated with each achievement level; and
   3. Assessment scores (“cut scores”) that differentiate among the achievement levels.
3. Documentation that the State has reported separately the number and percentage of students with disabilities assessed using an alternate assessment based on alternate academic achievement standards, an alternate assessment based on grade-level academic achievement standards, and the general HSA assessment with and without accommodations.
4. Evidence that the State has documented the involvement of diverse stakeholders in the development of its alternate academic achievement standards.

4.0 – TECHNICAL QUALITY

HSA:

1. The report of the consequential validity study.
2. Documentation on how Hawaii will address inter-rater agreement rates for the short-answer and extended-response items that were less than the established target.
3. Procedures for the standardization of the accommodation that allows for the explanation of directions using simplified vocabulary.
4. Results of a study conducted to show that the simplified language procedure does not compromise the validity of the HSA score.
5. Evidence that the “read-aloud” accommodation is allowed only for those students who receive that accommodation in instruction and that the score reports indicate the assessment of an altered construct.
6. Results of a study conducted to show that the “read-aloud” accommodation does not compromise the validity of the HSA reading score.

HSAA:

1. Evidence that the State has documented validity (in addition to the alignment of the HSAA with the academic content standards) as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999).
2. Evidence that the State has provided documentation of the standards-setting process, including a description the selection of judges, methodology employed, and final results.
3. Evidence that the State has considered the issue of reliability, as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999).
4. Evidence that the State has ensured that its alternate assessment system is fair and accessible to eligible students, including students with limited English proficiency.
5. Evidence that the State has taken steps, such as bias review of items, to ensure fairness in the development of the alternate assessment.
6. When different test forms or formats are used for the alternate assessment, evidence that the State has ensured that the meaning and interpretation of results are consistent.
7. Evidence that the State has established:
   1. Clear criteria for the administration, scoring, analysis, and reporting components of its alternate assessment; and
   2. A system for monitoring and improving the on-going quality of its alternate assessment.

Hawaii Aligned Portfolio Assessment (HAPA):

1. Evidence that the cut scores for 2007 operationalize the content-specific performance level descriptors that are used for both the HAPA and HSA. This evidence should include documentation related to the process of reviewing the 2006 cut scores for use in 2007, including a description of the preparation of the panel(s) for their task, a demographic description of the participants, and results of their deliberations.
2. Evidence that Hawaii’s scoring for oral fluency is valid and consistent in the same way across scorers.
3. Evidence showing that all students are being included in the assessment program.
4. Documentation of the comparability of the oral fluency tasks for the HAPA and the items from the HSA.

5.0 – ALIGNMENT

HSA:

1. The blueprints for reading and mathematics for each required tested grade with the number of items developed for each of the standards.
2. Documentation to show that the full range of knowledge is assessed for each required tested grade for reading and mathematics.
3. Results of the item review panel including number of items reviewed, rejected, and revised, as well as alignment to benchmark and depth of knowledge levels for each required tested grade for reading and mathematics.
4. Documentation of how Hawaii addressed each of the cells in the alignment studies that resulted in ratings of “weak” or “no” alignment including the number and extent of changes made.

HSAA:

1. Evidence that the State has taken steps to ensure alignment between its alternate assessments and the State’s academic content and alternate achievement standards.
2. Evidence that the State has developed on-going procedures to maintain and improve alignment between the alternate assessment(s) and academic content and alternate academic achievement standards over time, particularly if gaps have been noted.

HAPA:

1. Evidence demonstrating how Hawaii has addressed or will address the concerns from the Rob Ely Report.
2. Documentation specifying how Hawaii addressed or will address the comments related to Source of Challenge identified by reviewers participating in the alignment studies.

6.0 – INCLUSION

HSA:

1. Statewide spring 2007 participation rates for reading and mathematics in the required grades. Totals should be disaggregated by gender, ethnicity, special education status, limited English proficiency status, economically disadvantaged status, and migrant status.

HSAA:

1. Evidence that the State has implemented alternate assessments for students whose disabilities do not permit them to participate in the regular assessment even with accommodations.
2. Evidence of guidelines and training that the State has in place to ensure that all students with disabilities taking the alternate assessment are included appropriately in the State assessment system.
3. Evidence that the State has developed clear guidelines for Individualized Education Program (IEP) teams to apply in determining which assessment is most appropriate for a student.
4. Regarding the alternate academic achievement standards:
   1. a. Evidence that the State has developed clear guidelines for IEP teams to apply in determining when a child’s cognitive disability justifies assessment based on alternate academic achievement standards; and
   2. Evidence of the steps the State has taken to help regular and special education teachers and other appropriate staff know how to administer the alternate assessment(s), including making use of accommodations, for students with the most significant cognitive disabilities.

7.0 – REPORTING

HSA:

1. Samples of the school-level achievement reports disaggregated by required subgroups by grade level and subject area for 2007.
2. Statewide achievement results disaggregated by the required subgroups by grade level and subject areas for 2007.

HSAA:

1. Evidence that the State’s reporting system facilitates appropriate, credible, and defensible interpretation and use of its alternate assessment data.
2. Evidence that the State has provided for the production of individual interpretive, descriptive, and (non-clinical) diagnostic reports that indicate relative strengths and instructional needs, including:
   1. Evidence that these individual student reports express results in terms of the State’s academic achievement standards rather than numerical values such as scale scores or percentiles.
   2. Evidence that these individual student reports provide information for parents, teachers, and principals to help them understand and address a student’s specific academic needs. This information must be displayed in a format and language that is understandable to parents, teachers, and principals, for example, through the use of descriptors that describe what students know and can do at different performance levels. The reports must be accompanied by interpretive guidance for these audiences; and
   3. Evidence that the State ensures that these individual student reports will be delivered to parents, teachers, and principals as soon as possible after the assessment is administered.

HAPA:

1. Complete set of reports for individual students, schools, and districts.

Return to state-by-state listing