Idaho Assessment Letter 3
December 9, 2005
Honorable Dwight Johnson
Interim Executive Director
Idaho State Board of Education
PO Box 83720
Boise, Idaho 83720-0037
Honorable Marilyn Howard
Superintendent of Public Instruction
Idaho Department of Education
650 West State Street
P.O. Box 83720
Boise, Idaho 83720-0027
Dear Mr. Johnson and Dr. Howard:
I am writing to provide my decision regarding the peer review of Idaho’s standards and assessment system under the Improving America’s Schools Act of 1994 (IASA). As you know, section 1111(b) of Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the IASA, required each State that received Title I, Part A funds to develop and implement challenging State content and performance standards that included the knowledge and skills it expected of all children and an aligned Statewide assessment system administered at some time during grades 3-5, 6-9, and 10-12. Section 1111(b) required a State to complete development of its standards and assessments by the end of the 2000-2001 school year. Idaho was unable to meet this deadline and entered into a compliance agreement with the Department on March 29, 2002.
Since that time, the Idaho State Board of Education (ISBE) has taken several steps to strengthen various aspects of its education system, all aimed at increasing student achievement. Idaho is one of thirteen states who, in coordination with Achieve, has committed to significantly raising the rigor of its high school standards, assessments and curriculum and better aligning these expectations with the demands of postsecondary education and the workforce (Achieve, 2005). The State has aligned teacher certification standards to student content standards. It has also shown a commitment to data-based decision making as is evidenced by the Idaho Basic Education Data System (IBEDS), a robust and adaptable data system that allows districts to reference a teacher’s highly qualified status.
In September 2005, Idaho was peer reviewed for compliance with IASA. In preparation for the peer review, ISBE aggressively studied the alignment of the Idaho Standards Achievement Test, the State content standards, and the State’s alternate assessment, providing the State with valuable feedback on how to further refine its standards and assessment system. The peer review also showed that once ISBE submits evidence of its formal adoption/approval of the State’s alternate academic achievement standards, Idaho will be among the first States to have constructed alternate assessments based on alternate achievement standards acceptable under current statute and regulations.
However, for the reasons discussed below, I am unable to approve Idaho’s system as meeting the IASA requirements. Under the terms of the compliance agreement, Idaho agreed, among other things, to develop and implement performance standards that are aligned with Idaho’s content standards. Idaho also agreed to develop an assessment system that fully meets the requirements in section 1111(b)(3) of the ESEA, as contained in the IASA. Specifically, Idaho agreed to develop a system that is aligned with its content and performance standards; is cognitively complex, with multiple measures that improve the validity, reliability, and fairness of the assessments; and is of high technical quality. Idaho had until Spring 2005 to develop and implement its system of standards and assessments.
Despite the extra time provided by the compliance agreement, Idaho has not satisfied the fundamental IASA requirements for a standards-based assessment system in several areas.
Performance standards: Idaho calculated performance standards in grades 2-10. However, the method it used to define those performance standards did not result in performance standards aligned with Idaho’s content standards. Specifically, Idaho set performance standards at 10th grade and applied a mathematical formula to then set the standards in grades 2 through 9. That is, Idaho set the 9th grade performance standards by subtracting 0.9 units from 10th grade cut scores and repeated this process at each of the remaining grade levels (2-8). This process is flawed for several reasons. First, Idaho’s process used to set the 10th grade standards, at least as documented in the evidence provided, is not consistent with currently accepted methods of setting performance standards. Second, using a mathematical model to set standards at grades 2 though 9 is inconsistent with the IASA because it does not account for mastery of State content standards at specific grades. In other words, the performance standards (as calculated) do not provide information about how well a student learned the content at any particular grade, which is a fundamental requirement of those standards.
Alignment: Idaho has not provided evidence that the components of the Idaho Assessment Program are aligned with Idaho’s content standards. An external alignment study was contracted by the Idaho State Board of Education (ISBE) and completed in April 2005. The ISBE noted in its submission for peer review that test blueprints provided lesser degrees of alignment than were acceptable to the State. ISBE provided some examples that help explain the alignment problems: (1) the Idaho math content standards include a large number of standards in one area, which make it difficult to fully assess the range of knowledge within those standards; and (2) the Idaho Standards Achievement Tests (ISAT) for reading are built on one standard so it is difficult to know whether the test is covering all aspects of reading (e.g., comprehension, vocabulary).
Technical quality: Idaho failed to provide convincing evidence that ISAT forms are comparable from one year to the next year. In order to attribute year-to-year changes in scores to either improvements or declines in student achievement, rather than differences in test difficulty, assessments must be equated from one year to the next. In the absence of adequate equating, it is difficult to interpret the meaning of the results over time.
Because Idaho’s compliance agreement expired on March 29, 2005, there is no additional time to meet IASA requirements and Idaho is out of compliance. If there is additional evidence that the areas of IASA non-compliance outlined above have been addressed, please submit it to my office, and no later than one week from your receipt of this letter. Idaho will need to take immediate action to correct the remaining areas of non-compliance. We look forward to working with Idaho to support a high-quality assessment system. Please do not hesitate to call David J. Harmon (202-205-3554) or Darla Marburger (202-260-2032) of my staff with any further questions.
Sincerely,
Henry L. Johnson
