Montana Assessment Letter

June 22, 2006

The Honorable Linda McCulloch
Superintendent of Public Instruction
Montana Office of Public Instruction
1227 11th Avenue
Helena, Montana 59620-2501

Dear Superintendent McCulloch:

Thank you for your participation in the U.S. Department of Education’s (Department) standards and assessment peer review process under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). I appreciate the efforts required to prepare for the peer review. As you know, with the implementation of NCLB’s accountability provisions, each school, district, and State is held accountable for making adequate yearly progress (AYP) towards having all students proficient by 2013-14. An assessment system that produces valid and reliable results is fundamental to a State’s accountability system.

I am writing to follow up on the letter that was sent to you on May 18, 2006. In that letter we presented the results of the peer review of the Montana standards and assessment system and detailed the additional evidence necessary for Montana to meet the statutory and regulatory requirements of Section 1111(b)(1) and (3) of the ESEA. The need for the remaining outstanding evidence, as listed in the enclosure to this letter, remains.

As you will recall, the Department laid out new approval categories in the letter to the Chief State School Officers on April 24, 2006. These categories better reflect where States collectively are in the process of meeting the statutory standards and assessment requirements and where each State individually stands. Based on these new categories, the current status of the Montana standards and assessment system is Approval Pending. This status indicates that Montana’s standards and assessment system administered in the 2005-06 school year has three or more fundamental components that are missing or that do not meet the statutory and regulatory requirements, in addition to other outstanding issues that can be addressed more immediately. These deficiencies must be resolved in a timely manner so that the standards and assessment system administered next year meets all requirements. The Department believes that Montana can address the outstanding issues by the next administration of its assessment system, that is, by the end of the 2006-07 school year.

Montana’s system has a number of fundamental components that warrant the designation of Approval Pending. Specifically, the Department cannot approve Montana’s standards and assessment system due to outstanding concerns with the development of general and alternate content and achievement standards, technical quality, including inter-rater agreement and appropriate accommodations, alignment of the State assessments to grade-level content standards, and the alternate assessment based on alternate achievement standards (the alternate Criterion-Referenced Tests, or CRT-Alt). Please refer to the enclosure for a detailed list of the evidence Montana must submit to meet the requirements for an approved standards and assessment system.

Accordingly, Montana is placed under Mandatory Oversight, as authorized under 34 C.F.R. §80.12. Under this status, there will be specific conditions placed on Montana’s fiscal year 2006 Title I, Part A grant award. In addition, Montana must provide, not later than 25 business days from receipt of this letter, a plan and detailed timeline for how it will meet the remaining requirements to come into full compliance by the end of the 2006-07 school year. Beginning in September 2006, Montana must also provide bi-monthly reports on its progress implementing the plan.

Due to the number of outstanding items that Montana needs to complete to come into compliance with NCLB, the Department intends to withhold 10 percent of the State’s fiscal year 2006 Title I, Part A administrative funds, totaling $41,020, pursuant to Section 1111(g)(2) of the ESEA. Montana has the opportunity, within 20 business days of receipt of this letter, to show cause in writing why we should not withhold these funds. If Montana cannot show cause, the Department will withhold 10 percent of Montana’s fiscal year 2006 Title I, Part A administrative funds, which will then revert to local educational agencies in Montana. Moreover, if, at any time, Montana does not meet the timeline set forth in its plan, the Department will initiate proceedings to withhold an additional 10 percent of the State’s fiscal year 2006 Title I, Part A administrative funds.

I know you are anxious to receive full approval of your standards and assessment system and we are committed to helping you get there. Toward that end, let me reiterate my earlier offer of technical assistance. We remain available to assist you however necessary to ensure you administer a fully approved standards and assessment system. We will schedule an additional peer review when you have evidence available to further evaluate your system. If you have any questions or would like to discuss this further, please do not hesitate to contact Patrick Rooney (Patrick.Rooney@ed.gov) or Carlos Martínez (Carlos.Martinez@ed.gov) of my staff.

Sincerely,

Henry L. Johnson

Enclosures

cc: Governor Brian Schweitzer
Nancy Coopersmith

Summary of Additional Evidence that Montana Must Submit to Meet ESEA Requirements for the Montana Comprehensive Assessment System

1.0 – ACADEMIC CONTENT STANDARDS

1. Evidence that it has approved content and skill expectations for each grade, 3-8 and 10.
2. Evidence of how its standards were developed to ensure that they contain rigorous content and encourage the teaching of advanced skills or provide evidence of standards analysis (such as Depth of Knowledge analysis) that address the rigor and challenge of the content expectations in reading and mathematics in grades 3, 5, 6, and 7 and the content expectations in science in grades 4, 8, and 10.
3. The state should provide evidence that the committees of stakeholders involved in the standards development process adequately represented the diversity of the State’s student population.

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

1. Documentation of completed achievement descriptors for grades 3, 5, 6, and 7.
2. Evidence that it has alternate academic achievement standards (achievement level descriptors and cut scores) that can be applied to each tested grade.
3. Evidence of the development of alternate achievement descriptors in science for the alternate assessment.
4. A specific plan and timeline for setting and approving cut scores in reading and mathematics on the general assessments and the alternate assessments in grades 3, 5, 6, and 7.

3.0- FULL ASSESSMENT SYSTEM

1. Evidence requested in section 1.0 (Content Standards) regarding rigor and challenge, and section 5.0 (Alignment) would satisfy this section.

4.0 – TECHNICAL QUALITY

1. Test blueprints with sufficient detail to support the creation of substantively comparable test forms that are also aligned with the content and skill aspects of its standards.
2. 2. A plan for evaluating how the results from the statewide assessments are used to evaluate and improve curriculum and instruction, evaluating the degree to which the Criterion-Referenced Tests (CRT) and the alternate Criterion-Referenced Tests (CRT-Alt) results converge with measures of similar constructs and diverge from measures of different constructs, and for identifying and evaluating unintended consequences associated with its statewide assessment system.
3. Evidence that it has evaluated the inter-rater agreement and consistency for constructed-response items on the CRT and CRT-Alt. This evidence should include a clear description of the scoring processes for constructed-response items on the CRT.
4. A clear articulation of how accommodations must be aligned with instructional approaches a student experiences in the classroom and how the state will monitor the use of accommodations at the time of testing.
5. A clarification of how translated and "Sheltered English" versions of its tests are developed and administered, the eligibility criteria for these test versions, and evidence that these test versions yield results that are comparable to those from other test versions. In addition, the state should submit a plan for evaluating the meaning of scores from translated and "Sheltered English" versions of the tests and for ensuring the comparability of these scores with scores from non-accommodated test conditions.

5.0 – ALIGNMENT

1. A plan for addressing the gaps and weaknesses alignment cited in the 2005 external alignment study.
2. An impartial study that evaluates the quality of alignment among its reading and mathematics standards and assessments in grades 3, 5, 6, and 7, and among its science standards and assessments in grades 4, 8, and 10. These plans should address the degree to which its assessments yield scores that adequately reflect the full range of performance defined in the performance level descriptors.
3. Evidence that the CRT-Alt extended content standards are linked to content standards at each grade level, 3-8 and 10, and will yield scores aligned with grade-specific expectations applicable to each tested grade.

6.0 – INCLUSION

1. Provide enrollment and number tested data for each student group within each grade and content area.
2. Separate lists of allowable accommodations for students with individualized education programs (IEPs), students with Section 504 plans, and English language learners (ELLs) to help ensure that accommodations are aligned with students’ specific, individual, academic needs. These lists must provide a clear distinction among which accommodations are allowable for students with IEPs, students with Section 504 plans, and ELLs.

7.0 – REPORTING

1. Provide participation data for each grade level and content area as addressed in section 6.0.

Return to state-by-state listing