Mississippi Assessment Letter dated June 13, 2008

June 13, 2008

The Honorable Hank M. Bounds
Superintendent
Mississippi Department of Education
P. O. Box 771
Jackson, Mississippi 39205-0771

Dear Superintendent Bounds:

I am writing regarding our latest review of the Mississippi standards and assessment system under the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB).

In September 2007 and February 2008, the Department reviewed evidence of the Mississippi assessment system. That evidence consisted of development information and pilot test results for the general assessments, the Mississippi Curriculum Test (MCT2) for grades 3-8 and the Mississippi Subject Area Test Program (MSATP) for high school, and the first full administration of the alternate assessments based on alternate academic achievement standards for students with the most significant cognitive disabilities, the Mississippi Alternate Assessment of Extended Curriculum Frameworks (MAAECF) for grades 3-8 and the High School Alternate Assessment for high school. Based on the evidence we received, we find that the Mississippi assessment system does not meet the statutory and regulatory requirements of sections 1111(b)(1) and 1111(b)(3) of the ESEA. The development work and pilot tests for the MCT2 and MSATP appear to be on the right track, but the Department requires final evidence following the first full administration of these assessments. With respect to the alternate assessments, the MAAECF and High School Alternate Assessment, we have concerns with the technical quality and alignment with grade-level content standards. We are concerned that the structure of these alternate assessments, and the support and training provided to teachers, is not sufficient to ensure alignment with grade-level content.

I have enclosed the list of evidence that Mississippi must provide to demonstrate full compliance with the ESEA. I have broken the list of evidence into two lists, one for the MCT2 and MSATP and one for the MAAECF and High School Alternate Assessment. While the list pertaining to the MCT2 and MSATP is long, this is due to the fact that complete final evidence is still necessary for these assessments now that they have been administered for the first time in spring 2008. Consequently, the list includes almost every aspect of the Department’s Peer Review Guidance. Regarding the MAAECF and the High School Alternate Assessment, it will be helpful for you to keep the first list in mind as you revise the alternate assessments for 2008-09.

Because Mississippi was not able to demonstrate that its full standards and assessment system meets all ESEA requirements, the Mississippi assessment system is designated Approval Pending. Accordingly, Mississippi is placed on Mandatory Oversight, as authorized by 34 C.F.R. § 80.12, and a condition will be placed on the fiscal year 2008 Mississippi Title I, Part A grant award. Mississippi may request reconsideration of its Mandatory Oversight status by submitting in writing to me, within 10 days of receipt of this letter, a detailed discussion setting forth the basis for its belief that this designation is improper, including the specific facts that support its position.

In light of this Mandatory Oversight status and due to the fact the assessment system will not be compliant for the 2007-08 school year, Mississippi must enter into a Memorandum of Agreement (MOA) with the Department demonstrating its commitment and investment of resources necessary to address all outstanding issues with the MAAECF and High School Alternate Assessment in order to administer a fully compliant standards and assessment system during the 2008-09 school year. This MOA must include a mutually acceptable timeline for how, when, and by whom the remaining work will be completed and submitted for peer review, and must provide for the submission of quarterly reports of Mississippi’s progress along this timeline. I am asking Grace Ross and Patrick Rooney of my office to work closely with your staff to develop this timeline to ensure that it includes all necessary evidence.

Because the Mississippi assessment system does not meet all of the requirements of sections 1111(b)(1) and 1111(b)(3) of the ESEA, the Secretary is authorized, pursuant to section 1111(g)(2) of the ESEA, to withhold all or a portion of the Mississippi Title I, Part A administrative funds. The Secretary will refrain from exercising her authority under that provision so long as Mississippi complies with all material terms of the MOA, including complying with significant deadlines set forth in the timeline. If, however, Mississippi fails to comply with any material term of the MOA, the Secretary may initiate proceedings pursuant to her withholding authority under section 1111(g)(2) of the ESEA. Both parties to the MOA will identify its material terms prior to finalizing and signing the agreement. If the Secretary should initiate withholding proceedings, in so doing and in determining the proper amount to be withheld, the Secretary will take into consideration the number of violations of the MOA by Mississippi as well as any other relevant circumstances.

I appreciate the steps Mississippi has taken toward meeting the requirements of the ESEA and I know you are anxious to receive full approval of your standards and assessment system. We are committed to helping you get there and remain available to provide technical assistance as you continue to complete the work on the MAAECF and the High School Alternate Assessment. We will schedule an additional peer review when you have the evidence available to further evaluate your system. If you have any questions or would like to discuss this further, please do not hesitate to contact Grace Ross (Grace.Ross@ed.gov) or Patrick Rooney (Patrick.Rooney@ed.gov) of my staff.

Sincerely,

Kerri L. Briggs, Ph.D.

Enclosures

cc: Governor Haley Barbour
Kris Kaase

SUMMARY OF ADDITIONAL EVIDENCE REGARDING MISSISSIPPI’S ALTERNATE ASSESSMENTS THAT MISSISSIPPI MUST SUBMIT TO MEET ESEA REQUIREMENTS FOR THE MISSISSIPPI ASSESSMENT SYSTEM

NOTE: The following list of evidence pertains to the Mississippi alternate assessments based on alternate academic achievement standards, the MAAECF and the High School Alternate Assessment.

1.0 – ACADEMIC CONTENT STANDARDS

Documentation showing stakeholder diversity, especially expertise in students with disabilities and limited English proficiency, in the panels that developed the frameworks in language arts, mathematics and science.

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

Evidence that the state has approved/adopted alternate academic achievement standards in reading/language arts and mathematics for each of grades 3 through 8 and high school.
Evidence that the academic achievement standards include, for each content area:
1. At least three levels of achievement, including two levels of high achievement (proficient and advanced) that determine how well students are mastering a State’s academic content standards, and a third level of achievement (basic) to provide information about the progress of lower-achieving students toward reaching the proficient and advanced levels of achievement;
2. Descriptions of the competencies associated with each achievement level; and
3. Assessment scores (“cut scores”) that differentiate among the achievement levels.
Evidence that the state has documented the involvement of diverse stakeholders in the development of its alternate achievement standards.

4.0 -TECHNICAL QUALITY

Has the state documented the issue of validity (in addition to the alignment of the assessment with the content standards), as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999), with respect to all of the following categories:
1. Has the state specified the purposes of the assessments, delineating the types of uses and decisions most appropriate to each? Has the state ascertained that the decisions based on the results of its assessments are consistent with the purposes for which the assessments were designed? and
2. Has the state ascertained whether the assessment system produces intended and unintended consequences? and
For each assessment:
1. Has the state ascertained that the scoring and reporting structures are consistent with the subdomain structures of its academic content standards (i.e., are item interrelationships consistent with the framework from which the test arises)? and
2. Has the state ascertained that test and item scores are related to internal or external variables as intended (e.g. scores are correlated strongly with relevant measures of academic achievement and are weakly correlated, if at all, with irrelevant characteristics, such as demographics)?
3. For each assessment, provide documentation of the standards-setting process. Describe the selection of judges, methodology employed, and final results.
For each assessment, has the state considered the issue of reliability, as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999), with respect to all of the following categories:
1. Has the state determined the reliability of the scores it reports, based on data for its own student population and each reported subpopulation? and
2. For all assessments, has the state reported the conditional standard error of measurement and student classification that are consistent at each cut score specified in its academic achievement standards? and
3. Has the state reported evidence of generalizability for all relevant sources, such as variability of groups, internal consistency of item responses, variability among schools, consistency from form to form of the test, and inter-rater consistency in scoring?
Evidence that the state has ensured that its assessment system is fair and accessible to all students, including students with disabilities and students with limited English proficiency, including a DIF analysis or bias review of items to ensure fairness in the development of the assessments.
When different test forms or formats are used, evidence that the state has taken steps to ensure consistency of test forms over time to ensure that the meaning and interpretation of results are consistent.
Evidence of a system for monitoring and improving the on-going quality of its assessment system.
Evidence that the State has evaluated its use of accommodations.

5.0 – ALIGNMENT

Documentation of the alignment of student work on the alternate assessments with the extended curriculum frameworks.
Evidence that the state has developed ongoing procedures to maintain and improve alignment between the alternate assessments and standards over time, particularly if gaps have been noted.

SUMMARY OF ADDITIONAL EVIDENCE REGARDING MISSISSIPPI’S GENERAL ASSESSMENTS THAT MISSISSIPPI MUST SUBMIT TO MEET ESEA REQUIREMENTS FOR THE MISSISSIPPI ASSESSMENT SYSTEM

NOTE: The following list of evidence pertains to the Mississippi general assessments, the MCT2 and the MSATP.

2.0 – ACADEMIC ACHIEVEMENT STANDARDS

Evidence that the state has approved/adopted academic achievement standards in reading/language arts and mathematics for each of grades 3 through 8 and high school.
Evidence that the academic achievement standards include, for each content area:
1. At least three levels of achievement, including two levels of high achievement (proficient and advanced) that determine how well students are mastering a state’s academic content standards, and a third level of achievement (basic) to provide information about the progress of lower-achieving students toward reaching the proficient and advanced levels of achievement;
2. Descriptions of the competencies associated with each achievement level; and
3. Assessment scores (“cut scores”) that differentiate among the achievement levels.
Evidence that the state has documented the involvement of diverse stakeholders in the development of its achievement standards.

3.0 – FULL ASSESSMENT SYSTEM

Documentation of the results of alignment studies of the assessments to show higher-order thinking skills.

4.0 – TECHNICAL QUALITY

Has the state documented the issue of validity (in addition to the alignment of the assessment with the content standards), as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999), with respect to all of the following categories:
1. Has the state specified the purposes of the assessments, delineating the types of uses and decisions most appropriate to each? Has the State ascertained that the decisions based on the results of its assessments are consistent with the purposes for which the assessments were designed? and
2. Has the state ascertained whether the assessment system produces intended and unintended consequences? and
For each assessment:
1. Has the state ascertained that the scoring and reporting structures are consistent with the subdomain structures of its academic content standards (i.e., are item interrelationships consistent with the framework from which the test arises)? and
2. Has the state ascertained that test and item scores are related to internal or external variables as intended (e.g., scores are correlated strongly with relevant measures of academic achievement and are weakly correlated, if at all, with irrelevant characteristics, such as demographics)?
3. For each assessment, provide documentation of the standard setting process. Describe the selection of judges, methodology employed, and final results.
For each assessment, has the state considered the issue of reliability, as described in the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999), with respect to all of the following categories:
1. a. Has the state determined the reliability of the scores it reports based on data for its own student population and each reported subpopulation? and
2. For all assessments, has the state reported the conditional standard error of measurement and student classification that is consistent at each cut score specified in its academic achievement standards? and
3. Has the state reported evidence of generalizability for all relevant sources, such as variability of groups, internal consistency of item responses, variability among schools, consistency from form to form of the test, and inter- rater consistency in scoring?
Has the state ensured that its assessment system is fair and accessible to all students, including students with disabilities and students with limited English proficiency, with respect to each of the following issues:
1. Has the state ensured that the assessments provide an appropriate variety of accommodations for students with disabilities? and
2. Has the state ensured that the assessments provide an appropriate variety of linguistic accommodations for students with limited English proficiency? and
3. Has the state taken steps, such as DIF analysis or bias review of items, to ensure fairness in the development of the assessments? and
4. Does the use of accommodations and/or alternate assessments yield meaningful scores?
When different test forms or formats are used, the state must ensure that the meaning and interpretation of results are consistent.
1. Has the state taken steps to ensure consistency of test forms over time?
2. If the state administers both an online and paper and pencil test, has the state documented the comparability (achievement level for individual student) of the electronic and paper forms of the test?
Does the state have a system for monitoring and improving the on-going quality of its assessment system?
Has the state evaluated its use of accommodations?
1. How has the state determined that scores for students with disabilities that are based on accommodated administration conditions will allow for valid inferences about these students’ knowledge and skills and can be combined meaningfully with scores from non-accommodated administration conditions?
If appropriate accommodations are available to limited English proficient students has the state documented that these accommodations are used to yield accurate and reliable information about what limited English proficient students know and can do? Does the state monitor availability of accommodations during test administration?
How has the state determined that scores for limited English proficiency students that are based on accommodated administration circumstances will allow for valid inferences about these students’ knowledge and skills and can be combined meaningfully with scores from non-accommodated administration circumstances?
What actions has the state taken to monitor the implementation of accommodations during testing?

5.0 – ALIGNMENT

What ongoing procedures does the state use to maintain and improve alignment between the assessments and standards over time, particularly if gaps have been noted?
Are the assessments and the standards aligned comprehensively, meaning that the assessments reflect the full range of the state’s academic content standards? Are the assessments as cognitively challenging as the standards? Are the assessments and standards aligned to measure the depth of the standards? Do the assessments reflect the degree of cognitive complexity and level of difficulty of the concepts and processes described in the standards?
Do the assessments reflect both the content knowledge and procedural or process skills (such as the application of knowledge to solve problems, or understanding of scientific method) as represented in the state’s academic content standards?
Do the assessments reflect the same degree and pattern of emphasis as are reflected in the state’s academic content standards?
Has the state provided detailed assessment specifications and/or a description of the test development process to illustrate how the assessment system reflects both the content knowledge and skills specified in the academic achievement standards? For instance, if the standards require “writes in different genres,” does the assessment require writing samples from different genres?

6.0 – INCLUSION

Evidence that shows that all students in the grades tested are included in the assessment system (e.g., students with disabilities, students with limited English proficiency, economically disadvantaged students, all race/ethnicity groups, migrant students, homeless students, etc.).

7.0 – REPORTING

Evidence that the state’s reporting system facilitates appropriate, credible, and defensible interpretation and use of its assessment data.
Evidence that the state has provided, for each content area, grade and subgroup, a summary report that includes the number of students enrolled or number tested/not tested as evidence that all students are tested.
Evidence that the state has provided final copies of individual interpretive, descriptive, and [non-clinical] diagnostic reports that indicate relative strengths and instructional needs.

Return to state-by-state listing

Top