Awards

The Development of Alternate English Language Proficiency Assessment Procedures for English Language Learners with Significant Disabilities

The Washington, DC Office of the State Superintendent of Education (OSSE), on behalf of the 17-state World-class Instructional Design and Assessment (WIDA) Consortium, proposes to develop and implement a feasible, accessible, valid, and efficient standards-based English language proficiency (ELP) alternate assessment system that yields technically sound results and facilitates the inclusion of English language learners (ELLs) with significant disabilities in educational accountability systems across the WIDA Consortium. This performance-based alternate assessment system will compliment and parallel the University of Wisconsin – Madison and WIDA’s evidence-based collection alternate ELP approach that is currently being field-tested within WIDA Consortium states. This new performance-based approach will give WIDA states the flexibility to implement alternate ELP assessments consistent with their existing alternate academic content assessments. The WIDA Consortium, which is located within the Wisconsin Center for Education Research (WCER) at the University of Wisconsin – Madison, will lead the development of this assessment through a cooperative agreement with the Washington, DC OSSE.

The WIDA Consortium, originally established with funding from a U.S. Department of Education Enhanced Assessment Grant, currently includes Washington, DC and 16 additional states. Combined, the 17 WIDA partner states contain approximately 550,000 K-12 ELLs. Since 2003, WIDA has created and adopted comprehensive English language proficiency standards (2004, 2007) that represent the second language acquisition process and the language in the content areas of language arts, mathematics, science, and social studies. Based on these standards, WIDA developed a K-12 ELP test battery–ACCESS for ELLs–that approximately 420,000 students took in spring 2007. The ACCESS for ELLs is currently being used by more states than any other ELP measure. WIDA also provides professional development activities and maintains a Web site (www.wida.us).

The proposed alternate ELP assessment system, named the Alternate ACCESS for ELLs with Significant Disabilities, will be designed to (a) meet the accountability requirements of the No Child Left Behind Act of 2001 and the Individuals with Disabilities Education Improvement Act of 2004, (b) meet the technical requirements of the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999), (c) facilitate the involvement of ELLs in participating states’ accountability systems, (d) provide a method for monitoring the ELP growth of ELLs with significant disabilities, and (e) provide guidance to individualized education program (IEP) teams in developing appropriate language proficiency IEP goals and objectives. The development of this parallel form of the Alternate ACCESS for ELLs will follow key principles that require the assessment to (a) identify and assess skills that are critical to language proficiency development; (b) be aligned with the WIDA Consortium’s language proficiency standards; (c) be sensitive to student growth and accurately reflect students’ abilities in language areas; (d) lead to instructional opportunities that meet student needs; (e) provide reliable and valid results; (f) be non-biased and sensitive to cultural differences; (g) produce results that are helpful to teachers, parents, and administrators in making educational decisions; and (h) be time- and resource-efficient, as well as consistent with participating WIDA Consortium states’ existing academic content alternate assessment systems. A multi-part investigation using a multi-method, multi-source approach to developing and evaluating the alternate assessment will be used. Throughout all phases of the proposed alternate ELP assessment development and validation procedures, the involvement of WIDA Consortium states will increase the likelihood of participation by a culturally, geographically, and structurally diverse population of schools that will increase the generalizability of findings from the project. It is anticipated that we will be collecting data from a minimum of 300 schools within the consortium, including schools from each WIDA Consortium state.

Although a variety of academic content assessments are available for use with ELLs with significant disabilities, there currently exist no alternate ELP assessments for ELL students with significant disabilities. Consequently, this project will advance theory, knowledge, and practice in the fields of assessment and instructional programs for ELLs with significant disabilities. We anticipate that the development and use of alternate ELP assessments for ELLs in the beginning stages of English language acquisition will prove to be a valid, reliable, and equitable way to assess the English language proficiency of ELLs with significant disabilities.

Modifications for a Better Assessment of What Students with Disabilities Know and Can Do

The state of Minnesota, in collaboration with the states of Ohio, Oregon, and with the American Institutes for Research propose a research and design study to improve our planned Alternate Assessment of Modified Achievement Standards (AA-MAS). The AA-MAS targets persistently low-performing students with disabilities.

Our proposed project addresses Absolute Priority 1 by collaborating with the American Institutes for Research and university-based cognitive psychologists to improve the reliability and validity with which state assessments can measure the academic achievement of students with a variety of disabilities whose skills are not appropriately measured through the general education assessment or the alternate assessment based on alternate achievement standards (AA-AAS). Although an AA-MAS is permitted under ESEA for this population, it is not required, and these improvements will extend the reliability and validity of the tests for this population beyond the ESEA requirements.

Development of our AA-MAS builds on the idea that deficits in specific cognitive traits may impede student performance on assessment tasks. Under an existing General Supervision Enhancement Grant (GSEG), our consortium is currently designing modifications to remove specific impediments that arise from deficits in working memory, executive function (planning), and focused and sustained attention. Successful modifications will lower barriers imposed by these deficits while minimizing changes to the construct being measured, thereby enabling students with disabilities to more accurately show what they know and can do.

This proposal describes a field test designed as an experiment. Students in the experimental group and the control group will respond to assessments including both modified and unmodified test items. Qualified evaluators will also administer brief, validated screeners to Minnesota, Ohio, and Oregon Consortium page 2 measure the four cognitive traits, deficits in which are often associated with cognitive disabilities. Analysis of the data will evaluate whether the modifications render the assessments more accessible without simply making them easier.

Our design directly addresses the Secretary’s three competitive preference priorities by 1) promising significant advancement in our understanding of how to validly test student with disabilities through alternate assessment and accommodations; 2) collaborating in this effort with a three-state consortium; and 3) proposing a dissemination plan that will reach all state assessment programs.

Integrated Simulation-Based Science Assessments into Balanced State Science Assessment Systems

The Nevada State Department of Education will lead a collaboration of seven states including Connecticut, Massachusetts, Nevada, North Carolina, Utah, Washington, and Vermont to study the feasibility of integrating computer simulation-based science assessments into balanced state science assessment systems. The collaboration will take place in partnership with WestEd, the Council of Chief State School Officers (CCSSO), and the Center for Research on Educational Standards and Student Testing (CRESST) at UCLA. The purpose of the project is to support the assessment of science knowledge and inquiry strategies not typically well-measured in paper-based large scale science tests by implementing local technology-based science formative, curriculum-embedded and end-of unit benchmark assessments that can augment district and state science test evidence of progress on science standards. The goals of the project are to study: (1) the technical qualities of the simulation-based science assessments; (2) the feasibility and utility of the assessments for formative, summative, and accountability purposes; (3) the effects of the simulation-based assessments for all students, English learners, and students with disabilities; and (4) propose alternative models for integrating simulation-based assessments into state science assessment systems.

Four of the states (NV, NC, UT, WA) will pilot test the assessments in three demographically diverse districts per state, 108 teachers and approximately 10,800 students. The project will study how the rich environments, multiple representations, flexible response formats, and accommodations in the technology-enhanced science assessments benefit the performance of disadvantaged students, English learners, and students with disabilities. CCSSO and Nevada will lead a State Science Assessment Design Panel of all seven states to monitor the pilot testing and develop specifications and models for integrating simulations into balanced state science assessment systems.

Examining the Feasibility, Effect and Capacity to Provide Universal Access through Computer-Based Testing

The project proposed here seeks to examine the feasibility, effect, and capacity to deliver state achievement tests using a computer-based test delivery system specifically designed to provide universal access to test content for students with disabilities or special needs. The proposed project is a direct outgrowth of prior work supported by EAG funding conducted by New Hampshire, Vermont, and Rhode Island in which the feasibility of using computers to provide specific test accommodations was examined. Based on this prior work, members of the New Hampshire Department of Education Curriculum and Assessment program conducted a statewide pilot test in which the interface used for the prior EAG project was used to provide a read aloud accommodation to students for its 2006 grade 10 mathematics test. This successful pilot led to a collaborative effort with Nimble Assessment Systems to develop a comprehensive test delivery system that employed principles of universal design to flexibly meet the accessibility and accommodation needs of individual students. The proposed project brings together 11 states to examine the feasibility and effect of using this comprehensive test delivery system to improve test validity for students with disabilities and special needs who are believed to benefit from one or more of the accessibility and accommodation tools built into the system. Specifically, members of this collaborative project include: New Hampshire, Vermont, Rhode Island, South Carolina, North Carolina, Georgia, Montana, Iowa, Connecticut, Maryland, and Florida. In addition, the proposed project includes partnerships with the National Center for Educational Outcomes, Nimble Assessment Systems, and the NECAP state contractor (currently Measured Progress).

The proposed project will undertake 5 major initiatives:

• Conduct a set of three computer-based test accommodation efficacy studies.
• Employ the UAS to deliver the operational Grade 11 Mathematics, Reading and Writing tests, and the operational Grade 4, 8 and 11 Science tests.

• Develop and validate a school computer-based test delivery capacity index.
• Analyze school computer-based test delivery capacity for schools in participating states.
• Conduct a cost analysis for preparing for and delivering a test using a computer-based test delivery system with embedded accommodations.

Given that three members of the collaborative project have jointly developed achievement tests that are used across their respective states, the project will focus specifically on the NECAP tests. This will enable efficient replication of the study findings across multiple states while allowing all participating states to refine research questions, analyses, and the development of the School Capacity Index. Collectively, the number and variety of studies undertaken through this project holds promise to rapidly advance assessment practices within each of the participating states, while also informing the practices in non-participating states.

Alternate Assessment Design—Mathematics (AAD-M)

The Utah State Office of Education is the applicant for the proposed Alternate Assessment Design—Mathematics project. With technical support from SRI International, the states of Utah, Idaho, and Florida will collaborate to achieve the following goals: (1) extend the conceptual framework of evidence-centered design (ECD) to alternate assessment based on alternate achievement standards (AA-AAS) using the Principled Assessment Design for Inquiry (PADI) model and (2) develop AA-AAS testing designs, blueprints, and assessment task specifications that address priority state academic standards in mathematics for students with significant cognitive disabilities.

The collaborating states have AA-AAS systems for students with significant disabilities (1%), and they have completed one peer review cycle (NCLB). They are now refining their assessments to improve technical quality. The states are seeking to improve quality by developing a system of structured or standardized performance tasks that are aligned with grade-level academic content, boosting expectations for student achievement, and increasing access to grade-level academic standards using ECD. The states are also seeking to increase the reliability of their alternate assessment systems.

Two obstacles hinder achieving these intentions: few states documented the procedures or the rationale they used to set priorities for what is tested in alternate assessment (the content domains) or used a systematic process to examine the characteristics of the content to be tested to guide the design of the assessment and the tasks or items. The proposed project will address these obstacles by taking the next logical step to integrate practice with grounded measurement principles—emulating and extending an ECD approach to designing alternate assessment (AA-AAS).

The foundation for this project, ECD, is a practical theory-based approach to developing quality assessments that combines developments in cognitive psychology and advances in measurement theory and technology. The purpose of the project is to apply ECD, as it is manifested in the PADI model, to designing, delivering, and scoring assessments of the academic achievement of students with significant cognitive disabilities. Although Utah, Idaho, and Florida have unique needs, the PADI model is robust and suitable for addressing each state’s needs.

In addition to emulating the PADI model, the scope of the project encompasses (1) developing design pattern (frameworks or schema used to design assessments), (2) producing assessment blueprints or templates that are based on the design patterns, (3) describing the conditions required to effectively present tasks and evaluate student performance, and (4) producing written descriptions of exemplars of assessment tasks and scoring systems. In conducting these activities for the alternate mathematics assessments, the collaborating states will gain the expertise needed to apply the principles of the ECD model to designing alternate assessments (AA-AAS) in other subject areas. The project will thereby produce procedural guidelines for designing future assessments. These guidelines, additional materials about the approach, and other project results will be disseminated to the field on a website and through multiple presentations at key conferences and meetings.

In addition to these important, concrete benefits to statewide assessment systems and the field of measurement, the proposed project offers an opportunity both to extend a contemporary approach to test design through a novel application and to evaluate this extension through pilot-testing of selected tasks across multiple states.