FY 2003

Knowing What Students with Significant Cognitive Disabilities Know: Defining and Disseminating Technical Criteria for Alternate Assessments through a Research and Practice Partnership

     States and testing companies have struggled to identify technically adequate but educationally sound methods of assessing the small group of students with significant cognitive disabilities for federal accountability purposes. Typically, experts in educational programming for these students, along with key stakeholders, have advised state assessment offices in defining what the best possible outcomes of standards-based instruction should be for the students. From those definitions, states and test company partners have developed assessments to measure the outcomes for school, district, and state accountability purposes.

     In the next year, state assessment systems will undergo Title I peer review to determine whether the systems meet NCLB requirements. Technical manuals will be important pieces of documentation. This project will address the short-term practical necessity of technical adequacy documentation, and the longer-term research commitment to building measurement models that “work” to measure achievement for this small group of students. We will work at the theoretical and the practice levels, addressing our research questions through collaborative cross-disciplinary study, reflection, discussion, and explication; through prototype development and testing; and through extensive real-world application and review by technical and policy experts from multiple states with varied approaches to alternate assessment.

     The project has three primary goals:
1. First, the project will address the immediate practical challenge of documentation of the technical adequacy of alternate assessment for students with significant cognitive disabilities, along with developing technical assistance processes and products for use with states during and after the project ends.
2. Second, the project will enhance fundamental knowledge of what the results of good teaching and learning look like for students with significant disabilities, allowing educational researchers, measurement experts, and practitioners to identify the kinds of evidence of standards-based learning that can yield valid and reliable inferences for accountability and school improvement purposes.
3. Finally, we will capture lessons learned that will help define areas for improvement of entire assessment systems. We will target areas where technical assistance is needed to document the technical adequacy of alternate assessments, we will identify gaps in our knowledge, and we will define needs for further research.

Validity of Accommodations for LEP Students and Students with Disabilities in Math and English

     The Oklahoma State Department of Education (OSDE) will serve as the lead state organization in collaboration with thirteen jurisdictions including Alabama, Georgia, Hawaii, Iowa, Kentucky, Louisiana, Nebraska, Nevada, New Jersey, Ohio, Oregon, Virginia, and West Virginia to investigate the validity of accommodations in math and English proficiency assessments for limited English proficient students with disabilities (LEP/SD). The proposed project addresses the need for states to identify valid accommodations for LEP students with disabilities in an effort to develop and implement reliable and valid English language proficiency tests as required by Title III, and to fairly assess all students in the math content area as required by Title I of the No Child Left Behind (NCLB) Act. As a result, the project will provide information on the validity of accommodations for future national and state assessments for LEP students with disabilities, a group of students that needs more attention due to their dual challenges, limited English proficiency and individual disabilities.

     This study examines the validity of accommodations in two ways: (1) comparing the performance of accommodated and non-accommodated non-LEP/non-SD students for whom accommodations are not intended, and (2) comparing the criterion-related validity of accommodated and non-accommodated assessment within a structural equation modeling approach. Researchers at the Advance Research & Data Analyses Center (ARDAC) and the California State University, Long Beach (CSULB) will assist the OSDE in the implementation of the project and be responsible for overseeing the research plans, data collection and analyses, and in part, the dissemination of the study’s outcomes. The Center for the Study of Assessment, Validity and Evaluation (C-SAVE) at the University of Maryland will serve as the independent evaluator of this project.

     The results of the study will help identify valid accommodations for LEP students with disabilities and may be applied to the general population of students with disabilities (with or without limited English proficiency). The large number of states participating in this project makes it possible for the results to be generalized to the nation with a greater level of confidence.

Reaching the “Students in the Gap” through Web-based Module Assessments

     Given the current emphasis on accountability, all states need to provide equitable access to assessment for every student, and typically do so through two large-scale assessment systems, the regular (grade-level) and the alternate. However, 3-4% of the overall student population fall between the two assessment systems. Students with moderate cognitive deficits or severe learning disabilities do not qualify for the alternate assessment, but perform poorly on current regular assessment tests. These students are capable of, and could demonstrate, greater proficiency on grade-level assessments if provided with the appropriate scaffolding and contextualization. To meet the needs of these “students in the gap,” the New England Compact Enhanced Assessment Instrument Project proposes to develop a web-based Task Module Assessment (TMAS) prototype.

     The TMAS will be web-based, allowing for local administration; it will include task- and performance-based assessments rather than on-demand assessments; computer-based accommodations where appropriate; and online and offline assessment options. It will be designed to measure student learning against grade-level standards in mathematics and/or English Language Arts at one grade level and will be aligned to the large-scale assessment. The final products will consist of common criteria across the four NE Compact states defining the process for identifying students who qualify for the Task Module Assessment; a TMAS prototype for one grade; online assessment architecture with computer accommodations and options for delivery and performance; validity and reliability results for the TMAS; “how to” guides to build the capacity of other agencies to deal with assessment issues for students in the gap; and a series of monographs to disseminate findings to help “demystify” validity and reliability issues.

     Dissemination will use strategically selected venues to ensure all states and interested researchers timely access to the products including, as examples, conference presentations, articles in newsletters, publications and establishing links from key websites to the New England Compact’s website.

Achieving Accurate Results for Diverse Learners: Accommodations and Access Enhanced Item Formats for English Language Learners and Students with Disabilities (AARDL)

     The overall goal of the Achieving Accurate Results for Diverse Learners: Accommodations and Access Enhanced Item Formats for English Language Learners and Students with Disabilities (AARDL) project is to obtain more accurate results about the academic achievement of English language learners (ELLs) and students with disabilities (SWDs). AARDL proposes to empirically investigate strategies designed to increase access to test content for these students and others who may encounter barriers to demonstrating their knowledge of content areas and cognitive ability under regular testing conditions. The proposed study addresses all four of the absolute priorities laid out in the Request for Proposals as well as all three competitive preferences.

     The AARDL project involves a consortium of states and jurisdictions including South Carolina, North Carolina, Maryland, Minnesota, Oregon, Pennsylvania, and the District of Columbia who will collaborate with the University of Maryland, the University of Oregon, the University of South Carolina, Data Recognition Corporation (DRC) and SERVE. This diverse collaborative of state departments of education, research institutions, testing contractors and external evaluators will ensure that the project has a firm grounding in both theory and practice.

     The main objectives of the AARDL project are to (1) develop Access Enhanced (AE) Items in four subject areas and 6 grade levels with accompanying scoring guides focusing on responses from diverse learners; (2) investigate the reliability and utility of the Accommodation Station; (3) determine the comparability and scalability of the Access Enhanced (AE) Items; and (4) disseminate results through reports and a handbook for developing AE items.The results of the AARDL will provide an important contribution to the growing body of research into providing appropriate testing accommodations for those students for whom regular testing conditions pose a barrier to accessing content. New strategies such as the ones AARDL investigates are critical to the valid, reliable, and accurate assessment of English language learners and students with disabilities. Having accurate results for these and all students is essential to ensure the accountability of the educational system, determine how to best meet students’ educational needs, and track student progress over time.

Project DAATA: Developing Alternate Assessment Technical Adequacy

     All states currently have alternate assessment systems for students with significant disabilities. The problem is that 50 different approaches and strategies have been taken in the development of tasks used in the alternate assessment, alignment of alternate assessments with state standards, development of state standards upon which to construct alternate assessments, training of teachers in the administration, scoring and reporting of alternate assessments, validation of alternate assessments, and policy guidelines. And little technical adequacy exists on any of the 50 systems. This proposal reflects the next logical step in integrating practice with technically adequate measurement through the use of rigorous research methodologies. In this application, three major institutions (WV Department of Education, the Council of Chief State School Officers, and Behavioral Research and Training-University of Oregon) will work with a consortium of states to develop a handbook on technical adequacy of alternate assessments. We also propose incorporation of validated instruments and reporting systems within this handbook so that states have easy access and practical examples for use in developing technical adequacy in their own state. We address five related components in our focus on technical adequacy, each of which is aimed to have impact on classroom practice and the improvement of student performance and progress: (a) content validity, which should support development of appropriate Individualized Educational Programs (IEPs); (b) generalizability which should help teachers target classroom instructional practice; (c) reliability which should provide a stable analysis of current and expected performance for students with significant disabilities in the most accurate manner; (d) criterion and predictive validity which should help situate performance and allow teachers to trust the outcomes; and (e) consequential validity which should help states report outcomes and improve training and practice. The handbook is designed for state level use in training and policy development. West Virginia Department of Education is the lead state and applicant for this grant. The Council of Chief State School Officers (CCSSO), via the Assessing Special Education Students (ASES) State Collaborative on Assessment and Student Standards (SCASS) brings several states together to actually conduct the research. Behavioral Research and Teaching in the College of Education at the University of Oregon is contracted to conduct much of the research operations. Nationally recognized researchers in alternate and large-scale assessment assist in dissemination: National Association of State Directors of Special Education (NASDSE), National Center on Educational Outcomes (NCEO) and the Regional Resource Centers (RRC)