FY 2016

Innovations in Science Map, Assessment, and Report Technologies (I-SMART) Project Objectives and Activities:

Abstract

Next Generation Science Standards (NGSS) reflect high expectations for students and are based on a multidimensional model of learning science. As states adopt the NGSS, high-quality assessments are needed to measure student learning of more rigorous standards and provide timely and useful feedback about student performance. I-SMART’s ultimate goal is to maximize science achievement and progress across grades for students with significant cognitive disabilities (SCD) who take alternate assessments and for students with or without disabilities who are not yet meeting grade-level standards.

Goal 1: Develop and evaluate a learning map model for science. The project will build on existing local neighborhood maps around science grade-level targets for students with SCD by integrating science map neighborhoods with a multidisciplinary learning map that includes knowledge and skills in English language arts and mathematics. Activities include developing and evaluating the learning map model.

Goal 2: Design, develop, and evaluate assessments that incorporate science disciplinary content and science and engineering practices in highly engaging, universally designed, technology-delivered formats. Using an evidence-centered design approach, we will develop testlets (short assessments) that measure students’ knowledge and skills in science content aligned to the learning map. Universal Design for Learning principles will be incorporated to maximize student engagement and minimize barriers. After prototyping innovative items and testlets and receiving stakeholder input, refined testlets will be externally reviewed, pilot tested, and evaluated for their potential to support reliable, valid, and fair assessment.

Goal 3: Design, develop, and evaluate a dashboard that provides diagnostic feedback based on student performance on science assessments. Using iterative prototype designs and with input from stakeholders, we will develop a reporting dashboard that provides feedback on individual student performance on the new testlets. Using information from the learning maps and connections with other content areas, results will support teaching, learning, and communication with parents. The dashboard will include recommendations for instruction and embed just-in-time assessment literacy supports to facilitate appropriate interpretations and uses of results.

Goal 4: Broadly disseminate project materials and findings to a variety of audiences. The project’s dissemination plan includes dissemination of materials and products developed in goals 1-3, lessons learned during the design process, and research outcomes to stakeholder organizations, educators in the field, professional organizations, researchers, and policy makers.

Priorities: I-SMART addresses all four absolute priorities and competitive preference priorities 1(a&c) and 2(a&c). The project will be in collaboration with five states (Maryland-lead, Missouri, New York, New Jersey, Oklahoma), the University of Kansas Center for Educational Testing and Evaluation, the Center for Applied Special Technology (CAST), and BYC Consulting to produce assessments and materials to support comprehensive alternate assessments that include multiple measures of student progress over time. The project delivers innovative science assessments and score reports that improve the utility of information about student performance. I-SMART includes a comprehensive dissemination plan for materials, processes, and results.

Outcomes: The science learning map model includes multiple pathways for students to learn science and reach challenging grade-level expectations. Assessments aligned to the learning map model will measure student learning. The reporting dashboard would be appropriate for within-year uses and may also be useful for fine-grained reporting of summative results.

Participants and sites: Approximately 4,500 students and their teachers across the partner states.

SCILLSS Project Objectives and Activities:

Abstract

In the Strengthening Claims-based Interpretations and Uses of Local and Large-scale Science Assessments (SCILLSS) project, we propose to establish a foundation from which a broad range of enhanced science assessments that yield valid score interpretations can be built, evaluated, and shared across states, local education agencies, schools, and classrooms using a principled-design approach. To address this objective, SCILLSS is organized into six phases. Phase 1 includes project management activities to ensure that the project is managed appropriately. Phase 2 includes needs assessments to gather important information about the status and characteristics of state and local assessment systems. Phase 3 involves the creation of a validity evaluation framework that can be tailored to specific assessment types and contexts. Phase 4 focuses on the intra-assessment examination of performance level descriptors, task models, items, and blueprints and the creation of large-scale assessment design and development tools that target standards-based concepts and skills. Phase 5 involves the creation of classroom-based evidence and tools to support effective interpretations and uses of large-scale assessment results. Phase 6 involves project evaluation and reporting to evaluate states’ progress, guide next steps, and provide useful reports.

Applicable Priorities: Through the SCILLSS project, we propose to address each of the Secretary’s four absolute priorities (APs) and three competitive preference priorities (CPPs). We address AP1 (collaboration) by bringing together three states, three independent organizations, and an external evaluator to improve the quality of statewide assessment systems in science. To address AP2 (multiple measures) we will establish a means for states to strengthen the meaning of statewide assessment results and to connect those results with local assessments in a complementary system. We will collect aggregated statewide assessment data and individual exemplars in a body of evidence that supports analysis of cross-sectional and within-student progress, as emphasized in AP3 (charting student progress over time). For AP4 (comprehensive assessment instruments) we will build principled-design tools to guide educators through a replicable process aimed at strengthening their assessment systems in science.

SCILLSS will address CPP1 (developing innovative assessment item types and design approaches) by using principled-design methodologies to evaluate current science assessment items and to develop task models for new innovative science items. We will address CPP2 (improving assessment scoring and score reporting) by engaging state and local educators to clarify the intended interpretations and uses of assessment scores, and to create a repertoire of tools aimed at improving the utility of student performance results for all stakeholders. We will address CPP3 (inventory of state and local assessment systems) by administering a needs assessment for each state to review their statewide and local assessments for quality, standards and instructional alignment, purpose, utility, and equity.

Proposed Project Outcomes: A primary goal of SCILLSS is to leverage existing tools and expertise to generate more broadly applicable resources and to strengthen the knowledge base among stakeholders for using principled-design approaches to create and evaluate quality science assessments that generate meaningful and useful scores. The SCILLSS tools and resources will be designed to have applicability and use beyond the participating project states.

Number of Participants to be Served: The SCILLSS project will involve key state and local education agency staff, approximately 120 educators, and a broad representation of students representing the three participating states (NE, MT, and WY), and will generate widely applicable tools and resources for use and dissemination beyond the participating states.

Number and Location of Proposed Sites: Project activities will be conducted virtually as well as on-site at local school districts and state education agencies within the partner states.