Evaluation Resources

Below, we offer a set of resource documents to help you understand, design, and implement evaluation methods for educator development programs in a way that is consistent with the Department of Education What Works Clearinghouse (WWC) standards.

How to use this page: To find resources that have the most relevance to your interests, use the filters on the right side of the page. You can find the definition for each field as you hover over the field name. Click here for more detailed definitions of these fields (or click here to download a PDF version). To learn more about each resource, click on the resource title. Click here to download a User Guide on this database.

Resource # Title Author(s) Hosting/Publishing Organization Short Description Year of Publication Weblink Type 1 Type 2 Length Audience References WWC Standards Short Cycle Low Cost Educator Development Other Education Topics RCT QED RDD SCD Identifying Comparison Groups Determining Sample Sizes Recruiting Study Participants Addressing Changes in your Sample Reducing Bias in Comparison Groups Acquiring Administrative Data Selecting Appropriate Outcome Measures Collecting New Data Combining Data Systems Understanding Data Analytic Models Addressing Analysis Challenges Reporting Findings Visualizing Data Student Achievement(measure) Student Behavior(measure) Teacher(measure) Principal/School(measure) District(measure) Associated Keywords Length Value Group Any Examples of Outcome Measures Is Brief or Summary? Is Guide? Is Tool? Is Methods Reports? Is Video? Is Webinar? Is Slide Presentation? Initial Planning
1 5-minute evaluation resource series None Office of Innovation and Improvement, U.S. Department of Education This series is designed to provide a non-technical audiences information on evaluation design in short, informative, and easy to read resources. Each resource is meant to be read or watched in 5 minutes or less. The series is organized into suites of resources that focus on a specific evaluation component. The first suite focuses on the evaluation life cycle and the What Works Clearinghouse, the U.S. Deparment of Education’s reliable source for rigorous research. The suites that follow focus on different stages within the evaluation life cycle. 2019 Link to Resource Brief or summary guide 5 P True False False True True True True True False True True True True True True True False False False True True False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 1 False True True False False False False False True
2 Pathway to Effective Evaluation: Stage 1–Plan and Design Armstrong, K.R.; Brandt, C. Office of Innovation and Improvement, U.S. Department of Education This guide describes the first stage of the evaluation life cycle, the plan and design stage. The steps include 1. Determine your research question; 2. Create your logic model; 3. Carefully describe the people and context of your program; and 4. Select your research design. Each step is explained in further detail on the second page of the resource, with suggestions and links for resources that provide readers with more in-depth information about each step. 2018 Link to Resource Guide Brief or Summary 2 P True False False True True True True True False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 1 False True True False False False False False True
3 Pathway to Effective Evaluation: Stage 2–Identify and Recruit Participants Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This guide describes the second stage of the evaluation life cycle, identify and recruit your participants. The steps include 5. Identify comparison groups; 6. Determine sample size; 7. Recruit people to participate in your study; 8: Address changes in your sample, and 9. Reduce bias in your comparison groups. Each step is explained in further detail on the second page of the resource, with suggestions and links for resources that provide readers with more in-depth information about each step. 2019 Link to Resource Guide Brief or Summary 2 P True False False True True True True False False True True True True True False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, sample size; confounding factors, attrition, baseline equivalence, reducing bias, changes in your sample, 1 False True True False False False False False True
4 Evaluation Life Cycle Armstrong, K.R.; Brandt, C. Office of Innovation and Improvement, U.S. Department of Education This infographic provides an overview of the stages of the evaluation life cycle: plan and design; identify and follow participants; collect and store data; analyze data; and report and use findings. 2017 Link to Resource Guide Brief or Summary 1 P True False False True True True True True False True False False False False False False False False False False False False False False False False False Evaluation Life Cycle; Evaluation Design; 1 False True True False False False False False True
5 Group Evaluation Comparison Chart Armstrong, K.R.; Brandt, C. Office of Innovation and Improvement, U.S. Department of Education This infographic provides an overview of three group designs for evaluation: randomized controlled trials, regression discontinuity design, and matched comparison design. 2017 Link to Resource Guide Brief or Summary 1 P False False False True True True True True False False False False False False False False False False False False False False False False False False False Evaluation Design; QED; RDD; RCT 1 False True True False False False False False False
6 5-Minute Evaluation Resource Series: Illustrated script for Video #5, The Power of Sample Size, Part 5 Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This illustrated script provides the text and screen shots of the vdieo of the same name. The short video builds on the ‘The Power of Sample Size’ part 1 video, and discusses how decisions around cluster designs and pre-intervention data affect the number of people you need in your study to detect a meaningful difference. The video is designed for a non-technical audience. 2019 Link to Resource Video 5 P False False False True True True False False False True True False False False False False False False False False False False False False False False False Comparison Groups; sample size; power analysis; MDE; Minimum Detectable effect; confidence intervals; simple experimental design 1 False False False False False True False False True
7 5-Minute Evaluation Resource Series: Video #1, Evaluation for Educators Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This introductory video begins a series designed to help practitioners understand the basic concepts behind program evaluation, particularly educator effectiveness programs (although the concepts can be applied to other areas of education as well). The introductory video explains the goal of the series, which is to help practitioners engage with their evaluators, be more involved in the evaluation process, and understand WWC standards. The video also introduces the suite of resources that accompany it. 2018 Link to Resource Video guide 5 P True False False True True True True True False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 1 False False True False False True False False True
8 5-Minute Evaluation Resource Series: Video #2, Life cycle of an evaluation Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education The life cycle evaluation video features an overview of the stages of the life cycle: plan and design, identify and follow participants, collect and store data, analyze data, report and use findings. This is the second video in a series designed to help practitioners understand the basic concepts behind program evaluation, particularly educator effectiveness programs (although the concepts can be applied to other areas of education as well). 2018 Link to Resource Video guide 5 P True False False True True True True True False True False False False False False True False False False False True False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 1 False False True False False True False False True
9 5-Minute Evaluation Resource Series: Video #3, The case for comparison groups Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education The case for comparison groups video explains the concept of comparison groups, why they are necessary, and how they are chosen to ensure a valid evaluation that can meet WWC standards. This is the 3rd video in the 5 minute evaluation resources series, designed to help practitioners understand the basic concepts behind program evaluation, particularly educator effectiveness programs (although the concepts can be applied to other areas of education as well). 2018 Link to Resource Video guide 5 P True False False True True False False False False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, ; 1 False False True False False True False False True
10 5-Minute Evaluation Resource Series: Video #4, The Power of Sample Size, Part 1 Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This short video illustrates why having a large enough sample size is essential for identifying whether your program made a meaningful difference. It also addresses some of the decisions that are made in conducting a power analysis, including what a minimum detectable effect (MDE) is and how it affects the number of people you need in your study. The video is designed for a non-technical audience. 2019 Link to Resource Video 5 P True False False True True True False False False True True False False False False False False False False False False False False False False False False Comparison Groups; sample size; power analysis; MDE; Minimum Detectable effect; confidence intervals; simple experimental design 1 False False False False False True False False True
11 5-Minute Evaluation Resource Series: Video #5, The Power of Sample Size, Part 5 Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This short video builds on the ‘The Power of Sample Size’ part 1 video, and discusses how decisions around cluster designs and pre-intervention data affect the number of people you need in your study to detect a meaningful difference. The video is designed for a non-technical audience. 2019 Link to Resource Video 5 P True False False True True True False False False True True False False False False False False False False False False False False False False False False WWC; Comparison Groups; sample size; power analysis; MDE; Minimum Detectable effect; confidence intervals; simple experimental design 1 False False False False False True False False True
12 5-Minute Evaluation Resource Series: Video Script #1, Evaluation for Educators Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This pdf is an illustrated script of the introductory video that begins a series designed to help practitioners understand the basic concepts behind program evaluation, particularly educator effectiveness programs (although the concepts can be applied to other areas of education as well). The introductory video explains the goal of the series, which is to help practitioners engage with their evaluators, be more involved in the evaluation process, and understand WWC standards. The video also introduces the suite of resources that accompany it. 2018 Link to Resource Guide Brief or Summary 8 P True False False True True True True True False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 2 False True True False False False False False True
13 5-Minute Evaluation Resource Series: Video Script #2, Life cycle of an evaluation Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This pdf provides an illustrated script to the life cycle evaluation video which features an overview of the stages of the life cycle: plan and design, identify and follow participants, collect and store data, analyze data, report and use findings. This is the second video in a series designed to help practitioners understand the basic concepts behind program evaluation, particularly educator effectiveness programs (although the concepts can be applied to other areas of education as well), and WWC standards. 2018 Link to Resource Guide Brief or Summary 8 P True False False True True True True True False True False False False False False True False False False False True False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 2 False True True False False False False False True
14 5-Minute Evaluation Resource Series: Video Script #3, The case for comparison groups Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This pdf is an illustrated script for the case for comparison groups video. It explains the concept of comparison groups, why they are necessary, and how they are chosen to ensure a valid evaluation that can meet WWC standards. This is the 3rd video in the 5 minute evaluation resources series, designed to help practitioners understand the basic concepts behind program evaluation, particularly educator effectiveness programs (although the concepts can be applied to other areas of education as well). 2018 Link to Resource Guide Brief or Summary 9 P True False False True True False False False False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, ; 2 False True True False False False False False True
15 5-minute evaluation resource series–Suite 1: The evaluation life cycle None Office of Innovation and Improvement, U.S. Department of Education The first suite of the 5-minute evaluation series focuses on the evaluation life cycle and the What Works Clearinghouse, the U.S. Deparment of Education’s reliable source for rigorous research. This series is designed to provide a non-technical audiences information on evaluation design and WWC standards in short, informative, and easy to read resources. Each resource is meant to be read or watched in 5 minutes or less. The series is organized into suites of resources that focus a specific evaluation issue. 2019 Link to Resource Brief or summary guide 5 P True False False True True True True True False True True True True True True True False False False True True False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 1 False True True False False False False False True
16 5-minute evaluation resource series–Suite 2: The Planning and Design Stage None Office of Innovation and Improvement, U.S. Department of Education The second suite of the 5-minute evaluation series focuses on the first stage of the evaluation life cycle –the planning and design stage. This stage includes 4 steps: 1. Determine your research questions; Create your logic models; Carefully describe the people and context of your program; and 4 select your research design. This series is designed to provide a non-technical audiences information on evaluation design and WWC standards in short, informative, and easy to read resources. Each resource is meant to be read or watched in 5 minutes or less. The series is organized into suites of resources that focus a specific evaluation issue. 2019 Link to Resource Brief or summary guide 5 P True False False True True True True True False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 1 False True True False False False False False True
17 5-minute evaluation resource series–Suite 3: Identify and Follow Participants. None Office of Innovation and Improvement, U.S. Department of Education The third suite of the 5-minute evaluation series focuses on the second stage of the evaluation life cycle –identify and follow participants. This stage includes five steps: 1. Identify comparison groups; 2. Determine sample sizes; 3. Recruit study participants; 4.Address changes in your sample; 5. Reduce bias in comparison groups. This series is designed to provide a non-technical audiences information on evaluation design and WWC standards in short, informative, and easy to read resources. Each resource is meant to be read or watched in 5 minutes or less. The series is organized into suites of resources that focus a specific evaluation issue. 2019 Link to Resource Brief or summary guide 5 P True False False True True True True True False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, sample size; confounding factors, attrition, baseline equivalence, reducing bias, changes in your sample, 1 False True True False False False False False True
18 Identifying a Valid Comparison Group Armstrong, K.R.; Brandt, C. Office of Innovation and Improvement, U.S. Department of Education This guide defines what the counterfactual condition is and why it is important. The second page of the guide describes the factors that need to be considered when selecting appropriate comparison groups. 2018 Link to Resource Guide Brief or Summary 2 P True False False True True True True True False True False False False False False False False False False False False False False False False False False WWC, Comparison groups, Evaluation life cycle, standards, evaluation design, RCT, QED, RDD; 1 False True True False False False False False True
19 The Power of Sample Size Armstrong, Karen R. Office of Innovation and Improvement, U.S. Department of Education This 2-page brief addresses why its important to have enough people in your sample to detect that your program made a meaningful difference. The second page of the brief identifies some of the evaluation decisions that will increase or decrease the number of people you will need. It provides a quick resource for seeing how sample size needs change depending on (1) the size of the minimum detectable affect; (2) whether you are conducting a simple or cluster evaluation design, (3) how many individuals are in each cluster (4) whether students within a cluster represent a homogenous or heteorgenous group, and (5) the predictive strength of pre-intervention data that you have available. 2019 Link to Resource Brief or summary guide 2 P False False False True True True False False False True True False False False False False False False False False False False False False False False False Comparison Groups; sample size; power analysis; MDE; Minimum Detectable effect; confidence intervals; simple experimental design 1 False True True False False False False False True
20 Definitions for Database Fields: Evaluation Resources for Supporting Effective Educator Development None Office of Innovation and Improvement, U.S. Department of Education This document provides definitions for each field used in the SEED Evaluation Resources Database. 2017 Link to Resource Guide 3 P True False False True True True True True False True True True True True True True True True True True True True False False False False False None 1 False False True False False False False False True
21 Recruiting participants: Excerpt from “Recognizing and conducting opportunistic experiments in education; a guide for policymakers and researchers Resch, A., Berk, J., & Akers, L. REL This excerpt of the guide for policy makers and researchers focuse on strategies to consider when recruiting participants for an education rigorous evaluation. 2014 Link to Resource Brief or summary guide 1 P False False False False True False False False False True False True False False False False False False False False False False False False False False False Recruiting participants 1 False True True False False False False False True
22 Technical Terms for a Non-Technical Audience None Office of Innovation and Improvement, U.S. Department of Education This glossary lists evaluation terms that may not be familiar to non-technical readers and gives brief definitions for each term. 2017 Link to Resource Guide Brief or Summary 3 P True False False True True True True True True False False False False False False False False False False False False False False False False False False Attition, Confounding Factors; Evaluation design; Evaluation terms; baseline equivalence, effect size, WWC 1 False True True False False False False False True
23 5-Minute Evaluation Resource Series: User’s Guide: Getting the most out of this series IMPAQ International, U.S. Dept of Education Office of Innovation and Improvement, U.S. Department of Education This user’s guide accompanies the 5-Minute Evaluation Resource Series. The guide explains what is contained in each suite within the series and offers suggestions for the easiest way to use the suites. It also points readers to the Evaluation Resource Database that accompanies the series if they want more in-depth information about a particular topic. 2017 Link to Resource Guide 1 P True False False True True False False False False False False False False False False False False False False False False False False False False False False None 1 False False True False False False False False True
24 User’s Guide: Database of Evaluation Resources for Supporting Effective Educator Development None Office of Innovation and Improvement, U.S. Department of Education This guide provides the reader with instruction on the use of the SEED Evaluation Resource Database, including how to use the filter and keyword search features. 2017 Link to Resource Guide 2 P True False False True True False False False False False False False False False False False False False False False False False False False False False False None 1 False False True False False False False False True
25 Theory of Change: Excerpt from ‘Recognizing and conducting opportunistic experiments in education’ Resch, A., Berk, J., & Akers, L. None This short excerpt illustrates how to use a theory of change as you develop your research design. 2014 Link to Resource None None None False False False False False False False False False False False False False False False False False False False False False False False False False False False None 0 False False False False False False False False False
26 Evaluation Life Cycle Armstrong, K.R.; Brandt, C. Office of Innovation and Improvement, U.S. Department of Education This infographic provides an overview of the stages of the evaluation life cycle: plan and design; identify and follow participants; collect and store data; analyze data; and report and use findings. 2017 Link to Resource Guide Brief or Summary 1 P True False False True True True True True False True False False False False False False False False False False False False False False False False False Evaluation Life Cycle; Evaluation Design; 1 False True True False False False False False True
50 90-Day Cycle Handbook Park, S.; & Takahashi, S. Carnegie Foundation for the Advancement of Teaching The Carnegie Foundation for the Advancement of Teaching developed this handbook as an introduction to 90-day, short-duration research cycles. This handbook provides a broad overview of the purpose and essential characteristics of a 90-day cycle, followed by a discussion of the processes and steps involved in conducting a 90-day cycle (post-cycle). In addition, the handbook includes tips and examples to facilitate their execution. The handbook concludes with a description of the roles and responsibilities of participants in a 90-day cycle. 2013 Link to Resource Guide 32 pages P False True False False False False False False False False False False False False False False False False False False False False False False False False False SANDRA PARK SOLA TAKAHASHI driver diagrams literature short list big picture picture goal broad overview cycle day team product phase scan process topic 3 False False True False False False False False False
51 A Composite Estimator of Effective Teaching Mihaly, K.; McCaffrey, D.; Staiger, D.; and Lockwood, J. Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this report to help educator evaluation program designers select weights to place on multiple measures of effectiveness in an effort to most accurately assess teaching quality. The report focuses on a statistical method to select optimal weights and then compares those weights to others, such as ones used in state educator evaluation systems. It also examines the extent to which optimal weights vary depending on how much data are available. This guide is a valuable resource for program designers who want to understand and replicate the statistical analyses that underlie the non-technical “Ensuring Fair and Reliable Measures of Effective Teaching” (Resource # 205). 2013 Link to Resource Methods Report 51 pages R False False False True True False True False False False False False False False True True False False True True True False True False True False False None 3 True False False False False False False False False
52 A Conceptual Framework for Studying the Sources of Variation in Program Effects Weiss, M.J.; Bloom, H.S.; & Brock, T. MDRC This MDRC paper presents a conceptual framework for designing and interpreting research on variation in program effects and the sources of this variation. The goals of the framework are to enable researchers to offer better guidance to policymakers and program operators on the conditions and practices that are associated with larger and more positive results. Throughout the paper, the authors include concrete empirical examples to illustrate points. 2013 Link to Resource Methods Report 59 pages R False False False False False True True False False False False False False False False False False False False False False False True False False False False Latino males eleventh graders prima faci Head Start Construct Possible Home Visiting Scholarship Demonstration Thomas Brock cognitive behavioral program treatment effects implementation services group contrast example variation research 3 True False False False True False False False False
53 A Guide for Monitoring District Implementation of Educator Evaluation Systems Cherasaro, T.; Yanoski, D.; & Swackhamer, L. Regional Education Laboratory Program, Institute of Education Sciences This Regional Educational Laboratory (REL) Central-developed guide walks users through a three-step process that states can use to monitor district implementation of educator evaluation systems to comply with the requirements of the Elementary and Secondary Education Act flexibility requests. The guide also offers example tools developed together with the Missouri Department of Elementary and Secondary Education that states can adapt to provide districts with greater clarity on expectations about educator evaluation systems. 2015 Link to Resource Guide 38 pages P False False False True True False False False False False False False False False False False True False False False False False False False False False False Description Early childhood Educational Laboratory NEE/MU Regional Educational adequate duration either formally goals aimed starting point Authors’ creation checkbox evaluation district teacher practice criteria student system principal 3 False False True False False False False False False
54 A Guide for Monitoring District Implementation of Educator Evaluation Systems Cherasaro, T. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Central developed this video to help professionals determine whether the associated report, The Examining Evaluator Feedback Survey, would be useful to them. The video provides an overview of the iterative process used to develop the survey, a three-step process which includes developing state guidelines that align district systems to state expectations, developing data collection methods for both policy and practice data primarily through surveys, and developing adherence criteria and review data against criteria. The video includes a description of the contents of the survey and how states and districts can use the survey. Finally, there is a short discussion about resources that are available to help those interested in using the survey. This video will be useful to program staff interested in developing their own monitoring systems or adapting the tools provided. 2016 Link to Resource Video Brief or Summary 4min P False False False True False False False False False False False False False False False False False False False False False False False False False False False None 1 False True False False False True False False False
55 A Guide to Finding Information on Studies Reviewed by the What Works Clearinghouse What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this video to help researchers, program evaluators, and program staff effectively use the WWC website. The WWC evaluates research studies that look at the effectiveness of education programs, products, policies, and practices, and generally employ randomized or quasi-experimental designs. The video goes through screen shots to help the viewer find whether a particular study of interest has been reviewed by the WWC and why (e.g., grant competition) or what studies the WWC has reviewed on a topic of interest. It provides information on study ratings, sample characteristics, statistically significant positive findings, protocols under which studies are reviewed (which can help with outcome selection), and associated products such as practice guides that describe evidence-based interventions. It describes search strategies with examples, starting at 1:05, and types of search terms that can be used (4:54). 2017 Link to Resource Video Guide 6:54 P True False False False True True True False False False False False False False False False False False False False True False False False False False False References WWC Standards, Any Education Topic, Initial Planning, RCT, QED, Reporting, Interpreting, and Using Findings 2 False False True False False True False False True
56 A Practical Approach to Continuous Improvement in Education Rodriguez, S.; Shakman, K. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast and Islands developed this webinar to help professionals considering implementing a continuous improvement process potentially as part of a network. Presenters describe continuous improvement as a way of thinking about systems change and quality improvement and its underlying six principles. Listeners for whom this approach is appropriate will then receive a set of implementation tools they can use in their own settings, such as how to define a problem, select key questions, set goals, and leverage tools such as Plan-Do-Study-Act cycles, aim statements, and fishbone and driver diagrams. The webinar provides a hands-on opportunity to practice as well as examples from school districts. This webinar will be useful for those interested in leveraging tools to help in the planning of a continuous monitoring process. 2016 Link to Resource Webinar Guide 1h21min P False True False False True False False False False False False False False False False False False False False False False False False False False False False None 3 False False True False False False True False True
57 A Practical Guide to Regression Discontinuity Jacob, R., Zhu, R., Somers, M-A., & Bloom, H. MDRC This MDRC methodology paper is intended to serve as a practitioners’ guide to implementing regression discontinuity (RD) designs. It uses an approachable language and offers best practices and general guidance to those attempting an RD analysis. This paper discusses in detail the following: 1) approaches to estimation, 2) assessing the internal validity of the design, 3) assessing the precision of an RD design, and 4) determining the generalizability of the findings. In addition, the paper illustrates the various RD techniques available to researchers and explores their strengths and weaknesses, using a simulated dataset. 2012 Link to Resource Methods Report 100 pages R False False False False False False True True False True False False False True False False False False True True True True False False False False False Internal Validity decision makers independently academic warning broadly generalizable difference observations Web site gender regression rating discontinuity data simulated ratings treatment distribution plot bin 3 False False False False True False False False False
58 A Primer for Continuous Improvement in Schools and Districts Shakman, K.; Bailey, J; Breslow, N. Teacher Incentive Fund and Teacher & School Leader Incentive Program The Teacher Incentive Fund and Teacher & School Leader Incentive Program developed this guide to orient educational practitioners to the continuous improvement process in educational settings. It discusses implementing and studying small changes and making revisions in iterative cycles with the goal of making lasting improvement. The brief offers a model for improvement that consists of three essential questions: What problem are we trying to solve? What changes might we introduce and why? How will we know that a change is actually an improvement? It shares six principles of improvement the Carnegie Foundation for the Advancement of Teaching developed for education-focused audiences (page 4). It then describes in depth the components of the Plan-Do-Study-Act (PDSA) model for improvement. The PDSA cycle provides a structure for testing a change and guides rapid learning through four steps that repeat as part of an ongoing cycle of improvement: 1) Plan: defining the problem (using the Fishbone Diagram tool in Appendix A) and establishing an aim (using the Driver Diagram tool in Appendix B); 2) Do: implementing an initiative and measuring success using “practical measures,” i.e., those that practitioners can collect, analyze, and use within their daily work lives; 3) Study: investigating the data; and 4) Act: determining next steps. The brief presents an example in the context of teacher turnover. Finally, it points the reader to additional resources. 2017 Link to Resource Guide Tool 14 P False True False True True False False False False False False False False False False True True False False False False False False False True False False None 2 True False True True False False False False True
59 Addressing Attrition Bias in Randomized Controlled Trials: Considerations for Systematic Evidence Reviews Deke, J.; Sama-Miller, E.; & Hershey, A. U.S. Department of Health and Human Services This U.S. Department of Health and Human Services paper discusses whether an attrition standard based on information from education research is appropriate for use with research that the Home Visiting Evidence of Effectiveness Review (HomVEE) examines. The paper also provides an example of how to assess whether the attrition standard for one systematic evidence review fits other systematic reviews, along with considerations for adopting or modifying the standard for alternative contexts. 2015 Link to Resource Methods Report 21 pages R True False False False False True False False False False True False True False False False False False False False False False False False False False False Controlled Trials Primary caregiver’s Randomized Controlled What Works Clearinghouse follow-up Evidence Reviews Human Services Systematic Evidence attrition bias homvee boundary standard effect program rate outcomes 3 False False False False True False False False False
60 Addressing Challenges and Concerns about Opportunistic Experiments Hartog, J. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) developed a video series that explains how schools, districts, states, and their research partners can use a cost-effective approach, known as “opportunistic experiments,” to test the effectiveness of programs. Opportunistic experiments take advantage of a planned intervention of policy change. This video describes four common concerns that come up before starting an opportunistic experiment; such as fairness, access, expense, and disruption, and three key challenges that are common once the experiment in underway; study-exempt participants, building support, and observing long-term outcomes. This video will be useful to professionals considering opportunistic experiments and using research to make decisions and improve programs. 2016 Link to Resource Video 11min P False True True False True True False False False True False False False False False False False False False False False False False False False False False None 2 False False False False False True False False False
61 An educator’s guide to questionnaire development Harlacher, J. Institute of Education Sciences This Regional Educational Laboratory Central guide designed for education administrators offers a five-step, questionnaire-development process: determining the goal(s) of the questionnaire, defining the information needed to address each goal, writing the questions, reviewing the questionnaire for alignment with goals and adherence to research-based guidelines for writing questions, and organizing and formatting the questionnaire. 2016 Link to Resource Tool 22 pages P False False False True False False False False False False False False False False False True True False False False False False False True True True False New York emotionally charged released publicly Authors compilation Public Instruction Retrieved September Sage Publications Thousand Oaks personality traits mutually exclusive questions questionnaire information respondents example question response responses data may 3 True False False True False False False False False
62 An Environmental Scan of Tools and Strategies that Measure Progress in School Reform Griffin, P.; Woods, K.; & Nguyen, C. Victorian (AU) Department of Education and Training The Victorian (AU) Department of Education and Training developed this guide to help professionals evaluate school reform. The guide begins with an overview of tools and methods for measuring the progress and impact of school reform initiatives in the U.S. and other countries. It reviews the trend for “evidence- based” reform in education and discusses the importance of rigor in the design of evaluation studies, sampling and data collection procedures, analysis and interpretation of evidence. After focusing on what not to do, the guide switches to providing detailed guidelines for evaluation that can be applied to a range of reform programs. One section discusses the evaluation of the implementation, processes, and outcomes of learning standards in Victoria. The final section briefly describes three case studies in which measurement and monitoring of student outcomes have been incorporated into program evaluation. This guide is a valuable resource for evaluators who need to identify appropriate indicators of successful implementation, including student academic outcomes. 2005 Link to Resource Guide 87 Pages P False True False False False False False False False False False False False False False True False False True True True False False False False False False Manpower Bureau Environmental Scan General Accounting Primary Native Flagship Strategy Family Trust Fischer Family Quality Assurance Development Branch Native English-speaking reform outcomes evaluation data indicators section factors implementation students report 3 False False True False False False False False False
63 An Evaluation of Bias in the 2007 National Households Education Surveys Program: Results From a Special Data Collection Effort Van de Kerckhove, W.; Montaquila, J.M.; Carver, P. R.; Brick, J.M. National Center for Education Statistics, U.S. Department of Education The National Center for Education Statistics developed this report to examine bias in estimates from the 2007 National Household Education Surveys Program due to nonresponse from both refusals and noncontact cases, as well as bias due to non-coverage of households that only had cell phones and households without any telephones. The most relevant section for this data base are appendices A through F which provide advance, refusal, and community Letters, as well as four main tools for conducting in-person follow-ups: a household folder, an interviewer observation form, a “sorry I missed you” card, an appointment card, and a non-interview report form. 2008 Link to Resource Tool 53 pages P False False False False True False False False False False False True False False False False True False False False False False False False False False False Bias Analysis; Interviews by Telephone; Random Digit Dialing; Response Rates 3 False False False True False False False False False
64 An Evaluation of the Data From the Teacher Compensation Survey: School Year 2006–07 Cornman. S.Q.; Johnson. F.; Zhou, L; Honegger,S.; Noel, A.M. National Center for Education Statistics, U.S. Department of Education The National Center for Education Statistics developed this report to provide researchers with compensation, teacher status, and demographic data about all public school teachers from multiple states. Three sections are of use. Section 2 describes the data elements the survey collects. Section 4 describes data quality and section 6 survey limitations. The link to access the data is at the bottom of page 55. This survey is a valuable resource for researchers who want to analyze teacher compensation data and are concerned with the sampling error and self-reporting bias present in sample surveys, as this dataset contains universe data at the teacher level for multiple states. Sections 4 and 6 are also useful for program directors and evaluators who want to assess the quality and completeness of data by providing examples of data quality checks and red flags. 2009 Link to Resource Guide Tool 10 pages P False False False False True False False False False False False False False False False False True False False True False False False False True False False None 2 True False True True False False False False False
65 An SEA Guide for Identifying Evidence-Based Interventions for School Improvement Lee, L.; Hughes, J.; Smith, K;. Foorman, B. Florida Center for Reading Research, Florida State University The Florida Center for Reading Research developed this guide to help state education agencies consider the evidence supporting intervention options. Of particular use for evaluation purposes, the guide describes levels of evidence as defined by ESSA and informed by WWC on pages 11-18. The guide’s main focus is to describe the steps of a collaborative self-study process to identify interventions. Self-study is a process that facilitates thoughtful investigation and discussion of an issue or topic so that decisions can be made collaboratively. The guide also provides a wealth of tools, including role definitions, an intervention scoring template, a scoring guide, a consensus rating form, a planning form for providing guidance to districts, and a sample logic model. Individual scoring guides are provided in a range of areas — systemic change, leadership, instruction, staff development and retention, and school climate. 2016 Link to Resource Guide Tool 89 pages P True False False True True False False False False False False False False False False False False False False False False False False False False False False FLORIDA CENTER, Florida Department of Education, Grant Foundation, Casey Foundation, Self-Study Guide, FLORIDA STATE UNIVERSITY, Overdeck Family Foundation, SEA Guide, Kevin Smith, READING RESEARCH, School Improvement, Self-Study Process, Tennette Smith, Barbara Foorman, John Hughes, Laurie Lee, Identifying Evidence-Based Interventions, UPDATED NOVEMBER, Funding Foundations, Sandra Dilger, Holly Edenfield, Jason Graham, Shannon Houston, Eileen McDaniel, Sonya Morris, Melissa Ramsey, Michael Stowell, Jennifer Morrison, Kim Benton, Robin Lemonis, Roy Stehle, Nathan Oakley, Sonja Robertson, Table of Contents, Mississippi Department of Education, Responsibility, Roles, South Carolina Department of Education, Annie, Flexibility, Acknowledgments, iii, Introduction 3 False False True True False False False False True
66 Analysis Models and Baseline Equivalence For Impact Studies Price, C.; Wolf, A. Abt Associates This video provides guidance on the selection of analysis models for estimating impacts and assessing baseline equivalence in evaluations of education interventions. It uses practical examples from K-12 and postsecondary education. Specific analysis models are presented for common study designs along with a discussion of individual vs. cluster assignment, blocking, and inferences to a sample vs. generalization to a population. The video is also an orientation to a set of materials including a set of example data sets and programs that can be modified for use with SAS, Stata, SPSS, and R to meet the specific needs of an evaluation. See the “Analysis Models and Baseline Equivalence for Impact Studies: Guidance, Example Data Sets and Programs for SAS, Stata, SPSS, and R Users” resource in this database for these materials. This video is a valuable resource for evaluators who plan, conduct, and report results from rigorous impact evaluations of the sort that can meet WWC standards with or without reservations. 2018 Link to Resource Video Guide 47:40 R True False False False True True True False False True False False False False False False False False True True False False True False False False False None 3 True False True False False True False False True
67 Analysis Models and Baseline Equivalence For Impact Studies: Guidance, Example Data Sets and Programs for SAS, Stata, SPSS, and R Users Boulay, B.; Miller, H.; Price, C.; Wolf, A. Abt Associates This link provides a package of materials to support the selection of analysis models for estimating impacts and assessing baseline equivalence in evaluations of education interventions. It includes: a two-page overview of the contents of the zip file (000_Read Me First.pdf); 56-page long PowerPoint and PDF versions of a video (called webinar) that discusses analysis models for estimating impacts and assessing baseline equivalence, specifically using examples within postsecondary education (the video can be downloaded from the same link; a 28-page practical guide for evaluators on establishing baseline equivalence (or the similarity of treatment and comparison groups at the beginning of an evaluations so intervention effects can be attributed to the intervention); a 32-page guide to help choose an appropriate analysis model for evaluations using a group design—either a randomized controlled trial (RCT) or a quasi-experimental design (QED)—to evaluate the effectiveness of an intervention; and five data sets and programs that can be modified for use with SAS, Stata, SPSS, and R to meet the specific needs of an evaluation. This packet of materials is a valuable resource for evaluators who plan, conduct, and report results from rigorous impact evaluations that can meet WWC standards with or without reservations. (Note: In case of difficulty opening a file, rename the downloaded zip file with a shorter title e.g., Analysis Models.zip) 2017 Link to Resource Guide Tool 176 R True False False False True True True False False True False False False False False False False False True True False False True False False False False General Design Guidance, QED Tools and Resources, RCT Tools and Resources, Guidance 3 True False True True False False False False True
68 Analyzing Teacher Retention by Performance Level and School Need Examples From Maricopa County Nicotera, A.; Pepper, M.; Springer, J.; Milanowski, A. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to provide district staff with simple analyses to help identify teacher retention challenges. The body of the guide goes through various measures and presents results for one school district in user-friendly graphics. The appendices describe the data, measures, and analyses. This guide is a valuable resource for district research and human resources staff who want to understand how to understand and address teacher retention patterns. 2017 Link to Resource Guide 13 pages P False False False True True False False False False False False False False False True True False False False False True False False False True True False None 2 True False True False False False False False False
69 Applying the WWC Standards to Postsecondary Research What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this webinar to help researchers conduct effectiveness evaluations in postsecondary settings. The facilitators provide an overview of the WWC and discuss how the WWC determines study ratings, with special considerations for postsecondary settings. They recall eligible designs (randomized control trials and some quasi-experimental designs) and their potential to meet WWC standards. They also explain how certain aspects of a study, such as attrition, group comparability, and outcome measurements, can impact the WWC’s rating of a study. This webinar provides guidance on common, but frequently avoidable, challenges in postsecondary research that prevent studies from meeting standards. This webinar will be helpful for researchers who want to know what to look for in strong studies in postsecondary research. 2016 Link to Resource Video Guide 37 min R True False False False False True True True False True False False True True False True False False False False True False False False False False False None 3 False False True False False True False False False
70 Approaches to Evaluating Teacher Preparation Programs in Seven States Meyer, S.J.; Brodersen, R.M.; & Linick, M.A. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory Central conducted this qualitative study to inform policymakers, researchers, educators, and others who are concerned with the quality of preparing teachers for their profession. The report focuses on how seven central states evaluate their teacher-preparation programs and the changes they are making to improve the states’ approaches to evaluation. Most changes involve paying more attention to the performance of program graduates, developing common data-collection tools and data systems, and developing new ways to report evaluation data. This report is a valuable resource for policymakers and practitioners and provides a good example of a qualitative research study that can inform decision-making. 2014 Link to Resource Guide 40 pages P False True False True False False False False False False False True False False False False True False False True True False False False True False False Colorado Public Instruction American Progress Washington standards bachelors degree on-site visit lack formal count multiple teacher programs state preparation program education evaluation data states department 3 True False True False False False False False False
71 Approaches to TIF Program Evaluation: Cost-Effectiveness Analysis Finster, M. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to help program directors and evaluators understand cost effectiveness analysis. The one-page guide focuses on a simple, concrete example; then lists the question this type of analysis answers, the data required, how it works, and how to use results. This guide is a valuable resource for practitioners who must choose among different programs. It is also useful for evaluators in the early stages of planning an evaluation. 2016 Link to Resource Guide 1 page P False False False True True False False False False False False False False False False False False False True False False False False False False False False None 1 False False True False False False False False True
72 Approaches to TIF Program Evaluation: Difference in Differences Design Finster, M. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to help program directors and evaluators understand difference-in-differences designs. The one-page guide focuses on a simple, concrete example; then lists the question this type of analysis answers, the data required, how it works, and how to use results. This guide is a valuable resource for program directors who want to measure the impact of programs by comparing the difference in outcomes between the intervention and comparison groups before and after the intervention group experienced the intervention. It is also useful for evaluators in the early stages of planning an evaluation. 2016 Link to Resource Guide 1 page P False False False True True False True False False False False False False False False False False False True False False False False False False False False None 1 False False True False False False False False True
73 Approaches to TIF Program Evaluation: Interrupted Time-Series (ITS) Designs Finster, M. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to help program directors and evaluators understand interrupted time series designs. The one-page guide focuses on a simple, concrete example; then lists the question this type of analysis answers, the data required, how it works, and how to use results. This guide is a valuable resource for program directors who want to measure the impact of programs by comparing multiple observations on an outcome before and after an intervention. It is also useful for evaluators in the early stages of planning an evaluation. 2016 Link to Resource Guide 1 page P False False False True True False True False False False False False False False False False False False True False False False False False False False False None 1 False False True False False False False False True
74 Approaches to TIF Program Evaluation: Regression Discontinuity Design Finster, M. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to help program directors and evaluators understand regression discontinuity designs. The one-page guide focuses on a simple, concrete example; then lists the question this type of analysis answers, the data required, how it works, and how to use results. This guide is a valuable resource for program directors who want to measure the impact of programs by comparing outcomes of similar participants on either side of a cut-off point that determines whether the participant participates. It is also useful for evaluators in the early stages of planning an evaluation. 2016 Link to Resource Guide 1 page P False False False True True False False True False True False False False False False False False False True False False False False False False False False None 1 False False True False False False False False True
75 Asking Students About Teaching: Student Perception Surveys and Their Implementation – Policy and Practice Brief Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help program directors use student perception surveys as part of educator evaluations. The guide describes how to measure what matters — what teachers do, the learning environment they create, the theory of instruction that defines expectations for teachers in a system; how to ensure accuracy in student responses; how to ensure reliability with adequate sampling and items number; how to support improvement in teaching; and how to engage stakeholders. The guide provides real-life examples, tools, and graphics. This guide is a valuable resource for program directors and evaluators who want to leverage student perception as part of program evaluations. 2012 Link to Resource Guide 4 pages P False False False True True False False False False False False False False False False False True False False False False True False False False False False None 1 False False True False False False False False False
76 Attrition Knowledge Checks: Module 2, Chapter 5, WWC Training What Works Clearinghouse Institute of Education Sciences This video provides examples of study design decisions related to IES standards on attrition. Viewers are provided time to choose the best answer before the correct answer is provided. Reasons for answers are well described. 2016 Link to Resource Video 6 min P True False False False True True False False False False False False True False False False False False False False False False False False False False False None 2 False False False False False True False False False
77 Attrition Thresholds: Module 2, Chapter 3, WWC Training What Works Clearinghouse Institute of Education Sciences The video discusses WWC’s attrition thresholds, distinguishing between liberal threshold and conservative threshold. The video also points the viewer to additional WWC resources to calculate and understand the attrition thresholds. 2016 Link to Resource Video 8 min P True False False False True True False False False False False False True False False False False False False False False False False False False False False None 2 False False False False False True False False False
78 Baccalaureate and Beyond Longitudinal Study (B&B:08/09) Field Test Methodology Report – Working Paper Series Wine. J.; Cominole, M.; Janson, N.; Socha, T.; National Center for Education Statistics, U.S. Department of Education The National Center for Education Statistics developed this report to describe the methodology and findings of the Baccalaureate and Beyond Longitudinal study. The relevant section for this data base is appendix C which provides examples of materials related to recruiting and maintaining participants, including parent letters, a data collection announcement letter and flyer, and thank you letters. 2015 Link to Resource Guide Tool 284 pages P False False False False True False False False False False False True False False False False True False False False False False True False False False False None 3 True False True True False False False False False
79 Baseline Equivalence Knowledge Checks: Module 3, Chapter 5, WWC Training What Works Clearinghouse Institute of Education Sciences This video includes a knowledge check on how to define baseline equivalence, when it’s appropriate to use statistical adjustments, and in what baseline conditions can studies receive the ‘meets standards with reservations’ rating. 2016 Link to Resource Video 10 min R True False False False True True True False False True False False False True False False False False False False False False False False False False False None 2 False False False False False True False False False
80 Benchmarking Education Management Information Systems Across the Federated States of Micronesia Cicchinelli, L. F.; Kendall, J.; Dandapani, N. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Pacific developed this report to provide information on the current quality of the education management information system (EMIS) in Yap, Federated States of Micronesia, so that data specialists, administrators, and policy makers might identify areas for improvement. The relevant sections of the report are in the appendices: data sources and methodology (appendix B) and data collection and instruments (appendix C). These sections are valuable resource for program directors who want to assess the quality of their education management information system (EMIS) or who want to see an example of a data collection effort that attempts to quantify answers from interviews. In particular, this helps ensure that education policy, planning, and strategy decisions are grounded in accurate information. 2016 Link to Resource Guide Tool 34 Pages P False False False False True False False False False False False False False False False False True False False False False False False False False False False education management information system, aspects of system quality, Accessibility of education data, overall system, system overall, reliability of education statistics, Department of Education, Integrity of education statistics, aspects of quality, Institute of Education Sciences, data reporting, scores, place, t i o, Data specialists, r t m e n t o f E d u c, state of Yap, D e p, John, Prerequisites of quality, King, benchmark levels, relevance, timeliness, Deputy Director, Policy, Acting Secretary, example of best practice, McREL International, Key findings, Louis, Cicchinelli, Nitara Dandapani, Delegated Duties, Ruth Neild, Serviceability, consistency, Federated States of Micronesia, indicators, rubric, Research, institutional frameworks, process of implementation, Accuracy, supporting resources, stakeholders, meeting standards, Kendall, follows, February 3 False False True True False False False False False
81 Better Feedback for Better Teaching: A Practical Guide to Improving Classroom Observations Archer, J.; Cantrell, S.; Holtzman, S..; Joe, J..; Tocci, C.; Wood, J. Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide focuses on the components of high-quality evaluator training including priorities, delivery methods, the creation of training videos, the effective use of an observation rubric, and data use for continuous improvement. It also provides a series of tools from states and districts to support each step of the process. This guide is a valuable resource for program directors who want to understand how to design and implement a high quality evaluation. It is also useful for evaluators of educator evaluation systems. 2016 Link to Resource Guide Tool 364 pages P False False False True True False False False False False False False False False False False True False False False True False False False True False False None 3 True False True True False False False False True
82 Building Trust in Observations: A Blueprint for Improving Systems to Support Great Teaching [Policy and Practice Brief] Wood, J.; Tocci, C.; Joe, K.; Holtzman, S.; Cantrell, S; Archer, J. Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide focuses on the components of observation systems that help build trust – observation rubric, evaluator training, observer assessment, and monitoring. Each component comes with a specific set of action steps in a user-friendly table format. This guide is a valuable resource for program directors who want to design and implement a high quality observation system with an eye towards building trust and buy-in. 2014 Link to Resource Guide Tool 28 pages P False False False True True False False False False False False False False False False False True False False False True False False False True False False None 3 True False True True False False False False True
83 Causal Inference and the Comparative Interrupted Time Series Design: Findings from Within-Study Comparisons St. Clair, T.; Hallberg, K.; Cook, T.D. Society for Research on Educational Effectiveness This resource is designed to help researchers understand the circumstances under which a comparative interrupted time series (CITS) design produces estimates that are comparable to those from a randomized controlled trial (RCT). In a simple interrupted time series design, researchers compare the pre-treatment values of a treatment group time series to post-treatment values in order to assess the impact of a treatment, without any comparison group to account for confounding factors. In the CITS design, a treatment and a comparison group are evaluated before and after the onset of a treatment. The authors employ two datasets to determine whether the treatment and comparison group pretreatment time series gets estimates comparable to those from an RCT. They find very different results with the two datasets. They consider multiple approaches to correctly model pre-treatment trends as the treatment and comparison groups may have different slopes in the pretreatment period, and which approach to select may depend on how much data are available. 2013 Link to Resource Brief or summary Methods report 14 pages R False False False False True True True False False False False False False False False False False False True True False False True False False False False Measurement Techniques, Time, Randomized Controlled Trials, Comparative Analysis, Performance, Benchmarking, Grade 4, Grade 5, Grade 6, Scores, Achievement Tests, English, Language Arts, Mathematics 2 True True False False True False False False False
84 Challenges and Strategies for Assessing Specialized Knowledge for Teaching Orrill, C.H.; Kim, O.K.; Peters, S.A.; Lischka, A.E.; Jong, C.; Sanchez, W.B.; & Eli, J.A. Mathematics Teacher Education and Development Journal The authors of this Journal of Mathematics Teacher Education article wrote it to help policy makers, grant-funding agencies, mathematics teachers, and others to understand the knowledge necessary for effective teaching of mathematics and how to assess it. The guide begins with an overview of what is known about measuring teachers’ knowledge. It then highlights the challenges inherent in creating assessment items that focus specifically on measuring teachers’ specialized knowledge for teaching: creating items with appropriate difficulty levels, creating items for the target constructs, using precise language, incorporating pedagogical concerns appropriately, and writing clear stems and distracters. The article also offers insights into three practices that the authors have found valuable in their own work: creating an interdisciplinary team to write the assessment, using interviews to understand whether an item is measuring the appropriate construct and whether it is at an appropriate level of difficulty by providing insight into participants’ thinking, and adopting an iterative process to item writing. This guide is a valuable resource for those who create assessment items for measuring teachers’ knowledge for teaching. 2015 Link to Resource Guide 18 pages P False False False True False False False False False False False False False False False True False False False False False False False False True False False Teacher Knowledge, Mathematics’ Teacher Knowledge, Measurement, Item Development, TEDS-M Testing Service United States high-quality Ferrini-Mundy subject matter rational numbers concerns appropriately story problem Elementary School knowledge teacher mathematics item items teachers teaching teachers’ assessment education 2 True False True False False False False False False
85 Checklist for Assessing USAID Evaluation Reports USAID USAID The U.S. Agency for International Development Evaluation Report Checklist helps evaluators review and strengthen draft evaluation reports. The report details 15 critical factors (noted in boldface type) that should be addressed in early drafts of the evaluation report. The authors recommend assessing the final report against all 76 factors to ensure high technical quality, a strong executive summary, and the targeting of recommendations for decision-making purposes. 2013 Link to Resource Tool 7 pages P False False False False False False False False False False False False False False False False False False False False False False False False False False False evaluation report project data information questions include evaluation report evaluation questions data collection report identify evaluation team collection instruments cost structure evaluation design 2 False False False True False False False False False
86 Checklist For Reviewing an RCT of a Social Program or Project, To Assess Whether It Produced Valid Evidence Coalition for Evidence-Based Policy Coalition for Evidence-Based Policy The Coalition for Evidence-Based Policy developed this tool to identify key items to help researchers assess whether the results of a randomized controlled trial of a social program, project, or strategy (“intervention”) produced valid evidence on the intervention’s effectiveness. The tool is a (non-exhaustive) checklist that addresses designing a study, gauging the equivalence of the intervention and control groups, selecting outcome measures, reporting the intervention’s effects, and determining whether there is enough evidence to assert that the intervention is effective. This checklist is a valuable aid, but good judgment may still be needed to gauge, for example, whether a deviation from one or more checklist items is serious enough to undermine the study’s findings. 2010 Link to Resource Tool 8 pages R True False False False False True False False False True True False False True False True False False False False True False False False False False False intervention study group groups program effects randomized sample control outcomes illustrative examples controlled trial random assignment randomized controlled randomly assign housing projects WWC stamdards 2 False False False True False False False False False
87 City Connects: Building an Argument for Effects on Student Achievement with a Quasi-Experimental Design Walsh, M.; Raczek, A.; Sibley, E.; Lee-St. John, T.; An, C.; Akbayin, B.; Dearing, E.; Foley, C. Society for Research on Educational Effectiveness This Society for Research on Educational Effectiveness (SREE) report shares the results from three analyses that provide different pieces of evidence for a causal relationship between a City Connects intervention and student achievement. The authors provide an overview of a multi-year effort for the evaluation of a large-scale, school-based intervention for which there are rich, longitudinal data and many opportunities to exploit design features to ask: does the intervention promote the achievement of children in high-poverty, urban schools? The authors recommended that other school-based interventions that cannot feasibly use a randomized control design instead employ a range of methods aimed at reducing endogeneity and getting closer to understanding whether a causal relationship might exist between the intervention and its outcomes. 2015 Link to Resource Brief or Summary 6 pages R False False False False False False True True False True False False False True False False False False False True False False True False False False False Academic Achievement, Quasiexperimental Design, Intervention, Elementary School Students, Urban Schools, Low Income Students, Longitudinal Studies, Program Effectiveness, Hierarchical Linear Modeling, Regression (Statistics), Student Support Not included causal inference interrupted time 2015 Conference SREE Spring Spring 2015 report card Abstract Template city connects students school intervention student schools comparison effects outcomes 2 True True False False False False False False False
88 Classroom Assessment for Student Learning: Impact on Elementary School Mathematics in the Central Region Randel, B.; Beesley, A. D.; Apthorp, H., Clark, T.F.; Wang, X.; Cicchinelli, L. F.; Williams, J. M. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Central developed this report to provide evaluators with a detailed description of an implementation and impact evaluation of a professional development program in classroom and formative assessment. The report describes the intervention, study design and implementation, implementation fidelity, and impact on student and teacher outcomes. In the process it raises a number of challenges and limitations such as attrition, non-response, missing data, inability to detect very small effects, and generalizability. Appendices provide a rich level of detail as well as data collection instruments This report is a valuable resource for program evaluators who seek an example of a program evaluation that addresses both implementation and impact and that examines a range of threats to the reliability and validity of results. 2010 Link to Resource Methods report Tool 153 pages R False False False True True True False False False True True True True True False False True False True True True False True False True False False None 3 True False False True True False False False False
89 Cluster Assignment Studies: Data Collection and Reporting Issues Price, C. Abt Associates This slide presentation introduces researchers to What Works Clearinghouse revised cluster standards. It explains the concepts of risk of bias from individuals who enter the intervention after the study has begun. The PPT also addresses: (1) how to anticipate what data elements to consider collecting and reporting on for impact studies; (2) clusters and individuals as units of assignment; (3) causal inferences regarding impacts on individuals and clusters; (4) clear sequencing of questions and conditions that lead to determining whether revised standards make a difference; (5) stayers and early and late joiners and the risk of bias from individuals entering clusters after randomization. This slide presentation is a valuable resource for researchers planning and implementing randomized controlled trials or quasi-experimental studies. 2017 Link to Resource Slide Presentation 64 R True False False False True True True False False True False False True True False False False False False False False False True False False False False Implementation Study Tools and Resources 3 True False False False False False False True True
90 College Enrollment Patterns for Rural Indiana High School Graduates Burke, M. R.; Davis, E.; Stephan, J. L. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Midwest developed this report to describe the findings from a descriptive study of rural and non-rural differences in college enrollment patterns among Indiana’s 2010 public high school graduates enrolling in Indiana public colleges. The relevant sections of the report are the limitations section on pages 18-19 and Appendix B on data and methodology. Limitations include incomplete data, restricted ability to generalize, and inability to make causal inferences. Appendix B describes data sources, analytic sample description, missing data handling, variable creation, descriptive statistics, statistical tests, mapping analyses, a rubric on college selectivity, and regression models. These sections are a valuable resource for program directors and evaluators who want to learn or review a range of analyses that can be conducted with administrative data. 2016 Link to Resource Methods report 16 pages R False False False False True False True False False False False False False False True True False False True True True False True False False True False None 2 True False False False True False False False False
91 Common Circumstances in WWC Reviews: Module 4, Chapter 3, WWC Training What Works Clearinghouse Institute of Education Sciences This video explains that for a study to be judged as having a confounding factor, all three conditions of confounding factors must hold: (1) the study component must be observed (2) the component is aligned completely with only of the study conditions, and the component is not part of the intervention that the study is testing. If any of these conditions do not apply, then WWC defines it as a ‘non-confounding factor’ 2016 Link to Resource Video 6 min P True False False False True False False False False False False False False True False False False False False False False False False False False False False None 2 False False False False False True False False False
92 Compliance-Effect Correlation Bias in Instrumental Variables Estimators Reardon, S. F. SREE Spring 2010 Conference Abstract The study author developed this brief (SREE abstract) to help researchers’ appropriately use instrumental variables estimators to identify the effects of mediators in multi-site randomized trials under conditions of heterogeneous compliance. The brief describes how and when instrumental variables can be used under these conditions, provides a stylized example of a randomized trial investigating the impact of teacher professional development on student achievement, presents the structural model and findings, and concludes with the potentially substantial bias misuse of these estimators can yield. 2010 Link to Resource Methods report Brief 8 pages R False False False False False True False False False False False False False True False False False False True True False False False False False False False Social Science Research, Least Squares Statistics, Computation, Correlation, Educational Research, Statistical Bias, Sampling, Research Design, Evaluation Methods, Intervention, Research Methodology, Research Problems, Measurement 2 False True False False True False False False False
93 Conducting Implementation-Informed Evaluations: Practical Applications and Lessons from Implementation Science REL Mid-Atlantic Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Mid-Atlantic developed this webinar to help districts and schools define new initiatives and measure changes needed to ensure that new skills are learned and used. It begins with a reminder of why implementation is difficult and which three components are required for educational outcomes to change. The focus then turns to two active-implementation frameworks to change a system and effectively implement evidence-based practices. The first is implementation drivers: the critical supports needed to make change happen and what the infrastructure is. The second is improvement cycles: how we can create more hospitable environments, efficiently solve problems, and get better. The webinar also explores practical applications and examples of an implementation-informed, decision-support-data system, including fidelity assessments. It is a valuable resource for educators who want to identify, implement, and evaluate initiatives with a high level of fidelity and meet the goal of improving student outcomes. The webinar is useful for identifying how improvement cycles (rapid-cycle problem solving and usability testing) can be used to put effective practices into operation. 2015 Link to Resource Webinar Slide Presentation 71 slides, 87 minutes P False True False False False False False False False False False False False False False True True False False False False False False False False False False Administration Decision Support Data System Change Integrated Improved Student Infrastructure Improved Intervention Facilitative Administration Decision Support Selection Systems Intervention Facilitative Coaching Training Selection Systems Regional Educational next set Division H 3 False False False False False False True True False
94 Confounding Factors Knowledge Check: Module 4, Chapter 4, WWC Training What Works Clearinghouse Institute of Education Sciences The video provides scenarios of different studies, and the viewer is asked to determine if the scenario illustrates a study with confounding factors. The video carefully explains each correct answer. 2016 Link to Resource Video 6 min P True False False False True False False False False False False False False True False False False False False False False False False False False False False None 2 False False False False False True False False False
95 Content Knowledge for Teaching and the MET Project – Report Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement educator evaluation systems that use multiple measures of effectiveness. The guide focuses on pedagogical content knowledge, a subject-specific professional knowledge that bridges content knowledge and knowledge about the practice of teaching. The guide introduces assessments of content knowledge for teaching mathematics and English language arts. These assessments can be administered to teachers in order to establish the links among content knowledge for teaching, instructional practice, and student achievement. This guide is a valuable resource for program directors who want to design and implement a high quality educator evaluation system. 2010 Link to Resource Guide 4 pages P False False False True True False False False False False False False False False False False True False False False False False False False True False False None 1 True False True False False False False False False
96 Continuous Improvement in Education Excerpt 1: The Model for Improvement REL program Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a webinar series to help professionals put in place a process to implement and continually monitor programs. This first excerpt introduces the three essential questions that guide the model for improvement: what problem are we trying to solve, what changes may we introduce and why, and how will we know that the change is actual improvement. It provides an example in an education setting. This video will be helpful to program staff interested in implementing a continuous improvement process and wanting a very basic introduction. 2016 Link to Resource Video Webinar 2 min P False True False False True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False False False True True False False
97 Continuous Improvement in Education Excerpt 2: The Plan Do Study Act (PDSA) Cycle REL Northeast & Islands Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a webinar series to help professionals put in place a process to implement and continually monitor programs. This second excerpt provides a brief overview of the Plan-Do-Study-Act process, a four-step approach to systematically implement, monitor, and improve programs. It mentions tools that help in its implementation and provides an example in an education setting. This video will be helpful to program staff interested in implementing a continuous improvement process and wanting a very basic introduction. 2016 Link to Resource Video Tool 3 min P False True False False True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False True False True False False True
98 Continuous Improvement in Education Excerpt 3: Fishbone Diagram REL Northeast & Islands Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a webinar series to help professionals put in place a process to continually monitor programs. This third excerpt introduces the participants to the fishbone diagram, a tool that helps identify the causes of a problem. It illustrates its use with an example. This video will be helpful to program staff interested in implementing a continuous improvement process and anyone looking to explore several layers of causes of an issue. 2016 Link to Resource Video Tool 3 min P False True False False False False False False False False False False False False False False False False False False False False False False False False False None 1 False False False True False True False False False
99 Continuous Improvement in Education Excerpt 4: Practical Measures and the Driver Diagram REL Northeast & Islands Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a webinar series to help professionals put in place a process to implement and continually monitor programs. This fourth excerpt provides guidance on selecting process and outcome measures with a focus on measures that are useful to practitioners and embedded in existing practice. It introduces a tool that can be used to identify measures – the driver diagram – and illustrates its use with examples. This video will be helpful to program staff interested in implementing a continuous improvement process and anyone looking to identify a sequence of testable items that have an effect on outcomes. 2016 Link to Resource Video Tool 5 min P False True False False False False False False False False False False False False False True False False False False False False False False False False False None 1 False False False True False True False False True
100 Continuous Improvement in Education Excerpt 5: Practitioner Testimonial REL Northeast & Islands Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a webinar series to help professionals put in place a process to implement and continually monitor programs. This fifth excerpt features a principal who discusses her school’s use of continuous improvement to improve math discourse. She notes the types of data teachers collected, the challenges they encountered, and how they addressed them through multiple Plan-Do-Study-Act cycles. This video will be helpful to program staff considering implementing a continuous improvement process and who could benefit from a real-life example and how the experience was valuable. 2016 Link to Resource Video Webinar 4 min P False True False False True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False False False True True False False
101 Contrast Tool Goodson. B.; Price, C.; Wolf, A.; Boulay, B. Abt Associates Abt Associates developed this tool to help researchers design program evaluations. This “contrast tool” is an Excel spreadsheet designed to support and supplement the development of an evaluation plan. It lists each impact that an evaluation will estimate to test program effectiveness. Individual worksheets can be used to enter information on research questions, outcome measures, baseline measures, samples, and contrasts (i.e., impacts) to be tested. Four worksheets can be used for reporting results: impacts, attrition (for RCTs only), and Baseline Equivalence (for RCTs with high attrition and quasi-experimental designs), and representativeness at baseline. There is an instructions tab for each worksheet. This tool is a valuable resource for evaluators who want to align planned analyses with research questions and to outline key information about the plan’s outcome measures, baseline measures, and analytic samples. The database also includes two examples (https://ies.ed.gov/ncee/projects/xls/ExContrastToolStudentRCT.xlsm and https://ies.ed.gov/ncee/projects/xls/ExContrastToolQED.xlsm) that can help complete the template. This resource can be used in combination with the evaluation plan template also included in this database (https://ies.ed.gov/ncee/projects/pdf/EvaluationPlanTemplate.pdf). 2017 Link to Resource Tool 24 R True False False False False True True False False False False False False False False False False False False False False False False False False False False None 3 False False False True False False False False True
102 Cost-Analysis Methods: What Are These Methods & How Can School Districts Benefit From Using Them? DeCesare, D.; Fermanich, M. Institute of Education Sciences The Institute for Education Sciences developed this webinar to introduce professionals to cost analyses in education research. The webinar focuses on three key types of cost analyses which are valid and answer different questions: cost effectiveness, cost feasibility, and cost benefit. The first eleven minutes of the webinar provide introductions and an agenda for the session; recall that education research has focused on the effectiveness of interventions more than on their costs and benefits and why it is important to consider costs in tight fiscal environments; point interested participants to the Center for Benefit Cost Studies at Columbia University for resources and training; and mention cost-utility analyses, an approach this webinar does not cover. At about 11:00, the webinar goes in depth into cost-effectiveness analyses, which are used to compare alternative programs. A discussion cost-feasibility analysis begins at about 22:25. This is the method that is employed when cost is the only or primary decision factor. At 24:10, the webinar moves on to a discussion of cost-benefit analysis, which is used to identify the cost of a program and the range of its benefits. Pros and cons of each type of analyses are discussed and hypothetical examples provided (a series begins at about 32:00). At about 50:00, the webinar introduces the ingredients method for identifying all components of a program and their cost. At 58:46, the webinar goes into three methods to measure benefits: experimental or quasi-experimental or correlational studies (58:56), self-reported willingness to pay (59:48), and comparable marketplace services (60:08). More examples begin at about 61:00. A summary starts just before 64:00 before the webinar moves on to a real-world example at 67:20. The webinar concludes starting at 86:48 with a short Q&A. 2017 Link to Resource Webinar Guide 1:29:33 P False False False False True True True False False False False False False False False False True False False False False False True False True False False None 3 True False True False False False True False True
103 Danielson’s Framework for Teaching for Classroom Observations Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide presents the elements that constitute the Danielson framework, a research-based protocol to evaluate math and English language arts lessons in all grades. This guide is a valuable resource for program directors who want to select a teacher observation tool. 2010 Link to Resource Guide 4 pages P False False False True True False False False False False False False False False False False True False False False False False False False True False False None 1 True False True False False False False False False
104 Data Collection and Use in Early Childhood Education Programs: Evidence from the Northeast Region Zweig, J.; Irwin, C. W.; Kook, J. F.; Cox, J. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast and Islands developed this report to describe preschools’ collection and use of data on early learning outcomes, the amount of time children spend in early childhood education, and classroom quality. The relevant sections of the report are the limitations section on page 18 and the appendices. Limitations include selection bias, small sample size, and missing data. Appendix A describes the sample, recruitment strategy, interview protocol, and procedures for analyzing the interviews. Appendix B describes the sample, variables, and methodology used to analyze the data (descriptive statistics and statistical tests), including how missing data were handled. Figure C1 in Appendix C illustrates the process used to combine data from multiple sources. Appendices D and E provide interview protocols. These sections are a valuable resource for program directors and evaluators who want to learn as much as possible from a limited sample. 2015 Link to Resource Methods report Tool 18 pages R False False False False True False False False False False True True False False True True True True False True False False True False True False False None 2 True False False True True False False False False
105 Data Quality Essentials – Guide to Implementation: Resources for Applied Practice Watson, J.; Kraemer, S.; Thorn, C. Center for Educator Compensation Reform The Center for Educator Compensation Reform published this article to help school systems identify, address, and plan for data quality problems before performance decisions are put under scrutiny of system stakeholders. The goal of this article is to identify the dimensions of data quality from a compensation reform perspective, provide Teacher Incentive Fund (TIF) project leaders with a data quality focus, and describe common data quality problems and solutions. The second goal is to help TIF leaders focus their attention on the quality of student-teacher linkage data. The article concludes with a brief discussion of continuous improvement methods through follow-up and feedback cycles for teachers. 2016 Link to Resource Guide 18 pages P False False False False True False False False False False False False False False True False False True False True False False True False True False False None 2 True False True False False False False False False
106 Defining and Measuring Baseline Equivalence: Module 3, Chapter 2, WWC Training What Works Clearinghouse Institute of Education Sciences This video defines and describes the importance of baseline equivalence between intervention and comparison groups in the analytic sample. It provides WWC’s standard for equivalence (if the difference between baseline’s effect size is above 0.25, it does not satisfy baseline equivalence). The video also describes the use of Hedges’ g and Cox’s Index to calculate effect sizes. 2016 Link to Resource Video 10 min R True False False False True True True False False True False False True True False False False False False False False False False False False False False None 2 False False False False False True False False False
107 Defining Confounding Factors: Module 4, Chapter 2, WWC Training What Works Clearinghouse Institute of Education Sciences Confounding factors are components of a study that make it difficult or impossible to isolate the effect of the intervention. To be identified as a confounding factor, the study component must be observed, aligned completely with only one of the study conditions, and is not part of the intervention that the study is testing. The video provides examples of these conditions. 2016 Link to Resource video 8 min P True False False False True False False False False False False False False True False False False False False False False False False False False False False None 2 False False False False False True False False False
108 Demystifying the What Works Clearinghouse: A Webinar for Developers and Researchers Constantine, J.; Cody, S.; Seftor, N.; Lesnick, J.; and McCallum, D. Institute of Education Sciences: What Works Clearinghouse at the U.S. Department of Education The What Works Clearinghouse (WWC) developed this webinar with materials to increase researchers’ knowledge of the key features of the WWC review process, including how they identify research studies and evaluate them against WWC standards. This discussion also provided details on: the role of the WWC, why it is important to have standards for research, how WWC reviews are conducted, what is done to ensure the quality and accuracy of WWC reviews and reports, how to submit questions to the WWC, how to submit a study for review, and WWC resources for researchers and developers. This webinar is a valuable resource for researchers who want to understand the features of high quality research, especially if they intend to submit their work for WWC review. 2013 Link to Resource Webinar 15 slides, 1 hour R True False False False False False False False False False False False False False False False False False False False False False False False False False False None 3 False False False False False False True False False
109 Designing and Conducting Strong Quasi-Experiments in Education. Version 2 Scher, L.; Kisker, E.; & Dynarski, M. Institute of Education Sciences The Institute of Education Sciences (IES) developed this guide to help researchers design and implement strong quasi-experimental designs (QED) when assessing the effectiveness of policies, programs, or practices. The guide first discusses the issues researchers face when choosing to conduct a QED, as opposed to a more rigorous randomized controlled trial design. Next, it documents four sets of best practices in designing and implementing QEDs, including: 1) considering unobserved variables, 2) selecting appropriate matching strategies for creating comparison groups, 3) following general guidelines for sound research, and 4) addressing the What Works Clearinghouse (WWC) standards. The guide then presents 31 frequently asked questions related to QEDs meeting WWC standards with reservations, and discusses common pitfalls that cause studies not to meet WWC standards. Topics covered in the frequently-asked questions section include: study design/group formation, outcomes, confounding factors, baseline equivalence, sample loss and power, and analytic techniques. A detailed checklist provides valuable items for researchers to consider when designing a strong QED study, and a table clarifies why each issue is important and how it relates to WWC standards. The guide is also useful for researchers looking for additional detail, as it provides links to related IES resources. 2015 Link to Resource Guide Tool 27 pages R True False False False False False True False False True True False True True False True False False True True False False False False False False False Educational Research, Quasiexperimental Design, Best Practices, Predictor Variables, Check Lists, Comparative Analysis, Standards, Clearinghouses, treatment administrative records random assignment Design Characteristics developmental math mirror image completely aligned field settings best practices wwc study treatment standards comparison group groups outcomes researchers intervention 3 False False True True False False False False False
110 Designing Quasi-Experiments: Meeting What Works Clearinghouse Standards Without Random Assignment: What Works Clearinghouse Lesnick, Joy; Seftor, Neil; Knab, Jean Institute of Education Sciences: What Works Clearinghouse at the U.S. Department of Education The Webinar aims to respond to questions from the What Works Clearinghouse about standards and procedures for studies that do not use random assignment. The intent is to inform high-quality non-experimental research design. Within a QED, baseline equivalence must be established for the control and treatment groups. WWC determines that effect size (ES) units 0-.05 ES is satisfactory, between .05 and .25 ES is moderate, and < .25 ES does not satisfy baseline equivalence. The Webinar overview includes the What Works Clearinghouse and Standards, Quasi-experimental designs (QEDs) characteristics as well as tips and cautions, resources, and staying informed. The Webinar closes with questions and answers. 2015 Link to Resource Webinar 1 hour, 13 slides R True False False False True True True False False True False False False False False True True False True True False False False False False False False None 3 False False False False False False True False False
111 Designing Strong Studies: Developing Studies Consistent with What Works Clearinghouse Evidence Standards Agodini, R.; Constantine, J.; Seftor, N.; & Neild, R.C. Institute of Education Sciences An Institute of Education Sciences webinar to help researchers design and execute studies more likely to meet What Works Clearinghouse’s (WWC) standards for research studies. The webinar focuses on several key aspects of successfully conducting randomized controlled trials and quasi-experimental study designs in an educational setting, including such issues as attrition. The webinar draws on WWC’s resources to explain how to identify and maintain a research sample, how to collect the necessary data, how to conduct analysis consistent with WWC standards, and strategies to address some of the common pitfalls in effectiveness research. 2014 Link to Resource Webinar 88 min., 34 slides R True False False False False True True False False True True True True True False True False False False True True False True False False False False you study what standards attrition intervention treatment wwc assignment comparison quasi experimental confounding factors qualitative conclusion causal statements 3 True False False False False False True False False
112 Designing Teacher Evaluation Systems: New Guidance from the Measures of Effective Teaching Project Kane, T.; Kerr, K.; Pianta, R. Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation published this book to share the lessons learned from its multi-year Measures of Effective Teaching project. The book is a compilation of research papers that use a large database the Foundation created in the course of the project. It focuses on three aspects of the study: using data for feedback and evaluation, connecting evaluation measures with student learning, and the properties of evaluation systems (quality, frameworks, and design decisions). This guide is a valuable resource for professionals designing and evaluating educator evaluation systems. The guide is also useful for program designers and evaluators who want to anticipate issues that are likely to arise due to their design decisions. 2015 Link to Resource Methods Report 617 pages R False False False True True False True False False False False True False False True True True False True True True True True False True False False None 3 True False False False False False False False False
113 Developing Surveys Using the Collaborative Survey Development Process Irwin, C.; Stafford, E. Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a video series to help professionals develop, administer, and analyze surveys in practical education contexts. The series builds off on a series of guides available online. The presentation (from about 0:08:55 to 1:15:00) addresses collaborative survey development. It defines it as a process that includes educators, researchers, and content experts, which allows the leveraging of expertise, reduces the burden on anyone individual, and promotes survey quality. The video describes the establishment of a development team and the five steps in the process: identifying topics of interest, identifying survey items, drafting new items and adapting existing ones, reviewing draft items with stakeholders and content experts, and refining the draft survey with pretesting using cognitive interviewing. It provides practical examples, tools, and activities for each step. This video will be helpful to those interested in developing and administering surveys as a means of collecting data to inform education policy and practice. 2016 Link to Resource Video Guide 1 hour 32 min P False False False False True False False False False False False False False False False False True False False False False False False False False False False None 3 False False True False False True False False False
114 Developing Surveys Using the Collaborative Survey Development Process Excerpt 2: Rewriting Items Irwin, C.; Stafford, E. Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a video series to help professionals collaboratively develop and administer surveys. The series briefly summarizes information from a related guide. This video proposes three ways to phrase a survey question and asks participants to list their weaknesses. This video will be helpful to those wanting to build their awareness of the complexity of item writing. 2016 Link to Resource Video Brief 4 minutes P False False False False True False False False False False False False False False False False True False False False False False False False False False False None 1 False True False False False True False False False
115 Developing Surveys Using the Collaborative Survey Development Process Excerpt 1: Survey Research Process and Collaborative Survey Development Irwin, C.; Stafford, E. Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a video series to help professionals collaboratively develop and administer surveys. The series very briefly summarize information from a related guide. This video mentions the three stages of the survey research process: survey development, sample selection and survey administration, and data analysis and reporting. It then asks participants why they think the development process should be collaborative. Next, it defines collaborative survey development as a process that includes educators, researchers, and content experts, which allows the leveraging of expertise, reduces the burden on anyone individual, and promotes survey quality. This video will be helpful to those considering the utility of developing surveys as part of a collaboration. 2016 Link to Resource Video 5 minutes P False False False False True False False False False False False False False False False False True False False False False False False False False False False None 1 False False False False False True False False False
116 Developing Surveys Using the Collaborative Survey Development Process Excerpt 3: Q&A. Irwin, C.; Stafford, E. Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Northeast & Islands developed a video series to help professionals collaboratively develop and administer surveys. The series briefly summarizes information from a related guide. This video defines cognitive interviewing as a process to identify and correct problems with surveys by watching respondents take one. This can reveal issues such as confusing items, selection bias, people who open the survey and close it almost immediately. The authors have used this process as a way to pilot/pre-test a survey and recommended conducting this prior to a full-on pilot. This video will be helpful to those who have absolutely no knowledge of cognitive interviewing. 2016 Link to Resource Video 4 min P False False False False True False False False False False False False False False False False True False False False False False False False False False False None 1 False False False False False True False False False
117 Dimensions of Dosage: Evaluation Brief for TIF Grantees Bailey, J.; Shakman, K.; Ansel, D.; Greller, S. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to introduce evaluators to dosage as a tool that can be used when it is not possible to identify a good comparison group for evaluating the effects of a program. Dosage refers to the amount of the intervention that is delivered—for example, the number of hours teachers spend with a teacher leader. The guide describes dosage, its dimensions, associated data requirements, and considerations to keep in mind when thinking about measuring dosage. It provides an example from a school district and a list of questions to guide discussions around dosage. This guide is a valuable resource for program directors planning a high quality evaluation. 2016 Link to Resource Guide 8 pages P False False False True True False False False False True False False False False False False False False False False False False False False False False False None 2 False False True False False False False False False
118 Dynamic Effects of Teacher Turnover on the Quality of Instruction. Working Paper 170 Hanushek, E.; Rivkin, S.; Schiman, J. National Center for Analysis of Longitudinal Data in Education Research The National Center for Analysis of Longitudinal Data in Education Research developed this report to provide researchers with a thorough analysis of the confounding factors that affect conclusions about the relationship between teacher turnover and quality instruction. The report focuses on analyses that aim to account for nonrandom sorting of students into classrooms, endogenous teacher exits, and grade-switching. A literature review describes the steps taken in previous research, lessons learned, and limitations that this paper addresses. 2016 Link to Resource Methods report 52 pages R False False False False True False False False False False False False False False True True False False True True False False True False True False False Labor Turnover, Teacher Persistence, Educational Quality, Urban Schools, Disadvantaged Youth, Low Achievement, Teacher Distribution, Teacher Placement, Teaching Experience, School Districts, Teacher Effectiveness, Productivity, Public Schools, Academic Achievement, Comparative Analysis, Instructional Program Divisions, Records (Forms), Student Records, Statistical Analysis 3 True False False False True False False False False
119 Educator Evaluation System Data: What Exists and How do we use it? Lemke, M.; Livinston, T.; & Rainey, K. Regional Education Laboratory Program, Institute of Education Sciences This Regional Educational Laboratory (REL) Midwest webinar presents research on different designs of teacher evaluation systems and provide examples of how educator evaluation data is being used to support decision-making and inform professional development. The presenters also share insights on the implementation of evaluation systems and use of data at the state and district levels. 2015 Link to Resource Webinar Slide presentation 68 slides, 78 minutes P False True False True True False False False False False False False False False False False False False False False False False False False False False False Educator Effectiveness implementation research data assessment effective leads cohort practices circle 3 False False False False False False True True False
120 Estimating causal effects using experimental and observational designs Schneider, B.; Carnoy, M.; Kilpatrick, J.; Schmidt, W.; & Shavelson, R. American Educational Research Association This American Educational Research Association (AERA) report is intended to help researchers, educators, and policymakers understand causal estimation by describing the logic of causal inference and reviewing designs and methods that allow researchers to draw causal inferences about the effectiveness of educational interventions. The purpose of the report is to explain the value of quasi-experimental techniques that can be used to approximate randomized experiments. The report does not address many of the nuances of experimental and quasi-experimental designs. 2007 Link to Resource Methods Report 158 pages R True False True False False True True True False True False False True True True False False False True True False False True False False False False American Children’s Cognitive Cognitive Growth Left Behind North Carolina Rosenbaum characteristics coverage curriculum kinder effects students research causal treatment school achievement education student schools 3 True False False False True False False False False
121 Evaluating Principals Work: Design Considerations and Examples for an Evolving Field Kimball, S.; Clifford, M.; Fetters, J.; Arrigoni, J.; Bobola, K. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to help program directors design principal evaluation systems. The first section addresses leadership effectiveness literature and the documented need for principal evaluation. The second section includes the key components of principal evaluation systems supplemented with examples from innovative school districts and states. This section addresses challenges reformers often confront in the design process and solutions for some of those challenges. The third section summarizes the critical areas of evaluator training and pilot testing. The fourth section highlights stakeholder engagement and communication. The fifth section summarizes training and pilot testing. This guide is also a valuable resource for researchers who want to assess the quality of principal evaluation systems. 2012 Link to Resource Methods Report 23 pages P False False False True True False False False False False False False False False False True True False False False False False True False True True False None 3 True False False False False False False False False
122 Evaluating Programs for Strengthening Teaching and Leadership Milanowski, A.; Finster, M. Teacher and Leadership Programs The Teacher Incentive Fund developed this guide to help grantees evaluate their programs. The guide focuses on key aspects of the evaluation process, including identifying the logic of how the program will lead to the desired outcomes, developing evaluation questions that examine this logic, exploring methods for measuring these evaluation questions, choosing an appropriate evaluation design for assessing impacts, disseminating evaluation findings, and choosing the right evaluator. This guide is a valuable resource for program directors who want to understand how to design and implement a high-quality evaluation. The guide is also useful for seasoned evaluators who want to review important evaluation issues. 2016 Link to Resource Guide 67 pages P False False False True True True True False True True False False False True False True False False True True True False True True True True True None 3 True False True False False False False False True
123 Evaluation of the Effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI) Newman, D.; Finney, P. B.; Bell, S.; Turner, H.; Jaciw, A.; Zacamy, J.; Feagans G. L. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southeast developed this report to describe the findings from a study of a two-year intervention intended improve student achievement in math, science, and technology. An overview of findings starts in the last paragraph of page xxiv and ends at the bottom of page xxvii. The relevant sections of the report are selection and random assignment of schools (appendix C on pages C-1 to C-3), a teacher survey of math and science classroom practices and professional development and availability of technology (appendix G on pages G-1 to G-15), data cleaning and data file construction (appendix H on page H-1), and sections on attrition (appendix I — steps through which the analytic sample was selected for each confirmatory outcome on pages I-1 to I-8, N — attrition associated with the SAT reading outcomes, teacher content knowledge in mathematics and in science, and student engagement in mathematics and science through study stages for samples used in Year 1 exploratory analysis on pages N-1 to N-8, and R — on attrition through study stages for samples contributing to estimation of two-year effects on pages R-1 to R-9). 2012 Link to Resource Methods Report 44 Pages R False False False False True True False False False True False False True True True False False True True True False False True False True False False Student Achievement, Mathematics, Science 3 True False False False True False False False False
124 Evaluation of the Teacher Incentive Fund: Final Report on Implementation and Impacts of Pay-for-Performance Across Four Years Chiang, H.; Speroni, C.; Herrmann, M.; Hallgren, K.; Burkander, P.; Wellington, A. Institute of Education SciencesD3:D7 National Center for Education Evaluation and Regional Assistance The Institute of Education Sciences developed this report to disseminate evaluation findings on the Teacher Incentive Fund’s (TIF) implementation and impacts of pay-for-performance. The relevant sections of the report are the overview in Chapter II and the methodology appendices. Appendix A provides information on random assignment and matching of schools, attrition, selection of the teacher survey sample, sampling, nonresponse, and treatment of missing data. Appendix B provides the rationale for and technical details of the methods used in the report. It covers standardization of educator performance ratings and student test scores across districts; the technical approach for describing the distribution of performance ratings and TIF payouts in evaluation districts; the analytic methods in detail; interpretation of student achievement impacts; methods used to explore differences in impacts across teacher and student subgroups, districts, and schools; estimation of year-to-year changes in average teacher perceptions of TIF; imputation of educators’ beliefs about bonus amounts; and minimum detectable impacts. These sections are a valuable resource for program directors and evaluators who are designing and implementing high quality evaluations that involve random assignment and both qualitative and quantitaive data collection. 2017 Link to Resource Methods report 63 pages R False False False False True True False False False True True False True True True True True False True True False False True False True True False Incentive, teachers 3 True False False False True False False False False
125 Evaluation Plan Template Price, C.; Goodson, B.; Wolf, A.; Boulay. B. Abt Associates Abt Associates developed this guide to help researchers design program evaluations. It identifies the key components of an evaluation plan and provides guidance about the information typically included in each section of a plan for evaluating both the effectiveness and implementation of an intervention. It suggests which sections should be filled during initial evaluation planning and which can be filled later in the process. Guidance appears in italics in a box under each section heading. After the document is downloaded, saved, and opened in a pdf editor, text can be typed into each box and the guidance can be deleted. Throughout, there are references to additional resources or tools that can help develop an evaluation plan. This tool is a valuable resource for evaluators who want to align planned analyses with research questions and to outline key information about the plan’s outcome measures, baseline measures, and analytic samples. The database includes example plans for student-level (https://ies.ed.gov/ncee/projects/pdf/ExEPStudentLevelRCT.pdf) and cluster (https://ies.ed.gov/ncee/projects/pdf/ExEPClusterRCT.pdf) randomized controlled trials and quasi-experimental designs (https://ies.ed.gov/ncee/projects/pdf/ExEPQuasiExperimentalDesign.pdf). The template can be used in combination with a Contrast Tool also included in the database (https://ies.ed.gov/ncee/projects/xls/ContrastTool.xlsm) to document each impact that the evaluation will estimate to test program effectiveness. 2016 Link to Resource Guide Tool 15 R True False False False False True True False False True False False True False False True True False False False False False False False False False False None 2 False False True True False False False False True
126 Example Contrast Tool for a Quasi-Experimental Design Wolf, A.; Barbara Goodson, B.; Gan, K.; Price, C.; Boulay, B. Abt Associates Abt Associates developed this tool to help researchers design program evaluations. This “contrast tool” is an Excel spreadsheet designed to support and supplement the development of an evaluation plan. It contains information from an example study that investigates the impact of a mathematics intervention that aims to improve students’ successful completion of developmental math courses in order to prepare them to be able to meet core course requirements for college-level mathematics. Individual worksheets contain information on research questions, outcome measures, baseline measures, samples, and contrasts (i.e., impacts) to be tested. Two worksheets have reporting information on attrition and baseline equivalence. There is an instructions tab for each worksheet. This tool is a valuable resource for evaluators who want to align planned analyses with research questions and to outline key information about the plan’s outcome measures, baseline measures, and analytic samples. It can be used in combination with the matching example evaluation plan included in this database (https://ies.ed.gov/ncee/projects/pdf/ExEPQuasiExperimentalDesign.pdf) included in this database. A template version is included as well (https://ies.ed.gov/ncee/projects/xls/ContrastTool.xlsm). 2016 Link to Resource Tool 20 R True False False False True False True False False True False False True False False True False False False True True False True False False False False None 2 True False False True False False False False True
127 Example Contrast Tool for a Student-Level Randomized Controlled Trial Wolf. A.; Epstein, C.; Goodson, B.; Price, C.; Boulay, B. Abt Associates Abt Associates developed this tool to help researchers design program evaluations. This “contrast tool” is an Excel spreadsheet designed to support and supplement the development of an evaluation plan. It contains information from an example study that investigates the impact of a peer support community on college students at risk of failing to complete a Bachelor’s degree. Individual worksheets contain information on research questions, outcome measures, baseline measures, samples, and contrasts (i.e., impacts) to be tested. Three worksheets can be used for reporting results: impacts, attrition, and Baseline Equivalence. There is an instructions tab for each worksheet. This tool is a valuable resource for evaluators who want to align planned analyses with research questions and to outline key information about the plan’s outcome measures, baseline measures, and analytic samples. It can be used in combination with the matching evaluation plan example (https://ies.ed.gov/ncee/projects/pdf/ExEPStudentLevelRCT.pdf) included in this database. A template version is included as well (https://ies.ed.gov/ncee/projects/xls/ContrastTool.xlsm). 2016 Link to Resource Tool 20 R True False False False True True False False False True False False True False False True False False False True True False True False False False False None 2 True False False True False False False False True
128 Example Evaluation Plan for a Quasi-Experimental Design Wolf. A.; Goodson, B.; Gan, K.; Price, C.; Boulay. B. Abt Associates Abt Associates developed this guide to help researchers design quasi-experimental studies (QEDs). This evaluation plan template identifies the key components of an evaluation plan, provides guidance about the information typically included in each section of a plan for evaluating the effectiveness of an intervention, and illustrates each section with an example. It can be used in combination with a “contrast tool” available in this database (https://ies.ed.gov/ncee/projects/xls/ExContrastToolQED.xlsm), which is an Excel file that lists each impact that the example evaluation will estimate to test program effectiveness. The database also includes a template (https://ies.ed.gov/ncee/projects/pdf/EvaluationPlanTemplate.pdf). This guide is a valuable resource for evaluators who want to align planned analyses with research questions and to outline key information about the plan’s key measures and analytic samples. 2016 Link to Resource Guide Tool 21 R True False False False True False True False False True True False True True True True False False True True False False True False False False False None 3 True False True True False False False False True
129 Example Evaluation Plan for a Student-Level Randomized Control Trial Wolf, A.; Epstein, C.; Goodson, B.; Price, C.; Boulay, B. Abt Associates Abt Associates developed this guide to help researchers design student-level randomized controlled trials (RCTs). It identifies the key components of an evaluation plan, provides guidance about the information typically included in each section of a plan for evaluating both the effectiveness and implementation of an intervention, and illustrates each section with an example. The plan is for an RCT in which students are randomly assigned to an intervention or a control condition. It can be used in combination with a “contrast tool” available in this database (https://ies.ed.gov/ncee/projects/xls/ExContrastToolStudentRCT.xlsm), which is an Excel file that lists each impact that the example evaluation will estimate to test program effectiveness. The database also includes a template (https://ies.ed.gov/ncee/projects/pdf/EvaluationPlanTemplate.pdf). This guide is a valuable resource for evaluators who want to align planned analyses with research questions and to outline key information about the plan’s outcome measures, baseline measures, and analytic samples. 2016 Link to Resource Guide Tool 27 R True False False False True True False False False True True True True True True True True False True True False False True False False False False None 3 True False True True False False False False True
130 Face Validity and Reliability: Module 5, Chapter 3, WWC Training What Works Clearinghouse Institute of Education Sciences This video defines and provides examples of face validity and reliability. Face validity is ensuring that the chosen measure actually captures data on the outcome of interest. The reliability of a measure ensures that the measure would yield similar scores when used several times, and has a minimum risk of measurement error. 2016 Link to Resource video 6 min P True False False False True False False False False False False False False False False True False False False False False False False False False False False None 2 False False False False False True False False False
131 First in the World Project Directors Meeting: Implementing a Successful Quasi-Experimental Design Moss, M. Institute of Education Sciences Abt Associates developed this guide to help researchers design quasi-experimental designs (QEDs). The guide begins with a review of randomized controlled trials (RCTs), which are considered the strongest program evaluation design in the field. It focuses on QEDs as the next best alternative when RCTs are not feasible and describes their limitations. The guide suggests steps to take and pitfalls to avoid when selecting a comparison group and provides a link to a list of software for implementing matching methods. The final section of the guide discusses baseline equivalence. 2016 Link to Resource Slide Presentation Guide 26 R True False False False True True True False False True False False False True False False False False False True False False False False False False False References WWC Standards, Any Education Topic, Initial Planning, RCT, QED, Identifying Comparison Groups, Reducing Bias in Comparison Groups, Addressing Analysis Challenges 3 False False True False False False False True True
132 First in the World Project Directors Meeting: Understanding Challenges in Random Assignment Wolf, A. Institute of Education Sciences Abt Associates developed this guide to help researchers plan for a randomized controlled trial, focusing on conducting and monitoring random assignment (RA). It provides suggestions for five major activities in the RA process: designing it (identifying participants, timing the RA, and documenting the process); creating buy-in (with examples of communications strategies); identifying eligible participants; conducting the RA (including dealing with late identification and correct placement as well as documenting those); and monitoring it. The guide provides a link to a diagram for tracking sample numbers throughout the study (click on “CONSORT 2010 Flow Diagram”). Click on “CONSORT 2010 Checklist” for a checklist of information to include when reporting on an RCT. 2016 Link to Resource Slide Presentation Guide 31 P False False False False True True False False False True False True True True False False False False False False False False False False False False False Any Education Topic, Initial Planning, RCT, Identifying Comparison Groups, Recruiting Study Participants, Addressing Changes in Your Sample, Reducing Bias in Comparison Groups 3 False False True False False False False True True
133 Forming a Team to Ensure High-Quality Measurement in Education Studies Kisker, E.; & Boller, K. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) program developed this brief to help identify the expertise needed to formulate an appropriate and effective measurement strategy and to collect high-quality data. The primary audience for this brief is research teams – such as those at RELs – who work in partnership with school districts or states. The brief provides tips for finding staff and consultants with the needed expertise, and outlines their main responsibilities. It also outlines typical measurement tasks and discusses how measurement team members can work together to complete these tasks successfully. 2014 Link to Resource Guide 18 pages P True False False False False False False False False False False False False False False False False False False False False False False False False False False data professional networks Analytic Technical Works Clearinghouse study U.S. Department statistical power strategies listed measurement measures collection expert team measure constructs 2 False False True False False False False False False
134 Forum Guide to Decision Support Systems: A Resource for Educators. NFES 2006-807 U.S. Department of Education U.S. Department of Education This document was developed by educators for educators to remedy the lack of reliable, objective information available to the education community about decision support systems. The authors hope it will help readers better understand what decision support systems are, how they are configured, how they operate, and how they might be implemented in an education institution. The Decision Support System Literacy Task Force hopes that readers will find this document useful, and that it helps improve data-driven decision making in schools, school districts, and state education agencies across the nation. This document addresses the following broad questions: (Part I) What is a Decision Support System, and how does a decision support system differ from a data warehouse and a data mart? (Part II) What components, features, and capabilities commonly comprise a decision support system? How does each broad category of these components and features contribute to system operation? (Part III) How does an education organization buy or develop a decision support system, and how are stakeholders trained to use it? 2006 Link to Resource Guide 34 pages P False False False True False False False False False False False False False False True False True True True False True False False False False False False Department of Education, Education Statistics website, secondary education, NATIONAL COOPERATIVE EDUCATION STATISTICS SYSTEM, National Forum, National Center, local education agencies, early childhood education, state, Forum World Wide, Cooperative System, data, NCES World Wide Web Home Page, NCES World Wide Web Electronic Catalog, uniform information, local levels, principles of good practice, meeting, activities, Publications, policymaking, purpose, products, views, endeavors, formal review, resources, opinions, September 3 False False True False False False False False False
135 Gathering Feedback for Teaching: Combining High-Quality Observations with Student Surveys and Achievement Gains (Study) Kane,T.J.; & Staiger, D.O. Bill & Melinda Gates Foundation The Bill & Melinda Gates Foundation developed this guide to help policymakers and practitioners effectively implement observations of teacher practice. The guide reviews minimum requirements for high-quality classroom observations and investigates two properties of five instruments: reliability (the extent to which results reflect consistent aspects of a teacher’s practice and not the idiosyncrasies of a particular observer, group of students, or lesson) and validity (the extent to which observation results are related to student outcomes). This guide is a valuable resource for policymakers and practitioners at every level who are intensely focused on improving teaching and learning through better evaluation, feedback, and professional development. It is one outcome of the Gates’ Measures of Effective Teaching project, which investigates a number of alternative approaches to identifying effective teaching: systematic classroom observations, surveys collecting confidential student feedback, a new assessment of teachers’ pedagogical content knowledge, and different measures of student achievement. Those wanting to explore all the technical aspects of the study and analysis are referred to a companion research report. 2012 Link to Resource Guide 36 pages P False False False True False False False False False False False False False False False False False False False False False False True False False False False Teacher Effectiveness, Achievement Gains, Evaluation Methods, Teaching Methods, Observation, Feedback (Response), Video Technology, Research Papers (Students), Student Surveys, Teacher Competencies, Student Attitudes, Academic Achievement, Faculty Development, Teacher Improvement, Standards, Scoring, Language Arts, Mathematics Instruction, Test Validity, Test Reliability, student teachers scores teaching project met teacher observation observations instruments met project value added combined measure effective teaching student achievement observation scores competencies classroom observations achievement gains 3 True False True False False False False False False
136 Gold-Standard Program Evaluations, on a Shoestring Budget Baron, J. Education Week An Education Week article advocating for the use of low-cost randomized controlled trials (RCT) in education research. The article describes two examples of recent RCTs that were conducted at low cost, yet produced findings of policy and practical importance. The author suggests that with improvements in information technology and the availability of high-quality administrative data, it is now more feasible to conduct gold-standard randomized evaluations on a shoestring budget. 2011 Link to Resource Brief or Summary 2 pages P False False True False False True False False False False False False False False True False False False False False False False False False False False False already collected student achievement low cost key outcomes program study cost education student data group achievement conducted 1 False True False False False False False False False
137 Graphic Design for Researchers Institute of Education Sciences (ED); Decision Information Resources, Inc.; Mathematica Policy Research, Inc. Regional Education Laboratory Program, Institute of Education Sciences The Institute of Education Sciences developed this guide that offers a basic overview on how researchers can effectively use design to create engaging and visually appealing Regional Educational Laboratory products. This guide also touches on how researchers can use data visualization to make complex concepts accessible. 2014 Link to Resource Guide 14 pages R False False False False False False False False False False False False False False False False False False False False False True False False False False False What Works Clearinghouse approaching benchmarks benchmarks category transition courses visually appealing design elements example http data visual clip art 2 False False True False False False False False False
138 Graphical Models for Quasi-Experimental Designs Kim, Y.; Steiner, P.M.; Hall, C.E.; Su, D. Society for research on Educational Effectiveness This conference abstract was written to introduce researchers to graph-theoretical models as a way to formalize causal inference. They describe mechanics and assumptions of the causal graphs for experimental and quasi-experimental designs — randomized controlled trials, and regression discontinuity, instrumental variable, and matching and propensity score designs. For each design, they show how the assumptions required for identifying a causal effect are encoded in the graphical representation. They argue that causal graphs make the assumptions more transparent and easier to understand for applied researchers. This abstract will help users in getting a better understanding of the designs’ assumptions and limitations and conducting better causal studies. It will be useful for researchers who want to represent the features of the study designs and data collection process that affect the identification of causal effects. 2016 Link to Resource Brief or Summary 11 pages R False False False False False True True True False False False False False False False False False False True False True True False False False False False causal estimands of, threats, University of Wisconsin-Madison, Quasi-Experimental Designs, ubin Causal Model, causal inference, Abstract Title Page, Abstract Body Limit, Conference Abstract Template, Campbell, convenient way of defining causal estimands, conceptualization of internal validity, RCM, deriving identification assumptions, Mexico Public Education Department, SREE Spring, fields, central role, estimating cause-effect relationships, history, maturation, Hall, page count, Authors, Affiliations, Background, Context, stable-unit-treatment-value assumption, instrumentation effects, strong ignorability, potential outcomes frameworks, Shadish, behavioral sciences, pages single-spaced, practical point of view, selection, tradition of ruling, researchers, psychology, propensity scor, SUTVA, Cook, Steiner, Holland, formal representation, Courtney, Yongnam Kim, Dan Su, Graphical Models, Peter 2 False True False False False False False False False
139 Group Design Knowledge Checks: Module 1, Chapter 4, WWC Training What Works Clearinghouse Institute of Education Sciences This video includes a knowledge check for viewers to assess whether they have a clear understanding of what types of designs are eligible for WWC review. Randomized Controlled trials (RCTs) and Quasi-experimental designs (QED) are addressed. 2016 Link to Resource Video 12 min P True False False False True True True False False False False False False False False False False False False False False False False False False False False None 2 False False False False False True False False False
140 How (and What) to Train Teachers and Administrators on Education Data Privacy Vance, A. National Association of State Boards of Education The National Association of State Boards of Education developed this presentation to train teachers and administrators on the protection of student data. This video reviews state trends, common reasons for data breaches, legislation, what administrators need to know when using, saving, and sharing data and working with vendors, and ways to prevent and handle student data disclosure. This video provides several resources, most of them available online; lessons learned; and a communications toolkit for best practices while communicating with various stakeholders. Key points and resources can be easily accessed by scrolling along the timing line on screen and pausing on the slides the presenter shows. This video will be helpful to those working with sensitive data. 2016 Link to Resource Video Guide 44 min P False False False False True False False False False False False False False False True False False False False False False False False False False False False None 3 False False True False False True False False False
141 How to Facilitate the Logic Models Workshop for Program Design and Evaluation Rodriguez, S. & Shakman, K. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast & Islands developed this webinar as a guide for educational leaders who want to introduce logic models for program design, implementation, and evaluation using a workshop toolkit available from REL Northeast & Islands (the toolkit is also in this database – see “Logic models for program design, implementation, and evaluation: Workshop toolkit”. During the webinar, the authors of the toolkit focus on the facilitation, organization, and resources necessary to conduct the workshop virtually or in person. 2015 Link to Resource Webinar Slide presentation 1 hour 13 minutes P False False False True True False False False False False False False False False False False False False False False False False False False False False False None 3 False False False False False False True True False
142 I Wish I Could Believe You: The Frustrating Unreliability of Some Assessment Research Hunt, T.; Jordan, S. Practitioner Research in Higher Education The authors developed this guide to help professionals understand the limitations of assessment research and some mitigation strategies. The challenge is determining the impact of assessment practices on learning in authentic educational settings. The guide uses examples to highlight some of the difficulties: assuming that if two effects are correlated, then one must have caused the other; confounding variables obscuring the true relationships; experimental approaches that are too far removed from reality; and the danger that self-reported behavior and opinion is sometimes different from student’s actual behavior. As practical solutions, the guide proposes using: experimental or pseudo-experimental approaches; mixed methods; and meta-analyses. 2016 Link to Resource Guide 9 P False False False False True True True False False True False False False False False False False False False True False False False False False False False Research methodology 2 False False True False False False False False False
143 Identifying and Measuring Attrition: Module 2, Chapter 2, WWC Training What Works Clearinghouse Institute of Education Sciences This video defines attrition in RCTs (WWC does not consider attrition in QEDs) and discusses the risk of bias in a study due to attrition. The video provides examples of different types of attrition, defines overall versus differential attrition, and discusses how the WWC calculates attrition rates. WWC defines high attrition as bias if it is equal or greater than 0.05 standard deviations. 2016 Link to Resource Video 15 min P True False False False True True False False False False False False True False False False False False False False False False False False False False False None 2 False False False False False True False False False
144 Impact of a Checklist on Principal–Teacher Feedback Conferences Following Classroom Observations Mihaly,K.; Opper, I.; Rodriguez, L.; Schwartz, H.; Grimm, G.; Mariano; L. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southwest developed this report to describe the findings from a study of a principal-teacher feedback conferences. This study was a statewide experiment in New Mexico in 2015-16. It provided principals and teachers a checklist to use in the feedback conferences that principals had with teachers following formal classroom observations and tested whether it would improve the quality and impact of the conferences. The relevant section of the report is Appendix D. Appendix D presents the data, sample, and methodology. It includes a detailed description of how outcome measures were constructed, sample recruitment, the model, sensitivity and subgroup analyses, and treatment of missing data and crossovers. This section is a valuable resource for evaluators planning and implementing randomized controlled trials. 2018 Link to Resource Methods report 18 R False False False True True True False False False True False True True True True True True False True True False False True False True True False Principals (Administrator Role), Teacher 2 True False False False True False False False False
145 Impact of a Checklist on Principal–Teacher Feedback Conferences Following Classroom Observations Mihaly,K.; Opper, I.; Rodriguez, L.; Schwartz, H.; Grimm, G.; Mariano; L. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southwest developed this report to describe the findings from a study of a principal-teacher feedback conferences. This study was a statewide experiment in New Mexico in 2015-16. It provided principals and teachers a checklist to use in the feedback conferences that principals had with teachers following formal classroom observations and tested whether it would improve the quality and impact of the conferences. The relevant sections of the report are Appendices B and D. Appendix B provides a principal and a teacher checklist including a list of documents for principals and teachers to bring and items on how to start the conversation, acknowledging successes, identifying challenges, generating ideas for addressing them, and ending on a positive note. These checklists could be used to suggest elements to include in the development of a tool (e.g., survey or interview) to collect data to assess implementation fidelity and effectiveness of feedback conferences as part of evaluation systems. Appendix D presents the data, sample, and methodology. It includes a detailed description of how outcome measures were constructed, sample recruitment, the model, sensitivity and subgroup analyses, and treatment of missing data and crossovers. These sections are a valuable resource for evaluators planning and implementing randomized controlled trials. 2018 Link to Resource Tool Methods report 72 R False False False True True True False False False True False True True True True True True False True True False False True False True True False Principals (Administrator Role), Teacher 3 True False False True True False False False False
146 Impacts of Comprehensive Teacher Induction: Final Results from a Randomized Controlled Study Glazerman, S.; Isenberg,E.; Dolfin, S.; Bleeker, M.; Johnson,A.; Grider, M.; Jacobus, M. National Center for Education Statistics, U.S. Department of Education The National Center for Education Statistics developed this report to provide researchers with a description of an evaluation of the impact of a comprehensive teacher induction initiative. The report focuses on the design and implementation of a randomized experiment using data collection from multiple stakeholders. It addresses issues of comparison group selection and bias. Appendix A provides detailed information on the model and how it addresses potential bias. Figures are provided throughout that are examples of various ways of displaying results. 2010 Link to Resource Methods report 272 pages R False False False True True True False False False True True True True True True False True False True True True True True False True True False None 3 True False False False True False False False False
147 Incomplete Reporting: Addressing the Prevalence of Outcome-Reporting Bias in Educational Research Trainor, B.; Polanin, J.; Williams, R.; Pigott, T. SREE Spring 2015 Conference Abstract Study authors developed this brief (SREE abstract) to raise researchers’ awareness of biases due to incomplete outcome reporting — the practice of omitting outcome variables from primary studies that were actually collected. Primary studies not reporting on all outcomes may lead to an incomplete understanding of a phenomenon that may be compounded when the study is included in a systematic review of research. The brief focuses on how authors plan to conduct analyses of a set of dissertations to identify factors that might be related to outcome-reporting bias such as whether the outcome was considered a primary or a secondary focus of an intervention, whether the outcome was a negative or potentially harmful effect of the intervention, and whether the measurement strategy for the outcome was reliable and valid. 2015 Link to Resource Methods report Brief 5 pages R False False False False True False False False False False False False False False False False False False False True True False False False False False False None 1 False True False False True False False False False
148 Indicators of Successful Teacher Recruitment and Retention in Oklahoma Rural School Districts Lazarev, V.; Toby, M.; Zacamy, J.; Lin, L.; Newman, D. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southwest developed this report to describe teacher, district, and community characteristics in rural Oklahoma that predict which teachers are most likely to be successfully recruited and retained longer term. Three sections of the report are relevant. The first one is study limitations on page 16, which discusses unavailability of data on key variables that could predict recruitment and retention; a ten-year data time span, which does not allow for long-term analyses; and the non-experimental design, which prevents making causal inferences. Appendix B describes how variables were identified. In Appendix C on data and methodology, the authors describe the sample, datasets, outcome variable creation, treatment of missing data, accounting for the censoring of duration data, and the approach they used for each research question, including descriptive statistics, probabilities, statistical tests, and regression analyses (logistic regression and discrete-time survival analysis). It also describes how results across models were compared to understand the predictive capacity of specific groups of variables, robustness checks, and the calculation of regression-adjusted marginal probabilities to account for the fact that variables are likely to be significant simply because of the size of the database. These sections are a valuable resource for program evaluators and researchers who want to identify patterns in large datasets, especially those that involve duration variables. 2017 Link to Resource Methods report 17 pages R False False False True True False False False False False True False False False True True False False True True False False False False True True True Retention (Of Employees), States, Teachers Attrition/Mobility 2 True False False False True False False False False
149 Introducing the Education Logic Model (ELM) Application REL Pacific Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Pacific developed this guide tool to provide program implementers and evaluators with a logic model template to capture information about program resources, activities, outputs, and short-, mid-, and long-term outcomes. Logic models are used to visually present the relationships among program elements and help document a path to reach expected outcomes. They can be used for program design, implementation, evaluation, and communications. To access the resource, open the weblink, click on the arrow by “Download ELM;” save the file (it takes several minutes to download); right click on elm.zip; select “extract all;” in the window that appears, select a folder for storing; click “Extract;” in that folder, click on the ELM folder; double-click on “index.” To test the process, click on “New Model,” enter “test” under program name and click next; follow the instructions to obtain a logic model that can be saved as a pdf file and printed by clicking on “View/Print.” Written and audio instructions are provided throughout. 2018 Link to Resource Tool 5 P False False False False True False False False False False False False False False False True False False False False False False False False False False False Any Education Topic, Initial Planning, Selecting Appropriate Outcome Measures 1 False False False True False False False False True
150 Introduction to Group Designs: Module 1, Chapter 1, WWC Training What Works Clearinghouse Institute of Education Sciences This video provides an overview of research designs reviewed under WWC Group Design Standards–randomized controlled trials (RCT) or Quasi-Experimental Designs (QED) 2016 Link to Resource Video 3 min P True False False False True True True False False True False False False False False False False False False False False False False False False False False None 1 False False False False False True False False False
151 Introduction to WWC Training: Intro–Chapter 1, WWC Training What Works Clearinghouse Institute of Education Sciences An introduction to WWC Group Design Training Modules, this video provides a brief summary of the purpose and the products of the WWC. 2016 Link to Resource video 6 min P True False False False True False False False False False False False False False False False False False False False True False False False False False False None 2 False False False False False True False False False
152 IWWC Group Design Standards: Intro–Chapter 3, WWC Training What Works Clearinghouse Institute of Education Sciences This video provides a brief overview of the reason for WWC standards, and the three ratings that a study might receive after being reviewed by WWC: (1) Meets standards without reservations (2) Meets standards with reservations, and (3) Does not meet standards. 2016 Link to Resource video 2 min P True False False False True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False False False True False False False
153 Key Items to Get Right When Conducting Randomized Controlled Trials of Social Programs Evidence-Based Policy Team of the Laura and John Arnold Foundation Laura and John Arnold Foundation The Laura and John Arnold Foundation developed this checklist to help researchers and sponsors of research ensure that items critical to the success of a randomized controlled trial (RCT) in producing valid findings about a social program’s effectiveness are included. Items in this checklist are categorized according to the following phases of an RCT: planning the study, carrying out random assignment, measuring outcomes for the study sample, and analyzing the study results. This tool is a valuable resource for researchers who need a list of key items to get right when conducting an RCT to evaluate a social program or practice. 2016 Link to Resource Tool 12 pages P False False False False False True False False False True True True True True False True True False True True True False False False False False False Evaluating Public Experimental Methods With Experimental Larry L. Orr Programs With Public Programs get right into account take into evaluation educator teacher student data teachers learning district effectiveness system 2 False False False True False False False False False
154 Limitations of Experiments in Education Research Schanzenbach, D. Association for Education Finance and Policy The Association for Education Finance and Policy developed this brief as a caution to the growing traction of randomized experiments in education circles in recent years. The brief succinctly describes significant limitations of experiments, including feasibility, a program’s complexity, time, generalizability of results, lack of insight into what makes a program work, behavior changes due to participation in an experiment, cost, ethics, fidelity of implementation, the temptation to mine the data to obtain desired results, and the fact that experiments may not be superior to other evaluation approaches. This brief also suggests ways to mitigate these limitations when designing a study. 2012 Link to Resource Brief or Summary 14 pages P False False True False False True False False False False False False False False False True False False True True False False False False False False False experiments may impact program school policy group research experimental evaluation 2 False True False False False False False False False
155 Logic models for program design, implementation, and evaluation: Workshop toolkit Shakman, K, & Rodriguez, S. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast & Islands developed this toolkit to help practitioners use logic models as a tool for building a relevant evaluation design or working more effectively with evaluators whom they engage to conduct evaluations on their behalf. The toolkit describes two workshops, one (two hours long) to learn about logic models, and the other (one and half hour long) to move from logic models to program evaluation. Appendix one provides a very simple graphical representation of a logic model (page A-1). Appendix B (page B-1) is a template. Appendix C (page C-1) is a sample. Appendix D (page D-1) is an example of making the connection with a theory of action as part of using a logic model to evaluate a program. This toolkit is a useful resource for practitioners who want to understand how to use a logic model as an effective tool for program or policy design, implementation, and evaluation. It provides an opportunity to practice creating a logic model and using it to develop evaluation questions and indicators of success. Finally, it provides guidance in how to determine the appropriate evaluation for a specific program or policy. 2015 Link to Resource Tool 118 Pages P False False False True True False False False False False False False False False False False False False False False False False False False False False False Education Evaluation, Department of Education, Education Development Center, Institute of Education Sciences, Regional Assistance, program evaluation, t i o, r t m e n t o f E d u c, research-based technical assistance, unbiased large-scale evaluations of education programs, Logic models, National Center, D e p, participant workbook, Associate Commissioner, results of research, facilitator workbook, Regional Educational Laboratory Northeast, Workshop toolkit, Sue Betka, Acting Director, Chris Boccanfuso, Officer, Amy Johnson, Editor, Arne Duncan, Secretary, Karen Shakman, Rodriguez, Sheila, overall purpose, Joy Lesnick, Ruth Curran Neild, appropriate steps, policymakers, educators, practices, widespread dissemination, help practitioners, different elements, implementation, slide deck, ED-IES, Islands, synthesis, NCEE, funds, United States, Contract, design, report, Tools, Overview, REL 3 False False False True False False False False False
156 Logic Models: A Tool for Designing and Monitoring Program Evaluations Lawton, B., Brandon, P. R., Cicchinelli, L., & Kekahio, W. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Pacific developed this tool to help educators understand how to use logic models as they plan and monitor program evaluations using logic models. This tool provides an introduction to how program components are connected using a logic model, and provides an example to demonstrate how an education program’s resources, activities, outputs, and outcomes are illustrated through a logic model. 2014 Link to Resource Tool 5 pages P False False False True True False False False False False False False False False False False False False False False False False False False False False False Evaluation, Outcome 1 False False False True False False False False False
157 Making the Most of Opportunities to Learn What Works: A School District’s Guide. Akers, L.; Resch, A.; & Berk, J. Regional Education Laboratory Program, Institute for Education Sciences The Regional Educational Laboratory program developed this guide to help school district leaders assess the effectiveness of programs through a randomized controlled trial (RCT). The guide provides details on conducting an RCT, describing three key steps: 1) identifying opportunities to conduct an RCT, while minimizing the time and resources required, 2) gauging the feasibility of conducting an RCT, and 3) following the key steps in conducting an RCT. 2014 Link to Resource Guide 6 pages P False False True False False True False False False True True True False True False False False False False False False False False False False False False U.S. Department moderate risk What Works apples-to-apples comparison staggered rollout what works program students district may schools rct group school reading districts 2 False False True False False False False False False
158 Measurement instruments for assessing the performance of professional learning communities Blitz, C. L., & Schulman, R. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Mid-Atlantic developed this tool to help researchers, practitioners, and education professionals plan, implement, and evaluate teacher professional learning communities (PLCs). The PLC-related measurement instruments identified in this project include 31 quantitative and 18 qualitative instruments that assess a range of teacher/principal-, team-, and student-level variables. Each is described in detail in appendix D (pages D-1 to D-52). Appendices B (page B-1) and C (pages C-1 to C-2) are most relevant for users interested in identifying and applying an appropriate measurement instrument that can answer their questions about teacher PLCs. The document suggests proceeding in three steps: step 1) consult appendix B (the PLC logic model) to determine what information you want and for what purpose (planning, implementation, or evaluation); step 2) depending on your goal (planning, implementation, or evaluation), use the table in appendix C to identify the key tasks you are interested in (column 1) and select the key indicators you are most interested in measuring (column 2) — the class or classes of measurement instruments most relevant to measuring the indicators of interest are in column 3; step 3) click on the hyperlink of a specific instrument in appendix D’s table of contents to retrieve the information about the instrument, determine whether it meets your needs, and find out how to obtain a copy. Notably the body of the document is relevant for readers interested in better understanding logic models. 2016 Link to Resource Tool 72 pages P False False False False True False False False False False False False False False False False True False False False False False False False True True False Evaluation, Professional Learning, Teachers 3 True False False True False False False False False
159 Measuring Program Quality, Part 2: Addressing Potential Cultural Bias in a Rater Reliability Exam Richer, A.; Charmaraman, L.; Ceder, I. Afterschool Matters Afterschool Matters developed this guide to provide program implementers and evaluators with an example of a process to validate an observation tool with an eye towards cultural bias. The authors base their discussion on the Assessment of Program Practices Tool (APT). The guide focuses on (1) why cultural bias is an issue; (2) background for rater reliability, rater bias, and the APT; (3) the goals of the study (to develop rater reliability exams and understand what factors affect performance on the exams); (4) the rating process, including selecting videos, language, and master scorers; (5) the pilot and field tests, including participant selection; and (6) data analysis. The paper concludes with findings that although no significant differences were observed among raters of different characteristics, evidence suggests that familiarity (gained by training) with the APT anchors (detailed descriptions of what each point on the rating scale looks like) increases rater accuracy. They recommend further study. 2018 Link to Resource Guide Methods Report 9 P False False False True True False False False False False False True False False False False True False False False True False False False False False False Cultural Differences, Social Bias, Interrater Reliability, Program Evaluation, Measurement Techniques, Evaluation Research, Follow-up Studies, Educational Assessment, Educational Quality, Cultural Relevance, Program Validation, Cohort Analysis, Familiarity, Prior Learning, Evaluation Methods, Test Reliability, Test Validity, Educator Development, Any Education Topic, Recruiting Study Participants, Collecting New Data, Reporting, Interpreting, and Using Findings 2 False False True False True False False False False
160 Methods for Minimizing Differences Between Groups in Quasi-Experimental Designs Parsad, A.; Price, C. Institute of Education Sciences This video discusses methods to help improve causal inference in quasi-experimental evaluations, including methods for forming similar treatment and comparison groups, deciding on the best matching approach for your study, selecting sample characteristics for matching, and meeting the What Works Clearinghouse standards. The introduction recalls why equivalent groups are important and what matching is, provides an outline for the session, and reviews terminology. The video then describes how to select characteristics on which to match (6:45) It examines matching methods – propensity score matching, Mahalanobis distance, many-to-many matching or stratification, and weighting – and discusses matching with and without replacement (13:50). It provides direction on determining which comparison unit is the best match for a treatment unit (28:35). It concludes with a reminder to account for any matching method in analysis and points the audience to additional resources (36:10). 2016 Link to Resource Video Guide 39:29 R True False False False True False True False False True False False False True False False False False False False False False True False False False False References WWC Standards, Any Education Topic, QED, Identifying Comparison Groups, Reducing Bias in Comparison Groups, Student Achievement Measure, 3 True False True False False True False False False
161 Modern Regression Discontinuity Analysis Bloom, H. Journal of Research on Educational Effectiveness MDRC developed this paper to provide researchers with a detailed discussion of the theory and practice of modern regression discontinuity (RD) analysis. The paper briefly chronicles the history of RD analysis and summarizes its past applications; it explains how, in theory, an RD analysis can identify an average effect of treatment for a population and how different types of RD analyses — “sharp” versus “fuzzy” — can identify average treatment effects for different conceptual subpopulations; it introduces graphical methods, parametric statistical methods, and nonparametric statistical methods for estimating treatment effects from RD data plus validation tests and robustness tests for assessing these estimates; it considers generalizing RD findings and presents several views on and approaches to the issue; and it notes some important issues to pursue in future research about or applications of RD analysis. This paper is a valuable resource for researchers who want to use RD analysis for estimating the effects of interventions or treatments. 2009 Link to Resource Methods Report 62 pages R False False False False False False False True False False False False False False False False False False True True True True False False False False False treatment discontinuity regression point cut group effect control average outcomes 3 False False False False True False False False False
162 Module 8, Chapter 1: Introduction to Cluster-Level Assignment What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this series of seven recorded webinars to help researchers design and implement high quality studies that assign clusters, or groups, to conditions instead of individuals, and that are eligible for review using group design standards. The structure of these types of studies requires a special application of the WWC Group Design Standards. Each chapter focuses on a different aspect of the WWC Group Design Standards. Chapter 1 introduces Module 8 by describing when a study is eligible to be reviewed under the cluster design. The WWC recommends viewing the modules in numerical order. 2017 Link to Resource Video 06:00 R True False False False True True True False False True False False False False False False False False False False False False False False False False False None 2 False False False False False True False False False
163 Module 8, Chapter 2: Compositional Changes in Cluster RCTs What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this series of seven recorded webinars to help researchers design and implement high quality studies that assign clusters, or groups, to conditions instead of individuals, and that are eligible for review using group design standards. The structure of these types of studies requires a special application of the WWC Group Design Standards. Each chapter focuses on a different aspect of the WWC Group Design Standards. Chapter 2 presents compositional changes in cluster randomized control trials, specifically discussing differences between initial and analytic samples, both differences in clusters and individuals as they leave or enter the analytic sample, and how this affects the integrity of the random assignment and the rating of the study. The WWC recommends viewing the modules in numerical order. 2017 Link to Resource Video 10:00 R True False False False True True False False False False False False True True False False False False False True False False False False False False False None 2 False False False False False True False False False
164 Module 8, Chapter 3: Overview of Review Process for Cluster Studies What Works Clearinghouse Institute of Education Sciences This video lists the seven steps in the What Works Clearinghouse (WWC) process for reviewing cluster studies. In the first four steps, cluster studies are reviewed for evidence of effects on individuals. If the cluster study cannot satisfy WWC standards for evidence of effects on individuals, then the study is also reviewed for evidence of effects on clusters (steps 5 to 7). A study that does not satisfy WWC standards for evidence of either type of effect does not meet WWC group design standards. 2017 Link to Resource Video 01:00 R True False False False True True True False False True False False True False False False False False False False False False False False False False False None 1 False False False False False False False False False
165 Module 8, Chapter 4: Review for Effects on Individuals What Works Clearinghouse Institute of Education Sciences This video describes the first four steps of the What Works Clearinghouse (WWC) review of a cluster study. These steps examine the credibility of the study’s evidence of effects on individuals. The first three steps determine whether a cluster randomized controlled trial (RCT) is eligible to meet WWC group design standards without reservations. These steps assess three requirements. First, the cluster RCT has to have low cluster-level attrition. Second, it has to limit the risk of bias due to joiners – entrants to the sample after random assignment. Third, it has to have low individual non-response. The fourth step provides one way for other cluster RCTs, those that do not meet these three requirements, and cluster quasi-experimental designs to meet WWC group design standards with reservations. To receive this rating and satisfy WWC standards for evidence of effects on individuals, these studies must establish that the individuals in the analytic sample were equivalent at baseline. Studies that cannot establish equivalence of individuals may still meet WWC group design standards with reservations by satisfying WWC standards for evidence of effects on clusters. The video concludes with a knowledge check. 2017 Link to Resource Video 20:00 R True False False False True True True False False True False False True False False False False False False False False False False False False False False None 2 False False False False False True False False False
166 Module 8, Chapter 5: Review for Effects on Clusters What Works Clearinghouse Institute of Education Sciences This video describes the steps in the What Works Clearinghouse (WWC) review of studies for evidence of effects on clusters. A cluster study that has not satisfied WWC standards for evidence of effects on individuals can meet WWC group design standards with reservations for evidence of effects on clusters if it meets two requirements. First, the individuals in the analytic sample must be representative of the clusters. If the analytic sample includes only a small fraction of the individuals in the clusters, a study cannot satisfy WWC standards for evidence of effects on clusters. This representativeness requirement is assessed in step 5. Second, the study must either be a randomized controlled trial with low cluster-level attrition, which is assessed in step 6, or demonstrate equivalence of the analytic sample of clusters, as assessed in step 7. 2017 Link to Resource Video 14:00 R True False False False True True True False False True False False True False False False False False False False False False False False False False False None 2 False False False False False True False False False
167 Module 8, Chapter 6: Other Considerations in Cluster Studies What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this series of seven recorded webinars to help researchers design and implement high quality studies that assign clusters, or groups, to conditions instead of individuals, and that are eligible for review using group design standards. The structure of these types of studies requires a special application of the WWC Group Design Standards. Each chapter focuses on a different aspect of the WWC Group Design Standards. Chapter 6 discusses the implications for bias, study rating, and WWC reports of findings when a researcher excludes sample units from analysis . The WWC recommends viewing the modules in numerical order. 2017 Link to Resource Video 07:00 R True False False False True True False False False False False False True True False False False False False False False False False False False False False None 2 False False False False False True False False False
168 Module 8, Chapter 7: Wrap-up and Knowledge Checks What Works Clearinghouse Institute of Education Sciences This video provides a series of knowledge checks related to the What Works Clearinghouse (WWC) group design standards. Each question is based in an example study and each answer comes with a justification. The video ends with a short overview of the WWC process to identify and review cluster studies. 2017 Link to Resource Video 13:00 R True False False False True True True False False True False False True False False False False False False False False False False False False False False None 2 False False False False False True False False False
169 Multiple Regression as a Practical Tool for Teacher Preparation Program Evaluation Williams, C. Journal of Case Studies in Accreditation and Assessment The Journal of Case Studies in Accreditation and Assessment developed this report to help program evaluators understand the usefulness of regression analysis as a practical, quantitative evaluation method. The study describes its theoretical program evaluation framework, the input-environment-output theory regarding education (pages 2 and 3) and a graphic representation on page 30. The model and methodological considerations are presented on pages 9-12 and include power, sample size, effect size, and statistical significance; replicability and reliability; normality, omitted, confounding, and missing variables; the choice of interpretation of bivariate correlations rather than structure coefficients; suppressor variables; outcome selection; participants, instrumentation, variables, and data collection; and threats to internal and external validity. 2012 Link to Resource Methods report 7 pages R False False False True True False False False False False False False False False False True True False True True True False False False True False False Program Evaluation, Accountability, Assessment, Teacher Preparation, Multiple Regression, Higher Education 2 True False False False True False False False True
170 National Board Certification as Professional Development: What Are Teachers Learning? Lustick, D.; Sykes, G. Education Policy Analysis Archives This Education Policy Analysis Archives article evaluates the contribution of the professional development aspect of the National Board Certification to teacher learning, providing an example evaluation approach in the process. The report focuses on handling the biases introduced by the fact that teachers self-select into this certification process. It leverages the Recurrent Institutional Cycle Design (RICD), a quasi-experimental design that accounts for the voluntary, self-selected nature of the subjects’ participation while maintaining the pre-post collection of data. RICD helps control for non-random threats to internal validity while providing a means of establishing some degree of causality between the treatment and observed results. The report also discusses how to use interview data to measure evidence of learning. Appendix A lists limitations of the research and ways they could have been mitigated with appropriate resources. This report is a valuable resource for program evaluators who are concerned with self-selection bias and those planning to collect data through interviews. 2006 Link to Resource Methods report 46 pages R False False False True True False True False False True False True False True False False True False True True True False False False True False False Intervention, National Standards, Young Adults, Program Effectiveness, Teacher Certification, Faculty Development, Teacher Competencies, Interviews, Science Instruction, Knowledge Base for Teaching, Teacher Effectiveness, Teaching Skills, Science Teachers, Outcomes of Education, Teacher Improvement, Interrater Reliability, Inquiry, Portfolio Assessment 3 True False False False True False False False False
171 New Empirical Evidence for the Design of Group Randomized Trials in Education Jacob, R.; Zhu, P.; & Bloom, H.S. MDRC This MDRC working paper is a practical guide for designing group randomized studies to measure the impacts of educational interventions. The paper provides information about the values of parameters that influence the precision of impact estimates (intra-class correlations and R-squared values), and includes outcomes other than standardized test scores and data with a three-level, rather than a two-level, structure. The paper also discusses the role of error in estimates of design parameters and the implications error has for design decisions. 2009 Link to Resource Methods Report 57 pages R False False False False True True False False False False True False False False True True False False True True True False True True False False False not estimable Pediatric Symptom Symptom Checklist gene reliability Sage Publications Students Per randomized square root level class intra school schools outcomes correlations sample number correlation 3 True False False False True False False False False
172 Opportunities for Teacher Professional Development in Oklahoma Rural and Nonrural Schools Peltola, P.; Haynes, E.; Clymer, L.; McMillan, A.; Williams, H. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southwest developed this report to describe the findings from a principal survey on opportunities for teacher professional development in Oklahoma rural and non-rural schools. The relevant sections of the report are the appendices. Appendix A describes survey development and administration, sample selection, data file processing, and an analysis of nonresponse bias. Appendix B provides the survey. 2009 Link to Resource Tool 50 pages P False False False True True False False False False False False True False False True False True False False True False False False False True False False Rural Education, Rural Schools, Schools: Characteristics of, Teachers, Characteristics of, Professional Development 3 True False False True False False False False False
173 Other WWC Review Issues: Module 5, Chapter 6, WWC Training What Works Clearinghouse Institute of Education Sciences This video describes how the outcome measure standards apply to baseline measures. For example, pre-intervention measures must satisfy the same reliability criteria as outcome measures. 2016 Link to Resource Video 2 min R True False False False True False False False False False False False False False False True False False False False False False False False False False False None 1 False False False False False True False False False
174 Over alignment and Data Collection Differences: Module 5, Chapter 5, WWC Training What Works Clearinghouse Institute of Education Sciences The video discusses the problems associated with (1) over alignment of a measure with the intervention and (2) when a measure is collected differently from the intervention and the comparison group. A measure is considered to be over aligned to the intervention when it has been specifically tailored to the intervention. In these cases, the measure can provide an unfair advantage to the intervention group. Measures collected differently between intervention and comparison groups, such as different modes, timing, administering personnel, or construction of scored, will result in a ‘does not meet WWC standards’ rating. 2016 Link to Resource Video 4 min P True False False False True False False False False False False False False False False True False False False False False False False False False False False None 1 False False False False False True False False False
175 Partial Identification of Treatment Effects: Applications to Generalizability Chan, W. Society for Research on Educational Effectiveness This resource is designed to help researchers address the lack of generalizability of RCT results under some circumstances. Generalizability requires that samples and populations be similar. Methods that address these issues tend to require multiple assumptions, some of which cannot be tested. This paper proposes an approach that uses interval estimation or partial identification instead of point estimation to estimate treatment effects without the need for assumptions. The author derives partial identification bounds on the expected treatment effect for a population under three different frameworks: 1) Worst-case framework with no assumptions on the data; 2) Treatment optimization where subjects select the treatment that optimizes the expected outcome, and 3) Monotone treatment response where the response function is assumed to be weakly increasing in the treatment. 2016 Link to Resource Brief or summary Methods report 8 pages R False False False False True True False False False False False False False False False False False False True True False False True False False False False Generalization, Intervention, Validity, Identification, Guidelines, Outcomes of Treatment, Evidence Based Practice, Benchmarking, Language Arts, English, Mathematics Achievement, Achievement Tests, State Standards, Elementary School Students, Middle School Students 2 True True False False True False False False False
176 Partially Nested Randomized Controlled Trials in Education Research: A Guide to Design and Analysis Lohr, S.; Schochet, P.Z.; & Sanders, E. Institute of Education Sciences An Institute of Education Sciences report for applied education researchers to provide guidance to education researchers on how to recognize, design, and analyze data from partially nested, randomized controlled trials to rigorously assess whether an intervention is effective. The paper addresses design issues, such as possibilities for random assignment, cluster formation, statistical power, and confounding factors that may mask the contribution of the intervention. The paper also discusses basic statistical models that adjust for the clustering of treatment students within intervention clusters, associated computer code for estimation, and a step-by-step guide, using examples, on how to estimate the models and interpret the output. 2014 Link to Resource Methods Report 179 pages R False False False False False True True True False True True False True True False True False False True True False False False False False False False 21st Century Cov Parm Deviation Component softball team treatment group students control school design rct squared variance sigma squared control group 3 False False False False True False False False False
177 Peer Evaluation of Teachers in Maricopa County’s Teacher Incentive Fund Program Milanowski, A.; Heneman, H.; Finster, M. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to help program designers and evaluators implement a peer observation system. The guide describes a peer observation system in Arizona, including advantages, challenges, and mitigation strategies; peer evaluator recruitment, selection, training, evaluation, and compensation; initial perceived effects; and issues of costs and sustainability. This guide is a valuable resource for program directors who want to include peers among their evaluators to increase the total number of evaluators or reduce the time burden on current evaluators. The guide is also useful for evaluators who want to leverage teachers to collect data on educator evaluations. 2015 Link to Resource Guide 16 pages P False False False True True False False False True False False True False False False False True False False False True False False False True False False None 2 True False True False False False False False False
178 Performance Management: Aligning Resources to Reforms Reform Support Network Reform Support Network The Reform Support Network (RSN) developed this document as part of a series designed to help education agencies pursue performance management of their key education reforms. Performance management is a systematic approach to ensure quality and progress toward organizational goals by aligning structures, processes, and routines through a set of reinforcing activities that enable an agency to methodically and routinely monitor the connection between the work underway and the outcomes sought. This brief addresses “alignment of resources,” i.e., how to direct or redirect resources (time, money, technology and people) to priority efforts that produce results and establish clear roles and responsibilities. Alignment of resources is the second of the four elements of performance management described in a Sustainability Rubric that RSN created to help education agencies improve their performance management practices (the rubric’s three categories are system capacity, performance management, and context for sustaining reform; the performance management category includes four elements: clarity of outcomes and theory of action, alignment of resources, collection and use of data, and accountability for results). The rubric offers a template through which agencies can identify key elements of sustainability and assess strengths and weaknesses to address in their sustainability planning. While developed for state education agencies (SEAs), the rubric is a valuable resource for policymakers and practitioners who want to gauge how well they are answering key questions related to aligning resources for student improvement. The brief also describes early, promising work from Hawaii and Tennessee that embodies the basic elements of performance management. 2014 Link to Resource Guide 4 pages P False False False False False False False False False False False False False False False False False False False False False False False False False False False effective teachers This brief must have produce results complex area staff members education agencies Harris Top States complex areas program study control group treatment sample members assignment random outcomes 1 False False True False False False False False False
179 Performance Management: Setting Outcomes and Strategies to Improve Student Achievement Reform Support Network Reform Support Network The Reform Support Network (RSN) developed this document as part of a series designed to help education agencies pursue performance management of their key education reforms. Performance management is a systematic approach to ensure quality and progress toward organizational goals by aligning structures, processes, and routines through a set of reinforcing activities that enable an agency to methodically and routinely monitor the connection between the work underway and the outcomes sought. This brief addresses “clarity of outcomes and theory of action,” i.e., how to establish and widely communicate priorities and set ambitious, clear, and measurable goals and outcomes with aligned strategies and activities. Clarity of outcomes and theory of action are the first of the four elements of performance management described in a Sustainability Rubric that RSN created to help education agencies improve their performance management practices (the rubric’s three categories are system capacity, performance management, and context for sustaining reform; the performance management category includes four elements: clarity of outcomes and theory of action, alignment of resources, collection and use of data, and accountability for results). The rubric offers a template through which agencies can identify key elements of sustainability and assess strengths and weaknesses to address in their sustainability planning. While developed for state education agencies (SEAs), the rubric is a valuable resource for policymakers and practitioners who want to gauge how well they are answering key questions related to clarifying expected outcomes for student improvement. The brief also describes early, promising work from Massachusetts and Tennessee that embodies the basic elements of performance management. 2014 Link to Resource Guide 6 pages P False False False False False False False False False False False False False False False False False False False False False False False False False False False Delivery Unit Strategic Plan after high This brief grade reading grade look at Kang aligned strategies must have outcomes plan strategic student performance four management goals Tennessee education 2 False False True False False False False False False
180 Planning for High-Quality Evaluation of Professional Learning Bocala, C., Bledsoe, K. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast & Islands developed a series of three webinars to help practitioners design and implement high-quality evaluations of educator professional development programs. Session 1 defines program evaluation, its purpose, its types (formative and summative), and steps in the process. It provides guidance on how to collaborate with stakeholders. It introduces logic models and includes a template for them. Finally, it explores how to develop evaluation questions. 2017 Link to Resource Webinar Guide 01:21:00 P False False False True True False False False False False False False False False False False False False False False False False False False False False False None 3 False False True False False False True False True
181 Planning for High-Quality Evaluation of Professional Learning, Session 2 Bocala, C., Bledsoe, K. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast & Islands developed a series of three webinars to help practitioners design and implement high-quality evaluations of educator professional development programs. Session 2 explains how to craft evaluation questions, leveraging logic models, and describes types of study designs available to answer different types of questions. It presents various data collection methods, quantitative and qualitative, and their appropriate uses. Finally, it providing guiding questions to assess the utility, feasibility, propriety, and accuracy of collected data. 2017 Link to Resource Webinar Guide 01:23:00 P False False False True True False False False False False False False False False False False True False False False False False False False False False False None 3 False False True False False False True False True
182 PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-experimental Design Studies Maynard, R.A., & Dong, N. Journal of Research on Educational Effectiveness This paper complements exisitng power analysis tools by offering tools to compute minimum detectable effect sizes (MDES) and estimating minimum required sample sizes (MRSS) for studies under design. The tools that accompany the paper supports estimates of MDES or MSSR for 21 different study designs. 2013 Link to Resource tool guide 44 R False False False False False True True False False False True False False False False False False False False False False False False False False False False power analysis, minmum detectable effect, MDE, sample size 3 False False True True False False False False True
183 Practical Resource Guide for Evaluating STEM Master Teacher Programs Frechtling, J. Teacher Incentive Fund The Teacher Incentive Fund developed this guide to help grantees develop evaluations. It is written in the context of programs to improve science, technology, engineering, and mathematics education but is relevant to a broad range of topics. The guide focuses on key aspects of the evaluation process, including identifying evaluation questions, selecting data collection approaches, developing the evaluation design and analysis plan, and disseminating findings. This guide is a valuable resource for program managers, project implementation teams, and their evaluators who want to understand how to design a high quality evaluation. The guide is also useful for seasoned evaluators who want to introduce practitioners to evaluation. 2014 Link to Resource Guide 22 pages P False False False True True True True False False False False False False False False False False False True False True False False False False False False None 3 False False True False False False False False True
184 Practice Guide Level of Evidence What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this video to help researchers understand the levels of evidence associated with educational interventions included in the WWC practice guides. Practice guides provide educators with evidence-based practices, recommendations, and concrete steps they can implement in classrooms and schools to tackle current challenges. Each recommendation is assigned a level of evidence that characterizes the research supporting it and the extent to which improved outcomes for students can be attributed to the recommended practice. The video describes the three levels of evidence (00:27), how they are determined (1:03), what earns a recommendation a higher rating (01:45), and what a lower rating means (02:36). Lower ratings are common and mean that the intervention is supported by research but that further research is needed to examine its effects. The video concludes with a description of the information in practice guides’ appendices on ratings and underlying studies (03:19). 2017 Link to Resource Video 4:07 P True False False False True False False False False False False False False False False False False False False False True False False False False False False References WWC Standards, Any Education Topic, Initial Planning, Reporting, Interpreting, and Using Findings 1 False False False False False True False False True
185 Precision Gains from Publically Available School Proficiency Measures Compared to Study-Collected Test Scores in Education Cluster-Randomized Trials Deke, J.; Dragoset, L.; & Moore, R. Institute of Education Sciences The Institute of Education Sciences developed this report to help researchers determine whether pre-test scores or publicly available, school-level proficiency data yield the same precision gains in randomized controlled trials. The paper defines the key measures used and describes the data sources. It compares the precision gains associated with study-collected data of students’ test scores to the gains associated with school-level proficiency data. It explores approaches to reducing the costs of collecting baseline test scores. Finally, it examines the potential implications for attrition bias of not collecting pre-test data. This series is a valuable resource for those who fund and commission research, as publicly available, school-level proficiency data may be available at lower cost than pre-test scores are. 2010 Link to Resource Methods Report 44 pages R False False True False True True False False False False False False True False True True False False True True False False True False False False False Publically Available Random Assignment School-Level proficiency Bandeira de Effect Size New York Percent Subsample School-Level de Mello level school test proficiency study data baseline students model 3 True False False False True False False False False
186 Professional Experiences of Online Teachers in Wisconsin: Results from a Survey about Training and Challenges Zweig, J.; Stafford, E.; Clements, M.; & Pazzaglia, A. M. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Midwest developed this report to increase policymakers’ and educators’ knowledge of the range of professional learning experiences available to online teachers and the challenges they face teaching online. The survey instrument (Appendix A) is a valuable resource for state online learning programs, state departments of education, and employers of online teachers who want to examine the training and professional development provided to online teachers in their jurisdictions. The data and methodology section (Appendix D) is a useful example of how to develop a survey, administer it, and collect and analyze the data. The body of the report focuses on the results of the survey, which was administered to a virtual school in Wisconsin, and information about how online teacher training can vary in timing, duration, format, and content. In the current climate, in which few states have specific requirements about training for online teachers, these findings may interest policymakers; district and school administrators; and parents in other states, districts, and schools. 2015 Link to Resource Tool Guide 40 pages P False False False True False False False False False False False False False False False True True False False True True False False False True False False Online Courses, Teaching Experience, Teacher Surveys, Training, Teacher Competencies, Barriers, Teacher Education, Faculty Development, Student Behavior, Educational Methods, Elementary School Teachers, Secondary School Teachers, Technology Uses in Education, Academic Persistence, Student Responsibility, Act 222 Advanced Placement development little guidance grade levels professional working conditions online checkbox professional development percent teachers school training virtual Wisconsin 3 True False True True False False False False False
187 Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective Interventions. Report to Congressional Requesters. GAO-10-30 Kingsbury, N. U.S. Government Accountability Office The U.S. Government Accountability Office (GAO) initially developed this report to help congressional requesters choose which federal social programs to fund. After the private, nonprofit Coalition for Evidence-Based Policy undertook the Top Tier Evidence initiative to help federal programs identify interventions for which randomized experiments (REs) show sizable, sustained benefits to participants or society, GAO was asked to examine (1) the validity and transparency of the Coalition’s process, (2) how its process compared to that of six federally supported efforts to identify effective interventions, (3) the types of interventions best suited for assessment with randomized experiments, and (4) alternative rigorous methods used to assess effectiveness. The report assesses the Top Tier initiative process and standards, confirms that REs can provide the most credible evidence of effectiveness under certain conditions, and describes available rigorous alternatives (quasi-experimental comparison groups including regression discontinuity designs, statistical analyses of observational data including interrupted time-series, and in-depth case studies). This report is a valuable resource for researchers who aim to demonstrate program effectiveness using a variety of rigorous methods, especially as they seek to understand when a RE is necessary versus when other methods are suitable for demonstrating program effectiveness. 2009 Link to Resource Methods Report 49 pages R True False False False False True True True True True False False False True False False False False True True False False False False False False False Nancy Kingsbury Task Force Academies Press Conference proceedings Steps Seven broadcast media remain separate Major journals Nominations solicited evidence interventions intervention program evaluation studies top tier effectiveness outcomes 3 False False False False True False False False False
188 Properties of the Multiple Measures in Arizona’s Teacher Evaluation Model. REL 2015-050 Lazarev, V.; Newman, D.; Sharp, A. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) West developed this report to explore the relationship among assessments made through teacher observations, student academic progress, and stakeholder surveys to investigate how well the model differentiated between high- and low-performing schools. The study found only a few correlations between student academic progress and teacher observation scores, suggesting that a single aggregated observation score might not adequately measure independent aspects of teacher performance. This report can be useful to evaluators and education stakeholders who are considering using teacher observation scores as an evaluation measure or wish to gauge the quality of their agency’s educator evaluation system. In addition, appendix E in the report provides instructional information on how to detect nonlinear relationships between observation item scores and student academic progress metrics. 2014 Link to Resource Methods Report 42 pages R False False False True True False False False False False False False False False False True False True False True False False False False True False False Teacher Evaluation, Observation, State Departments of Education, Correlation, Teacher Competencies, Academic Achievement, Teacher Influence, Scores, Surveys, Mathematics Tests, Reading Tests, Mathematics Achievement, Reading Achievement, Teacher Effectiveness, Teacher Characteristics, Evaluation Methods, Public Schools, School Districts, Charter Schools, Planning, Classroom Environment, Teaching Methods, Teacher Responsibility, Standardized Tests, Elementary Schools, Secondary Schools, Grade 3, Grade 4, Grade 5, Grade 6, Grade 7, Grade 8, Grade 9, Grade 10, Grade 11, Grade 12, Student Attitudes, Teacher Attitudes, Parent Attitudes 3 True False False False True False False False False
189 Protocol for a Systematic Review: Teach For America (TFA) for Improving Math, Language Arts, and Science Achievement of Primary and Secondary Students in the United States: A Systematic Review Turner, H.; Boruch, R.; Ncube, M; & Turner, A. The Campbell Collaboration This Campbell Collaboration protocol provides researchers with an example of a proposal for a systematic review of a program. The purpose of a systematic review is to sum up the best available research on a question, which is done by synthesizing the results of several studies. A systematic review uses transparent procedures to find, evaluate, and synthesize the results of relevant research. Procedures are explicitly defined in advance to ensure that the exercise is transparent and can be replicated. This practice is also designed to minimize bias. Studies included in a review are screened for quality, so that the findings of a large number of studies can be combined. Peer review is a key part of the process; qualified independent researchers check the author’s methods and results. This protocol is a valuable resource for researchers who want guidance on structured guidelines and standards for summarizing research evidence. This example focuses on the Teach For America peer preparation program and its effect on student academic outcomes. As such, the protocol is also useful for professionals interested in the strategies and goals of interventions that aim to recruit teachers and improve their effectiveness. 2016 Link to Resource Guide 33 pages R False False False True False False False False False False False False False False False True False False True True True False True False True False False Language Arts Postal Code USA Phone accept responsibility minimum dosage United States absolute value confidence intervals take place Mackson Ncube tfa studies review effect study analysis outcome research teacher outcomes 3 True False True False False False False False False
190 Quasi-experimental Designs: Module 1, Chapter 3, WWC Training What Works Clearinghouse Institute of Education Sciences This video provides the WWC definition of quasi-experimental design (QED), and describes designs that are potentially acceptable to WWC. All eligible QEDs must compare the intervention group with distinct (non-overlapping) groups that are determined by a non-random process. Comparison groups can be determined through a convenient sample, national rates, or by using statistical methods. The highest WWC rating that a QED can receive is ‘Meets Standards With Reservations.” 2016 Link to Resource video 3 min P True False False False True False True False False True False False False False False False False False False False False False False False False False False None 1 False False False False False True False False False
191 Randomized Control Trials: Module 1, Chapter 2, WWC Training What Works Clearinghouse Institute of Education Sciences This video goes into detail on what WWC considers as ‘well executed randomization’ and provides examples of randomization that are well executed, and those that are not well executed. For example, choosing students by name, birth date, or class schedule is not considered randomization. The video also discusses how randomization can be compromised. For example, if a student is originally assigned to control, even if s/he is later moved into the intervention, the study must treat this student as a ‘control’ student. 2016 Link to Resource video 16 min P True False False False True True False False False True False False False False False False False False False False False False False False False False False None 2 False False False False False True False False False
192 Rapid Cycle Evaluation: Race to the Top-District Tools & Strategies Ceja, B.; & Resch, A. Reform Support Network The Reform Support Network (RSN) developed this webinar to help practitioners evaluate personalized learning tools using Rapid Cycle Evaluation and Opportunistic Experiments. The webinar first recognizes how these quick turnaround methods can be used to build knowledge in a context of scarce resources. It defines these methods, the conditions under which they are appropriate, and the identification of comparison groups. The webinar then goes into an example using Cognitive Tutor, a software intended to improve high school student math achievement. It focuses on an opportunity for Race to the Top grantees to apply for the RSN to conduct an evaluation, but this webinar is also a valuable resource for practitioners who want to understand how the results of quick turnaround experiments might inform the continued implementation of learning tools and/or future purchasing decisions. 2015 Link to Resource Webinar 27 slides; 55 minutes P True True False False False False False False False True False True False True True True False False False False False False True False False False False Leadership: Evaluation of Program Outcome (Revisions); Program Evaluation, random assignment Cycle Evaluation Improve achievement Rapid Cycle Cognitive Tutor comparison group can be subset of or schools rsn cognitive comparison students 3 True False False False False False True False False
193 Recognizing and Conducting Opportunistic Experiments in Education: A Guide for Policymakers and Researchers. REL 2014-037 Resch, A.; Berk, J.; & Akers, L. Institute of Education Sciences National Center for Education Evaluation and Regional Assistance An Institute of Education Sciences-developed guide designed to help researchers recognize and conduct opportunistic experiments. The authors begin by defining opportunistic experiments and providing examples, then discuss key steps and issues to consider when choosing potential experiments. The authors also outline the critical steps to successfully completing an opportunistic experiment and the potential to conduct such studies at a relatively low cost. 2014 Link to Resource Guide 28 pages P False False True False False True False False False True False True False True False True False False False False True False True False False False False Pell Grants Power Program Mathematica Policy Works Clearinghouse random causal chain Regional Assistance Completion Project Data use schools data research random assignment district opportunistic study researchers intervention 3 True False True False False False False False False
194 Recognizing Opportunities for Rigorous Evaluation Tuttle, C. Regional Education Laboratory Program, Institute of Education Sciences and Mathematica Policy Research The Regional Educational Program (REL) developed a video series to help schools, districts, states, and their research partners use a cost-effective approach, known as “opportunistic experiments,” to test the effectiveness of programs. This video describes key characteristics of opportunistic experiments and introduces situations that provide good opportunities for embedding experiments in existing programs with minimal disruption. The video will be useful to professionals considering rigorous evaluation that will generate evidence for informing their education decisions in a cost-effective manner. 2016 Link to Resource Video 8 min P False False True False False True False False False False False False False False False False False False False False False False False False False False False None 2 False False False False False True False False True
195 Redesigning Teacher Evaluation: Lessons from a Pilot Implementation Riordan, J.; Lacireno-Paquet, N.; Shakman, K.; Bocala, C.; Chang, Q. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast & Islands developed useful surveys as part of this report, which describes a study of the implementation of new teacher evaluation systems in New Hampshire’s School Improvement Grant (SIG) schools. They find that, while the basic system features are similar across district plans, the specifics of these features vary considerably by district. Key findings are listed on the second half of page i. The relevant section is in appendix D (pages D-1 to D-11), which provides survey protocols. The goal of the surveys is to understand what is working well and what is challenging during the pilot implementation of a new evaluation system. There is a teacher professional climate and implementation survey and an evaluator survey. Both ask for background information and about implementation and stakeholder support. The teacher survey also includes questions on the professional school climate (i.e., inclusive leadership, teacher influence, teacher–principal trust, coordination and quality of professional development, reflective dialogue, and focus on student learning). 2015 Link to Resource Tool 11 Pages P False False False False True False False False False False False False False False False False True False False False False False False False True False False Evaluation, Teachers 2 True False False True False False False False False
196 Reporting on Impacts in Anticipation of a WWC Review Anne Wolf, A.; Goodson, B. Institute of Education Sciences This video provides an overview about what to include in a report on the findings from an evaluation or impact study of an education intervention to support a systematic evidence review by the What Works Clearinghouse. It will help researchers minimize misinterpretation during the review process, get the highest rating the study has the potential to receive, and ensure that consumers of the study report can clearly understand and interpret the results. The video can be used as a checklist to review each section of a report. The presenters recognize that researchers may write different reports for other audiences and recall that the WWC is an important audience as it tells readers of the study report how confident they can be about the reported evidence. The first five minutes describe the purpose and organization of the video. Next, it goes into what information to include on: the intervention program and activities, participants, location, time period, and intervention and comparison groups (5:00); the study design, including method and unit of assignment (7:30); outcomes and baseline measures (12:00); evidence that the methodology for data collection was consistent across the intervention and comparison condition (14:00); the analytic approach, including models and handling of missing data (14:35); the baseline equivalence model (20:45); and findings including impacts, attrition, baseline equivalence, and representativeness (22:35). The video concludes at 45:54 with a list of additional resources about reporting. Examples are included. The underlying guidance document can be found at https://fitw.grads360.org/#communities/pdc/documents/15060. 2018 Link to Resource Video Guide 50:57 R True False False False True True True False False True False False True True False True False False False True True False False False False False False References WWC Standards, Any Education Topic, Initial Planning, RCT, QED, Identifying Comparison Groups, Addressing Changes in your Sample, Reducing Bias in Comparison Groups, Selecting Appropriate Outcome Measures, Addressing Analysis Challenges, Reporting, Interpreting, and Using Findings 3 False False True False False True False False True
197 Reporting What Readers Need to Know about Education Research Measures: a Guide Boller, K.; & Kisker, E.E. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) program developed this guide to help researchers make sure that their research reports include enough information about study measures to allow readers to assess the quality of the study’s methods and results. The guide provides five checklists to assist authors in reporting on a study’s quantitative measures, a sample write-up to illustrate the application of the checklists, and a list of resources to further inform reporting of quantitative measures and publication of studies. The checklists address the following measurement topics: (1) measure domains and descriptions; (2) data collection training and quality; (3) reference population, study sample, and measurement timing; (4) reliability and construct validity evidence; and (5) missing data and descriptive statistics. 2014 Link to Resource Guide Tool 26 pages R True False False False False False False False False False False False False False False False False False False False True False False False False False False Child Trends Early Childhood overly aligned New York Certification criteria Standards Handbook measures reliability study evidence training 3 False False True True False False False False False
198 Request for Proposals Evaluation Guide Reform Support Network Reform Support Network The Reform Support Network developed this guide to help state and local education agencies in defining the evaluation process for a Request for Proposal (RFP). The guide addresses the major components of an RFP, including involving stakeholders in its evaluation, scoring proposals, considering costs, planning vendors’ demonstrations, checking references, and establishing timelines. Tools valuable to education agencies include sample-scoring matrices for different components and overall, a checklist for the agency to evaluate vendors’ written proposals and one for vendors to identify the extent to which they meet key functional requirements, a committee signoff sheet, and a sample evaluation timeline. 2012 Link to Resource Guide 12 pages P False False False False False False False False False False False False False False False False False False False False False False False False False False False Scoring Matrix rather than Technical local education Rank Order Final Rank Functional Requirements vendor evaluation rfp cost agency team score phase 2 False False True False False False False False False
199 Retaining High Performers: Insights from DC Public Schools’ Teacher Exit Survey Pennington, K.; Brand, A. Bellwether Education Partners Bellwether Education Partners developed this tool to help practitioners and researchers better understand how to retain high-performing teachers. The relevant sections of this slide presentation are pages 8-10 and 36 which list survey questions to ask teachers on why they leave their school district, what might have retained them, and what their characteristics are. This tool is available to practitioners and researchers want to collect data on teachers’ experiences. 2018 Link to Resource Tool Slide Presentation 4 P False False False False True False False False False False False False False False False False True False False False False False False False True False False Collecting New Data, Teacher Measure 1 True False False True False False False True False
200 Retaining K-12 Online Teachers: A Predictive Model for K-12 Online Teacher Turnover Larkin, I.; Lokey-Vega, A.; Brantley-Dias, L. Journal of Online Learning Research The authors developed this tool to help practitioners and researchers better understand the factors that influence teacher job satisfaction and teacher commitment to their job. The relevant section of this paper is the survey tool in Appendix A. It includes questions about teacher demographic characteristics, education, past and current experience, workload, job satisfaction, organizational commitment, and turnover intention. 2018 Link to Resource Tool 9 P False False False False True False False False False False False False False False False False True False False False False False False False True False False Elementary School Teachers, Secondary School Teachers, Online Courses, Teacher Persistence, Labor Turnover, Job Satisfaction, Teacher Surveys, Teacher Attitudes, Intention, Scheduling, Mentors, Teacher Student Ratio, Teaching Conditions, Teaching Experience, Predictor Variables, Regression (Statistics), Statistical Analysis, Collecting New Data, Teacher Measure 2 True False False True False False False False False
201 Review Issues Related to Attrition: Module 2, Chapter 4, WWC Training What Works Clearinghouse Institute of Education Sciences This video discusses how ways in which researchers exclude sample members may or may not cause an attrition bias. This video also discusses in what conditions ‘nonparticipation’ is or is not considered attrition. If researchers continue to track individuals who did not participate, and keep them in their originally assigned condition, then it is not considered attrition. 2016 Link to Resource Video 10 min P True False False False True True False False False False False False True False False False False False False False False False False False False False False None 2 False False False False False True False False False
202 Review Protocol For Teacher Training, Evaluation, And Compensation Version 3.2 (July 2016) What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this protocol to help program directors and evaluators select WWC-appropriate outcome measures. It defines key terms including categories of relevant research (preparation, induction, evaluation, compensation, and professional development). It describes the procedures for conducting a literature search, including search terms and searched databases. It lists eligibility criteria for populations, interventions, research, and student and teacher outcomes. It provides the eligible standards against which studies are reviewed: sample attrition, baseline equivalence, and outcomes. 2016 Link to Resource Guide 15 pages R False False False True True True True True False True False False True False False True False False False False False False True True True False False None 2 True False True False False False False False False
203 Rigorous Program Evaluations on a Budget: How Low-Cost Randomized Controlled Trials Are Possible in Many Areas of Social Policy Coalition for Evidence-Based Policy Coalition for Evidence-Based Policy The Coalition for Evidence-Based Policy developed this guide to illustrate the feasibility and value of low-cost randomized controlled trials (RCT) for policy officials and researchers, by providing concrete examples from diverse program areas. The guide provides a brief overview of what RCTs are and how to reduce their cost. It then presents five examples of quality, low-cost RCTs in four areas: criminal justice, child welfare, a community-wide parenting intervention, and a teacher-incentive program. This guide is a valuable resource for policymakers, researchers, and others who automatically discard RCTs as an approach because of the perception that such studies are always too costly and too administratively burdensome on schools to be practical. 2012 Link to Resource Guide Brief or Summary 10 pages P False False True False True True False False False False False False False False True False False False False False False False True True False False False Task Force Waiver Demonstration Community Supervision New York Recovery Coach Services Task carried out program study cost data group low control administrative outcomes randomized 2 True True True False False False False False False
204 RTT-D Guidance: Implementing Performance Metrics for Continuous Improvement that Support the Foundational Conditions for Personalized Learning Johnson, J.; Kendziora, K.; & Osher, D. American Institutes for Research The American Institutes for Research developed this guide to help districts implement a continuous improvement process to personalize education for all students in their schools, focusing on classrooms and the relationship between educators and students. The guide begins by recalling the three foundational conditions for personalized learning – building the capacity of teachers to create highly effective classroom learning environments, developing a rigorous capacity for student support, and establishing the organizational efficacy of leadership and management to implement personalized learning environments. It then describes an approach to continuous improvement that builds on the conditions for learning, personalization, and achievement for all students. The guide proposes a key set of performance metrics and summarizes them in a table on pages 21-23. The appendix presents a sample on-track measure. This guide is a valuable resource for district and school staff who want to gauge how well they provide teachers with the information, tools, and supports that enable them to meet the needs of each student and substantially accelerate and deepen each student’s learning. Through this guide, district and school staff can gauge whether they have the policies, systems, infrastructure, capacity, and culture to enable teachers, teacher teams, and school leaders to continuously focus on improving individual student achievement and closing achievement gaps; whether they set ambitious yet achievable performance measures that provide rigorous, timely, and formative information; and the extent to which implementation is progressing. 2012 Link to Resource Guide 40 pages P False True False False True False False False False False False False False False False True True False False False False False True True True True False Civil Rights Disease Control Healthy Kids Longer Term Term Sustainability juvenile justice Action Brief BMI school students student track learning received classes above high 3 True False True False False False False False False
205 Seeing It Clearly: Improving Observer Training for Better Feedback and Better Teaching Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide focuses on training observers of educators being evaluated. The guide describes how to recruit and prepare observers; what skills to develop; and how to use data to monitor implementation. Each section provides guiding questions on how to lay the foundation and how to improve. The guide also includes a checklist, tips, tools from districts and states, and a planning worksheet. This guide is a valuable resource for program directors who want to design and implement a high quality observation system. 2015 Link to Resource Guide Tool 128 pages P False False False True True False False False False False False True False False False False True False False False False False False False False False False None 3 False False True True False False False False False
206 Self-study Guide for Implementing Early Literacy Interventions Dombek, J.; Foorman, B.; Garcia, M.; Smith, K. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southeast developed this guide to help district and school-based practitioners conduct self-studies for planning and implementing early literacy interventions for kindergarten, grade1, and grade 2 students. Although the guide is not designed for rigorous evaluations, it can be used for continuous ongoing improvement. The guide is designed to promote reflection about current strengths and challenges in planning for implementation of early literacy interventions, spark conversations among staff, and identify areas for improvement. Schools and districts interested in implementing an early literacy program can use the guide to help assess needs in areas such as student selection, content and instruction, professional development, communication and other areas of importance. 2016 Link to Resource Guide Tool 27 Pages P False True True False True False False False False False False False False False False False False False False False False False False False False False False Early Interventions, Education Interventions, Literacy 3 False False True True False False False False False
207 Sensitivity Analysis for Multivalued Treatment Effects: An Example of a Cross-Country Study of Teacher Participation and Job Satisfaction Chang, C. Society for Research on Educational Effectiveness This resource illustrates the importance of conducting sensitivity analyses in the context of examining average treatment effects of multiple treatments (multivalued treatments). The study focuses on the sensitivity of a multivalued treatment effect; the degree to which test statistics and sample size affect the Impact Threshold of a Confounding Variable(ITCV) index (the size of the impact of a potential confounding variable that can change an inference); and whether different propensity score weighting methods are sensitive enough to capture different levels of the treatment and allow for robust inferences on each value treatment. 2015 Link to Resource Brief or summary Methods report 17 pages R False False False False True False True False False False False False False False False False False False False True False False False False True False False Job Satisfaction, Intervention, Sample Size, Weighted Scores, Robustness (Statistics), Simulation, Inferences, Surveys, Teacher Attitudes, Cross Cultural Studies, Statistical Analysis, Multivariate Analysis, Validity, Teacher Participation, Likert Scales 2 True True False False True False False False False
208 Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-Cycle Evaluation to Improve Program Development and Outcomes Cody, S.; & Asher, A. Mathematica Policy Research The Mathematica Policy Research report proposes using predictive modeling and rapid-cycle evaluation to improve program services while efficiently allocating limited resources. The authors believe that these methods – both individually and together – hold significant promise to improve programs in an increasingly fast-paced political environment. They propose two actions: first, agency departments with planning and oversight responsibilities should encourage the staff of individual programs to conduct a thorough needs assessment, and second, federal agencies should take broad steps to promote predictive analytics and rapid-cycle evaluation. 2014 Link to Resource Brief or Summary 12 pages R False False False False False False False False False False False False False False False False False False False False False False False False False False False Predictive Analytics, Eligibility Assessment Resources Administration Andrew Asher frontline workers Address Poverty Hamilton Projecrookings U.S. Department Scott Cody decision making program predictive analytics programs data rapid cycle evaluation 2 False True False False False False False False False
209 Standardized Measures: Module 5, Chapter 4, WWC Training What Works Clearinghouse Institute of Education Sciences Standardized tests are considered to have face validity and reliability if administered in the way intended. This video describes how reviewers can identify standardized tests and provides examples of how researchers may deviate from the intended administration of the standardized tests. In these cases, the measure would not be considered reliable. 2016 Link to Resource Video 5 min P True False False False True False False False False False False False False False False True False False False False False False False False False False False None 1 False False False False False True False False False
210 Statistical Adjustments: Module 3, Chapter 4, WWC Training What Works Clearinghouse Institute of Education Sciences This video discusses acceptable methods to adjust for baseline pretest differences when baseline differences are between .05 -.20 standard deviations. Acceptable methods include regression covariate adjustments (including covariates in HLM), and analysis of covariance (ANCOVA). 2016 Link to Resource Video 3 min R True False False False True True True False False True False False False True False False False False False True False False False False False False False None 1 False False False False False True False False False
211 Statistical Power Analysis in Education Research Hodges, L.V.; & Rhoads, C. Institute of Education Sciences An Institute of Education Sciences paper that provides an introduction to the computation of statistical power for education field studies and to discuss research-design parameters that directly affect statistical power. The paper shows how to use the concepts of operational effect sizes and sample sizes to compute statistical power. It also shows how clustering structure, described by intra-class correlation coefficients, influences operational effect size and therefore, statistical power. In addition, this paper details how the use of covariates can increase power by operational effect size. 2010 Link to Resource Methods Report 88 pages R False False False False False True False False False True True False False False False False False False False True False False False False False False False design power effect sample designs level size statistical covariates Hierarchical Potential Conflicts Psychological Methods RR assigned costs while effect randomized-block while maintaining 3 False False False False True False False False False
212 Studying the Sensitivity of Inferences to Possible Unmeasured Confounding Variables in Multisite Evaluations. CSE Technical Report 701 Seltzer, M.; Kim. J.; Frank, K. CRESST/UCLA The University of California, Los Angeles developed this report to help researchers identify and deal with factors that may be confounded with differences in implementation. The report focuses on assessing the impact of omitted confounding variables on coefficients of interest in regression settings in the context of hierarchical models in multisite settings in which interest centers on testing whether certain aspects of implementation are critical to a program’s success. It provides a detailed illustrative example using the data from a study focusing on the effects of reform-minded instructional practices in mathematics. It also addresses how widely applicable their approach to sensitivity analysis is, specifically: in what situations or settings will cluster-level regression analyses yield point estimates and standard errors for fixed effects of interest that are highly similar to those produced via a restricted maximum likelihod/empirical Bayes estimation strategy? This report is a valuable resource for researchers who conduct research in multiple sites and those who implement hierarchical models. 2006 Link to Resource Methods report 42 pages R False False False False True True False False False False False False False False False False False False True True False False True False False False False Program Effectiveness, Teaching Methods, Inferences, Educational Environment, Educational Practices, Educational Experience 3 True False False False True False False False False
213 Supporting a Culture of Evidence Use in Education Institute of Education Sciences Institute of Education Sciences This video describes the different research resources that individuals can access through the Institute of Education Sciences (IES). The video describes the products offered through the What Works Clearinghouse (WWC) and the online search tool for thousands of research studies through the Education Resource Information Center (ERIC). It also provides information about the 10 Regional Education Laboratories (RELS) and Statewide Longitudinal Data Systems (SLDSs). This video will be useful to participants who want to make better use of evidence for education improvement and to leverage IES resources in that effort. 2016 Link to Resource Video 5 min P True False False False True False False False False False False False False False False False False False False False True False False False False False False None 1 False False False False False True False False False
214 Survey methods for educators: Analysis and reporting of survey data (part 3 of 3) Pazzaglia, A. M., Stafford, E. T., & Rodriguez, S. M. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast and Islands developed this guide to help educators analyze survey data and report the results in a way that is accessible and useful to interested stakeholders. This guide will help readers use survey data to inform decisions about policy or practice in schools and local or state education agencies. The document describes a five-step survey analysis and reporting process: step 1 – review the analysis plan, step 2 – prepare and check data files, step 3 – calculate response rates, step 4 – calculate summary statistics, and step 5 – present the results in tables or figures. This guide is the third in a three-part series of survey method guides for educators. The first guide in the series covers survey development, and the second guide in the series covers sample selection and survey administration. 2016 Link to Resource Guide Tool 37 pages P False False False False True False False False False False False False False False False False True True True True True True True False False False False Achievement, Achievement (student), Data Interpretation, States 3 True False True True False False False False False
215 Survey methods for educators: Collaborative survey development (part 1 of 3) Irwin, C. W., & Stafford, E. T. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast and Islands developed a series of three guides to help educators administer surveys to collect unavailable data they need. This guide, the first in the series, focuses on collaborative survey development. The document describes a five-step process: step 1 – identify topics of interest, step 2 – identify relevant, existing survey items, step 3 – draft new survey items and adapt existing survey items, step 4 – review draft survey items with stakeholders and content experts, and step 5 – refine the draft survey using cognitive interviewing. Appendix A (page A-1) provides additional resources for sampling and survey administration, including references to textbooks and professional organizations and university departments with expertise in these topics. Appendix B (pages B-1 and B-2) provides a survey blueprint or table of specifications i.e., a document that outlines the topics and subtopics to be included on a survey and serves as an outline for developing survey items. Appendix C (page C-1) describes how to develop an analysis plan, to be used prior to calculating summary statistics and ensure that the resulting information will help address the survey development team’s topics of interest, and provides a sample. Appendix D (pages D-1 and D-2) provides a sample feedback form, which allowed a survey development team to obtain input on item wording and importance across all stakeholders despite limited time to meet with the stakeholder group. Appendix E (pages E-1 to E-4) provides a sample cognitive interview protocol, which includes is the step-by-step instructions for gathering feedback during cognitive interviews. Appendix F (pages F-1 to F-3) provides a sample codebook i.e., a record of the codes used to categorize feedback gathered during cognitive interviewing. 2016 Link to Resource Guide Tool 33 Pages P False False False False True False False False False False False False False False False False True False False False False False False False False False False Achievement, States 3 False False True True False False False False False
216 Survey methods for educators: Selecting samples and administering surveys (part 2 of 3) Pazzaglia, A. M., Stafford, E. T., & Rodriguez, S. M. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast and Islands developed a series of three guides to help educators administer surveys to collect unavailable data they need. This guide, the second in the series, focuses on selecting a sample and administering a survey. The document describes a five-step sample selection and survey administration process: step 1 – define the population, step 2 – specify the sampling procedure, step 3 – determine the sample size, step 4 – select the sample, and step 5 – administer the survey. Appendices provide steps on how to use Microsoft Excel to obtain a random sample (appendix B on pages B-1 to B-5), and sample survey invitations and reminders (appendix D on pages D-1 to D-11). This guide is a valuable resource for educators who want to understand a larger population by surveying a subgroup of its members. The first guide in the series covers survey development, and the third guide in the series covers data analysis and reporting. 2016 Link to Resource Guide Tool 29 pages P False False False False True False False False False False True True False False False False True False False False False False False False False False False Achievement, Academic Achievement,Achievement (student), Data Interpretation, States 3 False False True True False False False False False
217 Teacher Effectiveness Data Use Regional Education Laboratory Program Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) West developed this video to help district and school leaders build capacity to better understand, analyze, and apply teacher effectiveness data to make strategic decisions regarding teacher assignment, leadership, and professional development, and to facilitate local planning around the use of these data. The video defines different types of effectiveness data, and discusses ways to use data and interpret research. It introduces a five-step process to analyze teacher effectiveness data, which includes: identifying the question, collecting data, organizing data, making data-based decisions, and communicating the data findings and decisions. The video can be used as a working session by using handouts from the Great Teachers and Leaders Supporting Principals Using Teacher Effectiveness Data Module (note that handouts are numbered differently in the video). This video will be helpful to stakeholders interested in making data-based decisions to help increase teacher effectiveness. 2016 Link to Resource Video Guide 25 min P False False False True False False False False False False False False False False False False False False False False True False False False True False False None 3 True False True False False True False False False
218 Teacher evaluation and professional learning: Lessons from early implementation in a large urban district Shakman, K.; Zweig, J.; Bailey, J.; Bocala, C.; Lacireno-Pacquet, N. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast and Islands developed this report to describe the findings from a study of the alignment of educator evaluations and professional development. The relevant sections of the report are the survey and interview tools (Appendices D, F, and G). Appendix D describes the coding dictionary that – a list of professional development activities with descriptions – allowed the study team to summarize and quantify the professional activities in the narratives and to match the activities prescribed to the activities reported in the survey to study the alignment between what evaluators prescribed and what teachers reported. Appendix F lists the survey items, which are about the types of professional development that teachers engaged in during the study period; the standards and indicators that the professional development addressed; the type and duration of the professional development activities; and school climate. Appendix G provides the interview protocol. Teachers were asked about the prescriptions they received, how they were selected, what they did, and their experience with the resulting professional development. Educators were asked similar questions about the prescriptions they wrote, their role in the process, what teachers did and how successfully, and their evaluation of the resulting professional development. These sections are a valuable resource for program directors who want to better gauge the quality of the feedback administrators give teachers regarding how to improve. They also help track whether teachers take the recommended steps and how these may affect subsequent evaluation results. 2016 Link to Resource Tool 8 pages P False False False True True False False False False False False True False False False True True False True False False False False False True False False Teachers, Education/Preparation, Performance Evaluations, Professional Development, Urban Schools 2 True False False True False False False False False
219 Teaching and Learning International Survey (TALIS) 2013: U.S. Technical Report Strizek, G. A.; Tourkin, S.; Erberber, E.; Gonzales, P. National Center for Education Statistics, U.S. Department of Education The National Center for Education Statistics developed this report to provide researchers with an overview of the design and implementation of a large-scale teacher survey that covers professional development, evaluation, school leadership and climate, instructional beliefs, and pedagogical and professional practices. The report focuses on a comprehensive description of survey development and administration, and data processing. It addresses common, important survey considerations such as sampling and non-response. Appendices provide the survey tool and recruitment documents. This series is a valuable resource for program directors and evaluators who want an introduction to or a review of all the steps involved in collecting survey data. The series is also useful for evaluators who want to use the data from this particular survey (the Teaching and Learning International Survey) to answer some of their research questions. 2014 Link to Resource Guide Tool 257 pages R False False False True True False False False False False True True True False True False True False False True False False False False True True False International Comparisons, Labor Force Experiences, Principals, Professional Development, Public Schools, Schools, Secondary Education, Elementary and Secondary Schools Staff, Teachers:Characteristics of, College Major or Minor, Education/preparation, Job Satisfaction, Perceptions and Attitudes 3 True False True True False False False False False
220 Technical Assistance Materials for Conducting Rigorous Impact Evaluations Institute of Education Sciences National Center for Education Evaluation and Regional Assistance Institute of Education Sciences National Center for Education Evaluation and Regional Assistance The National Center for Education Evaluation and Regional Assistance developed this website to help program evaluators design, implement, conduct analyses, and report findings from impact studies. The site links to multiple resources: an evaluation plan template with guidance, an Excel spreadsheet designed to support and supplement the development of an evaluation plan (the contrast tool), and examples for randomized controlled trials (RCTs) and quasi-experiments (QEs); a toolkit to understand and design logic models and use them to plan a program evaluation; resources that support the design and implementation of RCTs and QEs; briefs on baseline equivalence and attrition; and guidance on reporting findings. 2010–2018 Link to Resource Guide Tool None P True False False False True True True False False True True True True True True True False True True True True False True False False False False References WWC Standards, Any Education Topic, Initial Planning, RCT, QED, Determining Sample Sizes, Recruiting Study Participants, Addressing Changes in your Sample, Reducing Bias in Comparison Groups, Acquiring and Using Administrative Data, Selecting Appropriate Outcome Measures, Combining Data Systems, Understanding Data Analytic Models, Addressing Analysis Challenges, Reporting, Interpreting, and Using Findings, Student Achievement Measure 0 True False True True False False False False True
221 The CLASS Protocol for Classroom Observations Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide presents the elements that constitute the CLASS framework, a research-based protocol to evaluate math and English language arts lessons in all grades. This guide is a valuable resource for program directors who want to select a teacher observation tool. 2010 Link to Resource Guide 4 pages P False False False True True False False False False False False False False False False False True False False False False False False False True False False None 1 True False True False False False False False False
222 The core analytics of randomized experiments for social research Bloom, H. MDRC This MDRC paper on research methodology examines the core analytic elements of randomized experiments for social research. Its goal is to provide a compact discussion for researchers of the design and analysis of randomized experiments. Design issues considered include choosing the size of a study sample and its allocation to experimental groups, using covariates or blocking to improve the precision of impact estimates, and randomizing intact groups instead of individuals. 2006 Link to Resource Methods Report 41 pages R False False False False False True False False False False True False True True False False False False True True False False False False False False False Evidence Head Start Local Average Web site covariates identically distributed Evolving Analytic Thousand Oaks student achievement Improve Precision treatment group effect randomized groups sample research randomization impact experimental 3 False False False False True False False False False
223 The Effectiveness of a Program to Accelerate Vocabulary Development in Kindergarten Goodson, B.; Wolf, A.; Bell, S.; Turner, H.; Finney, P. B. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southeast developed this report to describe the findings from the study of the 24 week long K-PAVE program and its impact on students’ vocabulary development and academic knowledge and on the vocabulary and comprehension support that teachers provided during book read-aloud and other instructional time. The relevant sections of the report are the random assignment and the survey appendices. Appendix C describes the matching of schools within blocks for random assignment and Appendix G includes the teacher survey. These resources are valuable for grantees who wish to conduct similar research. Other appendices focus on quantitative research methods that grantees might find useful. 2010 Link to Resource Methods Report 9 Pages R False False False False True True False False False True False False False False False False True False False False False False True False True False False Education Interventions, Kindergarten, Reading 2 True False False False True False False False False
224 The Effects of Connected Mathematics 2 on Math Achievement in Grade 6 in the Mid-Atlantic Region Martin, T.; Brasiel, S.; Turner, H.; Wise, J. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Mid-Atlantic developed this report to describe the results of a study of the effect of a mathematics intervention on the mathematics achievement and engagement of grade six students. It found that students who received the intervention did not have greater mathematics achievement or engagement than comparison students who experienced other curricula. The relevant sections of the report are appendices on the statistical power analysis conducted to determine the number of schools, class sections, and students needed to detect a minimum detectable effect size for an outcome (appendix B on pages B-1 and B-2), an illustration of how random assignment procedures were implemented using Microsoft Excel (appendix C on pages C-1 to C-3), the student math interest inventory used to understand what students think about the work they do for mathematics class and the associated confirmatory factor analysis (appendix D on pages D-1 to D-4), and teacher surveys including an initial survey for background information and then monthly surveys and a final surveys about how intervention elements were used and perceived (appendix E on pages E-1 to E-4). 2012 Link to Resource Methods Report Tool 13 Pages R False False False False True True False False False True True False False False False False True True True True False False True False False False False Student Achievement, Mathematics, Professional Development 2 True False False True True False False False False
225 The English Language Learner Program Survey for Principals. REL 2014-027. Grady, M. W.; O’dwyer, L. M. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast & Islands developed this survey tool – The English Language Learner Program Survey for Principals – to help state education departments collect consistent data on the education of English language learner students. Designed for school principals, the survey gathers information on school-level policies and practices for educating English language learner students, the types of professional development related to educating these students that principals have received and would like to receive, principals’ familiarity with state guidelines and standards for educating these students, and principals’ beliefs about educating these students. 2014 Link to Resource Tool 28 Pages P False False False False True False False False False False False False False False False False True False False False False False False False False True False Education Evaluation, Department of Education, Education Development Center, Institute of Education Sciences, t i o, National Center, r t m e n t o f E d u c, Regional Assistance, unbiased large-scale evaluations of education programs, D e p, research-based technical assistance, results of research, commercial products, Grady, O’Dwyer, English Language Learner Alliance, policies of IES, REL report, Boston College, mention of trade names, ED-IES, Learner Program, Survey, Regional Educational Laboratory Northeast, organizations, policymakers, educators, public domain, widespread dissemination, practices, content, endorsement, Laura, Matthew, publication, Islands, synthesis, NCEE, DC, funds, United States, Principals, Contract, Washington, views, permission, June, collaboration, Government 3 True False False True False False False False False
226 The Impact of Providing Performance Feedback to Teachers and Principals: Final Report Garet, M.; Wayne, A.; Brown, S.; Rickles, J.; Song, M., Manzeske, D. Institute of Education Sciences The Institute of Education Sciences developed this report to provide researchers and program evaluators with a description of the design and findings of an evaluation of a program providing performance feedback to teachers and principals. The relevant sections of the report are the tools used to measure classroom practice, principal leadership, and student achievement, and the description of the outcomes of the study. The report includes an overview of the measures (pages 2-5), more detail on pages 22-24, 33-34, and 41-42, and sample reports in Appendix K. Outcome measures are presented on pages B-4 to B6. Appendix E provides technical detail on the value-added model used to estimate student achievement and Appendix H technical details about analyses assessing treatment-control differences in educators’ experiences and impacts on outcomes. 2017 Link to Resource Tool Methods report 61 pages R False False False True True True False False False True False True True True True True True False True True True True True False True True False Principals, Teachers’ Performance Evaluations 3 True False False True True False False False False
227 The Investing in Innovation Fund: Summary of 67 Evaluations Boulay, B.; Goodson, B.; Olsen, R; McCormick, R.; Darrow, C.; Frye, M.; Gan, K.; Harvill, E.; Sarna, M.; Institute of Education Sciences National Center for Education Evaluation and Regional Assistance The National Center for Education Evaluation and Regional Assistance developed this report to summarize the findings of a set of evaluations of programs designed to improve student outcomes. The report includes sections that are useful for professionals designing program evaluations. Section 3.1. describes how the authors adopted criteria to assess the strength of the i3 evaluations using What Works Clearinghouse (WWC) evidence standards and a review protocol that ensured the consistent application of those standards (Appendix B). The protocol lists a number of elements that program evaluators might want to consider as they design evaluations, including example outcomes (these are also summarized in section 2.3.3.). Section 3.2.2. lists four standardized tools and templates the authors designed: a study design template to fully describe impact and implementation evaluation design plans, with embedded supports and links to relevant WWC Standards; a Contrast Tool to document each of the planned impact analyses, including research questions, samples, outcomes and baseline measures, and timing of the analysis; a Fidelity Measure Tool to describe the methodology for measuring the fidelity of implementation of the key components of the intervention’s logic model; and reporting templates that include all the information necessary to review the strength of, and summarize the findings from, the evaluations to guide evaluators in drafting reports. These tools are available at http://ies.ed.gov/ncee/projects/evaluationta.asp. Section 3.2.3. describes potential risks to the strength of evaluations such as threats to baseline, power, and implementation fidelity. Section 3.3.1. describes how the team prioritized among multiple evaluation findings. Finally, Chapter 4 and Appendix I can be skimmed to explore different ways of summarizing findings. 2018 Link to Resource Guide 166 R True False False False True True True False False True False False True True False True False False False True True False True True False False False Education Interventions, Educational Research, References WWC Standards, Any Education Topic, Initial Planning, RCT, QED, Identifying Comparison Groups, Addressing Changes in Your Sample, Reducing Bias in Comparison Groups, Selecting Appropriate Outcome Measures, Addressing Analysis Challenges, Reporting, Interpreting, and Using Findings, Student Achievement Measure, Student Behavior Measure 3 True False True False False False False False True
228 The MQI Protocol for Classroom Observations Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide presents the elements that constitute the MQI framework, a research-based protocol to evaluate math lessons. This guide is a valuable resource for program directors who want to select a teacher observation tool. 2010 Link to Resource Guide 4 pages P False False False True True False False False False False False False False False False False True False False False False False False False True False False None 1 True False True False False False False False False
229 The Next Big Leap for Research-Practice Partnerships: Building and Testing Theories to Improve Research Use Tseng, V. W. T. Grant Foundation The W. T. Grant Foundation developed this guide to introduce researchers and practitioners to theories of action (TOAs) as a tool to promote evidence use. Research-practice partnerships (RPPs) have been flourishing as opportunities for researchers to do more impactful research, and for agency staff to get more useful research that can inform their work to improve student outcomes. However, RPPs have focused on producing evidence more than they have on using it. This guide describes how to articulate TOAs for research use, empirically test them, and then iteratively improve on RPP work and refine theories. The guide describes the purpose of a TOA and how to focus initial conversations on different types of research use (instrumental, conceptual, political, process, and imposed uses). Next it highlights considerations around change, decision-making processes, processes by which research would come to be used, and the conditions that would support its use. The last section moves beyond the use of TOAs to support research use and highlights emerging research on TOAs and RPPs including: the relationships between RPP structures, strategies, and capacity and practitioner research use; how and under what conditions RPPs influence the use of research; and the measurement of RPP effectiveness employing five key dimensions: 1) building trusting relationships, 2) conducting rigorous research to inform action, 3) supporting practice change, 4) informing education efforts outside the partnership, and 5) building capacity among partners. 2017 Link to Resource Guide 12 P False True False False True False False False False False False False False False False False False False False False False False False False False False False None 2 False False True False False False False False True
230 The PLATO Protocol for Classroom Observations Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide presents the elements that constitute the PLATO framework, a research-based protocol to evaluate English language arts lessons in grades four through nine. This guide is a valuable resource for program directors who want to select a teacher observation tool. 2010 Link to Resource Guide 5 pages P False False False True True False False False False False False False False False False False True False False False False False False False True False False None 1 True False True False False False False False False
231 The Reliability of Classroom Observations by School Personnel Ho, A.; Kane, T. Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this report to help evaluators of educator observation programs assess the accuracy and reliability of school personnel in performing classroom observations. A particularly useful section of the report is the description of the study design, which includes random assignment (pp. 5-8). The authors examine different combinations of observers and lessons observed, including variation in the length of observations, and the option for teachers to choose the videos that observers see. The bulk of the report (pp. 9-31) presents study results including clear tables and graphs. This guide is a valuable resource for program evaluators who want to understand the breadth of statistical analyses that underlie the non-technical “Ensuring Fair and Reliable Measures of Effective Teaching” (Resource # 205). 2013 Link to Resource Methods report 34 pages R False False False True True True False False False True False True False False False True True False True True True True False False True False False None 3 True False False False True False False False False
232 The Role of Between-Case Effect Size in Conducting, Interpreting, and Summarizing Single-Case Research. NCER 2015-002 Shadish,W.R.; Hedges, L.V.; Horner, R.H.; & Odom, S.L. National Center for Education Research The National Center for Education Research developed this report to increase professionals’ knowledge of, and interest in, single-cased design (SCD) as a strong experimental-design option. The report addresses when to use an SCD, the different forms it can take, its ability to yield causal inference, analysis issues, and how to calculate, use, and report on effect sizes. This paper adds a tool to the repertoire of SCD researchers. Those who review literatures to identify what works (SCD researchers, other academic scholars, research firms, or governmental bodies interested in effective interventions) will benefit from statistical recommendations about effect-size calculations and their use in meta-analysis and pertinent examples to encourage the inclusion of SCD studies in evidence-based practice reviews. Suggestions for future directions are useful for those in policy positions with resources and an interest in improving the use of SCDs both as a method in their own right and as one of many methods that can contribute to knowledge of what works. Finally, the report can help scholars with a statistical and methodological bent to address tackle the theoretical issues the analysis and meta-analysis of SCD research raise. 2015 Link to Resource Methods Report 109 pages R False True False False False False False False True False False False False False False False False False True True True False False False False False False Effect Size, Case Studies, Research Design, Observation, Inferences, Computation, Computer Software, Meta Analysis, Ruling Out Thousand Oaks shed light Mean Difference Sir Ronald Van den Noortgate Heterogeneity Testing Pivotal Response effect case size sizes scd studies treatment analysis scds 3 False False False False True False False False False
233 The Role of Effect Size in Conducting, Interpreting, and Summarizing Single-Case Research William R. Shadish, Larry V. Hedges, Robert H. Horner, Samuel L. Odom IES The National Center for Education Research (NCER) and National Center for Special Education Research (NCSER) commissioned a paper by leading experts in methodology and SCD. Authors William Shadish, Larry Hedges, Robert Horner, and Samuel Odom contend that the best way to ensure that SCD research is accessible and informs policy decisions is to use good standardized effect size measures—indices that put results on a scale with the same meaning across studies—for statistical analyses. Included in this paper are the authors’ recommendations for how SCD researchers can calculate and report on standardized between-case effect sizes, the way in these effect sizes can be used for various audiences (including policymakers) to interpret findings, and how they can be used across studies to summarize the evidence base for education practices. 2016 Link to Resource Methods Report 109 pages R False True True False False False False False False False False False False False True False True True False False True True False False False False False Education Research Thomas, Special Education Research Joan McLaughlin, University of California, Institute of Education Sciences Ruth Neild, Northwestern University, University of Oregon, National Center, University of North Carolina, report, Department of Education Arne Duncan, Summarizing Single-Case Research, single-case design research, Case Effect, Project Officer, Meredith Larson, Phill Gagné, Kimberley Sprague, public domain, Deputy Director, Shadish, use of effect sizes, Odom, Commissioner, Chapel Hill, Policy, Interpreting, Robert, Horner, Larry, Hedges, paper, William, Merced, Samuel, Contract ED-IES, Conducting, Delegated Duties, Authors, views, Brock, Secretary, positions, NCSER, NCER, Westat, Role, opinions, December, Disclaimer, Authorization 3 False False False False True False False False False
234 The Role of Review Protocols and Other Review Issues: Module 3, Chapter 3, WWC Training What Works Clearinghouse Institute of Education Sciences This video discusses in detail how to determine which variables must show baseline equivalence on the analytic sample. Most studies must demonstrate baseline equivalence on outcome variables. However, when the outcome variable is impossible to collect at baseline, WWC provides content area ‘review protocols’ that provide acceptable proxies. The video also discusses why the use of propensity score matching cannot be used to determine baseline equivalence. It also addresses when imputed baseline data may or may not be acceptable. 2016 Link to Resource Video 13 min R True False False False True True True False False True False False False True False True False False False False False False False False False False False None 2 False False False False False True False False False
235 The Texas Teacher Evaluation and Support System Rubric: Properties and Association With School Characteristics Lazarev, V.; Newman, D.; Nguyen, T.; Lin, L.; Zacamy, J. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southwest developed this report to describe the properties of the Texas Teacher Evaluation and Support System rubric and its association with school characteristics. Two sections of the report are relevant. The first one is study limitations on pages 11 and 12, which discusses collecting data during early stages of implementation, small sample size, single teacher observations, and school- vs. classroom-level analyses. The other is data and methodology in Appendix C where authors describe how they combined datasets and the approach they used for each research question, including descriptive statistics, correlations, factor analysis, and statistical tests. These sections are a valuable resource for program directors and evaluators who want to create or evaluate teacher evaluation rubrics and especially determine whether they truly differentiate among teachers. Appendix C will be useful to evaluators working with multiple datasets. 2015 Link to Resource Methods report Tool 6 pages R False False False True True False False False False False False False False False True False False True True False False False False False True True False Schools: Characteristics of, Teachers: Characteristics of, Performance Evaluations, Quality of Instruction 2 True False False True True False False False False
236 The Utility of Teacher and Student Surveys in Principal Evaluations: An Empirical Investigation Liu, K.; Springer, J.; Stuit, D.; Lindsay, J.; Wan, Y. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Midwest developed this report to describe the findings from a study of a district’s principal evaluation model and the relationships between component measures and student growth. The relevant sections of the report are: the analyses appendix (C) and the Tripod survey tool (appendix E). Appendix C reports on supplemental analyses: the reliability and validity of the principal evaluation instrument, additional correlations, and power analysis. Appendix E lists the publicly-released Tripod Student Perception Survey items. These sections are a valuable resource for program directors who want to better gauge the quality of their agency’s educator evaluation system. In particular, they provide a way to assess how well evaluation components predict student growth. 2014 Link to Resource Methods Report Tool 50 pages R False False False True True False False False False False True False False False False True True False True True False False True True True True False Evaluation, Principals, Students, Student Evaluation, Teachers 3 True False False True True False False False False
237 The What Works Clearinghouse: Connecting research and practice to improve education What Works Clearinghouse What Works Clearinghouse This guide goes into further depth on what the WWC does. It features links to WWC publications and resources, such as intervention reports, practice guides, and reviews. The guide also provides answers to frequently asked questions and information about the WWC review process. 2017 Link to Resource Guide Brief or Summary 2 P True False False False True False False False False False False False False False False False False False False False True False False False False False False WWC 1 False True True False False False False False True
238 The What Works Clearinghouse: New Strategies to Support Non-Researchers in Using Rigorous Research in Education Decision-Making Seftor, N.; Monahan, S.; McCutcheon, A. SREE Spring 2016 Conference Abstract Study authors developed this brief (SREE abstract) to raise researchers’ awareness of What Works Clearinghouse (WWC) activities to communicate findings of rigorous research to a wide audience in clear and engaging ways. The brief focuses on three tools the WWC publishes: briefs that explain the rules the WWC uses to assess the quality of studies, a database of intervention reports, and practice guides and accompanying materials that describe low-cost actionable recommendations to address common challenges, such as writing or algebra instruction. This brief is a valuable resource for researchers and evaluators who want to leverage rigorous research to guide and support their work and want to access them easily. 2016 Link to Resource Guide 7 pages P False False False False True False False False False False False False False False False False False False False False False False False False False False False Clearinghouses, Federal Programs, Educational Research, Decision Making, Evidence Based Practice, Information Dissemination, Research and Development, Theory Practice Relationship 2 False False True False False False False False True
239 The WWC Outcome Standard: Module 5, Chapter 2, WWC Training What Works Clearinghouse Institute of Education Sciences This video begins by providing WWC’s definition of an outcome, outcome measure, and outcome domain, and then provides examples of how these three terms are used differently. The video then briefly reviews 4 ways in which the WWC rates and outcome study: (1) demonstrate face validity (2) demonstrate reliability (3) avoid over alignment, and (4) use consistent data collection procedures. 2016 Link to Resource Video 3 min P True False False False True False False False False False False False False False False True False False False False False False False False False False False None 1 False False False False False True False False False
240 Theoretical and Empirical Underpinnings of the What Works Clearinghouse Attrition Standard for Randomized Controlled Trials Deke, J.; Chiang, H. SREE Spring 2014 Conference Abstract Study authors developed this brief (SREE abstract) to explain the WWC attrition model, the process for selecting key parameter values for that model, and how the model informed the development of the WWC attrition standard, and can be used to develop attrition standards tailored to other substantive areas. The brief focuses on the model, how it can be used to estimate bias and determine whether differential attrition rates exceeds a maximum tolerable level. 2014 Link to Resource Brief Methods report 9 pages R True False False False False True False False False False False False True True False False False False False True False False False False False False False Attrition (Research Studies), Student Attrition, Randomized Controlled Trials, Standards, Evidence, Models, Statistical Bias, Error of Measurement, Intervention, Educational Research 2 False True False False True False False False False
241 Tips and Tricks for Successful research Recruitment: A Toolkit for a Community-Based Approach Kubicek,K and Robles, M. Southern California Clinical and Translational Science Institute This toolkit provides an array of advice on how to recruit participants. Even though the main focus is on medical trials, many tips can be applied to educational research as well. The tool discusses the role of partners in recruitment, tips about how to successfully communicate your study, and discusses strategies to help you address common recruiment challenges. 2016 Link to Resource tool guide 30 P False False False False False False False False False True False True False False False False False False False False False False False False False False False Recruiting participants; Recruitment; Digital platforms; compensation 3 False False True True False False False False True
242 Toolkit for a Workshop on Building a Culture of Data Use Gerzon, N.; Guckenburg, S. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Northeast & Islands developed this guide to help practitioners foster a culture of data use in districts and schools. It is grounded in a conceptual framework that draws on five research-based elements known to support an effective culture of data use: participating in the flow of information for data use; communicating professional expectations for data use; providing resources and assistance to make meaning from data; providing professional development on data-use knowledge and skills; and providing leadership to nurture a culture of data use. The guide provides a set of structured activities for facilitators to undertake during a workshop, guiding ideas to scaffold participant learning, and suggestions for participant activities. The facilitator’s guide includes instructions for use and preparation, an agenda for a one-day professional development session (or a series of shorter sessions), and slides with guidance for each of the 11 components of the workshop(s). The handouts offer research reviews, vignettes, tools, and resources that highlight effective practices in each of the five framework elements. This guide is a valuable resource for program directors who may want to implement or improve on data use in their setting. The guide is also useful for evaluators who want to add important elements related to data use to logic models including activities and indicators. 2015 Link to Resource Guide Tool 129 P False False False False True False False False False False False False False False True False False False False False False False False False False False False None 3 False False True True False False False False False
243 Tools For Making it Real: Pre-Scoring Video to Clarify Expectations for Effective Teaching Archer, J.; Cantrell, S.; Holtzman, S.; Joe, J.; Tocci, C.; Wood, J. Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. The guide focuses on creating and using pre-scored videos of teaching practice to build a common understanding of what teaching looks like at particular levels of effectiveness. The guide describes how to identify and acquire appropriate videos; how to recruit and prepare coders; and how to monitor for continuous improvement. Each section provides guiding questions on how to lay the foundation and how to improve. The guide also includes a planning worksheet. This guide is a valuable resource for program directors who want to design and implement a high quality observation system. 2014 Link to Resource Guide Tool 41 pages P False False False True True False False False False False False True False False False False False False False False False False False False False False False None 3 False False True True False False False False False
244 Tools For Making it Real: Pre-Scoring Video to Clarify Expectations for Effective Teaching – DCPS Advice for Facilitating Reconciliation Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this tool to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. Videos of teaching practice are a tool used to train observers. Master observers pre-score videos with a rubric. This tool is a script for reaching agreement on correct ratings when pre-scoring videos of teaching practice. This tool is a valuable resource for program directors who want to design and implement a high quality observation system. 2014 Link to Resource Tool 3 pages P False False False True True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False True False False False False False
245 Tools for Making it Real: Pre-Scoring Video to Clarify Expectations for Effective Teaching – DCPS Guidelines for Score Rationale Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. Videos of teaching practice are a tool used to train observers. Master observers pre-score videos with a rubric. Clear rationales for ratings help observers in training understand what a rubric’s indicators of performance look like in practice, and it lets them compare their own attempts to identify relevant evidence with the correct evidence for rating a segment. This tool is a set of guidelines and specific examples to draft quality rationales when pre-scoring videos of teaching with a rubric. This tool is a valuable resource for program directors who want to design and implement a high quality observation system. 2014 Link to Resource Guide 3 pages P False False False True True False False False False False False False False False False False False False False False False False False False False False False None 1 False False True False False False False False False
246 Tools for Making it Real: Pre-Scoring Video to Clarify Expectations for Effective Teaching – DCPS Video Quality Checklist Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this tool to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. Videos of teaching practice are a tool used to train observers. Master observers pre-score videos with a rubric and observers in training compare their own attempts to the master observers’. This tool is a checklist that helps determine whether a video includes sufficient evidence to rate practice and should therefore be pre-scored and used in the training of observers. This tool is a valuable resource for program directors who want to design and implement a high quality observation system. 2014 Link to Resource Tool 2 pages P False False False True True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False True False False False False False
247 Tools For Making it Real: Pre-Scoring Video to Clarify Expectations for Effective Teaching – RIFT Master Coding Worksheet Measures of Effective Teaching (MET) project Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this tool to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. Videos of teaching practice are a tool used to train observers. Master observers pre-score videos with a rubric. This tool is a form for coders to organize relevant evidence for rating while pre-scoring videos of teaching. This tool is a valuable resource for program directors who want to design and implement a high quality observation system. 2014 Link to Resource Tool 2 pages P False False False True True False False False False False False False False False False False True False False False False False False False False False False None 1 False False False True False False False False False
248 Toward School Districts Conducting Their Own Rigorous Program Evaluations: Final Report on the “Low Cost Experiments to Support Local School District Decisions” Project Newman, D. Empirical Education Inc. Empirical Education Inc. developed this report to help school systems increase the efficiency and lower the cost of randomized experiments. The report focuses on how to design and conduct a randomized experiment, on school system decision-making processes, and on the challenges to using research findings. This report is targeted to three audiences: “1) educators with little training or interest in research methodologies but who are in decision-making positions, such as senior central office administrators, 2) district research managers with some, but often rusty or outdated, understanding of research methodology and statistical analysis, and 3) technical experts who may be called upon to review the evidence.” (p. 2) 2008 Link to Resource Methods Report 47 pages R False False True False False True False False False True True True False False True False True False True False True False False False False False False Bay Area Child Left Behind Paired Randomization Professional Development research district program decision evidence school education experiment report 3 False False False False True False False False False
249 Understanding Program Monitoring: The Relationships among Outcomes, Indicators, Measures, and Targets. REL 2014-011 Malone, N.; Mark, L.; Narayan, K. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Pacific developed this guide to help educators, program managers, administrators, and researchers monitor program outcomes more effectively. The guide provides concise definitions of program monitoring components, explains how these components relate to one another, and illustrates a logic model framework for assessing the progress of a program. The guide demonstrates through examples the relationships between outcomes, indicators, measures, benchmarks, baselines, and targets. This guide is a useful resource for educators who want to identify the achievement of program goals. This guide is one of a four-part series on program planning and monitoring. 2014 Link to Resource Guide 5 Pages P False False False False True False False False False False False False False False False True False False False False False False False False False False False program components, program monitoring framework, program managers, monitoring program outcomes, Understanding program monitoring, program evaluation, definitions of program monitoring components, assessing program progress, Mid-continent Research, measures, educators, indicators, better data-informed decisions, Education, effectiveness, Krishna Narayan, Island Innovation, LLC, better monitor, researchers, resource, baselines, building capacity, targets, including benchmarks, Learning, administrators, crucial step, guide, relationships, Nolan Malone, Lauren Mark, means, practitioners, ongoing plan, programs, Examples, following, Policymakers 1 False False True False False False False False False
251 Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods Schochet, P. Z.; Puma, M.; & Deke, J. Institute of Education Sciences An Institute of Education Sciences-developed report to help education researchers evaluate programs using randomized controlled trials or quasi-experimental designs with treatment and comparison groups. This report summarizes the research literature on quantitative methods for assessing the impacts of educational interventions on instructional practices and student learning and provides technical guidance about the use and interpretation of these methods. This report is a valuable resource for education researchers with an intermediate to advanced knowledge of quantitative research methods who could benefit from summary information and references to recent papers on quantitative methods. The report examines variations in treatment effects across students, educators, and sites in education evaluations. 2014 Link to Resource Methods Report 50 pages R False False False False False True True True False False False False False False False False False False True True False False False False False False False National Center Working Paper intervention Mathematica Policy take-up exclusion restriction Regional Assistance U.S. Department job training treatment subscript effects intervention methods example subgroup control group student 3 False False False False True False False False False
252 Using Administrative Data for Research: A Companion Guide to “A Descriptive Analysis of the Principal Workforce in Florida Schools.” REL 2015-049 Folsom, J.S.; Osborne-Lampkin, L.; Herrington, C. D. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southeast developed this guide to help professionals extract and describe information from a State Department of Education database. The document describes the process of data cleaning, merging, and analysis, including: identifying and requesting data, creating functional datasets, merging datasets, and analyzing data. Although focused on the demographic composition, certifications, and career paths of Florida school leaders, the processes described in this guide can be modified to explore other education datasets. The document also provides an example of a data request and data dictionary. This guide will be useful to anyone using administrative datasets for to create summary statistics and charts, or conducting other analyses from raw administrative data. 2014 Link to Resource Guide 38 pages P False False False True True False False False False False False False False False True True False True True True False True False False False True False Education Evaluation, Department of Education, Regional Assistance, Institute of Education Sciences, Reading Research, results of research, research-based technical assistance, National Center, Regional Educational Laboratory Southeast, publication, unbiased large-scale evaluations of education programs, Florida schools, commercial products, Osborne-Lampkin, REL report, Florida State University, policies of IES, ED-IES, principal workforce, Using administrative data, mention of trade names, descriptive analysis, organizations, policymakers, practices, educators, widespread dissemination, content, permission, companion guide, endorsement, Herrington, Folsom, synthesis, public domain, DC, NCEE, funds, United States, Contract, Washington, views, December, Government, edlabs 3 True False True False False False False False False
253 Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates Fortson, K.; Verbitsky-Savitz, N.; Kopa, E.; Gleason, P. Institute of Education Sciences This National Center for Education Evaluation and Regional Assistance (NCEE) report estimates the impacts of charter schools using the four common comparison group methods and assesses whether these nonexperimental impact estimates are the same as the experimental impact estimates. The authors find that the use of pre-intervention baseline data that are strongly predictive of the key outcome measures considerably reduces but might not completely eliminate bias. Regression-based nonexperimental impact estimates are significantly different from experimental impact estimates, though the magnitude of difference is modest. 2012 Link to Resource Methods Report 68 pages R False True True False False True True False False True False False True True False False False False True True False False True False False False False DEPARTMENT OF EDUCATION Emma Kopa Natalya Verbitsky-Savitz ad justed experimental Supplemental Tables Working Paper group comparison treatment students group test experimental nonexperimental impact baseline estimates 3 True False False False True False False False False
254 Using Classroom Observations to Measure Teacher Effectiveness Regional Education Laboratory Program Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Mid-Atlantic developed this webinar to help participants use classroom observations as one measure of teacher effectiveness in a comprehensive educator support system. The featured presentation (about 0:04:56 to 1:15:00) begins with an overview of the context and general principles for educator evaluations (until about 21:00) and then goes into the effective practice of observations and professional conversations. This video will be helpful to stakeholders interested in putting in place, or improving upon, classroom observations for evaluation purposes and increased professional learning. 2016 Link to Resource Video 1 hour 42 min P False False False True False False False False False False False False False False False False True False False False True False False False False False False None 3 False False False False False True False False False
255 Using Cost Analysis to Inform Decisions about Professional Learning REL Northeast and Islands Regional Education Laboratory Program, Institute of Education Sciences The Institute for Education Sciences developed this webinar to introduce professionals to cost analyses in education research with a focus on the costs of professional development offerings. The goals for the webinar are to demonstrate ways to estimate resource requirements and costs of implementing educational strategies, the questions different cost analyses can answer and with what data, and how a district or school can use these analyses to inform resource allocation decisions. An introduction to cost analysis and purposes begins at 4:03. Two examples are presented starting a 7:29—class size reduction and Read 180. The importance of cost analyses for professional learning starts at 10:16, and addresses which costs to include (district, school, teachers, taxpayers, state education department). At 15:28, the webinar describes CostOut, a free online toolkit to estimate the cost and the cost-effectiveness of educational programs. A description of four types of cost analyses begins at 16:41: cost feasibility, cost effectiveness , cost benefit, and cost utility. Costs to be considered in the context of professional learning is at 30:52. At 38:44, a case study in Maine includes a description of cost analyses considering various professional development offerings and lessons learned (65:42). 2018 Link to Resource Webinar Guide 86:37 P False False False True True False False False False False False False False False True True True True True False True False True False True True False Educator Development, Any Education Topic, Initial Planning, Acquiring and Using Administrative Data, Selecting Appropriate Outcome Measures, Collecting New Data, Combining Data Systems, Understanding Data Analytic Models, Reporting, Interpreting, and Using Findings, Student Achievement measure, Teacher Measure, Principal/School Measure 3 True False True False False False True False True
256 Using Matching Methods: A Practical Guide to Minimizing Differences between Groups in Quasi-Experimental Designs Litwok, D.; Wolf, A.; Price, C.; Unlu, F. Abt Associates This set of resources offers practical guidance to evaluators interested in using matching methods to improve estimates of intervention impacts. The zip file includes a 39-page guide that explains why a matched comparison group should be used and what makes a comparison group well matched. It describes the four steps for implementing a matching process to create equivalent treatment and comparison groups in a quasi-experimental evaluation: selecting the matching variables, choosing the distance metric and matching method, assessing baseline equivalence, and adjusting the match. It goes through five applications of propensity-score matching and two examples of Mahalanobis matching, and compares methods. The zip file includes a dataset and SAS and Stata programs that can be used to implement the matching techniques described in the guide. (Note: the Length field is the number of pages in the guide) 2017 Link to Resource Guide Tool 39 R True False False False True False True False False True True False False True False False False False False False False False True False False False False QED Tools and Resources, Guidance 3 True False True True False False False False False
257 Using Student Surveys to Monitor Teacher Effectiveness Regional Education Laboratory Program Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Mid-Atlantic developed this webinar to give participants access to a student survey to monitor teacher effectiveness. The featured presentation (from about 00:8:00 to 1:35:00) focuses on the Tripod survey, a student survey about teaching practices, student engagement, and school climate. At its core are the 7 Cs (caring, captivating, conferring, clarifying, consolidating, challenging, and controlling), a set of teaching practices that peer-reviewed research has linked to student engagement (effort and behavior) and achievement (gains on standardized tests). The video explores each measure using survey examples, results from the field for various student groups, and graphical representations. It discusses an example of use of the survey in the Measures of Effective Teaching project, which examined how evaluation methods could best be used to identify and develop great teaching. This video will be helpful to schools and districts interested in using student surveys to improve teaching and learning. 2016 Link to Resource Webinar 1 hour 55 min P False False False True True False False False False False False False False False False False True False False False True True False True True False False None 3 True False False False False False True False False
258 Using the WWC to Support Data Collection for Meta-Analyses Josh Polanin, J.; Scher, L. What Works Clearinghouse The What Works Clearinghouse (WWC) developed this webinar to help researchers conduct meta-analyses – the webinar assumes basic familiarity with meta-analysis and the WWC website. The webinar begins with a brief overview of meta-analysis and systematic reviews differentiating them from other research; it then describes the key steps involved in conducting a meta-analysis: formulating a research question, developing a protocol, searching the literature, gathering study-specific information, preparing a meta-analytic dataset, and analyzing and reporting results. The webinar describes how researchers can use WWC resources to support their meta-analytic review efforts; how to export study-specific details from the WWC’s individual studies database, including data on effect sizes that may not be available in published study reports; how to extract information from the database files and conduct a meta-analysis using R, a free statistical software package. It also points to researchers and software that could be resources for those interested in single case designs in the meta-analysis context (this starts at 71:18). 2017 Link to Resource Webinar Methods report 01:19:00 R False False False False True False False False True False True False False False True True False True False False True False False False False False False None 3 False False False False True False True False False
259 Validity and Reliability of a Questionnaire on Primary and Secondary School Teachers’ Perception of Teaching a Competence-Based Curriculum Model García, L.; Gutiérrez, D.; Pastor, J. ; Romo, V. Journal of New Approaches in Educational Research The authors developed this paper to help researchers develop a valid and reliable questionnaire to measure teacher perception, in this case of a competence-based curriculum model. The article describes the sample and how it was recruited; how the questionnaire was drafted based on a review of the literature and an expert panel; how it was administered; and how it was finalized using exploratory and confirmatory factor analysis, resulting in a rapidly and easily administered scale with good criterion validity. This paper is a valuable resource for program evaluators who want to develop quality questionnaires or those who are looking for a tool to measure teacher perceptions of competence-based curriculum 2018 Link to Resource Methods report Tool 6 R False False False False True False False False False False False False False False False True True False False False False False False False True False False Validity, Reliability, Questionnaire, Teacher, Competence 2 True False False True True False False False False
260 Ways to Evaluate the Success of Your Teacher Incentive Fund Project in Meeting TIF Goals Milanowski, A.; Finster, M. U.S. Department of Education, Teacher Incentive Fund The U.S. Department of Education developed this guide to help Teacher Incentive Fund (TIF) grantees collect evidence about how well they are accomplishing TIF goals (improving student achievement by increasing teacher and principal effectiveness; reforming teacher and principal compensation systems so that teachers and principals are rewarded for increases in student achievement; increasing the number of effective teachers teaching poor, minority, and disadvantaged students in hard-to-staff subjects, and creating sustainable performance-based compensation systems). The document provides examples of charts that users can create to visualize trends and comparisons. It also suggests survey questions that can be added to existing surveys to get at stakeholder perceptions of the program. It provides links to additional resources on program evaluation available on the TIF Community of Practice website. This guide will be useful to program staff who want to track progress toward program goals. However, the methods described in the guide will not provide definitive evidence to measure the program’s impact on outcomes. 2016 Link to Resource Tool Guide 12 pages P False False True True True False False False False False False False False False False True True False False False True True True False True False False Program Evaluation, Incentives, Grants, Compensation (Remuneration), Program Effectiveness, Academic Achievement, Sustainability, Teacher Effectiveness, Equal Education, Management Systems, Human Capital, Teachers, Principals, Evaluation Methods 2 True False True True False False False False False
261 We are Better Together: Researchers & Educators Partner to Improve Students’ Math Skills REL Midwest Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Midwest developed this podcast to help practitioners use quick turnaround research to get feedback on their improvement efforts in a timely manner. This podcast describes a continuous improvement process implemented through a networked improvement community – a group of individuals who are committed to working on the same problem of practice. It describes an example from a network of Michigan schools. This video will be helpful to practitioners and researchers interested in implementing and collaborating on a continuous improvement process and wanting an overview and a real-life example. 2016 Link to Resource Video Guide 18 min P False True False False True False False False False False False False False False False False False False False False False False False False False False False None 2 False False True False False True False False True
262 Webinar Series on Developing a Successful Professional Development Program Evaluation Webinar 4 of 5: Going Deeper into Analyzing Results REL Southeast Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory Southeast developed a series of five recorded webinars with materials designed to help professionals rigorously evaluate professional-development programs. The series focuses on key aspects of the evaluation process, including planning and design, identifying and measuring target outcomes, analyzing results, interpreting results, and presenting findings. This series is a valuable resource for program directors who want to understand how to design and implement a high-quality evaluation. The series is also useful for seasoned evaluators who want to review important evaluation issues. This fourth webinar mainly focuses on the considerations for quantitative and qualitative analyses. For quantitative analysis, the webinar focuses on calculating attrition, calculating baseline equivalence, and statistical adjustments. For qualitative analysis, the webinar focuses on triangulating data by using multiple methods of data on the same question, using different interviews to avoid biases of different data collectors and interviewers working alone, and using multiple perspectives to interpret a set of data. 2016 Link to Resource Webinar 31 slides, 38 minutes P True False False True False True True False False False False False True True False True False False True True False False False False False False False observed data/number without observed strong studies estimated effects evidence standards must have have low statistical significance studies that baseline characteristics data attrition wwc ncee professional development 3 False False False False False False True False False
263 Webinar Series on Developing a Successful Professional Development Program Evaluation: Webinar 1 of 5: Critical Elements of Professional Development Planning & Evaluation REL Southeast Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Southeast developed a series of five recorded webinars with materials designed to help professionals rigorously evaluate professional-development programs. The series focuses on key aspects of the evaluation process, including planning and design, identifying and measuring target outcomes, analyzing results, interpreting results, and presenting findings. This series is a valuable resource for program directors who want to understand how to design and implement a high-quality evaluation. The series is also useful for seasoned evaluators who want to review important evaluation issues. This first webinar introduces practitioners to What Works Clearinghouse standards for randomized controlled trials and quasi-experimental designs. It also provides recommendations so practitioners have the resources necessary to design an evaluation aligned to best practices for making causal conclusions. 2016 Link to Resource Webinar Slide presentation 35 slides, 62 minutes P True False False False False True True False False True False False True True False False False False True True True False False False False False False Development Planning Guide for Study Authors Logic Model Professional Development Reporting Guide for a Logic Barbara Foorman study # WWC Reporting professional development wwc http gov study ies ncee sid groups design 3 False False False False False False True True False
264 Webinar Series on Developing a Successful Professional Development Program Evaluation: Webinar 2 of 5: Going Deeper into Planning the Design REL Southeast Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory Southeast developed a series of five recorded webinars with materials designed to help professionals rigorously evaluate professional-development programs. The series focuses on key aspects of the evaluation process, including planning and design, identifying and measuring target outcomes, analyzing results, interpreting results, and presenting findings. This series is a valuable resource for program directors who want to understand how to design and implement a high-quality evaluation. The series is also useful for seasoned evaluators who want to review important evaluation issues. This second webinar builds on the first in the series and provides an in-depth discussion of the critical features of randomized controlled trials and quasi-experimental designs, including the random assignment process, attrition, baseline equivalence, confounding factors, outcome eligibility, and power analysis. 2016 Link to Resource Webinar Slide presentation 22 slides, 49 minutes P True False False False False True True False False True False False True True False True False False True True True False False False False False False Cluster-level RCT professional development QED 3 False False False False False False True True False
265 Webinar Series on Developing a Successful Professional Development Program Evaluation: Webinar 3 of 5: Going Deeper into Identifying & Measuring Target Outcomes REL Southeast Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory Southeast developed a series of five recorded webinars with materials designed to help professionals rigorously evaluate professional-development programs. The series focuses on key aspects of the evaluation process, including planning and design, identifying and measuring target outcomes, analyzing results, interpreting results, and presenting findings. This series is a valuable resource for program directors who want to understand how to design and implement a high-quality evaluation. The series is also useful for seasoned evaluators who want to review important evaluation issues. This third webinar mainly focuses on identifying evaluation outcomes and describes an example from Mississippi which created two measures – a classroom observation tool, and a teacher knowledge measure – to evaluate professional development around a literacy initiative. The third webinar also discusses participant recruitment. 2016 Link to Resource Webinar Slide presentation 20 slides, 49 minutes P False False False True False False True False False True False True False False False True True False False False False False False False True False False items test reliability may measure teachers tool schools professional study professional development teacher knowledge initiative target schools point biserial the assessment coaches observation tool target state 3 True False False False False False True True False
266 Webinar Series on Developing a Successful Professional Development Program Evaluation: Webinar 5 of 5: Going Deeper into Interpreting Results & Presenting Findings REL Southeast Regional Education Laboratory Program, Institute for Education Sciences The Regional Educational Laboratory (REL) Southeast developed a series of five recorded webinars with materials designed to help professionals rigorously evaluate professional-development programs. The series focuses on key aspects of the evaluation process, including planning and design, identifying and measuring target outcomes, analyzing results, interpreting results, and presenting findings. This series is a valuable resource for program directors who want to understand how to design and implement a high-quality evaluation. The series is also useful for seasoned evaluators who want to review important evaluation issues. This fifth webinar mainly focuses on interpreting results, especially using effect sizes and statistical significance. It also briefly addresses the presenting findings to different groups of practitioners and mentions a REL Northeast and Islands resource for those interested in the details on this topic. 2016 Link to Resource Webinar Slide Presentation 15 slides, 24 minutes P True False False True False True True True False False True False False False False True False False False False True False False False False False False Districts Working External Researchers Islands Toolkit REL Northeast Results Externally Disseminating Research Externally http Module 5 Research Results Working with professional development effects external student positive wwc results working effect research researchers 3 False False False False False False True True False
267 Webinar: Examining Evaluator Feedback in Teacher Evaluation Systems Cherasaro, T.; Martin, T. Regional Education Laboratory Program, Institute of Education Sciences The Regional Education Laboratory (REL) Central developed this webinar to give professionals access to a tool to assess the quality of evaluator feedback in a teacher evaluation context. Presenters describe the survey, guiding research questions, the underlying theoretical model, and definitions. They provide an overview of the iterative steps in the survey development process, including selecting an advisory panel, testing the survey with teacher cognitive interviews, examining the reliability and validity of the survey, piloting it, and getting to a final version. They go over sample research questions the survey can help address and how to analyze data. They also provide links to pdf, Word, and Google versions of the survey. Finally, they provide a sample data display and how to interpret it. This video will be useful more generally to those interested in creating their own evaluator feedback survey or even surveys in other areas. 2016 Link to Resource Webinar 28 min P False False False True False False False False False False False False False False False False True False False False True True False False True False False None 3 True False False False False False True False True
268 What is the WWC? What Works Clearinghouse What Works Clearinghouse This one-page brief provides a quick overveiw of the what, why, who, how, and where of the What Works Clearinghouse, the U.S. Department of Education’s trusted source for rigorous research in education. 2017 Link to Resource Brief or summary guide 1 P True False False False True False False False False False False False False False False False False False False False True False False False False False False WWC 1 False True True False False False False False True
269 What It Looks Like: Master Coding Videos for Observer Training and Assessment — Summary McClellan, C. Bill and Melinda Gates Foundation The Bill and Melinda Gates Foundation developed this guide to help professionals design and implement high-quality observation systems as part of educator evaluation initiatives. Observers must know what it looks like when a particular aspect of teaching is demonstrated at a particular level. Master-coded videos—videos of teachers engaged in classroom instruction that have been assigned correct scores by professionals with expertise in both the rubric and teaching practice—help ensure this competency. The guide describes the goals of master coding and the recruitment and training of coders. This guide is a valuable resource for program directors who want to design and implement a high quality observation system. 2013 Link to Resource Guide 4 pages P False False False True True False False False False False False True False False False False True False False False False False False False False False False None 1 False False True False False False False False False
270 What Works Clearinghouse Procedures and Standards Handbook Version 3.0 What Works Clearinghouse Institute of Education Sciences The Institute of Education Sciences developed this guide to help researchers design studies that meet the standards to claim causal inference. The guide focuses on randomized controlled trials and quasi experimental designs (it also briefly introduces standards for regression discontinuity and single case designs). The guide defines the basic steps used to develop a review protocol, identify the relevant literature, assess research quality, and summarize evidence of effectiveness. It also briefly describes the set of products that summarize the results. This guide is a valuable resource for researchers and practitioners who want to understand why studies meet or do not meet the standards and why education programs, policies, and practices are deemed effective or not. 2014 Link to Resource Guide 91 pages R True False False False False True True False False True True False True True False True False False False True True False False False False False False peer-reviewed Learning Disabilities Pilot Single Single-Case New York Electronic Databases Works Clearinghouse full text intervention wwc study effect review studies data group level design 3 False False True False False False False False False
271 What Works Clearinghouse. Single-Case Design Technical Documentation Kratochwill, T. R.; Hitchcock, J.; Horner, R. H.; Levin, J. R.; Odom, S. L.; Rindskopf, D. M.; & Shadish, W. R. Institute of Education Sciences: What Works Clearinghouse at the U.S. Department of Education The What Works Clearinghouse (WWC) developed this guide to help researchers implement effective single-case designs (SCDs). The guide focuses on the features of SCDs, threats to internal validity, standards for SCDs, the visual analysis of the data on which SCD researchers traditionally have relied to answer research questions, and the estimation of effect sizes. This guide is a valuable resource for researchers who want to understand how to design, implement, and evaluate an SCD that meets WWC standards. 2010 Link to Resource Guide Methods Report 34 pages R True False False False False False False False True False False False True False False False False False True True False True False False False False False Projected Comparison straight line best fitting distribution theory give reasons randomized controlled Examine Observed error structures intervention data phase baseline effect design standards case designs phases 3 False False True False True False False False False
272 When Is It Possible to Conduct a Randomized Controlled Trial in Education at Reduced Cost, Using Existing Data Sources? A Brief Overview Coalition for Evidence-Based Policy Council for Excellence in Government The Council for Excellence in Government developed this guide to advise researchers, policymakers and others on when it’s possible to conduct a low-cost, high-quality randomized controlled trial (RCT) in education. The guide includes two main sections: 1) describing the conditions that enable researchers to conduct an RCT at reduced cost (e.g. high-quality administrative data and an active partnership with senior school or district officials), and 2) providing examples of well-designed RCTs conducted at a reduced cost. This guide is a valuable resource for policymakers, researchers, and others who automatically discard RCTs as an approach because they believe such studies are always too costly and too burdensome on schools to be practical. 2007 Link to Resource Guide Brief or Summary 13 pages P False False True False False True False False False False False False False False True False False False False False False False True False False False False Costs, Scores, Data, Research Design, Evaluators, Evaluation Methods, Guides, Outcomes of Education, Program Evaluation, Research Methodology, Evidence, Educational Policy, Federal Programs, Controlled Trial Randomized Controlled grade retentions struggling readers Project Officials professional development special education Mathematica Policy Solicit Rigorous Evidence-Based Policy data school trial students cost intervention randomized study measure 2 True True True False False False False False False
273 Which comparison-group (“quasi-experimental”) study designs are most likely to produce valid estimates of a program’s impact: A brief overview and sample review form Coalition for Evidence-Based Policy Coalition for Evidence-Based Policy The Coalition for Evidence-Based Policy developed this guide to help researchers select comparison-group (quasi-experimental) studies that are most likely to provide the same level of rigor as a randomized controlled trial. The guide describes examples of these studies and how to identify comparison groups in a way that increases the likelihood of producing valid results. It identifies regression-discontinuity designs as an example of approaches that can meet the required conditions for validity of results. The guide also provides a form that can be used to help determine whether a comparison-group study produces scientifically valid estimates of a program’s impact. 2014 Link to Resource Guide 7 pages P False False False False False False True True False True False False False True False False False False True True False False False False False False False Brief Overview composite rating been carried carried out controlled trials one composite give one Please give program comparison group groups study studies valid estimates methods design 2 False False True False False False False False False
274 Which Study Designs Are Capable of Producing Valid Evidence About A Program’s Effectiveness? A Brief Overview Coalition for Evidence-Based Policy Coalition for Evidence-Based Policy The Coalition for Evidence-Based Policy developed this guide to help policy officials, program providers, and researchers go identify effective social programs and ways to assess a program’s effectiveness. The guide provides a brief overview of which studies (randomized controlled trials and quasi-experiments, such as matched comparison groups or regression discontinuity designs) can produce valid evidence about a program’s effectiveness. The final section identifies resources for readers seeking more detailed information or assistance. This guide is a valuable resource for policy officials, program providers, and researchers who want to understand the conditions under which different evaluation approaches are most likely to provide an accurate assessment of a program’s effectiveness. 2014 Link to Resource Guide Brief or Summary 8 pages P False False False False False True True True False True False False False True False True False False False False False False False False False False False Regional Assistance Services Task Study Designs Task Force Designs Are carried out linked here well-conducted pre-program program groups comparison studies study education evidence group rcts 2 False True True False False False False False False
275 Why Use Experiments to Evaluate Programs? Resch, A. Regional Education Laboratory Program, Institute of Education Sciences and Mathematica Policy Research The Regional Educational Program (REL) developed a video series to help schools, districts, states, and their research partners use a cost-effective approach, known as “opportunistic experiments,” to test the effectiveness of programs. This video describes why schools, districts, states, and their research partners might want to use experiments to evaluate programs and policies, and what they can learn from them. The video will be useful to professionals considering rigorous evaluation that will generate evidence for informing their education decisions in a cost-effective manner. 2016 Link to Resource Video 6 min P False False True False False True False False False False False False False False False False False False False False False False False False False False False None 2 False False False False False True False False True
276 Workshop on Survey Methods in Education Research: Facilitator’s guide and resources Walston, J.; Redford, J.; Bhatt, M. P. Regional Education Laboratory Program, Institute of Education Sciences The Regional Educational Laboratory (REL) Midwest developed this report for practitioners who want to conduct training on developing and administering surveys. The report provides materials related to various phases of the survey development process, including planning a survey, borrowing from existing surveys, writing survey items, pretesting surveys, sampling, survey administration, maximizing response rates, and measuring nonresponse bias. It also contains a section on focus groups (as part of the survey development process or as a supplementary or alternative data collection method). The materials include a sample workshop agenda, presentation handouts, activities, additional resources, and suggestions for adapting these materials to different contexts. This report is a valuable resource for program directors and evaluators who want to design surveys and focus groups to collect new data. 2017 Link to Resource Guide Tool 146 pages P False False False False True False False False False False True True False False False False True False False False False False False False False False False Educational Research 3 False False True True False False False False False
277 WWC Online Training, Module 6: Systematic Reviews, Part 2 What Works Clearinghouse Institute of Education Sciences This video describes how the What Works Clearinghouse (WWC) conducts systematic reviews of education research, which they do under a variety of topic areas. It mentions the WWC-developed standards, procedures, and review protocol, which are described in handbooks available on the WWC website. The protocol outlines additional review parameters and procedures specific to a number of topic areas of interest, including elgible outcomes, and serves as a guide for every step of the systematic review process. The video then describes the five steps of the WWC systematic review process: define the scope of the review, search the literature, assess the research, combine the findings, and summarize the review. It then presents the components of the protocols: what studies are eligible for review, how the WWC will search for them, and how it will review them. The video illustrates each step of the systematic review process by examining them trhough an example of the systematic review of the intervention Read 180, which is in the adolescent literacy topic area. 2018 Link to Resource Video Guide 14:47 R True False False False True False False False False False False False False False False True False False False False False False True False False False False None 2 True False True False False True False False True
278 WWC Online Training, Module 6: Systematic Reviews, Part 3 What Works Clearinghouse Institute of Education Sciences This video presents the What Works Clearinghouse’s (WWC) definition of a study, an important concept to understand since studies are the building block of systematic reviews. It recalls that a single manuscript can include one study or more and that multiple manuscripts might all report on the same study. It explains when analyses of the same intervention share enough (at least three) characteristics for the WWC to consider them parts of the same study. These study characteristics include the study sample, the assignment or selection process used to create the intervention and comparison groups, the data collection and analysis procedures, and study findings. It provides a couple of example and a knowledge check. It concludes on how ratings are finalized when reviewers initially disagree on them. 2018 Link to Resource Video 6:36 R True False False False True False False False False False False False False False False False False False False False False False False False False False False None 2 False False False False False True False False False
279 WWC Online Training, Module 7: Reporting, Part 2 What Works Clearinghouse Institute of Education Sciences This video describes the key elements for reporting intervention effectiveness findings stemming from the What Works Clearinghouse (WWC) systematic review process. For findings that meet WWC group design standards with or without reservations, these elements are three measures of the magnitude of findings: a mean difference, an effect size, and an improvement index. The video refers the viewer to the WWC study review guide and the WWC procedures handbook for more information and calculations. The WWC also reports the statistical significance of the findings and the video provides an example from a reading recovery intervention report. 2018 Link to Resource Video Guide 7:25 R True False False False True True True False False False False False False False False False False False False False True False True False False False False None 2 True False True False False True False False False
280 WWC Online Training, Module 7: Reporting, Part 3 What Works Clearinghouse Institute of Education Sciences This video explains how to report on findings from within studies that meet the eligibility requirements specified in the review protocol and meet What Works Clearinghouse (WWC) design standards with or without reservations. It describes how to report findings for studies that include multiple findings, for the same outcome measure on different samples, at different points in time, or from different measurement approaches, or for multiple outcomes within a single outcome domain. It explains how decisions are made regarding which are the main findings and which are supplemental, e.g., based on timing or sample vs. subgroup-based findings and how decisions are made in expedited reviews. It describes individual finding ratings vs. full-study ratings. Finally it presents the distinction between statistically significant and substantively important positive effects. The video concludes with an example from a reading recovery intervention report. 2018 Link to Resource Video Guide 13:25 R True False False False True True True False False False False False False False False False False False False False True False True False False False False None 2 True False True False False True False False False
281 WWC Online Training, Module 7: Reporting, Part 4 What Works Clearinghouse Institute of Education Sciences This video describes the statistical adjustments made during a What Works Clearinghouse (WWC) study review. There are three situations in which the WWC will adjust study findings rather than reporting measures of magnitude and statistical significance from a study or calculated based on statistics reported in a study. Two of these post hoc statistical adjustments affect the statistical significance of the reported results. First the WWC applies an adjustment to studies that assign clusters, that is, a set of individuals assigned as a group to a condition, if the study authors did not perform their own adjustment. Second, the WWC applies an adjustment when a study makes multiple comparisons in a group design study. The last adjustment, a difference-in-differences adjustment, affects the magnitude of the findings reported by the study. This adjustment accounts for differences that existed between the intervention and comparison groups prior to implementing the intervention and is applied when the authors did not perform their own adjustment for baseline differences. The video discusses these adjustments including when and how the WWC makes them and it uses an example to walk through each step of the process. 2018 Link to Resource Video Guide 14:21 R True False False False True True True False False False False False False False False False False False False True True False True False False False False None 2 True False True False False True False False False
282 WWC Online Training, Module 7: Reporting, Part 5 What Works Clearinghouse Institute of Education Sciences This video explains how the What Works Clearinghouse (WWC) characterizes the evidence of effectiveness for an intervention across studies, including individual characterizations for each outcome domain. There are six possible characterizations an intervention can receive: positive effects, potentially positive effects, no discernable effects, mixed effects, potentially negative effects, and negative effects. The video refers the viewer to the procedures handbook for detail on the criteria. The video also describes an extent of evidence rating for each domain to show how much evidence the WWC used to determine the intervention effectiveness rating, an indication of how likely the findings are to generalize to other settings. This is also called the external validity of the findings and is based on the number of studies, settings, students, and classrooms. The video displays a summary table from a reading recovery intervention report and describes each of its components. 2018 Link to Resource Video Guide 7:05 R True False False False True True True False False False False False False False False False False False False False True False True False False False False None 2 True False True False False True False False False
283 WWC Online Training, Module 7: Reporting, Part 6 What Works Clearinghouse Institute of Education Sciences The video is a quiz that checks for viewer’s knowledge of how the What Works Clearinghouse (WWC) reports findings stemming from the systematic review process of studies of intervention effectiveness that use a group design. Questions address statistical adjustments the WWC makes to reported findings under certain circumstances, statistical significance before and after adjustments, main vs. supplemental findings, and statistically significant, substantively important, and indeterminate effects. The last minute of the video summarizes a WWC module on reporting in the context of group design standards and points the viewer to additional resources on the WWC website, including detailed responses to the knowledge check. 2018 Link to Resource Video Guide 9:02 R True False False False True True True False False False False False False False False False False False False True True False True False False False False None 2 True False True False False True False False False
284 WWC Online Training, Module 9: Study Review Guide, Part 2 What Works Clearinghouse Institute of Education Sciences The relevant section of this video (up to 1:15) describes the What Works Clearinghouse (WWC) study review guide (SRG), a user-friendly tool for reviewing studies, which flags important information on studies and helps understand ratings. The video describes how the tool helps ensure the reporting of pertinent information about a study and the uniform and quality application of the logic of the WWC design standards and procedures to assign a study rating and report findings. It mentions the existence of two versions of the SRG, a public version and another available to contracted reviewers. They are identical in that they will lead to identical ratings, but they have different functionalities (the rest of the video provides tips on conducting a WWC study review and then gets into the details of completing the SRG). 2018 Link to Resource Video 1:15 R True False False False True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False False False True False False True
285 WWC STANDARDS Brief: Baseline Equivalence What Works Clearinghouse Institute of Education Sciences The What Works Clearinghouse (WWC) developed this brief to help program evaluators and researchers understand and determine baseline equivalence in education research. Baseline equivalence indicates how similar groups that received and did not receive an intervention were before the intervention began. The brief explains why baseline equivalence matters, provides an example with a graphical representation, and describes how the WWC determines baseline equivalence by calculating an effect size. The brief includes a glossary and suggestions for further reading. 2017 Link to Resource Guide 2 P True False False False True True False False False True False False False True False False False False False False False True False False False False False Control Groups, Experimental Groups, Intervention, Comparative Analysis, Individual Characteristics, Research Methodology, Computation, Standards, References WWC Standards, Any Education Topic, RCT, Identifying Comparison Groups, Reducing Bias in Comparison Groups, Visualizing Data 1 False False True False False False False False False
286 WWC Standards Brief: Attrition Standards What Works Clearinghouse Institute of Education Sciences Two-page WWC brief that defines attrition, identifies common causes, discusses why matters, and describes how the WWC handles attrition when considering if an evaluation has met WWC standards. 2017 Link to Resource brief or summary 2 P True False False False True True True False False False False False True True False False False False False False False False False False False False False WWC; Comparison Groups; attrition; bias; differential attrition; overall attrition 1 False True False False False False False False True
287 WWC Standards Brief: Confounding Factors What Works Clearinghouse Institute of Education Sciences Three-page WWC brief that defines confounding factors, (when it is not possible to tell whether the difference in outcomes is due to the intervention or to another cause), discusses why matters, identifies the types of confounding factors that commonly occure in WWC-reviewed studies, provides examples, and detials how a confounding factor can affect a study’s WWC rating. 2017 Link to Resource brief or summary 2 P True False False True True True True False False True False False False True False False False False False False False False False False False False False WWC; Comparison Groups; ; bias; confounding factors; 1 False True False False False False False False True
288 WWC Training Logistics: Intro–Chapter 2, WWC Training What Works Clearinghouse Institute of Education Sciences This video, designed to provide logistics related to the WWC Group Design Training Modules, introduces different helpful resources that are available on the WWC website. 2016 Link to Resource video 3 min P True False False False True False False False False False False False False False False False False False False False False False False False False False False None 1 False False False False False True False False False
289 WWC Version 4.0 Procedures Handbook What Works Clearinghouse Institute of Education Sciences The Institute of Education Sciences developed this guide to help researchers understand the What Works Clearinghouse (WWC) study review process. The guide describes the steps used to develop a review protocol for a newly-prioritized topic. It explains how relevant literature is identified, how eligible studies are selected within that literature, and how they are assessed against WWC design standards. It also details the process of reporting on findings, including the magnitude and statistical significance of study-reported estimates of the effectiveness of interventions. Appendices provide additional details and examples. This guide is a companion to the WWC Version 4.0 Standards Handbook available in this database. 2017 Link to Resource Guide 76 R True False False False True True True True True False False False False False False False False False False False True False False False False False False References WWC Standards, Initial Planning, RCT, QED, RDD, SCD, Reporting, Interpreting, and Using Findings 3 False False True False False False False False False
290 WWC Version 4.0 Standards Handbook What Works Clearinghouse Institute of Education Sciences The Institute of Education Sciences developed this guide to help researchers design studies that meet the standards to claim causal inference. The first section of the guide deals with randomized controlled trials (RCTs) and quasi experimental designs. It covers individual- and cluster-level assignment, propensity score analyses, Analyses in Which Subjects Are Observed in Multiple Time Periods, Analyses with Potentially Endogenous Covariates, Analyses with Missing Data, and Complier Average Causal Effects (CACE) — when subjects do not comply with their assigned conditions in an RCT). The second section focuses on regression discontinuity designs (RDDs) including eligibility, ratings, standards for different RDDs, and reporting requirements. Single case designs are addressed in an appendix. Additional appendices dive deeper into imputed outcome and baseline data and CACE estimates. This guide is a valuable resource for researchers and practitioners who want to understand why studies meet or do not meet the standards and why education programs, policies, and practices are deemed effective or not. It is a companion to the WWC Version 4.0 Procedures Handbook available in this database. 2017 Link to Resource Guide 130 R True False False False True True True True True True False False True True False True False False True True True False False False False False False References WWC Standards, Initial Planning, RCT, QED, RDD, SCD, Identifying Comparison Groups, Addressing Changes in your Sample, Reducing Bias in Comparison Groups, Selecting Appropriate Outcome Measures, Understanding Data Analytic Models, Addressing Analysis Challenges, Reporting, Interpreting, and Using Findings 3 False False True False False False False False True

How to use these filters. As you check individual field checkboxes, the documents that appear will only be documents that satisfy all of those checkboxes. The more boxes you check, the fewer number of documents.

Last Modified: 04/22/2020

Home

Programs

Grantees & Applicants

Families

Educators

Resources

Contact OESE