Scientifically Based Research — U.S. Department of Education– Pg 9
Comprehensive School Reform—Becki Herman
MS. NEUMAN: Finally, we are delighted to have Becki Herman who is a Senior Research Analyst from AEIR talking about comprehensive school reform. Becki?
MS. BECKI HERMAN: Well, thank you very much for the opportunity to come here and talk with you about scientifically based research and the Comprehensive School Reform Demonstration Program.
I am going to cover three areas in my short time. Give you an overview of the research on CSR, and I won’t delve too much into the actual findings that really focus on the quality of the methods in the research. And talk about what it means to apply the definition of scientifically based research to the Comprehensive School Reform Demonstration Program, CSRD. And also, to suggest some possible effects of using this view of research standards on the CSRD program.
First, I want to start off with a brief explanation of Comprehensive School Reform. What is Comprehensive School Reform? Comprehensive School Reform is a school level reform that’s built around a unifying theme. It should be touching all grades and key subjects, English and math for starters, and it should touch all aspects of the school, and this is a key piece: instruction, curriculum, management, parent involvement, community involvement, school organization. There are a number of aspects of the school that need to be covered in Comprehensive School Reform.
Now, to facilitate Comprehensive School Reform many universities and private organizations have developed models that can be selected by schools and adopted by schools. But CSR is not just models. CSR can involve schools developing their own approach where they’re thinking of how they’re going to revise and revamp their instruction and curriculum and their management around this unifying theme, or if they chose to adopt a model, it might be adopting a model and working with other separate practices that they want implement in conjunction with this model, they all fall under this unifying theme.
Since 1997, the Department of Education has supported Comprehensive School Reform with a Comprehensive School Reform Demonstration program. It’s not the only support, but it’s one of the biggest.
I want to touch briefly on the state of the research. Much of the outcomes research focuses on models and so that’s really what I’m going to focus on when I talk about the research but I want to remind you not to lose sight of the fact that models are only part of the story. There’s a missing part of the story that’s not necessarily being told because the research is a little weak there.
In the year 2000, the American Institutes for Research produced the Educators Guide to School Reform which profiled and reviewed the research on 24 of the most prominent CSR models in the country.
What we found was that there was limited research. We only found 130 outcome studies, and we set some limits for what we called an outcome study. It had to be focused on academic achievement and a few other criteria. And the new models have little to no research.
As part of the study, we rated the quality of each studies methodology. We used criteria such as what I have listed there under study methodology. We looked at the design. Was it random assignment? Was it causal, experimental? Did they use controls? What kind of construct, internal, external, validity evidence was there? What’s the duration of the study? Was it longitudinal? What about the sample? The size of the sample, attrition, those sorts of issues. And measures? Independent and are they well-respected, high quality measures of outcomes? Independence of the researcher. Those are some of the areas that we looked at to rate the quality of the studies.
Of 130 outcome studies in 24 models, we found one study that met the gold standard which is true random assignment and also strong in all these other dimensions of quality.
We found 61 studies that met the silver standard, that were quasi-experimental and strong in the other dimensions.
So, there’s not a lot of gold standard, high quality, random assignment research. There is some research that uses quality experimental methods.
As Lisa and Valerie have pointed out before, the quality of the research base overall matters. It’s not just the methodology used in the independent studies, but it’s a replication of findings. It’s that all of the research converges in a certain direction and points a way to a finding that can be useful to schools.
We found that there were very few models that had more than ten strong outcome studies and no models had absolutely consistent findings. There was always a school or a grade or a set of students that didn’t do well with a certain approach. We were unable to come up with conclusive findings that said something worked well every single time.
But we were able to find that the bulk of the research, limited though it was, pushed in certain directions and that there were some models that seemed more consistent in producing strong student achievement outcomes.
It’s important to look at the replication of findings, especially when you don’t have a lot of gold standard studies, when you don’t have a lot of random assignment studies because if you have hundreds of studies that are quasi-experimental study and no random assignment study, you might want to put some weight to those findings.
So, as I’ve said, I was focusing on research on CSRD models, there is very little CSR outcomes research that’s not focused on models. OERI is currently sponsoring a set of studies that look at some of the issues that transcend models. They look at models and the study says well, but there are some issues that are greater. For example, some of these studies are looking at what is the impact of comprehensiveness? Is the whole greater than the sum of the parts? Does a comprehensive reform work better than a set of discrete reforms within a school? Or some of the studies together are looking at the relative effectiveness of different approaches to CSR and some of the factors that help explain the variation.
In the last few years, there has been a marked increase in the amount of CSR research, including some random assignment experimental designs. The two Cook studies that studies that Steve Raudenbush mentioned earlier, a Success for All study that Steve Raudenbush described is actually one of the OERI funded studies where they’re using random assignment and the issues that they’re running into in conducting the study are too numerous to mention. But suffice it to say that they’re committed to doing it and they’ve worked out a strategy for doing it, but there are real world issues with trying to do this.
So, now I’ve touched on some of the highlights of the state of the research on CSR, I’d like to turn to the circumstances under which the definition of scientifically based research should apply to CSRD. I’m borrowing from Baruch’s chapter in an in-press book, Evidence Matters, for these five criteria for when you would apply the standard of—for him he was saying random assignment studies, when you would use that standard.
The first criteria, the problem is serious. The second, the solution is unproven, other study designs will not provide satisfactory results, the results will inform policy decisions and the rights of participants can be protected.
Three of these criteria are easily met for CSRD: the problem is serious and the solution has not been unequivocally proven, although there’s some evidence moving in some directions. And the results will probably inform such policy decisions.
However, the third criteria (that other studies will not provide satisfactory results) well, that depends on the question you’re asking as almost every speaker today has said. If the research question is outcomes, does CSR improve student achievement, a causal question, yes, you’ll get more defensible results using scientifically based research than using say case studies or some alternative design. If the question is what contributes to successful implementation, well, scientifically based research is not necessarily the only or the best strategy but certainly is part of the strategy for answering that question. But case studies can provide some very good information on what are issues with implementation and what are possible solutions.
The final criteria for applying the standards of scientifically based research to CSRD, is that the rights of participants can be protected. In this high stakes, outcome oriented environment for reforming schools that’s a difficult criterion to meet. It’s hard to ask a school to maintain a comprehensive school approach that does not seem to be working when they are under incredible pressure to produce results quickly for the duration of the study that you need to conduct. The study needs to be more than a few minutes.
(Laughter.)
It’s also difficult—and this is a problem with some of the CSR studies that are trying to use random assignment, there’s the problem of getting and maintaining adequate comparisons. If you use random assignment, how do you guarantee that there’s no slippage that they don’t go ahead and adopt either exactly the condition you were testing or a competing condition, but, in other words, somehow tainting your comparison?
It’s difficult to ask schools to either maintain or to not use a Comprehensive School Reform approach for the duration of a study, but there are ways of doing it.
In situations where you’re looking at outcomes and you’re looking for causal effects and where you’re able to protect the rights of participants, then it may be appropriate to apply the standards based research to Comprehensive School Reform Demonstration Programs.
CSRD in the No Child Left Behind legislation has eleven components. Only two of these components are explicitly tied to scientifically based research in the legislation. The first component which is “proven methods and strategies are based on scientifically based research” means the strategy for instruction should have some evidence using scientifically based research.
Then there are a series of components that talk about, say, professional development, measurable goals and benchmarks, that the design is comprehensive, which are less testable within experimental design. They are more about the development and the implementation and they are different sorts of issues.
But, the final component “that the CSR program results in significant improvements in academic achievement,” the idea that the practices that you’re using in your CSR program work and they work as a set collectively. That idea is also held to the standard of requiring evidence from scientifically based research or other evidence of effects.
I was talking to a few people before starting and some said that they were curious about what I was going to say and I said one of the first things I want to say is I’m not a soothsayer. I can’t tell you how this new definition of scientifically based research will effect the program. But, I can make some suggestions of possible effects and I’d be interested to see what actually pans out.
One of the possible effects, focusing on the first component of the CSRD, the expectation that CSR programs use proven practices, one of the effects may be the burden on the schools.
If you have a CSR program that includes a set of practices, you might have a practice like parent involvement. You might have a set of practices around curriculum. You might have a set of practices around instruction and a set of practices around management.
All these practices need to be proven. Somebody needs to go out there and do the research on them. There’s no single source that says this is the best way to go about instruction or this is the only effective curriculum. So, a school that’s thinking about adopting CSR needs to be able to investigate all these various areas of research and that’s a huge burden.
That’s a burden that can be eased with a lot of resources and I know that there’s been mention already of the What Works Clearinghouse which will hopefully be able to provide some support for schools in this area. There are organizations, the Department of Education is not the least, that provide a lot of information to help schools look at the research. But, it’s still very modest. That might deter some schools that are considering applying for CSRD grants if they’re expected to look at all of these aspects.
If a school is considering adopting a model, there might be a positive effect of this new definition of scientifically based research that focuses on the practices. Schools will be looking at the practices within the model, not just the model. They might be able to see whether there is evidence for all of the practices, the curriculum practice, the instruction practices, the management practices, to see whether they think that this is the right approach for them and that there’s evidence that this will work for them.
It might also cause them to question whether the model itself if comprehensive, whether there might not be some practices that are not part of the model, say parent involvement, that they might want to investigate themselves.
Further, it might encourage schools to think about developing their own approach to comprehensive school reform that is inclusive of a larger series of practices.
So, this focus on finding effective practices may really cause them to rethink how they are using models and what practices they would like to be using in their reform approach.
A second positive effect of this definition of standards based research is the possibility that it might encourage schools to be critical consumers of research for them to look at whether something works. That is, provided, that, as I said, they have the resources to help them collect the research and have the resources to help them understand and interpret the research.
A third possible effect of the research standard is a possibly detrimental effect on externally developed CSR models which at this point is one of the most prominent subsets of Comprehensive School Reform.
There are a lot of different models, some that are more mature. They are in a lot of schools and have a strong research base. And then there are some that are smaller. They’re newer. They aren’t in a lot of schools. There’s not a lot of evidence at this point.
With this kind of selection, schools can find a good fit for their own situation. They can find models developed around a theme that works for them. They can find a model that has a series of, a set of effective practices that they believe are right for their own strengths and weaknesses.
But new models may be strongly effected by the requirement for scientifically based research. If they are in few schools and they have not had time to develop a strong research base, this might prune the field. If you hold new models to the same standards it might foreclose the development of approaches, so that you only have one or two big approaches that are mature for schools to be able to turn to.
It might be appropriate to think about a schedule of evaluations where you hold a different standard to the more mature models than to the newer models or the practices that comprise the models, or to support newer demonstration approaches differently from the more mature models in some way.
Finally, for all of this to work, for the research to be meaningful to practitioners, it’s important to be able to build a bridge from the research to schools. I think I’ve mentioned this several times, researchers are trying to make decisions and they’re held to the requirement that these decisions need to have some scientific evidence. So, one of the biggest movements I could see is providing more support for helping schools access the research and for helping them understand and discern between the various levels of research and quality of research.
(Applause.)
MS. NEUMAN: Well, as I look at the clock, I realize that all this prepared discussion time just has ended actually to be blunt.
I thought this was great evidence that the topic of scientific based evidence is truly a fascinating one. I was fascinated to see how many of you all stayed throughout the discussions, as well as the wonderful papers. I had read every one of these papers prior to today, and yet, I found the delivery of those papers still fascinating. The issues you raise are just really important.
We will be thinking about that as we give guidance throughout our various programs.
I’m sure people are willing to stick around a little bit after.
Again, I want to thank all for these wonderful presentations today. They will be up on the web and please feel free to contact me or these wonderful speakers.
Again, thank you for coming.
(Applause.)
(Whereupon, the above-entitled matter was concluded at 11:58 a.m.)
