Scientifically Based Research — U.S. Department of Education– Pg 4
The Logic and the Basic Principles of Scientific Based Research—Michael Feuer and Lisa Towne
MS.NEUMAN: I’d like now to introduce Michael Feuer and Lisa Towne. They have just completed a wonderful project on scientifically based evidence. I am wondering if you have that report with you?
MS. LISA TOWNE: I didn’t anticipate to have to provide this many copies.
MS. NEUMAN: Lisa is a Senior Program Officer at the Center for Education at the National Research Council. Michael is the director of the Center for Education at the National Research Council of the National Academy of Sciences.
We are delighted to have them work with us in talking about the logic and the basic principles of scientific based research, as well as help us focus later on on the implications of this research for practice.
MR. MICHAEL FEUER: Thank you very much for this invitation, Susan, and thank you to all of you for coming out to listen to lectures about science on this Wednesday morning.
We’re here to tell you a little bit about a report that was released at the end of November in this handsomely bound pre-publication form. It’s called “Scientific Research in Education.” I want to spend a few minutes telling you some of the highlights of both why we were asked to do this and what you would find if and when you opened the book and read it which I hope you do.
First of all, the National Research Council of the National Academy of Sciences is, as I’m sure you know, an independent organization. We are not part of the government, although we work closely with the government and on behalf of the American people. This is an idea that goes back actually to the 19th century when President Lincoln looked around and discovered that there were some serious problems that perhaps science and technology could help him with. I’ll just tell you one quick story which my poor staff hears this so much that they tend to nod off when I get into this, but if they’ll indulge me.
One of the very first problems that this new Academy was confronted with had to do with a problem in the Civil War which was the ironclad ship. This, as you recall from history class, was an invention that actually ultimately helped the north win the war.
There was a problem with the ironclad ship, however, and that is that they couldn’t get the compasses to work because of the magnetic fields. Now, if you are ever interested in a sort of classic case of the collision between science and public policy just think about a ship that you can’t get to—you know, knowing the difference between north and south with the Civil War at hand is not a trivial matter.
This was one of the first problems that the Academy was asked to solve and, indeed, a small committee of physicists and engineers was brought together and they actually solved the problem, and I am actually happy to tell you that the report is nearly through review.
(Laughter.)
Now, with respect to education and education research, this is not the first time the Academies have been asked to weigh in on this. There were reports going back even to the late 1950s, and then later through the ’70s and ’80s and ’90s. And that, in itself, I would submit, is an interesting little bit of evidence (perhaps anecdotal, but maybe not) of the very perception that education research has, at least in part, an important scientific component. Because, after all, we are not the National Academy of Poetry, we are the National Academy of Sciences, and when we were asked to take on a question of the scientific quality of education research, I don’t think that was coincidental. I think that is part of a very important perception in the land about the nature of education research.
Indeed, when we were about to launch this study most recently, I began speaking with some of the distinguished scholars around the country. And, when I mentioned that we were about to do a project on the scientific quality of education research, I have to tell you that one of these very distinguished scholars said, “Well, that’s great. Finally, we’ll have a short report from the academy.”
That’s another important perception that we had to deal with and that is that the general perception of a low level of quality in education research writ large.
We don’t have any evidence and we didn’t try to get evidence to support or refute the claim of the overall quality of education research being poor. But we did take as a datum that the perception that it is poor is important and that it is, therefore, worthy of the attention of some very distinguished scientists and educators to think about this whole question.
One more bit of context. I don’t think it is coincidental that requests for study of the scientific nature of education research should come at a time when we probably have more information, more data and a more relentless flow of ideas about how to fix the schools than perhaps at any time in history. Again, I haven’t done the empirical research on this, but I would bet that education policy gets more headline attention than almost any other item on the domestic agenda. To some extent, I think the Administration, and Congress have conveyed an incredibly powerful message in the passage of No Child Left Behind, in particular just after this horrible season of terrorism that we have just come through. It is again an indication of the overwhelming importance of education and education policy in the agenda.
But, that said, there are lots and lots of folks who have gone to school and who therefore have very firm opinions about how to fix the schools. What we get is a cacophony of ideas, solutions, reform initiatives, standards—I mean, we’re responsible for some of the standards documents. And, I sympathize with people in the real world such as yourselves and with teachers and educators all around who have to sift through this morass and make something significant and effective. That’s where the appeal of science becomes very strong. It is after all an enterprise that attempts to distill from the cacophony of ideas and anecdotes and impressions, the nuggets of really enduring value, and that kind of knowledge upon which you would want to base important decisions about kids, about schools and about, ultimately, ourselves.
Having said all that, let me just offer a little bit of a foundation here for what Lisa is going to tell you more specifically and that is some of what’s actually in this report.
As I said, we are an independent organization. We were asked to take on a set of questions having to do, really, with first principles: What is science? That in itself took a few weeks to sort through. What are the principles of science and how do they apply to the science of education? These are very tough questions. What you will hear about is some of the key findings of an interdisciplinary group of scholars, not all educators: cell biologists, a chemist, education scientists, statisticians. This is the way we do our work. We bring these types of people together. And, after all, the National Academy of Sciences obviously exists in some measure to promote the values and the ethos of science and it’s utility in public policy decisions.
So, much of what Valerie has said resonates with the underlying purposes and—are we trying to follow along with the slide show? Because nothing I’ve said so far is on any of these slides. We have a unit at the Academy that specializes in improvisational theater.
(Laughter.)
Let me make this one little attempt at a slightly more cautious definition, or a more cautious statement about the nature of scientific reasoning in education research.
On the one hand, I think what you would see in the report and what you’ll hear about is a great deal of enthusiasm and encouragement for the notion of bringing scientific reasoning, the culture of science, to bear on the important decisions we make about kids and schools.
After all, science is intendedly rational, it is disciplined, it is honest, it is open, we aspire to a kind of dispassionate, politically neutral distillation of evidence to make decisions. That’s why we are enthusiastic about the underlying proposition here that has been articulated in the law and that most of you now are going to have to turn into the real practical day to day.
At the same time, I want to tell you that what scientists themselves often acknowledge is that there is a dimension of human judgment that can be missed with an overzealous focus on the rigors of scientific method.
It was, in fact, a psychologist who won the Nobel prize, Herbert Simon (unfortunately he passed away about a year or so ago) whose contributions to this I think are quite significant because of his work on what human rational decision making is really all about.
The story that he liked to tell was about the traveling salesman who had the following problem: to visit 15 cities and to work with customers in 15 different cities and wanted to minimize the costs of visiting those customers, fuel costs, time and so forth. What’s the rational way to approach that problem?
Well, one rational way to do it is to figure out the different routes you could take and then calculate how much it would cost because of the mileage and the fuel consumption.
Is that, however, really rational? And, the answer is not necessarily. And that’s because by the time you lay out all of the different routes and you mathematicians out there will figure this out pretty quickly that 15 factorial routes is a pretty large number. And, so by the time you have gotten to the end of the list, 20 years have passed. Your competitor who is using a less rigorous, less optimal approach has gone to Cleveland and then figured out that the next stop ought to be Buffalo because that’s closer than Houston. And, you’re back there on the back of your envelope doing the science.
The question becomes what really constitutes rational decision-making? And, the answer is: it depends on context, it depends on technology, it depends on the time you have, and, frankly, as Valerie has I think so eloquently reminded us all, a lot of the decisions that have to be made are going to be made with less than perfect evidence.
And, therefore, you have a double challenge. One of your challenges is to encourage the field of research to provide you with better and better useful evidence. And, don’t think for a minute that we researchers have figured all this out and the only problem is you people in the real world aren’t using it. We know that’s not true. The research community has a lot to do to shape up in order to provide you with useful evidence.
At the same time, the challenge is to continue to make reasonably good decisions based on the evidence that you have.
I don’t want to take time away from Lisa because the real messages of this report are what I think are going to count at the end of the day.
So, I thank you for letting me give you a little sermon about rational decision-making. And, now I will try to sit down rationally and let you hear the rest of this.
(Applause.)
MS. TOWNE: Hi, everybody. It is a pleasure to be here. I just want to, with time considerations, just sort of pick up where Michael left off and like he said just give you a brief sort of tour through what’s in a somewhat longer volume.
As Susan suggested, I am happy to make copies available to people. I wasn’t able to bring them here today but I will work with her to make sure that we can do that and that you will have the pleasure and the privilege of reading every page.
In the meantime, what I am going to do is just talk through, give you a brief idea of what’s in here with respect to the question we were asked to talk briefly about today which is what are the basic principles of science. As you might expect and as I’m very grateful to report, they do reflect in many ways what Valerie has already said.
MS. REYNA: Good.
MS. TOWNE: Yes, this is good. So, Steve, get ready!
If we could go to a slide that says, “Principle One,” that would be great.
What I’m going to do, just to give you a sense of what I’m going to talk about today is talk briefly about the principles of science that actually are common across all disciplines and fields. This is, again what Valerie said, that at a fundamental level, medicine (that was the example that she used), ecology, economics, all of the applied fields like medicine and agriculture, that there is a lot that is actually shared between them.
The principles of science that I’ll talk about today is what the committee who wrote this report believes are those common elements.
Then, I’ll spend a few minutes at the end talking about what is it about education that makes the application of these principles look a little different. Because you might be sitting there thinking, “Wow, looking at something that a physicist does sure does seem a heck of a lot different than what an education researcher does.”
So, I’ll talk a little bit about how these principles play out in studying education and why it is that they tend to look very different.
So, the first principle here relates to posing questions. It seems very straight forward, perhaps self evident, but actually the process of posing a new and different question is often times itself what is the basis of a scientific breakthrough, someone thinking about a problem in a new way and asking a new question.
There’s a couple of words here that I’ll just touch on briefly that give a little bit more detail about what this means.
“Significance,” this again goes back to what Valerie was saying with respect to education. The significance of a question can be judged in terms of its relevance to the core problems of teaching and learning and schooling.
In a more traditional scientific sense, the significance of the question can also derive from what has come before it. In other words, does this question help to advance the field and consensus, and the cumulative nature of science which is a theme that Valerie touched on and that this report also tries to stress very strongly.
The second one, I’ll just touch on briefly, is “empirical.” That simply in very straightforward terms means can be observed. The only reason this is relevant here is because there are some questions that are relevant to what teachers do every day that can’t be answered by science. Should students be asked to say the Pledge of Allegiance every day, for example, has to do with our values as a society and whether we think that is appropriate and good. It is not something that can be subjected to scientific study.
I will go on to the next slide, and talk about the principle that has to do with theory, and again Valerie alluded to this as well. The importance of theory is really very important in education research and the other sciences as well. In fact, much of science is fundamentally concerned with the development and testing of theories that helps you explain some aspect of the world.
In hard sciences, so-called hard sciences, we know of theories like evolution. Grand theories like that don’t typically pop up in education but certainly they are relevant and they certainly are kind of an organizing conception for scientific work. Valerie mentioned a theory of how children learn, that’s a great example. A theory of how educational resources translate into outcomes in schools is another example.
So, theory is really kind of an organizing idea for scientific investigation. The important point here is that data in an of themselves aren’t really relevant to a scientific investigation unless they are related to some sort of conceptual idea that you have going in like about how children learn or about how educational resources translate into, hopefully, better outcomes for schools and for students.
Even in program evaluation, which is a lot of what has to do with the implementation of this law, what works, there is some implicit theory about how the program is supposed to actually translate into better outcomes for kids. Should that point to the basis of a program evaluation? That’s what Carol Weiss calls “a program theory.” So, sometimes it’s explicit and sometimes it’s implicit, but it’s always there.
I will go onto the next principle on the next slide. This has to do with methodology, which Valerie has already, thankfully, covered very well for me.
I will just make three main points about the role of methodology in scientific research.
First of all, that there are a range of legitimate methods in the field. Education is studied from a lot of different disciplinary lenses: economists study this, developmental and popular psychologists, sociologists and anthropologists, they’re sort of studying a different part of the animal and they all bring their tools of the trade to bear on that. So, by definition, there are a range of legitimate methods that are within this domain.
A related point is that when you’re looking at questions in education research, that multiple methods used together tends to strengthen the inferences or the conclusions that one can draw when studying these things scientifically.
The last point that I will make about methodology and this gets to Valerie’s hierarchy of evidence, is that although there is a range of valid and legitimate methods that can be used in studying education, some methods are better than others for particular purposes. Valerie, actually, kind of very nicely laid out kind of a hierarchy of evidence within the class of questions that are causal.
There are other kinds of research in education. There’s descriptive research. There’s research that looks at mechanism. And, within those classes of questions, there’s also different kinds of methods that can be used. So that the method itself, taken out of the context of a particular study, can’t really judge to be good, bad or indifferent. A method is only as good as it addresses a particular question that is being addressed.
I’ll go on to the next slide. I have three minutes and I have several more principles.
Principle four is: a coherent chain or reasoning. This is sort of the logic behind science which, again, Valerie, has talked about and handled quite well.
So, I’ll go on to the next slide which is principle five, and this has to do with replication and generalization.
“Replicating” is a very core notion in science. It has to do with the fact that since in any particular study you’re only relying on a limited set of observations, to what extent does what you’re looking at here and now generalize to other times, places and contexts. In education, as you know, this is a critical question. Teachers and researchers alike have been knowing for years that something that works in a particular classroom may not work in the classroom next door and may not work in the same classroom a year later. So attention to sort of what’s going on in the classroom at that time can help you understand the conditions under which things tend to work and therefore how to think about how findings can generalize from one time to another.
I’ll go on to the last principle here, which has to do with the transparency of the scientific enterprise. Valerie alluded to this as well. This just has to do with the role of the scientific community actually working together to try and make sense of all of the findings and all of the conclusions that come from individual studies. Educators often bemoan what there perceive as bickering among the research community and we’ll grant you that there is some bickering. But there is actually something important to say about that and that is that researchers are actually trained and employed and paid money to be skeptical observers and to ask critical questions. That’s their job. So, this critical kind of work, critiquing other peoples findings and trying to make sense of them is actually an indication of the health of the scientific enterprise, not its failure.
So, those are the basic principles of what actually binds scientific inquiry together across domains and disciplines.
I am going to just touch briefly on a couple of things in education that help understand how these principles are actually translated in the study of education. How much time do I have for that? One minute? I am obviously going to just whiz through these.
One issue has to do—at one level there is a difference between the so-called hard and soft sciences. And, that has to do with differences that emanate from studying inanimate objects and studying people, which are complex and do crazy things that we often can’t understand or predict very well.
So, there are some things that are different. Broadly, research or control is one of them. Think of it this way, a petri dish of heart cells is a heck of a lot better behaved than a classroom of third graders. Anyone whose tried to study education research and has done cell biology, as one of my committee members did, can attest to this.
There’s other things that are different. I’ll just touch on this last one on the slide which has to do with certainty. Valerie said, and the committee completely agrees, that science is by definition an uncertain enterprise. The key is understanding the degree of uncertainty that is associated with what we know. In general terms, in the physical sciences we because of this ability to control the environment tend to have more certainty associated with them than sciences that have to do with people, like education research.
Moving on to the next couple of slides, there’s a couple of things in education, specifically, that actually explain and help understand the nature of education research. Values and politics, Valerie talked about this as well, the role of schooling in our democracy is one that is appropriately and historically grounded in our values as a people. What we decide to do with respect to schools is inevitably and appropriately going to be grounded in those values. Scientific research is one part of that decision process and it should be, but interacts in a very significant way with our values.
Human volition, I’ve alluded to this already. This has to do with the fact that people don’t always have the same agenda as a researcher might and they might move around and mess up samples and do things like that. So, there’s some messiness that researchers have to deal with.
Variability of education programs, I don’t have to tell all of you about the differences in the implementation of programs that happens in different districts and schools.
And, the organization of education, the fact that we have sort of this nested hierarchy matters in education research because understanding what’s going on in a school, you have to have some understanding of what’s going on in the districts, in the state and even at the federal level to really have a good sense of what’s happening at school.
Just go on to the last slide, there’s a couple of remaining points that I’ll just touch on and then wrap up, about what characterizes education research as a profession that tends to help understand its nature as a whole?
One is something I’ve alluded to already and that is the fact that education is not a traditional scientific discipline. It is an applied field, like agriculture, like medicine. So there are a lot of disciplines that legitimately bear on our understanding what is going on in education and that is a key piece to understanding it.
Ethical considerations. Most sciences, but not all have really to be concerned with the ethical implications of what they’re doing. Studying kids who are a vulnerable population sometimes entails things that you have to do with methodology and plan for research in order to make sure that they’re protected. Most of the time education research doesn’t pose any risk and is exempt from the federal regulations that govern them, but, none the less, it is something that factors into the research process and shapes it in a significant way.
Finally, I’ll end with this notion of relationships. Researchers can’t do their job without the cooperation of schools and students and all the different actors who are in the education system. At the very least, they need the cooperation for them to go in and collect data, to test them occasionally and increasingly we’re seeing full blown partnerships being developed where researchers and educators who are on the ground doing education day to day so to speak, actually work collaboratively in a way that tries to both improve practice through research, but also inform and improve the research process by better understanding of what’s going on in practice.
With that, I will conclude.
(Applause.)
MS. NEUMAN: There’s nothing worse than feeling rushed. I hate to do that, but unfortunately we do have a lot to cover.
