Scientifically Based Research — U.S. Department of Education– Pg 11
Submitted Paper—The Logic and the Basic Principles of Scientific Based Research—Michael Feuer and Lisa Towne
Our presentation today is based on a recently released study authored by the National Research Council’s Committee on Scientific Principles in Education Research. That report can be read online for free and additional hard copies are available for sale at http://www.nap.edu/catalog/10236.html.
Context
The study was sponsored by the U.S. National Educational Research Policy and Priorities Board of OERI amid the very interesting context that brings us all here today: the simultaneous enthusiasm for bringing the power of rigorous, objective, scientific understanding to bear on improving decisions about educational programming and thus student achievement, and the controversy and lack of consensus about what it actually means for something to be based on “scientific research” in education. As most of you know, on the one hand, before No Child Left Behind there was the Comprehensive School Reform Demonstration Act and the Reading Excellence Act—two federal education programs driven by the desire to ground program decisions in the best available evidence of their effectiveness. NCLB extends that reach with its myriad references to “scientifically based research” across the full range of programs the act covers.
At the same time, you are probably aware that not only does there not seem to be any working consensus on what exactly is meant by scientific research in education, but that there is also deep skepticism about the quality of existing work available for decision makers in complying with this requirement. Some of you may be aware that in the summer of 2000, Representative Castle (R-DE) introduced a bill to reauthorize OERI that included definitions of “scientifically valid quantitative methods” and “scientifically valid qualitative methods”. While elected officials have long engaged in the federal research effort in establishing priorities, rarely if ever have they instructed researchers on the tools of their trade and codified it in federal statute. This report was intended to engage a group of prominent researchers to articulate the nature of their work.
It is important to make a few introductory notes about what the committee–and thus the report–did and did not do. Without detailing the exact charge or how the committee went about fulfilling it, we do want to underscore a few points relevant to today’s discussion. First, the committee was asked to describe the principles of scientific research in education as well as to draw out the implications of those findings for the future of a federal education research agency. We will not be talking about the findings related to the research agency today, but are hopeful that they can inform the pending reauthorization of OERI. Also, it is important to understand that the committee did not evaluate the quality of existing research in the field. We will no doubt be hearing about the quality of existing research with respect to particular program areas later this morning. This report, and thus our remarks this morning, adopt a forward-looking approach: describing the enterprise in the ideal and highlighting its successes.
Key Messages
If you were hoping to come to this meeting to get a hard-and-fast definition of what constitutes scientific research in education, your expectations will not be fulfilled. There is no algorithm for science, nor is there a checklist for how to evaluate its quality. Of course, No Child Left Behind had to include definitions, but no statute could adequately define science in education or otherwise. Philosophers of science and researchers have been debating such questions for hundreds of years. The main point here is not that we haven’t figured out the algorithm yet, but rather that it does not exist; science is in part a creative enterprise.
The other key point is that there will rarely be any one study that should be taken as the definitive “answer” to questions about education. Science, by definition, is an uncertain enterprise that evolves over time. Things we take for granted as true (e.g., the earth is flat) can be completely reversed by subsequent inquiry. And science progresses as individual studies and their findings are integrated into current understanding. The NRC committee report emphasizes this accumulation of knowledge, arguing that it is sustained inquiry over time that produces insights, not typically any single investigation.
Although it is creative, science is also a very disciplined enterprise that is supported by norms and practices, what the NRC committee called a “culture of inquiry.” It is these norms, or principles, that we will describe today.
Guiding Principles of Scientific Inquiry
These norms apply to all sciences—to cell biology, ecology, economics, developmental psychology, and scientific education research. Although all sciences share these principles, the way they are applied varies depending on the objects of study (e.g., schools) and the context in which they are studied (e.g., highly mobile populations of students, multiple languages spoken). Thus, we will also briefly describe some of the features of education that shape the ways in the principles play out in the scientific study of education.
Principle 1: Pose Significant Questions That Can Be Investigated Empirically
The committee emphasizes here the idea that simply asking a question in a new way can lead to scientific breakthrough. The significance of the question relates to, for example, the extent to which inquiries inform core problems in education, or builds on prior knowledge. This does not mean that basic research is irrelevant; fundamental research in neuroscience, for example, has clear implications for how we educate our children. This underscores an important point that program evaluation and more specifically the estimation of the effects of a particular intervention are only one part of a larger research base that is potentially useful for informing policy and practice. What makes a question significant, then, can derive both from the practical problems of teaching, learning, and schooling as well as the state of knowledge in a particular area.
The word empirical basically means observation, and the use of this word simply signals that science can only address questions that can be answered through systematic investigation or observation. Some important questions lie solely in realms outside of science (e.g. should students be required to recite the pledge of allegiance each school day?).
Principle 2: Link Research to Theory
Much of science is fundamentally concerned with developing and testing theories that can help explain some aspect of the world. Evolution and quantum theory are examples of well-known theories. In the social sciences and education, such “grand” theories are rare, but the goal is still theoretical understanding. An important point here is that data are used in the process of scientific inquiry to relate to a broader framework that drives the investigation. Data about achievement or school spending alone are not useful in a scientific investigation unless they are explicitly used to address a specific question with a specified theoretical model or to generate a theory or conjecture that can be tested later. Even in program evaluations, program developers have at least an implicit conceptual model in mind of how a particular program is supposed to achieve its objectives, and thus this theoretical frame drives the evaluation.
Principle 3: Use Methods That Permit Direct Investigation of Question
Methodology is a key feature of science, but it does not uniquely define it. The method or design used in a particular investigation does not itself make the study scientific, and methods in the abstract cannot be judged to more or less scientific either. There is a wide range of legitimate methods available to researchers in all fields-the NRC report demonstrates this diversity of method in several examples inside and outside education research.
More specifically, it is often the case that the use of multiple methods can significantly strengthen the certainty with which conclusions can be drawn. Think of this idea as being an extension of the more general notion of thinking about a problem from a number of different perspectives: if you can convince yourself that a particular course of action is ideal from several angles, your confidence that it is the “right” thing to do increases.
The last point related to method is that some methods are better than others for particular purposes. Thus, the quality of a particular method can only be judged with respect to how well it addresses the question at hand. The NRC report provides a good bid of detail and several examples of this idea. The committee described a set of common questions in education research and discusses the most commonly used methods for addressing them under various conditions. So for example, when the research base on a particular area is weak-like in the case of understanding how children come to learn the mathematical concepts of ratio and proportion-in-depth, longitudinal, qualitative methodologies will likely be the most appropriate to start to develop theoretical models of student learning of these ideas. On the other hand, when investigators are trying to estimate the effectiveness of a fairly well defined educational intervention-say, for example, a comprehensive school reform model-the use of random assignment is especially well suited. To reiterate the earlier point about the use of multiple methods, it is also true that such evaluations of programs using random assignment methodologies are often significantly strengthened by qualitative methods that focus on what is happening inside classrooms. These tools can often help researchers identify and rule out alternative explanations for why student achievement may be different in classrooms receiving the intervention versus others.
Principle 4: Provide Coherent Chain of Rigorous Reasoning
This is largely what Valerie talked about in the previous presentation: the logic behind scientific reasoning. Again, there is no one, linear way to reason scientifically, but in general terms it must be coherent, explicit, and persuasive to the skeptical reader. The process of reasoning is conducted to produce what John Dewey called a “warrant”—a scientific justification—for inferences and conclusions. And it is important to point out that this logic is fundamentally the same for both quantitative and qualitative research. Scientific reasoning is characterized by clearly stating the assumptions present in the analysis, how evidence was judged to be relevant, how data relate to theoretical conceptions, how much error or uncertainty is associated with conclusions, and perhaps most importantly, how alternative explanations for what was observed were treated.
Principle 5: Replicate and Generalize
Scientific inquiry emphasizes checking and validating individual findings and results in different times, places, and contexts. Since all studies rely on a limited number of observations, a key question is how scientific inferences-that is, the conclusions of scientific work-generalize to a broader population or setting. Successfully replicating findings in different contexts can strengthen a theory or working consensus over time. In education research, contextual factors often are very important. Teachers and researchers alike have long noted that a particular program that works in one classroom may not replicate in a classroom just down the hall or within the same classroom but with a different group of students the next year. In education research, then, attention to the conditions under which a particular classroom being studied is quite important in understanding the extent to which findings will generalize beyond it.
Principle 6: Transparency and Scholarly Debate
A final principle of science relates directly to the “culture of inquiry” we described earlier. Researchers must engage in ongoing scrutiny of each other’s work: by publishing in peer reviewed journals, presenting findings at conferences, and the like. Educators often bemoan what they perceive as bickering within the research community as evidence that the community has somehow failed. On the contrary, researchers are trained and employed to ask critical questions and be skeptical observers. Of course, they also engage in such critique to try and forge consensus about the current state of knowledge in a particular area. The community of researchers has to collectively make sense of new findings to integrate them into the existing corpus of work. Indeed, the objectivity of science derives from these self-enforced norms, not the attributes of a particular person or method.
How is Education Research Special?
We have argued that the preceding principles apply to all sciences. Here, we provide a flavor for how these principles get applied in education research. Education research is most closely associated with the social and behavioral sciences, so some broad differences between the physical and social sciences help to understand the nature of education research.
For example, researcher control is often stronger in the physical sciences. Think of it this way: a Petri dish of heart cells is typically better behaved than a classroom of third-graders! The role of the researcher in sometimes different in hard vs. soft sciences-in the natural sciences it is customary for the researcher to be removed from the process of inquiry so as to minimize bias; in the social sciences, in some cases is not desirable. Also, theory in the social sciences tends to be used to model past behavior rather than to predict the future. And finally, it is very important to recognize that the level of uncertainty is typically higher when studying humans as compared to studying inanimate objects. All sciences have a degree of uncertainty associated with them, but our theoretical understanding of human behavior is still pretty elementary. That is why estimating the certainty of results is so important in education research. And this is important for consumers of research to understand as well: research can be an incredibly powerful tool for helping to make practical decisions, but rarely will it ever “prove” beyond a doubt that one strategy or another will be successful.
In education specifically, the NRC committee briefly described five features of education that influence research in it. For example, it talked about the proper role of values in making education decisions in our democracy and its influence on things like the choice of what is studied and how findings are interpreted and used.
Human volition is another factor. People are complex beings who often have priorities that may not comport with those of researchers trying to study them. This can result in samples changing over time due to high rates of student mobility or parents taking their children out of a particular program to which they were assigned for research purposes.
The local control of schools also means that the nature of programs—even those that are called the same thing—can be implemented very differently across the country and can change substantially year to year. Anyone who has ever tried to evaluate federal programs knows the issue of “fidelity” is critical to understanding its impact. Here again, the point is that paying close attention to this context is important in doing research. The hierarchical nature of schools means that for researchers to understand what is going on in a particular school, they typically must study it with a good understanding of what is going on at the district, state, and even federal levels that influence it. Finally, the cultural, language, racial, ethnic, and geographic diversity that characterizes our nation also, of course, characterizes people in education institutions. And programs may work for some populations but not others, so researcher must explicitly attend to these factors.
Finally, education research itself is characterized by multiple disciplines—for example: developmental psychologists study fundamental processes of cognition, language and socialization; economists study the incentive structures of schools and their relationship to behavior; political scientists study the implementation of large-scale institutional changes, like charter schools. That means again that lots of different methods are used in education research and that the challenge is to integrate what is known from each of these perspectives into some shared understanding.
Also, education research is sometimes curtailed due to justifiable ethical considerations to ensure proper treatment of children (although education research rarely presents any risk to research participants). Finally, education research depends critically on its relationships to educational practitioners. Researchers typically at least need the cooperation of schools and students to conduct their work and increasingly practitioners are entering into full partnership with researchers.
