Guiding Principles of Guiding Principles of Scientific ResearchScientific Research

G. Reid Lyon, PhDPresident and CEO

Synergistic Education SolutionsDallas, TX

Scientific ResearchScientific Research A process of rigorous reasoning based on interactions

among theories methods, and findings;

Builds on understanding derived from the objective testing of models or theories;

Accumulation of scientific knowledge is laborious, plodding, circuitous, and indirect;

Scientific knowledge is developed and honed through critique contested findings, replication, and convergence;

Scientific knowledge is developed through sustained efforts;

Scientific inquiry must be guided by fundamental principles.

Fundamental Principles Fundamental Principles (NRC, (NRC, 2002)2002)

I. Ask significant questions that can be answered empirically.

• “The formulation of a problem is often more essential than it’s solution, which may be merely a matter of mathematical or experimental skill. To raise new questions, new possibilities, to regard old questions from a new angle, requires creative imagination and marks real advance in science” (Einstein & Infeld, 1938);

• The research questions must be asked in a way that allows for empirical investigation.

Fundamental Principles Fundamental Principles (cont’d)(cont’d)

II.Link research to relevant theory.

• Scientific research can be guided by a conceptual framework model, or theory that generates questions to be asked or answers to the questions posed;

• Theory drives the research question, the use of methods, and the interpretation of results.

Fundamental Principles Fundamental Principles (cont’d)(cont’d)

III. Select and apply research designs and methods that permit direct investigation of the question.

• The trustworthiness of any research study is predicated initially on several major elements:

oThe suitability of the proposed research design or methodology to address the specific questions posed by the study;

oThe scientific rigor by which the methodology is applied;

Fundamental Principles Fundamental Principles (cont’d)(cont’d)

• The trustworthiness of any research study is predicated initially on several major elements (cont’d):

o The link between question and methodology must

be clear and justified;

o Detailed description of the method, measures, data collection procedures, data analyses, and subjects must be available to permit replication.

Fundamental Principles Fundamental Principles (cont’d)(cont’d)

IV.Provide a coherent and explicit chain of reasoning that can be replicated.

•What assumptions underlying the inferences were made? Were they clearly stated and justified?

•How was evidence judged to be relevant?

•How were alternative, competing hypotheses, and explanations identified, considered, and accounted for (accepted or discarded)?

Fundamental Principles Fundamental Principles (cont’d)(cont’d)

IV.Provide a coherent and explicit chain of reasoning that can be replicated (cont’d).

• How were the links between data and the conceptual or theoretical framework made?

• The chain of reasoning depends upon the design which depends on the type of question:

oDescription – what is happening?oCause – is there a systematic effect?oProcess/mechanism- why or how does the

effect occur?

Fundamental Principles Fundamental Principles (cont’d)(cont’d)

V. Replicate and generalize across studies.

• Internal Validity: The observations made are consistent an generalize from one observer to another, from one task to a parallel task from one measurement occasion to anther occasion.

oStatistical methods – e.g. correlation;oNon-statistical methods – e.g.

triangulation, comparative analysis.

• External Validity: The extent to which the treatment conditions and participant population reflect the “world” to which generalization is desired.

Fundamental Principles Fundamental Principles (cont’d)(cont’d)

VI. Report research publicly to encourage professional scrutiny, critique and replication.

• Criticism is essential to scientific progress;

• The extent to which new findings can be reviewed contested, and accepted or rejected by scientific peers depends upon accurate, comprehensive, and accessible records of:

oDataoMethodsoInferential reasoning

Common Common Conceptions/Misconceptions About Conceptions/Misconceptions About Scientific Quality and RigorScientific Quality and Rigor

Experimental research is more “scientific” than descriptive or qualitative research…

oNOT TRUE: The type of design/method does not render the study scientific.

Common Common Conceptions/Misconceptions About Conceptions/Misconceptions About Scientific Quality and Rigor Scientific Quality and Rigor (cont’d)(cont’d) A study is deemed to be “scientific” when:

oThere are a clear set of testable questions underlying the design;

oThe methods are appropriate to answer the questions and falsify competing hypotheses and answers;

oThe study is explicitly linked to theory and previous research;

oThe data are analyzed systematically and with the appropriate tools;

oThe data are made available for review and criticism.

Common Common Conceptions/MisconceptionsConceptions/Misconceptions Research in education is fundamentally

different than in the “hard” sciences.o NOT TRUE: Scientific research in education,

psychology, biochemistry, astrophysics, cultural anthropology, mathematics, etc., all:o Seek conceptual/theoretical understanding;

o Pose empirical and testable and refutable hypotheses;

o Design studies that test and rule out competing counter hypotheses;

o Use observational methods that are linked to theory and can be publicly assessed for accuracy;

o Recognize the importance of independent replication and generalization.

Features Common To Educational Features Common To Educational ResearchResearch It is a challenge to design and implement randomized

treatments and repeated measures under highly controlled conditions;

“Double-blind” controls are not feasible when using particular designs;

The level of certainty of research conclusions is lower than in the physical sciences;

Error limits associated with scientific inferences are larger in social, behavioral, and educational research than in the physical sciences;

The influential role of context makes interpretation of data “messy”;

Converging evidence is critical.

Some Reasons Why Educational Practices and Some Reasons Why Educational Practices and Policies Are Not Guided and Informed by Policies Are Not Guided and Informed by ScienceScience

Education Research is young – 100 year history;

Battered by different epistomological perspectives;

Skepticism Concerning the value of a science of education:

o NIE should conduct an evaluation on effectiveness of instructional programs emphasizing an ethnographic or descriptive case study approach because the audience for follow through evaluations is an audience of teachers that doesn’t need statistical findings of experiments to decide how best to teach children. They decide such matters on the basis of complicated public and private understandings, beliefs, motives, and wishes”. (Gene Glass, 1981)

Some Reasons Why Educational Practices and Some Reasons Why Educational Practices and Policies Are Not Guided and Informed by Policies Are Not Guided and Informed by ScienceScience

Lack of Federal financial support for educational research;

Lack of public support for education research

Inadequate translation of trustworthy research findings into applied classroom practices;

Teacher reliance on practical experience rather than data;

Expertise based on subjective judgments of the individual professional rather than student learning and achievement;

Some Reasons Why Educational Practices and Some Reasons Why Educational Practices and Policies Are Not Guided and Informed by Policies Are Not Guided and Informed by Science Science (cont’d)(cont’d)

Tendency to embrace “fads” in instruction rather than identify and implement policies and instructional practices based on data;

“DAP was never seen as needing to be exclusively or even primarily based on research literature…Folklore and personal accounts of best practices passed on from one generation of teachers to the next counted a great deal…The types of citations used to reference the NAEYC publications of DAP guidelines clearly indicate a reliance on sources other than articles reporting original empirical data (i.e. bona fide research)…only 13 of 25 references cited in the DAP report were original reports of research”. (Kontos, 1989)

Limitations in research training.

Principals For Fostering Science In Education: Principals For Fostering Science In Education: What The Federal Government Can DoWhat The Federal Government Can Do

Provide highly experienced research leadership and research management;

Develop rigorous and transparent peer-review processes;

Insulate the program from political interference;

Develop and manage coherent research programs/portfolios that incorporate agency-initiated programs, investigator/field-initiated programs, and multiple funding mechanisms;

Adequately fund the agency;

Stress trans-agency initiatives to increase community of researchers.