Some experimental methodsin Educational Research

QuestionsDesign

Analysis

Adam Dubrowski, HSC Learning Institute

Not enough?

You need to

perform a systematic review of literature

You need to

generate an answerable question

You need to

generate a

You need to

design of a “clean” experiment

You need to

be able to use the stats

You need to

generate an answerable question

Types of Questions

• Health Care Intervention (or treatment, prevention therapy): Determining the effect of different interventions

• Harm (causation): Ascertaining the effect of potentially harmful agents

• Prognosis: Estimating the future course of a patient’s disease or condition

• Diagnosis (or assessment): Establishing the power of a diagnostic tool

• Meaning: Describing, exploring, and explaining phenomena

• Economics: Studying the economic efficiency of health care programs or interventions

Types of Questions

• Education: Determining the effect of educational interventions

Matching Question to Study Design

• Quantitative studies are most useful for answering questions of “how many” or “how much”

Qualitative studies are most appropriate for answering questions about how people “feel about” or “experience” certain situations and conditions.

Matching Question to Study Design

• Quantitative studies are most useful for answering questions of “how many” or “how much”

• Qualitative studies are most appropriate for answering questions about how people “feel about” or “experience” certain situations and conditions.

Asking Structured Questions (WHY?)

Asking Structured Questions (WHY?)

Asking Structured Questions (WHY?)

Asking Structured Questions (WHY?)

Asking Structured Questions (WHY?)

Quantitative Questions (PICO)

• Population• Intervention or exposure• Comparison• Outcome

Quantitative Questions (PICO)

• The Population

– Who are the trainees?– Why are they enrolled?– Is there a particular age or sex grouping?– Is there a particular profession grouping?– What else?

Quantitative Questions (PICO)

• The Intervention

– What interventions are we interested in?– Examples?

Quantitative Questions (PICO)

• The Comparison

– What are the current educational practices or what are other possible educational practices that we are interested in comparing?

Quantitative Questions (PICO)

• The Outcome

– What are the relevant consequences of the intervention in which we are interested?

• Satisfaction• Performance

– Individual– Team– Patient

• System changes• Economics

Quantitative Questions (PICO)

• Group work

Hypothesis

Hypothesis

A hypothesis [from Greek] consists either of a suggested explanation of a phenomenon or of a reasoned proposal suggesting a possible relationship between multiple phenomena.

In science: The Scientific Method requires that one can test a scientific hypothesis.

Generally base such hypotheses on previous observations or on extensions of scientific theories.

Hypothesis

• For evaluation of a hypothesis, one needs to define operational terms.

“I predict that A will lead to D only if B and C are in place”

• A hypothesis should enable predictions by testing.

• A hypothesis requires work by the researcher in order to either accept or reject it.

Hypothesis

• A hypothesis may take the form of asserting a causal relationship (such as "A causes D") but not always.

• For example, if a particular independent variable changes, then a certain dependent variable also changes.

However …… …. what we do not know is whether there is a direct causal relationship, or whether there is another variable in the middle that causes the change.

Hypothesis: Variables

An independent variable is the variable whose value one actively controls and can change.

A dependent variable is the thing whose value then changes as a result.

Example:

In a study of how introduction of eLearning modules affect the adherence to a program, a researcher could compare the course drop out rate (the ? variables) with the presence of eLearning module (the ? Variable), and attempt to draw a conclusion.

Hypothesis: Variables

An independent variable is the variable whose value one actively controls and can change.

A dependent variable is the thing whose value then changes as a result.

Hypothesis testing

In statistics, there are only two kinds of hypothesis that are being tested:

The null hypothesis or H0. The alternative hypothesis or H1.

Hypothesis testing

In education research, the null hypothesis may be used to test differences in intervention and control groups at the end of a program, and the assumption is that no difference exists between the two groups for the outcome variable being compared.

Not guilty unless proven otherwise

Hypothesis testing: Statistical significance

The “p” value or alpha is the chance you are willing to take to reject the Ho [not guilty], while you actually should have retained it.

The choice of the level is up to the experimenter and may be determined as a function of the phenomenon being studied.

PausePause

Experimental design

You have

performed a systematic review of literature

You have

generated an answerable question

You have

generated a

What is the next step?

You need to

design of a “clean” experiment

Example

You may wish to examine if a new hands on simulation based educational program leads to adherence to hand washing protocols [by nurses ….newly hired nurses …etc].

Is this a good research question?

Does it have all the parts?

What are the hypotheses?

Ho

H1

Experimental design

Example

You may wish to examine if a new hands on simulation based educational program leads to adherence to hand washing protocols.

Is this a good research question? OK, but not great

Does it have all the parts? No control, participants, outcomes.

What are the hypotheses?

Ho: Old and new programs are the same

H1: Old and new are different (note no directionality)

Experimental design

To test this hypothesis requires 2 conditions to be met (Cook

and Campbell; 1979).

1. Changes in the outcome occur after, rather than before the institution of the program/intervention.

2. The program/intervention are the only reasonable explanation for the changes in the outcome.

If there are any other explanations for the observed changes in outcomes the researcher cannot be confident that the presumed cause effect relationship is correct.

How do you know if there are other explanations?

Experimental design

Eliminating these alternative explanations is the purpose of a proper experimental design.

These are also known as threats to internal validity (Cook and Campbell 1979).

Experimental design

Minimizing Threats to Internal Validity:

Argument. This is the least effective ways to argue threats to internal validity.

- In a paper it is in the intro

Design. This is by far the most powerful method to rule out alternative explanations.

- In a paper it is in the methods

Analysis. The researcher can use various statistical analysis performed on the collected data.

- In a paper it is in the methods and results

Experimental design

The three ways of Minimizing Threats to Validity are not mutually exclusive and a good research plan should make use of multiple methods for reducing threats.

Experimental design

Design construction

• Most research designs can be conceptualized and represented graphically from four basic elements

• Time, randomization, groups, interventions, observations, etc.

XOXOXOXO

Experimental design

Time. In design notation time is represented horizontally.

Intervention (s). In design notation the intervention is depicted with the symbol "X".

Observation (s). Assessments and observations are depicted as the symbol "O".

XOXOXOXOX

Experimental design

Groups. Each group is indicated on a separate line.

Most importantly however, the manner in which groups are assigned to the conditions can be indicated by a letter:

• "R" represents random assignment, • "N" represents non-random assignment (i.e., a

nonequivalent group or cohort) • "C" may represent an assignment based on a cutoff

score.

Group 1: R O X OGroup 2: N O X O

Experimental design

The most basic causal relationship between educational intervention and outcome can be described using the following annotation:

XO

This is the most simple design and serves as a starting point for the development of better strategies.

What is wrong with it?

No control for threats to internal validity: • Level before intervention?• Historical events?

Experimental design

When the researcher is faced with being able to deliver an intervention to all participants one can include additional observations either before or after the intervention.

O O X O O

• Provides a "baseline"

• No intervention vs. intervention

• Additional posttest assessments: decay, or a lag.

What is wrong with this deign?

Still we do not know if X (or the intervention) caused change in O

Experimental design

Use a control group!

N O X ON O O

• The interpretation: The study was comprised of two groups, with non-random assignment of participants to each of the groups. There was an initial assessment before the intervention implementation. Subsequently, participants in the first group receive the intervention (indicated by X), while participants in the second group do not. Finally, all participants are reassessed.

Why is the first “O” important?

To ensure no bias

Experimental design

• In education especially the initial tests may be viewed as a contaminating factor.

R X OR O

• The posttest-only randomized experimental design allows the researcher to assume that there are randomized assignment of the participants to the two groups will ensure similar patient distribution.

Experimental design

• These designs assess the effectiveness of intervention when compared to a lack of alternative intervention.

• That is the C group did not get any intervention.

R X1 OR X2 O

Experimental design

• Inclusion of additional groups in the design may be necessary in order to rule out specific threats to validity.

• The researcher may be inclined to add an additional nonequivalent group from a similar institution.

R O X O R O ON O O

Experimental design

Another possibility is to use pre-post cohort groups:

N O X ON O

N O

The treatment group consists of current learners, the first comparison group last year’s learners assessed in the same year, and the second comparison group consists of the following year's learners.

Experimental design

The Nature of Good Design

• Be linked with theory (know your threats)

• Be innovative

• Be realistic

• Be flexible

• Do not over-design

• Word of the ay: parsimonious

Experimental design

PausePause

You have

performed a systematic review of literature

You have

generated an answerable question

You have

generated a

You have

designed a “clean” experiment

What is the next step?

You need to

be able to use the stats

• The differences are important, but more important is the variability (noise)

• Control confounding

variables!

Basic stats - variability

How do we measure that?

Independent-Samples T Test• This procedure compares means for two groups of cases

R X OR X O

• Example. Learners are randomly assigned to a new educational approach group and the rest follows standard curriculum.

T-test

Standard

New

Skill

T-test

Standard

New

Skill

T-test

Standard

New

Skill

T-test

Paired-Samples T Test• This procedure compares the means of two variables for

a single group. The procedure computes the differences between values of the two variables for each case and tests whether the average differs from 0.

O X O

• Example. Learners participate a new curriculum.

T-test

Post

Pre

Skill

O X O

T-test

One-Sample T Test• This procedure tests whether the mean of a single

variable differs from a specified constant.

• Examples. A researcher might want to test whether the skill improvement for the group participating in a new curriculum differs from a known average.

T-test

LimitationsOnly 2 groups or valueDoes not correct for number of tests

T-test

The One-Way ANOVA

• This technique is an extension of the independent-sample t test.

• In addition to determining that differences exist among the means, you may want to know which means differ. Post hoc tests are run after the experiment has been conducted to test for trends across categories.

ANOVA

Example. Learners are participating in 2 new programs. You want to know which one is better, but you are also interested to know how does the standard curriculum compare.

Independent variable:Program (New A, New B and Standard)NOTE: This is also known as between group or between

subject variable

Dependent variable:Skill

ANOVA

Standard

Skill

New A New B

ANOVA

LimitationAllows for only one independent variable

Q: What if one of the two programs is an eLearning module and the other is f2f? You may suspect that the cyber natives and cyber shy people may have different benefits.

ANOVA

Multifactor ANOVA

• Using this procedure, you can test for the effects of individual factors.

• You can also investigate interactions between factors.

• After an overall test has shown significance, you can use post hoc tests to evaluate differences among specific means.

ANOVA

Cyber shy

Cyber native

Skill

ANOVA

Standard

Skill

New A New B

ANOVA

Standard

Skill

New A

New A

New BNew B

ANOVA

Each of these tests has a non-parametric equivalent.

Why? When?

If 2 assumptions are violated:

1. The data does not follow normal distribution.

2. N<15

Non-parametric test

You have

performed a systematic review of literature

You have

generated an answerable question

You have

generated a

You have

designed a “clean” experiment

What is the next step?

You have

Been introduced to stats