1

In Search of “What Works” in In Search of “What Works” in Online and Distance Education: A Online and Distance Education: A Report of Two Meta-AnalysesReport of Two Meta-Analyses

In Search of “What Works” in In Search of “What Works” in Online and Distance Education: A Online and Distance Education: A Report of Two Meta-AnalysesReport of Two Meta-Analyses

Robert M. Bernard, Philip C. Abrami and Robert M. Bernard, Philip C. Abrami and

Eugene BorokhovskiEugene Borokhovski

Centre for the Study of Learning and Centre for the Study of Learning and PerformancePerformance

Concordia UniversityConcordia Universityhttp://doe.concordia.ca/cslphttp://doe.concordia.ca/cslp

Research supported by grants from FQRSC and SSHRC to Bernard and Abrami

2

Purposes of meta-analysisPurposes of meta-analysis

• Estimate the population central tendency Estimate the population central tendency and variability of effect sizes between an and variability of effect sizes between an intervention (treatment condition) and a intervention (treatment condition) and a control condition.control condition.

• Explore unexplained variability through the Explore unexplained variability through the analysis of methodological and substantive analysis of methodological and substantive coded study features.coded study features.

3

What is an effect size?What is an effect size?

• Standardized difference between a Standardized difference between a treatment mean and a control meantreatment mean and a control mean

• Positive if treatment > controlPositive if treatment > control

• Negative if treatment < controlNegative if treatment < control

• Basic Equations:Basic Equations:

dCohen YExperimental YControl

SDPooled

gHedges d 1 3

4(NE NC 9

4

What is a weighted effect size?What is a weighted effect size?

• Effect size units of analysis are samples that differ Effect size units of analysis are samples that differ in their size (i.e., number of participants)in their size (i.e., number of participants)

• Inverse variance (1/seInverse variance (1/se22) sample size weighting is ) sample size weighting is applied to each effect size so that large samples applied to each effect size so that large samples are counted more than small samplesare counted more than small samples

• Statistical synthesis (averages Statistical synthesis (averages [g+],[g+], variability variability [Q-statistic],[Q-statistic], ANOVA, regression) take weighting ANOVA, regression) take weighting into accountinto account

5

What about variability among effect sizes?What about variability among effect sizes?

g+

Distribution1: Homogeneous

Distribution 2: Heterogeneous

Gray shaded area is variation left to be explained by moderators.

No variation left to be explained by moderators.

Q-statistic is low and not significant

Q-statistic is high and significant

The Q-statistic is used to measure variability of effect size (it is similar to a sum of squares and is evaluated with the 2 sampling distribution)

6

Project 1: 2000 – 2004Project 1: 2000 – 2004

• Question:Question: How does distance education compare to How does distance education compare to classroom instruction? classroom instruction? (inclusive dates 1985-2002)(inclusive dates 1985-2002)

• Total number of effect sizes: Total number of effect sizes: k = 232k = 232

• Measures:Measures: Achievement, Attitudes and Retention Achievement, Attitudes and Retention (opposite of drop-out)(opposite of drop-out)

• Divided into Divided into AsynchronousAsynchronous and and SynchronousSynchronous DE DE

Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. meta-analysis of the empirical literature. Review of Educational Research, 74Review of Educational Research, 74(3), 379-439.(3), 379-439.

7

Summary of results: AchievementSummary of results: Achievement

Type of DEType of DE kk g+g+ Sig.Sig.

CombinedCombined 318*318* 0.0130.013** > 0.05> 0.05

SynchronousSynchronous 9292 ––0.1020.102** < 0.05< 0.05

AsynchronousAsynchronous 174174 0.0530.053** < 0.05< 0.05

Achievement Outcomes

*Significantly heterogeneous average effect

8

Summary of results: AttitudesSummary of results: Attitudes

Type of DEType of DE kk g+g+ Sig.Sig.

CombinedCombined 154154 ––0.0810.081** < 0.05< 0.05

SynchronousSynchronous 8383 ––0.1850.185** < 0.05< 0.05

AsynchronousAsynchronous 7171 ––0.0340.034** > 0.05> 0.05

Attitude Outcomes

*Significantly heterogeneous average effect

9

Summary of results: RetentionSummary of results: Retention

Retention Outcomes

Type of DEType of DE kk g+g+ Sig.Sig.

CombinedCombined 103103 ––0.0570.057** < 0.05< 0.05

SynchronousSynchronous 1717 0.0050.005 > 0.05> 0.05

AsynchronousAsynchronous 5353 ––0.0930.093** < 0.05< 0.05

*Significantly heterogeneous effect sizes

10

Primary findingsPrimary findings

• DE and CI are essentially DE and CI are essentially equalequal (g+ ≈ 0.0 to low (g+ ≈ 0.0 to low average effect) on all measuresaverage effect) on all measures

• Effect size distributions are heterogeneous; some Effect size distributions are heterogeneous; some DE >> CI, some DE << CIDE >> CI, some DE << CI

• Generally poor methodological qualityGenerally poor methodological quality• Pedagogical study features account for more Pedagogical study features account for more

variation than media study features (Clark, 1994)variation than media study features (Clark, 1994)• Interactive DE an important variable*Interactive DE an important variable*

*Lou, Y., Bernard, R.M., & Abrami, P.C. (2006). Media and pedagogy in undergraduate distance education: A theory-based meta-analysis of empirical literature. Educational Technology Research & Development, 54(2), 141-176.

11

Project 2: 2005 – presentProject 2: 2005 – present

Question:Question: What are the effects of three types of interaction in What are the effects of three types of interaction in DE vs. DE studies? (Inclusive dates 1985 to 2006)DE vs. DE studies? (Inclusive dates 1985 to 2006)

Two definitions of interaction:Two definitions of interaction:

• Interaction should refer “in a restrictive manner to cover Interaction should refer “in a restrictive manner to cover only those activities where the student is in two-way only those activities where the student is in two-way contact with another person (or persons)” contact with another person (or persons)” (Daniel & (Daniel & Marquis, 1988, p. 339)Marquis, 1988, p. 339)

• Interactions are “reciprocal events that require at least two Interactions are “reciprocal events that require at least two objects and two actions. Interactions occur when these objects and two actions. Interactions occur when these objects and actions mutually influence one another objects and actions mutually influence one another (Wagner, 1994, p. 8)(Wagner, 1994, p. 8)

12

Project 2: 2005 – presentProject 2: 2005 – present

Moore (1989) distinctions are:Moore (1989) distinctions are:

• Three types of interactionThree types of interactiono student-student interactionstudent-student interactiono student-teacher interaction student-teacher interaction o Student-content interactionStudent-content interaction

Anderson (2003) hypotheses state:Anderson (2003) hypotheses state:

• Deep, meaningful learning is produced from 2 out of 3 Deep, meaningful learning is produced from 2 out of 3 interactions at a high levelinteractions at a high level

• High levels of more than 1 out of 3 interactions will produce High levels of more than 1 out of 3 interactions will produce satisfying educational experiencesatisfying educational experience

• Increasing satisfaction through interaction may not be as time Increasing satisfaction through interaction may not be as time or cost-effective as less interactive learning sequencesor cost-effective as less interactive learning sequences

13

The unique problem in this meta-analysisThe unique problem in this meta-analysis

• In most meta-analyses, treatment and control In most meta-analyses, treatment and control designations are unambiguous (e.g., DE vs. CI)designations are unambiguous (e.g., DE vs. CI)

• In DE vs. DE studies, which condition is the In DE vs. DE studies, which condition is the treatment and which is the control?treatment and which is the control?

• Sorted 74 achievement and 44 attitude effects into Sorted 74 achievement and 44 attitude effects into SS, ST and SC categoriesSS, ST and SC categories

• Two judges determined which condition was the Two judges determined which condition was the “best type” for each category; that group became “best type” for each category; that group became the treatment and the other the control the treatment and the other the control

14

Do the three types of interaction differ? Moore’s distinctionsDo the three types of interaction differ? Moore’s distinctions

Achievement and Attitude Outcomes

Achievement Attitudes Interaction Categories k g+adj. k g+adj. Student-Student 10 0.342 6 0.358 Student-Teac her 44 0.254 30 0.052 Student-Content 20 0.339 8 0.136 Total 74 0.291 44 0.090 Between-class 2.437 6.892*

Moore’s distinctions seem to apply for achievement (equal importance), but not for attitudes (however, samples are low for SS and SC)

15

Investigating “treatment strength”Investigating “treatment strength”

• Anderson’s hypotheses involve improving Anderson’s hypotheses involve improving the ‘strength’ of interaction treatmentsthe ‘strength’ of interaction treatments

• We defined treatment strength as ratings of We defined treatment strength as ratings of difference between two conditionsdifference between two conditions

Coded strength as:Coded strength as: • • 0) conditions are equal0) conditions are equal• • 1) treatment is > control1) treatment is > control• • 2) treatment is >> control2) treatment is >> control

16

Does strengthening interaction improve achievement and attitudes? Anderson’s hypothesesDoes strengthening interaction improve achievement and attitudes? Anderson’s hypotheses

Anderson’s first hypothesis about achievement appears to be supported

Anderson’s second hypothesis about satisfaction (attitude) appears to be supported, but only to an extent (i.e., only 5 studies in High Category)

Achievement and Attitude Outcomes

Achievement Attitudes Interaction Strength k g+adj. SE k g+adj. SE Low Strength 30 0.163 0.043 21 0.071 0.042 Med Strength 29 0.418 0.044 18 0.170 0.043 High Strength 15 0.305 0.062 5 -0.173 0.091 Total 74 0.291 0.027 44 0.090 0.029 (Q) Between-class 17.582* 12.060*

17

Do interaction types differ in strength?Do interaction types differ in strength?

Achievement Outcomes

Attitude Outcomes

Student-Student Student-Teacher Student-Content Levels of Treatment Strength k g+adj. k g+ k g+adj. Treatment = Control 36 0.350 18 0.414 35 0.247 Treatment > Control 19 0.223 29 0.212 27 0.212 Treatment >> Control 19 0.277 27 0.310 12 0.515 Total 74 0.291 74 0.291 74 0.291 (Q) Between-class 4.025 8.721* 18.523**

Student-Student Student-Teacher Student-Content Levels of Treatment Strength k g+adj. k g+adj. k g+adj. Treatment = Control 17 0.140 6 0.319 27 0.018 Treatment > Control 19 0.072 21 0.064 14 0.226 Treatment >> Control 8 0.030 17 0.032 3 0.183 Total 44 0.090 44 0.090 44 0.090 (Q) Between-class 1.897 11.361** 10.872**

18

Do interaction combinations differ?Do interaction combinations differ?Achievement Outcomes

Attitude Outcomes

SS + ST SS + SC ST + SC Levels of Treatment Strength k g+adj. k g+adj. k g+adj. Equal (0) 2 0.264 9 -0.033 2 0.378 Low (1) 21 0.057 25 0.110 22 0.103 Medium (2) 17 0.161 9 0.107 20 0.072 High (3) 4 -0.184 1 0.040 0 0.000

Total 44 0.090 44 0.090 44 0.090 (Q) Between-class 15.415* 2.833 2.833

SS + ST SS + SC ST + SC Levels of Treatment Strength k g+adj. k g+adj. k g+adj. Equal (0) 11 0.492 11 0.147 4 0.099 Low (1) 29 0.163 34 0.220 32 0.211 Medium (2) 22 0.413 28 0.382 37 0.367 High (3) 12 0.190 1 1.047 1 0.558

Total 74 0.291 74 0.291 74 0.291 (Q) Between-class 25.698* 13.329* 12.629*

19

What have we learned about the effects of interaction on achievement?What have we learned about the effects of interaction on achievement?

• The presence of any type of interaction enhances The presence of any type of interaction enhances achievement outcomesachievement outcomes

• Increasing cognitive engagement (i.e., providing Increasing cognitive engagement (i.e., providing the conditions for interaction to occur) improves the conditions for interaction to occur) improves achievement (i.e., learning)achievement (i.e., learning)

• This is especially true for student-content This is especially true for student-content interaction and any combination that involves interaction and any combination that involves student-content interactionstudent-content interaction

• Strengthening student-student interaction also Strengthening student-student interaction also appears to influence achievementappears to influence achievement

20

What have we learned about the effects of interaction on attitudes (satisfaction)?What have we learned about the effects of interaction on attitudes (satisfaction)?

• The relationship between attitudes and interaction The relationship between attitudes and interaction is more complex than for achievementis more complex than for achievement

• Student-student interaction seems importantStudent-student interaction seems important

• Strengthening interactions (in general) has a Strengthening interactions (in general) has a modest impact modest impact

• The role of the teacher in DE seems to have a The role of the teacher in DE seems to have a variable effect on attitudesvariable effect on attitudes

• There is a slight suggestion that increasing SS + There is a slight suggestion that increasing SS + ST has an effect on attitudesST has an effect on attitudes

21

ReferencesReferencesAnderson, T. (2003). Getting the mix right: An updated and theoretical rationale for interaction. Anderson, T. (2003). Getting the mix right: An updated and theoretical rationale for interaction. IRRODLIRRODL, 4(2) , 4(2)

[Online][Online]

Bernard, R.M., Abrami, P.C., Lou, Y. Borokhovski, E., Wade, A. Wozney, L., Wallet, P.A., Fiset, M., & Huang, Bernard, R.M., Abrami, P.C., Lou, Y. Borokhovski, E., Wade, A. Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. literature. Review of Educational Research, 74Review of Educational Research, 74(3), 379-439.(3), 379-439.

Bernard, R.M., Abrami, P.C., Bernard, R.M., Abrami, P.C.,

Daniel, J., & Marquis, C. (1979). Interaction and independence: Getting the mixture right. Daniel, J., & Marquis, C. (1979). Interaction and independence: Getting the mixture right. Teaching at a Teaching at a DistanceDistance, , 1515, 25-44., 25-44.

Lou, Y., Bernard, R.M., & Abrami, P.C. (2006). Media and pedagogy in undergraduate distance education: A theory-based meta-analysis of empirical literature. Educational Technology Research & Development, 54(2), 141-176.

Moore, M.G. (1989). Three types of interaction. Moore, M.G. (1989). Three types of interaction. American Journal of Distance Education, 3American Journal of Distance Education, 3(2), 1-6.(2), 1-6.

Wagner, E.D. (1994). In support of a functional definition of interaction. Wagner, E.D. (1994). In support of a functional definition of interaction. American Journal of Distance American Journal of Distance Education, 8Education, 8(2), 6-26.(2), 6-26.

Borokhovski, E., Wade, A., Tamin, R. & Surkes, M. Borokhovski, E., Wade, A., Tamin, R. & Surkes, M. (2008). Examining Three Forms of Interaction in (2008). Examining Three Forms of Interaction in Distance Education: A Meta-Analysis of Between-DE Distance Education: A Meta-Analysis of Between-DE Studies. Manuscript in preparation.Studies. Manuscript in preparation.