Researching for Better O nline I nstructional M ethods using AB Testing

Researching for Better Online Instructional

Methods using AB Testing

--- Zhongzhou Chen, Christopher Chudzicki, Peter Bohacek, Neset Demirci and David Pritchard

Teaching is an old practice…

We enter the age of the Internet

Now we can teach like this

Can we find better ways to teach in an online

environment?

In a normal MOOC

Entire MOOC Population

Course Material

A/B Testing

Traditional Teaching Method Experimental Teaching Method

A/B Testing

Simple Design of an A/B Expriment

Treatment Control

Common Assessment

Multiple A/B Experiments in a Course

Unit 1Experiment 1


Unit 1 Unit 2

Independent Random Splits

Experiment 1 Experiment 2


Unit 1Experiment 1 (part 1)

Unit 2Experiment 2

Unit 3Experiment 1 (part 2)

Summary of A/B Experiment Features

• No limit on the number of groups, nor on the number of experiments in a course.

• Each group can receive a different number of materials.

• Group assignment is random.

• No student can "opt-out" of any of the experiments.

Examples of A/B Experiments in our MOOC

Our MOOC: 8.MReV

Content: College introductory level mechanics (calculus based)

Length: 14 Weeks (12 required, 2 optional) May 29th ------- Sep 14th

Initially Registered: 10709

Attempted Second Week’s Quiz: ~4100 (estimated)

Our MOOC: 8.MReV

Total Number of AB experiments: 7

Experiment 1: Does length of video make a difference?

Does length of video make a difference?

Office hour videos: ~10+ minutes

Talk about a challenging homework problem


~2 minute video segments

Short Video


~2 minute video segments

Short Video

~10 minutes whole video

Long Video

Experiment Procedure

Challenging Homework Problem (Graded)

Experiment Procedure

Long Video Short VideoChallenging Homework Problem (Graded)


Research Questions:

• Difference in time spent watching video?

• Difference in number of questions answered?

• Difference in successfully solving the challenging homework problem?

Increasing Interactivity of Course Material

Interactivity of Course Material

Traditional problem vs. Direct measurement video

Traditional Physics problem

Provides all necessary information

Direct measurement video (by Peter Bohacek)

Time measured by counting frames

Player can play frame by frame

Distance measured by on screen ruler.

Direct Measurement Video

Common Homework Problem

Traditional Homework


Survey question


Research Questions:

• Does the extra interactivity result in more participation, or does the extra work lead to less participation?

• Can we identify any positive impact on students’ knowledge to justify the cost of creating these videos?

Interactivity of Course Material

Multiple Choice vs. Drag and Drop Problem

Multiple Choice

Extraneous Cognitive Load

Traditional Multiple Choice question

Drag and Drop Example

Drag and Drop Example 2

Drag and Drop and Deliberate Practice

Deliberate Practice: (Erickson)


Deliberate Practice:

• Focus on practicing one skill at a time




• Providing immediate specific feedback





• Repetitive and highly focused training





• Repetitive and highly focused training

Domain Specific Problem Design

Problem Design/Auto Grading/ Customized Feedback

Drag and Drop format

Experimental Setup

Entire MOOC population

Experimental Setup

Multiple Choice vs. Deliberate Practice

Research Questions:

• Does increasing interactivity result in more engagement? (i.e. Are students more “deliberate” in their practice?)

• Compared to traditional problems, do Drag and Drop problems serve as better/worse assessments?

• Do “deliberate practice” result in any detectable learning gain?

Thank you

Comments and Questions?

Discussion Questions

• Other innovative instructional methods worth putting online and testing via AB experiment?


• How should we design our experiments to suit the (unique) MOOC population?

• How can we transfer the experimental results obtained through a MOOC to a "flipped" classroom despite the difference in population?


• What criteria should we use to judge whether one pedagogy is better than another? Learning? Engagement? More discussion? Less attrition?


• How do we design effective assessments of learning when students have the freedom to access any external resources?

Researching for Better O nline I nstructional M ethods using AB Testing

Documents

Transcript of Researching for Better O nline I nstructional M ethods using AB Testing