Researching for Better Online Instructional Methods using AB Testing --- Zhongzhou Chen, Christopher...
-
Upload
claribel-cook -
Category
Documents
-
view
218 -
download
0
Transcript of Researching for Better Online Instructional Methods using AB Testing --- Zhongzhou Chen, Christopher...
Researching for Better Online Instructional
Methods using AB Testing
--- Zhongzhou Chen, Christopher Chudzicki, Peter Bohacek, Neset Demirci and David Pritchard
Multiple A/B Experiments in a Course
Unit 1 Unit 2
Independent Random Splits
Experiment 1 Experiment 2
Multiple A/B Experiments in a Course
Unit 1Experiment 1 (part 1)
Unit 2Experiment 2
Unit 3Experiment 1 (part 2)
Summary of A/B Experiment Features
• No limit on the number of groups, nor on the number of experiments in a course.
• Each group can receive a different number of materials.
• Group assignment is random.
• No student can "opt-out" of any of the experiments.
Our MOOC: 8.MReV
Content: College introductory level mechanics (calculus based)
Length: 14 Weeks (12 required, 2 optional) May 29th ------- Sep 14th
Initially Registered: 10709
Attempted Second Week’s Quiz: ~4100 (estimated)
Does length of video make a difference?
Office hour videos: ~10+ minutes
Talk about a challenging homework problem
Does length of video make a difference?
~2 minute video segments
Short Video
~10 minutes whole video
Long Video
Does length of video make a difference?
~2 minute video segments
Short Video
~10 minutes whole video
Long Video
Does length of video make a difference?
Research Questions:
• Difference in time spent watching video?
• Difference in number of questions answered?
• Difference in successfully solving the challenging homework problem?
Direct measurement video (by Peter Bohacek)
Time measured by counting frames
Player can play frame by frame
Distance measured by on screen ruler.
Direct Measurement Video
Common Homework Problem
Traditional Homework
Direct Measurement Video
Survey question
Direct Measurement Video
Research Questions:
• Does the extra interactivity result in more participation, or does the extra work lead to less participation?
• Can we identify any positive impact on students’ knowledge to justify the cost of creating these videos?
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
• Repetitive and highly focused training
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
• Repetitive and highly focused training
Domain Specific Problem Design
Problem Design/Auto Grading/ Customized Feedback
Drag and Drop format
Multiple Choice vs. Deliberate Practice
Research Questions:
• Does increasing interactivity result in more engagement? (i.e. Are students more “deliberate” in their practice?)
• Compared to traditional problems, do Drag and Drop problems serve as better/worse assessments?
• Do “deliberate practice” result in any detectable learning gain?
Discussion Questions
• Other innovative instructional methods worth putting online and testing via AB experiment?
Discussion Questions
• How should we design our experiments to suit the (unique) MOOC population?
• How can we transfer the experimental results obtained through a MOOC to a "flipped" classroom despite the difference in population?
Discussion Questions
• What criteria should we use to judge whether one pedagogy is better than another? Learning? Engagement? More discussion? Less attrition?