Towards a new model for measuring success in MOOCs: The role of participants’ intentions
-
Upload
open-education-consortium -
Category
Education
-
view
30 -
download
2
Transcript of Towards a new model for measuring success in MOOCs: The role of participants’ intentions
![Page 1: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/1.jpg)
Marco Kalz, Maartje Henderikx & Karel KreijnsOpen University of the Netherlands
Chair of Open EducationWelten Institute/Faculty MST
OE Global, 8 March 2017
AN ALTERNATIVE APPROACH FOR MEASURING MOOC SUCCES BASED
ON PARTICIPANT’S INTENTIONS
![Page 2: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/2.jpg)
SOONER TEAM
dr. Jeroen Janssen
dr. ir. Karel Kreijns
dr. Peter van
Rosmalenprof dr.
Marco Kalzprof dr.
Liesbeth Kester
dr. Anoushka
van Leeuwen
drs. Renee Jansen
drs. Maartje Henderikx
drs. Julia Kasch
drs. Martine Schophuize
n
![Page 3: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/3.jpg)
MICRO, MESO, MACRO
PhD B
Motivation and intentions as key
to drop-out
PhD C
Scaling of support,
feedback and interaction
PhD D
Organizational development
and educational innovation
PhD A
Self-regulated learning skill acquisition
MACRO:Organizational
Level
MESO:Course level
MICRO: Learner
level
Fundamental research Accompanying research
![Page 4: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/4.jpg)
DROP
![Page 5: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/5.jpg)
DROPOUT: THE CATEGORY ERROR
![Page 6: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/6.jpg)
DROPOUT: THE CATEGORY ERROR
Jordan, 2015
![Page 7: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/7.jpg)
WHAT STUDENTS SAY
![Page 8: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/8.jpg)
www.sooner.nu
A […] more important limitation […] is the tendency to ignore the perspective of the individual […]Such definitions of dropout […] imply connotations of inferiority […] of the individual dropping out.
Tinto & Cullen, 1973
![Page 9: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/9.jpg)
Individual goal achievement as new measurement approach
• How can we measure goal achievement?
• What is a valid model for goal achievement in MOOCs?
• Do intentions in MOOCs predict behaviour?
• Do implementation intentions play a role?
![Page 10: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/10.jpg)
INTENTIONS
Reich, 2014Kizilcek, Piech, & Schneider, 2013
![Page 11: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/11.jpg)
THEORETICAL MODEL
(Kalz et al., 2015)
![Page 12: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/12.jpg)
INTENTIONS
Average correlation between intention and behaviour in other domains of study between .48 (Sutton, 1998) and .53 (Sheeran, 2002).
![Page 13: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/13.jpg)
Intention formation
(Gollwitzer, 1990)
![Page 14: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/14.jpg)
INTENTION-BEHAVIOUR CLUSTERSConsistent Inconsistent
Inclined actors
Intend to act and act
Inclined abstainers
Intend to act, but do not act
Disinclined actors
Don’t intend to act, but act
Disinclined abstainers
Don’t intend to act and don’t act
Gollwitzer & Sheeran, 2006
![Page 15: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/15.jpg)
www.sooner.nu
“Forming a goal intention does not prepare people sufficiently for dealing
with self-regulatory problems in initiating, maintaining, disengaging
from,or over-extending oneself in goal
striving.”
Tinto & Cullen, 1973
![Page 16: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/16.jpg)
Implementation intentions
Implementation intentions define the • when,• where and • howof goal striving.
Meta-analysis by Gollwitzer and Sheeran (2006):• 94 studies, N=8000• Effect size d = .65
Simple plans, huge impact.
![Page 17: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/17.jpg)
Implementation intentions
“When situation X arises, I will perform response Y!”
![Page 18: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/18.jpg)
Implementation intentions“I will reserve each Tuesday
afternoon to learn at my desk ”“When I cannot learn on Tuesday at
home, I will skip football on Saturdayto learn then”
“I will connect each morning for anhour to fellow student of the MOOC
to discuss questions and assignments”
![Page 19: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/19.jpg)
Intention formation in MOOCs
![Page 20: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/20.jpg)
3 main research questions
• 1.) Intention-behaviour correlation• 2.) The alternative measurement approach• 3.) The role of implementation intentions
![Page 21: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/21.jpg)
TWO CASESMOOC “The Adolescent Brain”
(1500 participants)EMMA platform
MOOC “Marine Litter” (6500 participants)
OpenEdX
![Page 22: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/22.jpg)
METHODS• Construction of items that fit to the design of the MOOC (Pre-
and Post) according to guidelines by Sutton (2008) and Fischbein & Ayzen (2010)
• Reuse of items by Rise, Thompson, & Verplanken (2003) for implementation intentions
• Construction of items for barriers (to which degree their were expected, had an impact and could be resolved)
• Background variables (socio-economic profile, prior knowledge, lifelong learning profile, ICT skills)
![Page 23: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/23.jpg)
CORRELATION BETWEEN INTENTION AND BEHAVIOUR
MOOC “The Adolescent Brain” (1500 participants)
rs = .16, p>0,05, n=63
MOOC “Marine Litter” (6500 participants)
rs = .34, p<0,01, n=86
![Page 24: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/24.jpg)
MEASURING THE INTENTION-BEHAVIOUR GAP
MOOC “The Adolescent Brain” (n=86)MOOC “Marine Litter” (n=63)
![Page 25: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/25.jpg)
COMPARING SUCCESS PARADIGMSMOOC “Marine Litter”
The traditional approach
6500, 4000 starters, 400 certificates
10 % successful learners
90 % dropouts
The intention approach
49,2 % Inclined abstainers23,8 % Inclined actors27 % Disclined abstainers
![Page 26: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/26.jpg)
COMPARING SUCCESS PARADIGMSMOOC “The Adolescent Brain”
The traditional approach
1500 participants, 80 certificates
5,5 % successful learners
95,5 % dropouts
The intention approach
23,3 % Inclined abstainers8,1 % Inclined actors68,6 % Disclined abstainers
![Page 27: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/27.jpg)
Influence of implementation intentions
(Kreijns, Kalz, Castaño-Munoz, & Punie, accepted)
![Page 28: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/28.jpg)
CONCLUSIONS & LIMITATIONS• Taking into account participants’ intention opens new
options for understanding success in MOOCs more objectively
• Individual success measurement might help to identify powerful instructional design/ challenging designs
• Not all drop-out has implications for course design• Self-reported behaviour needs to be combined with real behavioural data to check reliability (data-triangulation through learning analytics)
![Page 29: Towards a new model for measuring success in MOOCs: The role of participants’ intentions](https://reader033.fdocuments.us/reader033/viewer/2022051707/58ce7a041a28abdc578b750b/html5/thumbnails/29.jpg)
LIMITATIONS AND FUTURE WORK• Timespan between measurements influences
correlation between intention and behaviour• Potential survival effects through survey
participation• Dynamics in intention-formation (adapting
intentions during the course)• Quality of the intention-behaviour gap• This measurement needs to be combined with
direct measurement of impact on learning• Identification of most important barriers
responsible for the intention-behaviour gap (Henderikx, Kreijns, & Kalz, accepted)