Speakers: Nishan Joomun, Senior Multimedia Designer & Dr. Patrick Devey, Chief Learning Officer.
The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer...
-
Upload
leona-shelton -
Category
Documents
-
view
215 -
download
0
Transcript of The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer...
![Page 1: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/1.jpg)
The Impact of Learning Analytics on eLearning Design
Patrick Devey, Ph.D.Chief Learning Officer
KnowledgeOne Inc.Montréal, Québec, Canada
![Page 2: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/2.jpg)
THE MISSING STUDENTS CAPER
![Page 3: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/3.jpg)
“I seem to be missing a significant number of students at the final exam…”
-FINA 200 Professor
![Page 4: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/4.jpg)
0%
5%
10%
15%
20%
25%
30%
Voluntary Student Dropout
eConcordia FINA 200
“help…?”-FINA 200 Professor
![Page 5: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/5.jpg)
WHERE DO WE BEGIN?
![Page 6: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/6.jpg)
Student Logs (from LMS)
Over 50,000 entries!!!
![Page 7: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/7.jpg)
Hazard Rate using Registration Data
1 2 3 4 5 6 7 8 9.000
.010
.020
.030
.040
.050
.060
FINA 200
Time (Week)
Haz
ard
Rate
![Page 8: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/8.jpg)
2 3 4 5 6 7 8 9 10.000
.010
.020
.030
.040
.050
.060
Time (Week)
Haz
ard
Rate
Ass
ign
men
t 1
Ass
ign
men
t 2
Ass
ign
men
t 3
Hazard Rate using Registration Data
![Page 9: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/9.jpg)
THE MISSING STUDENTS CAPER
…will return after these informative messages!
![Page 10: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/10.jpg)
![Page 11: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/11.jpg)
![Page 12: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/12.jpg)
Learning Analytics
Data must be:• relevant to the learners and to the instructors,• focuses on individual learners or a course,• leads to informed decisions to improve learning
services or products at the learner/course level.
“The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.”
-The Society for Learning Analytics Research (SoLAR)
![Page 13: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/13.jpg)
13
Pascarella & Chapman (1983)
Longitudinal Data: Path Analysis
R2 = .120
![Page 14: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/14.jpg)
Data Sources
LMS
Google Analytics
Surveys
Industry Statistics
& Reports
EnrolmentHR System
xAPICustom
![Page 15: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/15.jpg)
INDUSTRY STATISTICS & REPORTS
Easy to get, but do they apply to your situation?
![Page 16: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/16.jpg)
http://en.wikipedia.org/wiki/Usage_share_of_operating_systems
![Page 17: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/17.jpg)
Nov-07 Mar-09 Aug-10 Dec-11 May-130.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Browser History
Internet Explorer Firefox Chrome Safari Opera
![Page 18: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/18.jpg)
Google Chrome Safari MS Internet Explorer Firefox Other0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Browser Comparison
W3 Concordia University
![Page 19: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/19.jpg)
Windows Linux Mac Mobile0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
OS Comparison
w3 eConcordia
![Page 20: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/20.jpg)
How could this data be used?
• What are the main browsers used by our learners?– Quality assurance testing– Compatibility (HTML5, plug-ins, etc…)
• What are the primary operating systems being used by your learners?
• Are your learners making use of mobile devices?• What are the trends?
– Should you be making preparations because of them?
![Page 21: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/21.jpg)
LEARNING MANAGEMENT SYSTEM
It’s already being collected, might as well use it!
![Page 22: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/22.jpg)
![Page 23: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/23.jpg)
![Page 24: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/24.jpg)
![Page 25: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/25.jpg)
![Page 26: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/26.jpg)
How could this data be used?
• When are learners most likely to be active?– Schedule live office hours– Schedule server maintenance
• Are learners achieving the learning objectives?– Adjustments to assessments– Adjustments to existing content– Addition of remedial content
• Is time-on-task correlated with performance?– Log-ins vs. performance– Identifying learners who have “stopped out”
![Page 27: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/27.jpg)
GOOGLE ANALYTICS
Tuning in to the “Greatest Hits”
![Page 28: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/28.jpg)
Demographic Data
Browser Statistics, OS, Mobile, etc…
![Page 29: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/29.jpg)
![Page 30: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/30.jpg)
![Page 31: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/31.jpg)
![Page 32: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/32.jpg)
![Page 33: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/33.jpg)
Qui
z 1
Qui
z 2
Qui
z 3
Fina
l Exa
m
![Page 34: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/34.jpg)
![Page 35: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/35.jpg)
![Page 36: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/36.jpg)
Qu
iz 1
Ass
ign
men
t 1O
rie
nta
tion
Se
ssio
n
Ass
ign
men
t 2
Qu
iz 2
Ass
ign
men
t 3
Qu
iz 3
Fin
al E
xamMidterm Break
![Page 37: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/37.jpg)
![Page 38: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/38.jpg)
![Page 39: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/39.jpg)
031.aspx 042.aspx
![Page 40: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/40.jpg)
How could this data be used?
• Is my visual design effective?– Do I have buttons that are not being used? Can the site be
optimized?• How are learners navigating through the course website?
– Are the navigation buttons being used as designed? Can they be improved or adapted?
• When are learners using particular course materials?– Is there an opportunity to send remedial content when it is
relevant?• How long are learners spending on individual sections?
– Is there an opportunity to improve the content?
![Page 41: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/41.jpg)
DESIGN YOUR OWN DATA COLLECTION
If the data you want is not being collected, design a way to get it!
![Page 42: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/42.jpg)
• Who are the students enrolling in online courses at Concordia University? How do they compare to the general student population at Concordia as well as to the distance education literature?
• Why are students enrolling in online courses? What expectations do they have about their online course?
• Why do students drop out of online courses? Are there demographic, attitudinal, integration, and/or behavioural variables related to persistence?
• When do students drop out of online courses?
• What interventions can the institution make to curtail attrition in their online courses?
What Patrick wanted to know
![Page 43: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/43.jpg)
Timeframe of the Study
1 2 3 4 5 6 7 8 9 10 11 12 13
Weeks of the Semester
1. Web-Based Learning
Questionnaire
4. Registration Data3. Exit Survey
DNE DISCDropout Zone
2. Weekly Survey
![Page 44: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/44.jpg)
44
![Page 45: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/45.jpg)
![Page 46: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/46.jpg)
THE MISSING STUDENTS CAPER
…and now, back to:
![Page 47: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/47.jpg)
Finance: Course Satisfaction
2 3 4 5 6 7 8 9 1064
66
68
70
72
74
76
78
Week
Aver
age
![Page 48: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/48.jpg)
Finance: Hazard Rate/Satisfaction
2 3 4 5 6 7 8 9 10.000
.010
.020
.030
.040
.050
.060
Time (Week)
Haza
rd R
ate
48
Ass
ign
men
t 1
Ass
ign
men
t 2
Ass
ign
men
t 3
2 3 4 5 6 7 8 9 1064
66
68
70
72
74
76
78
Week
Aver
age
![Page 49: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/49.jpg)
The Facts
• Significant portion of the class were non-business students– Struggling with basic concepts such as time value
of money calculations• Older, worked full-time, studied part-time
– Did not have time to seek out instructor “in-between classes” or to go to campus
• Dropping out because of a fear of poor performance in the course
![Page 50: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/50.jpg)
Interventions
• Live virtual office hours and tutorials scheduled when they were most needed– Dealt with the assignments– Sessions were recorded and posted on the course website
• Development of additional remedial content to address problem areas
• Provided links to the remedial content in the course content when they were relevant
• Altered assessments (rubric) for non-business students
CASE
CLOSED
(kind of)
![Page 51: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/51.jpg)
WHAT DO YOU WANT TO IMPROVE?
![Page 52: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/52.jpg)
What do you want to improve?
• Retention rate• Transfer of training• Achievement of learning objectives• Performance on assessments• Usability of the course website• Services to instructors and learners• ???
![Page 53: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/53.jpg)
Take Action
1. Go beyond pass/fail to validate your learning solution.
2. Don’t let the system limit you. 3. Design the instruments you need to collect
the metrics you want.4. Don’t be intimidated by the numbers.5. Continuously seek ways to improve.
![Page 54: The Impact of Learning Analytics on eLearning Design Patrick Devey, Ph.D. Chief Learning Officer KnowledgeOne Inc. Montréal, Québec, Canada.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649eb65503460f94bc00ed/html5/thumbnails/54.jpg)
Thank you! Merci!
Questions?
[email protected] @patrickdevey
http://survivingelearning.blogspot.ca
http://www.knowledgeone.ca