Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class...
Transcript of Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class...
![Page 1: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/1.jpg)
#THETA2017
This work is licensed under a Creative Commons Attribution 4.0 International License
Does Echo360‐ALP analytics predict the relationship between
student behaviours and learning?
Colin Montpetit, Sonya Sabourin
![Page 2: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/2.jpg)
Leveraging the Echo360-ALP to Transform A Large Enrollment
Lecture Classroom into an Active Learning Zone
Colin Montpetit, Ph.D., Assistant Professor
Sonya Sabourin, B.Sc. Honour’s student
Department of Biology, University of Ottawa
2015 TECHTEACH CONFERENCE - UIC
Does Echo360-ALP analytics predict the relationship
between student behaviours and learning?
![Page 3: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/3.jpg)
2015 TECHTEACH CONFERENCE - UIC
Modules (self-paced)• Targeted readings
• Interactive act./Videos
• Reflective act.
• Module quiz OnlineBlackBoard LMSInitial exposure
Summative• Two-Stage Collaborative Exams
(2 midterms + 1 final)
Assessments
Labs• Experiments
• Worksheets, reports
• Quizzes
F2F-Labs
Defined Learning Outcomes, 5E Learning Cycle
Class• Clicker case studies
• Peer-Instruction
• Just-in-Time Teaching
F2F-class
Echo360-ALP
65% 20%
15%
![Page 4: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/4.jpg)
Echo360- Active Learning Platform
2015 TECHTEACH CONFERENCE - UIC
![Page 5: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/5.jpg)
2015 TECHTEACH CONFERENCE - UIC
What is the class usage profile?
![Page 6: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/6.jpg)
2015 TECHTEACH CONFERENCE - UIC
Engagement data provides insight via dashboards
Confusion
Questions
posed
Activity
answers/
participation
Notes
Attendance
Presentation
views
Learning Analytics Database
Lecture capture
![Page 7: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/7.jpg)
2015 TECHTEACH CONFERENCE - UIC
Attendance, Participation, Activity Score
![Page 8: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/8.jpg)
2015 TECHTEACH CONFERENCE - UIC
Note Taking
![Page 9: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/9.jpg)
2015 TECHTEACH CONFERENCE - UIC
Presentation Views
![Page 10: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/10.jpg)
2015 TECHTEACH CONFERENCE - UIC
Video Views
![Page 11: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/11.jpg)
Caroline Petit-Turcotte Winter 2015
447 different students asked at least 1 question
2015 TECHTEACH CONFERENCE - UIC
Questions to the professorBIO1140 – Introduction to Cell Biology
![Page 12: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/12.jpg)
153 different students flagged a slideSOS call out! (flagging slides)BIO1140 – Introduction to Cell Biology
2015 TECHTEACH CONFERENCE - UIC
![Page 13: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/13.jpg)
2015 TECHTEACH CONFERENCE - UIC
Item Analysis – Difficulty Index
![Page 14: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/14.jpg)
2015 TECHTEACH CONFERENCE - UIC
Item Analysis – Discrimination Index
![Page 15: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/15.jpg)
0-10 10_19 20-29 30-39 40-49 50-59 60-69 70-79 80-89 90-99
Pro
po
rtio
n o
f C
las
s (
%)
0
5
10
15
20
Distribution of Learning Gains
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
2011: 30%
2014/16: 60%
Normalized LG = Post – Pre / 100 - pre
![Page 16: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/16.jpg)
What Are Two-Stage Exam?
2015 TECHTEACH CONFERENCE - UIC
Stage 1: IndividualMCQs, T/F, Match
Short Answer
Stage 2: GroupsMCQs, T/F, Match
Short Answer
Average: 65%
Average: 88%
![Page 17: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/17.jpg)
>00-
10
10_1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
Pro
po
rtio
n o
f C
las
s (
%)
0
10
20
30
40
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
Learning Gains in Groups? 2016
<Gain> : Pretest (Individual) vs Post-test (Individual)Avg: 50%
![Page 18: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/18.jpg)
>00-
10
10_1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
Pro
po
rtio
n o
f C
las
s (
%)
0
10
20
30
40
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
Learning Gains in Groups? 2016
<Gain> : Post-test (Individual) vs Post-test (Individual)Avg: 66%
![Page 19: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/19.jpg)
>00-
10
10_1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
Pro
po
rtio
n o
f C
las
s (
%)
0
10
20
30
40
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
Learning Gains in Groups? 2016
<Gain> : Pretest (Individual) vs Post-test (Group)Avg: 80%
![Page 20: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/20.jpg)
Final Grade
A+ A A- B+ B C+ C D+ D E F
Cla
ss
Pro
po
rtio
n (
%)
0
5
10
15
20
252016
Final Exam Scores
2015 TECHTEACH CONFERENCE - UIC
2015
2014
2013
20122011
![Page 21: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/21.jpg)
Summary - Learning Outcomes(2011 versus 2012-2015)
2015 TECHTEACH CONFERENCE - UIC
Student performance
on exams
Learning gains pre/post assessments(Smith et al., 2008)
Final grades
7-10%
~10%
50-90%
2012-2015 – eInstruction, LectureTools and Echo360 Lecture Captures
enabled classrooms
![Page 22: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/22.jpg)
Research Questions
2015 TECHTEACH CONFERENCE - UIC
1) Does the use of an Echo360-ALP feature influence the use of
another?
2) Is there a relationship between each individual metric and
exam outcomes?
3) Are there interactions between the metrics that lead to
differences in student performance and learning?
4) Is there a relationship between user profiles and exam
outcomes?
![Page 23: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/23.jpg)
2015 TECHTEACH CONFERENCE - UIC
Does the use of an Echo360-ALP feature
influence the use of another?
![Page 24: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/24.jpg)
R2 = 0.5449 (p<0.05)
2015 TECHTEACH CONFERENCE - UIC
Model generated by “stepwise
backward linear regression”.
• 6 of 256 interactions
were eliminated from
the model.
• Interactions between
the metrics are many
and complex.
Fit of the Linear Regression Model
Composite Exam Score (%)
Pre
dic
ted
Co
mp
osit
e
Exam
Sco
re (
%)
Are there interactions between the metrics that lead
to differences in exam outcomes?
![Page 25: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/25.jpg)
2015 TECHTEACH CONFERENCE - UIC
40%
45%
50%
55%
Low Usage High Usage
Pre
dic
ted
learn
ing
gain
sLinear regression model
was used to evaluate the
effect size of each metric
on learning gains.
Effect size of each metric
calculated by setting all
metrics at their median
values except the one
under study.
Presentation engagement
video views
Activity score
Video engagement
Attendance
Activity participation
Note taking
Presentation views
Low usage (1st quartile) High usage (3rd quartile)
All metrics at median values
(2nd quartile)
What are the effects of the metrics
on the learning gains?
![Page 26: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/26.jpg)
Using the Linear Regression Model to Evaluate the Effect Size
of Each Metric on Exam Outcomes
2015 TECHTEACH CONFERENCE - UIC
![Page 27: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/27.jpg)
Is there a relationship between user profiles and
exam outcomes?
2015 TECHTEACH CONFERENCE - UIC
Example
Engagement score =
![Page 28: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/28.jpg)
Weighting schemes which give the smallest interquartile range of engagement scores within each grade level
F
APP ATT NOT PVE VVE
0 1 0 0 0
D
APP ATT NOT PVE VVE
10 3 0 2 4
C
APP ATT NOT PVE VVE
8 0 0 0 1
B
APP ATT NOT PVE VVE
9 2 1 0 1
A
APP ATT NOT PVE VVE
9 2 0 2 0
2015 TECHTEACH CONFERENCE - UIC
![Page 29: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/29.jpg)
2015 TECHTEACH CONFERENCE - UIC
What does it all mean to teaching and learning?
![Page 30: Does Echo360‐ALP analytics predict the relationship ... · • Just -in Time Teaching F2F-class Echo360-ALP 65% 20% 15%. ... Confusion Questions posed Activity answers/ participation](https://reader034.fdocuments.us/reader034/viewer/2022042913/5f49de8b2e17496f4e270090/html5/thumbnails/30.jpg)
Acknowledgements
Teaching and Learning Support Services
Funding sources:
TLSS Pedagogical Grants
Vice-Rector Academics – Initiative Grants
Echo360 grant
uOttawa Blended Initiative
Ontario Consortium of Undergraduate Biology Educators
2015 TECHTEACH CONFERENCE - UIC