Post on 15-Jul-2020
#THETA2017
This work is licensed under a Creative Commons Attribution 4.0 International License
Does Echo360‐ALP analytics predict the relationship between
student behaviours and learning?
Colin Montpetit, Sonya Sabourin
Leveraging the Echo360-ALP to Transform A Large Enrollment
Lecture Classroom into an Active Learning Zone
Colin Montpetit, Ph.D., Assistant Professor
Sonya Sabourin, B.Sc. Honour’s student
Department of Biology, University of Ottawa
2015 TECHTEACH CONFERENCE - UIC
Does Echo360-ALP analytics predict the relationship
between student behaviours and learning?
2015 TECHTEACH CONFERENCE - UIC
Modules (self-paced)• Targeted readings
• Interactive act./Videos
• Reflective act.
• Module quiz OnlineBlackBoard LMSInitial exposure
Summative• Two-Stage Collaborative Exams
(2 midterms + 1 final)
Assessments
Labs• Experiments
• Worksheets, reports
• Quizzes
F2F-Labs
Defined Learning Outcomes, 5E Learning Cycle
Class• Clicker case studies
• Peer-Instruction
• Just-in-Time Teaching
F2F-class
Echo360-ALP
65% 20%
15%
Echo360- Active Learning Platform
2015 TECHTEACH CONFERENCE - UIC
2015 TECHTEACH CONFERENCE - UIC
What is the class usage profile?
2015 TECHTEACH CONFERENCE - UIC
Engagement data provides insight via dashboards
Confusion
Questions
posed
Activity
answers/
participation
Notes
Attendance
Presentation
views
Learning Analytics Database
Lecture capture
2015 TECHTEACH CONFERENCE - UIC
Attendance, Participation, Activity Score
2015 TECHTEACH CONFERENCE - UIC
Note Taking
2015 TECHTEACH CONFERENCE - UIC
Presentation Views
2015 TECHTEACH CONFERENCE - UIC
Video Views
Caroline Petit-Turcotte Winter 2015
447 different students asked at least 1 question
2015 TECHTEACH CONFERENCE - UIC
Questions to the professorBIO1140 – Introduction to Cell Biology
153 different students flagged a slideSOS call out! (flagging slides)BIO1140 – Introduction to Cell Biology
2015 TECHTEACH CONFERENCE - UIC
2015 TECHTEACH CONFERENCE - UIC
Item Analysis – Difficulty Index
2015 TECHTEACH CONFERENCE - UIC
Item Analysis – Discrimination Index
0-10 10_19 20-29 30-39 40-49 50-59 60-69 70-79 80-89 90-99
Pro
po
rtio
n o
f C
las
s (
%)
0
5
10
15
20
Distribution of Learning Gains
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
2011: 30%
2014/16: 60%
Normalized LG = Post – Pre / 100 - pre
What Are Two-Stage Exam?
2015 TECHTEACH CONFERENCE - UIC
Stage 1: IndividualMCQs, T/F, Match
Short Answer
Stage 2: GroupsMCQs, T/F, Match
Short Answer
Average: 65%
Average: 88%
>00-
10
10_1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
Pro
po
rtio
n o
f C
las
s (
%)
0
10
20
30
40
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
Learning Gains in Groups? 2016
<Gain> : Pretest (Individual) vs Post-test (Individual)Avg: 50%
>00-
10
10_1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
Pro
po
rtio
n o
f C
las
s (
%)
0
10
20
30
40
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
Learning Gains in Groups? 2016
<Gain> : Post-test (Individual) vs Post-test (Individual)Avg: 66%
>00-
10
10_1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
Pro
po
rtio
n o
f C
las
s (
%)
0
10
20
30
40
2015 TECHTEACH CONFERENCE - UIC
<gains> (ranges)
Learning Gains in Groups? 2016
<Gain> : Pretest (Individual) vs Post-test (Group)Avg: 80%
Final Grade
A+ A A- B+ B C+ C D+ D E F
Cla
ss
Pro
po
rtio
n (
%)
0
5
10
15
20
252016
Final Exam Scores
2015 TECHTEACH CONFERENCE - UIC
2015
2014
2013
20122011
Summary - Learning Outcomes(2011 versus 2012-2015)
2015 TECHTEACH CONFERENCE - UIC
Student performance
on exams
Learning gains pre/post assessments(Smith et al., 2008)
Final grades
7-10%
~10%
50-90%
2012-2015 – eInstruction, LectureTools and Echo360 Lecture Captures
enabled classrooms
Research Questions
2015 TECHTEACH CONFERENCE - UIC
1) Does the use of an Echo360-ALP feature influence the use of
another?
2) Is there a relationship between each individual metric and
exam outcomes?
3) Are there interactions between the metrics that lead to
differences in student performance and learning?
4) Is there a relationship between user profiles and exam
outcomes?
2015 TECHTEACH CONFERENCE - UIC
Does the use of an Echo360-ALP feature
influence the use of another?
R2 = 0.5449 (p<0.05)
2015 TECHTEACH CONFERENCE - UIC
Model generated by “stepwise
backward linear regression”.
• 6 of 256 interactions
were eliminated from
the model.
• Interactions between
the metrics are many
and complex.
Fit of the Linear Regression Model
Composite Exam Score (%)
Pre
dic
ted
Co
mp
osit
e
Exam
Sco
re (
%)
Are there interactions between the metrics that lead
to differences in exam outcomes?
2015 TECHTEACH CONFERENCE - UIC
40%
45%
50%
55%
Low Usage High Usage
Pre
dic
ted
learn
ing
gain
sLinear regression model
was used to evaluate the
effect size of each metric
on learning gains.
Effect size of each metric
calculated by setting all
metrics at their median
values except the one
under study.
Presentation engagement
video views
Activity score
Video engagement
Attendance
Activity participation
Note taking
Presentation views
Low usage (1st quartile) High usage (3rd quartile)
All metrics at median values
(2nd quartile)
What are the effects of the metrics
on the learning gains?
Using the Linear Regression Model to Evaluate the Effect Size
of Each Metric on Exam Outcomes
2015 TECHTEACH CONFERENCE - UIC
Is there a relationship between user profiles and
exam outcomes?
2015 TECHTEACH CONFERENCE - UIC
Example
Engagement score =
Weighting schemes which give the smallest interquartile range of engagement scores within each grade level
F
APP ATT NOT PVE VVE
0 1 0 0 0
D
APP ATT NOT PVE VVE
10 3 0 2 4
C
APP ATT NOT PVE VVE
8 0 0 0 1
B
APP ATT NOT PVE VVE
9 2 1 0 1
A
APP ATT NOT PVE VVE
9 2 0 2 0
2015 TECHTEACH CONFERENCE - UIC
2015 TECHTEACH CONFERENCE - UIC
What does it all mean to teaching and learning?
Acknowledgements
Teaching and Learning Support Services
Funding sources:
TLSS Pedagogical Grants
Vice-Rector Academics – Initiative Grants
Echo360 grant
uOttawa Blended Initiative
Ontario Consortium of Undergraduate Biology Educators
2015 TECHTEACH CONFERENCE - UIC