Learning Analytics Research Findings Update

“Dr.John”WhitmerDirector,Analy6csandResearchJISCRemotePresenta6on|2.22.2017

1.  LearningAnaly6csOverview&BbDataScience

2. MajorFindingsin20171.  Varia6oninLMS“Effec6veness”

2.  ToolUse3.  CourseCategories4.  StudentPercep6onsofDataDashboards(6me

permiYng)

3.  BlackboardAnaly6csSolu6onsPor\olio4.  Discussion

Learning Analytics Overview

Educational Technology Assessment Hierarchy

Doesitimpactstudentlearning?(LearningAnaly6cs)

Howmanypeopleuseit?(Adop6on)

Doesitwork?(SLAs)

What is Learning Analytics?

Learning and Knowledge Analytics Conference, 2011

“...measurement,collec6on,analysisandrepor6ngofdataaboutlearnersandtheircontexts,forpurposesofunderstandingandop2mizinglearningandtheenvironmentsinwhichitoccurs.”

Meta- questions driving our Learning Analytics research @ Blackboard

1.Howisstudent/facultyuseofBbpla\orms(e.g.Learn,Collab,etc.)relatedtostudentachievement?[orsa6sfac6on,orrisk,or…]

3.Whatdataelements,featuresets,andfunc6onalitycanwecreatetointegratethesefindingsintoBbproductstohelpfacultyimprovestudentachievement?

2.Dothesefindingsapplyequallytostudents‘atpromise’duetotheiracademicachievementorbackgroundcharacteris6cs?(e.g.race,class,familyeduca6on,geography)

Techniques

•  Simula2onifX,whatY?(“WiththisUltraLearningAnaly6cstriggerrule,howmanystudentswouldtripno6fied?”)

•  Hypothesistes2ng:inves6gateifaspecificrela6onshipistrue(“What’stherela6onshipbetween6mespentinacourseandstudentgrade”?)

•  Datamining:analyzeunderlyinglatentpakernsindata(“WhattypicalpakernsintoolusecharacterizesBBLearncourses?”)

KeyDataSources

•  LearnManagedHos6ng

•  LearnSaaS

•  CollaborateUltra

Main Big Data Sources & Techniques

Commitment to Privacy & Openness

•  AnalyzedatarecordsthatarenotonlyremovedofPII,butde-personalized(individual&ins6tu6onallevels)

•  Shareresultsandopendiscussionproceduresforanalysistoinformbroadereduca6onalcommunity

•  Respectterritorialjurisdic6onsandsafeharborprovisions

Major Findings in 2016 (and one from 2017)

Relationship Student Use Learn vs. Grade

Bb Study: Relationship Time in Learn & Grade

•  Distribution in Time Spent is highly skewed toward low access

•  Transforming data (log

transform) can produce normal curves for analysis

•  Of course, huge variation of

quality within that time spent (of course materials, of student activity)

Findings: Relationship Time in Learn & Grade •  Question: what is the

relationship between student use of Learn and their course grade?

•  Investigate at student-course level (one student, one course)

•  1.2M students, 34,519 courses, 788 institutions

•  Significant, but effect size < 1%

Finding: Tool Use & Grade TooluseandFinalGradedonothavealinearrela6onship;thereisadiminishingmarginaleffectoftooluseonFinalGrade

Interpreta6ons•  Studentsabsentfromcourseac6vityareat

greatestriskoflowachievement.•  Thefirst6meyouread/seeaPowerPoint

presenta6on,youlearnalot,butthesecond6meyouread/seeit,youlearnless.

•  GeYngfroma90%toa95%requiresmoreeffortthangeYngfroma60%toa65%.

Logtransforma2onshowsstrongertrend

But strong effect in some courses (n=7,648, 22%)

What makes some for a stronger or weaker relationship? Tools used? Course design? Quality of activity/effort?

Learn Tool Use vs. Grade

Investigation Grade by Specific Tools Used Ques6on:whatistherela6onshipbetweenuseofLearnandstudentgrade,basedonthetoolused?AnalysisSteps1.  Filterdataforcourseswithpoten6almeaningful

use(>60minaverage,enrollment>10<500,gradebookused)

2.  Iden6fymostfrequentlyusedtools3.  Separatetooluseintonouse&quar6les4.  Dividestudentsinto3groupsbycoursegrade

•  High(80+)•  Passing(60-79)•  Low/Failing(0-59)

Finding: MyGrades Ateverylevel,probabilityofhighergradeincreaseswithincreaseduse.Causal?Probablynot.Goodindicator?Absolutely.

Finding: Course contents Moreisnotalwaysbeker.Largejumpnonetosome;thennorela6onship

Finding: Assessments/Assignments Studentsabovemeanhavelowerlikelihoodofachievingahighgradethanstudentsbelowthemean

Implications

•  MovebeyondLMSuseasproxyforeffort(wheremoreisalwaysbeker),andgetatfiner-grainedlearningbehaviorsthataremoreuseful(e.g.studentswhoarestrugglingtounderstandmaterial,studentswhoarenotprepared).

•  Majormissingelementsfromresearch

– fine-grainedunderstandingofac6vityover6me(e.g.crammingvs.consistenthardworking)

– qualityofcoursematerialsandcoursedesign

Patterns in Course Design

Research Questions Ques2ons

1.  Aretheresystema6cwaysthatinstructorsuseLMStoolsintheircoursesthatspaninstructorsandins6tu6ons?

2.  Whatrecommenda6onscanbedrawnforfaculty,instruc6onaldesigners,andotheracademictechnologyleadersseekingtoincreasetheimpactofLMSuseattheirins6tu6on?

Methods

1.  Usesamefiltereddatasampleofstudent-coursedata

2.  Calculaterela6vestudent6mepertool(as%oftotalcourse6me),forcomparisonbetweencourses

3.  Clusterbypakernsinthebalanceof6mespentineachtool(unsupervisedmachine-learning;kmeansclusteranalysis)

4.  Adddataasrelevanttopakernsaboutenrollment,total6me,etc.

5.  Makeupcoolnamesforeachclusterandinterpretmeaning

Distribution of Courses by Type

Finding: Discussions with low/high avg use Comparecourseswithlowforumusetocourseswithforumuse>1hour/studentaverage

Summary & Future Directions for DS Research

Summary

•  Tremendousvaria6oninuseofLearn;mostuseskewedtowardlow/verylowuse.

•  Importanceof6mespentinLearnforlearningisalsotremendouslyvaried(“necessary”and“effec6ve”useofLearn)

•  Cri6caltoaccountforthisvaria6ontounderstandpoten6alimportanceofLearnac6vity

FutureDirec2ons

•  Analyzequalityofac2vityingreaterdepth(e.g.contentofassignments,wordsinforumposts)togetinsightsintoqualityofinterac6ons

•  Conduct6me-seriesanalysis(quan6ta6vemethods,designalsoneeded);whensomeoneaccessesismoreimportantthaniftheydo.

•  Createproxies/derivedvaluesforbehavior(aboveaverage,ataverage,etc.)bytool

3. Blackboard Analytics Portfolio

Blackboard Analytics – Product Naming

Blackboard Analytics Data warehouse products

Blackboard Analytics Suite of analytics products

BlackboardAnaly.cs

Product portfolio

Blackboard Intelligence •  Analy6csforLearn–LMSdata•  StudentManagement–SISdata•  Finance,HR,Advancement–ERPdata

Blackboard Predict •  Predic6veanaly6csandearlyalertsforreten6on•  Providesdataforfacultyandadvisorsaboutat-riskstudents•  FormerlyBlueCanary

X-Ray Learning Analytics •  Classroomengagementdataforfaculty•  Ac6vityaggregatedinto30+visualiza6ons•  CurrentlyavailableforMoodlerooms&Self-HostedMoodlersonly

Pastview Currentview Futureview

Pastview Currentview Futureview

Pastview Currentview Futureview

Blackboard Analytics Solution Portfolio

X-RayLearningAnaly6cs

BlackboardAnaly6cs

BlackboardPredict

Analy6csforLearn

StudentManagement Finance HR Advancement

BlackboardIntelligence

Blackboard Analytics – Our Approach & Philosophy

Productsthatprovideinsightintotheteachingandlearningprocess

OurPhilosophy:Datacomplements

humandecision-making

Corecompetency:Learningso{wareandacademicdata

Ateamofexpertsintheanaly6csfield

Discussion Thank you!

John Whitmer, Ed.D. john.whitmer@blackboard.com@johncwhitmerwww.johnwhitmer.info/research