Blackboard Learning Analytics Research Update
-
Upload
john-whitmer-edd -
Category
Data & Analytics
-
view
425 -
download
2
Transcript of Blackboard Learning Analytics Research Update
![Page 1: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/1.jpg)
Learning Analytics Research Findings Update
“Dr.John”WhitmerDirector,Analy6csandResearchJISCRemotePresenta6on|2.22.2017
![Page 2: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/2.jpg)
1. LearningAnaly6csOverview&BbDataScience
2. MajorFindingsin20171. Varia6oninLMS“Effec6veness”
2. ToolUse3. CourseCategories4. StudentPercep6onsofDataDashboards(6me
permiYng)
3. BlackboardAnaly6csSolu6onsPor\olio4. Discussion
![Page 3: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/3.jpg)
Learning Analytics Overview
![Page 4: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/4.jpg)
Educational Technology Assessment Hierarchy
Doesitimpactstudentlearning?(LearningAnaly6cs)
Howmanypeopleuseit?(Adop6on)
Doesitwork?(SLAs)
![Page 5: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/5.jpg)
What is Learning Analytics?
Learning and Knowledge Analytics Conference, 2011
“...measurement,collec6on,analysisandrepor6ngofdataaboutlearnersandtheircontexts,forpurposesofunderstandingandop2mizinglearningandtheenvironmentsinwhichitoccurs.”
![Page 6: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/6.jpg)
Meta- questions driving our Learning Analytics research @ Blackboard
1.Howisstudent/facultyuseofBbpla\orms(e.g.Learn,Collab,etc.)relatedtostudentachievement?[orsa6sfac6on,orrisk,or…]
3.Whatdataelements,featuresets,andfunc6onalitycanwecreatetointegratethesefindingsintoBbproductstohelpfacultyimprovestudentachievement?
2.Dothesefindingsapplyequallytostudents‘atpromise’duetotheiracademicachievementorbackgroundcharacteris6cs?(e.g.race,class,familyeduca6on,geography)
![Page 7: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/7.jpg)
Techniques
• Simula2onifX,whatY?(“WiththisUltraLearningAnaly6cstriggerrule,howmanystudentswouldtripno6fied?”)
• Hypothesistes2ng:inves6gateifaspecificrela6onshipistrue(“What’stherela6onshipbetween6mespentinacourseandstudentgrade”?)
• Datamining:analyzeunderlyinglatentpakernsindata(“WhattypicalpakernsintoolusecharacterizesBBLearncourses?”)
KeyDataSources
• LearnManagedHos6ng
• LearnSaaS
• CollaborateUltra
Main Big Data Sources & Techniques
![Page 8: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/8.jpg)
Commitment to Privacy & Openness
• AnalyzedatarecordsthatarenotonlyremovedofPII,butde-personalized(individual&ins6tu6onallevels)
• Shareresultsandopendiscussionproceduresforanalysistoinformbroadereduca6onalcommunity
• Respectterritorialjurisdic6onsandsafeharborprovisions
![Page 9: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/9.jpg)
Major Findings in 2016 (and one from 2017)
![Page 10: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/10.jpg)
Relationship Student Use Learn vs. Grade
![Page 11: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/11.jpg)
Bb Study: Relationship Time in Learn & Grade
• Distribution in Time Spent is highly skewed toward low access
• Transforming data (log
transform) can produce normal curves for analysis
• Of course, huge variation of
quality within that time spent (of course materials, of student activity)
![Page 12: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/12.jpg)
Findings: Relationship Time in Learn & Grade • Question: what is the
relationship between student use of Learn and their course grade?
• Investigate at student-course level (one student, one course)
• 1.2M students, 34,519 courses, 788 institutions
• Significant, but effect size < 1%
![Page 13: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/13.jpg)
Finding: Tool Use & Grade TooluseandFinalGradedonothavealinearrela6onship;thereisadiminishingmarginaleffectoftooluseonFinalGrade
Interpreta6ons• Studentsabsentfromcourseac6vityareat
greatestriskoflowachievement.• Thefirst6meyouread/seeaPowerPoint
presenta6on,youlearnalot,butthesecond6meyouread/seeit,youlearnless.
• GeYngfroma90%toa95%requiresmoreeffortthangeYngfroma60%toa65%.
Logtransforma2onshowsstrongertrend
![Page 14: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/14.jpg)
But strong effect in some courses (n=7,648, 22%)
![Page 15: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/15.jpg)
What makes some for a stronger or weaker relationship? Tools used? Course design? Quality of activity/effort?
![Page 16: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/16.jpg)
Learn Tool Use vs. Grade
![Page 17: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/17.jpg)
Investigation Grade by Specific Tools Used Ques6on:whatistherela6onshipbetweenuseofLearnandstudentgrade,basedonthetoolused?AnalysisSteps1. Filterdataforcourseswithpoten6almeaningful
use(>60minaverage,enrollment>10<500,gradebookused)
2. Iden6fymostfrequentlyusedtools3. Separatetooluseintonouse&quar6les4. Dividestudentsinto3groupsbycoursegrade
• High(80+)• Passing(60-79)• Low/Failing(0-59)
![Page 18: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/18.jpg)
Finding: MyGrades Ateverylevel,probabilityofhighergradeincreaseswithincreaseduse.Causal?Probablynot.Goodindicator?Absolutely.
![Page 19: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/19.jpg)
Finding: Course contents Moreisnotalwaysbeker.Largejumpnonetosome;thennorela6onship
![Page 20: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/20.jpg)
Finding: Assessments/Assignments Studentsabovemeanhavelowerlikelihoodofachievingahighgradethanstudentsbelowthemean
![Page 21: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/21.jpg)
Implications
• MovebeyondLMSuseasproxyforeffort(wheremoreisalwaysbeker),andgetatfiner-grainedlearningbehaviorsthataremoreuseful(e.g.studentswhoarestrugglingtounderstandmaterial,studentswhoarenotprepared).
• Majormissingelementsfromresearch
– fine-grainedunderstandingofac6vityover6me(e.g.crammingvs.consistenthardworking)
– qualityofcoursematerialsandcoursedesign
![Page 22: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/22.jpg)
Patterns in Course Design
![Page 23: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/23.jpg)
Research Questions Ques2ons
1. Aretheresystema6cwaysthatinstructorsuseLMStoolsintheircoursesthatspaninstructorsandins6tu6ons?
2. Whatrecommenda6onscanbedrawnforfaculty,instruc6onaldesigners,andotheracademictechnologyleadersseekingtoincreasetheimpactofLMSuseattheirins6tu6on?
Methods
1. Usesamefiltereddatasampleofstudent-coursedata
2. Calculaterela6vestudent6mepertool(as%oftotalcourse6me),forcomparisonbetweencourses
3. Clusterbypakernsinthebalanceof6mespentineachtool(unsupervisedmachine-learning;kmeansclusteranalysis)
4. Adddataasrelevanttopakernsaboutenrollment,total6me,etc.
5. Makeupcoolnamesforeachclusterandinterpretmeaning
![Page 24: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/24.jpg)
Distribution of Courses by Type
![Page 25: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/25.jpg)
![Page 26: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/26.jpg)
![Page 27: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/27.jpg)
![Page 28: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/28.jpg)
![Page 29: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/29.jpg)
![Page 30: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/30.jpg)
![Page 31: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/31.jpg)
Finding: Discussions with low/high avg use Comparecourseswithlowforumusetocourseswithforumuse>1hour/studentaverage
![Page 32: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/32.jpg)
Summary & Future Directions for DS Research
Summary
• Tremendousvaria6oninuseofLearn;mostuseskewedtowardlow/verylowuse.
• Importanceof6mespentinLearnforlearningisalsotremendouslyvaried(“necessary”and“effec6ve”useofLearn)
• Cri6caltoaccountforthisvaria6ontounderstandpoten6alimportanceofLearnac6vity
FutureDirec2ons
• Analyzequalityofac2vityingreaterdepth(e.g.contentofassignments,wordsinforumposts)togetinsightsintoqualityofinterac6ons
• Conduct6me-seriesanalysis(quan6ta6vemethods,designalsoneeded);whensomeoneaccessesismoreimportantthaniftheydo.
• Createproxies/derivedvaluesforbehavior(aboveaverage,ataverage,etc.)bytool
![Page 33: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/33.jpg)
3. Blackboard Analytics Portfolio
![Page 34: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/34.jpg)
Blackboard Analytics – Product Naming
Blackboard Analytics Data warehouse products
Blackboard Analytics Suite of analytics products
![Page 35: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/35.jpg)
BlackboardAnaly.cs
Product portfolio
Blackboard Intelligence • Analy6csforLearn–LMSdata• StudentManagement–SISdata• Finance,HR,Advancement–ERPdata
Blackboard Predict • Predic6veanaly6csandearlyalertsforreten6on• Providesdataforfacultyandadvisorsaboutat-riskstudents• FormerlyBlueCanary
X-Ray Learning Analytics • Classroomengagementdataforfaculty• Ac6vityaggregatedinto30+visualiza6ons• CurrentlyavailableforMoodlerooms&Self-HostedMoodlersonly
Pastview Currentview Futureview
Pastview Currentview Futureview
Pastview Currentview Futureview
![Page 36: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/36.jpg)
Blackboard Analytics Solution Portfolio
X-RayLearningAnaly6cs
BlackboardAnaly6cs
BlackboardPredict
Analy6csforLearn
StudentManagement Finance HR Advancement
BlackboardIntelligence
![Page 37: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/37.jpg)
Blackboard Analytics – Our Approach & Philosophy
Productsthatprovideinsightintotheteachingandlearningprocess
OurPhilosophy:Datacomplements
humandecision-making
Corecompetency:Learningso{wareandacademicdata
Ateamofexpertsintheanaly6csfield
![Page 38: Blackboard Learning Analytics Research Update](https://reader033.fdocuments.us/reader033/viewer/2022051502/58b884541a28ab44078b6d2f/html5/thumbnails/38.jpg)
Discussion Thank you!
John Whitmer, Ed.D. [email protected]@johncwhitmerwww.johnwhitmer.info/research