1

OpenLearningAnalytics:anintegrated&modularizedplatformProposaltodesign,implementandevaluateanopenplatformtointegrateheterogeneouslearninganalyticstechniques

GeorgeSiemens&DraganGasevicAthabascaUniversity,CanadaCarolineHaythornthwaite&ShaneDawsonUniversityofBritishColumbia,CanadaSimonBuckinghamShum&RebeccaFergusonOpenUniversity,UnitedKingdomErikDuval&KatrienVerbertKatholiekeUniversiteitLeuven,BelgiumRyanS.J.d.BakerWorcesterPolytechnicInstitute,UnitedStates

July28,2011

2

ProjectOverviewTheSocietyforLearningAnalyticsResearch(SoLAR–www.solaresearch.org)isaninter-disciplinarynetworkofleadinginternationalresearcherswhoareexploringtheroleandimpactofanalyticsonteaching,learning,traininganddevelopment.Ourmissionasanorganizationisto:a)pursuethedevelopmentofresearchopportunitiesinlearninganalyticsandeducationaldatamining,b)increasetheprofileoflearninganalyticsineducationalcontexts,andc)serveasanadvocateforlearninganalyticstopolicymakersSignificantpotentialexistsforanalyticstoguidelearners,educators,administrators,andfundersinmakinglearning-relateddecisions.Learninganalyticsrepresenttheapplicationof“bigdata”andanalyticsineducation1.Thisproposalexpressestheimportanceofaplannedandintegratedapproachtodevelopinginsightfulandeasy-to-uselearninganalyticstools.Threecriticalbeliefsunderpinourproposal:

1. Opennessofprocess,algorithms,andtechnologiesisimportantforinnovationandmeetingthevaryingcontextsofimplementation.

2. Modularizedintegration:coreanalytictools(orengines)include:adaptation,learning,interventions,anddashboards.Thelearninganalyticsplatformisanopenarchitecture,enablingresearcherstodeveloptheirowntoolsandmethodstobeintegratedwiththeplatform.

3. Reductionofinevitablefragmentationbyprovidinganintegrated,expandable,open

technologythatresearchersandcontentproducerscanuseindatamining,analytics,andadaptivecontentdevelopment.Educators,learners,andadministratorsbenefitfrommodularizedfunctionality:withcustomizableandextendablecoreanalytics,intervention,andcontenttoolstomeetneedsoflearnersandeducators(particularlyinidentifyingat-riskstudents).Administratorsbenefitfromintegratedtoolsthattracklearning-relatedactivityandtheninfluenceresourceallocationacrossmultipletoolsandspacesoflearning.Learnerswillbenefitfromhavingtimelyandrelevantfeedbackontheirperformance,aswellascontent,activity,andsocialnetworkrecommendationstoimproveandguidetheirlearning.

1JohnP.Campbell,PeterB.DeBlois,andDianaG.Oblinger(2007),EDUCAUSEReview,vol.42,no.4(July/August2007),pp.53–54

3

AnalyticsinEducation.............................................................................................................4BenefitsofAnalytics................................................................................................................5TheChallengeandtheOpportunity.........................................................................................6Envisioninguserscenarioswewillsupport..............................................................................7

Learneruserscenario.....................................................................................................................................7Educatoruserscenario...............................................................................................................................10Administratoruserscenario....................................................................................................................11Researcheruserscenario:Analyticsonlearninganalyticsuse................................................12

Technicalstrategy..................................................................................................................13AnalyticsEngine............................................................................................................................................13LearningAdaptationandPersonalizationEngine..........................................................................14TheInterventionEngine............................................................................................................................14TheDashboard...............................................................................................................................................15ContextandDepthofAnalysis.................................................................................................................16Datamart...........................................................................................................................................................17

ModularizedDevelopment.....................................................................................................18InvestigatorBiographies.........................................................................................................18Considerations.......................................................................................................................19GettingInvolved.....................................................................................................................19ConcludingThoughts..............................................................................................................19

4

AnalyticsinEducationDataandanalyticshavecapturedtheattentionofbusinessleadersandtechnologycompanies.Thebroadpromiseofanalyticsisthatnewinsightscanbegainedfromin-depthanalysisofthedatatrailsleftbyindividualsintheirinteractionswithothers,withinformation,withtechnology,andwithorganizations.Therapiddevelopmentof“bigdata”methodsandtoolscoincideswithnewmanagementandmeasurementprocessesincorporations.Theterm“businessintelligence”isusedtodescribethisintersectionofdataandinsight.Whenappliedtotheeducationsector,analyticsfallintotwobroadsectors(Table1):learningandacademic.Learninganalytics(LA)isthemeasurement,collection,analysisandreportingofdataaboutlearnersandtheircontexts,forpurposesofunderstandingandoptimizinglearningandtheenvironmentsinwhichitoccurs2.Learninganalyticsarelargelyconcernedwithimprovinglearnersuccess.Academicanalyticsistheimprovementoforganizationalprocesses,workflows,resourceallocation,andinstitutionalmeasurementthroughtheuseoflearner,academic,andinstitutionaldata.Academicanalytics,akintobusinessanalytics,areconcernedwithimprovingorganizationaleffectiveness3.

Typeofanalytics

Levelorobjectofanalysis WhoBenefits?

Learninganalytics

Personallevel:analyticsonpersonalperformanceinrelationtolearninggoals,learningresources,andstudyhabitsofotherclassmates.Course-level:socialnetworks,conceptualdevelopment,discourseanalysis,“intelligentcurriculum”

Learners,educatorsandteachingstaff

Departmental:predictivemodeling,patternsofsuccess/failure

Learners,educators

AcademicAnalytics

Institutional:learnerprofiles,performanceofacademics,knowledgeflow,resourceallocation

Administrators,funders,marketing

Regional(state/provincial):comparisonsbetweensystems,Qualityandstandards

Funders,administrators

National&International Nationalgovernments,UNESCO,OECD,LeagueTables

Table1:LearningandAcademicAnalytics 21stInternationalConferenceonLearningAnalytics&Knowledge3Siemens,G.,Long,P.(2011).PenetratingtheFog:Analyticsinlearningandeducation.EDUCAUSEReview,vol.46,no.4(July/August2011)

5

InspiteofthepotentialofLA,itisimportanttoemphasizethatanalyticsarenottheendgoal.WeenvisionLAasameanstoprovidestakeholders(learners,educators,administrators,andfunders)withbetterinformationanddeepinsightintothefactorswithinthelearningprocessthatcontributetolearnersuccess.Analyticsservetoguidedecisionmakingabouteducationalreformandlearner-levelinterventionforat-riskstudents.ThebroadgoalsofLAaretoimprovecompletionrates,providedecisionsmakerswithneededinformation,andassistlearnersindevelopingincreasedresponsibilityfortheirlearningactivity.Aslearningandinteractionareincreasinglydistributedacrossdifferenttoolsandenvironments4,analyticscanservepulltogetherandanalyzetheimpactofthosevariousinteractionsandidentities.Additionally,theuseofLAwillhelpresearchersgroundtheirresearchinlearningonrealdata,enablingwhatiscurrentlyacollectionofbeliefsandopinionstobecomemorescientificandempiricallygrounded.

BenefitsofAnalyticsLearninganalyticsholdthepromiseofimprovinglearningefficiencyandeffectivenessinprimary,secondary,andpost-secondaryeducation.LAaredirectedtowardprovidingeducators,learners,andchairs/deanswithactionableinsighttoclassroomandcourselevelactivities.ThebenefitsofLAinclude:

- Reduceattritionthroughearlydetectionofat-riskstudentsandgeneratingalertsforlearnersandeducators.

- Personalizeandadaptlearningprocessandcontent,ensuringthateachlearnerreceivesresourcesandteachingthatreflectstheircurrentknowledgestate.

- Extendandenhancelearnerachievement,motivation,andconfidencebyprovidinglearnerswithtimelyinformationabouttheirperformanceandthatoftheirpeers,aswellasprovidingsuggestionsonactivitiesandcontentthataddressidentifiedknowledgegaps.

- Improveteachertimeandeffortbyprovidinginformationonwhichstudentsneedadditionalhelp,whichstudentsarecandidatesformentoringothers,andwhichteachingpracticesaremakingthebiggestimpact.

- Higherqualitylearningdesignandimprovedcurriculumdevelopmentprocessesthroughtheutilizationofdatageneratedduringreal-timeinstructionandlearningactivities.

- Interactivevisualizationsofcomplexinformationwillgivelearnersandeducatorstheabilityto“zoomin”or“zoomout”ondatasets,dependingontheneedsofaspecificteachingorlearningcontext.

- Morerapidachievementoflearninggoalsbygivinglearnersaccesstotoolsthathelpthemtoevaluatetheirprogressanddeterminewhichactivitiesareproducingthebestresults.

4Siemens,G.(2007)“Connectivism:CreatingaLearningEcologyinDistributedEnvironment,”inDidacticsofMicrolearning:Concepts,discourses,andexamples,inT.Hug,(ed.),WaxmannVerlag,NewYork,pp.53-68

6

Academicanalyticstargethigher-levelanalysisandtargettheneedsofsenioradministrators,policymakers,governmentofficials,andprospectivefunders.Thebenefitsofacademicanalyticsinclude:

- Improvedknowledgeflowacrosstheorganization- Benchmarkingandcomparisonswithotheruniversities- Learnersuccessinrelationtootherschoolsystems- Reducedcosts- Betterdecisionmakingthroughincreasedinsightintofactorsimpactinglearning

achievement- Betterresourceallocationasaconsequenceofaccurate,up-to-dateinformationof

activitieswithintheorganization

TheChallengeandtheOpportunityInspiteoftheattentiongiventoanalyticsasaconceptandthedevelopmentofmethods(statisticalanalysis,“bigdata”),educatorsdonothaveaccesstointegratedtoolsetsthatallowforvariedandcomplexevaluationsoflearnerperformanceandcomparisonsbetweendifferentsetsoflearners.Learnersalsolackneededinformationabouttheirperformance.Numeroussocialmediatoolshaveadvancedsearch,ratings,andrelatedanalyticsthatcanbemotivatingforcontinuedinvolvementinacommunityoraroundaspecifictopic.LAcansimilarlycontributetolearnermotivationbyprovidingdetailedinformationaboutherperformance.Inresponsetothisweakness,weproposethedevelopmentofanintegratedandextensibletoolsetthatcanassistacademicsandorganizationsevaluatelearneractivity,determineneededinterventions,andimproveadvancementoflearningopportunities.Thistoolsetwillbelearnerfacingaswell,permittingindividualstotracktheirownprogressandtakeadvantageofanalyticsinimprovingtheirlearningactivities.Tobeeffectiveandreduceinaccurateassessment,analyticsneedtobebroad-based,multi-sourced,contextualandintegrated.Theanalysisofanyinteractiondatamustbeinalignmentwiththeintendedteachingpracticeandcurriculumgoalsandoutcomes.Thisprovidesformoreaccurateanalysesandsubsequentlearnerandinstructorrecommendationsaswellasidentifyingparticularlearningandteachingactivitiesthatpromoteindividualstudentsuccess.Anunderstandingofthelearningandteachingcontextforthecourseofferingswillalsoassistinaddressingtheneedforinstitutionalcomparisonsandbenchmarking.Thisfundingproposaladdressestheneedforintegratedtoolsetsthroughthedevelopmentoffourspecifictoolsandresources,whichwillbeelaboratedonlaterinthisproposal:

1. Learninganalyticsengine2. Adaptivecontentengine3. Interventionengine:recommendations,automatedsupport4. Dashboard,reporting,andvisualizationtools

7

EnvisioninguserscenarioswewillsupportInthefollowingexamplesweenvisionthekindsofuserscenariosthattheproposedplatformwillsupport.Wefocusontoolsforfourkeystakeholdergroups:

1. Learners2. Educators3. Administrators4. Researchersanddataanalysts

Weprovidemockupdashboardsofvisualanalyticstoserveasindicatorsofthekindsofend-userexperienceweaimtodesign.Akeypartoftheproject’sworkis,ofcourse,todevelopmoredetailedanalysesofuseractivitiesinconsultationwiththewidercommunitytoelicitthefeaturestheywouldliketosee.

LearneruserscenarioKrisisafirstyearpoliticsandenvironmentstudentatXYZUniversity.Likeallstudents,hehasaccesstoadashboardofanalyticswhichprovidehimwithfeedbackonhisprogress.Krishassetthedashboardtosendhimaweeklysummaryofwhat’schanged,sinceitisaggregatinghisactivityonquiteafewsitesprovidedbyhiscourse,aswellassomeofthesocialwebtoolshehasgivenaccessto,sincehedoesalotofhislearningusingexternaltoolssuchasGoogleScholar,severalonlinelibraries,severalopeneducationsites,Wikipedia,FacebookandSocialLearn.Thedashboard,image1,providesbasicstatisticsonhisattendanceatlecturesandonlineactivities,participationratesinforums,passratesontheonlinetests,andhismarksonformalwrittenassignmentsandexams.However,whathefindsmostusefulforreflectionarethevisual‘mirrors’thatthesystempresentstohim,plussuggestionsofwaysinwhichhemightbecomeamoreeffective,strategiclearner.Heknowsthatwhenitcomestoapplyingforenvironmentalpolicyjobs,employerswillbelookingnotonlyathisexamscores,butathimasaperson,andhewantstobuildanevidencebaseonthisfronttoo:hiscreativity,criticalthinking,collaborationandpresentationskills,emotionalintelligence,agilityincopingwithunexpectedevents,andsoforth.5Krisalsobenefitsfromreflectingonhisachievementscomparedtoothersinhisgrouporhispeers.PossiblelearningpathsarealsorecommendedbasedonpreviousactivitiesofhispeersorlearnerswithaprofilesimilartoKris’.IfKrisbeginstofallbehind,thesystemcanalsorecommendsmallstrategiesonhowtocatchupwiththerestoftheclass.Hisdashboardthereforeusesarangeofanalyticstechnologiestunedtosuchskillsanddispositions,asillustratedandnumberedinthemockupbelow.1. ThisisanexampleDiscourseAnalytic,providingfeedbackfromcomputational

linguisticstoolsthatcandifferentiateverysimpletextsfromthoseusingthekindsofargumentativeformsexemplifiedbymoreadvancedstudents.Krisneedstolearnto

5ThebasisforthisenvisionedLearnerDashboardcomesfromtechniquesreviewedin:BuckinghamShum,S.andFerguson,R.(2011).SocialLearningAnalytics.Availableas:TechnicalReportKMI-11-01,KnowledgeMediaInstitute,TheOpenUniversity,UK.http://kmi.open.ac.uk/publications/pdf/kmi-11-01.pdf

8

makethisthinkingvisibleinhiswriting,inordertodemonstratethatheismasteringscholarlygenresofdiscourse.

2. ThisshowsKrishisprofilefromELLI,theEffectiveLifelongLearningInventory,awebsurveytoolinwhichlearnersreflectontheirskillsanddispositionsaslearners.6ThespiderdiagramshowstheextenttowhichtheyperceivethemselvesondimensionssuchasCriticalCuriosity,LearningRelationships,CreativityandResilience.Underthisisananalyticwhichinterpretspatternsintheonlinebehaviorwhichseemtoevidenceactivityagainstdifferentdimensions.7

3. ThisshowsKriswherehesitsinrelationtothesocialnetworkofco-learnersonthecourse,basedonanalgorithmwhichweightsdifferentlevelsofinteractionwithpeersviadifferentmodalitiesandmedia.Hecanjumpbackintimetoseehowthisnetworkhaschangedinshapeanddensity.Thenetworkmapprovidesawaytolinktotheonlinerecordofinteractionshehashadwithagivenperson.

4. ThisanalyticshowsanembeddedknowledgemapwhichexpressesmeaningfulrelationshipsbetweenideasonwhichKrisisworking.

5. Thefifthanalyticfocusesontheemotionaldimensionoflearningwhichisincreasinglyrecognizedtobecentraltoeffectivelearning.ThismoodchartreflectsbacktoKrisatimelineofblogpostsinhislearningjournalinwhichheindicatedhowhewasfeelingabouthiswork,andwhy.

6DeakinCrick,R.,Broadfoot,P.andClaxton,G.(2004).DevelopinganEffectiveLifelongLearningInventory:TheELLIProject.AssessmentinEducation,11,(3),pp.247-2727AdaptedfromEnquiryBlogger:http://learningemergence.net/tools/enquiryblogger

9

Image1:Learnerdashboard,variousanalyticsmethods

10

EducatoruserscenarioJennyisrunningthecourseKrisistakingandhasadifferentdashboard,designedforeducators.Herpriorityistotrackandgaininsightintotherangeofdifferentfactorsknowntoimpactsustainedlearnerengagementinthefirstyearofthischallengingcourse.First,thevisualizations(image2,leftcolumn)showtheresultsof“atrisk”analytics,asdefinedbyarangeofalgorithmsthatmatchonlinebehaviortopredictivemodelsbasedonpastcohorts.8Thesealgorithmsareacombinationofwhatshehasfoundtobebestsuitedtothecharacteristicsofcoursesrunatheruniversity,thestudentdemographicandlearningstyles,andthemixoftechnologiestheyuse.Second,Jenny’sdashboardalsoshows(ontheright)aggregatesofallthepersonallearneranalyticsdescribedinKris’sdashboard,providingadeeperlevelofinsightintohowstudentsareself-reportingtheirsenseofprogressandevidencingitthroughtheironlineengagement.Onothertabs,shecanseehowagivenstudenthasmovedwithinthesocialnetworkgeneratedfromcourseactivity.Thisisofparticularinterestsincethiscourseteachestheimportanceofsocialcapitalandsocialmediainnextgenerationenvironmentalpolicystrategy.Third,Jennyisnotlockedintoasingleplatformfortheseanalytics.Ashintedabove,asshegetsfamiliarwiththenewcourse,andgetstoknowherstudents,shecanconfiguretheanalyticsthatbestilluminateproblemsandprogressofdifferentsorts.Thisispossiblebecausetheplatformisthedefactoplacefordeveloperstoaddtheirtoolsasnewplugins,whichareeasytoaddasanewpluginsinherWordPressblog.Shecanchooseonlytogowithtriedandtestedalgorithmsfromwellknowngroups,orshecanexperimentwithnewones.Ratingsandreviewsfromfelloweducatorsproviderapidalertstonewreleasesandfeatures.Fourth,theopenplatformmeansshecanalsocomparecurrentanalyticswithother,anonymizeddatasetswithinthefaculty,butalsowithotherfacultydata,andindeed,withdatafromsimilarcoursesinotheruniversities.Fifth,Jennyhasaccesstotoolsthatreflectthesomewhatlesstangibleaspectsoflearning,suchassentimentsoflearnersinrelationtoatopic,thelivelinessofdebatearoundatopic,orengagementlevelsofdifferentlearners.Sheisalsoabletoreflectontheeffectofher(andtheLAplatform’s)interventionswherestudentsshowsignsofdifficultyorneedingadditionalhelp.Finally,analyticsareofnouseifshecan’tfigureoutwhattodowiththem.Shehasbeenonprofessionaldevelopmentcoursesfor2yearsnowtounderstandthedifferentapproachesasgenerictools,butalsoworkingcloselywithdisciplinarycolleaguestocontextualizethemtoPolicyandEnvironmentanalytics,andtodesignandmappedagogicalinterventionstodifferentanalyticspatterns.Someoftheseinterventionsare‘manual’(e.g.shesimplyneedstoprioritizeaconversationwithagivenstudent),whileothersarepatternsamenabletocodingintherecommenderengineassuggestedactionsforthestudenttoconsider,andinsomecases,dynamicchangestothesequenceandmannerinwhichcoursematerialsarepresented,bettersuitedtostudents’characteristics. 8ThesetwomockupexamplesareadaptedfromDesire2Learn’sdemonstrationmovie:http://www.desire2learn.com/demos/Analytics

11

Asareflectivepractitioner,shehasanewkindofworkbenchtoreflectonhereffectiveness.Akeypartofheracademicresearchisthedesignandevaluationofnewanalytics,contributingtothedebateontheethicsandpedagogyofanalytics-drivenpractice,andthedesignofappropriateinterventions.

Image2:Instructordashboard,variousanalyticsmethods

AdministratoruserscenarioTheadministratorsofthefaculty,andofthebroaderinstituteofwhichthisfacultyisamember,alsohaveaviewontoJenny’scourse.TheyareunderstandablylessinterestedinthefinedetailofKris’progress–whichisrightlythejointconcernofJennyandKris.They

12

areinterestedinhighlevelsummarystatisticsonJenny’scourse,andhowthisconnectswiththeuniversity’strategicprioritiesonstudentsuccessandsatisfaction,coursecompletionratesandresourceallocations–forwhichthereisanotherdashboard.Administratorswillhavegreaterabilitiestoanalyze,acrosstheirinstitutions,theeffectivenessandefficiencyofprogramsandlearningresources.Suchwillgreatlyfacilitateimprovedqualityinthedevelopmentandprovisionoflearningmaterials,coursedesign,andteaching.Whilecrossprogramcomparisoniscomplex,thedatawillallowareasofcommonalitytobedeterminedallowingforcross-institutionalandinter-programimprovement.AnewdimensionforthemisthatalluniversitiesintheNextGenerationAnalyticsConsortium,andallfacultieswithinagiveninstitution,shareanonymizeddatawitheachother,inordertofacilitatebenchmarking,andpromoteethical,effectivepracticeswithinandbetweeninstitutions.Wehavenotmockedupthisdashboard,butenvisagevisualanalyticssummarizingkeyoutputstunedtotheadministrators’priorities.

Researcheruserscenario:AnalyticsonlearninganalyticsuseThebroader“bigdata”movementineducation,andspecificallylearninganalytics,educationaldatamining,andrecommendersystems,arestillintheearlystagesofdevelopment.Theproposedplatformisintendedtoensurethatassoonaspossible,wehavea‘commons’wherethequestionsandtechnicalpossibilitiesthatresearchersfrommanycommunitiesbring,canmeetwiththeprioritiesandconstraintsthatlearners,educatorsandadministratorsbringintheirdailylives(e.g.inskills,andinthetechnologiestowhichtheyhaveaccess).Thefollowinghighlightthreedistinctivewaysinwhichtheplatformwillenablethis:BringingresearchersintoJenny’s‘classroom’.Theplatformprovidesabridgeoverthetraditionalresearch/practicedivide,sincethoseresearchers/developerswhobuildanddeveloptoolsgoodenoughthatUniversityXYZ,KrisorJennychoosetousethemarerewardedwithauthentic,timelydata–butinthespiritoftheinitiative,thisisdataopentoall.Wehaveamechanismformorerapidfeedbackloopsanddesigniterations.WhenresearchersareconnectedwiththedifferentscenariosofLAuse,theycanquicklydevelopanddeployprototypestoimprovetheanalyticsprocess.Essentially,theresearcherispartofafeedbackloopthatquestionsandevaluatestheanalyticstoolsandstrategiesusedbystakeholders.Meta-analysiswithinandacrosstools.Theresearchcommunityaddsameta-analyticlayerontopoftheplatform,ensuringthatthesystemitselfisbeinganalyzedandevaluated.Naturally,theplatformwillitselfgenerateanalyticstoinformusability,technicaldecisions,andquestionsaroundthedynamicsofadoptionofsuchnoveltools.WhichanalyticsaremosthelpfulinassistinglearnerslikeKrisinmanagingtheirlearning?Howdoadministratorsalterprocessesbasedonnewinsightsgainedfromanalyticstools?WhichtoolsareusedbyeducatorslikeJenny,anddoweseehowhersocialnetworkmayinfacthelpspreadherexperiencesandpeers’adoptionofherpreferredtools?Whatarethemosturgentfeaturerequests,asdeclaredbylearners,educators,andadministrators?

13

Combinatorialanalytics.Givenmultipleanalyticsonthesamedataset,eachofwhichprovidesalensthataddsmorecontextualinsight,researchers(andlikelypolicyanalystsandreflectiveeducators)willhave,forthefirsttime,thepossibilitytolookforpatterns.IsKris’spositioninhissocialnetworkcorrelatedwiththewayheuseslanguage?Ishispassratecorrelatedwiththenumberofquestionsheasks?WhenJennyfocusesstudentattentionontheirteam-working,doesthishaveanyimpactontheircollaborationbehavior,andhowdoesthisimpactdropoutrates?

TechnicalstrategyHowcansuchaplatformbedelivered,whichasemphasized,enablesthemtocompareandcontrasttools,anddatasets,fromdiversesources?Fundamentally,werequireanopenplatformwithstandardsforaddingnew“plugins”(seeimage3).Aslongasdevelopersofanalytics,recommenderservices,visualuserinterfaces,andinterventionstrategies,complywiththesestandards,theirworkcanbecomepartofthisecosystem.(Wedonotdetailinthisoutlinedocumentthetechnicalissuesindefiningsuchstandards,thoughnumerousexamplesofsimilarapproachesexist,mostnotably,Moodle,theopensourcelearningmanagementsystem).

Image3:Integratedlearninganalyticssystem

AnalyticsEngineTheanalyticsengineisthecentralcomponentintheLAsystem.Itisaframeworkforidentifyingandthenprocessingdatabasedonvariousanalysismodules.Forexample,theanalysisofadiscussionforuminanLMSwouldinvolveidentifyinganddetailingthescopeoftheforumsandthenapplyingvarioustechniques(seeimage4)suchasnaturallanguageprocessing,socialnetworkanalysis9,processmining(toconsiderthedegreeofcompliance 9Haythornthwaite, C. (2008). Learning relations and networks in web-based communities. International Journal of Web Based Communities, 4(2), 140-158.

14

betweeninstructionaldesignandthelogdataoflearneractivities),traceanalysisofselfregulatedlearning10,thedevelopmentofpredictionmodelsbasedonhumanassessmentofinteractions11,ortheprocessofconceptdevelopmentinsmallpeergroups.AsLAdevelopsasafield,pluginsdevelopedbyotherresearchersorsoftwarevendorscanbeaddedasmodulesforanalysis.Theanalyticsengineincorporatesdatafromlearningmanagementsystems,socialweb,andphysicalworld-data(suchasclassroomattendance,useofuniversityresources,GPS-datawhencompletingactivitiessuchassurveying),andwillleveragebestpracticesfromboththelearninganalyticsandeducationaldatamining1213communities.

LearningAdaptationandPersonalizationEngineThelearningadaptationandpersonalizationwillincludeadaptivityofthelearningprocess,instructionaldesign,andlearningcontent.Forexamplethisadaptationenginecouldconnecttheanalyticsenginewithcontentdevelopers.DeveloperscouldincludeexistingpublisherssuchasPearsonorMcGraw-Hillaswellasinstitutionaldeveloperssuchasinstructionaldesignersandanyimplementedcurriculumdocumentationprocessesandtools.Whenlearningmaterialsaredesignedtoreflecttheknowledgearchitectureofadomain,thecontentdeliveredtoindividuallearnerscanbecustomizedandpersonalized.Thepersonalizationandadaptationenginedrawsfromthelearner’sprofileasdefinedinthelearningmanagementsystemandsocialmediasources(whenpermittedbythelearner).

TheInterventionEngineTheinterventionenginewilltracklearnerprogressandprovidevariousautomatedandeducatorinterventionsusingpredictionmodelsdevelopedintheanalyticsengine.Forexample,alearnerwillreceiverecommendationsfordifferentcontent,learningpaths,tutors,orlearningpartners.Thesesoftinterventionsarenudgestowardlearnersuccessbyprovidinglearnerswithresources,socialconnections,orstrategiesthathavebeenpredictivelymodeledtoassistothers.Recommendationshavebecomeanimportantpartoffindingresourcesonline,asexemplifiedbyAmazon(books),Spotify(music),andBingorGoogle(search).Ineducation,recommendationscanhelplearnersdiscoverrelated,butimportant,learningresources.Additionally,theinterventionenginecanassistlearnersbytrackingprogresstowardlearninggoals.Automatedinterventionsalsoincludeemailsandremindersaboutcourseworkorencouragementtologbackintothesystemwhenlearnershavebeenabsentforaperiodoftimethatmightindicate“riskybehavior”.Interventionswillalsobetriggeredforeducatorsandtutors.Whenalearnerhasbeennotifiedbyautomatedemail,buthasfailedtorespond,theinterventionenginewillescalate

10Hadwin,A.F.,Nesbit,J.C.,Code,J.,Jamieson-Noel,D.L.,&Winne,P.H.(2007).Examiningtracedatatoexploreself-regulatedlearning.MetacognitionandLearning,2,107-12411Baker,R.S.J.d.,deCarvalho,A.M.J.A.(2008)LabelingStudentBehaviorFasterandMorePreciselywithTextReplays.Proceedingsofthe1stInternationalConferenceonEducationalDataMining,38-47.12Baker,R.S.J.d.,Yacef,K.(2009)TheStateofEducationalDataMiningin2009:AReviewandFutureVisions.JournalofEducationalDataMining,1(1),3-17.13Romero,C.Ventura,S.(inpress)EducationalDataMining:AReviewoftheState-of-the-Art.IEEETransactiononSystems,Man,andCybernetics,PartC:ApplicationsandReviews.

15

thesituationbysendingeducatorsandtutorsnoticestodirectlycontactthestudent.Thevalueofdirectinterventionbyateacherasamotivatingconditionforreturntolearningtasksiswelldocumentedbyexistingeducationresearch.

TheDashboardThedashboardisthesensemakingcomponentoftheLAsystem,presentingvisualizeddatatoassistindividualsinmakingdecisionsaboutteachingandlearning.Thedashboardconsistsoffourviews:learner,educator,researcher,andinstitutional.Learnerswillbeabletoseetheirprogressagainstthatoftheirpeers(nameswillbeexcludedwhereappropriate),againstlearnerswhohavepreviouslytakenthecourse,againstwhattheythemselveshavedoneinthepast,oragainstthegoalsthattheteacherorthelearnerherselfhasdefined.Educatorswillbeabletoseevariousrepresentationsoflearneractivity,includingconceptualdevelopmentofindividuallearners,progresstowardmasteringcoreconceptsofthecourse,andsocialnetworkstoidentifylearnerswhoarenotwellconnectedwithothers.Analyticsforeducatorswill,dependingonthecontext,begeneratedrealtimeaswellashourlyordailysnapshots.Thedashboardwillprovideinstitution-levelanalyticsforsenioradministratorstotracklearnersuccessandprogress.Whencombinedwithacademicanalytics,thismodulewillbevaluableforanalyzinginstitutionalactivities(businessintelligence).Basedoncriteriaestablishedthroughresearchofthelearninganalyticssystem(suchastheimpactofsocialconnectivityoncoursecompletion,warningsignalssuchaschangesinattendancepatterns,predictivemodeling),automatedandhumaninterventionswillbeactivatedtoprovideearlyassistancetolearnersdemonstratinga)difficultywithcoursematerials,b)strongcompetenceandneedingmorecomplexordifferentchallenges,andc)atriskfordropout.

Image4:Sampletechniquesfortheanalyticsengine

16

ContextandDepthofAnalysisLearnersandeducatorsrequiredifferentdepthsofinformationindashboards,basedonthecontextandtypeofanalytics.Asaresult,dashboardswillbeconstructedtoallowfor“drillingdown”togetmoredetailedandnuancedanalysisoflearningtrends.Image5detailsdrilldownoptionforlearners,helpingtoenhancetheirexperienceinacourse14andimage6detailsdrilldownoptionsforinstructors,particularlyinidentifyinglearnerswhomightbeatriskfordropoutorwhofaceacademicchallenges.15.

Image5:Learnerdashboard,showing“drilldown”options

14Dawson,S.,Heathcote,L.andPoole,G.(2010),“HarnessingICTPotential:TheAdoptionandAnalysisofICTSystemsforEnhancingtheStudentLearningExperience,”TheInternationalJournalofEducationalManagement,24(2),pp.116-129.15Macfayden,L.P.,&Dawson,S.(2010).MiningLMSdatatodevelopan"earlywarning"systemforeducators:aproofofconcept.Computers&Education,54(2),588–599.

17

Image6:Instructordashboard,showing“drilldown”options

DatamartInordertoapplylearninganalyticsatabroaderscale,itisimperativethataninfrastructurebecreatedthatcansupportthedevelopmentoftoolsandservices.Suchaninfrastructurewillneedbasictechnicalagreementoncommonstandardsandprotocols16.Afirstquestionishowtomodeltherelevantdata.OurearlyworkonContextualizedAttentionMetadata(CAM)1718definesasimplemodeltostructureattentionmeta-data,i.e.theinteractionsthatpeoplehavewithobjects.Theontology-baseduserinteractioncontextmodel(UICO)19focusesmoreonthetasksthatpeoplecarryoutwhileinteractingwithresources.ThePittsburghScienceofLearningCenterDataShop20providesaframeworkforstructuringdatafromtheinteractionbetweenlearnersandvarioustypesofinteractivelearningenvironments,potentiallyprovidinglessonsthatcanbeadaptedtothedifferent

16Duval,E.Verbert,K.“Ontheroleoftechnicalstandardsforlearningtechnologies.”IEEETransactionsonLearningTechnologies1,no.4(October2008):229-234.https://lirias.kuleuven.be/handle/123456789/234781.17J.Najjar,M.Wolpers,andE.Duval.Contextualizedattentionmetadata.D-LibMagazine,13(9/10),Sept.2007.18M.Wolpers,J.Najjar,K.Verbert,andE.Duval.Trackingactualusage:theattentionmetadataapproach.EducationalTechnologyandSociety,10(3):106–121,2007.19A.S.Rath,D.Devaurs,andS.Lindstaedt.UICO:anontology-baseduserinteractioncontextmodelforautomatictaskdetectiononthecomputerdesktop.InProceedingsofthe1stWorkshoponContext,InformationandOntologies,CIAO’09,pages8:1—-8:10,NewYork,NY,USA,2009.ACM.20Koedinger,K.R.,Baker,R.S.J.d.,Cunningham,K.,Skogsholm,A.,Leber,B.,Stamper,J.(2010)ADataRepositoryfortheEDMcommunity:ThePSLCDataShop.InRomero,C.,Ventura,S.,Pechenizkiy,M.,Baker,R.S.J.d.(Eds.)HandbookofEducationalDataMining.BocaRaton,FL:CRCPress,pp.43-56.

18

typesofdatathisprojectwillencompass.Moreworkisneededtodevelopawidelyapplicablesuchparadataschema.Similartothewaythatlearninglearningobjectsandtheirmetadatacanbemanaged21,wewillneedaservicearchitecturethatcanpoweraplethoraoftoolsandapplications.Oneinterestingapproachistorelyonwidgettechnologiesthatenablethedynamicembeddingofsmallapplicationcomponents-anapproachatthecoreofPersonalLearningEnvironments(PLE’s),researchedintheROLEprojectonResponsiveOpenLearningEnvironments(seehttp://www.role-project.eu).AnotherapproachistheLearningRegistryarchitecturethatmakes”userdatatrails”availablethroughanetworkofnodesthatprovideservicestopublish,access,distribute,brokeroradministerparadata(seehttp://www.learningregistry.org/).Whererelevant,itmaybefeasibleanddesirabletoexportdatatoexistinganalyticaltools,forinstancethetoolsforprofilinglearnerperformanceandmodelinglearnerskillinthePittsburghScienceofLearningCenterDataShop(seehttps://pslcdatashop.web.cmu.edu).

ModularizedDevelopmentTheproposedOpenLearningAnalyticsplatformwillbedevelopedinseveralresearchstages,includingthedevelopmentofsamplecourses(forpersonalizationandadaptivity),thedeploymentoftheanalyticsengineinonlineandblendeduniversitylevelcourses,andtheevaluationofdashboardswithmultiplecoursesandprogramstoensuredataisclearlypresentedandevaluated.Thescopeoftheintegratedlearninganalyticstoolsetisbroad,butthemodularizeddevelopmentoftheanalyticsengines,andthespecificwidgetsorpluginsthatextendthoseengines,willbehandledbyindividualresearchersandresearchteams.Forexample,thedashboardwillbeleadbyresearcherswithexpertiseinthevisualdisplayofinformation.Thisresearchteamwillinteractextensivelywiththecontent,analytics,andinterventionresearchteams.Theapproachofmodularized,butintegratedallowsforconnectedspecializationwhereresearchteamsareconnectedwithabroadervision,butstilldeveloptheirspecificresearcharea.

InvestigatorBiographiesTheinvestigatorsareattheforefrontoftheLearningAnalyticsfield,leadingataninternationallevel,theconferences,researchandpublicationsthatareshapingthescopeofthisemergingfield.Theyrepresentacross-sectionofresearchersinbothtechnicalandsocialdomainsfromaninternationalsetofuniversities.

21S.Ternier,K.Verbert,G.Parra,B.Vandeputte,J.Klerkx,E.Duval,V.Ordonez,andX.Ochoa.TheAriadneInfrastructureforManagingandStoringMetadata.IEEEInternetComputing,13(4):18–25,July2009.

19

Moreinformationoninvestigatorscanbefoundonthefollowingsites:

GeorgeSiemens:https://tekri.athabascau.ca/content/george-siemensDraganGasevic:http://www.sfu.ca/~dgasevic/SimonBuckingham-Shum:http://people.kmi.open.ac.uk/sbs/RebeccaFerguson:http://kmi.open.ac.uk/people/member/rebecca-fergusonCarolineHaythornthwaite:http://www.slais.ubc.ca/people/faculty/faculty.htmShaneDawson:http://www.arts.ubc.ca/faculty-amp-staff/arts-isit/staff.htmlErikDuval:http://erikduval.wordpress.com/about/KatrienVerbert:http://people.cs.kuleuven.be/~katrien.verbert/RyanS.J.d.Baker:http://users.wpi.edu/~rsbaker/

ConsiderationsOurintentforthisprojectistocreatethearchitectureforresearcherandpractitionerinnovationineducationthroughaccesstotoolsthatprovideinsightintotheteachingandlearningprocess.Aspartofthiswork,inadditiontothebroadconceptualoverviewofourintegratedlearninganalyticsplatform,weemphasizethefollowinggoals:

- Developmentofcommonlanguagefordataexchange.- Analyticsengine:transparencyinalgorithmssolearnersandeducatorsareawareof

databeinggatheredandresearcherscancustomizeanalyticsmethodstoreflecttheneedsofdifferentcontexts(schools,circumstances,policyoradministrativepriorities).

- Dashboardsandreportingtoolstovisualizeinformationandprovidereal-timeinformationtolearners,educators,administrators,andresearchers.

- Openrepositoryofanonymizeddatafortrainingandresearchdevelopment- ConnecttoandamplifytheexistingresearchbeingconductedbyEDUCAUSE,the

InternationalEducationalDataMiningSociety,NextGenerationLearningChallenge,andrelatedanalyticsinitiativessuchastheEUdataTELinitiative.

GettingInvolvedTheSocietyforLearningAnalyticsResearch(www.Solaresearch.org)welcomespartners–universitiesasfoundingmembersandresearchers,tojoinintheOpenLearningAnalyticsinitiative.Pleasecontactusatinfo@solaresearch.orgformoreinformation.

ConcludingThoughtsAllstakeholdersintheeducationsystemhaveaccesstomoredatathantheycanpossiblymakesenseoformanage.Inspiteofthisabundance,however,learners,educators,administrators,andpolicymakersareessentiallydrivingblind.Newtechnologiesandmethodsarerequiredtogaininsightintothecomplexabundantdataencounteredonadailybasis.Thehistoryoftechnologyadoptionineducationsuggestsaconsistentandchallengingmodel:importantideasandinnovationsdevelopedpiecemealandinisolation,resultinginafragmentationandconfusionforenduserswhoaremostinneedofefficientsolutions.Our

20

proposedintegratedlearninganalyticsplatformattemptstocircumventthepiecemealprocessofeducationalinnovationbyprovidedanopeninfrastructureforresearchers,educators,andlearnerstodevelopnewtechnologiesandmethods.Intoday’seducationalclimate–greateraccountabilityinaclimateofreducedfunds–suggestsnewthinkingandnewapproachestochangearerequired.Analyticsholdtheprospectofservingasasensemakingagentinnavigatinguncertainchangebyofferingleaderswithinsightfuldataandanalysis,displayedthroughuser-controlledvisualizations.