1

DigitalisationofEducation:ApplicationandBestPracticesOlgaViberg&AnnaMavroudi

KTHRoyalInstituteofTechnology,SchoolofElectricalEngineeringandComputerScienceResearchGroupinTechnology-EnhancedLearning

e-mail:oviberg@kth.se;amav@kth.se

TableofContentsDigitalisationofEducation:ApplicationandBestPractices......................................................1

Introduction..........................................................................................................................1

1. LearningManagementSystems...................................................................................2

2. MassiveOpenOnlineCourses......................................................................................3

3. Learninganalyticsdashboardsandvisualisations........................................................4

4. Data-supportedfeedback.............................................................................................5

5. Studentmisconceptions...............................................................................................6

6. Identifyingat-riskstudentsandpredictiveanalytics....................................................7

7. Studentstrategiesandattitudes..................................................................................8

8. Learninginasocialcontext..........................................................................................9

9. Learninganalyticsandlearningdesign......................................................................12

References...........................................................................................................................14

IntroductionThetopicofdigitalisationofeducationhasattractedtheinterestoftheresearchcommunityworldwide due to unprecedented capabilities provided by the technology to capture the‘digitaltraces’oftoday’sstudentswho,being‘digitalnatives’,areactive intechnology-richlearningenvironments.TheaimofthisreportistopresentsolutionsonthetopicthatcanbeapplicabletoSwedishcontext.Thesesolutionsemergeasbestpracticesorexamplesderivedfromsearchingtherecentinternationalliterature.TheycanbeadoptedormodifiedtofittheSwedishcontext,giventhatitisraretofinduniversalsolutionsthatcanbeapplied‘as-is’(i.e.,directly)ineverysocio-culturalcontext.

Thisreportcomprisestendifferentapplicationsondigitalisationofmoderneducation,which in their majority are relevant to Learning Analytics, defined as “the measurement,collection,analysisandreportingof[big]dataaboutlearnersandtheircontexts,forpurposesofunderstandingandoptimizinglearningandtheenvironmentsinwhichitoccurs”(Long&Siemens,2011,p.34).Learninganalyticsevidenceforlearninghasearlierbeenvalidatedandpresentedaccordingtofourpropositions:

- Learninganalyticsimprovelearningoutcomes- Learninganalyticsimprovelearningsupportandteaching- Learninganalyticsaretakenupandusedwidely,includingdeploymentatscale- Learninganalyticsareusedinethicalway(seeFerguson&Clow,2017;Vibergetal.,

2018).

2

First,thisreporttouchesuponsomebestpracticesofdigitalisationinconjunctionwithsome prominent learning technology, in particular with Learning Management Systems(LMSs),MassiveOpenOnlineCourses(MOOCs),andlearningdashboards(sections1-3).Thesecondcategorydiscussestheuseoflearninganalyticswithafocusonaddressingparticularproblems (sections 4-7): provide timely and relevant feedback, destabilise studentmisconceptions, help students develop effective learning strategies, and identify at-riskstudentswiththeaimofreducingdrop-outs.Finally,twointerestingandrelevantthemesarediscussedwhicharecloselyrelatedtocurriculumdesign(sections8-9):theinterplaybetweenlearninganalyticsandlearningdesign,learninganalyticsinasociallearningcontext.Itshouldbenotedthattheresearchstudiesmentionedinonecategorymightalsofittosomeothercategory.

In addition to the relevant literature, the report is particularly inspired by recentreviewsoflearninganalytics:Vibergetal.(2018),Misiejuk&Wasson(2017),andLangetal.(2017). These reviews suggest that thereare several researchgaps in theareaof learninganalytics. They currently include the application of learning analytics in K-12 education,researchoneverydayanalyticsinclassrooms,researchonassessment,researchonlearning-centric analytics, as opposed to learner-centric analytics, implementation and impact oflearninganalyticsanddataliteracy,withtheexceptionsofafewstudiesthataddresswhetherornotstakeholders (i.e., teachersand learners)canunderstandthevisualisations theyarepresented(Misiejuk&Wasson,2017).Mostofthelearninganalyticsresearchhassofarbeenconductedinhighereducation.

1. LearningManagementSystemsLearningManagementSystems(LMS;e.g.,Moodle,Blackboard,Canvas)arevirtuallearningenvironmentsthatarebeingwidelyusedinmanydifferenteducationalsettings.Tracedataaboutstudents’activityinaLMSareoftenattheheartoflearninganalyticsapplications.Forexample,Gaševićetal.(2016)haveshownthatonlythefollowingvariables-numberofloginsandnumberofoperationsperformedondiscussionforumsandresources-weresignificantpredictors of academic performance, predicting approximately 21% of the variability inacademic performance. Hence, it is difficult to translate these predictors into actionablerecommendationstofacilitatestudentlearning.AnanswertothisissueisofferedbyNguyenetal.(2018),whoarguethatthekeytoactionableinsightsistheassociationbetweenlearninganalyticswithlearningdesign.Thisassociationenablestheteachertocompareandcontrastthe actual online behaviour of the students with the teacher’s intentions. It is furtherdescribedinsection9onlearninganalyticswithlearningdesign.

LearninganalyticshavebeenshowntomakeextensiveuseoftracedatafromlearnersinteractingwithLMSs,andoneofthemostcommonusesistoderiveanestimateofthetimeeachlearnerspentontask,thatisengaginginparticularlearningactivities.Kovanovicetal.(2015)presenttwoexperimentsexploringdifferentmeasuresoftimeontask;oneusingdatafromafully-onlinecourseataCanadianuniversity,andanotherusingdatafromablendedcourse at an Australian university. Based on modelling different student performancemeasureswithpopularstatisticsmethodsintwodatasets,theresultsshowthattime-on-taskmeasures typically outperformed countmeasures for predicting studentperformance, butmore importantly, identified that the precise choice of time-on-taskmeasure "can have asignificant effect on the overall fit of the model, its significance, and eventually on theinterpretationofresearchfindings"(p.105).Insummary,time-on-taskestimationmethodswerefoundtoplayaconsiderableroleinshapingthefinalsstudyresults,especiallyinonlineenvironmentswheretheamountofinteractionwithLMSisusuallyhigher.Fromapracticalperspective, researchers recommend todevelop“standardizedplugin for theextractionof

3

tracedatafrompopularLMSsystems(e.g.,Moodle,WebCT,Canvas)thatcouldprovidefastandeasytotime-on-taskandcountmeasures”(Kovanovicetal.,2015,p.105).Today,westillneedsuchautomatizedtoolsthatcouldbesmoothlyintegratedintoexistingLMSstobetterunderstandhow learners spend timeon their learning activities and thus to createbetteropportunitiesfortheireffectivelearning.

2. MassiveOpenOnlineCoursesMassiveOpenOnlineCourses(MOOCs)areavailabletothousandsofstudentsviasomeweb-basedplatform,increasingaccesstohigh-qualitymaterialfordistanceandlifelonglearners.MOOCscreatelargeamountsofdatacanfeedthevariouslearninganalyticstools.However,mostMOOCsstilladoptatop-downteachingapproach,ignoringthepotentialforfacilitatingawareness,self-regulationandpersonalisation (Drachsler&Kalz,2016).DrachslerandKalz(2016)describedthepotentialof learninganalytics inthecontextofMOOCs. Inparticular,theydescribeaMOOCLearningAnalyticsInnovationCycle(MOLAC;Figure1)whichcanbeappliedatthemicro,meso,andmacrolevel.

Figure1.TheMOOCLearningAnalyticsInnovationCycle(adoptedfromDrachsler&Kalz,2016)

On the micro-level, the data collection and analytics activities are focused on individualreflectionandindividualprediction.Onthemeso-level,datafromseveralopencoursesarecombined to support benchmarking and to create insights about behavior of groups oflearners rather than the individual. Finally, on themacro-level, cross-institutional learninganalyticsenablestodeveloplearningandteachinginterventionsthatcanbetestedinaclusterof educational institutions to analyse the impactof such interventions, beyond contextualfactors.IntheirreviewofMOOCsandlearninganalytics,theauthorspinpointthatpresentlymostoftheconductedstudiesfocusedonthemicro-leveloftheMOLACcycle.

AchallengewithMOOCSisthat,unliketraditionaleducationalsystems,measureslikedetaileddemographicsandpriorknowledgearenotcollected inorder tomaintaina lowbarrier toentry. Many MOOC-providing institutions thus have begun to collect this data throughoptional surveys. Students completing their courses are also thosewho tend to completethesesurveys.Reich(2014)suggestthatroughlyonequarterofenrolledlearnersfilloutsuchcoursesurveys.Thevolumeofcollectedresponsescanbehigh(oftentensofthousands)butmightrepresentaskewedsampleofgenerallymoremotivatedlearners.

4

Withrespecttostudents’affectivedomain(i.e.,thedomainthatincludesthemannerinwhichindividualsmanagethingsemotionally,forexamplevalues,feelings,andappreciation),Leonyetal. (2015)built fourmodels todetectboredom, frustration,happiness,andconfusion inMOOCs.Themodelswere testedonagroupof studentsand thecorrelationbetween theemotionsandstudent’sinteractiondatawascalculated.ThefoursuggestedmodelshavebeenimplementsintoALAS-KA,alearninganalyticsmodulefortheKhanAcademyplatform.Thismodule includes new metrics (except those already provided by the platform) andvisualisationsofinformationtakenfromtheanalysisoftheuseractivities’patterns.AlloftheimplementedmodulestakeintoaccounttheProblemLogprovidedbytheKhanAcademy.Itincludesalloftheinformationaboutusers’interactionwithexercisesandthetimestampinwhichtheytookplace.Thefourmodelswereappliedonthe90studentstakingthecoursesinmathematics,physicsandchemistryin2013.Theinitialanalysisofemotionvaluescalculatedforaperiodof10dayshasshownseveralexpectedpatterns,suchasincrementsofemotionlevelsaslearnerswereinteractionmorewiththecoursematerials.Theuseofrulesforthedetectioncanhelptoovercomeseveralissues,includingaslowstart,giventhatataninitialmomenttheremightnotbeenoughdatatocalculatethelearners’emotionsvalues.Byaddingthelogictotheinferenceofeachemotion,educationalinstructorscanbeawareofthemetricsthatcouldhaveinterferedtobeabletodetectachosenemotioninalearnerandthustomakebetterinformeddecision.

3. LearninganalyticsdashboardsandvisualisationsLearning-analyticsdashboards(LAD)arecurrentlythemaintrendintheliteraturesincetheymirror the students’ behaviour back to the students and/or/or to the teachers. Learningdashboardisdefinedas“asingledisplaythataggregatesmultiplevisualisationsofdifferentindicatorsaboutlearner(s),learningprocess(es),and/orlearningcontext”(Schwendimannetal.,2016). Millecampetal.(2018)presentaLADthatsupportsthedialoguebetweenastudentand a study advisor. To ensure transfer to other contexts (in terms of scalability), thedashboardvisualisesdatathatiscommonlyavailableinhighereducation,likethegradesofthe student, study progress, and academic ranking in comparison with peers (i.e., socialcomparison), and a prediction of the duration of study for the bachelor program for thisstudent,basedonhistoricdata.AdashboardwasdeployedatKULeuven(Belgium)tosupportdiscussion sessions. The results indicate that the dashboard primarily triggers reflectiveconversationswithstudentsdoubtingtocontinuethebachelorprogram,indicatingthatthedashboardisusefultosupportdifficultdecision-makingprocesses. AnotherexamplepresentstheStudentActivityMeter(SAM)forawarenessandself-reflection (Govaerts et al., 2011). SAM visualises learning activates within online learningenvironments for learners and teachers to help increase awareness and to support self-reflection.Thisdashboardsupportsthefollowingteacherobjectives:

- Awareness for teachersofwhatandhow learnersaredoing is important toassesslearnersprogress

- SAMvisualisesthetimelearnersspentandtheresourcestheyused- Timetrackinginformationallowsteacherstoassesstheirinitialtimeestimateswith

therealtimespendingofthestudentsandfintheexercisesthatconsumemosttime- The resourceusagecanshowthepopular learningmaterialsandenables resource

discovery,throughalistofmostusedormosttimespentonresourcesinSAM.

5

TheresultsoftheconductedcasestudyshowthatSAMcontributestocreatingawarenessforteachers.Itenables:i)tofindstudentsdoingwellandatrisk,ii)abettercourseoverview,andiii)understandingstudenttimespending.MostoftheparticipantswantedtocontinueusingSAM. Broosetal. (2017) introducedaLADforactionablefeedbackonlearningskills.Thedashboard provides actionable feedback about five of the learning skills assessed by theLearningAnalyticsStudyStrategiesInventory(LASSI):concentration,anxiety,motivation,teststrategies,andtimemanagement.Itusesdataaboutstudents,theirlearningskillsandscoresthatwerealreadyavailablewiththeinstitution(KULeuven),butfragmentedacrossservicesandnotfedbacktothedatasubjects(i.e.,learnersandteachers)inadirectway.14061styearstudentsin12differentstudyprogramsatKULeuvenwereinvitedtousethedashboardtosupportthemintheirtransitionfromsecondarytohighereducation.TheLASSIwasusedtoassess learningskillsusingasurveytakenwithinthefirstweeksof theacademicyear.Thedashboardsystem (for thehigh-leveloverviewof thesystemseeFigure2)didnot requiredirectaccesstothestudents’namesorothercharacteristicsthatallowforstraightforwardidentification,astheyweremadeavailableataccesstimebythesingle-signoninfrastructure.

Figure2.Overviewofsystemsanddataflowstosupportthedashboard(adoptedfromBroosetal.2017)

ThesystemloggedaccesstoandbehaviorwithintheLADandanalysedtheirrelationshipwiththese learning skills. The results show that while 8 out of 10 students accessed the LAD,studentswithlowertimemanagementscorestendtohavealowerclick-throughrate.Oncewithinthedashboard,studentswithlowerscoresforspecificlearningskillsareaccessingthecorresponding information and remediation possibilities more often. Regardless of theirscoresforanyoftheotherlearningskills,learnerswithhighermotivationscoresarereadingtheremediationpossibilitiesfortheotherfourlearningskillsmoreoften.GenderandstudyprogramwerefoundtohaveaninfluenceonhowlearnersusetheLAD.

4. Data-supportedfeedbackTheprovisionoftimelyandrelevantfeedbackisanothertopiccloselyassociatedtotheuseoflearning analytics. Kickmeier-Rust et al. (2014) developed a gamified learning app, ‘SonicDivider’1tohelpprimaryschoolstudentsinAustralialearnmathematics.Itfeaturesformativeassessment and feedback functions and gives the teacher a quickly overview about theachievements, scores, and competency levels. The authors examined the link betweenformativefeedbackandlearningperformance,andanalysedtheresultsbygenderandoverall

1SoniceDivider,http://next-tell.eu/portfolios/sonicdivider/

6

bystatisticalmethods.Theyfoundevidenceforthemotivationalaspectofthegamificationelements,inparticularscoring.Theyalsoidentifiedpositiveeffectsofanindividualizedandmeaningful,elaboratedfeedbackabouterrorsmade.Withrespecttogenderdifferences,theauthors found evidence that girls were less attracted by competition elements (e.g., bycomparing high scores) than boys, however, they weremore attentive towards feedbackcomingfromthetool.

“Learning ispersonal (LIP)” 2 isanotheravailabletoolthatoffersdata-supportedfeedback.TheLIPtoolcaptureslearningactivitiesinreal-timeandprovidesinstantstatisticalfeedback.Itmainly targets teachers, studentsandstudents’peers. Inparticular, it supports learners’peer-andself-assessmentsaswellasteachers’observations.Itisoptimizedtofitclassroomactivitieswhere timeandattentionare scarce resources. The tool’s focus is toprovideanoverviewoverall learningactivitiesinallsubjectsoverthewholeterm.Itdoesnotprovidespecificinsightsorin-depthanalyses.However,itprovidestheteacherwithsimple,real-timeinsightsandallowstheteachertoenhancethedatamodel(e.g.,usedmaterial)onthefly.Thistoolisoptimisedforperformanceandtobeontablets,mobilesandlaptops.

5. StudentmisconceptionsStudents’misconceptionsaboutasubjectmightbedeeply-rootedandcanimpedelearning,whichsuggeststhatthepracticeofonlyconsideringthefinalsolutionoftenresultsinstudentknowledge gaps andmisconceptions (Davies et al., 2015). The area of intelligent tutoringsystems3 (ITS)hasa long tradition inclarificationofmisconceptionsandpersonalisationoflearning(e.g.,personalisedfeedback).

Davies et al. (2015) compared differences in detecting students’ knowledge gaps andmisconceptionsabouttheuseofabsolutereferences4whenusingassessment-leveldata(i.e.,thefinalsolutionstudentssubmitwhensolvingaproblem)tothatofusingtransactional-leveloractivity-tracedata(i.e.,theprocessstudentstaketoarriveattheirfinalsolution).Intheirstudy, the researchers – based on the analysis of the student assessment data (i.e., theirwrittenhomework,mid-termexaminationandfinalexams)collectedfromthreeuniversitiesin the western United States in 2014 (995 students) - found higher levels of knowledgecomponents gaps and misunderstandings when assessing transactional-level knowledgecomponentdatathantask-levelfinalsolutiondata.

Data was gathered through an ITS, in this case a tool developed in the Microsoft Excelplatform:foreachassignment,thedevelopedsystemcreatedaspecifiedlogofeachstepthestudenttooktocometoasolution,i.e.,notonlythefinalsolutiongradedbytheprogram.Thelogwasthenrecordedonahiddenworksheetwithintheworkbooksothatwhenthestudent’ssolutionissubmittedthelogissubmittedaswell.Thislogcanthenbeextractedforanalysisofthestudent’slearningprocessandthefinalsolution.

Examiningthe finalsolutions thatstudentssubmittedshowed littleevidencethatstudentshad any misunderstandings or knowledge gaps about the use of absolute references.Nevertheless,analysingdataat the transactional level researchers identified that farmorestudentsstruggledusingabsolutereferencesthantheanalysisofonlythefinalsolutions.In2LIPtool,http://next-tell.eu/portfolios/learning-is-personal-lip/3Intelligenttutoringsystemisacomputersystemthataimstoprovideimmediateandcustomisedfeedbacktolearners.4Formoreinformationaboutabsoluteandrelativereferences,seehttps://edu.gcfglobal.org/en/excel2016/relative-and-absolute-cell-references/1/

7

particular, the findings revealed that students not only struggled to solve the problemrequiringtheuseofabsolutereferences,butalsotheytriedtouseabsolutereferenceswhentheywerenotneeded.Thiswasnotdetectedwhenanalysingjustfinalassessmentdata.

All thissuggeststhatoncedata isunderstoodthroughdatamining itcanbeusedtomakedecisions in regard to when and how to intervene in the learning process. The study’simplicationshighlightthepotentialofmakingadaptationstoinstruction,providingfeedbackand remediation, and also improvements in the ability to diagnose knowledge gaps thatinfluence student comprehension. For researchers and practitioners there are severalchallengestobeconsideredwhenattemptingtoanalysestudents’misconceptions:

- Getting the rightdata isvital! (i.e., in theaforementionedcase, transactional leveldata)

- Whencollectingdata,meaningfuldataloggingproceduresshouldbeestablishedinadvance; this includes capturing pertinent data that can be linked to otherinformation, e.g., systems level data. At this point data must be extracted andanalysedmanually(Daviesetal.,2015).

6. Identifyingat-riskstudentsandpredictiveanalyticsTheidentificationofat-riskstudentshasreceivedmuchattentionandithasbeenfrequentlyassociatedwithaspecialcategoryof learninganalyticsnamedpredictiveanalyticswhich isfocusingspecificallyonthataspect.However,thenumberofinitiativesthathavebeenabletotransitionfromconcepttoimplementationisstillscare(Jayaprakashetal.,2014,Vibergetal.2018).TheCourseSignalsprojectatPurdueUniversity(Arnold&Pistilli,2012)isasuccessfulapplicationofpredictivemodelling to student completion inhighereducation.TheCourseSignals system has been developed on the idea that students do not have a goodunderstandingofhowtheyareprogressingintheirstudies/courses.Purduedevelopedtheirapproachto“earlywarningsystems”whichshowsat-riskstudentsoverseveralyears.

The predictive model used is based on four components: demographic characteristics,previousacademichistory,interactionwiththeLMS/VLEduringthecourseandperformanceon thecourse todate.Thepredictions fromthemodelare translated intoa signal:green,denotingahighchanceofsuccess;yellow,denotingpotentialproblems;orred,denotingahighchanceoffailure.Teachersrunthemodelandgeneratesignalsforthestudentsontheircourse.Theteachercanthenchoosewhat interventions to trigger: sendingapersonalisedemailor text,posting the signalon the LMS.PurduepilotedCourseSignals in2007and itbecameautomated in2009. In2012around24000studentsand145educatorshaveusedCourseSignalsinatleastoneofthecourses.Keypointsinclude:i)problemsareidentifiedasearly as the second week in the semester, iii) feedback is categorised as motivational orinformative, iii)studentsusingSignalsseekhelpearlierandmoreoften,and iv)oneofthestudy’sresultsshowed10%moreAsandBsweregivenforcoursesusingSignalscompareforpreviouscourseswhichdidnotusetheSignalssystem.Overall,studentslikedSignals:89%ofthemconsidereditapositiveexperienceandfoundthecombinationofthetrafficsignalsandinstructor feedback tobe informativeandmotivational.Anticipatedproblemswith Signalsincludedmaintaining theprivacyof teachingperformanceandensuring thevalidityof thealgorithm.Aspointedoutbytheearlierresearch(JICS,2017),severalissueshavetobetakenintoconsiderationwhendesigningandimplementingsimilarsystems:

• Studentsdidnotexpress concernsaboutprivacyabout theirdatabut thatdidnotmeanthattheinstitutionshouldnotneglecttoprotectit.

8

• Institutions need to be able to extract data fromdifferent systems efficiently andtimely.

• Giventhecentralityofstudentsuccesstotheinstitutionalmission,learninganalyticseffortsmightbebetter initiatedandledfromaneducation-focusedunitthanfromtheIT-department.

Based on the success of the Signals systems, researchers at theMarist College in the US(Jayaprakashetal.,2014)implementedtheOnlineAcademicAnalyticsInitiative(OAAI)thataimedtohelpstudents,identifiedas‘needinghelp’,toengagetheminanonlinecommunitydesignedtohelpthemtosucceedacademically.TheOAAIdevelopedaprototypeofanopen-source academic earlier alert system that feeds from the Sakai CLE (Collaboration andLearningEnvironment),andincludesopenpredictivemodelbasedonthePentahoBusinessAnalytics suite (an open source analytics suite with data mining and data integrationcapabilities), and intervention strategies that leverage Open Educational Resources. Thepredictivegoalofthedevelopedsystemwastodetectearlythoseundergraduateswhowerein academic difficulty in the course by using student data. Four sources of data wereconsidered: i) studentdemographicandaptitudedata, ii) coursegradeandcourse relateddata, iii) Sakai-generated data on student interaction with the LMS and iv) partialcontributionstothestudent’sfinalgradecollectedbySakai’sgradebooktool,i.e.,gradesonspecificgradingevents,e.g.,assignmentsandexams.ThepilotswereruninSpring2012andFall2012.Studentidentifiedatrisk,weresubjectedto2interventionsstrategies:“AwarenessMessaging”andparticipatinginon“OnlineAcademicSupportEnvironment”.InonecourseatMaristCollegetherewasasignificantimprovementinfinalgrade(6%)withthoseat-riskstudentswhoweresubjecttoaninterventioncomparedwiththoseinthecontrolgroup,whowerenot.

7. LearningstrategiesDevelopingstudents’learningstrategiesandattitudesisintriguingbecauseitinvolveshigher-orderthinkingskillsonbehalfof thestudents, likeself-regulated learning(SRL)andcriticalthinking.While the importanceof SRL to learning is recognised,understandinghowSRL isappliedincontextisnotasimpletask(Roll&Winne,2015).Torealisethepotentialoflearninganalytics foradvancedof the learning sciences,weneed: i) todevelop instrumentation torecord traces of SRL across all its phases, ii) to develop and to testmethods that identifystructureswithindata,iii)toexploretheeffectsofinterventions(Roll&Winnie,2015).

Oneofthesuccessfulapplicationsoflearninganalyticstoimprovestudents’SRLskillsrefers to the study by Tabuenca et al. (2015). In this study, the researchers investigatedstudents’ timemanagement skills as thepart of their SRL, and in particular examined theeffectsoftrackingandmonitoringtimedevotedtolearnwithamobiletoolonSRL.26studentsenrolled intwoonlinecoursesfromtwouniversities intheNetherlandsparticipated inthestudy.TheLearnTrackerBackendDatabasemodelhasbeenused.Fromthestudentactivitypointofview,thesystemhoststhetimestampanddurationofthelearningactivity(formoreseeTabuencaetal.,2015).LearnTrackeroffersmobilesupportforstudentsenrolledinonlinecourses in which the learning goals are predefined by teachers. Courses deployed inLearnTracker are retrieved from the remote database at LearnTracker Backend. Similarly,time-logs are also recorded in the backened. LearnerTracker also provides social learninganalyticscontrastingthetimedevotedbythestudentwiththetimedevotedbyhispeers,aswellasthetimeoriginallyestimatedbytheteacher.Theexperimentaimedtoestimatehowaccurate estimations by instructional designers are with regard to the time needed to

9

accomplisheachlearningactivityscheduledinacourse.Time-logsrecordsduringthecourseandthegradesobtainedinthefinalevaluationweretakenasindicator.Thestudy’sfindingsshowthatusingmobiledevicesto logandtrackthetimedevotedtostudyacrosscontexts(formalandinformal)mightleadtoanimprovementontimemanagementskillsandshowedthatnotificationspushedatrandomtimeofthedaydonotproducesignificantimprovementsintimemanagement.

Researchers have also examined the development of students' reflection skillsthroughagame-basedapproachcalledthe“TrailsofIntegrityandEthics”(TIEs;Kwongetal.,2017). A total of 658 students from four Hong Kong institutions participated in the studybetweenOctober2016toDecember2016.Thisapproachusesfocusedscenariosonethicaldilemmas - coupledwithAugmentedReality (AR)andmobile technologies - todesignTIEswherescenariosaretriggeredinvariouslocationsandstudentsarechallengedtoreviewtheargumentspresentedandreflectontheirethicalchoices.Inparticular,studentsplayedouttheconsequencesoftheirdecisionswhichhelpstrengthenthelinksbetweenthetheoreticalconceptofacademicintegrityandethicsaswellasthepracticalapplicationindailysettings.Contents of all scenarios are designed based in the 3C model, i.e., each decisionmakingscenariohastoincludeaChallenge,Choices,andConsequences.AllTIEsaredeployedusingamobileapplicationMobxz5,aservicedesignedspecificallyforlearningtrailsandsupportingbothiOSandAndroidsmartdevices.Studentsusetheirmobiledevicestowalkthroughatrailandactivatethe learningactivitieswithineachcheckpointscenariomarker-based(e.g.,QRcode)andmarker-less(e.g.,GPSlocationmapping),ARtechnologiesthatarebuilt-ininthismobile app. To evaluate the effectiveness of TIEs, triangulation of different data setswasadopted,includingexperiencesurveys,qualitativefeedback,clickstreamdataandtextminingofpre/post-traildiscussion.The results indicated that students’ interests in learningaboutissuesofacademicintegrityandethicsinclassroomswerestimulated.

Thisstudy-usingclickstreamdatacollectedfrommobiledevices-presentsanexampleoftheresearch, employed at scale, in the area of learning analytics and shows positive learningevidence.Suchstudiesarelimited(Chanetal.2015)andshouldbeenthusseenasexemplars.“WithenoughevidenceshowingstudentscaneffectivelylearntheconceptsofAIE[academicintegrity and ethics] using the TIEs [Trails of Integrity and Ethics], others in the educationsector,suchassecondaryschoolscanbeencouragedtogetinvolved”(Kwongetal.,2017,p.367).

8. LearninginasocialcontextWithrespecttodigitalisationofdata,learninginasocialcontextpinpointstheemergenceofsocial learning analytics (SLA). SLA emphases the concept of social network analysis onlearningactivitiesandemploysmethodsof learninganalytics to studentgroups insteadofindividualsasascopeforanalysisoflearningactivities(Hayaetal.,2015,p.303).FergusonandBuckinghamShum(2012)suggestfiveapproachestoSLA.

- Sociallearningnetworkanalytics:asuccessfulexampleofthiscategoryistheSNAPP6(SocialNetworksAdaptingPedagogicalPractice)tool,whichisavisualisationtoolforLMSforums.Itcanbeusedtoidentifydisconnectedstudentsandindicatetheextenttowhichalearningcommunityisdevelopingwithinavirtuallearningenvironment.

5Mobxz,https://mightysignal.com/a/ios/6182956476SNAPPtool,https://github.com/aneesha/SNAPPVis

10

- Sociallearningdiscourseanalytics:anexampleistheCoheretool,7whichisawebapplicationprovidingamediumforengaginginstructuredonlinediscourse,orforsummarising/analysingit,e.g.asamoderator,educatororresearcher.Figure2showshowCohereaugmentscommononlinedialoguetext:theiconsatthesideofeachpostshowsrhetoricalroleofthepostandthesemanticconnectionstherhetoricalmovebetweenposts(DeLiddoetal.,2011).

- Sociallearningcontentanalytics:whenviewingonlineresources,SocialLearn’s‘Backpack’–atoolbarofappsandresources–canbeusedonanyInternetsite.TheBackpackcurrentlyincludesthebasiccomponentsofsociallearningcontentanalytics.Itprovides,amongothers,theoptionofviewingthekeywords,hotlinksorimagesconnectedwiththeopenwebpage.Inparticular,whenclickingintheBackpack’slightbulbiconprovidestheoptionofviewingthekeywords,hotlinksorimagesconnectedwiththeopenwebpage(asinthelargeboxontherightofFigure3).Theinformationaboutimagescanbemergedwithvisualsimilaritysearchtoidentifyandrecommendotherresourcesthatmakeuseoftheseimages,

Figure3.TheSocialLearnBackpackopenatthefootofaBBCNewspage,showingalistofimagesonthepage(adoptedfromFerguson&BuckinghamShum,2012)

- Sociallearningdispositionanalytics.Learningdispositionscanbemodelledasamulti-dimensional construct called Learning Power, currently assessed by learner self-report via aweb questionnaire called ELLI (Effective Lifelong Learning Inventory8),whose data warehouse platform supports a range of analytics (Ferguson &Buckingham Shum, 2012). ELLI generates a spider diagram visual analyticwhich isusedtosupportself-reflectionandchange.ELLIhasbeenvalidatedandpilotedattheOpenUniversity(UK;Edwards,2011).

- Implementingsociallearningcontextanalytics:theSocialLearnapprecommendsandprovides access to learningmaterials in response to search terms. The app allowsresourcestoberatedandrecommendedtoindividualsortogroups.Ifuserschoosetomaketheirlocationdataavailable,thiscanbeusedtoinfluencerecommendations;forexample,ifSimonisworkingon‘ClimateChange’theappmightsuggestapodcastoncoastaldefenceswhenhevisitsaseasideresort,andcouldprovideamapshowingalocalsitewhereSimonwouldbeabletoviewtheeffectsoferosion.

7Coheretool,http://cohere.open.ac.uk/8EffectiveLifelongLearningInventory,https://www.elli.global/

11

Figure 4. Dialogues augmented with the Cohere tool

Hayaetal. (2015)demonstratedaSocialLearningAnalytics(SLA)toolkitthatappliessocialnetworkanalysisandcontentanalysistechniquesonforummessagestoanalysecollaborationamong students and to support teacher inquiry. The SLA toolkit was integrated in theJuxtaLearnlearningprocessandplatform(seeFigure5).

Figure5.LearningAnalytics-enhancedJuxtaLearnProcess(adoptedfromHayaetal.2015).

The authors adopted the Teacher-led Inquiry into Student Learning (TISL) pedagogicalapproachandpracticethatencouragesteacherstosystematicallyexaminetheeffectsoftheirlearning design on student performance and learning. Learning Design refers to “theapplication of methods, resources and theoretical frameworks to achieve a particularpedagogicalgoalinagivencontext”(p.302)andfromaLearningDesignpointofview,themain challenge lies in integrating design tools used for lesson planning with tools formonitoringandanalysis.

TheresearchpresentscasestudiesthatexemplifythekeyfeaturesoftheTISLapproachinthecontextofvideocreatedlearningscenarioandthekindofoutcomesthatcanbeobtained.Intheirstudy,Hayaetal.(2015)exploredthepotentialofcomputationalmethodstosupport

12

teachersandlearnersparticipatingincollaborativescenarios.Theresearchersimplementedawebspacethatsupportsnetworksofpeople,contentandservicesbasedontheopen-sourcesocialplatformElgg(https://elgg.org).Thisspaceprovidesobject-orientedprogramming–thechosenforthisstudycourse–relatedcommunicationthroughvideosandensuingdiscussionsand comments. Theplatform supportsuploading, archivingand sharingof learner-createdcontent.Overall,thelearningprocessisgroundedinthestudentreflectioninthethreekeyareas:I)collaborativevideoproduction,ii)discussionandiii)peerevaluation.Discussionandevaluationaredirectlysupportedbythewebspace.

Thecasestudies’resultsshowthattheteacherscanderivevaluableinsightsintothestudents’learningactivitiesfromanalyticsresults.Theauthors(Hayaetal.,2015)exemplifyindicatorsforthequalityofcooperationpresentedbycommunication(seee.g.,Figure6)orportfoliodiagrams(seee.g.,Figure).

Figure6.Discussionnetwork Figure7.Reciprocity

Suchdiagramsallowforintroducingthresholdsforexpectedbehaviorindicatingtheneedforinterventiontoteachershowarenowfamiliarwiththesetypesofdiagrams.Forresearchersand learning designerswho become used to interpreting such diagrams they provide theopportunitytoobtaindetailedinformationaboutirregularitiesintheprocess,whichmayandinsomecases,shouldleadtoadaptations.Thecomplementingapplicationofcontentanalysismethods,hereNetworkTextAnalysishelpedtodetectfrequentmisconceptionsfromstudentcommentswhichinformlearningdesignersandteachersabouttopicsofinterests.Moreover,additionalindicators,suchasthesemanticrichnessmeasure,providethebasesforscaffoldingstudentstokeepfocusedonthedomain-relatedlearningtaskratherthanaimingatproducing‘perfect’ videos. In summary, the toolkit supportsmultiple levels of analysis that providedeeperinsightsintothecollaborativelearningprocess

Berlandetal.(2015)introducedAMOEBA,acollaborationorchestrationtooltosupportprogramminglearningactivitiesinhighschoolbysupportingteacherstopairstudentsbasedonrealtimeanalysesofstudents’programmingprogressions.TheresultsshowedthatusingAMOEBAtohelpwithpairingstudentsresultedinimprovementsinthecomplexityanddepthofthestudent’sprograms.

9. LearninganalyticsandlearningdesignThe interplay between learning analytics and learning design has recently attracted theinterest of researchers mainly for one reason: to check whether there are discrepanciesbetweenthestudents’actualbehaviourandtheteacher’s intentions,andinturn, tomakecorrectiveactions.Forexample,thestudyofNguyenetal.(2018)usedtracedatafromthe

13

Moodle LMS to investigate to what extent students’ timing of engagements aligned withinstructor learning design, and how engagement varied across different levels ofperformance.Theanalysiswas conductedover30weeksusing tracedata (fromanonlinecourseattheOpenUniversity,UK)on387studentsandreplicatedovertwosemestersin2015and2016.

Thedataanalysiswasperformedviavisualisationsandmultilevelmodelling.Tounderstandtowhat extent do students’ timing of engagement align with instructors’ learning designvisualisationswereemployed.Firstly,actuallystudypattersagainstthelearningdesignwerevisualised. Secondly, the study patters for respective individual study materials acrossexcellent, passed, and failed group. The visualisations were conducted using JupyterNotebook9andTableau10.Tocomparestudypattersacrossthreegroupsofperformanceovertime,amultilevelmodelling(ormixed-effectmodelling)approachwasused.Thisapproach,compared to the traditional repeatedmeasure ANOVA approach, allows formissing data,tolerates differently spaced waves of data (e.g., due to Christmas break) and allows fornonlinearrelationships(Nguenetal.,2018,p.144).

Theirfindingsrevealedamismatchbetweenhowinstructorsdesignedforlearningandhowstudents studied in reality. Inmostweeks, students spent less time studying the assignedmaterials on the Moodle platform compared to the number of hours recommended byinstructors.Thetimingofengagementalsovaried,frominadvancetocatchinguppatterns.Highperformingstudents(i.e.,thepassedandtheexcellentgroups)spentmoretimestudyingin advance, while low-performing students spent a higher proportion of their time oncatching-up activities. Towards the end of the course, the gap between failed andpassed/excellentstudentsincreasedconsiderably.

In summary, by comparing and contrasting the assumptions in learning design made byinstructorswithactualstudentbehavior,learninganalyticscouldactasareflectivesourceandprovide actionable feedback.Onepotential implication of this study is if students tend tospendmoretimeoncatchingupaparticular learningmaterial, the instructorscouldcheckwhetherthematerialwasclearlyexplained,andprovideaquickreview.

Rodríguez-Trianaetal.(2015)presentthemonitoring-awaredesignprocess,whichdescribesthe steps that teachers should undertake during the design of computer-supportedcollaborative learning scenarios, inorder to reflecton theirmonitoringneedsandexpressexpectationsaboutstudents’ interaction.Fromalearninganalyticsperspective,monitoringexaminesstudents’interactionsduringtheenactmentofthelearningscenariosandfacilitatestheinterventiononthesituationtowardamoreproductivedirection(Rodríguez-Trianaetal.,2015).Theauthorsadoptthefollowinglearningdesign-awaremonitoringprocess:

1. Collectinteractiondata,guidedbytheunderlyinglearningdesign(e.g.,participants,resources,and/ordeadlines).

2. Createindicatorstoconstructamodelofdesiredinteraction(i.e.,students’interactionswiththelearningsystem)basedonthelearningdesigndefinitionandconstraints,andtheteachers’decisions.

3. Compareactualanddesiredstatesofinteractioni.e.,comparingthegatheredevidencewiththelearningdesigndefinitionandconstraints.

9https://jupyter.org10https://www.tableau.com

14

4. Adviceorguidetheinteraction,byinformingteachersaboutdiscrepanciesabouttheactualandthedesiredstate.

Todeploythestrategymentionedabove,theauthorsusedanumberoftools;inparticular,First,theWeb-Collagetool11tocreatethelearninginadigitalformat;then,theGLUE-PS12tool(i.e.,GroupLearningUnifiedEnvironment-PedagogicalScripting) -usedbytheteachers-deployedautomaticallythedesignintothedistributedlearningenvironments,i.e.,differentexistingVirtual Learning Environments (e.g.,Moodle); and a custom-made third tool, theGroup Learning Interaction Monitor for Pedagogical Scripting Environments (GLIMSE)prototype,totrackandreporttheinteractionstakingplaceintheLMSplatform.GLIMSEtookthescriptinstantiationfilegeneratedbyGLUE-PSandtheXMLversionoftheformstoguidethedatagatheringandanalysis.

In summary,we have provided several relevant examples thatmight be applicable in theSwedishcontext,but tobeable tounderstand furtherhow learninganalyticscan improvedifferentaspectsoflearningprocessandlearningoutcomes,practitionersneedtoconsiderseveral important questions such as:What problems do we want to solve?What learneractivitiesdowewanttosupport?Whatkindofdata,thatwemay,ormaynothavetoday,wouldbevaluabletoanalyse?Howtheavailablemethodsandtoolsthatareapplicableononeeducationalcontextberelevantandusefulinanothercontext?Whatarethechallenges?Arethereanyimportantethicalconsiderationsthatshouldbetakenintoaccount?

ReferencesArnold,K.E.,&Pistilli,M.D.(2012,April).CoursesignalsatPurdue:Usinglearninganalytics

toincreasestudentsuccess.InProceedingsoftheSecondInternationalConferenceonLearningAnalyticsandKnowledge(pp.267-270).ACM.

Berland,M.,Davis,D.,&Smith,C.P.(2015).AMOEBA:DesigningforCollaborationinComputerScienceClassroomsthroughLiveLearningAnalytics.InternationalJournalofComputer-SupportedCollaborativeLearning,10(4),425-447.

Broos,T.,Peeters,L.,Verbert,K.,VanSoom,Langie,G.,&DeLaet,T.(2017).DashboardforActionableFeedbackonLearningSkills:HowLeanerProfileAffectsUse.Proceedingsofthe7thWorkshoponAwarenessandReflectioninTechnologyEnhancedLearning,co-locatedwiththe12thEuropeanConferenceonTechnologyEnhancedLearning.Retrievedfromhttp://ceur-ws.org/Vol-1997/paper7.pdf2018-12-15

Chan,J.,Chiu,R.,Ng,G.,&Kwong,T.(2015).Howclickstreamtrackinghelpsdesignmobilelearningcontent.InternationalJournalofHumanitiesSocialSciencesandEducation,2(7),95–104.

Davies,R.,Nyland,R.,Chapman,J.,&Allen,G.(2015,March).Usingtransaction-leveldatatodiagnoseknowledgegapsandmisconceptions.InProceedingsoftheFifthInternationalConferenceonLearningAnalyticsandKnowledge(pp.113-117).ACM.

DeLiddo,A.,Shum,S.B.,Quinto,I.,Bachler,M.,&Cannavacciuolo,L.(2011).Discourse-centriclearninganalytics.InProceedingsofthe1stInternationalConferenceonLearningAnalyticsandKnowledge(pp.23-33).ACM.

11WebCollage,http://www.ld-grid.org/resources/tools/webcollage12GLUE!-PStool,https://www.gsic.uva.es/glueps/

15

Drachsler,H.,&Kalz,M.(2016).TheMOOCandLearningAnalyticsInnovationCycle(MOLAC):AReflectiveSummaryofOngoingResearchanditsChallenges.JournalofComputerAssistedLearning,32(3),281-290.

Gašević,D.,Dawson,S.,Rogers,T.,&Gasevic,D.(2016).Learninganalyticsshouldnotpromoteonesizefitsall:Theeffectsofinstructionalconditionsinpredicatinglearningsuccess.TheInternetandHigherEducation,28,68-84.

Govaerts,S.Verbert,K.,&.Duval,E.(2011).Evaluatingthestudentactivitymeter:Twocasestudies.InH.Leung,E.Popescu,Y.Cao,R.Lau,andW.Nejdl(Eds).AdvancesinWeb-BasedLearning-ICWL2011,volume7048ofLectureNotesinComputerScience,pp.188-197.SpringerBerlinHeidelberg.

Edwards,C.(2011).Embeddingreflectivethinkingonapproachestolearning-movingfrompilotstudytodevelopinginstitutionalgoodpractice.In:The16thAnnualConferenceofEducation,Learning,Styles,IndividualdifferencesNetwork,29Jun-1Jul2011,UniversityofAntwerp.

Ferguson,R.,&BuckinghamShum,S.(2012).SocialLearningAnalytics:FiveApproaches.InProceedingsofthe2ndInternationalConferenceonLearningAnalyticsandKnowledge,(29Apr-2May,Vancouver,BC).ACMPress:NewYork.Eprint:http://oro.open.ac.uk/32910

Ferguson,R.,&Clow,D.(2017).Whereistheevidence?Acalltoactionforlearninganalytics.Proceedingofthe7thinternationallearninganalyticsandknowledgeconference,pp.56-65.ACM.

Jayaprakash,S.,Moody,E.,Lauria,E.,Regan,J.,&Baron,J.(2014).Earlyalertofacademicat-riskstudents:anOpenSourceAnalyticsInitiative.JournalofLearningAnalytics,1(1),6-47.

Haya,P.,Daems,O.,Malzahn,N.,Castellanos,J.,&Hoppe,H.(2015).AnalysingContentandPatternsofInteractionforImprovingtheLearningDesignofNetworkedLearningEnvironments.BritishJournalofEducationalTechnology,46(2),300-316.

JICS(2017).Learninganalyticsinhighereducation:AreviewofUKandinternationalpractice.CaseStudyA:Trafficlightsandinterventions:SignalsatPurdueUniversity.Retrievedfromhttps://analytics.jiscinvolve.org/wp/files/2016/04/CASE-STUDY-A-Purdue-University.pdf2018-12-06

Kickmeier-Rust,M.,Hillemann,E.,&Albert,D.(2014).Gamificationandsmartfeedback:Experienceswithaprimaryschoollevelmathapp.InternationalJournalofGame-BasedLearning,4(3),35-46.

Kovanovic,V.,Gasevic,D.,Dawson,S.,Joksimovic,S.,Baker,R.,&Hatala,M.(2015).Doestime-on-taskestimationmatter?Implicationsforthevalidityoflearninganalyticsfindings.JournalofLearningAnalytics,2(3),81-110.

Kwong,T.,Wong,E.,&Yue,K.(2017).Bringingabstractacademicintegrityandethicalconceptsintoreal-lifesituations.Technology,KnowledgeandLearning,22(3),353-368.

Lang,C.,Siemens,G.,Wise,A.,&Gasevic,D.(Eds;2017).HandbookofLearningAnalytics(firstedition),DOI:10.18608/hla17

Leony,D.,Merino,P.J.M.,Valiente,J.A.R.,Pardo,A.,&Kloos,C.(2015).DetectionandEvaluationofEmotionsinMassiveOpenOnlineCourses.JournalofUniversalComputerScience,21(5),638-655.

Long,P.,&Siemens,G.(2011).Penetratingthefog:Analyticsinlearningandeducation.EducauseReview,46(5),31-40.

16

McNamara,D.,Allen,L.,Crossley,S.,Dascalu,M.,&Perret,C.A.(2017).Naturallanguageprocessingandlearninganalytics.Handbookoflearninganalytics,93-104.Retrivedfromhttps://solaresearch.org/wp-content/uploads/2017/05/chapter8.pdf2018-09-28

Misiejuk,K.,&Wasson,B.(2017).StateoftheFieldReportonLearningAnalytics.Onlineat:http://bora.uib.no/bitstream/handle/1956/17740/SoF%20Learning%20Analytics%20Report.pdf?sequence=1

Millecamp,M.,Gutiérrez,F.,Charleer,S.,Verbert,K.,&DeLaet,T.(2018,March).Aqualitativeevaluationofalearningdashboardtosupportadvisor-studentdialogues.InProceedingsofthe8thInternationalConferenceonLearningAnalyticsandKnowledge(pp.56-60).ACM.

Nguyen,Q.,Huptych,M.,&Rienties,B.(2018).Linkingstudents'timingofengagementtolearningdesignandacademicperformance.InProceedingsofthe8thInternationalConferenceonLearningAnalyticsandKnowledge(pp.141-150).ACM.

Reich,J.(2014,December8).MOOCcompletionandretentioninthecontextofstudentintent.EDUCAUSEReviewOnline.http://er.educause.edu/articles/2014/12/mooc-completion-and-retention-in-the-context-of-student-intent

Rodríguez-Triana,M.,Martínez-Monés,A.,Asensio-Pérez,J.,&Dimitriadis,Y.(2015).Scriptingandmonitoringmeeteachother:AligninglearninganalyticsandlearningdesigntosupportteachersinorchestratingCSCLsituations.BritishJournalofEducationalTechnology,46(2),330-343.

Roll,I.,&Winne,P.H.(2015).Understanding,evaluating,andsupportingself-regulatedlearningusinglearninganalytics.JournalofLearningAnalytics,2(1),7–12.

Schwendimann,B.,Rodríguez-Triana,M.,Vozniuk,A.,Prieto,L.,Boroujeni,M.,Holzer,A.,Gillet,D.,&Dillenbourg,P.(2016).Understandinglearningataglance:anoverviewoflearningdashboardstudies.LAK.Retrievedfromhttps://infoscience.epfl.ch/record/216917/files/2016___LAK___Dashboard_systematic_review__Poster_.pdf2018-12-15

Simsek,D.,BuckinghamShum,S.,Sandor,A.,DeLiddo,A.,&Ferguson,R.(2013).XIPDashboard:visualanalyticsfromautomatedrhetoricalparsingofscientificmetadiscourse.In:1stInternationalWorkshoponDiscourse-CentricLearningAnalytics,8Apr2013,Leuven,Belgium.

Tabuenca,B.,Kalz,M.,Drachsler,H.,&Specht,M.(2015).TimewillTell:TheRoleofMobileLearningAnalyticsinSelf-RegulatedLearning.Computers&Education,89,53-74.

Viberg,O.,Hatakka,M.,Bälter,O.,&Mavroudi,A.(2018).TheCurrentLandscapeofLearningAnalyticsinHigherEducation.ComputersinHumanBehavior,89,98-110.