SHEILA policy framework: informing ins9tu9onal strategies and pol-icy processes of learning analy9cs

Yi-ShanTsai TheUniversityofEdinburgh

OldCollege,SouthBridge,EdinburghEH89YL UK

yi-shan.tsai@ed.ac.uk

PedroManuelMoreno-Marcos UniversidadCarlosIIIdeMadrid AvenidaUniversidad,30,28911Le-

ganes,Madrid Spain

pemoreno@it.uc3m.es

KairitTammets TallinnUniversity

Narvamnt25,10120Tallinn Estonia

kairit@tlu.ee

KaireKollom TallinnUniversity

Narvamnt25,10120Tallinn Estonia

kairit@tlu.ee

DraganGasevic TheUniversityofEdinburgh

OldCollege,SouthBridge,EdinburghEH89YLUK

MonashUniversity Scenic Blvd, Clayton VIC 3800

Australiadragan.gasevic@ed.ac.uk

ABSTRACTThispaperintroducesalearninganalyticspolicydevelopmentframework developed by a cross-European research projectteam – SHEILA (Supporting Higher Education to IntegrateLearningAnalytics),basedoninterviewswith78seniorman-agers from51Europeanhighereducation institutionsacross16countries.TheframeworkwasdevelopedusingtheRAPIDOutcomeMappingApproach(ROMA),whichisdesignedtode-velopeffectivestrategiesandevidence-basedpolicyincomplexenvironments.Thispaperpresentsthreecasestudiestoillus-trate the development process of the SHEILA policy frame-work,whichcanbeusedtoinformstrategicplanningandpol-icy processes in real world environments, particularly forlarge-scaleimplementationinhighereducationcontexts.

CCSCONCEPTS• Security and privacy~Social aspects of security and privacy • Applied computing~Education

KEYWORDSLearning analytics, policy, higher education, strategy, ROMAmodel

ACMReferenceformat:Y.-S.Tsai,P.M.Moreno-Marcos,K.Tammets,K.Kollom,andD.Gasevic.2018.SHEILApolicyframework:informinginstitutionalstrategiesandpolicy processes of learning analytics. InProceedings of the Interna-tional Conference on Learning Analytics and Knowledge, Sydney, Aus-tralia, March 2018 (LAK’18), 11 pages. DOI:https://doi.org/10.1145/3170358.3170367

1 INTRODUCTIONANDBACKGROUNDHigherEducationInstitutions(HEIs)areconstantlycollect-

inglargeamountsofdataintheformofstudents’digitalfoot-printsduringtheirstudies.AlthoughHEIsstrivetoincreasethequality of teaching and learning by exploiting the collecteddata,thereareoftenbarriersthatpreventdatafrombeingusedsystematicallyandeffectively.Forexample,dataquality,own-ershipandaccess,organisationalculture,andexpertiseavaila-bletoimplementlearninganalytics(LA)areprevalentissuesthatneedtobeaddressedbeforeimplementation[4].Accord-ingtoFergusonandothers[10],althoughfundingopportuni-tiesforLAresearchandactivitieshaveincreased,thereisstillalackofsystematicandlarge-scaleimplementationsofLAinhighereducation.ThepreliminaryfindingsofaEuropeanpro-ject–SHEILA(SupportingHigherEducationtoIntegrateLearn-ingAnalytics)havedemonstratedthatnumerousHEIs inEu-ropeareeitherobservingthedevelopmentofLAorhaveen-gagedwithitpracticallywithoutadefinedstrategyormonitor-ingframeworktoensuretheeffectivenessandlegitimacyofLA

Permissiontomakedigitalorhardcopiesofpartorallofthisworkforper-sonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationonthefirstpage.CopyrightsforcomponentsofthisworkownedbyothersthanACMmustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orrepublish,topostonserversortoredistributetolists,requirespriorspecificpermissionand/orafee.RequestpermissionsfromPermissions@acm.org.LAK'18,March7–9,2018,Sydney,NSW,Australia©2018AssociationforComputingMachinery.ACMISBN978-1-4503-6400-3/18/03…$15.00

LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.

2

practices [30]. Drachsler and Greller identified uncertaintiesamonginstitutionsaboutlegalboundariesandethicallimitsre-gardingtheuseofpersonaldataforLA,inadditiontoprevalentfearofnegativeconsequencesfromtheapplicationofLA[8].Asaresult,theyproposedtheDELICATEchecklisttorebrandtheprivacyburdenthattheLAcommunitycarriedwithaqualitylabel.OtherfamousmodelsthatexisttoguidetheadoptionofLAcanbefoundinJisc’sCodeofPractice[5]andtheOpenUni-versity’s“PolicyonEthicaluseofStudentDataforLearningAn-alytics” [28]. However, these ethical and privacy guidelinesmaynotalwaysapplytoeveryinstitution’sownuniquecon-text.Inlightoftheneedforasoundpolicythatistailored`tomeetindividualinstitutions’uniquecontextsandensuresare-sponsibleandeffectiveuseofstudentdataforLA,theSHEILAprojectwaslaunchedwiththegoaltoassistHEIstodevelopin-stitutionalpoliciesforLA.Todoso,theprojectwillproduceapolicyframework(addressedastheSHEILApolicyframeworkhereafter)byengagingendusersofLAdirectlytounderstandtheir perceptions, expectations and concerns, as Knight andothers[15]havesuggestedthatusersareinthemostaccuratepositiontoidentifytheirownneedsandtoindicatehowtheirpracticescanbesupportedandimprovedbeforesolutionsaredesignedandimplemented.[20].Withdatacollectedfromthedirect engagement with stakeholders, the project team hasusedtheRAPIDOutcomeMappingApproach(ROMA)toscopeexistingpracticesofLAamongHEIs inEurope,and tomakesuggestionsforpolicydevelopment.AlthoughtheliteraturehassuggestedthatROMAmodelisaneffectivetooltosupportsys-tematicadoptionof learninganalytics inHEIs[10,17], therehas been limited work that purposively involved differentstakeholdergroupstovalidatethefeasibilityofthistoolforLApolicydevelopment.Thecontributionofourworkistobridgethisgap,andextendtheuseoftheROMAmodeltoaddresschal-lenges recognised in the literature and raised by differentstakeholdergroups.WhilethefinalproductoftheSHEILApolicyframeworkwill

reflecttheperspectivesofvariousstakeholders,includinginsti-tutionalleadersanddecisionmakers,teachingstaff,students,andLAexperts,thispaperwillfocusonthefirstSHEILApolicyframework,whichwasdevelopedbasedon64interviewswithsenior managers from 51 European HEIs. Considering thescopeofthepaper,wewillpresentthreerepresentativecasestoillustratetheconceptoftheframework,aswellaspotentialwaystouseitforinstitutionalstrategicplanningandpolicyfor-mationforLA.

2 LITERATUREREVIEWIn spiteof thepotential toprovidebetter informationaboutstudent learningbehaviourandprogress, thereby improvingthequalityofeducationalofferingsandoptimisinglearning,LAhasmetanumberofchallengesthatimpedeitsadoptionataninstitutionallevel.Themostfrequentlyidentifiedissuesare(1)thedemandonresources,(2)issuesofethicsandprivacy,and(3)stakeholderengagementandbuy-in.Thesechallengesneed

to be tackled through strategic planningand a sound policyframework.Inthissection,weoutlineissuesidentifiedintheliterature under the three themes and introduce the ROMA(RAPID Outcome Mapping Approach) model, on which theSHEILApolicyframeworkisbased.

2.1 LearningAnalyticsChallenges2.2.1 DemandonResources.Thefirstmainissuecoverschal-lengesassociatedwithdataandtechnologicalinfrastructure,fi-nancialresources,andhumanresources.TheimplementationofLAtypicallyinvolvescomplexcomputingandaggregatingoflargeamountsofdata,inadditiontomanagementchallenges,suchastheintegrationofresearchtoolsintoexistinglearningenvironments[13].Thesetaskscanbedifficulttoperformwithtraditionaldatamanagementtechnologies[14].Asurveycar-riedoutbyEDUCAUSEtoinvestigateanalyticslandscapesinUShighereducationrevealedthatdata-qualityconcernsandsys-tem-integrationdifficultieswerepartofthemajorchallengestoembeddingtheuseofLAintoinstitutions[3].Thesefindingssuggestthatthere isaneedfora financial investment inad-vancing institutional data infrastructure to enable LA. How-ever,thesamestudybyEDUCAUSEalsofoundthatLAremainsaninterestratherthanamajorpriorityatmostinstitutions[3].Thisfindinghighlightsthechallengeofobtainingsufficientfi-nancialsupporttodevelopatechnologicalenvironmentforLAor appointinganalytics specialists inmanyHEIs if LAhas tocompetewithother institutional priorities. For example, an-otherEDUCAUSEreportbasedonthesamesurveydatapointedout that institutional analytics was twice as likely to be de-scribedasamajorpriorityaswaslearninganalytics,and4in10institutionsreportedlittleornoinvestmentinlearningan-alytics[32].Anotherkeydimensionishumanresources,whichincludes

boththeavailabilityofstafftimeandexpertisethatisrequiredtoimplementLA.Inacomplexeducationalsystem,theintro-ductionofasubtlechangecanmeetsubstantialresistancebe-causeoftheperceivedincreaseinworkloadforstaff[17].AsLAmakesuseofdatafromvarioussources,institutionsnotonlyneeddataexpertstoobtainandanalysegoodqualitydata,buttheyalso need theusers (e.g., administrators, teaching staff,andstudents)tohavebasicdatainterpretationskillsandtheabilitytoreflectondatacritically,inorderthatLAmayhavepositiveimpactoninformingdecisionsandchangingbehaviour[2,19,31].Thishasbeenidentifiedasacommongapbetweenneedsandsolutionsininstitutionalanalyticscapacity[18,25].

2.2.2 IssuesofEthicsandPrivacy.The secondmain issuehas been identified as amajor obstacle to gain buy-in fromstakeholders, especiallywhen the collection and use of dataseemtoriskintrudingprivacy[23,27].LikeallBigDataappli-cations,LAreliesonconstantandubiquitouscollectionofdatafromstudents.Thewiderangeandtypesofdatacollectedcouldinducediscomfortamongdatasubjectsduetoasenseofsur-veillance,leadingtoresistancetoLA[19].Moreover,whilean-

SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia

3

onymity policies are commonly enforced inHEIs when per-sonaldataisused,itcanbedifficulttodelivercustomisedin-terventions without retaining a certain degree of individuallinkages [24]. Similarly,Greller andDrachsler acknowledgedthedilemmabetweenkeepingdataanonymousandexploitingthemostvalueofdata[12].Theyalsoarguedthatfearinducedbyethicsandprivacyissuescaneasilyleadtomisunderstand-ingsanddistrustininstitutions[8].Anotherkeyissueassociatedwithethicsandprivacyisin-

formedconsent [26].Rubeland Jonesquestion theextent towhich students canmake informedconsent [24].Theypointout thateducational institutionsmaybe transparent in theirdatapractices,butthecomplexityofalgorithmsstillmakesan-alytics a ‘black box’ formany.Moreover, the inherent infor-mationasymmetriesbetweendatacollectorsanddatasubjectsmeanstudentstendtohavelimitedknowledgeaboutwhocanaccesstheirdata,whattheydowiththedata,andwhatconse-quences intrusionsofprivacymaybe [8]. Similarly,PrinslooandSladeareconcernedaboutthebesttimetoseekconsentfromstudents.Theysuggestthatconsentseekingshouldfocusondownstreamusersratherthanonthetimeoftheinitialcol-lectionofdata,becausethebenefitsofopting-inoroutmaynotbeapparentat themomentwhenaLA service is introduced[22].Theconflictsbetweenmaximisingtheefficiencyandeffi-cacyofLAandrespectingdatasubjects’rightstocontroltheirown data can be challenging to institution adopting LA at alargescale.

2.2.3 StakeholderEngagementandBuy-in.Thethirdmainissue has been highlighted in a systematic literature reviewwhereTsaiandGaševićpointedoutthatHEIsstruggletofindcommongroundsamongdifferentstakeholdersregardingtheadoptionofLA,duetodiscrepanciesinexistingexperienceandknowledgeofdata,thereforeresultingindifferentunderstand-ingofpossiblebenefitsandoutcomesofLA[29].Moreover,ac-cordingtoTsaiandGašević,onlyahandfulofstudieshavetriedtoexplorestudentperspectivesregardingtheuseoftheirdataforlearninganalyticsortheimpactontheirlearningjourneys,despitethefactthatLAchampionsforalearningenvironmentthatislearner-centredandlearner-concerned[11].Thediffer-encesinperceptionsofLAamongstakeholderscanleadtoun-equalbuy-iniftheirneedsarenotmet,furtherresultingindis-trustinLAifconcernsarenotaddressed.Forexample,PrinslooandSladespecificallycalledforresearcherstoexplorepoten-tialconflictsbetweenstudents’concernswiththeirrighttoopt-outandtheimplicationsofpersonal-levelinterventionsfromHEIs[21].Adirectimpactofunequalengagementwithteachingpro-

fessionals is theweakpedagogicalgroundingofLAtechnolo-giesandimplementationdesign.Forexample,AliandotherspointedoutthatLA tools still needed tomovefromspottingstudents at risk toprovidingpedagogically informedsugges-tions[1],andMacfadyenandDawsonsuggestedthat institu-

tionsshouldbalancesolvingtechnicalchallengesanddevelop-ingpedagogicalplans[16].Similarly,FergusonandcolleagueshighlightedthatmuchworkonLAhasconcentratedonthesup-plyside,andconsiderablylessonthedemandside,forexampleconnectingLAwitheducationinwaysthatcantrulysupporttheeverydaylearning,teachingandassessmentwork[9].Fail-ingtoconsiderthepedagogicalcontextinwhichdataisgener-atedandinterpretedwillaffectteachingstaff’sperceptionsofthe usefulness of LA, thereby impeding broader buy-in andscalableactionsofLA[25].Thephenomenonofunequalengagementwithstakeholders

isalsoreflectedbytheabsenceofclearleadershiptodefinedi-rectionsforLAadoptionamongmanyHEIs[13],whichiscon-sideredakeyfactorassociatedwiththematurityofLAprac-ticesatan institutional level[6,18,25]. Inparticular, thein-volvementofinstitutionalleadersiscrucialtothedevelopmentofstrategiesandpoliciesforLA,whichcouldhelpmitigatethechallengesidentifiedsofar.Asnewpracticesinacomplexedu-cational system potentially disrupt traditional managementandorganisationalstructures,andthereforelikelytomeetre-sistance [17], it has been suggested that institutions shouldstartLAimplementationbydefiningastrategicplan[2,7,10].Moreover,studieshaveidentifiedthatexistingpoliciesrelatedtotechnicalstandardsforinteroperabilitydonotfullyapplytoLApractices[9],andtailoredLApoliciesforindividualinstitu-tionswillbeneededinordertoproperlyconsiderindividualinstitutionalcontextsineveryphaseofadoption[29].Withoutdedicatedinputfromhigh-leveldecisionmakers[7],itcanbedifficulttopressforthedevelopmentofLAspecificstrategiesandpoliciesthatmeettheneedsofindividualinstitutionsandthememberstherein.Inresponsetotheneedforastrategicframeworkandpolicy

toadoptLAsystematically,theSHEILAprojectusedtheRAPIDOutcomeMappingApproach(ROMA)toproduceapolicyde-velopment framework. The ROMAmodel was adopted as afoundation for thedevelopmentof theSHEILApolicy frame-workduetotheoriginalpurposeofROMAtosupportevidence-basedpolicydevelopmentandchangethroughactiveengage-mentwithrelevantstakeholders.ThemodelhasalreadybeensuggestedforsystemicadoptionofLAinHEIs[10,17].Thefol-lowingsubsectionintroducestheconceptoftheROMAmodel.

2.2.TheROMAModelinLearningAnalyticsCon-texts

TheROMAmodelwasdesignedbytheODI(OverseasDevelop-mentInstitute)toinformpolicyprocessesinthefieldofinter-nationaldevelopmentusingresearchevidence[33].Themodelbeginsbydefininganoverarchingpolicyobjective,whichisfol-lowedbysixstepsdesignedtoprovidepolicymakerswithcon-text-basedinformation:1)mappoliticalcontext,2)identifykeystakeholders,3)identifydesiredbehaviourchanges,4)developengagement strategy, 5) analyse internal capacity to effectchange,and6)establishmonitoringandlearningframeworks.

LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.

4

Unlike traditional linear tools and approaches, ROMA is de-signed tobeused iteratively to informstrategic choicesandmeetunexpectedchanges(orchallenges)inacomplexsetting.Thismodelhasbeenadaptedtoguidetheplanningandimple-mentationofLAataninstitutionallevel[10,17](Figure1).

Figure1:TheRAPIDOutcomeMappingApproach[10]

Fergusonand colleaguesprovided twocase studiesofLApracticefromtheUKandAustraliatodemonstratehowtheo-reticalframeworkscouldbeoperatedintherealworldand,inparticular,howROMAcouldbeusedfortheplanningandim-plementationofLAinhighereducationcontextstomaximisethesuccessandimpactofLA.OurworkbuildsontheapproachadoptedbyFergusonandothers[10]tomapoutthestateofLAadoptionamongHEIsinEuropeusingROMA,andfurtherpro-videssuggestionstoguidepolicydevelopment.ThefollowingsectionaccountsfortheabovementionedmethodsadoptedtodeveloptheSHEILApolicyframework,followedbythreecasestudiesthathavecontributedtothispolicyframework.

3 METHODOLOGYTheSHEILApolicyframeworkwillbebasedonevidencefromawiderangeofdataincludinganinstitutionalsurveyadminis-teredtouniversitiesinEuropetounderstandthestateofadop-tion of LA (n=46), a Group Concept Mapping activity thatsoughtopinionsfromLAexpertsonessentialfeaturesofaLApolicy(n=30),64 institutional interviewswithmostly seniormanagers(e.g.,provosts,rector,deans,principals,viceprinci-pals,andvice/pro-vicechancellor)from51highereducationinstitutionsacross16countriesinEurope,andlocalconsulta-tionswithteachingstaffandstudentsatfourEuropeanhighereducationinstitutionsusingasurveymethodandafocusgroupmethod. The SHEILA policy frameworkwill be developed inphasesbasedonthefindingsfromtheabovementioneddata.Thispaperwillfocusontheoutputofthefirstphasedevel-

opment. The first SHEILA policy framework was developedbasedontheresultsofananalysisof64institutionalinterviewsthattookplacebetweenAugust2016andFebruary2017.Eachoftheseinterviewslastedfor30to60minutes.Thenumberofparticipants ineach interviewrangedfrom1to3, and someparticipantsfromthesameinstitutionattendedtheinterviewsseparately.Thisresultedinatotalnumberof78participants

from51institutions.Teninterviewquestionsweredevelopedtoinvestigate1)institutionalplansforLA,2)motivationsforLA,3)adoptedstrategy,4)strategydevelopmentprocesses,5)readinesspreparations,6)successandevaluation,7)successenablers,8)challenges,9)ethicalandprivacyconsiderations,and10)theinterviewee’sviewsofessentialelementsinaLApolicy.WeusedtheROMAmodelasatooltoanalyseeachinstitu-

tionalcasebymappingouttheirLArelatedactivitiesandchal-lengestothesixkeydimensionsofROMA.Duringthisprocess,we identifieda strong connectionbetween the sixROMAdi-mensions.Thatis,thesamechallengemaybeidentifiedinmul-tipledimensions,andanactionmaybeinformedbyconsidera-tionofmultipledimensionsatthesametime.WhiletheROMAmodelshouldbeappliediteratively,theredoesnotseemtobea definite order between the dimensions. Therefore, we de-cidedtotreatthemas ‘dimensions’ratherthan ‘steps’as ini-tially suggested by Young and Mendizabal [33], so as toacknowledgethefluiditybetweenthesixdimensions.Wesynthesisedthemappingresultsofthe51casesandcre-

atedacomprehensivetableofallactionsandchallengesidenti-fiedintheinterviews.Thisprocessresultedinalistof42actionpoints and 59 challenges across the six ROMA dimensions.Basedonthisresultandtheinterviewees’responsestoQues-tion10,wegenerated47policyquestionstoaddressthekeyactions and challenges. Thus, the SHEILA policy frameworkconsistsofacomprehensive listofadoptionactions,relevantchallengesandpolicyprompts,framedinthesixROMAdimen-sions. Figure 2 explains the concept and structure of theSHEILApolicyframework.

Figure2:TheSHEILApolicyframeworkstructure

Wegroupedtheactionpoints,challenges,andpolicyques-tionsbycommonthemesincludingcapabilities,culture,ethics&privacy,evaluation,financial&humanresources,infrastruc-ture,internal&externalsupport,management,methodology,purpose,andstakeholderengagement.Thesethemeshelpedus

SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia

5

to identifythemainfocusofactionineachROMAdimensionandprevalentissuestoaddress.Thefollowingsectionsdiscussthemappingresultsofthree

casesthataredifferentfromeachotherbyinstitutionalsize,lo-cation,goals,andapproachestoLA.Whilethedatapresentedbelowonlymakesuppartofourpolicyframework,ourinten-tionistousethemtoillustratethedevelopmentprocessoftheSHEILApolicyframework,andtodemonstratehowtheSHEILApolicyframeworkcouldbeusedtoguidethedevelopmentofinstitutionalpoliciesandstrategicplanningforLA.

4 RESULTSInthissection,wepresenttheactionpointsundertakenbythethreeselectedinstitutionsandthechallengesthattheyfaced,followedbyalistofquestionstoreflectonwhendevelopingaLApolicyinsimilarcontexts.Eachofthestatementsisassoci-atedwithatheme.Section4.1presentstheprofilesofthethreecases, includingtheir approaches toLA. Section4.2presentsthemappingresultsofthethreecasesusingtheROMAmodel.

4.1 ThreecasesInstitutionAisbasedintheUKandhasmorethan30,000stu-dentsenrolled.Atthetimeoftheinterview,institutionAhadonecentraluniversitysponsoredLAprojectandanumberofsmallprojectsinitiatedbyindividualteachingstaff.Intermsoftheinstitutionaluptake,institutionAtookanexperimentalap-proachtoLA.Thatis,LAwasadoptednotasatooltosolveiden-tifiedproblems,butasatooltoexplorenewpossibilitiesandinnovations to enhanceexisting practice. Institution A’s goalwastouseLAtoenhancecurriculumdesignandstudentexpe-rience.InstitutionBisbasedinEstoniaandhasmorethan10,000

students enrolled.This institutionhada fewcourse-levelLAprojectspreviously,andwaspreparinganinstitutionalLApro-jectatthetimeoftheinterview.InstitutionBtookaproblem-basedapproachtoLA,whichisperceivedasapotentialsolu-tiontodealwithstudentdropouts.Thegoalwastounderstandstudents’ learning progress and provide interventions whenneeded.InstitutionCisbasedinSpainandhasmorethan30,000stu-

dentsenrolled.Atthetimeoftheinterview,institutionCdidnothaveanyinstitutionalLAproject,althoughthereweresmall-scaleprojectscarriedoutbyindividualresearchers.Themaingoaloftheseprojectswastoexploredatacollectedfromcur-rentandpastcoursestoidentifyopportunitiesforteachingin-novations.

4.2 SixROMAdimensions

Ananalysisof thethree casesusing theROMAmodel showsthatthemostcommonthemesofchallenges identified inDi-mension2(stakeholders)areethicsandprivacyrelatedissues,whilethoseinDimension3(desiredchanges),4(engagementstrategy),and6(monitoringframework)aremethodologyre-lated.Dimension5(capacityforchanges)examinedtheinter-nalcapacityoftheinstitutions,resultinginalongerlistofchal-lengesbeingidentifiedcomparedtotheotherdimensions.Thecommonchallengesinthisdimensionarerelatedtoculture,ca-pability,andinfrastructure.Incontrast,themappingofDimen-sion 1 (political context) did not identified shared themesamongthecomparativelyshorterlistofchallenges.Thefollow-ing subsectionsareorganisedaccording to the sixROMAdi-mensions.EachsectionbeginswithacriticalreflectiononthestateofadoptionofLAamongthethreecases,followedbythreetablesprovidingfurtherinformationoncorrespondingactions,challenges,andpolicypromptsrespectively.ThesetablesalsopresentaselectivepartoftheSHEILApolicyframework,asil-lustratedinFigure2.

4.2.1 Dimension 1 – Map political context ThemappingofDimension1revealedinstitutionaldriversandneedsforLA.BothCaseAandBfacedexternalpressuretoper-formquality evaluation,whichusually formspartof thekeyperformanceindicators(KPI)inHEIs(Table1).Therefore,itisparticularly important for these institutionsto reflecton thereasonsforadoptingLA–whetheritisforthebenefitsoftheinstitutionorforlearnersandteachers(Table3).WhileLAac-tivitiesinCaseCwerestillatagrass-rootlevel,thesamepolicyquestionswouldbeusefultoreflectonwhenplanningastrate-gicmovementtowardsinstitution-leveladoption.Thatis,alignindividual-level research activitieswith thewider universitystrategy,soastogainsupportfromseniormanagers/decisionmakers.TheneedtogainsupportfromkeyleadershiptoenablesystematicadoptionofLAhasalsobeenconfirmedbytheiden-tifiedchallenges(Table2)

Table1:Mappoliticalcontexts-actions

Case Action ThemeA Theinternaldriverwastousedatatoinform

teachingandlearningrelateddecisions,andanexternaldriverwastoprovidedataforaudits(e.g.NationalStudentSurvey).

Pur-pose

Giventhesizeoftheuniversity,itwasdecidedthatapilotstudywasneededtofindthebestwaytoextractandintegratedata.

Meth-odology

B The internal driver was to increase teachingqualityandlearningmotivations.Theexternaldriverwastoprovidedataforstate-levelqual-ity evaluations, which had previously high-lightedtheproblemofstudentdropouts.

Pur-pose

LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.

6

C Akeydriverwastogainbetterunderstandingofcourserelatedactivitiessoastoimprovethecurriculumdesign.

Pur-pose

Table2:Mappoliticalcontexts-challenges

Case Challenges ThemeA Nochallengeswereidentified. N/AB Thereisnocentralguidancefromthegovern-

mentregardingtheuseofstudentdatainuni-versityfeedbacksystems.

Man-age-ment

C Decentralised leadership made it difficult totakeacentralisedapproachtoLA.

Meth-odology

Table3:Mappoliticalcontexts-policyprompts

Policy–questionstoreflecton ThemeWhatarethereasonsforintroducingLAtostudentsandstaff?Howdoinstitutionalobjectivesalignwithpersonalbenefitsforteachingstaffandstudents?

Pur-pose

4.2.2 Dimension 2 – Identify key stakeholders ThemappingofDimension2showedthattheadoptionofLAinthethreecasesinvolvedawiderangeofstakeholders,bothin-ternallyandexternally(Table4).Akeyimplicationforpolicyistoconsidertheresponsibilitiesandrightsofeveryoneinvolved,inadditiontotheimpactonthem(Table6).CaseB,inparticu-lar,facedanethicaldilemmaabouthowtomakeopt-outop-tionsavailablewhile addressing institutional challenges thatinvolveeverymemberoftheinstitution(Table5).Whilethereis no easy solution for this challenge, defining the circum-stancesofenforcingopt-out/-inoptions,anonymity,andlim-itedaccesstodatainapolicycaneffectivelyminimiseconflicts.Incontrast,CaseCwasconcernedaboutdatare-identification,whichwouldneedtobeaddressedbyevaluationactioninDi-mension6(seeSection4.2.6).Animplicationofthischallengeforpolicyistodefinerulesaboutsharingdatawithresearchersandexternalparties.

Table4:Identifykeystakeholders-actions

Case Action ThemeA The primary internal stakeholders included

students,teachingstaff,seniormanagersandaworkinggroupmadeofrepresentatives fromvariousunits.TheexternalstakeholderwasaLAserviceproviderthatofferedawarehouseandanalyticsexpertise.

Stake-holderengage-ment

B The primary internal stakeholders includedstudents,teachingstaff,ITofficers,seniorman-agers,andthedepartmentofacademicstudies.

Stake-holder

The need to involve external stakeholders,such as LA experts and data scientists, wasidentified.

engage-ment

C Themain stakeholderswereresearchersandIT officers. However, there was indirect en-gagement with external researchers throughthe engagement of LA literature and confer-ences.

Stake-holderengage-ment

Table5:Identifykeystakeholders-challenges

Case Challenges ThemeA It was difficult to define ownership and re-

sponsibilities among professional groupswithintheuniversity.

Man-age-ment

B Theprovisionofopt-outoptionsconflictswiththegoaltotackleinstitutionalchallengesthatinvolveallinstitutionalmembers.

Ethics&Privacy

C Anonymiseddatacouldpotentiallybere-iden-tifiedwhenmatchedwithotherpiecesofdata.

Ethics&Privacy

Table6:Identifykeystakeholders-policyprompts

Policy–questionstoreflecton ThemeWhoisthepolicyfor?Howwillresponsibilitiesbedefinedforeachstake-holder?

Stake-holderengage-ment

Whosedatawillbecollected? Meth-odology

Howwillconsentbeobtained?Isthereanoptiontoopt-outof(oroptinto)anydatacollectionandanalysis?Whocanaccessthedata?Howwill anonymitypoliciesbeappliedto thepro-cessingandpresentationofdata?Willdatabesharedwithresearchers?Willdatabesharedwithexternalparties?Isitjustifi-able?

Dataman-age-ment

4.2.3 Dimension 3 – Identify desired behaviour changes ThemappingofDimension3showedthattheexpectedchangesforCaseBwereparticularly ‘institution-focused’,whilethoseidentifiedinCaseCwereteacher-focused(Table7).AlthoughCaseAexpectedtoseebehaviourchangesamongallthreelev-elsofstakeholders,therewasaconcernthatexpectationsmaynotbemet(Table8).Asimilarconcernaboutreturnsoninvest-mentwasobservedinCaseBwhereLAwasalsodrivencen-trallybytheinstitution.Therefore,itisimportantthatthepol-icynotonlyguidesdecisionmakersto focusonchangesthatmeaningfullyreflectthegoalssetoutforLA(Table9),butalsoarangeofindicatorsthatcantrulyreflectthesechanges ina

SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia

7

specificinstitution’scontext.Thelattercouldbedefinedassuc-cessindicators,assuggestedlaterinDimension6(seeSection4.2.6).

Table7:Identifydesiredbehaviourchanges-actions

Case Action ThemeA Academic staff will better understand stu-

dents’ learning problemsand offer supportaccordingly.Studentswillbeabletoreflectonhowtheylearn,andmakelearningplansaccordingly.Theinstitutionwillbeabletomakebetterde-cisions to support learning and teachingbasedonanoverviewoflearningandteach-ingeffectiveness.

Purpose

B Studentdropoutrateswilldecrease.Students will be provided with regular re-portsabouttheirlearningprogress.Theinstitutionwillmakebetterdecisionstoenhanceteachingqualityandkeepstudentsmotivated.

Purpose

C Academic staff will better understand stu-dent learning behavior, thereby improvingthewaytheyteach.The institution will improve the quality oftheireducationalservices.

Purpose

Table8:Identifydesiredbehaviourchanges-challenges

Case Challenges ThemeA Anexperimentalapproachissusceptibletoa

senseofuncertaintyaboutthereturnonin-vestment.

Method-ology

B It is unclear if a problem-based approachguaranteesasolution.

Method-ology

C Nochallengeswereidentified.

Table9:Identifydesiredbehaviourchanges-policyprompts

Policy–questionstoreflecton ThemeWhatchangeswillLAbringtothecurrentsituation?Whyarethesechangesimportanttous?

Purpose

Whowillbenefitfromlearninganalytics?Howwillthepurposeoflearninganalyticsbecom-municatedtoprimaryusers?

Stake-holderengage-ment

4.2.4 Dimension 4 – Develop engagement strategy

Themapping of Dimension 4 showed that engagement datawasconsideredprimarydataforLAinthethreecases(Table10).Theimplicationforpolicyistodefinetherangeofdatabe-ingcollectedandencourage‘meaningfulselection’ofdata,sothatLAwillnotbedrivenbydata,butbylearningorteachinggoals(Table12).Itisalsocrucialtoincludestudentsandteach-ersintheinterpretationofdatasoastocontextualisedataandincrease thevalidityof analytics.The challenges thatCaseAandCfocusedonsuggesttheimportanceofincludingthesekeystakeholdersineffortstoimprovetheefficacyofLA(Table11)Acommonstrategysharedbyallthreecasesistosetupawork-inggrouptodriveLA.Itisimportantthatthepolicystatestheresponsibilitiesoftheworkinggroup,particularlytheirroleinensuringthatLAwillbeusedresponsiblywithintheinstitution.Forexample, theworkinggroupatCaseBwillneedtomakesurethatrelevantdataprotectionregulationshavebeencon-sulted,asitisnotevidentinthereportedactions.

Table10:Developengagementstrategy-actions

Case Action ThemeA TheinitialengagementwithLAwasguided

byJisc’sCodeofPracticeforLearningAnalyt-ics.Therewerepreparationstodevelopaninsti-tutional policy to provide a framework fortheuseofLAinthelocalcontext.

Ethics &privacy

TwoLAspecialistsandaworkinggroupweresetup to facilitateapilotprojectwithaLAserviceprovider,engagewithresearchactiv-ities,anddevelopinstitutionalstrategies.

Humanre-sources

TheinitialpreparationsincludedareviewofexistingLAcases.Thesourcesofdatausedinthepilotprojectincludedinteractionsinvirtuallearningenvi-ronments, Student Record Systems, andcoursemarks.Sixty-fiveonlineMSccourseswereinvolved.

Method-ology

B AdiverseworkinggroupwassetuptodriveLAactivities.

Humanre-sources

Theworkinggroupwillinitiatecommunica-tionsamongdifferentstakeholders.

Stake-holderengage-ment

Theinitialpreparationsincludedareviewofexisting LA cases and visits to other Euro-pean universities to learn from best prac-tices.ThedatasourcesincludedengagementdatainLMS(LearningManagementSystem)and

Method-ology

LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.

8

data held in SIS (Student Information Sys-tem).

C There were consultations on the SpanishLOPD(OrganicLawonProtectionofPersonalData).

Ethics &Privacy

TherewasaplantosetupaworkinggrouptopromoteLAamongteachingstaffandde-velopethicalguidelines.

Humanre-sources

Social interaction data was extracted fromdiscussionforumsintheLMS.

Method-ology

Table11:Developengagementstrategy-challenges

Case Challenges ThemeA Overrelyondataandfailtoconsidertheex-

perienceandknowledgeofinstructor/tutorsaboutstudents.

Method-ology

B While there was funding support from thegovernmenttodevelopstudentfeedbacksys-temsamongEstonianuniversities,therewasnostate-levelcoordinationtoinitiatecollab-oration among universities that have re-ceivedthegrant.

Manage-ment

C Focus on identifying students at risk andoverlook thepedagogicaldesignof curricu-lumorlearningsupport

Method-ology

Table12:Developengagementstrategy-policyprompts

Policy–questionstoreflecton ThemeWhataretheobjectivesforLA? PurposeWhatkindsofdatawillbecollectedtoachievetheseobjectives?Whatisthescopeofdatacollection?How will the results of analytics be interpretedwithinthecontext?Willteachingstafforstudentsbeinvolvedintheprocess?Whowilloverseeethicalconductsrelatedtolearn-inganalytics?

Method-ology

4.2.5 Dimension 5 – Analyse internal capacity to effect change ThemappingofDimension5showedthattheevaluationofin-ternalcapacityfocusedonfinancial,infrastructure,andhumancapacity(Table13).Acommonchallengesharedbythethreecases was in gaining wide support from the teaching staffamong whom analytical literacy and time availability weremainissuestodealwith(Table14).Theimplicationforpolicyis to ensure the availability of communication channels andsupport resources among different stakeholders (Table 15).While all cases identified the challenge of accessing certain‘useful’ data, Cases A and B recognised that ethical conduct

needsanenablinginfrastructure.Thus,itiscrucialthatthepol-icyprovidesguidelinestokeeptheinfrastructureupdatedwithregardtocurrentdataprotectionrequirements.

Table13:Analyseinternalcapacitytoeffectchange-ac-tions

Case Action ThemeA Ariskevaluationwasperformedtoanalyse

internalcapacity.Method-ology

B Therewasgovernmentfundingforthedevel-opmentoffeedbacksystemstosupportstu-dents.

Financialresources

C Therewas an evaluation of the availabilityandusefulnessofdatafromtheLMS.Interest was expressed in cross-institutioncollaborationonLAresearchprojectstoen-hancetheintegrationofLA.

Infra-structure

Table14:Analyseinternalcapacitytoeffectchange-chal-lenges

Case Challenges ThemeA 2018GDPR(EuropeanGeneralDataProtec-

tion Regulation) will bring changes to thewaytheuniversitydealtwithstudentdata.

Method-ology

The existing data infrastructure could notdealwithindividualopt-outs.Therewasno singlepermissiontouse stu-dentdataacrosstheinstitution.Some useful data remains inaccessible, e.g.the usage record of the digital library waskeptbypublishers.

Infra-structure

IfInstitutionAfailedtomanageonestudent’srequesttobeexcludedproperly,theunhap-pinessofonestudentmightspreadtoothersandstartaninstitution-wideobjection.Thebuy-infromteachingstaffwaspolarised.

Culture

B Thecultureofusingdatatoinformdecision-makingwasimmature.Although compulsory trainingwas plannedfor teaching and support staff, it was notclearhowtofosterownershipofLAamongstaff.ThebenefitofusingLAtosupportdecision-makingwascleartoseniormanagersbutnottoteachingstaff.

Culture

The existing infrastructure is not matureenough toprocessdata from theLMSortocopewithprivacyrequirements,suchasal-lowingindividualopt-outs.

Infra-structure

SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia

9

Data that ispotentiallyuseful for achievingthegoalsofLAmaynotbeaccessibleduetoprivacyissues.There was a skills gap in analytics and LAproject design, which posed questions re-gardingthevalidityofthecurrentapproachtoLA.

Capabili-ties

C Theskillsrequiredtounderstandandinter-pret visualised data needed to be installedamongteachingstaff.

Capabili-ties

Worriesaboutthetimedemandsinincorpo-ratingLAintoteachingoutweighedtheper-ceivedbenefitsofLA,andreducedthemoti-vationtoattendrelevanttraining.

Culture

CertaindataoutsidetheLMSishard toac-quire,suchassocialinteractionsinaphysicalclassroom.

Infra-structure

Table15:Analyseinternalcapacitytoeffectchange-pol-icyprompts

Policy–questionstoreflecton ThemeHowwilldataintegritybeachieved? Method-

ologyHowwillthedatabestoredanddisposed?Howoftenwilltheefficiencyandsecurityofexist-ingdatainfrastructurebeevaluated?

Datamanage-ment

Arethererelatedpolicies inthe institutional/na-tional/ international level that the LA policy sitsalongside/above/below?

Policymanage-ment

Whatcommunicationchannelsorfeedbackmecha-nismswillbeinplace?Whattrainingwillbedeployed?Willitbecompul-sory?

Stake-holderengage-ment

4.2.6 Dimension 6 – Establish monitoring and learning frame-works ThemappingofDimension6showedthatnoneofthethree

institutionshaddevelopedsuccesscriteriaordefinedmonitor-ingprocedures,perhapsdue to theearly stagesof adoption.However,thechallengesthatconfrontedthemindicatetheur-gency and importance to define success measures for LA intheircontexts,particularlywiththegroundingoflearningandteaching theories (Table 16). More importantly, the policyneedstoraiseawarenessaboutinadvertentconsequencesthatmayresult fromanalytics,andsuggestproceduretomonitoranddealwiththeserisks(Table17).

Table16:Establishmonitoringandlearningframeworks-challenges

Case Challenges Theme

A Therewasafearof failingtomeetexpecta-tions,resultinginabadnameforLA.

Method-ology

B It remains questionable whether studentdropoutrateisthebestsuccessindicatorfortheinstitutionalLAproject.

Method-ology

C Thecaptureddataoftimespentonlinemaynottrulyreflectlearning.The design and implementation of LAmayfailtoconsiderpedagogicaltheories.

Method-ology

Table17:Establishmonitoringandlearningframeworks-policyprompts

Policy–questionstoreflecton ThemeHowwillsuccessbemeasured?Whataresuccessindicators?Whatarethemechanismsthatdealwithinadvert-entconsequences?Whowillcarryouttheevaluationofimpact?

Evalua-tion

Howoftenwillthepolicybereviewedandupdated?Whowillberesponsibleforthepolicy?

Policymanage-ment

5 DISCUSSIONTheassociatedthemesthathaveemergedinthemappingre-sultsshowadifferentfocusforeachROMAdimension.Dimen-sion1 (mappingpolitical context) focuseson identifying the‘purpose’foradoptingLAinaspecificcontextsoastodriveac-tionsintheotherdimensions.Dimension2(identifykeystake-holders)isdrivenbytherecognitionthattheimplementationofLAinasocialenvironment involvescollectiveeffortsfromdifferent stakeholders. Dimension 3 (identify desiredbehav-iourchanges) setsobjectives,which reflectback to the ‘pur-pose’ofadoptingLA.Dimension4(developengagementstrat-egy)definesapproachestoachievingtheobjectivesbyaddress-ingaspectsthatcouldotherwisebecomechallenges,asidenti-fied in the literature: resources, ethics&privacy, and stake-holderengagementandbuy-in(seeSection2.1).Dimension5(analyse internal capacity to effect change) focuses on as-sessing the availability of existing resources (e.g., data andfunding)andidentifyingchallenges(risks).Dimension6(es-tablishmonitoringandlearningframeworks)iscurrentlyab-sentinallthreecases.ThismappingprocessillustrateshowtheROMAmodelcan

beusedtoexamineexistingLApracticesandrefinestrategies.Forexample,themappingresultsshowthatallthreecasesstillneedtoconsiderwhatitmeanstobesuccessfulwithLAandwhatsuccesslookslike(Dimension6),soastobetterinformactionsrelatedtootherdimensions.Theactionstakenbythethree cases also contributed to the action elements in theSHEILApolicy framework(Figure2),whichcouldbeusedtoinitiatestrategicplanningforearlyadopters.

LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.

10

Intermsofchallengesthatconfrontedthethreecases,themappingofDimension5identifiedkeythemesaroundculture,capability,andinfrastructure.Thisresultcoincideswithtwoofthe threekeyLAchallenges identified in the literature–de-mandonresourcesandstakeholderengagementandbuy-inasintroducedinSection2.1.Asaresult,thepolicyquestionsfocusonmanagementissuesarounddataintegrityandsecurity,andchannels for stakeholdertrainingandcommunicationwithintheinstitution.Theotherkeychallenge–ethicsandprivacy–wasparticularlyhighlighted in themappingofDimension2.Thisreafxirmstheimportanceandurgencyofaddressingethicsandprivacy issues that couldotherwise impedebuy-in fromstakeholders.Tothisend,thepolicyquestionsparticularlyfo-cusonmanagement issuesaroundprivacy, suchas consent-seeking,dataaccess,anonymityprinciples,anddatasharing.Whileapolicydoesnotnecessarilyprovidedirectsolutions

totheidentixiedchallenges,thequestionsintheSHEILApolicyframeworkintendtopromptanswersthatcouldserveassuit-ablecodeofpracticetomitigatethechallenges.Forexample,answerstothepolicyquestion–“howwillanonymitypoliciesbeappliedtotheprocessingandpresentationofdata”(seeTa-ble6)maynotprovidesolutionstothedatare-identixicationchallengeidentixiedbyCaseC(seeTable5),asitmaynotbeforeseenbeforedifferentdatasetsareintegrated.However,apolicy could suggest that a reviewand testprocess for suchrisks be carried out by data specialists before data ismadeavailabletoawiderpopulationofstakeholders.Thismayfur-therinformactionsofDimension4and5,astheavailabilityofdatacouldbedeterminedbytheassociatedrisksofprivacyandconsequentlyaffectengagementstrategy.Asidentifiedintheliterature,stakeholderengagementand

buy-inhasadirectimpactonthescalabilityandsustainabilityofLA,whichneedtobesupportedbystrategicplanning,ledbyinstitutionalleaders,andinformedbypedagogicalknowledgepossessedbyteachingprofessionals.ThisissueisreflectedinthemappingresultsofchallengesassociatedwithDimension1,3and4,where‘methodology’and‘management’arekeyissues.Asaresult,thepolicyquestionsfocusondefiningthepurposeofimplementingLAandconsideringthevalueofLAtoallrele-vant stakeholders and the specific context of the institution.Basedontheidentifiedpurpose,themethodologyadoptedtoachievethechosengoalshouldalsobestatedinapolicy,assug-gestedinDimension4.

6 CONCLUSIONWehavepresented three institutions’ approaches toLAandchallengesthatconfrontedtheminthispaper.UsingtheROMAmodel,weanalysedactionscarriedoutbythese institutions.WeextendedandadaptedtheuseofROMAfurtherbyincludingchallengesunderthesixdimensions.Thereafter,wedevelopeda set of questions to beaddressedwhen formulating policy.Thismapping process demonstrated the evidence-based ap-proachthatweadoptedtodeveloptheSHEILApolicyframe-work,which contributes three typesof informationvaluable

forasystematicadoptionofLA–actions,challenges,andpolicy.Theframeworkcouldbeusedtoguidethedevelopmentofin-stitutionalpoliciesandstrategicplanningforlearninganalyt-ics,toevaluateinstitutionalreadinessforLAandtobenchmarkbestpractices.ThispaperhaspresentedaselectivepartofthefirstSHEILA

policyframeworkthroughthreechosencases.Thelistofpolicypromptspresentedin thispaperwere selected to reflect thethreeparticularcases.TheframeworkwasdevelopedbasedonaseriesofinterviewswithpredominantlyseniormanagersinHEIs.Therefore,itparticularlyreflectstheperspectivesofthisgroup of stakeholders. Our futurework aims to incorporatefindingsfromotheron-goingresearchactivities,whichexploreviewsfromotherkeystakeholderssuchasteachersandstu-dents,regardingtheadoptionofLA.ACKNOWLEDGMENTSThisworkwassupportedbytheErasmus+ProgrammeoftheEuropean Union [562080-EPP-1-2015-1-BE-EPPKA3-PI-FOR-WARD].TheEuropeanCommission’ssupport fortheproduc-tionofthispublicationdoesnotconstituteanendorsementofthecontents,whichreflectstheviewsonlyoftheauthors,andtheCommissionwillnotbeheldresponsibleforanyusewhichmaybemadeoftheinformationcontainedtherein.Theprojectinvolvedcollaborativeinputfromallthepartnersinvolvedandtheircontributionsarehighlyappreciated.Wewouldalsoliketogivethankstoourresearchparticipants fortheirvaluablecontributions.

REFERENCES

[1] Ali,L.,Asadi,M.,Gašević,D.,Jovanović,J.andHatala,M.2013.Factorsinflu-encingbeliefsforadoptionofalearninganalyticstool:Anempiricalstudy.Computers & Education. 62, (Mar. 2013), 130–148.DOI:https://doi.org/10.1016/j.compedu.2012.10.023.

[2] Arnold,K.E.,Lynch,G.,Huston,D.,Wong,L.,Jorn,L.andOlsen,C.W.2014.BuildingInstitutionalCapacitiesandCompetenciesforSystemicLearningAnalyticsInitiatives.ProceedingsoftheFourthInternationalConferenceonLearningAnalyticsAndKnowledge(NewYork,NY,USA,2014),257–260.

[3] Arroway,P.,Morgan,G.,O’Keefe,M.andYanosky,R.2016.LearningAnalyt-icsinHigherEducation.ECAR.

[4] Bichsel,J.2012.AnalyticsinHigherEducation:Benefits,Barriers,Progress,andRecommendations.ECAR.

[5] Code of practice for learning analytics: 2015.https://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics. Ac-cessed:2016-06-07.

[6] Colvin,C.,Dawson,S.,Wade,A.andGašević,D.2017.AddressingtheChal-lengesofInstitutionalAdoption.HandbookofLearningAnalytics.C.Lang,G.Siemens,A.Wise,andD.Gaśević,eds.SocietyforLearningAnalyticsRe-search.281–289.

[7] Colvin,C.,Rogers,T.,Wade,A.,Dawson,S.,Gasevic,D.,Shum,S.B.,Nelson,K.,Alexander,S.,Lockyer,L.,Kennedy,G.,Corrin,L.andFisher,J.2015.Stu-dentRetentionandLearningAnalytics:ASnapshotofAustralianPracticesandaFrameworkforAdvancement.TheAustralianGovernmentOfficeforLearningandTeaching.

[8] Drachsler,H.andGreller,W.2016.PrivacyandAnalytics:It’saDELICATEIssueaChecklist forTrustedLearningAnalytics.ProceedingsoftheSixthInternationalConferenceonLearningAnalytics&Knowledge (NewYork,NY,USA,2016),89–98.

[9] Ferguson,R.,Brasher,A.,Clow,D.,Cooper,A.,Hillaire,G.,Mittelmeier, J.,Rienties,B.,Ullmann,T.andVuorikari,R.2016.ResearchEvidenceontheUseofLearningAnalytics:ImplicationsforEducationPolicy.

[10] Ferguson,R.,Macfadyen,L.P.,Clow,D.,Tynan,B.,Alexander,S.andDawson,S.2014.SettingLearningAnalyticsinContext:OvercomingtheBarriersto

SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia

11

Large-ScaleAdoption.JournalofLearningAnalytics.1,3(Sep.2014),120–144.

[11] Gašević,D.,Dawson,S.andSiemens,G.2015.Let’snotforget:Learningan-alytics are about learning. TechTrends. 59, 1 (Jan. 2015), 64–71.DOI:https://doi.org/10.1007/s11528-014-0822-x.

[12] Greller,W.andDrachsler,H.2012.TranslatingLearningintoNumbers:AGenericFrameworkforLearningAnalytics.EducationalTechnology&Soci-ety.15,3(2012),42–57.

[13] HigherEducationCommission2016.FromBrickstoClicks-ThePotentialofDataandAnalyticsinHigherEducation|HigherEducationCommission.

[14] Jeremic,Z.,Kumar,V.andGraf,S.2017.MORPH:Supportingtheintegrationoflearninganalyticsatinstitutionallevel.ProceedingsoftheSeventhInter-nationalLearningAnalytics&KnowledgeConference(NewYork,NY,USA,2017),596–597.

[15] Knight,D.B.,Brozina,C.,Stauffer,E.M.,Frisina,C.andAbel,T.D.2015.De-velopingaLearningAnalyticsDashboardforUndergraduateEngineeringUsingParticipatoryDesign.(Jun.2015),26.485.1-26.485.11.

[16] Macfadyen,L.andDawson,S.2012.Numbersarenotenough.Whye-learn-inganalyticsfailedtoinformaninstitutionalstrategicplan.FacultyofEdu-cation-Papers(Archive).(Jan.2012),149–163.

[17] Macfadyen,L.P.,Dawson,S.,Pardo,A.andGaševic,D.2014.EmbracingBigDatainComplexEducationalSystems:TheLearningAnalyticsImperativeandthePolicyChallenge.Research&PracticeinAssessment.9,(2014),17–28.

[18] Norris,D.M.andBaer,L.L.2013.BuildingOrganizationalCapacityforAna-lytics.

[19] Pardo,A.andSiemens,G.2014.Ethicalandprivacyprinciplesforlearninganalytics.BritishJournalofEducationalTechnology.45,3(May2014),438–450.DOI:https://doi.org/10.1111/bjet.12152.

[20] Park,Y.andJo,I.-H.2015.Developmentofthelearninganalyticsdashboardtosupportstudents’learningperformance.JournalofUniversalComputerScience.21,1(2015),110–133.

[21] Prinsloo,P.andSlade,S.2017.EthicsandLearningAnalytics:Chartingthe(Un)Charted.HandbookofLearningAnalytics.C.Lang,G.Siemens,A.Wise,andD.Gašević,eds.SocietyforLearningAnalyticsResearch.49–57.

[22] Prinsloo,P.andSlade,S.2015.StudentPrivacySelf-management:Implica-tionsforLearningAnalytics.ProceedingsoftheFifthInternationalConfer-enceonLearningAnalyticsAndKnowledge(NewYork,NY,USA,2015),83–92.

[23] Roberts,L.D.,Howell,J.A.,Seaman,K.andGibson,D.C.2016.Studentatti-tudestowardlearninganalyticsinhighereducation:“Thefitbitversionofthe learning world.” Frontiers in Psychology. 7, (Dec. 2016), 1–11.DOI:https://doi.org/10.3389/fpsyg.2016.01959.

[24] Rubel,A.andJones,K.M.L.2016.Studentprivacyinlearninganalytics:Aninformationethicsperspective.TheInformationSociety.32,2(Mar.2016),143–159.DOI:https://doi.org/10.1080/01972243.2016.1130502.

[25] Siemens, G., Dawson, S. and Lynch, G. 2013. Improving the Quality andProductivityoftheHigherEducationSector:PolicyandStrategyforSystems-levelDeploymentofLearningAnalytics.SocietyforLearningAnalyticsRe-searchfortheAustralianOfficeforLearningandTeaching.

[26] Slade,S.andPrinsloo,P.2013.LearningAnalytics:EthicalIssuesandDi-lemmas.American Behavioral Scientist. 57, 10 (Mar. 2013), 1510–1529.DOI:https://doi.org/10.1177/0002764213479366.

[27] Slade, S. andPrinsloo,P. 2014. Studentperspectiveson theuse of theirdata:Betweenintrusion,surveillanceandcare.ChallengesforResearchintoOpen&DistanceLearning:DoingThingsBetter–DoingBetterThings(Ox-ford,UK,Oct.2014),291–300.

[28] TheOpenUniversity2014.PolicyonEthicaluseofStudentDataforLearn-ingAnalytics.TheOpenUniversity.

[29] Tsai,Y.-S.andGašević,D.2017.Learninganalyticsinhighereducation—challengesandpolicies:Areviewofeightlearninganalyticspolicies.Pro-ceedingsoftheSeventhInternationalLearningAnalytics&KnowledgeCon-ference(NewYork,NY,USA,2017),233–242.

[30] Tsai,Y.-S.andGašević,D.2017.TheStateofLearningAnalyticsinEurope–ExecutiveSummary–SHEILA.

[31] Wolff,A.,Moore,J.,Zdrahal,Z.,Hlosta,M.andKuzilek,J.2016.DataLiteracyforLearningAnalytics.ProceedingsoftheSixthInternationalConferenceonLearningAnalytics&Knowledge(NewYork,NY,USA,2016),500–501.

[32] Yanosky,R.andArroway,P.2015.TheAnalyticsLandscapeinHigherEdu-cation.ECAR.

[33] Young,J.andMendizabal,E.2009.HelpingResearchersBecomePolicyEn-trepreneurs-HowtoDevelopEngagementStrategiesforEvidence-basedPol-icy-making.OverseasDevelopmentInstitute.