Post on 10-Jul-2020
TransformingexamsacrossAustralia:Processesandplatformfore-examsinhighstakes,supervisedenvironments
Finalreport 2019
Lead institution: MonashUniversity
Partner institutions: CentralQueenslandUniversity,EdithCowanUniversity,MacquarieUniversity,RMITUniversity,TheUniversityofQueensland,TheAustralianNationalUniversity,UniversityofNewSouthWalesatADFA,UniversityofSouthAustralia,UniversityofTasmania
Project leader: DrMathewHillier
Teammembers: AssociateProfessorMatthewBower,AssociateProfessorMichaelCowling,DrAndrewFluck,DrRuthGeer,DrScottGrant,DrBethHarris,MrKennethHowah,DrDavidMeacheam,MrDominicMcGrath,DrJeremyPagram,MrBruceWhite
Projectmanager: MsVilmaSimbag
SoftwareDeveloper: MrMartinColeman
TransformingExams.com
SupportfortheproductionofthisreporthasbeenprovidedbytheAustralianGovernmentDepartmentofEducation.TheviewsexpressedinthisreportdonotnecessarilyreflecttheviewsoftheAustralianGovernmentDepartmentofEducation.
WiththeexceptionoftheCommonwealthCoatofArms,andwhereotherwisenoted,allmaterialpresentedinthisdocumentisprovidedunderCreativeCommonsAttribution-ShareAlike4.0InternationalLicensecreativecommons/4.0/license.
ThedetailsoftherelevantlicenceconditionsareavailableontheCreativeCommonswebsite(accessibleusingthelinksprovided)asisthefulllegalcodefortheCreativeCommonsAttribution-ShareAlike4.0InternationalLicensecreativecommons/4.0/legalcode.
Requestsandenquiriesconcerningtheserightsshouldbeaddressedto:HigherEducationProgramManagementGovernanceQualityandAccessBranchHigherEducationGroupAustralianGovernmentDepartmentofEducationGPOBox9880LocationcodeC50MA7CANBERRAACT2601<learningandteaching@education.gov.au>
2019ISBN978-1-76051-756-4[PDF]ISBN978-1-76051-757-1[DOCX]ISBN978-1-76051-758-8[PRINT]
TransformingexamsacrossAustralia | iii
AcknowledgementsTheTransformingExamsAcrossAustraliaprojectteamwouldliketogivespecialthankstothestudentsandeducatorsfromourpartneruniversitieswhoparticipatedintriallinge-Examsintheircourses.EducatorswhohostedoneormoretrialssincetheSeedProject2014include:AlistairCampbell(ECU),AmyHubbell(UQ),AndrewFluck(UTAS),AndrewGilbert(UNSWADFA),AroshaWeerakoon(UQ),BruceWhite(UniSA),CharanjitKaur(Monash),DavidBooth(UQ),JeremyPagram(ECU),KatrinaWilliams(UQ),KennethHowah(CQU),LizSpringfield(UQ),LyndenGriggs(UTas),MalcolmJones(UQ),MaryTom(CQU),MatthewBower(MQU),MichaelCowling(CQU),MichaelStokes(UTas),MichaelTate(UTas),NathanielLyon(MonashCollege),RahatHossain(CQU),RebekahScotney(UQ),RobinFitzgerald(UQ),RuthGeer(UniSA),ScottGrant(Monash),ShaniTobias(Monash),TomDunning(UTas),WendyBalassa(UTas).Theirparticipationledtheprojectteamtowardssignificantfindingsandrecommendationstoimprovethee-Examtechnology,processesanddocumentation.WealsothankinitialteammembersandsupportersinvolvedearlyintheprojectincludingKimBlackmore(ANU),PaulNewhouse(ECU—deceased),DominicVerity(MQU),MarilynBaird(Monash),ShonaLeitch(RMIT),CarolineSteel(LaTrobe),GeoffreyCrisp(UC),GeraldCrowhurst(UQ),Sarah-RobertsThompson(UQ),HelenFarley(USQ).TheprojectteamwouldliketoacknowledgetheadditionalfinancialsupportprovidedbyMonashUniversityfrom(former)Pro-ViceChancellor,DarrellEvans,intheformofastrategicfundsallocation.Thisenabledtheacquisitionofmuch-neededspecialisttechnicalequipmentandinfrastructure.AspecialthanksisalsoduetoRobertFootefromtheNationalAccreditationAuthorityforTranslatorsandInterpretersforhissupportininitiatingafundedspin-offprojectinlanguagetranslatorcertificationtesting.Acknowledgementisdueforhighlyvaluableworkonthedevelopmentofe-ExamstechnologybyLubosRandek(late2015tomid-2016)andMartinColeman(early2016to2019);alsospecialthankstoourprojectmanagerVilmaSimbagforkeepingusallontrack.WewanttoexpressourgratitudetoDeanneGannaway,theprojectexternalevaluator,whohelpedusremainfocusedonworkingtowardstheproject’sdeliverables.
TransformingexamsacrossAustralia | ivT
Listofacronyms
ADFA AustralianDefenceForceAcademy
ANU TheAustralianNationalUniversity
AR AugmentedReality
ASCILITE AustralasianSocietyforComputersinLearninginTertiaryEducation
BYOD BringYourOwnDevice(alsoasinBYOlaptop)
CQU CentralQueenslandUniversity
DET DepartmentofEducationandTraining,AustralianGovernment
ECU EdithCowanUniversity
LMS Learningmanagementsystem
MCQ Multiple-choicequestion
Moodle ModularObjectOrientedDynamicLearningEnvironment(aLMS)
Monash MonashUniversity
MQU MacquarieUniversity
NAATI NationalAccreditationAuthorityforTranslatorsandInterpreters
NAPLAN NationalAssessmentProgram-LiteracyandNumeracy
OLT OfficeforLearningandTeaching
RMIT RMITUniversity
SES SocialEconomicStatus(asinlowSES)
TEQSA TertiaryEducationQualityandStandardsAgency
UniSA UniversityofSouthAustralia
UNSW UniversityofNewSouthWales
UQ TheUniversityofQueensland
UTas UniversityofTasmania
TransformingexamsacrossAustralia | vT
ExecutivesummaryProjectcontext
Aparadigmshiftinhighereducationisunderway.Wearemovingfromanalogue,disconnected,paper-centricassessmentpracticestowardstheinclusionofdigitallyconnected,relevantandauthenticapproachestolearningandteaching.Thisenablesustoaccreditgraduatesasreadyfortheworkandsocialworldofthe21stcentury.
Thismoveisnecessarybecausecurrentinvigilatedexaminationpracticesarestuckinthe20thcentury.Moderne-toolsofthetrade,intheformofsophisticatedsoftwareapplicationsandinformationresources,aredisallowedinfavourofanalogueisolationintheexamroom.Studentsaccustomedtoconnected,digitalwaysofworkingarestrugglingtodemonstratetheirskillsunderconditionsrarelyfacedintheprofessionalandvocationalsphere.Academicsarealsoheldbackfromdevelopingricher,more-authenticwaysofassessinglearninginlightofthecurrenttrade-offsbetweenpedagogicalefficacy,integrityandfeasibility.Althoughincreasingnumbersofdigitalassessmenttoolsarebeingoffered,manyneglectauthenticityinfavourofscalabilityorsecurity,orfocusonanarrowcontextofclassroomquizzesortestingcentres.Thesesystemsareoftenreliantonhavingperfectnetworkconnectivity.Thereiscurrentlynorobust,viablemethodtodoauthentice-assessmentthatwillworktoalignbothin-classprogressiveassessmentforlearningandhigherstakessummativeassessmentundertakeninlargescaleexamhalls.
Thisprojecttransformsexaminationsbyidentifyingthenecessaryunderlyingconditionsandcontextualfactorsforsuccessfulauthentice-assessmentpractice.Thereweretwocriticaldevelopmentsinthisproject.Firstly,therecognitionthatweb-browser-basedassessmentalonedoesnotpermitcandidatestodemonstrateskillsbasedonsophisticatedcomputersoftwareapplicationsfitforprofessionalwork.Secondly,theuseofacommonUSBbootingsystemprovidestotalinstitutionalcontrolofstudent-ownedcomputerswithoutinterferingwiththeirpersonaldataorsoftware.Thesecriticalideasledtheprojectteamtodesignarobuste-Examplatform,whichisalsoresilienttonetworkbreakdowns.Theprojectteamthenwentontoimplementandverify,acrossarangeofinstitutionalcontexts,aviabletechnologicalandproceduralapproachtoscaleauthentice-assessment.
Aims
Thisprojectaimedtoenablethedevelopmentanddeliveryofauthentice-assessmentinthesupervisedexamroomcontext,inamannerthatwasscalableandsustainable.Tofacilitatethemovefrompapertodigitalexams,theprojectsetouttodeliverapracticalsolutionintheformofane-Examtechnologyplatformsupportedbyguidesforstudents,teachersandadministrators.Evidenceastotheviabilityoftheapproachtoauthentice-Examswastobeprovidedby35research-ledtrialsacrossarangeofinstitutionalcontextsthatincludedstudent,teacherandadministratorexperiences.Thesefindingswillguideteachers,assessmentdevelopers,technologysupport,examinationsmanagersandinstitutionalleadershipinmattersofpolicy,practicalimplementationandpedagogicdesign.
Approach
Theprojectinvolvediterativephasesofdevelopmentalongaroadmap.Aseriesofe-examtrialswereconductedindifferentinstitutionalcontextswiththefindingsofeachroundfeedingforwardintothenextiteration.Eachinstitutionprogressedattheirownpace.
TransformingexamsacrossAustralia | viT
Start> >>> >>> >>> >>> >Future>GetReady Phase1 Phase2 Phase3 Phase4 Phase5
Institutionalapprovals,
researchethics,hardwareandinfrastructure
Paper-equivalentsmallscale
Post-papersmalltomedium
Mediumtolargescale
WhitelistedandloggedInternet
OpenbutfullyloggedInternet
Basicdocexamstobegin!
Expandingtheappandmedialandscape
Addingthepowerofan
LMSNetworkedBYODexam
NetworkmixedmodeBYODexam
Crawling Walking Running Jumping Flying!
Acrossthefouryearsoftheproject,10partnerinstitutionshadinput,35e-examswereconductedwithover3000students,teachers,andadministratorsparticipatinginexamsessions,insurveys,focusgroupinterviews,workshopsandaninternationalsymposium.
Projectdeliverables
Theprojectteamnotonlymetalldefineddeliverablesoftheprojectbutalsoprovidedadditionaloutputsthatcentredondisseminationandadoption.Theprojectteamfocuseduponactivitylikelytohavethegreatestimpactonthehighereducationsector,thisbeingdevelopingknowledgeaboutauthentice-assessmentpracticeandinprovidingthepracticalinfrastructuretoimplemente-examsinsupervisedsettings.Significantoutputsincluded:
• theTransformingExamsprojectwebsitehousesthemajorityofprojectoutputslistedbelowandservedasaportalforupdates,seehttp://transformingexams.com
• thee-Examstechnologyplatformconsistingofaclient,serverandadministrationcomponents.Thissoftwarewasupdatedastheprojectprogressedanddistributedtoprojectpartners.Demonstrationversionsareavailableontheprojectwebsite
• learningresourcesandguidesforstudents,teachers,technicaladministratorsandexaminationsmanagersweredevelopedandmadeavailableontheprojectwebsite
• anationalroadshowandsymposiumencompassedfivecapitalcitiesinwhichover300highereducationpersonnelparticipated
• disseminationoutputsincludedover60presentationsatconferencesandinstitutionalmeetings,withsessionrecordingsortheslidesmadeavailableviatheprojectwebsite.Thesesessionsadvisedthehighereducationcommunityoftherationaleandaroadmaptowardsauthentice-assessmentaswellastheresearchfindingsfrom35diverseresearch-lede-Examtrials
• atotalof15refereedpublicationswereproduced(withmoretocome)coveringresultsfrom35e-Examtrialsinwhich1700studentstypedtheirexam.Findingscoverede-Examsystemdesignanduse,students'perspectivesone-Examsandontheirwritingstrategies,possibledigitalassessmentfuturesandpotentialimpactsofauthentice-assessmentoncurriculumtransformation.
KeyFindings
Theprojectencompassedmultiplerealmsfromtheory,perceptionandpolicytopracticalmeasuressuchastechnologydevelopment,implementationanduse.Keyfindingsincluded:
• creationofaviable,robusttechnologyplatformfore-Exams.Itisflexible,supportingatransitionfrompaper-equivalentexamstorich,authenticdigitalassessment.Atechnologyplatformfore-Examsmustsupportpedagogicalalignmentatallstagesoflearningfromtheclassroomintotheexamroom.Therefore,itintegratesandleveragesexistingsystemssuchascommonwordprocessors,learningmanagement
TransformingexamsacrossAustralia | vii
systemsandsoftwarecommonlyusedinthediscipline.Aviablee-Examsystemmustworkintheroughspacesofclassroomsandeducationsettings.Thee-Examsystemwasfoundtobesecureandreliable,highlyresistanttonetworkoutagesandcanfunctionoffline.Thisallowsstudentstofocusoncompletingtheirexamandminimisesdisruptiontoexamtimetables
• studentsaccepte-Examsandfindthemsuitedtotheirlargelypreferredwaysofworking.Studentsreportthattypingispreferabletohandwritinginexams.However,somestudentsneedsupportinthetransitionwithasignificantminorityfacingchallengesinadjustmentofwritinghabitsandinsupplyingequipmentthatisuptostandard.Thetransitionroadmap,alongwithamixofinstitutionandstudenttechnology,willenabletheuptakeofmediumtohigherstakesdigitalassessmentacrosseducationalinstitutions
• logisticalsupportandattentiontodetailmattersinthehighstakes,time-pressuredexams.Whereorganisation,resources,processandcommunicationarelacking,achievinggoodoutcomeswithcomputerisedexamsismadesignificantlyharder.ExaminationsofficepersonnelareexpertsatlogisticssoleveragingthatexpertiseincollaborationwithotherkeyareassuchasITsupportisimportanttosuccess
• academicdevelopmentintermsofdigitalliteracyanddigitalpedagogywillbeimportanttoembeddingandscalingauthentice-assessmentacrossthecurriculum.Whilee-Examsareonepartofthepicture,ourcurrentanalogueexampracticesareoneofthe'blockers'tobroadersystemicchange.Theneedtoensurealignmentofpracticesinclassroomsandexamhallsmeansthatacademicteachingandlearningsupportstaffmustalsohavetheskillsandcapabilitiestodeployrelevante-examstechnologiesandtoolsintheirclassrooms
• policyisamechanismtosupportchange,asismanagerialsupport.Policiesonexaminationsthatsetouttoframee-Examsmustextendbeyondtraditionalconfines.Newsystemicconnectionsmustbemadewithbroaderareas,includingdevelopingaholisticdigitallearningstrategy,togetherwithaBringYourOwnDevice(BYOD)-focusedinfrastructuredevelopmentstrategy,ITsupport,pedagogicprofessionaldevelopmentandstudentsupporttoenableanequitabletransition
• universitiesdonotstandaloneintheassessmentspace.InAustralianschools,NAPLAN(nationalnumeracyandliteracytesting)andmatriculationexamsaretentativelymovingonline,OthercountriessuchasFinlandhavealreadycompletedthetransitionwithtestsandexamsinschoolforYears10–12,aswellasuniversityentranceexamsconductedentirelyoncomputers.Thee-Examplatformiswellplacedforuseinschool-basedassessmentsandmatriculationexaminations,aswellasinvocationalandprofessionaleducationsettings.
Futurework
Theprojectwasagreatsuccessindemonstratingwhatrobust,authenticdigitalassessmentcouldbelike.Butthefindingsshowourworkisnotyetcomplete.Furthertechnicalworkisrequiredtoimprovethee-Examtechnologyplatform,itsintegrationwithothersystemsanditsadministrativesupporttools.Furtherinvestigationandinvestmentintosystemicsupportmechanismsisrequiredtoscaledigitalassessmentathighereducationinstitutionsinareassuchasprocess,professionaldevelopment,policyandinfrastructure.
TransformingexamsacrossAustralia |viii
TableofContentsAcknowledgements...................................................................................................................iiiListofacronyms........................................................................................................................ivExecutivesummary....................................................................................................................vTablesandfigures.....................................................................................................................ixChapter1. Theproject...........................................................................................................1
1.1 Background....................................................................................................................11.2 Projectlogistics..............................................................................................................21.3 Projectmanagement.....................................................................................................31.4 Statusofprojectdeliverables........................................................................................3
Chapter2. Ane-Examplatformforauthenticassessment....................................................42.1 Rationaleforthedevelopmentofane-Examsolution.................................................42.2 Priore-Examssolutions.................................................................................................42.3 Historicaldevelopmentofarobuste-Examplatform...................................................52.4 Thelateste-Examplatform...........................................................................................7
Chapter3. e-Examtrialfindings...........................................................................................103.1 Researchdatacollection.............................................................................................103.2 Studentperspectives...................................................................................................113.3 Logisticsfindings..........................................................................................................123.4 Minicases....................................................................................................................14
Chapter4. Dissemination.....................................................................................................154.1 Websiteandsocialmedia............................................................................................154.2 Presentationsandpublications...................................................................................154.3 e-Examcasestudies....................................................................................................164.4 e-Examworkshops......................................................................................................164.5 e-Examroadshow........................................................................................................164.6 e-Examssymposium....................................................................................................17
Chapter5. Impact.................................................................................................................195.1 Changedpractices.......................................................................................................195.2 Othere-examprojects.................................................................................................195.3 PartnershipwithNAATI...............................................................................................195.4 Potentialjointventurepartnershipandcommercialisation.......................................195.5 Furtherresearchandjointprojectwork.....................................................................20
Chapter6. Projectevaluationandlessonslearnt................................................................216.1 Projectexternalevaluation.........................................................................................216.2 Policyadjustment........................................................................................................216.3 Logisticsandcommunicationswithinpartnerinstitutions..........................................22
Chapter7. Conclusion..........................................................................................................24References..............................................................................................................................25AppendixACertification.........................................................................................................27AppendixBProjectdeliverabledetails...................................................................................28AppendixCe-Examtrialdetails...............................................................................................30AppendixDUpdatedImpactMatrix.......................................................................................35AppendixEFormativemid-projectevaluationreport............................................................37AppendixFFinalexternalevaluationreport...........................................................................39
TransformingexamsacrossAustralia |ix
TablesandfiguresTablesTable1Statusofprojectdeliverables......................................................................................3Table2e-Examsystemfeaturesandaffordances...................................................................7Table3e-Examtrialsconductedacrossinstitutionsandyears..............................................10Table4e-Examsummarystatistics........................................................................................11FiguresFigure1Projectorganisation...................................................................................................2Figure2Tensionsandtrade-offsindevelopingorselectinge-Examssolutions......................5Figure3e-Examsystemdevelopmentandadoptionroadmap...............................................6Figure4e-ExamworkflowforrobustMoodleuse(phasethree)............................................8Figure5Photo:arobustonlinee-Examsessioninprogress..................................................10Figure6Aggregatedstudentfeedbackonthee-Examsystem..............................................11Figure7StudentsurveyresponsesfollowingrobustMoodleexams....................................11Figure8Handwritingneatnessandtypingprowess..............................................................12Figure9Powersupplyfore-Exams:layingcablesandbatterylife........................................13Figure10Websitevisitorsbycountry....................................................................................15Figure11Photo:e-Examsymposiumsessions24November2018.......................................17
TransformingexamsacrossAustralia |1
Chapter1. Theproject1.1 BackgroundThisproject‘TransformingexamsacrossAustralia:Processesandplatformfore-examsinhighstakes,supervisedenvironments’wasbornfromthegrowingdisconnectbetweenthewayhighstakestestingisconductedusingpen-on-paperexamsandstudents’everydayexperiencesofstudy,workandlife.Thecomplexityofproblemsfacedbyallindustriesandsocietydemandstheuseofsophisticatedproblem-solvingapproaches,typicallyemployingadvancedsoftwaretechnologies.Theuseofinformationandcommunicationtechnology(ICT)inhighereducationand,inparticular,invigilatedhigherstakesassessmentlagsbehindthecommonuseofICTbystudentsinotherareasoftheirstudy,workandsociallife.Thisisdespitehighereducationinstitutions’ambitionstoprovidestudentswithcoretechnologyliteraciesrequiredforsuccessinamodernworld.Thepaucityofauthenticassessmentattheinvigilatedmediumandhigherstakesendoftheassessmentspectrumisholdingbacktheabilityforhighereducationtoevaluatestudents'learninginlightofdesired21stcenturycapabilities(Fluck&Hillier,2016).TheAustralianhighereducationsectorneedsagreatlyexpandedcapabilityforhigherstakes'assessmentandpromotionofstudentlearning'ifourgraduatesaretoremaincompetitiveintheinformation-rich,technology-intensiveworldoftoday.
Thisprojectaimedtoexplorehowauthenticformsofassessmentcouldbeundertakenifsophisticated'e-toolsofthetrade'weremadeavailabletoassessmentdesignersandstudentsintheinvigilatedexamroomcontext.Theprojectteamsetaboutdevelopingandevaluatingacomprehensivetechnologyenvironmentforexamsthatwouldenableauthentic,complex,constructedassessment.Thee-ExamsprojectteamrecognisedthattherearecontextualchallengessurroundingtheimplementationofICTintheexamroom,particularlyintheformofasuitable,scalableandaffordableexamstechnologyplatform.Surroundingissueswerealsowithinthescopeofprojectdeliberationsincludedconsiderationoftheefficiencyofexammanagement,thelogisticsofcarryingoutcomputerisedexaminations,theavailabilityofsuitablespacesatinstitutions,theneedforcomputerequipmenttocaterforalargenumberofexamcandidatesandstudentacceptanceofe-exams.
Thepredecessorseedprojectidentifiedtheneedtofurtherenhanceasetofe-examsgoodpracticeguidelinesacrossmultipleinstitutionalsettings.Theseedprojectalsoprovideddirectionforthefurtherdevelopmentofarobuste-examsinfrastructurewithsector-widerelevanceandcross-disciplineapplicability.Theteamalsoarticulatedhowe-examscouldevolveintheAustralianhighereducationcontextoverthefollowing10years(Hillier&Fluck,2015).
Thisprojectsetoutto:
1. buildontheexistinge-examplatform(developedunderthepriorseedproject)intendedforuseinsupervised,'bringyourownlaptop'(BYOL)settingstoincludecomputer-markedquestionsandelectronicreticulationofstudentresponses
2. developguidelinestoassiststudents,educatorsandadministratorstoeffectivelyprepareandundertakee-exams,takingintoconsiderationthewholeassessmentworkflow.
TheprojecthasdeliveredarangeoffindingsonstudentpreferencesandageneraltoolsetforICT-enhancedinvigilatedmediumandhigherstakesassessmentusinganapproachthat
TransformingexamsacrossAustralia |2
isdesignedtobeopen,modularandastechnologyneutralaspossible.Thee-Examdeliveryplatformdevelopedbytheprojectteamenablesinvigilatedhighstakestestingoncampusata largescale.Itmustbenotedthatthisprojectdidnotspecificallyaddressonline-onlyordistanceeducationcontexts.
1.2 ProjectlogisticsTheprojectteamwasorganisedintoclustersateachpartneruniversitywiththeleadinstitutionhosting'e-ExamHQ'.TheorganisationshowingtherelationshipsbetweeninstitutionsandgroupsisdepictedinFigure1.
Figure1ProjectorganisationOnreceptofthegranttheprojectleaderpreparedthegroundworkforimplementation.Theactivitiesincluded:
• recruitmentofstaffincludingtheprojectmanagerandleadtechnicaldevelopers• obtainingethicalclearancefromtheleadinstitutionMonashUniversity,which
facilitatedtheapprovalbyprojectpartnerinstitutions:CentralQueenslandUniversity(CQU),EdithCowanUniversity(ECU),MacquarieUniversity(MQU),TheUniversityofQueensland(UQ),UniversityofSouthAustralia(UniSA),andUniversityofTasmania(UTAS).TheUniversityofNewSouthWales–AustralianDefenceForceAcademyjoinedtheprojectin2017.Note:TheAustralianNationalUniversity(ANU)andRMITUniversitydidnotparticipateine-examtrials
• developingsubcontractagreementsbetweenMonashUniversityandindividualpartnerinstitutionsthatoutlinedresponsibilitiesanddeliverables.Wheregrantpaymentswereplanned,theseweremadeconditionalupondeliverablesbeingmetateachstage
• holdingapreparationworkshopintheopeningmonthsoftheproject.HostedatMonashUniversity,CaulfieldCampus,itwasattendedby21projectcollaboratorsandteammembers.Theaimwastoprovideaprojectoverviewanddetailedinstructionsontheimplementationofthee-examstrialsandahands-onsessionusingthee-Examsystem.Attendeesincludedacademics,learningdesignersandexamsofficersfrompartnerinstitutions
• developingapackageoffilestoruntrialsthatwasprovidedtoeachpartner.Thisincludedtemplatesforapreliminarymeetingchecklist,ethicsprotocoldraft,participantinformedconsentinformationandform,surveyandfocusgroupquestionnairesanddatacollectionspreadsheettemplates.Inthelead-uptoeachtrial,meetingswereheldwithcourseacademicsbyprojectteammembersateachinstitutiontoarrangecontextspecificactionplans
• collectingdataduringandfollowingtrialsthatwasreportedtotheprojectleaderviaprojectpartnerinstitutionrepresentativesusingthedatacollectionspreadsheettemplates.
Core partner institution(s). Project-funded sites. Lead + RA staff. Trial sites.
Associate partner institution(s) Self-funded. a) Trial sites: receive mentorship; return data. b) Technology exchange partners.
Lead institution. Funded. 'e-Exam HQ': Project leader > project manager, technical development staff. Main trial site.
Reference group DET
Other key stakeholders.
External Evaluator
TransformingexamsacrossAustralia |3
1.3 ProjectmanagementThefollowingactivitiesenabledthenationalscaleprojecttobemanaged.
• Regularprojectmanagementteammeetings:TheprojectmanagementteamcomprisingtheprojectleaderandprojectmanagerheldregularweeklySkypemeetings.
• Projectteammeetings:Theteamhadfiveface-to-facemeetings:onein2016,twoin2017andtwoin2018.
• Regularemailcommunication:Teammembersandcollaboratorscommunicatedcloselyviaemail.Communicationwascentredonrunningofparticulare-examtrials,arrangingsitevisitsandpreparingjointpublicationsorpresentations.
• Useofprojectmanagementtools:Theprogresstowardsdeliverablesandoutcomeswastrackedusingtheonlinetool'Basecamp'.Thistoolallowedprojectmemberstocommunicatewitheachother,kepttrackoftheirtasks,timelinesandmilestones.Italsoservedasrepositoryforprojectdocumentsrelatingtoindividuale-examtrials.Ametaprojectwasestablishedwitheachpartnerinstitutionhavingitsownsubproject.TeamGanttsyncedwiththeBasecampaccountandprovidedanintegratedviewofalltasksandschedulesforthecoremanagementteam.CloudstorageservicessuchasDropbox,GoogledriveandAARNETCloudstorwereusedasrepositoriesforcommonprojectdocumentsandtodistributeupdatede-Examsystemfiles.
1.4 StatusofprojectdeliverablesFollowingonfromtheoriginallyplannedprojectdeliverablesitwasfoundthattheenvironmentandfocusofourworkevolvedovertheprojectperiod2015to2018.Alldeliverablesweremet,althoughsomeadjustmentsweremadetotheformdeliverablesasnewinformationcametohand.Thestatusofeachoftheoriginallyplannedprojectdeliverables,alongwiththeachievedprojectoutputs,isshowninTable1withfurtherexplanationofeachprovidedAppendixB.
Table1StatusofprojectdeliverablesOriginaldeliverable Deliveredoutpute-Examstechnicalinfrastructure Delivered.Arobustandmodulare-exams
platformresistanttonetworkoutagesGoodpracticeguides Delivered.Theseevolvedintoasetofuser
guides.Adraftsetofassessment/examspolicyrecommendationsforinstitutions
Alternativedelivered.Seesection6.1.foralternativefindingsanddiscussion
Asetofexamplee-exams Delivered.35trialsconductedacrossthreemodalitiesandmultipledisciplines
Researchfindingspublished Delivered.15refereedpublicationsand60presentations
Workshopmaterialstointroduceacademicstoe-exams
Delivered.Severalworkshops,nationalroadshowandaninternationale-ExamsSymposium
An'e-examsfoundation'drafttermsofreference
Alternativedelivered.Recommendationforjointventureorcommercialisation
TransformingexamsacrossAustralia |4
Chapter2. Ane-Examplatformforauthenticassessment2.1 Rationaleforthedevelopmentofane-ExamsolutionAtthebeginningoftheprojectanumberofpremiseswereoutlinedthatformedarationaleforthedevelopmentofane-Examssolution.Thechiefamongthesewasthegrowingdisconnectbetweencurrentexaminationpracticesandtheworldofwork.Similarly,examswerebecomingfurtherdisconnectedfromtheremainderofstudents'learningexperiencesandtheirextensiveuseofcomputersintheirstudies.Therewasanoverwhelmingsensethatexaminationsneededtoenable'authentice-assessment'whilemaintainingassessmentintegrity.Therewasalsorecognitionthatane-Examapproachhadtobescalableandsustainable.Thestartingrationalewasoutlinedinthreebroadareaspresentedinthepointsbelow,withfurtherexplanationandaconceptmapprovidedinHillierandFluck(2013).Studentrequirements:
• studentsrarelyhand-writeassessmentresponses• studentsarefamiliarwiththeirowndevices• highstudentownershipoflaptops• student-owneddevicesarediverse• studentsdislikeinterferencewiththeirdevice.
Pedagogicalrequirements:• technologycanenhancetheauthenticityofassessment• equivalentsoftwareenvironmentforallstudents• provideopportunitiestopractice.
Institutionalrequirements:• scalabilityofthesolutiontoservelargestudentgroups• institutionalcontextsdifferthatdemandsaflexiblesolution• resilienceandsecurity• unauthoriseddataandcommunicationmustbeexcludedfromtheexam• workflowefficiency.
2.2 Priore-ExamssolutionsAtthebeginningoftheinnovationanddevelopmentoftheprojectamarketscanwasconductedthatlookedatexistingcomputerisedexamsolutionsandapproachesbeingused.Thesewerefoundtobewanting.Typically,solutionsfocusedonsecurityorefficiencybutoftenneglectedpedagogicalsophistication.ThistensionisshowninFigure2(left).Theexistingapproachestoconductingcomputerisedexamshadanumberoflimitations,suchthatoneormoreofthefollowingelementswereevident:
• Limitedpedagogicscope.Limitationsinthetypeofquestionsandtasksthatarepossible.Systemsoftenfocusedonasingletypeofassessmentormodality.Typicallyasanisolatedquizinvolvingselectedresponsequestionsoressayquestiontypes.
• Candidateswerenotpreventedfromaccessingunauthorisedmaterials.TheuseofstandardquiztoolsembeddedinalearningmanagementsystemmeansthatstudentshaveaccesstoothertoolswithintheLMS.
• Candidatenumberswerelimitedtothecapacityofon-campuscomputerlabs.Testsundertakeninteachinglabswithafinitecapacitycanonlycatertoasmallnumberof
TransformingexamsacrossAustralia |5
seats,thisisinsufficientforthenormalhighvolumeofcandidates.Thelayoutoflabroomsisoftenunsuitableforhighstakesexamswithoutsomemodification.
• Proprietarysoftwareapplicationsandlimitedflexibility.Licensingconditionsmaylimitbroaderusewithoutadditionalcostsbeingincurred.Similarly,mostsystemsarepresentedasaseparateenvironmentthatisnotpartoftheregularlearningandteachingtoolset.Thiscreatesadiscontinuityofexperiencebetweenlearningandsummativetesting.This'blackbox'styleseparationalsolimitsthedatasharingtoafinaltaskgrade.Thus,itlimitsthepotentialforbigdatalearninganalyticstoincludefine-graineddetailwhenassessmenttasksareundertakeninseparatesystems.
• Invasivetechnology.Somesolutions,particularlytargetingstudentdevices,enforceundesirableconditionsonstudentssuchasrequiringinvasivelockdownsoftwaretobeinstalledintostudent'spersonaldevicesthatmaycreateconflictswithinsystems.
• Lackofholisticcontrol.Inparticular,solutionsthatutilisedremotesupervisionservicespresentedrisksofexamprotocolbreachesduetothelackofaholisticcontrolsphereandcommonlyutilisedinvasivetechnology(asperpreviouspoint).Ifoutsourcedtestingcentresorserviceswereused,thencontrolispassedtootherorganisationsorindividuals.Theseservicesweremainlyintendedforsmallnumbersofexternalordistanceeducationstudentsandwerefoundtobecostlyperstudent,negatingtheiruseatscale.However,ithasbeenfoundthatoverthedurationoftheprojecttheuseofsuchserviceshasbeenincreasing,withanever-largernumberofcoursesmovingtoonline-onlydelivery.Cautionisurgedintheraceforefficiencyandaccesswheretheintegrityoftheassessmentsystemmaybeplacedinjeopardy.
Itwasfeltnecessarytodevelopanewtypeoftechnologyplatformthatwouldovercomemanyoftheissuesoutlinedabove,whileenablingamuchgreaterdegreeofauthenticityinassessmenttaskswithinthesupervisedcontextofexamhallsandclassrooms.Giventheapparentgapinthemarket,theprojectteamhastakena'pedagogyfirst'approach,buthasmaintainedakeeneyeontheenablingcost-effectiveness,implementationandintegrity.Inmovingtowardsasolution,thewell-knownprojectmanagementaxiomof'good,quickandcheap–picktwo'wasapparentinimplementinge-Examsprojects.Thisisoftenshotintostarkreliefwherecorporatemanagerialprocessesareinstigatedtoselecta'solution'.Itwasobservedthatatensionbetweenthespeedoreaseofimplementation,costandpedagogicalaffordancesoftenledtothelatterlosingout.Thistensionasitappliestoe-ExamtechnologyisrevisitedinFigure2(right).
Figure2Tensionsandtrade-offsindevelopingorselectinge-Examssolutions
2.3 Historicaldevelopmentofarobuste-ExamplatformThee-Examplatformstartedlifein2007attheUniversityofTasmania.Initially,in-classe-testswereconductedbybootingfromCD-ROMandwritinganswerfilestoasecondaryUSBstick.Thismethodcentredonusingword-processingdocumentsasthequestionand
TransformingexamsacrossAustralia |6
responsecompositionenvironment.Thiscontinuedin2008and2009forin-classe-Exams(SeeFluck,citedinLane,2009;Fluck,Pullen,&Harper,2009).In2010,thefirstLiveLinuxUSBstickwasusedtobootWindowsPCswhileCD-ROMscontinuedtobeusedonApplecomputers.In2011,e-ExamswereformalisedwithUniversitySenateapproval.Atthistime,aUSBstickversionwasdevelopedforusewithbothWindowsandApplecomputers.Senateapprovalinitiatedamoveoutofcampuscomputerlabsandintocentrallyadministeredexaminationsvenues.In2013,theOfficeofLearningandTeachingapprovedaseedprojectanddevelopmentbegunatTheUniversityofQueenslandonversionfiveofthee-Examsystem.Thisphaseaimedtostreamlineandautomateaspectsoftheuserexperience,addadditionalsoftwareapplicationsandexpandtherangeofcompatiblehardware.AprototypeofflineMoodleinstancewithinthee-Examsystemwasalsodevelopedin2014(althoughitwasnotreadyforuseinexams).FollowingthesuccessfulawardofanAustralianGovernmentDepartmentofEducationandTrainingInnovationandDevelopmentgrantin2015,upgradestothecoreoperatingsystemandcapabilityofthee-Examsystemwererealised,producingversionsixofthee-Examsystem.EarlyintheInnovationandDevelopmentProjecttheteamarticulatedaphasedroadmap(seeFigure3)forthedevelopmentofsystemcapabilitiesandthenatureofexaminationsthatcouldbeundertakenateachphase.Theroadmapsetoutapathwaytowardsauthentice-assessmentwiththeintentionthateachinstitutionwouldprogressattheirownpace.
Start> >>> >>> >>> >>> >Future>GetReady Phase1 Phase2 Phase3 Phase4 Phase5
Institutionalapprovals,
researchethics,hardwareandinfrastructure
Paper-equivalentsmallscale
Post-papersmalltomedium
Mediumtolargescale
WhitelistedandloggedInternet
OpenbutfullyloggedInternet
Basicdocexamstobegin!
Expandingtheappand
medialandscape
Addingthepowerofan
LMSNetworkedBYODexam
NetworkmixedmodeBYODexam
Crawling Walking Running Jumping Flying!Figure3e-Examsystemdevelopmentandadoptionroadmap
Aseriesoftrialswerethenimplementedthatcoveredphasesonetothree.Thesetrialsincludedusingwordprocessordocuments(inbothphaseoneandtwo),includingmulti-languagetranslationexams;usingaspreadsheetasaformtoenableautomatedmarking(seeHillier&Grant,2018);andanewclient-servermodeforMoodlequiz-basedexams(phasethree).Furtherupdatesincludedcomprehensivesystemlogging(hardwareandsoftwareprocesses)andthedevelopmentofprototypesessionrecordingofuseractions(screenvideorecording,webcam,screenimagesandtextcapture).A'robust'integrationofSafeExamBrowserwithintheLiveLinuxenvironmentwasalsodevelopedthatallowsanexamsessiontocontinueintheeventofnetworkoutages.Livetrialsofthe'robust'Moodleapproachtoe-examsweresuccessful(seeHillier,Grant,&Coleman2018).In2018,theprojectteam,incollaborationwithcolleaguesatBondUniversity,developedaprototypeAugmentedRealitye-Examtocollectspatialdatafromuseractions(Cowling,Hillier,&Birt,2018).BytheendoftheInnovationandDevelopmentProjectin2019theprojectteamwasbeginningnegotiationstotrialthe'robust'e-ExamsystemapproachinalternativelearningmanagementsystemssuchasCanvasandearlydiscussionsofjointventurearrangementswithEuropeanorganisationshadalsobegun.
TransformingexamsacrossAustralia |7
2.4 Thelateste-ExamplatformAttheconclusionofthisprojectthee-ExamplatformhadreachedastateofmaturityhavingbeentestedinphasethreetrialswithMoodle-centricrobustonlineexams.Therearethreemaincomponentsavailableforthee-Examplatform:
• studentuserclient'e-ExamOS'(liveUSB)forofflineor'robust'onlinemode• servertohostMoodleLMSand'robust'plugins• administrationtool(alsoalivesystemUSB)tousewithlargeUSBhubs.
ThesystemasitstandshasanumberofsignificantfeaturesasoutlinedinTable2.
Table2e-ExamsystemfeaturesandaffordancesPertinentfeatures AffordancesA'wholecomputer'environment(operatingsystem,securebrowser,mediaplayersandarangeofsoftwareapplications)asanexamdeliveryenvironmentratherthanasingletestingapplication.
Vastlyexpandedpedagogicallandscapeoverthatofabrowserwindoworsingletestingapplication.Arangeof'e-toolsofthetrade'canbeusedtoconstructsophisticatedresponses.Examplesincludeofficesuite,spreadsheet,languagetranslationdictionaries,programmingkitsuchasScratchorPython,mathematicalapplicationssuchasRorGeoGebra,augmentedrealityapp.
Responsesviaafullwordprocessororconstructedviaspecialistapplications(mainlyhumanmarked)andresponsessecurelyreticulatedviaalearningmanagementsystem(LMS)quizwithcomputerisedmarking.
Catersforintroductiontoadvanceduses,frompaper-equivalenttopost-papermodalities.Softwarecomponentscanbeaddedorremovedtosuit.Computerisedresponsecollectionfacilitatesassessmentanalytics.
Robustagainstnetworkoutages—noorminimallivenetworkconnectivityisrequiredduringexam,evenforLMSquestions.
Notdependantonahighdegreeofnetworkreliabilitywhilemaintainingtheadvantagesofnetworkedcontentdeliveryandresponsesubmission.Canbeusedinisolatedareaswithpoorconnectivity.
Livealternativeoperatingsystemused—meansstudent-ownedequipmentcanbeusedtohosttheexamenvironmentbutisleftuntouched.
Anethicalapproachtoscalability.Thereisnoinvasivesoftwaretoinstall.Allcandidateshaveanequivalentsoftwaretoolsetwithsomeaccessibilityfeaturesbuiltin.
Modular,opensourcecodebaseandcommodity'offtheshelf'components.LeveragingLinuxsecuritywithextensiveloggingandsessionvideorecording.
Leveragingpopularandsustainableprojectsforbetterefficiency.Fully'known'totestadministrators(no'blackbox').Asecureenvironmentwithextensiveauditing.
OneversionworksonmostIntelbasedlaptops—Apple,Windows,Linux,thathaveaUSBport.CanbeusedincampuslabsandBYOlaptops.
Onesoftwareversioncanserveall.Streamlinesdevelopmentandmaintenance.Allexamcandidatesareequippedwiththesamesetofsoftwareapplications.
Userguidesareavailablefor:students(Preparationandpracticeguide),teachers(Preparingexammaterialsguide),examinationprocessadministration(Organisinge-Examsguide),technicaladministration(Administrationtoolguide,Alternativeadministrationguide,Hardwareguide,Laptoprequirementsguide),generaluserguides(Quickstartguides,Detailedstart-upguide).Thestudentclientportionoftheplatformanduserguidescanbeobtainedfromtheprojectwebsiteathttp://transformingexams.com.Theprocessforusingthee-ExamsysteminarobustonlineMoodledeployment(phasethree)isshowninFigure4
TransformingexamsacrossAustralia |8
withfurtherdetailavailableinHillier,GrantandColeman(2018).Detailsoftheworkflowusedforofflinedeployment(asperphaseoneandtwo)isavailableinHillierandFluck(2017).
Figure4e-ExamworkflowforrobustMoodleuse(phasethree)
Exammodalitiesusedtodatewithintheproject(inphasesonetothree)include:1. Paper-equivalentworddocument-basedexam.Primarilythesewereconductedon
opt-inbasisandusedforearlystagetrialsateachinstitution(asperphaseone).Nonetworkinfrastructurewasrequiredtoruntheseexams.Studentsweregivenachoiceoftypingorhandwrittenresponses.Academicspreparedexamsusingawordprocessorandstudentsrespondedwithinacopyofthedocument.
2. Post-paperworddocumentcentricexam.Theserequiredallcandidatestotype(asperphasetwo)andweregenerallyundertakenfollowingseveralroundsofopt-intrialsatagiveninstitution.Post-paperexamsexploredassessmenttasksthatcouldnotbeperformedonpaper(e.g.multimediaprompts,spreadsheettasksorcomputerprogrammingtasks).Nonetworkinfrastructurewasrequiredtoruntheseexams.Academicspreparedexamquestionsusingawordprocessor,butalsoselectedandpreparedadditionalpromptsandresourcestoaccompanythequestions(e.g.video,audioclips,oradditionalsoftwareapplications).Studentsrespondedusingamixoftheexamworddocumentandbysubmittedfilesconstructedusingothersoftwaretools(e.g.aScratchprogrammingfile).
3. Client-serverbased'robust'onlineMoodleexam.TheseweresimilarinnaturetothephasetwoexamsbutwereconductedbasedonaMoodlequizviaSafeExamBrowser.TheseexamsrequirednetworkinfrastructuretobeinplaceintheformofaMoodleserverandWi-Ficonnectivity.Multimediawasalsoutilisedinsomeexams(e.g.languagelisteningtestsusingaudiofilesandearphones).AcademicspreparedexamquestionsusingtheMoodlequiztoolsandembeddedmediafilesasrequired.Selectsupportingsoftwareapplicationswereaddedasrequired(e.g.aChineselanguagetranslationdictionaryapplication).Ademonstrationaugmentedreality(AR)
TransformingexamsacrossAustralia |9
e-ExamwasalsodevelopedwithanAndroidVirtualmachinecontaininganARapploadedintothee-Examsystem.TheARappcollectedspatialdatapertainingtouseractionsinrelationtomarkers.Thesedatawerethentransmittedtoaserverforanalysis.SeeCowling,HillierandBirt(2018)forfurtherinformation.
Additionalfeatureswereprototypedbuthaveyettobetrialledinliveevents.Theseinclude:4. Accesstoexternallywhitelistedinternetresourcesduringanexam,withaccess
logging.Thisrequiresahigherdegreeofnetworkavailabilityforthedurationoftheassessmenteventbecauseanyexternalresourceneedstobeavailablewhenthestudentcallsuponit.AdemonstrationexamfeaturingthistypeofquestionisprovidedontheprojectdemoMoodleserver.
5. Openonlineexamwithextensivesystemanduseractionlogging.Theremovalofrestrictionsoninternetresourcesandtheadditionofextensivesessionrecording.Userloggingcanincludevideoscreenrecording,screenimagecapture,webcamimagecapture,keystrokerecordingandcursormovementrecording.
Incomparingourprogresstothatofothere-examprojectsaroundtheworld(Fluck,2019;Fluck,Adebayo,&Abdulhamid,2017;Fluck&Hillier,2017),werealisewestillhavealottodo,yetitisalsoevidentthatthetechnicalstrategyandpedagogicalgoalswehavesetarealsoachievable.Wehighlightedtwosuchexamplesatoure-ExamSymposiumin2018.Thehome-grown'Abitti'LinuxUSB-basede-examapproachusedinFinlandseniorschoolsandmatriculationexamssaw47,000USBsdistributednationallyin2018.Thenetwork-booted'SEE'LinuxclientandMoodleserverapproachatauniversityinAustriareached50percentpenetrationforcourseexamsduring2018.Bothexamplesshowourideasarescalable.
TransformingexamsacrossAustralia |10
Chapter3. e-Examtrialfindings3.1 ResearchdatacollectionAnethicsprotocolwasapprovedattheleadinstitution.Ethicsapprovalwasalsosoughtbyeachpartnerinstitutionmemberfromtheirlocalethicscommittee.Bothqualitativeandquantitativedatawerecollectedfromstudentusersusinganinformedconsentprocess.Datacollectionmethodsincludedpre–postsessionsurveys,focusgroups,observationandtechnicaldatalogging.Examsessionswereconductedinclassroomsandinexamhalls.
Figure5Photo:arobustonlinee-Examsessioninprogress
Stakeholdersurveysincluded:a) Pre-examsurvey:Doneatpracticesession—twoparts:Technicaldetailsofstudent
laptopequipmentandfirstimpressionsfollowinginitialuseoftheexamsystem.b) Post-examsurvey:Donefollowingarealexamevent.Experienceofane-examand
opinionsaboutaspectsofrespondingtoassessmenttasksandexams.Sometrialsincludedaseriesofexameventsprovidingasmalldataserieswithingroups.
c) Focusgroups:Donefollowingarealexamevent.Semi-structureddiscussionaroundselectedthemesorquestionsrelatingtocandidate'sexperienceofthee-exam.
d) Stakeholderandcommunitysurveys:Donefollowingaworkshoporpresentationtogainfeedbackonsessionrelevance,e-examknowledgeandinterestintheproject.
Atotalof31e-examtrialswereconductedacrosseightinstitutionsfrom2016to2018(seeTable3).Themostrecenttrialsutilisedthe'robust'networkresiliencefeaturesforonlineexamshostedviaaMoodlequizthatincludedwrittenandlisteningaudioresponsetasks(seeFigure5).RefertoAppendixCofthisreportfordetailsofthesetrials,includinghostinstitution,date,course,exammodality,questiontypesused,assessmentweighting,durationandthenumberoftypistsandhand-writersundertakingeachexam.
Table3e-ExamtrialsconductedacrossinstitutionsandyearsInstit'n CQU ECU Monash MQU UniSA UNSW UQ UTAS Total2016 3 2 52017 2 2 9 1 1 1 2 1 192018 1 6 1 2 1 11Total 3 2 18 1 2 1 4 4 35
Participationacrossthe35trialsconsistedofover1700studentstypingtheirexamswith1300electingtohand-writetheirexam(whereachoicewasprovided).Examsrangedfrom15minutesto180minutesindurationandrangedinweightingfrom5percentto50percentofthecoursegrade.ThesefiguresaresummarisedinTable4.
TransformingexamsacrossAustralia |11
Table4e-ExamsummarystatisticsExams Typists Penonpaper Weight MinutesSum 1750 1309 ~ 4145Mean 40 37 32% 106Smallest 1 ~ 5% 15Largest 166 ~ 50% 180
3.2 StudentperspectivesStudentsurveyresultshaveshownpositivefeedbackfromstudentuserswithratingsof4+outof5acrossmostuseracceptancemeasures.Whenaskedinpost-examsurveysifstudentswouldrecommendthee-Examsystemtoothers,alargemajoritywereinagreementwith90percentsayingtheywoulduseitwithoutconcernand70percentpositivelyendorsingthee-examapproach(seeFigure6,left).
“Iwouldrecommendthee-Examsystemtoothers”
“Moodleworkedwellforexams”
Figure6Aggregatedstudentfeedbackonthee-Examsystem
TherecenttrialsalsoaskedforopinionsoftheuseoftheMoodleLMSasanexamenvironment—mostreportedapositiveexperience(seeFigure6,right).Studentswhoparticipatedinrecent'robust'Moodlee-examtrialsinsemesters1and22018providedresponsestoaseriesofsurveyswhereapracticesessionandthreeexameventswererunineachcourse.ThisseriesispresentedinFigure7withfurtherfindingsinHillier,GrantandColeman(2018).Additionalreportsofe-examtrialsincludeHillier(2015),HillierandGrant(2018),HillierandLyon(2018b)andPagram,Cooper,JinandCampbell(2018).
Figure7StudentsurveyresponsesfollowingrobustMoodleexams
TransformingexamsacrossAustralia |12
Arangeofotherissueswerealsoputtostudentsincludingassessment-writingstrategies,habitsandstyles(seeHillier&Lyon,2018a).Whenitcametowritingincomputerisedexamsthetwoissuesmostfrequentlyraisedbystakeholderswerehandwritingneatnessandtypingprowess,withthelatterthoughttobeapotentialsourceofdisadvantagewhenadoptinge-exams.AsseeninFigure8,studentsthoughtthatmessyhandwritingwasthemoreprevalentproblemincomparisontothevastmajorityofstudentswhofelttheirtypingwasspeedwasgoodenoughforane-exam.
“Myhandwritingisneat”
“Mytypingisfastenoughforexams”
Figure8Handwritingneatnessandtypingprowess
3.3 LogisticsfindingsOurexperiencewithrunninge-examsessionsraisedanumberoflogisticalmattersthatneedattention.Feedbackfromstudentsandacademics,aswellaspracticalexperiencefromteammembers,providedindicationsofthevarietyofissuesfacedbyinstitutionsoperatinginverydifferentcontexts,includingin-class,centralexamhalls,multi-campusandexternalvenues.ManyoftheparticipatinginstitutionswererunningaBYOlaptope-Examforthefirsttimeandthisisreflectedtosomedegreeinthefeedbackandintheminicasesofthesetrials(seesection3.4).TheteamsatMonashandUTashadpriorexperienceofexamsandtheirfocuswasonmovingonwardtousepost-paperandnetworkedexammodalities.
Themostrecentphaseofworkduring2018demonstratedtherobustnatureofthetechnicalapproachusedinthisprojectforclient-servercomputerisedexams.Weidentifiedthatonlineexamsfaceasignificantriskoffailureifnetworkconnectivityisrelieduponforthedurationoftheexamsession(Hillier,Grant,&Coleman2018).Wehaveshownhowthee-Examplatformaddressesthisriskwiththeabilitytorecoverfollowingsystemcrashesandnetworkoutageswithoutlossofstudentdataandonlyminimaldisruptiontoworkingtimeintheexam.Thee-Examplatformallowsworktocontinueinanexamevenifthenetworkconnectivityispermanentlylostinasession.Thismeansthatexaminationsneednotbeinterruptedorrescheduledandtimetablesneednotbedisrupted.
Theapproachofusinga'BYOlaptopfirst'approachtoequipmentprovisionhasmeantadditionalchallengesthanwouldhavebeenthecaseifusewerelimitedtoinstitution-ownedcomputers.Theaddedcomplexityisduetothedifferingstart-upstepsrequiredacrossdifferentdevices.Thishasbeenrecognisedwitharecommendationstorunpracticesessionsforstudentsandtheneedtofurtherdevelopsupportresourcessuchasanonlineself-helpserviceforindividualusers.Short'Quickstart'guideshavebeendevelopedtoadviseusershowtostarttheirlaptopwiththee-Examsystem.Anadditional'Detailedstart-upguide'wasdevelopedwithin-depthinstructionsandtroubleshootingadviceforawiderangeofhardwaretypes.Theseareavailablefromtheprojectwebsite.
TransformingexamsacrossAustralia |13
Arangeofguidelinesarerecommendedtominimiseproblemsatvenuesusedforpracticesessionsandexamevents:
Runpre-exampracticesessionstoensurestudentsandstaffareprepared.Checkeachstudenthasacompatiblelaptop,itsbatterylifeissufficientfortheplannedexamdurationandthatthestudentsarefamiliarwiththeprocedures.Keeprecordstoensureexameventscanbeadequatelyprovisionedwithsparelaptops,wiredmice,powersockets,enoughe-ExamUSBsticksandspareWi-Fidongles(whereonlineexamsaretoberun).
WhenBYOlaptopsareused,expecttosupplyapproximately10percentofinstitution-ownedcomputersintheearlystagesofe-examuse.Acomputerlabmaybesetasideforthispurposeoratrolleyoflaptopsmadeavailable.Allowingstudentstobookaloanlaptopisaviablestrategy,especiallyintheearlystagesofe-examintroductionorwherelargercohortsoflowSESstudentsarepresent.Ashiftofresponsibilitycanoccurascohortsbecomeaccustomedtoequipmentrequirements(i.e.placeresponsibilityonstudentstosourceanalternativeworkinglaptop).UTashasshownthatmovingtoapre-certificationprocess(viahelpdeskorself-check)andcompatibilitycertificateisalsopossiblewithmaturityofane-examprogram.
Providepowersocketstoreducerelianceonbatterypower—ideallyonesocketperstudent,butthisratiomaybereducedforshorterexamsandwherestudentshavebecomemoreaccustomedtotherequirementsofe-exampractice.Powerextensioncordsandmulti-outletpowerboardscansupplementsuppliesinspaceswhereinstalledsocketsmaybefewinnumber.Inexamhallswithrowsofdesksitisrecommendedtoreserveeachsecondrowforpowersupplycablestoavoidtrippinghazardswhilestillallowinginvigilatoraccesstoeachstudent(seeFigure9,left).Surveysof510studentparticipantsconductedduringprojecttrials(2016–2018)askedforanecdotalreportsregardingthebatterylifeoftheirlaptopfromafullcharge(SeeFigure9,right).Ameanof4.9hourswasfound;however,therewasawiderangefromzerotoeight(plus)hoursreportedwithastandarddeviationof2.2hours.Thedatawereabnormallydistributedandthislikelyreflectsthefactthatstudentscomealongwitholdandnewlaptopstotheexaminvaryingconditions.Ifweexcludenewcomputers(thespikeof8+hours)thenthemeanbatterylifeisfourhours.Thosereportinglowbatterylifeoftwohoursorlessaccountedfor16percentofthegroup.Itmustalsobenotedthatbatteryconsumptioncanbehigherwhenusingthee-ExamUSBsystemorWi-Fi.
Layingpowercablesinanexamhall Studentself-reportedlaptopbatterylifeFigure9Powersupplyfore-Exams:layingcablesandbatterylife
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8+
Num
berofusers
Bawerylifeinhours
Self-reportedlaptopba]erylife
TransformingexamsacrossAustralia |14
Roomsandseatingarrangements:• Utiliseflatfloorroomswithchairsandtables• Wheregrouptablesareusedhavestudentssitat90degreestoeachother• Possibility:addcardboardcrossscreensordividersontables• Possibility:userandomorderforquestionsanddistractorswhereavailable• Whererowsareused,seatingallfacingfront>>>>>aresatisfactoryoralternating
face-to-face,back-to-back<><><><>mayreducethevisibilityofother'sscreens• Avoid'flappychairs'duetothembeingunstableforlaptopsandprovideinsufficient
spaceforlaptopsandotherexammaterials• Avoidtightlypackedtieredroomsduetorestrictedaccesstoeachstudentandthe
potentialforseeingother'sscreens• Preferroomswithquietacoustics—carpetedfloorsarerecommended.Outof1033
respondents,51percentcouldheartyping,ofthose28percentfounditdistracting(12percentofthewholegroup)
• Considerspacetoallowtypistsandhand-writerstositseparately• Itisimportantthatroombookingsaretimelytoensureasuitablevenueissecured.
GoodWi-Fiorwirednetworkingisneededwhenusingnetworkede-exams.However,ifusingthe'robust'e-examsystem,networkconnectivityismostimportantatthestartoftheexameventandcertainlyconvenientattheendoftheeventforresponsesubmission.ThetrialsconductedatMonashUniversityfoundWi-FitobeaviableoptionforBYOlaptop-basedexams.TheWi-Ficompatibilityoflaptopswasfoundtobelowerthanforthee-Examsystemingeneral.AsetofUSBWi-Fidongles($12eachfromcomputerpartsretailers)canbeusedtoboostthestockofcompatiblelaptopsandarequicktodeployintheexamroom.
Multi-campusandexternalsitesneedextraprecision.Ensureadequatetimeisallowedforsendinge-ExamUSBstootherlocations,thatadditionalspareUSBsaresentandthatclearrulesandcommunicationchannelsareestablished.Trialsatmulti-campusinstitutionsoratoff-sitelocationsexperiencedproblemswhentheseelementswereneglected.
3.4 MinicasesAseriesofeightminicasesareavailablefromtheprojectwebsitethatrepresenttherangeofe-exammodalitiesthatwereundertaken.Thecasesinclude:
• CentralQueenslandUniversitysemester12017:Paper-equivalentICTmanagementexamconductedatmultiplecentrallycontrolledon-campusvenues.
• EdithCowanUniversitysemester12017:Paper-equivalentexamindesigneducationandpost-paperexaminprogrammingeducationconductedin-class,oncampus.
• MacquarieUniversitysemester12017:Paper-equivalentICTeducationexam.• MonashUniversitysemester12017:Paper-equivalentexamusingaspreadsheetas
adatacollectionformwith'airgap'separationofresponsecollectionandevaluation.• MonashUniversitysemester12018:Post-paper'robust'onlineMoodleexamsthat
wereresistanttonetworkoutages.Moodlequizincludingaudioprompts.• TheUniversityofQueenslandsemester12017:Paper-equivalentFrenchlanguage
translationexamconductedin-class,oncampus.• UniversityofSouthAustraliasemester12017:Paper-equivalentmathseducation
theoryexamconductedoncampus.• UniversityofTasmaniasemester12017:Post-paperworddocumentexamsinICT
educationconductedovermultiplecentrallycontrolledvenuesanddistancecentres.
TransformingexamsacrossAustralia |15
Chapter4. DisseminationAsignificantfeatureofthisprojectwasitsstrongdisseminationstrategy.Theprojectleaderandteammembersprovidednumerouspresentationsandconductedseveralworkshops,bothlocallyandinternationallyinconferencesandotherprofessionalforums.ThesedisseminationactivitiescreatedawarenessandgeneratedinterestfromhighereducationorganisationsacrossAustraliaandothercountries.Themaindisseminationstrategiesareoutlinedinthefollowingsections.
4.1 WebsiteandsocialmediaAprojectwebsitewasestablishedathttp://transformingexams.comtoprovidepublicinformationandmostoftheprojectdeliverables.Itincludeslinkstoinformation,softwaredownloads,documentation,presentations,publicationsandnotificationsforevents.Overthelifeofthewebsiteitreceived16,793visitorsand37,478pageviewswererecordedatthehomepage.VisitorsfromAustraliamadeupapproximatelyonethirdoftheaudience.Thetop10countriesareshowninFigure10.Figure10Websitevisitorsbycountry
SocialmediasuchasLinkedIn,TwitterandTransformingAssessmentemaillistwereusedtopromoteprojecteventssuchaspresentations,theroadshowworkshopsande-Examsymposium.
4.2 PresentationsandpublicationsFormaloutputsofresearchandprojectprogressweredisseminatedvia:
• over60presentationsofferedatconferencesandaspartofprivatebriefingsessionsatindividualinstitutions
• atotalof15refereedpublicationsproducedfrom2015to2018.Furtherpublicationsarebeingpreparedbytheprojectteam
• additionalinformalposters,documentsanddiagramsweremadeavailable.Afulllistofdisseminationeventsandpublishedoutputsarelistedathttp://transformingexams.com/research.html
Theseoutputsservedtospreadnewsabouttheprojectanddisseminatefindingsfromtrialsastheprojectprogressed.Thisalsoprovidedopportunitiesfornetworking,gatheringofinterestprojectamongcolleaguesandelicitingfeedbacktotheprojectteamonprogress.
Thewidereachofthepresentationsgivenbyprojectmembersisdemonstratedinthelistofconferenceswherepresentationswereoffered.Theseinclude:ACMInternationalComputingEducationResearch(ICER)Conference2016,AustralianCouncilforComputersinEducation2016,AustralianGovernmentOfficeforLearningandTeachingConference2016,AustralianLearningAnalyticsSummerInstitute2016,AustralasianSocietyforComputersinLearninginTertiaryEducationconference2017and2018,Assessmentintegrityandtheroleofexaminations2017,AssessmentinHigherEducationconference2017,EDTechPosium(ACT)2017and2018,ExaminationsNetworkConference2017and2018,HigherEducationResearchandDevelopmentSocietyofAustralasia2015and2018,InternationalConferenceone-LearningandInnovativePedagogiesConference2019,InternationalComputingEducationResearch2016,InternationalFederationofInformationProcessingOpenConferenceforComputersinEducation(Austria)2018,InternationalFederationof
Figure10Websitevisitorsbycountry
TransformingexamsacrossAustralia |16
InformationProcessingWorldConferenceonComputersinEducation(Ireland)2017,OpenandDistanceLearningAssociationofAustraliaBiannualconference2017,TertiaryEducationQualityStandardsAuthorityconference2018andMoodleMoot2018.
4.3 e-ExamcasestudiesAseriesofeightminicasestudiesweredevelopedtooutlinethecharacteristicsandoutcomesofselectede-examtrialsconductedatdifferentpartnerinstitutions.Eachone-pagecasestudyshowcasesane-examusingdifferentaformatincluding:offlineword-processingdocument,spreadsheetasaform,computerprogrammingandonlineLMSuse.Thecasescanbeobtainedfromtheprojectwebsite.
4.4 e-Examworkshopse-Examshands-onworkshopswereconductedforthehighereducationcommunityandforprojectteammembers.Thesesessionswere:
17February2017:‘e-Examprojectpartnersworkshop’,heldatMonashUniversity,Caulfieldcampus,Melbourne.Thisprivateworkshopprovidedanoverviewforprojectpartnerspriortorunninglivetrialsattheirsites.8September2017:‘Runningauthentice-examsworkshop’atExaminationsNetworkconferenceinHobart.Thispublicworkshopfocusedone-examadministrationandlogisticsforexamsofficersatAustralianuniversities.7December2017:‘Transformingexams—HowITworksforBYODe-Exams’,ASCILITEconference,Toowoomba.Thispublicworkshopprovidedanopportunityforsomehands-ontimewithdifferenttypesofassessmentsthatarepossiblewithe-Exams.Thesessiontargetedacademics,learningdesignersandadministrativestaff.
4.5 e-ExamroadshowAseriesofroadshowpresentationsandworkshopswereconductedin2018andwerehostedbyprojectpartnerinstitutionscoveringmostcapitalcitiesinAustralia.Sessionswerefreetotheeducationcommunity.Keyteachingandlearningleaders,academicsandsupportstaffwereinvitedfromlocalinstitutionswithinthehoststate.Invitationswereissuedtostaffinuniversities,vocationaleducationprovidersandstateeducationauthorities.
Eachroadshowconsistedoftwoone-hoursessions.Thefirstsessionexploredtherationaleandsurroundingissuesfore-Exams.Thesecondsessionexploredpracticalapplicationsofe-examsandincludedhands-ontime.Theprojectleaderledthesessionsandwasoftenjoinedbymembersofthelocalprojectteamatthehostinstitution.Anexampleprogramisbelow:
1hour:Part1:Rationaleforauthentice-Exams.Forleaders/decisionmakers,allwelcome• Anargumentforauthentice-Examsforemployability.• Abusinesscaseforauthentice-Exams.• Keyfindingsonstudentfeedback.• Latestdevelopmentson'robust'Moodleuse(phase3development).
30min.break.(cateredfood/drink).1hour:Part2:Exploringthedetailsofe-Exams.• Workshopthefindingsonstudentfeedback,logistics,examplesofassessments.• Workshopaselectionofproposedprojectresources(e.g.samplecasestudiesto
influencedecisionmakers).• Timeforpractical'hands-on'withour'robust'Moodlee-Examsystem.
TransformingexamsacrossAustralia |17
Thelocations,datesandregistrationnumbersforroadshowsessionswere:Adelaide(31August:50+30online),Brisbane(24September:27),Canberra(31October:29),Perth(17September:40),andSydney(21September:23),Hobart(5October:20).TheAdelaidesessionprovidedonlineaccessvialivewebvideoconferencingforthefirstsession.TheHobartsessionwasalsobroadcastviavideoconferencetoLaunceston(Newnham),BurnieandCradleCoastcampusesofUniversityofTasmania.
4.6 e-ExamssymposiumThee-ExamSymposiumwasorganisedbytheprojectteamandheldon24November2018asafull-dayeventinMelbourneatMonashUniversity,Caulfieldcampus.KeyrepresentativesfromeachAustralianuniversity,stateeducationdepartments,medicalcollegesandothereducationorganisationswereinvited.Theninespeakersand86participantsfromAustralia,Austria,Finland,NewZealandandSwedenrepresented23universities,twoprivateproviders,fourprofessionalmedicalcolleges,fourstategovernmenteducationdepartments(Victoria,NewSouthWales,Tasmania,WesternAustralia)andtheTertiaryEducationQualityandStandardsAgency(TEQSA).
Aseriesofplenary-stylepresentations,discussionsectionsandapracticalhands-onsessioncoveredmultipleaspectsofimplementinge-exams(seeFigure11).Twointernationalguestspeakerspresentedontheimplementationofe-examsintheircontextusinghome-developedapproachesandtechnologythatfocusedonauthentice-assessment.ThisincludedcomputerisednationalmatriculationexamsinFinlandanduniversityexaminationsinAustria.Themajorityofprojectpartnersalsopresentedorattendedtheevent.
Figure11Photo:e-Examsymposiumsessions24November2018
Thesymposiumprogramisoutlinedbelow.Welcome:Arationaleforafocusonauthentice-examsandprogramoverview.Session1:TheAustrianexperiencewithe-Exams.GabrieleFrankl(Alpen-Adria
Universität,Klagenfurt,Austria)Session2:Thepedagogyofe-Exams—examplesandtransition.MathewHillier(Monash
University),MatthewBower(MacquarieUniversity)andAndrewFluck(UniversityofTasmania).
Session3:Qualitativeanalysis,studentconcerns:thegoodthebadandtheugly.DavidMeacheam(UniversityofNewSouthWales,Canberra/ADFA),AmyHubbell(TheUniversityofQueensland)andCristinaSavin(Student,MonashUniversity).
Session4:Anationalrolloutofe-examsforhighstakesMatriculation.ThomasVikberg(FinnishMatriculationExaminationBoard,Finland)
TransformingexamsacrossAustralia |18
Session5:Trade-offs,benefitsandcosts.AndrewFluck(UniversityofTasmania)andMathewHillier(MonashUniversity)
Session6:e-Examssecurityandintegrity.MichaelCowlingandKennethHowah(CentralQueenslandUniversity)
Session7:Runninge-Examsoncampusandatadistance.AndrewFluck(UniversityofTasmania)andMathewHillier(MonashUniversity).
Session8:Handson!Technologyformovingfrompapertoauthentice-assessment.MathewHillier(MonashUniversity).
Session9:Ane-ExamsSociety?Anopendiscussiononwheretonextforthee-Examsproject.MathewHillier(MonashUniversity)andAndrewFluck(UniversityofTasmania)[Notrecorded].
Thesymposiumproceedingswerevideorecordedandplacedonlinewithopenaccess.Anoutlineofeachsession(1to8),slideset,linksandvideosareavailableat:http://transformingassessment.com/e-exam_symposium_2018.php
TransformingexamsacrossAustralia |19
Chapter5. ImpactTheprojecthashadadirectimpactintermsofdevelopingknowledgeofe-examsimplementationandtheadoptionofprojectdeliverablesinasmallnumberofcases,butlargerimpactsmaybecomeevidentinthefuture.Itisstillearlydaysforthefullimpactoftheprojectworktobecomeapparent.Thecurrentstateofimpactasatthecloseoftheprojectisoutlinedbelow.AnannotatedIMPELframeworkispresentedinAppendixD.
5.1 ChangedpracticesTheprojecthashaddirectimpactintermsofknowledgeofparticipatingacademicsthatwereengagedintherunningofe-examtrialsattheirrespectiveinstitutions.Theoutcomesvariedaccordingtocontext.Someacademicscontinuetousee-Examapproachesdevelopedaspartoftheproject.TheseincludeattheUniversityofTasmaniaineducation,atMonashUniversityinlanguagesandattheUniversityofSouthAustraliainmathematicseducation.Inothercases,lessonslearntprovidedavaluablebasisforfutureconsiderationofe-examsadoptionincontext.Theseoutcomeswerereportedaspartofthee-Examssymposiumandviapublications.
5.2 Othere-examprojectsTheprojectteamwasconsultedbyotherorganisationsintheAustraliancommunityonmattersrelatingtoe-exams.Projectsorevaluationsrelatingtoe-examsatotheruniversitiesandorganisationswereindependentlyinitiated,someoftheseonlydiscoveredafterthefact.Oftenproponentsfoundourpublications,ourprojectwebsiteorweretoldaboutourworkviawordofmouth.Examplesinclude:UNSWSydneywhereseparateprojectsoccurredinlawandmedicine.AtUNSWCanberrawholaterjoinedtheprojectformallyandatUQinmedicine,whosoughtadvice.AtNAATIwithwhomaspin-offprojectwasdevelopedtointegrateamulti-languagecapability.AcommercialcomputerisedtestingvendorintheUnitedStateshassoughtconsultancyandadvicefromtheprojectleader.Additionally,aprofessionalaccountingbodyrecentlysoughtconsultancy.Wehavealsoreachedagreementtopilot'robust'e-ExamswiththeLMS(Canvas)during2019ofastategovernmenteducationdepartment.
5.3 PartnershipwithNAATIAsignificantpartnershipwasdevelopedwiththeNationalAccreditationAuthorityforTranslatorsandInterpreters(NAATI)inparallelwiththisproject.NAATIcommissionedworktowardsbuildingadditionalmulti-languagecapabilitiesintothee-Examsystemtobettercaterformulti-languagetranslationexaminations.PreliminaryroundsoflivetrialswereconductedatMonashUniversityin2017and2018coveringadefinedsetoflanguages.Atthetimeofwriting,workisongoingtofurtherenhancee-Examsystemcapabilities.Lessonslearntfromtheapplicationofe-ExamstechnologyinthespecificcontextofprofessionalaccreditationforlanguagetranslatorshasalreadyhadadirectimpactofhowNAATIisconsideringtheconductofcomputerisedexamsinthefuture.Alarge,high-levelimpactcouldbeexpectedintheareaoflanguagetranslatortestinginAustraliashouldtheoutcomesofthisworkbeadaptedbyNAATI.
5.4 PotentialjointventurepartnershipandcommercialisationSignificantinterestwasgarneredfromseveralorganisationsoutsideofAustraliawithaview
TransformingexamsacrossAustralia |20
topartneringinajointventureorcommercialisationarrangement.Atthetimeofwriting,twomembersoftheprojectteamareindiscussionwithanorganisationfromtheEuropeanUniontopartnerinajointventurewithaviewtocommercialisingelementsofthetechnologyandapproachesdevelopedaspartofthisproject.
5.5 FurtherresearchandjointprojectworkSignificantinterestwasreceivedfromastate-leveleducationauthorityregardingextendingthe'robust'client-serveraspectoftheonlinee-examstechnologyforlocalusewiththeirLMS(Canvas).Asmallfeasibilitystudyisunderwayatthetimeofwritingwithplansforasmall-scaletrialinmid-2019.
TransformingexamsacrossAustralia |21
Chapter6. ProjectevaluationandlessonslearntWhiletheprojectwasultimatelysuccessfulinachievingouraimofenablingauthentice-assessmentinexamhallsandclassrooms,someareasofworkweremorechallengingthanfirstanticipated.Formalexternalprojectevaluationandinternaldeliberationshighlightedsuccesses,challengesandlessonslearnt.
6.1 ProjectexternalevaluationTheprojectmanagementteamheldmeetingswiththeexternalevaluatorswhowerebriefedontheprogressoftheproject.Anarms-lengthfinalevaluationwasalsoconducted.Theexternalevaluationsassistedinthelearningandimprovementofprojectprocessesandoutcomes.
Twoexternalevaluationswereproduced:
I. Aformativemid-projectreportwasproducedin2017.Thiswasformulatedasasurveyofstakeholderviewsbytheexternalprojectevaluator.Thereportoutlinedpointsofsuccessandissuesfortheprojectteamtoaddressgoingforward.Dataweregatheredviasurveyingandtelephoneinterviewswithprojectstakeholders.AsummaryofthisformativeexternalevaluationisincludedasAppendixE.
II. Asummativefinalreportproducedattheendoftheprojectin2019.RefertoAppendixFforthefinalevaluationreport.
Astheprojectprogressedtwomainareasemergedthatweredifferenttoinitialexpectations.Thefirstofthesewasthenatureofchangeneededininstitutionalpoliciesandtheotherwasinthedetailsoftheunexpectedlogisticalchallengeswefaced.
6.2 PolicyadjustmentOneoftheproposeddeliverablesfromtheprojectwastoprovidepolicychangerecommendations.Thiswasbotheasyanddifficultatthesametime.
Firsttheeasypart!Followingareviewofseveralinstitutionspoliciesitwasfoundthatverylittledirectpolicyonexaminationsstoodinthewayofusingcomputersinexamsatmostuniversities.Typically,currentpoliciesorproceduresspecificallyrelatedtotheconductofexaminationshadphrasessuchas‘Noelectronicdevicesarepermittedintheexaminationvenues’.Minoradjustmentstopolicywordingcanclearthewayforcomputerstobeusedinexams.Forexample,rewordingsuchphrasesto:‘Nounauthorisedelectronicdevicesarepermittedintheexaminationvenues’or‘Onlyspecificallyauthorisedelectronicdevicesarepermittedintheexaminationvenues’.Itisthennecessarytoestablishaprocessforauthorisingelectronicdevices.OncesuchexamplewasdevelopedattheUniversityofTasmania.Astudentself-helppracticeandlaptop-testingprocessprovideda'compliancecertificate'attestingtothestudent'ssuccessatstartingandusingthee-ExamUSBsystemwiththelaptop.Thecompliancecertificateisthenshownbythestudentattheexamvenue.
Institutionsmaybeinclinedtoshowactivesupportfortheuseofcomputersinexaminationsthroughaformofofficialendorsementatseniorlevels.Examplesinclude:theUniversityofTasmaniasenatepassingaresolutiontosupporttheuseofe-Exams.AtTheUniversityofQueenslandamotionwaspassedbythecentralassessmentcommitteetosupportandplaceconditionsaroundthetriallingofe-Exams(sincelifted).However,otherinstitutionsinvolvedintheprojecthavenotseenitnecessarytoformallyendorseorplace
TransformingexamsacrossAustralia |22
conditionsontrials.Inthesecases,theethicsprotocolandresearchsubcontractwasseenasadequatetoframeparticipationintheproject.
Difficultiesarise,however,increatingtheconditiontosupportofbroaderadoption.Moresystemicpolicyandmanagerialattitudeswerefoundtobesignificantinmovingonfromintroductionofe-Examsataninstitutiontobroader,centralisedsupport.Thepolicylandscapeisinterconnectedandinterdependent.Thiscansupportordetractfrome-Examsimplementationanduse.Contextualpolicies,proceduresandofficialendorsementisrequiredtoactivelyandsystemicallysupporttheuseofcomputersinexamsatinstitutions.Examplesincludedevelopinga'BYODstrategy'tocoverinfrastructure(power,Wi-Fiandnetworking),an'equipmentequityprogram','equipmentcertification',additionstoexamclassifications,additionstopermittedmaterialslists,newe-examtemplates,developmentofsupportresourcesandhelpdesksforstudentsaswellasprofessionaldevelopmentforacademicandprofessionalstaff.Muchofthisvaluablework,however,wasbeyondthescopeandresourcesofthecurrentprojecttoaddress.
Onapositivenoteforthefuture,ithasbecomeapparentthatsincethebeginningofthisprojectinstitutionshavebeguntodevelopbroader'digitallearning'strategiesthathaveencompassede-examsore-assessmentimplementation.AnexampleofthisisatMonashUniversitywherea'DigitalLearningStrategy'androadmapwasdevelopedthatincluded'Authentice-Assessment'asoneofthetargetitemsalongsidea'BYODstrategy'.
6.3 LogisticsandcommunicationswithinpartnerinstitutionsTheimplementationofe-examtrialsatpartnerinstitutionssawobstaclesencounteredthatgenerallyarosefromthedivergentobjectivesofindividualgroupswithinanorganisation,aswellaslongchainsofcommunicationwherekeymessagesbecamelostordiluted.Examplesofmanifestedproblemsthatoccurredattrialpartnersinclude(innoparticularorder):
Thecentralexaminationsofficehaditsownsurveyforstudents.Thiswasthecaseatonepartnerinstitutionthatimpactedacoupleoftrialsrunatthatsite.Inprevioustrialsatthesamesitetheissuedidnotarise;however,theexamsofficethemselvesdecidedtoevaluatee-examsandsotheyproducedtheirownstudentsurvey,whichdemonstratedhandwritingcandidatesadjacenttoe-Examcandidateswereunaffected.Alackoftwo-waycommunicationresultedinconfusionforstaffinexamroomsthatmeantthatoursurveywasnotprovidedtostudents.Thishighlightsaneedforcontinuedandclosecollaborationwithexamvenueoperatorsforeachtrialinstance.Itcannotbeassumedthatbecauseanarrangementwasinplacepreviouslythatitwouldcontinueinthefollowingsemester.Hadbettercommunicationoccurreditwouldhavebeenpossibletodevelopacombinedinstrumentthatwouldhaveservedtheneedsofbothparties.Fortunately,dataweresuccessfullycollectedduringthemajorityofthetrials.
Lowstudentsurveyresponseratesatsomevenues,oftenduetodelayedsurveyavailabilitytostudents.Insomecasessurveyswerenotadministeredtostudentsimmediatelyfollowingexameventsanddramaticallyreducedresponserateswereexperienced.Itwasrecommendedthatthesurveybehandedtoeachstudentatthetimetheycompletedtheassessment(i.e.intheexamroomitself).Thisneededtobearrangedwiththegrouprunningtheexamvenue(e.g.examinationsoffice)andforthemtocollaboratewiththeprocess,suchastoprovideanannouncementattheendoftheexamtocompletethesurveyformandallowamemberoftheprojectteamtobeon-site,ifpossible,todistributeandcollectsurveyforms.
TransformingexamsacrossAustralia |23
Equipmentprovisionandbackup.BYODwasfoundtobeasuccessfulequipmentprovisionstrategybutitwasnotwithoutitsproblems.Usingself-certificationoflaptopsandpre-exampracticesessionsishighlyrecommendedtofacilitatepreparednessofstudentsforBYODe-exams.Intheearlyphasesofimplementinge-examsthereislikelytobeaheightenedneedtoprovidespareorbackuplaptops,withusersinneedbestidentifiedviapracticesessions.Hardwarecompatibilitywasfoundtobeanissuein10to20percentofcases.Similarly,provisionofelectricitysocketstoeachstudentisrecommended;however,theratiosofpowerandbackuplaptopsmayberelaxedasstudentsbecomeaccustomedtoneedingareliablelaptopforexams.Mostpracticalissuesencounteredinexamroomsduringtheearlystagesofe-Examusecanbegreatlyalleviatedbyprovidingbackupequipment,powerandenforcingpracticesessionattendance.
Provisionofsparelaptopsatremoteexamcentres.Someinstitutionsfounditdifficultduetoresourcesnotbeingallocatedorundesirabletoprovidebackupandon-sitesupportweremultiplevenuesorexternalexamcentreswereused.Atoneinstitution,ane-examwasofferedindistancemodeatanoff-campusexternalexamcentre(e.g.anotheruniversitybyagreementorsomeotherpremisesbyarrangement),however,noarrangementwasputinplacefortheprovisionofsparelaptops.Thiswasexacerbatedbecausenopre-exampracticeorsystemcheckswereconducted.Thisresultedinsomestudentsneedingtoreverttopaper-basedexamsonthedayoftheexam.Itisacknowledgedthatitisalogisticallydifficultissuetoovercome,however,itissomethingthatmustbeaddressedife-examsaretobewidelyavailable.
Expectationsabouttechnicalmaturity.Theneedforguidanceanddocumentationforalluserswashighlighted(students,teachersandexamvenuestaff).Duringtheprojectthee-Examsystemwasstillundergoingactivedevelopmentwhereiterativeimprovementstothesystemwerebeingmadefollowingeachtrial.Thesecircumstancesmeantthatthosetriallingthesystemforthefirsttimeweremorelikelytoencounterproblemsthathadnotyetbeenencounteredbyothers.Furthermore,localorinstitutionaltechnicalsupportvariedmeaningthatthosewhowereleadingtrialshadtobesubstantiallyself-reliant,beyondwhatwouldbeexpectedifthee-Examsystemweretobeaninstitutionallysupportedtool.Althoughallpartnerswereclearlymadeawareofthesecircumstances,issuestendedtoarisewheretechnologyskillsorlocalsupportwaslackingorwherestakeholderssoughttocomparearesearchprojecttothatofamatureproductbackedbyafullservicesupportcontract.Wherethesecircumstancesprevailedthistendedtoresultinadegreeofdisappointment.Supportiveelements,suchasongoingrefinementoftheuserdocumentation,wasundertakentomatchchangesinsoftwareandhardwarecapabilitiesbuttheiterativenatureofthedevelopment,limitedprojectresourcesandthetimingofindividualtrialsdidnotalwaysresultinalignment.Furtherworkisalsoneededtoaddressthewishesandneedsofusers.Thereisalsoawordofcautionintheselectionofearlytrialpartnerswhoneedahighdegreeofresilienceintacklingtheunknowninnewcontexts.Thegoodnewsisthatthatforthosewhowerepersistent,goodoutcomeswereachieved!
TransformingexamsacrossAustralia |24
Chapter7. ConclusionThisprojecthadgreatsuccessindemonstratingwhatrobust,authenticdigitalassessmentcouldbelike.Butthefindingsshowourworkisnotyetcomplete—notechnologydevelopmentprojectisevercomplete!Furthertechnicalworkisrequiredtoimprovethee-Examtechnologyplatform,itsintegrationwithothersystemsanditsadministrativesupporttools.Whilewehavelaidthegroundworkandhavealreadyachievedmuchonthepedagogicalfront,moreremainstobedonetoenhancethescalabilityandefficiencyoftheapproach.Furtherinvestigationandinvestmentisrequiredtoensuresystemicsupportmechanismscanbedevelopedtoscaledigitalassessmentathighereducationinstitutions.Furtherworkisneededonrefiningprocesses,professionaldevelopmentcapabilities,studentsupport,alignmentofvariousinstitutionallearningandtechnologystrategiesandpolicy;aswellasenhancingtechnologyinfrastructuretocopewiththehigherdemandsoftime-limited,higherstakesauthentice-assessment.Finlandhasdemonstratedthepotentialofanationalapproach—thismayalsoberequiredinAustralia.Tappingintointernationalandcross-sectorcollaborationinvolvingschools,vocationaleducation,professionallearningandinternationalhighereducationsettingswillenablefurtherleveragingofthelessonslearntinthisproject,aswellaslearningfromother'ssuccessindeployingdigitalassessment.
Lookingtothefuture,thereismuchpotentialtoleveragedevelopmentsinfieldssuchaslearninganalytics,openknowledgesystems,mixedmediatechnologiesorusingblockchaintechnologytoenhancesecurityandauditcapabilitiesine-Exams.Whilethesearerelativelyunderexploitedareastheyholdgreatpromise,particularlyiftheintegrated,openarchitectureadvocatedinourapproachtothee-Examplatformcanbecomemorewidespread.However,Australianhighereducationisnowatacrossroadswithsignificantrisksforinstitutionsinhowtheytransitiontoagreateruseofdigitalassessment.Onesuchexampleistheuseofmanagerialisttechnologyselectionapproachesthattendtodiscountpedagogicalaffordancesforwhatappeartobequickandeasy-to-managesolutions.Similarly,adecisiontooutsourceassessmentsystemspresentsrealstrategicriskthatinstitutionscouldloseholdoftheassessmentprocesstoexternalinterests,ashasbeenthecaseinresearchpublishingandlearningcontent.Assessmentisoneofthejewelsinthecrownoflearningandteachingthatinstitutionsstillholdwithintheirgraspandisarguablythemostvaluableelementinthestudentlearninglandscape.Institutionswoulddowelltoconsiderverycarefullythestrategytheyadoptinmovingtodigitisingassessment.Ifthetruevalueandbenefitsofdigitalassessmentaretoberealised,institutionsmusttakebackownershipoftheireducationaltechnologyinfrastructureandreinvestinmorehome-growndevelopmentandinnovation,justastheydointhediscipline-basedresearchdomain.WeurgeAustralianhighereducationinstitutionstostakeaninvestedclaimindigitallearningandassessmentbecausethatwillgoalongwaytosecuringthefutureofeducationfortheirstudents,staffandwidersociety.
Theprojectgeneratedfindingsone-Examsystemdesignanduse,exammanagementandlogistics,students'perspectivesone-Examsandontheirresponsestrategies.Thesefindingsequipteachers,assessmentdevelopers,technologysupport,examinationsmanagersandinstitutionalleadershiptomakeinformeddecisionsinmattersofpolicy,practicalimplementationandtransformationalpedagogicdesign.Thiswillenableinstitutionstodeliverrich,relevant,assessmentfitforlearningandteachinginthe21stcentury.
TransformingexamsacrossAustralia |25
ReferencesCowling,M.A.,Hillier,M.,&Birt,J.R.(2018,Nov).Integratingmixedrealityspatiallearning
analyticsintosecureelectronicexams.InM.Campbell,J.Willems,C.Adachi,D.Blake,I.Doherty,S.Krishnan,S.Macfarlane,L.Ngo,M.O’Donnell,S.Palmer,L.Riddell,I.Story,H.Suri,&J.Tai(Eds.),ASCILITE2018ConferenceProceedings—OpenOceans:Learningwithoutborders(pp.330-334).Geelong,ASCILITE.http://transformingexams.com/files/cowling-hillier-birt-ASCILITE-2018.pdf
Fluck,A.(2019).AninternationalreviewofeExamtechnologiesandimpact.Computers&Education,132,1–15.https://doi.org/10.1016/j.compedu.2018.12.008
Fluck,A.,&Hillier,M.(2016,Sep/Oct).InnovativeassessmentwitheExams.PaperpresentedatAustralianCouncilforComputersinEducationConference,Brisbane.http://conference.acce.edu.au/index.php/acce/acce2016/paper/view/34/27
Fluck,A.,&Hillier,M.(2017,July).eExams:Strengthindiversity.PaperpresentedattheWorldConferenceforComputersinEducation(IFIPTC3),Dublin,Ireland.http://transformingexams.com/files/fluck_hillier_2017_eExams_strength_diversity_IFIP_WCCE_paper_45.pdf
Fluck,A.,Adebayo,O.S.,&Abdulhamid,S.M.(2017).Securee-examinationsystemscompared:Casestudiesfromtwocountries.JournalofInformationTechnologyEducation:InnovationsinPractice,16,107–125.https://doi.org/10.28945/3705
Fluck,A.,Pullen,D.L.,&Harper,C.(2009).Casestudyofacomputerbasedexaminationsystem.AustralasianJournalofEducationalTechnology,25(4),509–523.https://ajet.org.au/index.php/AJET/article/view/1126/379
Hillier,M.(2015,Nov/Dec).Totypeorhandwrite:Student'sexperienceacrosssixe-Examtrials.PaperpresentedatAustralasianSocietyforComputersinLearninginTertiaryEducationconference,Perth.http://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf
Hillier,M.,&Fluck,A(2013,December).Arguingagainfore-Examsforhighexaminations.Proceedingsofthe30thAustralasianSocietyforComputersinLearninginTertiaryEducation,(pp.385–397).MacquarieUniversitySydney,Australia.http://transformingexams.com/files/hillier_fluck_2013_ascilite_fullpaper.pdf
Hillier,M.,&Fluck,A.(2015,Nov/Dec).Apedagogicalendgameforexams:Alook10yearsintothefutureofhighstakesassessment.PaperpresentedatAustralasianSocietyforComputersinLearninginTertiaryEducationconference,Perth.http://transformingexams.com/files/hillier_fluck_2015_exam_futures.pdf
Hillier,M.,&Fluck,A.(2017,Dec).Transformingexams—HowITworksforBYODe-Exams.InH.Partridge,K.Davis,&J.Thomas.(Eds.),Me,Us,IT!ProceedingsASCILITE2017:34thInternationalConferenceonInnovation,PracticeandResearchintheUseofEducationalTechnologiesinTertiaryEducation(pp.100–105).Toowoomba.http://2017conference.ascilite.org/wp-content/uploads/2017/11/Concise-HILLIER.pdf
Hillier,M.,&Lyon,N.(2018a,June).Writinge-Examsinpre-universitycollege.PaperpresentedattheOpenConferenceonComputersinEducation(IFIPTC3),Linz,Austria.http://transformingexams.com/files/Hillier_Lyon_OCCE2018_writing_eexams_preuni_FP.pdf
TransformingexamsacrossAustralia |26
Hillier,M.,&Lyon,N.(2018b,June).Studentexperienceswithabringyourownlaptope-Examsysteminpre-universitycollege.PaperpresentedattheOpenConferenceonComputersinEducation(IFIPTC3),Linz,Austria.http://transformingexams.com/files/Hillier_Lyon_OCCE2018_student_experiences_eexams_preuni_FP.pdf
Hillier,M.,&Grant,S.(2018,Nov).Do-it-yourselfe-Exams.InM.Campbell,J.Willems,C.Adachi,D.Blake,I.Doherty,S.Krishnan,S.Macfarlane,L.Ngo,M.O’Donnell,S.Palmer,L.Riddell,I.Story,H.Suri&J.Tai(Eds.),ASCILITE2018ConferenceProceedings—OpenOceans:Learningwithoutborders,Geelong(pp.121–130).http://transformingexams.com/files/hillier-grant-diy-eexams-ASCILITE-2018.pdf
Hillier,M.,Grant,S.,&Coleman,M.(2018,Nov).Towardsauthentice-Examsatscale:RobustnetworkedMoodle.InM.Campbell,J.Willems,C.Adachi,D.Blake,I.Doherty,S.Krishnan,S.Macfarlane,L.Ngo,M.O’Donnell,S.Palmer,L.Riddell,I.Story,H.Suri&J.Tai(Eds.),ASCILITE2018ConferenceProceedings—OpenOceans:Learningwithoutborders,Geelong(pp.131–141).http://transformingexams.com/files/hillier-grant-coleman-Towards-authentic-e-Exams-at-scale-ASCILITE-2018.pdf
Lane,B,(2009,November18).Laptopspassthebigexam.TheAustralian[MediaInterview].Retrievedfromhttps://www.theaustralian.com.au/higher-education/laptops-pass-the-big-exam/news-story/ea5759df6b9d6dbca751b5e4f3c4b4c9
Pagram,J.,Cooper,M.,Jin,H.,&Campbell,A.(2018).Talesfromtheexamroom:TriallinganE-Examsystemforcomputereducationanddesignandtechnologystudents.EducationSciences,8(4),article188https://doi.org/10.3390/educsci8040188
TransformingexamsacrossAustralia |
27
AppendixACertificationCertificationbyDeputyVice-Chancellor(orequivalent)IcertifythatallpartsofthefinalreportforthisOLTgrantprovideanaccurate
representationoftheimplementation,impactandfindingsoftheproject,andthatthe
reportisofpublishablequality.
Name:ProfessorSusanElliott,Positiontitle:DeputyVice-ChancellorandVice-President(Education)MonashUniversityDate:24May2018
TransformingexamsacrossAustralia |
28
AppendixBProjectdeliverabledetailsDiscussionpointscomparingtheoriginallyproposedprojectdeliverablesandtheactualprojectdeliverableoutputsareoutlinedinthetablebelow.
Originaldeliverable Deliveredoutpute-Examstechnicalinfrastructure:developedtofacilitateanelectronicworkflowincluding:Set-uptemplatesforacademicstocreateexams,set-uptoolsforstudents,examsubmissionservertoreceiveexamsfromacademics,production/retrievalinfrastructureformanagingthee-examsclientUSBs,electronicreticulationofresponsesvianetworks(possiblywireless),electronicdistributionofstudentresponsestomarkersinformatsconducivetoe-marking(includescollationofstudentresponsesintoclassgroupsandmergerofMoodledatabasestoenablecomputerisedmarking).Howeverthescopecurrentlyexcludesthefacilitiesforelectronicmarkingitself.
Delivered.Arobuste-Examplatformwasdevelopedandusedin35liveexamevents.ItsupportsBYOlaptoptoMoodleservere-examsthatareresistanttonetworkoutages,evenoverWi-Ficonnections.ComputerbasedmarkingutilisedMoodlequestionswhilee-submissionofcomplexconstructedresponsefileswereachievedbyusingafileattachmenttoaMoodlequizquestion.Theoriginallyintendedon-boardLMSmodelevolvedtouseaclient-servermodelduetotherelativeeaseofimplementingitasworkingsystem.Howeverlivenetworkingalsopresentedtechnicalchallengeswithrespecttonetworkoutagevulnerabilitiesduringanexamevent.Theseweresolvedbytheprojectteam'se-Examplatform.
Goodpracticeguides:forstudents,academics,examsoffice(adminprocedures,roomsetupandinvigilation)andITsupportpersonnel.Thisbuildsontheseedgrantdraftversionsofsomeoftheseguidesbutaddsfurtherguidesforacademicsandexamsofficefunctions.
Alternativedelivered.Theseevolvedintoasetofuserguidesforstudents,teachers,examinationsofficersandtechnicaladministration.Theseweremadeavailableviatheprojectwebsite.
Adraftsetofassessment/examspolicyrecommendationsforinstitutions:Thesewillbeintheformofasetofprinciplesforwritingpolicieswithaccompanyingexamplesratherthanspecificfullywrittenpoliciesasitisexpectedinstitutionswillwanttowritetheirown.
Alternativedelivered.Followinganalysisofasampleofassessmentpoliciesathighereducationinstitutionsitwasfoundthatverylittlechangewasrequiredtoexitingpolicies.Simplewordchangeswouldnormallysufficetoenablee-assessmentstooccur.Seesection6.1.foradiscussionandrecommendationsfore-Examstobesystematicallysupportedbycontextualenablingpolicies.
TransformingexamsacrossAustralia |
29
Originaldeliverable DeliveredoutputAsetofexamplee-examtemplatesandquestions:Thiswillleverageandadaptwheresuitablethe'TransformingAssessment'onlinee-assessmentexemplars(Crisp'sALTCfellowship,Hillier's2014OLTExtensiongrant).Thiswillbuildonexample'paper-equivalent'questionformatsdevelopedduringtheseedgrantandaddfurtherexamplesgatheredfrompartnerinstitutions.
Delivered.Therewere35trialsconductedacrossthreemodalities(paper-equivalent,post-paperandrobustonline)inarangeofdisciplinesusingworddocuments,spreadsheetsandonlineMoodlequizzes.Examsincludedcomplexconstructedresponsessuchascomputerprogrammingfilesandeditedlanguagetranslationpieces.AprototypeAugmentedRealitye-Examwasalsodeveloped(yettobetrialled).Asetofeightminicasesweredevelopedtohighlightthedifferente-examtrials.Furtherdetailsofeachexamtypeareavailablefrompublications.Theseresourcesareavailablefromtheprojectwebsite.
Researchfindings:Thepublishedresultsfromtrialscompletedatamultipleinstitutionsthatincludesanalysisofresultsofstudentexperience.
Delivered.Therewere15referredpublicationsand60presentationsoverthedurationoftheproject.Thesepublicationsincludedtheresultsofstudentexperiencefeedback.Afulllistisprovidedontheprojectwebsite'Research'section.Furtherpublicationsareforthcoming.
Workshopmaterialstointroduceacademicstoe-exams:tobeofferedatnationalT&LfocusedconferencessuchasHERDSAorASCILITEinyearstwoandthree.
Delivered.Workshopsdeliveredatconferencesin2017,inanationalRoadshowforeachcapitalcityduring2018,thee-ExamsSymposiumheldon24Nov2018.Workshopsarelistedontheprojectwebsite.RecordingsononeroadshowsessionandallsymposiumpresentationsareavailableviatheprojectwebsiteandonYouTube.
An'e-examsfoundation'drafttermsofreferenceandacallformembership:ThefoundationistoprovideforthelongtermtechnicalsupporttomaintainICTinfrastructureandprovidelicencestocommercialserviceproviders.ThebusinessandlegalmodelwilllikelybebasedonthesuccessfulMoodleopensourcearrangements,whichhasproducedasustainableoutcomeforthelongtermsupportofthatAustralianinnovation.
Alternativedelivered.Itbecameapparentthatthesectorwasnotinthefinancialormanagerialpositiontosupportaneworganisation.Aplenarystylediscussionwasheldwiththe86attendeesate-ExamSymposiumon24Nov2018.Thisincludedrepresentationfrom23universities.Whilstsomeinterestwasshown,onbalancetheprojectteamconcludedthatthemostlikelywayforwardwouldbeviajointventurewithaEuropeanuniversityorcommercialisation.Theprojectleadershipisnowpursuingtheseoptions.
TransformingexamsacrossAustralia |
30
AppendixCe-ExamtrialdetailsThetablebelowdisplaysallknowntrialsconductedusingthee-Examplatformtodate.TrialsconductedundertheInnovationandDevelopmentprojectincludesthosesince2015.Priorworkundertakenwithpredecessorversionsofthee-Examsystemarealsoincludedforcompleteness.Examcandidatecountsarelistedfortypistsandhand-writerswhereknown.
Instit'n Year S Course Mode QuestionTypes
Weight Time(mins)
Context TypedHandTotalStaff Grant
CQU 2018 2 EthicsandSocialIssues(IT)(COIT11223)TBC
Worddoc Shortanswer 50% 180 Outofclass,central,multicampus,finalexam
? ? ? MichaelCowling
OLTID
Monash2018 2 IntroductiontoChineselanguage(ATS1002-2002)
Moodlequiz,audiofiles
Shorttext,TF,MCQ,fillinblank.Pluslisteningtestshortresponse.
15% 60 In-class,midsemesterexam(alltype).10minlisteningtest+70minutewritingtest.
74 0 74 ScottGrant OLTID
Monash2018 2 Chineseonlinemedia(ATS3038)
Moodlequiz,DimsumandSyngChinesedictionarytools
Longtexttranslation,shorttext,matching,fillinblank
15% 60 In-class,midsemesterexam(alltype).In-class1hourtests.25%(12.5%x2)Weeks7&11.Final2hourexam45%
22 0 22 ScottGrant OLTID
Monash2018 2 LanguageTranslation(AGP5048)
Worddocandmultilingual.Spellingon.
Shorttext,longtext
30% 180 InClass-3hr.15minfinalexam+separate14NAATIstudents(total48).
40 8 48 ShaniTobias
OLTID
CQU ? EthicsandSocialIssues(IT)(COIT11223)
Worddoc Shortanswer 50% Outofclass,central,multicampus,finalexam
115 115 MichaelCowling
OLTID
UQ 2018 2 FrenchlanguageB(FREN3113)
Worddoc Longtext 30% 120 In-class,Schoolrun,finalexam,collaborativelearningspace
? ? 66 AmyHubbell
OLTID
UQ 2018 2 FrenchlanguageB(FREN3380)
Worddoc Longtext 30% 120 In-class,Schoolrun,finalexam,collaborativelearningspace
? ? 38 AmyHubbell
OLTID
UniSA 2018 1 MathematicsforSecondaryTeaching1(EDUC5184)
Worddoc ShortAnswertextresponsesApprox600wordseach
40% 120 Inclass 31 2 RuthGeer,BruceWhite
OLTID
Monash2018 1 IntroductiontoChineselanguage(ATS1001-2001)
Moodlequiz,audiofiles
Shorttext,TF,MCQ,fillinblank.Pluslisteningtestshortresponse.
16% 80 In-class,midsemesterexam(limitedto20typingslots).10minlisteningtest+70minutewritingtest.
14 NotSev110
ScottGrant OLTID
TransformingexamsacrossAustralia |
31
Instit'n Year S Course Mode QuestionTypes
Weight Time(mins)
Context TypedHandTotalStaff Grant
Monash2018 1 Chineseonlinemedia(ATS3037)
Moodlequiz,DimsumandSyngchinesedictionarytools
Longtexttranslation,shorttext,matching,fillinblank
15% 70 In-class,midsemesterexam
27 4 ScottGrant OLTID
Monash2018 1 LanguageTranslation(AGP5690).PracticeFriday25thMayfrom10to12noon+examTuesday5thJunefrom9amto1pm
Worddocandmultilingual
Shorttext,longtext
30% 180 InClass-finalexam
26 5 ShaniTobias
OLTID
UTAS 2018 1 EducationDigitalTechnologies(ESH380)
Worddoc. Shorttext,longtext,Scratchprogramming,softwareappraisal
47% 120 Final 125 20 145 AndrewFluck
OLTID
CQU 2017 2 EthicsandSocialIssues(IT)(COIT11223)
Worddoc,PDF,videofile
Shortanswer 50% 180 Outofclass,central,multicampus,finalexam
123 0 MichaelCowling
OLTID
ECU 2017 2 TeachingIntroductoryComputerProgramming(CSE4102)
WorddocandPythonIDE
Shorttext,longtext,programming
15% 120 In-class,computerlab
8 0 JeremyPagram
OLTID
ECU 2017 2 MaterialsDesignandTechnologyOHS(DTE4271)
Worddoc Shorttext,longtext,images
30% 90 In-class,computerlab
9 0 JeremyPagram
OLTID
Monash2017 2 IntroductoryChineselanguage(ATS1002)
Spreadsheetasform.
MCQ,T/F,fillinblank,shorttext
16% 70 Outofclass,mid-sem
16 57 ScottGrant OLTID
Monash2017 2 LanguageTranslation(APG5048)
Worddocandmultilingual
Shorttext,longtext,editingtask
20% 180 In-class,midsemesterexam
23 8 ShaniTobias
OLTID
Monash2017 2 LanguageTranslation(APG5048)
Worddocandmultilingual
Shorttext,longtext
30% 180 In-class,finalexam,somesatasNAATIqualificationexam
26 5 ShaniTobias
OLTID
CQU 2017 1 KnowledgeManagementPrinciples(IT)(COIT12205)
Worddoc Shortanswer 50% 180 Outofclass,central,examhall/centres,multicampus,finalexam
30 50 RahatHossain
OLTID
Monash2017 1 BusinessStatistics(EFT5900)
Worddoc MCQ,shorttext
5% 30 In-classtest 20 Notsurveyed
CharanjitKaur
OLTID
TransformingexamsacrossAustralia |
32
Instit'n Year S Course Mode QuestionTypes
Weight Time(mins)
Context TypedHandTotalStaff Grant
Monash2017 1 LanguageTranslation(APG5690)
Worddocandmultilingual
Shorttext,longtext
30% 180 In-class,finalexam,somesatasNAATIqualificationexam
31 17 ShaniTobias
OLTID
Monash2017 1 Globalisationx4classes(Monashcollege)
Worddoc. Shorttext,longtext
10% 70 In-classtest 40 42 NathanielLyon
OLTID
MqU 2017 1 ICTinEducation(EDUC261)
Worddoc MCQ,extendedessay
40% 80 Outofclass,Schoolrun,finalexam
17 181 MatthewBower
OLTID
UniSA 2017 1 MathematicsforSecondaryTeaching(EDUC5184)
Spreadsheet
Formulae,calculationcharts
40% 120 In-class,finalexam(twosittings)
9 15 RuthGeer,BruceWhite
OLTID
UNSWADFA
2017 1 Airpower(ZHSS2424)
Worddoc Shorttext,extendedessayquestions
35% 120 Outofclass,Schoolrun,finalexam
42 8 AndrewGilbert
OLTID
UQ 2017 1 Frenchlanguagetranslation(FREN3310)
Worddoc Longtext 10% 50 In-class 16 2 AmyHubbell
OLTID
UTAS 2017 1 EducationDigitalTechnologies(ESH380)
Worddoc,media,Scratch
Shorttext,longtext,programming
47% 120 Outofclass,central,multicampus,finalexam
88 3 AndrewFluck
OLTID
Monash2016 2 Geographyx2classes(Monashcollege)
Worddoc Singleessay 10% 70 In-class 25 13 NathanielLyon
OLTID
Monash2016 2 LanguageTranslation(APG5048)
Worddocandmultilingual
Shorttext,longtext
Zero 45 In-class,mockexam
18 0 ShaniTobias
OLTID
UTAS 2016 2 EducationFoundationsofTeaching(ESH102)
Mcqs,worddoc
Shorttext,longtext,programming
40% 120 Outofclass,central,multicampus,finalexam
83 3 WendyBalassa
OLTID
UTAS 2016 1 EducationDigitalTechnologies(ESH380)
Worddoc,media,Scratch,PDF,Windowssoftware
Shorttext,longtext,programming
47% 120 Outofclass,centrallymanaged,multicampus,finalexam
110 14 AndrewFluck
OLTID
UTAS 2015 1 EducationDigitalTechnologies(ESH380)
Worddoc,media,PDF,Scratch,Windowssoftware
Shorttext,longtext,programming
44% 120 In-classtesteexam
110 7 AndrewFluck
OLTID
UQ 2015 1 PunishmentandSociety(CRIM2100)
Worddoc Longtext 20% 90 In-class,midsemesterexam(includes20%typedessaysectionand15%MCQusingbubblesheets).
4 33 RobinFitzgerald
UQTEL
TransformingexamsacrossAustralia |
33
Instit'n Year S Course Mode QuestionTypes
Weight Time(mins)
Context TypedHandTotalStaff Grant
UQ 2015 1 ResearchProcesses(DENT4092)
Worddoc Table,shorttext
20% 60 In-class,midsemesterexam
19 66 AroshaWeerakoon
UQTEL
CQU 2014 2 SystemsAnalysis(COIT11226)
Worddoc Shortanswer 50% 180 Outofclass,central,multicampus,finalexam,opt-in
6 50 MichaelCowling
OLTSeed
UQ 2014 2 Zoology(BIOL2204).BiologicalSciences
Worddoc MCQ,shorttext,diagram
20% 50 In-class,midsemesterexam
10 81 DavidBooth,
OLTSeed
UQ 2014 2 OccupationalTherapyforChildrenandYouth(OCTY7826)
Worddoc MCQ,casestudy,shorttext,longtext,table,listfill
25% 100 In-class,midsemesterexam
3 24 LizSpringfield
OLTSeed
UQ 2014 1 PhysiotherapySpecialitiesNeurology(PHTY3140/7814)(multiplesmallgroupsof12)
Worddoc,videoprojection
Table,shorttext
15% 15 In-class,midsemesterexam(groupsof12.Written15%sectiondonepriorto20%OSCE)
25 108 KatrinaWilliams
OLTSeed
UQ 2014 1 AnimalandVeterinaryBiology(ANIM1060)
Worddoc,image
MCQ,shorttext,imageidentification
15% 45 In-class,finalexam
5 109 MalcolmJones
OLTSeed
UQ 2014 1 PunishmentandSociety(CRIM2100)
Worddoc Longtextessay
10% 70 In-class,midsemesterexam(timeincludes10%typedessaysectionand25%MCQonpaper).
17 50 RobinFitzgerald
OLTSeed
UQ 2014 1 AnimalHealthTechnology(VETS2001)(twogroups:Internal,external)
Worddoc,image
MCQ,T/F,shorttext,imageidentification,diagramlabelling
15% 90 In-class,midsemester(internalandexternalgroupsdone2weeksapart).
11 78 RebekahScotney
OLTSeed
UTAS 2014 1 Education:DigitalTechnologies(ESH380)
Worddoc,media,PDF,Windowssoftware
Shorttext,longtext,
44% 120 Outofclass,multicampus,centrallymanaged,finalexam,multi-site
166 16 AndrewFluck
OLTSeed
UTAS 2013 1 Education:InformationandCommunicationTechnology(ESH380)
Worddoc,video,GIMP,PDF,Windowssoftware
Shorttext,longtext,
50% 120 Outofclass,multicampus,centrallymanaged,finalexam,multi-site
123 25 AndrewFluck
UTASTDG
UTAS 2012 3 Education:InformationandCommunicationTechnology(ESH380)
Worddoc,PDF,Windowssoftware
Shorttext,longtext,
50% 120 Outofclass,multicampus,centrallymanaged,finalexam,multi-site
48 5 AndrewFluck
UTASTDG
TransformingexamsacrossAustralia |
34
Instit'n Year S Course Mode QuestionTypes
Weight Time(mins)
Context TypedHandTotalStaff Grant
UTAS 2012 1 Education:InformationandCommunicationTechnology(ESH380)
Worddoc,PDF,Windowssoftware,animatedPPTwithaudio
Shorttext,longtext,
50% 120 Outofclass,multicampus,centrallymanaged,finalexam,multi-site
88 1 AndrewFluck
UTASTDG
UTAS 2011 2 Law:ConstitutionalLaw(LAW205)
Worddoc Shorttext,longtext,
50% 120 Outofclass,singlecampus,centrallymanaged,finalexam
69 179 MichaelStokes
UTASTDG
UTAS 2011 2 Law:PropertyLawB(LAW324)
Worddoc Shorttext,longtext,
50% 120 Outofclass,singlecampus,centrallymanaged,finalexam
26 ? LyndenGriggs
UTASTDG
UTAS 2011 2 Law:InternationalLaw(LAW631)
Worddoc Shorttext,longtext,
50% 120 Outofclass,singlecampus,centrallymanaged,finalexam
1 ? MichaelTate
UTASTDG
UTAS 2010 2 Arts:History-AtlanticWorlds1450-1807(HTA249/349)
Worddoc Shorttext,longtext,
50% 120 Outofclass,singlecampus,centrallymanaged,finalexam
3 ? TomDunning
UTASTDG
UTAS 2010 2 Education:ModesofcurriculumInquiryA(ICTandDesign&Technology)[EPC353]
Worddoc,video,PDF,Windowssoftware,PPTwithaudio
Shorttext,longtext,
25% 120 Outofclass,singlecampus,centrallymanaged,finalexam
69 4 AndrewFluck
UTASTDG
TransformingexamsacrossAustralia |
35
AppendixDUpdatedImpactMatrixTheIMPELmatrix(shownonthefollowingpage)providesanestimateofanticipatedchangesoutlinedintheprojectproposalalongwithtagstoindicatethestatusofeachproposedimpactpoint.Theseare:achieved,likely,possible,alternative(Theseareshownincapitallettersinthetablebelow).Itshouldbenotedthatduetochangesintheprojectfocusortheenvironmentsomeimpactpointsarenowconsideredunlikely,butinallsuchcasesweforeseeanalternativeoutcomebeingpossible.Themajorareaswherealternativesarepossibleinclude:
a)Theproposede-ExamsFoundationwasthoughtunsuitedtocurrentsectorconditions(financialandmanagerial)howeveranalternativeoutcomemaybeajointventureorcommercialisationarrangement.
b)Theadoptionofprojectdevelopede-examtechnologies.GiventheapparentstrategiesandprocessesofAustralianuniversitymanagement,theadoptionofe-ExamsatAustralianinstitutionsismorelikelytobeintheformofcommerciallyavailablesolutionsratherthan'homegrowninnovations'.Unfortunatelythecommercialmarkethasyettobeabletoofferasolutionwiththeextensiveauthentice-assessmentcapabilitiesofferedbythee-Examplatformdevelopedinthisproject.ThisisincontrasttoeducationorganisationsinregionssuchasEuropewhereanumberofhomegrowne-examapproachesincorporatingauthentice-assessmentarebeingadoptedwithgreatsuccess(Seee-ExamSymposiumsessions1and4forexamplesinFinlandandAustria).
TransformingexamsacrossAustralia |
36
AnnotatedIMPELMatrix
AnticipatedchangesatProjectcompletion 6Monthspost 12monthspost 24monthspost
1. TeamMembers
ACHIEVEDTeammembersknowledgeableaboutusinge-examstocreatepost-paperexaminationsintheircourses.
LIKELYResearchandpublicationsone-examsfindingsfromresearchproject.Planningnextresearchphase.Furtherinvitestocollaboratefrominstitutions.Continuedpresentationsgiven.
POSSIBLEApplicationunderwayforsubsequentgrant,fellowship(e.g.developlearninganalyticstoolsandtechniquestoleveragee-examsdata),orteachingawardapplicationsforteammembers.
POSSIBLE,ALTERNATIVEResearchandpracticeintoexamscontinues.Continueduseincoursesbyprojectteammembers.
2. Immediatestudents
ACHIEVEDSeveralgroupsofstudentsundertheinstructionofteammembersexperiencedmultiplee-exams.
POSSIBLE,ALTERNATIVELargernumbersofstudentsexperiencinge-exams
3. Spreadingtheword
ACHIEVEDWebsite,practiceguides,technicalinfrastructurereleased.Policyrecommendations.Resultsfromtrials.Softwaredistribution.
LIKELYDatapackages/archivessharedonlineforotherresearcherstoaccess.Websiteliveson.Publications,presentationscontinue.
LIKELYPublications,presentationsbeingcited.
4. Narrowopportuneadoption
ACHIEVEDSeveralcoursesemployingthee-exams.'Associate'partnerinstitutionscontinuetrials
POSSIBLEInstitutionsmakingcontacttotrialthee-examsystemonaself-fundedbasis.
5. Narrowsystematicadoption
ALTERNATIVEe-examsFoundationinitiated.Supportpartnersintereststart.Majorstakeholdersocieties,HERDSA,ASCILITE,CADADacknowledgingvalueoftheproject.
POSSIBLE,ALTERNATIVEOngoingdatacollection.Passiveonlinesurveyforreportinguse.Askinguserswhatimpactithadxmonthsafter.Corepartnerinstitutions'officially'supportinge-exams.
POSSIBLE,ALTERNATIVEFoundationunderwaytocontinuethetechnicaldevelopmentwithcommercialtechnicalsupport.Firstpartnersjoinedandpayingcontributionstothetechnicaldevelopment.
6. Broadopportuneadoption
LIKELY,ALTERNATIVESeveralinstitutionsemployingthee-exams.
LIKELY,ALTERNATIVEConsultancyservice(byprojectteammembers)assistinginstitutionstostarte-examsandcustomisetoneed.
POSSIBLE,ALTERNATIVEFirstcommercialproviderssigneduptofoundationandprovidingservicestothehighereducationsector.
7. Broadsystematicadoption
LIKELY,ALTERNATIVEPolicy'shape'recommendations.
LIKELYPolicychangeconsideredbyinstitutionsbeyondprojectcore.Someinstitutions'officially'supportinge-exams.
POSSIBLEConsultancyservicesofferingpedagogical/technicale-examsprocesssupport.
POSSIBLEPoliciesadoptedallowinge-examsinmultipleinstitutions.Moreinstitutions'officially'supportinge-exams.
TransformingexamsacrossAustralia |
37
AppendixEFormativemid-projectevaluationreportThee-ExamsystemItwastheopinionofallprojectparticipantsthattheprojecthasbeenhighlysuccessfulindevelopingasuitablee-examplatformforBYODsupervisedassessments.TheplatformhasbeenfoundtoworksuccessfullyonbothWindowsPCandAppleMacintoshlaptops.Someparticipantsnotedthatstudentsareincreasinglybuyingintotheidea,withfewerstudentsoptingforhandwrittensubmissions.Theguidelineshavebeenseentobeuseful,describedas"excellentdocumentationforthetechnicalaspectsofUSBproduction,answer-scriptcollectionandrecycling."Furtherthattheprojecthasbeenusedasimpetuousforarangeofconceptualandtechnologicaldevelopments:“Ourparticipationinthetrialprecipitatedunexpecteddemandsfromacademicsfore-assessments.Wehaveservicedthosedemandsbyrunningverylowteche-assessmentsthathavetaughtusalotaboutwhatneedstobeseentoinbackground,non-technicalarrangements.”
ChallengesandopportunitiesAllparticipantsviewedtheopportunitiesaffordedbythepotentialofe-examsasoutweighingthehindrances.Wherechallengeswerenoted,theywereidentifiedasareasthatcouldbeenhanced,ratherthanasrationalefordiscontinuinganyfurthertrials.Participantscitedchallengeswithinternalapprovalsandauthorisationsashindranceswiththeproject.Anexampleofonesuchchallengewastheexperienceinanearlierpilotprojectconductedin2014thatsawtoofewUSBsbeingprovided,resultinginstudentcomplaints.Thisexperiencedidprovideapositiveoutcomeinthat,asaresult,“betterbackupproceduresforlackofUSBorfailure”havebeendevelopedandimplementedinlatertrialsconductedaspartofthisproject.Intermsofthesoftware,thelimitedgraphicscapabilitywasidentifiedasbeingproblematicforsomeprogrammingexams.Alsoseentobeproblematicwastheinabilitytolimitaccesstoparticularwebsites.Thethesaurusfunctionwithinthee-examsystemwasnotedasbeingparticularusefulintranslation,andalsoacknowledgedtobeahithertounexploredresource.ThepotentialtoexpanddictionariestoincludebothEnglishandFrenchwasviewedaspotentiallybeneficial.
ProjectteamOverall,allparticipantswereverypleasedwiththeprogressoftheprojectandwereallpleasedwiththeserviceandresponsivenessoftheprojectteam,findingtheirengagementwiththeteamtobehighlyproductive.“Technicalsupportranseamlesslyinthepracticeexamsandtheactualtrials.Vilmawasveryorganisedandkeptusuptodateandontimewithvarioustasksthroughoutthetrials.Mathewisextremelyknowledgeableabouttheopportunitiesinthee-examsystemandhisbackgroundinpreparingtranslationexamswasusefulforme.”Thehelpfulnessandpositiveresponsivenesswascitedasthereasonbehindthesuccessfultrialsandimplementation.Exchangeswerenotedtoassistindeveloping“ourunderstandingofnotjustthetechnicalelementsofe-examsbutalsotheorganisationalelements.”
ConclusionsandRecommendationsTheprojectparticipantsat7ofthe9partnerinstitutionsrespondingtothesurveyindicatethattheyaresatisfiedwiththeprojectprogress.Theyareenthusiasticparticipants,seekingtocontinuetheirengagementforthesuccessfuloutcomesandexperiencesfortheir
TransformingexamsacrossAustralia |
38
students.Reviewingengagementdatainlightofthesurveyresponses,itisclearthattheprojectisprogressingasanticipated,andhasawiderreachandlevelofengagementthanpreviouslyanticipated.Workingwithconservativeinstitutionalpoliciesandacademicandtutoringstaffnotyetconvincedofthebenefitsofe-examtechnologieswillneedtobeconsideredinthefuture.Itisalsoworthconsideringtheuseofotherstoragetechnologiesasthistechnologycontinuestoevolveasanover-dependenceonUSBscouldbepotentiallylimitinginthefuture.
DrDeanneGannawaySFHEAAustralianLearningandTeachingFellow,Academiclead:Professionallearning.InstituteforTeachingandLearningInnovation(ITaLI),TheUniversityofQueensland.d.gannaway@uq.edu.au15December2017
TransformingexamsacrossAustralia |
39
AppendixFFinalexternalevaluationreportEXTERNALEVALUATIONREPORT:TRANSFORMINGEXAMSACROSSAUSTRALIAThisfinalexternalevaluationreportprovidesanoverviewoftheoutcomesfromtheTransformingExamsAcrossAustraliaprojectconductedbetween2016and2019andledbyDrMathewHillier.
1. EXTERNALEVALUATIONTEAM
TheexternalevaluationwasconductedbyDeanneGannawayandKarenSheppard.
DeanneGannawayBAHDipEd(Wits),MEd(USQ).PhD(Flinders)
DeanneGannawayisaseniorlecturerintheInstituteforTeachingandLearningInnovationattheUniversityofQueensland.Deanne’sresearchinterestsfocusonthedevelopmentofeducationalevaluationandenhancementofhighereducationcurriculum.Deanneisfrequentlycalledontoofferadviceonappropriateevaluationapproachesandmethodsforlargescaleteachingandlearningprojectsandcurriculum-basedactivitiesandhasbeencommissionedastheexternalevaluatoronanumberoflargescaleALTC/OLT-fundedprojectsandfellowshipprograms.
KarenSheppardBA(UWA),Dip.Ed.(WACAE),MEd(ProfessionalEducationandTraining)(Deakin)
KarenSheppardisalearningdesignerintheInstituteforTeachingandLearningInnovationattheUniversityofQueensland.
DeanneandKarenhavebeenactivelyinvolvedinanumberofnationalteachingandlearningfundedprojects,includingnationallyfundedevaluationofthedisseminationstrategiesusedintheAustralianLearningandTeachingCouncil(ALTC)grantsschemes:AReviewoftheDisseminationStrategiesusedbyProjectsFundedbytheALTCGrantsScheme.Thisprojectbuiltontheresearchcompletedin2004forthenationally-focused,DESTfundedresearchprojectondisseminationstrategiesforinnovationprojectoutcomeswhichdirectlyinfluencedthegrantschemesandprocessesdevelopedbytheALTC.
ThisfinalreporthasbeendraftedbyDrDeanneGannaway.
2. THEEVALUATIONAPPROACHThisexternalevaluationadoptedtheKeyEvaluationChecklistapproach(Scriven,2007)todevelopananalyticalframework.Thischecklisthasbeenusedtoevaluateprograms,plansandpoliciesandallowsforamulti-layeredmixedmethodmodeofinvestigation.(Stufflebeam,2001;Davidson,2005;Scriven,2007;Martz,2010).TheformativeevaluationstrategydirectedbythechecklistapproachpromotedbyScrivenfacilitatesprojectteamreflection,allowingforagileandresponsiveprojectprogressratherthansolelyasummativemeasurementofprojectimpactorattainment.
InkeepingwiththeScrivenapproach,theevaluationaimstoascertainthesuccessoftheprojectinrelationtoitsvision,mission,goals,deliverablesandplan,specifically,measurementandidentificationof:
• achievementofprojectgoals,objectivesandintendedoutcomes;• whetherneedsofstakeholderssuchasstudents,staff,projectparticipantsand
TransformingexamsacrossAustralia |
40
fundingbodyarebeingmet;• relevance,appropriatenessandqualityofthechoseninvestigationandproject
managementstrategiesusedintheproject;• goodpracticeandareasforimprovementacrosstheprojectlife;and• effectivenessofdisseminationandtake-upamongstakeholders.
Theformativeevaluationapproachadoptedrequiredacloseworkingrelationshipbetweentheprojectteamandtheexternalevaluationteaminordertomonitorandadviseontheevaluation,dissemination,resourcingandinvestigationstrategiesadoptedbytheprojectteam.Thisapproachenabledtheidentificationofgoodpracticeandareasforimprovementacrossthelifeoftheprojectand,mostimportantly,enablestheprojectteamtorespondtotheareasidentified.
EVALUATIONQUESTIONSANDACTIVITIES
Thefollowingdatacollectionstrategieswereusedtocollectandanalysetherequireddataagainsttherelevantevaluationquestions:
1. Wereprojectgoals,objectivesandintendedoutcomesachieved?
a. ReviewofprojectactivitiesconductedProjectdeliverablesarereviewedatregularintervalsacrossthelifeoftheproject.Theevaluationteamisnotacontentexpertandthereforenotinapositiontoevaluatetheaccuracyorrelevanceoftheresourcesdeveloped.Theexternalevaluationteamactedascriticalfriends,providingformativeinputacrossthedevelopmentprocesses.
b. EmailsurveyofprojectpartnersAnemailsurveywasconducted,targetingprojectparticipantswhoarealsopotentialadoptersofprojectoutcomes.Inacknowledgementofpossiblesurveyfatigue,ratherthananonlinesurvey,aquickemail-basedsurveywassenttoparticipants.AcopyofthequestionsisavailableasAppendix1.Thesurveywassentto15individualsresultingin7responses,providinginputfrom7ofthe9partnerinstitutions.Thetimingofthesurveymeantthatmanyofthepotentialparticipantswerealreadyonleaveforthesummerbreak.
c. InterviewswithprojectteammembersTheprojectleader,projectmanagerandtheexternalevaluatorhaveconductedregular“check-in”meetingstomonitorprojectprogressionandtoreviewprojectevaluationactivities.
2. Howrelevantandappropriatewerethechosenprojectplans,activitiesandstrategiesforachievingtheprojectaims?
a. MonitoringprojectprogressProgressreportswerereviewedandfeedbackgiven.Projectplanswerediscussedduringtheevaluationmeetingsandprojecttimelinesreviewed.
b. ReviewofevaluationdatacollectedEvaluationactivitiesconductedbytheprojectteamwerereviewedbytheevaluationteamtoensuretheveracityofprojectfindingsandconclusions.
3. Wereanticipatedprojectoutcomesmet?
TransformingexamsacrossAustralia |
41
a. ReviewofmaterialsdevelopedWeb-basedmaterialsandotherresourcesdevelopedasdeliverablesfromtheprojectwerereviewedandfeedbackprovidedinmeetingswiththeprojectcoreteam.
b. ObservationofprojectactivitiesKarenSheppardattendedandparticipatedinsomeofthenationalsymposia,collectingobservationaldataandcommentaryfromotherparticipants.
EvaluationOutcomes
1) Towhatextentdidtheprojectmeetand/orexceeditsobjectives?
Theoriginalprojectaimswere:1. Buildingane-examplatformtobeusedinsupervised,'bringyourowndevice'
(BYOD)settingsthatincludescomputermarkedquestionsandelectronicreticulationofstudentresponses.
2. Developingguidelinestoassiststudents,educatorsandadministratorstoeffectivelyprepareandundertakee-examstakingintoconsiderationthewholeassessmentworkflow.
Atotalof35trialswereconductedacrosseightinstitutionsfrom2016to2018withparticipationfromover1700students.Studentswereabletooptintothetrials.Projectparticipantsregularlycommentedthat,intheiropinion,theprojecthasbeenhighlysuccessfulindevelopingasuitablee-examplatformforBYODsupervisedassessments.TheplatformhasbeenfoundtoworksuccessfullyonbothWindowsPCandAppleMacintoshlaptops.Someparticipantsnotedthatstudentsareincreasinglybuyingintotheidea,withfewerstudentsoptingforhandwrittensubmissions.
Theguidelineshavebeenseentobeuseful,describedas“excellentdocumentationforthetechnicalaspectsofUSBproduction,answer-scriptcollectionandrecycling.”
Possiblymoreimportantly,theprojecthasbeenusedasimpetuousforarangeofconceptualandtechnologicaldevelopments:“Ourparticipationinthetrialprecipitatedunexpecteddemandsfromacademicsfore-assessments.Wehaveservicedthosedemandsbyrunningverylow-teche-assessmentsthathavetaughtusalotaboutwhatneedstobeseentoinbackground,non-technicalarrangements.”
2) Howrelevantandappropriatewerethechosenplans,activitiesandstrategiesforachievingtheprojectaims?
Overall,participantsindicatedthattheywerepleasedwiththeprogressoftheprojectandwereallpleasedwiththeserviceandresponsivenessoftheprojectteam,findingtheirengagementwiththeteamtobehighlyproductive.Projectpartnersunanimouslyagreedthattheprojectleaderandprojectmanagerwerehighlyeffective,accessibleandknowledgeable,facilitatingchangesinassessmentpolicyandpracticeandopeningupinstitutionalconversationsthatpotentiallyleadtoassessmenttransformationinthefuture.
“Technicalsupportranseamlesslyinthepracticeexamsandtheactualtrials.Vilmawasveryorganisedandkeptusuptodateandontimewithvarioustasksthroughoutthetrials.Mathewisextremelyknowledgeableabouttheopportunitiesinthee-examsystemandhisbackgroundinpreparingtranslationexamswasusefulforme.”
Thehelpfulnessandpositiveresponsivenessoftheprojectteamwasfrequentlycitedasthe
TransformingexamsacrossAustralia |
42
reasonbehindthesuccessfultrialsandimplementation.Exchangeswerenotedtoassistindeveloping“ourunderstandingofnotjustthetechnicalelementsofe-examsbutalsotheorganisationalelements.”
Theprojectparticipantsat7ofthe9partnerinstitutionsrespondingtothesurveyindicatethattheyaresatisfiedwiththeprojectprogress.Theyareenthusiasticparticipants,seekingtocontinuetheirengagementforthesuccessfuloutcomesandexperiencesfortheirstudents.Therewerenonegativeresponsestoanyofthequestions,butsomesuggestionsforfutureimprovementswereoffered.
Reviewingengagementdatainlightofthesurveyresponses,itisclearthattheprojecthashadawiderreachandlevelofengagementthanpreviouslyanticipated.
Participantscitedchallengeswithinternalapprovalsandauthorisationsashindranceswiththeproject.Anexampleofonesuchchallengewastheexperienceinanearlierpilotprojectconductedin2014thatsawtoofewUSBsbeingprovidedbytheinstitution’scentralexamsoffice,resultinginstudentcomplaints.Thisexperiencedidprovideapositiveoutcomeinthat,asaresult,“betterbackupproceduresforUSBlackorfailure”havebeendevelopedandimplementedatthisinstitutionwhichinlatertrialsconductedaspartofthisproject.Otherexamplesincludeinstitutionalpolicydecisionscappingtheweightingofassessmentallowedtobeincludedinsuchtrials,limitingengagement.Otherhindrancesidentifiedincludeinfrastructurelimitations,suchasshortagesinthenumbersofpowerpointsavailable;thepotentialforsomestudents’computerstonotbecompatible.Thefactthatthecurrentsystemdoesn'tworkonallmachines(particularlymoremodernmachines)andinvolvesthetime-consumingandtechnicallydifficultuseofUSBsticks,isaproblem.IamlookingforwardtofutureversionsofthesystemthattieinwithMoodleasamoreefficientwayofmanagingexams.ThechangingnatureofstoragetechnologyandtheneedtomovebeyondUSBinthefuturewasnotedasapotentialfuturedirection.Anotherpotentialhindrancewasthenegativeresponseevidentinteachingteammembersoutsideoftheprojectpartners.Somepartnersdescribedthechallengesofconvincingtutorsandacademicsnotfamiliarwiththesoftware,includinghavingtoprintstudentworksothattheycouldbemarkedasapaper-basedartefact.Thischallengesuggeststhatthereisstillmoreworktobedoneintermsofrefiningthemessagingandinformationprovidedtostakeholdersandpotentialusers.
Intermsofthesoftware,thelimitedgraphicscapabilitywasidentifiedintheinterimreportasbeingproblematicforsomeprogrammingexams.Alsoseentobeproblematicwastheinabilitytolimitaccesstoparticularwebsites.Thethesaurusfunctionwithinthee-examsystemwasnotedasbeingparticularusefulintranslation,andalsoacknowledgedtobeahithertounexploredresource.ThepotentialtoexpanddictionariestoincludebothEnglishandFrenchwasviewedaspotentiallybeneficial.
Participantsnotedtheneedtomonitortheevolutionofstoragetechnologiesinthefutureasanover-dependenceonUSBscouldbepotentiallylimitinginthefuture.
3) Howeffectivearethechosendisseminationstrategies?
TheTransformingexamsacrossAustraliawebsite,roadshowworkshops,conferencepresentations,journalpublicationsandthee-Examsymposiumallhaveservedtoraiseawarenessoftheprojectdeliverablesandoutcomes.However,workingwithconservativeinstitutionalpoliciesandacademicandtutoringstaffnotyetconvincedofthebenefitsofe-
TransformingexamsacrossAustralia |
43
examtechnologiesisafundamentalhindrancetotransformationofpracticeandfurtherspreadofinnovation.Participantsinthedisseminationactivitieshavetendedtobeearlyadoptersorthosealreadyconvincedthatthisisaviableoption.Thisprojectdidnothaveenterpriseculturalchangeasafundamentalaim,sotheawarenessraisingdisseminationstrategiesadoptedareentirelyappropriate.TheadoptionofthesolutionbyagenciesoutsideoftheAustralianuniversitysectorbodeswellforongoingcollaborationsandsustainabilitybeyondthelifeoftheproject.
CONCLUSIONSANDRECOMMENDATIONS
Insum,thisprojecthasmetandexceededitsobjectives.Thee-examsolutionusesallMoodlequestiontypes,andismovingtowardsmoreauthenticassessmentplusprovidingaccesstoarangeofsoftwaretools.AkeybenefitofthesolutionistheuseoftheBYODoverwifiorwirednetworks.Further,thecomprehensivenetworkoutageprotectionmeansthatstudentresponsesarefullycachedandautosavedtolocalstorageevenifthereisnonetwork.
Whiletheprojecthasdevelopedandtestedahardwareandsoftwaresolution,themostimportantcontributionoftheprojecthasbeentheinitiationofanationaldiscussion.Theprojecthasignitedactivitiesthathavealreadyseenfurtherinstitution-wideinitiatives(seeCMM17June,2019,forexample).Theprojectalsoleavesanimportantlegacyintheformofthehttp://transformingexams.comwebsitethatisattractingongoinginternationalandnationalengagement.
Thehigh-risk,highcostnatureofimplementingthetechnologyisstillalimitingfactoroffurtheradoption,despiteincreaseddesireonthepartofinstitutionalbodiestomoveintothisspace.Themovetowardsmorewidescaleadoptionrequiresawhole-of-institutioncommitment,thatincludesbuy-infromacademics,isembeddedinpolicy,istrustedbystudentsandhassubstantialresourcingallocatedtoensurethatthetechnologyisembeddedinamannerthatisatanenterpriselevel.Whiletheseaspectswereoutsideofthescopeofthisproject,thisprojecthasshiftedthesectortowardssuchafuture,bydevelopingaworkingtechnologyandanimplementationframework.Thereismuchmoreworktobedoneonasector-widelevelbeforewecanseewideradoption.Thisprojecthasleftbehindafertileground,readyforthatwork.DrDeanneGannawaySFHEAAustralianLearningandTeachingFellow,Academiclead:ProfessionallearningInstituteforTeachingandLearningInnovation(ITaLI),TheUniversityofQueenslandd.gannaway@uq.edu.auREFERENCESDavidson,E.(2005).Evaluationmethodologybasics:Thenutsandboltsofsoundevaluation:SagePublications,Inc.Martz,W.(2010).Validatinganevaluationchecklistusingamixedmethoddesign.EvaluationandProgramPlanning,33(3),215-222.doi:https://doi.org/10.1016/j.evalprogplan.2009.10.005Scriven,M.(2007).Thelogicandmethodologyofchecklists.Stufflebeam,D.L.(2001).EvaluationChecklists:PracticalToolsforGuidingandJudgingEvaluations.AmericanJournalofEvaluation,22(1),71-79
TransformingexamsacrossAustralia |
44
FinalEvaluationreport:APPENDIX1From:DeanneGannaway[mailto:d.gannaway@uq.edu.au]Sent:Tuesday,5December20177:10PMCc:DeanneGannaway<d.gannaway@uq.edu.au>Subject:Externalevaluationofe-ExamsprojectThee-ExamsprojectfundedbytheOLTandledbyDrMatHillierisdueforcompletioninDecember2018.Todate22testcaseshavebeenconductedacrossAustralia,acrossmultipleuniversitiesandover1400studentshaveengagedwiththetool.Thesoftwareandassociatedprocessesarenearingfinalisation,providinganopportunemomentintimetotakestockwithcollaboratorsandpartners,sothatanyfeedbackcanbeincorporatedintothefinalstagesoftheproject.ArequirementofOLTfundedprojectsisthattheprojectisexternallyevaluated.IhavebeenworkingwithDrHillierinthiscapacityacrossthelifeofthisprojectandamcurrentlydraftingtheinterimexternalevaluationreport.Iamcontactingyouasyouhavebeenidentifiedaseitherauser,collaboratororpartnerinthisproject.Theexternalevaluationofthisprojectwouldgreatlybenefitfromyourinputregardingthefollowing:
1. Yourviewsonwhetherthematerialsdevelopedachievetheprojectaims2. Opportunitiesandhindrancesyouhaveexperiencedorenvisageinimplementinge-
Examsintoyourassessmentpractice3. Yourexperienceswithengagementwiththeprojectteamtoensureoptimaltrial
experiencesAshortresponsetooneorbothoftheaboveviareplyemailwouldbemosthelpful.Youarealsowelcometoprovidecommentaryonanyotheraspectoftheprojectdeliverablesorprocesses.Yourresponseswillremainconfidentialandyouwillnotidentifiedinthereportunlessyouspecificallyrequesttobesoidentified.TheinterimexternalevaluationreportispubliclycirculatedonlytotheprojectleaderandtotheOLT.Icanforwardacopyoftheexternalevaluationtoyouifyouwouldliketoseeafinalcopy,withDrHillier'spermission.Ifyouwouldbewillingtorespond,couldyoupleasedosobythe15thofDecember?Pleasedonothesitatetocontactmewithanyquestionsorqueries.IlookforwardtohearingfromyouRegardsDeanne