Large-ScaleAssessments:Developingassessmentpublications

SallyRobertson,ACER,CentreforGlobalEducationMonitoring(GEM)

Processfordevelopingassessmentdocuments

• Planning• Writing• Reviewing• Re-writing• Proof-reading• Addpublicationdetails• Formattinganddesign• Finalreview• Printinganddissemination

Planningconsiderations

• Content(informedbydataanalysisandreportingplan)

• Styleandstructureofdocument• Expectationsoforganisation/client• Teaminvolved• Timeframes• Qualityassuranceprocesses

Writingconsiderations

Ensurethat:1.thedocumentisstructuredinalogicalway2.assessmentinformationisreportedaccuratelyandappropriately3.thekeymessagesstandout4.providingsufficientdetailisbalancedwithwritingconcisely5.theinformationisengaging,providinginformationthatisrelevanttoreaders6.clear,consistentandaccessiblelanguageisutilised

1.Documentstructure

• Presentinformationinalogicalorderthatisappropriatefortheparticulartypeofdisseminationproduct

• Useinformativeheadings• Getearlyfeedbackonthedraftstructure• Forreports,includeanexecutivesummary

http://www.keepeek.com/Digital-Asset-Management/oecd/education/low-performing-students_9789264250246-en#page1

2.Accuratereporting

• Includesufficientbackgroundinformationabouttheassessment(why,who,when,what,how)

• Considerwhatdataisavailableandwhatreportingisappropriatefortheanalysis

• Acknowledgeanylimitations• Providereaderswithinformationonhowtointerpret

theresults• Bewaryofover-interpretingtheresults• Referencepriorresearchorotherassessmentprograms

(whererelevant)

Source:https://xkcd.com/552/

3.Keymessages

Somewaystomakeimportantmessagesstandout:•Reiteratekeymessages(rephrasingtheinformation)•Formatting•Considertheplacementoftheinformation(e.g.,inexecutivesummary,chapterintroductionsandconclusions)•Usegraphsandimages•Useexamplestoillustratekeypoints•Limitthenumberofkeymessagesincludedinsummaries

http://img.asercentre.org/docs/Publications/ASER%20Reports/ASER%20TOT/fullasertrendsovertimereport.pdf

http://wwwstaging.acara.edu.au/docs/default-source/default-document-library/20160719-naplan-online-key-messages.pdf?sfvrsn=2

4.Levelofdetail

• Willdependonthepublicationtype,theaudienceandtheimportanceoftheinformation

• Takeintoaccountthelikelybackgroundknowledgeoftheaudience

• Backupstatementswithevidence• Explaingraphsandtablesinthetext• Providesufficientdetailtosupportunderstandingofthe

information• Providereaderswithdetailsaboutwheretheycanaccess

furtherinformation• Includelengthy/detailedtablesinappendix,whereappropriate

https://www.acer.edu.au/documents/PISA-2012-Report.pdf

5.Engagingreaders

• Firstimpressionscount!• Presenttheinformationinawaythatengagesreaders,sothattheywillwanttofindoutmore.

• Readersneedtofeelconvincedthattheresultsarevalidandreliable,andthattheinformationisimportantandrelevant.

• Addressquestionsandpolicyissuesthatwillbeofinteresttothetargetaudience.

• Usewelldesignedgraphs,tablesandimages.• Useawritingstyleandtonethatisappropriatefortheaudienceandtypeofdocument.

http://parlormultimedia.com/publishing/content/raise-bar

http://img.asercentre.org/docs/Publications/ASER%20Reports/ASER%202014/fullaser2014mainreport_1.pdf

http://epathshala.nic.in/wp-content/uploads/2015/NAS/Why%20NAS%20Matters.pdf

6.Clearcommunication

• Writeclearlyandconcisely• Explaintheresultsandpossibleimplications(makethereaders’jobaseasyaspossible)

• Limitacronymsandabbreviations• Explaintechnicaltermsanduseconsistentterminology

• Usetheappropriatetense• Usetheactivenotpassivevoice• Reviewandproofread!

http://research.acer.edu.au/cgi/viewcontent.cgi?article=1012&context=mteg

Reviewingprocess

• Mayincludereviewbyinternalteamandbyclient/externalreviewers

• Peoplewithdifferentareasofexpertisemayneedtoreviewdifferentsections

• Havesomeonereviewthewholedocumenttocheckforconsistency(aneditor)

• Identifywhatfeedbackisneededatwhatstage(forexample,feedbackonthestructure/bigpicture,detailedfeedbackonphrasing)

• Therearemanystagesofreviewingandre-writing–ensuresufficienttimeisallocated

Proof-readingprocess

• Detailedcheckforerrors(forexample,inspelling,punctuation,grammar,references)

• Willoftenincludecheckingdocumentagainstastyleguide

Publicationdetails

• Informationtoinclude:– Listofauthors– Copyrightdetails– Publisherdetails– ISBNand/orE-ISBN

• Thisinformationhelpswithidentification,marketingandreferencingthedocument

Formattinganddesign

• Utilisetheexpertiseofgraphicdesigners,wherepossible

• Identifywhereitcouldbeusefultouseinfographicsandimages

• Itisimportanttoscheduletimeforcheckingaftertheformattinghasbeendone

Printinganddissemination

Considerationsinclude:•Willtherebeonlineand/orprintversions?•Howmanycopiesshouldbeprinted?•Howshouldthedocumentbepublicised?•Howshouldthedocumentbedistributed?

Activity:Developingasummarypamphlet

Ingroups,youwillbedevelopingasummarypamphletbasedonanexistingthematicreport(seeactivitysheet)Thisactivitywillinclude:•readingthereport•writingthedraftpamphlet•receivingfeedbackfromothers(andprovidingotherswithfeedback)•revisingthepamphlet,inlightofthefeedback•reflectingontheactivity

Wewilltrynottotakethisapproach…

http://freshspectrum.com/in-defense-of-the-200-page-report/

Reflection

• Whatwasyourexperienceofthistask?• Whatarethestrengthsofthepamphletyourgroupdeveloped?

• Whatimprovementscouldbemadetothepamphletyourgroupdeveloped?

• Howwouldyouapproachdesigningasummarypamphletinthefuture,foradifferenttargetaudience?

UsefulresourcesCresswell,J.(2017).Disseminationandreporting.InP.Lietz,J.Cresswell,K.Rust&R.Adams(Eds.),Implementationoflarge-scaleeducationassessments.NewYork:Wiley.Forster,M.(2000).APolicyMakersGuidetoInternationalAchievementStudies.Availableat: http://research.acer.edu.au/policy_makers_guides/1Forster,M.(2001).APolicyMakersGuidetoSystemwideAssessmentPrograms.Availableat:

http://research.acer.edu.au/policy_makers_guides/2Kellaghan,T.,Greaney,V.,&Murray,T.S.(2009).Usingtheresultsofanationalassessmentofeducationalachievement.Nationalassessmentsofeducationalachievement(Vol.5).WorldBank,Washington,DC.Availableat:http://elibrary.worldbank.org/doi/abs/10.1596/978-0-8213-7929-5Masters,G.(2014).Assessment:GettingtotheEssence.Availableat: https://www.acer.edu.au/files/uploads/Assessment_Getting_to_the_essence.pdfMasters,G.(2013).Towardsagrowthmindsetinassessment.Availableat: http://research.acer.edu.au/cgi/viewcontent.cgi?article=1017&context=ar_miscRoss,K&Genevois,J.(Eds.).Cross-nationalstudiesofthequalityofeducation:Planningtheirdesignandmanagingtheirimpact.Paris:UNESCO,InternationalInstituteforEducationalPlanning.Availableat: http://unesdoc.unesco.org/images/0014/001470/147093e.pdf

Note:Fortheexamplesusedinthispresentation,pleaserefertotheURLprovidedontheslides