MeasuringOutcomes

Dr.CatherineWadePrincipalResearchSpecialistParentingResearchCentre

SamHaskardManagerChildrenandFamiliesProgram

AnglicareSA

Overview

Catherine:

• Approachestomeasuringprogramoutcomes

• Howtoeffectivelysourceandselecttherightmeasures

ü Interpretingpsychometrics

ü Measuresdatabases

ü Useofvalidatedtoolsvstailoredmeasures

Sam:

• Apracticalexample

Catherine&Sam:

• Exploringoptions

Definingprogramoutcomes

• Gettingclarityonoutcomes– importanceofprogramlogicinclearlyarticulatingthelinkbetweenprogramactivitiesandanticipatedoutcomes.

Whatoutcomestomeasure?

Whatoutcomestomeasure?

Trytoincludemeasuresforboththekeymediatingandintendedoutcomes,however:

• Considertimeframefortheinterventionandevaluation– whichintendedoutcomescouldwereasonablyexpecttoseeoccur?Someintendedoutcomesmightbebeyondthescopeoftheevaluation.

• Considerfeasibilityofmeasurementi.e.,willitbepossibletoobtainameasureofthetargetedintendedoutcome/s?

Whatoutcomestomeasure?

• Consideralsoincludingmeasuresofkeyimplementationandengagementoutcomes:

Implementationoutcomes considerwhatcriticalquantity,qualityandtimingoutcomesoftheprogramimplementationarenecessarytoachieverequiredengagementandadherencewithprogramsstrategies(e.g.,measuresofimplementationfidelity)

Engagementoutcomes considerwhatlevelsofengagementandadherencewithprogramstrategiesarethoughttobeessentialtoachievemediatingoutcomes (e.g.,measuresofclientattendanceandengagementwithprogram)

Whatoutcomestomeasure?

Approachestomeasuringoutcomes

• Quantitativeversusqualitative– wherepossiblecanbebeneficialtoincludeboth.

• Considerneedsoftheevaluationandthetargetparticipantgroup:

• Rigourofapproach

• Culturalappropriatenessofmethod(e.g.,useofyarningcirclesforevaluationsinvolvingAboriginalfamilies)

• Feasibilityofapproachandacceptabilityofmethodsbyfamiliesandprogramfacilitators

Identifyingandselectingmeasures

• Identifyatleastoneindicatorforeachoutcomeofinterest– engagement,implementation,mediatingandintendedoutcomes

• Astrongindicatorwillprovideameasureofthatoutcomethatisclosesttoa‘realworldresult’– morereliableandvalid

• Aweakindicatorwillbeanapproximationonlyandinvolvegreatererror– lessreliableandvalid

Identifyingandselectingmeasures

Identifyingandselectingmeasures

Typesofmeasures

Observational(video/live/audio):providedirectmeasuresoftheparentand/orthechild'sbehaviour,parent-childinteractions,skillacquisition,andgeneralisationofskillsoutsideoftheprogram.Checklists&toolsavailable,ordevelopyourown.

Self-monitoring:e.g.parentrecordsfrequencyordurationofanevent/behaviour,ortopographyofbehaviour(A’s,B’sandC’s).

Self-report:Checklists,ratingscales,questionnaires.

Interviews:structured/pre-preparedscheduleofquestions&prompts.Acquiescence&socialdesirabilitycanbeanissue.

Directtesting:knowledgeorskillassessmenttools(e.g.,vignettes,quizzeschilddevelopmentalassessments).

Measuresdatabases

CfC outcomesmeasurementmatrixonCFCAwebsite:https://aifs.gov.au/cfca/expert-panel-project/families-and-children-outcomes-measurement-matrix

OntarioCentreofExcellenceforChildandYouthMentalHealth: http://www.excellenceforchildandyouth.ca/resource-hub/measures-list-search-results?p=all

UniversityofMinnesotaEvaluationInstrumentSearch:https://cyfar.org/search_evaluation_instruments

MeasurementInstrumentDatabasefortheSocialSciences:http://www.midss.org/tools/search

TheCaliforniaEvidence-BasedClearinghouseMeasurementToolsforChildWelfare:http://www.cebc4cw.org/assessment-tools/

Usingvalidatedmeasuresversuscustomisedtools:

Prosofestablishedmeasures:

• Reliable

• Valid

• Normed

• Guidelinesforscoringandinterpretation

• Enablesbenchmarking

• Manyfreelyavailabletools

Usingvalidatedmeasuresversuscustomisedtools:Howeverneedtoconsiderfeasibilityofusingestablishedtools:

•Fitforpurpose

•Version

•Restrictions

•Time,cost,length,reportingburden

•Instructions

•Skillsarerequiredoftherespondent

•Populationnormsavailable

•Usedwithclientslikeyours?

Usingvalidatedmeasuresversuscustomisedtools:

Someimportantconsiderationsforthedevelopmentofcustomisedtools:

• Ensuresamequestionsareaskedatpreandposttest– nowordingchanges(thiscanleadtochangesinmeaningandmaythereforenotbemeasuringthesamethingpreandpost)

• Useclear,plainlanguageandavoiddouble-barrelledquestions

• Ensuregoodalignmentbetweentargetedprogramoutcomeandwordingofquestions.

Usingvalidatedmeasuresversuscustomisedtools:

Someimportantconsiderationsforthedevelopmentofcustomisedtools:

• Considerwhetherpossibletoco-designquestionswithclientgroupandincorporateapilottestbeforestartingyourevaluationtoobtainclientfeedback

• Acknowledgelimitationsofusingnon-validatedmeasureswheninterpretingfindings.

Dr.CatherineWadeLevel5,232VictoriaParadeEastMelbourne,Victoria,Australia3002

E:cwade@parentingrc.org.au

P:+61386603500F:+61386603599

@WadeCath