January 2015
developmentprogress.org
• Thecallfora‘datarevolution’hasspurreddebatearoundtheinclusionofnewdataandindicatorstomeasureprogresstowardsdevelopmentgoals.Indicatorsofperceptions–basedonaskingpeoplewhatmatterstothemmostandtheiropinionsofchange–couldhelptostimulatepublicdebateandholdpolicy-makersaccountable.
• Keystrengthsofperceptionsdataaretheirtimelinessandfrequency–suchattributescouldmakethemveryusefulaswarningsignalsforpolicyintervention.
• Weillustratethepotentialofperceptionindicatorsinthreepost-2015areas:socialnormsrelatedtogender,violenceandsecurity,andgovernance.Perceptionsandso-called‘objective’datacanmeasurecomplementaryaspectsoftheseareas.Analysinggapsbetweenperceptionsandobjectiveindicatorscanimproveunderstandingofhowpeoplearedissatisfied,orwhenthereareimplementationgapsinthepoliciesintendedtotackletheseareas.
• Mainlimitationsofthisdataarethechallengeofensuringthereliabilityoftheinformationobtained,anddifficultiesinmakingmeaningfulcomparisonsacrossgroupsofpeople.Wesuggestthatperceptionsdatawouldbemoreusefultomonitorchangingsituationsovertimewithincountries,ratherthantoestablishcomparisonsacrossthem.
Key messages
Working paper 413
Shaping policy for development odi.org
Asking people what they think Using perceptions data to monitor the post-2015 agendaLaura Rodríguez Takeuchi and Sébastien Hine with Cirenia Chávez
Contents
Acronyms and abbreviations 5
Acknowledgements 5
Introduction and context 6
The case for perceptions data 7
a) Perceptions data for guiding policy trade-offs 8
b) Perceptions data for accountability 10
Filling measurement gaps 14
a) Gender and social norms 14
b) Governance 15
c) Violence and security 17
Why not perceptions data? Common but differentiated challenges of perception surveys 19
a) Representation: whose responses are represented? 19
b) Reliability: how accurate are people’s responses? 21
c) Adaptation: have poor people come to terms with living in poverty? 22
d) Objective and subjective mismatch 23
Perceptions data in monitoring the post-2015 agenda: implementation issues 26
Conclusions 28
References 29
2 ODI Report
List of tables, figures and boxes
Tables
Table 1: Selected gender and social norms indicators 16
Table 2: Selected governance and effective institutions indicators 17
Table 3: Selected indicators to measure violence and insecurity 18
Table 4: Selected indicators to measure progress on eliminating violence against women and girls 18
Figures
Figure 1: Errors in the process of obtaining estimates from surveys 20
Boxes
Box 1: Income perception questions 8
Box 2: Adaptation, income poverty and subjective wellbeing 24
Asking people what they think 3
Acronyms and abbreviationsDHS Demographic and Health Surveys
FAO Food and Agriculture Organization
FGM female genital mutilation
GWP Gallup World Poll
HDI Human Development Index
HLP High Level Panel of Eminent Persons on the Post-2015 Development Agenda (UN)
IAEG-GS Inter-Agency and Expert Group on Gender Statistics (UN)
IEAG Independent Expert Advisory Group on a Data Revolution for Sustainable Development
IPFRI International Food Policy Research Institute
LSMS Living Standard Measurement Survey
MDG Millennium Development Goals
MICS Multiple Indicator Cluster Surveys
NSO National Statistical Office
OECD Organisation for Economic Co-operation and Development
OPHI Oxford Poverty & Human Development Initiative
OWG Open Working Group
SDG Sustainable Development Goals
SIGI Social Institutions and Gender Index (OECD)
SWB subjective wellbeing
UNDP United Nations Development Programme
WEIA Women’s Empowerment in Agriculture Index (IPFRI)
WVS World Values Surveys
Acknowledgements
WearegratefulforhelpfulcommentsandinputsfromEmmaSamman,JessicaHagen-ZankerandClaireMelamedfromODIaswellasfromLauraCamfield(UEA),HaykGyuzalyan(IpsosMori)andEnricoGiovanini
(UniversityofRome‘TorVergata’).NinaBehrmanprovidededitorialsupport.Theusualdisclaimersapply.
4 ODI Report
Introduction and context
TheprocessofdevelopingandassessingprogressagainsttheMillenniumDevelopmentGoals(MDG)hasbeencriticisedastechnocraticandtoomuch‘behindcloseddoors’.Incontrast,negotiationsaroundthesuccessorstotheMDGhavebeenwide-ranging,seekingtobeconsultativeandincorporatingtheviewsofallgovernmentsandofcitizensglobally.Thisspiritofmakingthepost-2015processmoreinclusiveandresponsivetocitizen’sviewsanddemandsshouldbecarriedintothemonitoringofthenewdevelopmentgoals.
Perceptionsdataareincreasinglywidelyavailable,andcouldbeusefulinmonitoringprogresstowardsnewdevelopmentgoals.Availabledataincludethosefrom:
• theWorldValueSurveys(WVS)conductedinover80countriesandtheGallupWorldPoll(GWP)coveringover160countries
• the‘barometers’(LatinAmerica,Asia,ArabregionandAfrica),theLatinAmericaPublicOpinionProjectandtheEuropeanSocialSurvey,whichhaveamoreregionalfocus
• questionsinLivingStandardMeasurementSurveys(LSMS),whichareinternationallycomparablemulti-topichouseholdsurveysconductedindevelopingcountries
• subjectivewellbeingdata(SWB)collectedbyNationalStatisticalOffices(NSOs)insomecountries1.
Whileexistingdatacouldbeusedascriteriafornarrowingdowntheplethoraofproposedgoalsandtargets,thevastinterestinexaminingnewformsofmeasuring–whattomeasure,andhowtomeasureit–providesagoodopportunitytoexaminenewpossibilities.TheUNHighLevelPanelofEminentPersonsonthePost-2015DevelopmentAgenda(HLP)hascalledfora‘datarevolution’,andtherecentreportoftheUN’sExpertGroupontheDataRevolution(IEAG,2014)promptedwidedebateonpossiblenewapproachesandincorporatingnewmethodsofcollectingandusingdifferenttypesofdata.
‘[A data revolution is] an explosion in the volume of data, the speed with which data are produced, the number of producers of data, the dissemination of data, and the range of things on which there is data, coming from new technologies such as mobile phones and the “internet of things”, and from other sources, such as qualitative data, citizen-generated data and perceptions data.’ (IEAG, 2014)
TheSecretary-General’ssynthesisreport(UN,2014b)alsoclearlyunderlinestheimportanceofsubjectivedatainmeasuringwellbeing.AsarguedbytheStiglitz,Sen,FitoussiCommission(2009),perceptionsdatacouldshiftthemeasurementdebatetowardsimprovingwellbeingofcitizensaroundtheworldastheyunderstandit.Nevertheless,toourknowledge,thereisnospecificassessmentofthepotentialofsubjectivedata,orthechallengesinvolved,inthiscontext.Thispaperaddressesthisissue,framedinthecontextofthemonitoringofthenewsetofdevelopmentgoals.Italsooutlinessomemethodologicalissuesmorebroadly.
Section2definesperceptionsdataandpresentsgeneralinitialthoughtsontheirpotential.Section3considersthepotentialofperceptionsdataformonitoringtheevolutionofgendernorms,violenceandsecurity,andgovernance.Section4concentratesonthechallenges,andinSection5wefocusonpracticalconsiderationsfortheinclusionofperceptionsdatainthemonitoringofpost-2015developmentgoals.
1 NationalStatisticalOfficesincludingSWBmeasuresare:Canada,France,Italy,Mexico,Morocco,NewZealand,UK,US,andmostEUcountries(O’Donnelletal.,2014).
Asking people what they think 5
The case for perceptions data
Perceptionsdata,oftencalledsubjectivedata,concerntheexpressedviews(opinionsorperceptions)2ofpeopleonaparticulartopic.3Incontrast,so-called‘objectivedata’arereportsofmaterialpossessions,eventsorsituationscollectedthroughdirectphysicalexaminationorobservation.Werefertothissecondkindofdataas‘objective’inthispaper,althoughwerecognisethattheycanalsobesubjecttobiasesanddifferentinterpretations.
Thedefinitionweadoptalsoexcludesself-reportedobjectivedata,orwhatsurveyresearcherscallanswersto‘factual’questions.Theserefertoinformationonpastactionsorobjectivelivingconditions,forexamplevotingbehaviourorself-reportedincome,althoughbothmaybecollectedthroughthesamemeansofsurveyquestionnaire.Thisdistinctionisimportant.Takeforexample,aquestionaskinganindividualtoreporttheirhouseholdincome(eitherasanexactfigureorwithinasetrange).Thistypeofquestionisself-reported,ratherthancollectedforexamplethroughtaxrecords,butitisnotaboutperceptionsasdefinedhere.
Whileclearlytherearesomeobjectivelymeasurablethingssuchasanamountoffood(orthenecessaryincometopurchasethatfood),thatpeopleneedtoleadfullandhealthylives,peoplecanalsomakesubjectiveassessmentsoftheseobjectiveconditions.Themainadvantageofperceptionsdataisthattheyprovide‘therespondents’ownviewsdirectly…perceptionsofthemselvesandtheirworld,whichareunobtainableinanyotherway’(Bakeretal.,2002).Forexample,aquestionabouttheperceptionofthehouseholdincomesufficiencywouldbeincludedinourdefinitionofperceptionsdata(Box1).Thesesubjectivejudgementsmatterregardlessoftheobjectivecircumstancesbecausepeoplemayreacttoitiftheycan–withconsequencesforhowtheyrespondtoincentives,andtopoliticalsystemsandprocesses,andforhowpeople
relatetoeachother(Coulthardetal.,2014).Moreover,thereareotherthingsthatarebestassessedonlyinsubjectivetermsbecausewhatreallymattersishowpeopleexperiencethese(Veenhoven,2004;GoughandMcGregor,2009;Coulthardetal.,2014).Veenhoven(2004)usestheexampleofxenophobia.Toadegree,thiscanbemeasuredbyareductionofracistattacksortheratesofinterethnicmarriage,forinstance,butperceptionsareneededtocapturethisconceptfully.Similarargumentsapplyfortheconceptsofhumandignityandsecurity(Coulthardetal.,2014).Perceptiondataarecollectedmainlythroughhouseholdsurveys.ThispaperfocusesoninternationallycomparabledatathatcouldbeusedformonitoringtheagreedSustainableDevelopmentGoals(SDG)/post-2015targets.Somemulti-topichouseholdsurveysincludeselectedquestionstoascertainperceptionsaroundspecificissues.Subjectivemodulescanbeappendedtobroadhouseholdsurveys,asinthe‘missingdimensions’projectoftheOxfordPoverty&HumanDevelopmentInitiative(OPHI).4Therearesomespecialisedsurveysfocusingexclusivelyoncapturingperceptionsdata.TwoofthelargestofthesearetheWVSandtheGWP,conductedinawidenumberofcountriesaroundtheworldfollowingastandardmethodologyandquestionnaire.Theyaskarepresentativesampleofparticipatingcountryresidentsabouttheirperceptionsontopicsrangingfromgovernmentandpolitics,tofamily,religion,ethicsandwellbeing.Theregionalbarometersurveysaresimilarbutthespecifictopicsandquestionsmayvaryintheregionalsurveys.
TheUNMyWorldsurveyisanotherlarge-scaleperceptionsurvey.Resemblingmoreashortopinionpollthanahouseholdsurvey,itconsistsofonlyonecorequestion(‘Which of these are most important for you and your family?’5),aimedatgatheringpeople’sprioritiesforglobaldevelopmentpost-2015.Inaddition,althoughthere
2 Evenwithinthedefinitionadopted,therecanbevarioustypesofdata.Adistinctioncouldbemadebetweenquestionsthataskaboutperceptions(‘howdoyoufeelabout…?’),opinions(‘doyouthinkthat…?’)andchoices(‘shouldthegovernmentdo...or…?’).Themainemphasisisonthefirsttwoofthesetypes,especiallysincethethirdmayhavedifferentmethodologicalproblems.Thatsaid,someissuesarecommontoallperceptionsdata,andevensometoallestimatesbasedonsurveymethods.
3 Experts’valuationscouldbeclassifiedasperceptionsdata,insofarastheyrefertoopinions/perceptions.Althoughexpertviewshavebeenwidelyused,forexampleinthemeasurementofgovernance,weexcludethemfromthedefinitionusedinthispaperasourmaininteresthereisperceptionsdatadrawnfrom(mostlyrepresentative)samplesofthegeneralpopulationratherthanfromselectedgroupsofexperts.
4 http://www.ophi.org.uk/research/missing-dimensions/
5 Thesurveyasksrespondentstochoose6issuesthatmattermostoutof16possibilities.
6 ODI Report
aresomenationallyrepresentativeMyWorldsurveys,formostpartthelargesampleofMyWorld–over7million–hasbeenpossiblebecauseithassoughttobeatrulyworldwidesurveyandcoverawiderangeofcountries,ratherthantobenationallyrepresentative.
a) Perceptions data for guiding policy trade-offs Perceptionsdatacanplaymultiplerolesinpolicyprocesses.Seaford(2013)definessixrolesforsubjectivewellbeingdata:tostimulatepublicdebate,todrawattentiontoimportantissueswhichmayotherwisebeignored,toinformthedevelopmentofeconomicmodels,toinfluencethechoiceofotherobjectiveindicators,toinputintoanewformofcost–benefitanalysis,6andtoholdpoliticiansaccountable.Theserolesapplymorebroadly
Asking people what they think 7
Box 1: Income perception questions
Householdorpersonalincomeisoftenusedasameasureofwelfareandsubsequentestimatesofmonetarypovertyandinequality.Itisalsopossibletoderivepovertyandinequalitymeasuresfromsubjectiveperceptionsofrelativewealth/povertyratherthanactualincome.Somehouseholdsurveyshaveincorporatedthistypeofquestion,forexample:
• Howdoesyourhouseholdincomecomparewithotherhouseholdsinyourvillage/neighbourhood?MuchaboveaverageincomeAboveaverageincomeAverageincomeBelowaverageincomeMuchbelowaverageincome(Donotknow)
• Whatmonthlyincomeleveldoyouconsidertobeminimalforyourhousehold,i.e.yourhouseholdcouldnotmakeendsmeetwithless?
• Isthetotalmonthlyincomeofyourhouseholdhigher,lowerormoreorlessthesameasthisfigure?MuchhigherHigherMoreorlessthesameLowerMuchlower
(SouthAfricanSocialAttitudesSurvey,2007)
• Yourhouseholdincome:IsnotenoughtomeettheminimumexpenditureIsjustenoughtomeettheminimumexpenditureIsabovetheminimumexpenditureDon’tknow/NA
(EncuestadeCalidaddeVidadeBogotá,2007)
• Isyourpayjust?Wearenotaskingabouthowmuchyouwouldliketoearn–butwhatyoufeelisjustgivenyourskillsandeffort.Ifyouarenotworkingnow,pleasetellaboutyourlastjob.
MuchlessthanisjustAlittlelessthanisjustAboutjustformeAlittlemorethanisjustMuchmorethanisjustNeverhadajobCan’tchoose
(ISSPSocialinequality-IV,2009)
• Concerningyourfamily’stotalincomeoverthepastonemonth,whichofthefollowingistrue?Itwaslessthanadequateforyourfamily’sneedsItwasjustadequateforyourfamily’sneedsItwasmorethanadequateforyourfamily’sneeds(TanzaniaKageraHealthandDevelopmentSurvey(LSMS),2004)
toothertypesofperceptionsdata.Welookindetailatthefirsttwointhissection,andtouchonsomeoftheotherrolesinSection2b.
Inthecontextofthepost-2015goals,therehasbeenasystematicattemptincludeassessmentsandperceptionsofdifferentgroupsintheprocessofdefiningthenewgoals.Butbeyondthisparticipatorydecision-makingprocess,thereisgreaterscopeforperceptionsindicatorstoguidepolicy.Whilewellbeingdataindevelopedcountrieshavebeenusedincreasinglytosupportpolicies,theyhavebeenlessusedtodrivethepolicyagenda(Seaford,2013).
Perceptionsdatacanhelptodrivethepolicyagendabyprovidinginformationonpeople’spriorities.Thiscanhelpanalysingtheinteractionsbetweendifferentpathwaysfrompoliciestowellbeing.Forexample,thisdatacanhelptomakevisiblehowpoliciesinhealth,forexample,arecomplementarywiththoseineducationorinsecurityandstimulatepublicdebateontherightmixofpoliciesappropriateinagivencountrycontext.Thisalsopointstothetrade-offsbetweenconflictingobjectivesoramultiplicityofobjectivesthatneedtobeprioritised.
Examplesinthepost-2015contextandbeyondshowhowsubjectiveindicatorscanfulfilthisrole.TheOECD‘How’slife’initiativeandtheBetterLifeIndex7areexamplesofthelatter.Theyidentify11dimensionsofmaterialwellbeingandqualityoflife8forOECDcountriesandothermajoreconomies.Theindexisaninteractiveweb-basedtoolthatinvitespeopletoexpresstheimportancetheygivetodifferentdimensionsofwellbeing.9Thekeyisthatthisexerciseallowsrespondentstocontrastcountriesaccordingtotherelativevaluetheyplaceoneachdimension,ratherthanbyhavingastaticpredeterminedcombination.Moreover,itallowsforanillustrativecomparativeanalysisofhowtheseprioritiesdifferacrosspeoplefromdifferentcountries,ageandgender.10People’sresponsesmakeitpossibletounderstandhowthehighimportanceattributedtohealth,forexample,isrelatedtootherdimensionsofwellbeingandtotheobjectivequalityofhealthcaresystemsinparticularcountries.Inthissense,theworkonnewmeasuresofprogress,suchastheBetterLifeIndex,isusefulinconsideringpublicprioritiesandstrategiesinpolicy-making(UNTaskTeam,2013),
identifyingpoliciesthatwouldcontributemoretooverallwellbeing.
Similarexercisescanbecarriedoutwithmoretraditionalmeasuresofwellbeing,toindicatewhatpeoplevalueandtowhatextent,andtoassesshowwellgovernmentsarerespondingtothosetrade-offsandfulfillingcitizens’expectationsofwellbeing.Forexample,Kroll(2013)proposesameasureofcountryperformanceinhumandevelopmentthatcombinestheobjectiveindicatorsoftheHumanDevelopmentIndex(HDI)butweightedaccordingtotheircontributiontoSWB.ThismeasureassessesthecountrycorrelationofthethreeHDIcomponents(materialconditions,healthandeducation)andpeople’sresponsesinalife-satisfactionquestion.BymeasuringwhichofthethreeHDIelementsaremorestronglyrelatedtolifesatisfaction,itbecomesclearwhichcountriesaremoresuccessfulingeneratingthegoodsthattrulymattertopeople’swellbeing.Krollfindssignificantheterogeneityintheimportanceofthesethreefactorsacross70countries.Forexample,inMoldova,thereisthehighestcorrelationbetweenSBWandincome.Incontrast,inFinlandorTurkey,thecontributionofincometoSBWislow.
Inhealth,useoftheQualityAdjustedLifeYears(QALY)andofPatientReportedOutcomeMeasures(PROMs)isapracticalexampleofhowtheseperceptionsontrade-offscanbeincorporatedinthedecision-makingprocessofpolicy(Melamedetal.,2012).PioneeredintheUK,theexerciseinvolveselicitingthe‘patient’sperspective’throughthousandsofsurveys(knownastheEQ5D11)askingpeopleabouttheirhealth.Foreachofthefivehealthdimensions,individualsratetheirhealthonthedaywhenthequestionnaireiscompleted–noproblems,someproblemsorsevereproblems–andvalueeachofthepossiblehealthoutcomes.12Foranygiventreatment,policy-makersneedtoknowhowmuchvalueisgained,forhowlong,andatwhatcost.OneQALYisequaltooneyearlivedinfullhealth.Ayearinanythinglessthanfullhealthisvaluedatlessthanone,dependingonthesocialvalue(derivedfromthePROMSprocessdescribedabove)attachedtothatparticularoutcome.Throughthisexercise,itispossibletoassessthetrade-offbetweenhealthtreatmentsthatimprovedifferentaspectsofhealth.Foranygiventreatment,thenumberofQALYsgained(eitherthroughimprovementsin
6 TheUKNationalHealthService(NHS)useofPatientReportedOutcomeMeasures(PROMS)asoneinputintomeasuringtheperformanceofdifferenthealthproviders/serviceswouldbepartofthiscost–benefitanalysis.
7 Seehttp://www.oecd.org/statistics/howslife.htmandhttp://www.oecdbetterlifeindex.org/
8 SelectedtomatchthoseproposedbyStiglitzetal.(2009),agreedbyOECDcountriesandreflectingdimensionsofwellbeingthatareuniversalandrelevantforallhumanbeingsratherthanspecificforaparticularcountry.Theyare:community,education,environment,civicengagement,health,housing,income,jobs,lifesatisfaction,safetyandwork–lifebalance.
9 Usershavetorateeachofthe11topicsfrom0(‘notimportant’)to5(‘veryimportant’).
10 Althoughtobeprecise,representativesamplesofthepopulationwouldberequired.
11 Eachofitsfivesectionscorrespondstoadifferenthealthoutcome:mobility(abilitytowalkaboutnormally);self-care(abilitytolookafteroneself);usualactivities(abilitytoperformusualactivities);painordiscomfort;andanxietyordepression.
12 Thereare243differentpossibleoutcomestothequestionnaire(threepossibleoutcomesoneachoffivedimensionsofhealth).
8 ODI Report
qualityorlengthoflife,orboth)canbecomparedwiththetotalcostsofthetreatment(Melamedetal.,2012).
Trade-offsinbudgetallocationsareespeciallyvisible.OneofthesuccessesoftheMDGwastoincreasedevelopmentaidfromdevelopedtodevelopingcountries(Clemensetal.,2007)buttherearedifferencesinwhatdifferentgroupsofpeopleperceivetobethebestuseforthoseresources.Usingperceptionsdatatoillustratehowpeopleperceivetrade-offscanbeparticularlyusefulininformingandmonitoringbudgetdecisionsofdonorcountries,acomparisonthatisinterestinginthepost-2015contextwhereasimilarexerciseofassigningbudgetprioritiesmaybeneeded.Leo(2013)investigateswhetherUSaidisprioritisingthoseissuesthatmattermosttopeopleinaid-receivingcountriesinAfricaandLatinAmerica.RegionalBarometersaskwhatpeoplebelievearethemostpressingproblemsfacingtheircountry.Only16%ofUSassistancehasbeenfocusedonwhatAfricansciteastheirtopthreemostpressingproblems(jobs,incomeandinfrastructure)since2002;nearly60%hasbeentargetedtowardssecondaryortertiaryconcerns(Leo,2013).Bycontrast,Prizzon(2014)findsthat,inEthiopia,CambodiaandZambia,citizens’preferencesmaynotbenecessarilyalignedtothetopprioritiesoftheirgovernments(infrastructure,energyandgrowthcreation)butthattheallocationofaidflowsfromdonorsdoesappearinthosecountriestobecloselyalignedtocitizens’preferences.
Inthecontextofdefiningthenewsetofdevelopmentgoals,informationonpeople’spreferencescouldbeextremelyhelpfulinestablishingprioritiesforimplementingthe17goalssofarproposedbytheOpenWorkingGroup(OWG).BothGallupandtheWVSaskedpeopletoprioritisetheMDGgoalsmostimportanttothem.In26countriesofsub-SaharanAfrica,accordingtothe2006Galluppoll,reducingpovertyandhungerwerethehighestranked,followedbyreducingthespreadofHIV,andprovidingjobs,withaveryconsistentpatternbypopulationsubgroupsbutsomevariationbycountry,particularlyinWestAfrica(Tortora,2009).
Otherinterestingmessagesemergefromperceptionsdatathataskpeopletoprioritiseglobalandcountrygoals.IntheWVS,amongotherproblems,‘peoplelivinginpoverty’wasoverwhelminglyrankedtopacrosscountriesasthemostseriousproblemfortheworld.However,alowershareofpeopleconsidereditthetopcountrypriority.13Moreover,whenaskedaboutwhetherthecountryshouldfocusonsolvingitsownproblemsorhelpingtoreducingpovertyintheworld,mostpeoplethinktheircountry’sproblemscomefirst.14Eveninthemostaltruisticcountries,15themajorityofpeoplewouldprefertoconcentrateontheirowncountryprioritiesrather
thanonglobalones.Theinclusionofsustainability,theenvironmentandclimateinthenewsetofdevelopmentgoalshasbroadenedthescopeoftheSDGbeyondthenationalspectrum.TheinformationpresentedintheWVSalignswiththeexperienceofMDG8‘globalpartnership’(KennyandDykstra,2013),suggestingthatitisgoingtobehardtoensureglobalcommitmenttoissuesthatareglobalinnature,orthatrequirecoordinationandcooperationfromvariouscountries.Apartfromthecoordinationissuesandthealignmentofpoliticalviewsbygovernmentsofdifferentcountries,ensuringpublicsupportwithincountriestotacklethoseissueswillnotbeeasy.
Insum,thevalueofsubjectiveindicatorsisthattheycanmakeexplicittheimplicitassumptionsthattypicallyguidepublicdebate,mainlyaroundtheissuesthatpeopleconsiderimportantinimprovingtheirlives(Seaford,2013;RodriguezTakeuchi,2014).Thiscanbeveryhelpfulininformingpolicydebatesbutmaynotsettlethem(Seaford,2013).Moreover,whileitisimportanttoknowtheseoverallpreferences,perceptionsdatacanalsobeusedtoassesstrade-offsfordifferentgroupsofpeople,forexampletoseethedifferencesbetweenwhatmenandwomen,oryoungerandolderpeople,thinkcouldmostimprovetheirwellbeing.Inthissense,perceptionsdatacandrawattentiontoimportantissuesorgroupsofpeoplewhichotherwisemaybeignored,becausetheywerenotpreviouslyidentifiedasaconcern(Seaford,2013).
b) Perceptions data for accountability‘To ask the people, with regularity, for their own thoughts strikes us as being both useful and a check on the claims of those in power’ (Dionne and Mann, 2003)Thereisnosingleagreeddefinitionofaccountability.Itinvolvesbothanswerability–theresponsibilityofduty-bearerstoprovideinformationaboutandjustificationoftheiractions–andenforceability–thepossibilityofpenaltiesorconsequencesforfailingtoansweraccountabilityclaims,althoughthissecondaspectisoftenleftasideintheunderstandingofaccountability(McGeeandGaventa,2011).Thereisstillmuchdebateabouthowtotranslateinformation,transparencyandparticipationintoaccountabilityandimprovedoutcomes,andwhether,infact,thereisaclearpositiveconnection(McGeeandGaventa,2011).Nevertheless,theverynatureofperceptionsdata–askingpeoplewhattheythinkonarangeofdimensionsoftheirwellbeing–hasthepotential
13 63%consideritaglobalpriorityand58%anationalone.
14 Inascalefrom1to10,theaveragescorewas7.8.About39%ofrespondentsmarked10(‘itisatopprioritytosolvemyowncountry’sproblems’).
15 Finland,Iran,Italy,Malaysia,Norway,Mali,Mexico,Spain,SwedenandSwitzerland,accordingtothesurveyresults.
Asking people what they think 9
togenerategreateraccountability.Thisworksintwoways:first,byopeningthedebatearoundmeasuringandactingonwhatistrulyimportanttopeople;and,second,byincreasingtheavailabilityofinformationandhence,opportunitiestotrackprogress.
Accountabilitystartswiththecreationofsystemsthatallowit.Makingsuretomeasurethingsthatpeopleactuallythinkareimportantandrelevantisafirststep.Thereisawidedebateonwhethersocialaccountabilitycanoccurbymakingrules,proceduresandplanstransparentandopenfordiscussionbeforetheyareimplemented(McGeeandGaventa,2011).Inthissense,perceptionsdatacouldbeusedtoopenthisdiscussionandeveninfluencethechoiceofindicatorsfortrackingoutcomesthatgovernmentswillbeheldaccountableforuntil2030.EvenifperceptionsdataarenotdirectlyincorporatedinthemonitoringindicatorsoftheSDG,theycaninfluencethechoiceanddesignofotherindicatorsandhighlightimportantissuesexpressedbypeoplewhichmayotherwisebeignored.
Thisexercisemayalsorevealthatcommonlyusedobjectiveindicatorsmaynotbereflectingtheappropriateissues.Forinstance,thereoftenseemstobeamarkeddistancebetweenstandardmeasuresofimportantsocioeconomicvariableslikeeconomicgrowth,inflationandunemployment,andwidespreadperceptionsofcountries’developmentprogress(Stiglitzetal.,2009).Insomecountries,thesegapshaveunderminedconfidenceinofficialstatistics,16andinotherstheyhaveresultedinapartialifnotmisguidedanalysisofcountries’andpeople’ssituations.
‘All institutes worldwide knew GDP was rising in Tunisia and Egypt. They knew what 11 million Tunisians and 80 million Egyptians were buying and selling — but they didn’t know what they were thinking. As a result, revolutions in those countries came as a shock. The UN didn’t see those revolutions coming, neither did the WEF nor the World Bank. The U.S. spends tens of billions on intelligence — and it missed those revolutions too.’ (Gallup, 2012)
Perceptionsdatacanthuscontributetothesocialrelevanceofindicatorsbyhelpingtodiscernthemostrelevantissuesforpolicy-makerstoaddress–forexample,byidentifyingobjectiveindicatorsthatbettercorrelatewithpeople’ssubjectivewellbeing.Seaford(2013)points
outthattomeasureacountry’seconomicprosperity,forexample,GDPpercapita(meanGDP)couldbereplacedbymedianincome,whichtakesintoaccountdistributionalissuesandhasagreaterimpactonwellbeingthanaverage(mean)income.Similarly,lookingattherelationshipbetween,say,perceptionsofsafetyandarangeofindicatorsonsafetyandsecuritymaybeusedtoinformtheselectionofanobjectiveindicatortomonitorapost-2015targetonsecurity.Moreover,perceptionsindicatorsaresociallyrelevantbecausetheycollectvaluableinformationondomainsofwellbeingthatareintrinsicallysubjective(GoughandMacGregor,2009)andthusdifficulttomeasurewithobjectivedata.
Inthisway,perceptionsdatacanhelptomakegovernmentsaccountableforsocialprogress,definedintermsofcontributorstomultidimensionalwellbeing,ratherthananarrowsetofoutcomes.Usingsubjectivedatathenmightimprovetheaccountabilityofglobalandnationaldecision-makerswithrespecttotheoverallaimofthepost-2015goals:toimprovepeople’slives.
Inthespecificcaseofindicatorsofgovernance,perceptionsdatacanbeusedtoholdgovernmentsaccountablefordemocraticprogress.Whilemanyobjectivemeasuresofgovernancetendtofocusonformalproceduresorinstitutions(e.g.whetherandhowoftenelectionsareheld,theformalseparationofpowers,theexistenceofananticorruptioncommission),governanceprogressrequiresmeasuresalongvariousdimensions(Forestietal.,2014)thatunderpintherelationshipsbetweenthestateandsociety.Perceptionsdatamayaddtheperspectiveofthosepeoplewhomgovernanceissupposedtobenefit.
Corruptionisakeyexample.Askingpeoplewhethertheythinkofficialswhocommitcrimesgounpunished,orfindingoutthenumberoffirmsthatreportthatinformalgiftsorpaymentsareexpectedtoobtainservices(UNDP,2014),providesinformationonhowpeopleexperiencecorruptionratherthanjustaboutprevalencelevels.Otherperceptionmeasuresimportantintermsofaccountabilityandgovernanceincludetheextenttowhichcitizensfeeltheyareabletoparticipateintheselectionoftheirgovernments,havefreedomofexpressionandliveinacountrywheremediaarefreeandofferavarietyofviews(Thomas,2009).Ifthesemeasuresareproperlyintegrated,theygiveacriticalvoicetopeoplewhopreviouslyhavehadlimitedornovoiceandhadthereforebeenexcludedfromaccountabilitymechanisms.Theimportantpointhereishow,andperhapsif,thedataareused.Dataalonearenotanaccountabilitymechanism.Butifthedataareappropriatelyusedby,forexample,freemediatoraiseissueshighlightedinpeople’sperceptionsthenthiscanmakeadifferencetohowpoliticiansactandrespondtopeople’sneeds.
16 Forexample,inFranceandintheUnitedKingdom.onlyonethirdofcitizenstrustofficialfigures,andthesecountriesarenotexceptions(Stiglitzetal.,2009).
10 ODI Report
Perceptionsdata,especiallyincombinationwithnewmethodsofdatacollection,canalsofillanotherimportantgapinmeasuringoutcomestotrackprogresstowardsthepost-2015targets:theavailabilityandtimelinessofdata.Increasingthefrequencyandcoverageofdataisakeycomponentofthedatarevolution.Thisiscrucialnotonlytoensureaccuracyofthestatisticsusedtoshowprogress(orlackofit),butalsotounderstandtheimpactsofshocks,suddenchangesinwellbeingandinstabilitysituations.Inparticular,perceptionsdatacanactassocialmonitorsandpointtochangesintrends,beforethosearereflectedinothertypesofstatistics.Forinstance,thesubjectivefoodinsecurityscaleoftheFoodandAgricultureOrganization(FAO)showsthepotentialofsuchindicatorstodetectearlyonsetofmalnutritioncrises.Unlikeameasureoffoodavailabilityfromagriculturalproductionortrade,oranthropometricindicatorssuchasunderweightorstuntingprevalence,whichevolveonlyaftermalnutritionbecomesmanifest,aperception-basedfoodinsecurityindexcanbeusedtoidentifyvulnerablepopulationsinatimelyway(Ballardetal.,2013)andevenhelptoidentifyseasonalitypatterns(HeadeyandEcker,2013).
ThissameideawasbehindtheSocialWeatherStation(SWS)project,oneofthefirstperceptionssurveysinthedevelopingworld–theideathat‘surveyscanservelikeobservationpoststomonitorsocialconditions,muchasmeteorologicalstationsmonitorweatherconditions’(Mangahas,2013).Assuch,perceptionsdataofferauniqueopportunitytocapturetheevolutioninpeople’sperceptionsandpotentiallyserveasan‘earlywarningsystem’(AlkireandSamman,2014)17fortheneedforpolicyintervention.Infact,anexplorationoftheSWSself-reportedpovertydataandtheofficialpovertystatisticsofthePhilippinesrevealthat,whileself-reportedpovertyandofficialpovertystatisticsdiffergreatlyintermsoflevels,thetrendstheydescribearebroadlyconsistent(AlkireandSamman,2014).
AlthoughsomeheadlineobjectiveindicatorssuchasGDParetrackedveryregularlyinmoredevelopedcountries,insomeothercountriesequivalentdatacanbeunreliable(seeforexampleJerven’s(2103)argumentaboutcorestatisticsinAfrica)ormuchdelayed.WithrespecttoMDGindicators,forexample,themostrecentglobalreportincludesdatauptofouryearsold:‘Thepovertyfiguresgobackto2010.We’renowinthemiddleof2014,sothat’squitefrustrating’(KeikoOsaki-Tomita18inSciDev,2014).
IncreasingthefrequencyofsurveyscanbecostlyandhighlydemandingforNSOswithlimitedcapacity.Morecompletesourcesofdatasuchascensusestendtobeveryinfrequent,oftenconductedonceeverydecade.Household
surveyshavebeenrisinginnumberbutarestillrelativelylimited,especiallyinlow-incomecountries.Chandy(2013)forexamplefindsthat,inlow-incomecountries,householdsurveysusedtomeasuremonetarypovertyhavebeencarriedoutonaveragejustfourtimessince1980.Similarly,DemographicandHealthSurveys(DHS),keysurveysusedfortrackingMDG,havebeencarriedoutin89countriesfromthelate1980stodate.However,ofthose89countries,26haveonlybeensurveyedonce.19Incontrast,currentperceptionsdatasurveysarecollectedmorefrequentlyandthushavethepotentialtobecollectedinatimelymannerandfrequentlyinawiderangeofcountries.TheSWSprojecthastrackedFilipinos’perceptionsatquarterlyintervalsforover30years.TheGWP,forexample,hasacoverageof160countriesandisfieldedannuallyin93ofthem(AlkireandSamman,2014).Thismaybepartlybecausecollectingperceptionsdataissimplerandlesscostly(seeBallardetal.,2013;HeadeyandEcker,2013,inthecontextoffoodsecuritydata).However,addressingsomeofthereliabilityissues(asdiscussedinSection4)requirestheapplicationofmethodologicalrigour,interviewertrainingandensuringtheappropriatesurveyprotocols,allofwhichcanincreasethelengthandcostofperceptionsurveysinthefuture.Thecostandtimeinvolvedincollectingperceptionsdataisalsocontingentinthespecificityoftheinstrumentsandwaysofdatacollection.20
Finally,acombinationofbigdataandsubjectivedatacanbeusedtoextrapolateharddata,for‘nowcasting’.Thisisimmediate-term(‘now’)forecastingonthebasisofreal-timedataflow(WorldBankandSecondMuse,2014).Thistechniqueisusedbystatisticalofficestofillinthegapsofserieswithlowfrequency,suchasinforecastsoffoodpricesordiseaseonsetusingGoogletrendsanalysis.Forexample,inthecontextofperceptionsdata,ajointWorldBankandUNGlobalPulsepilotexploredthepotentialofsocialnetworkcontent(fromTwitter)toanalysepublicperceptionsofagassubsidyreforminElSalvador.Thestudyusedtextanalyticstoseeiftheresultsfromthesocialmediaanalysis,whichcanbetrackedwithhighfrequency,matchedpublicopinionasmeasuredthroughhouseholdsurveysconductedbeforeandafterthereform.PreliminaryresultsconfirmedthatTwitterdataprovidedausefulcomplementtoanalysethepublicperceptionofapolicyreform(WorldBankandSecondMuse,2014)andthuscanbeusedtocapturetheevolutioninpeople’sperceptions,potentiallyservingasawarningsystem,asmentionedabove.Thiscouldpointtotheneedforpolicyinterventionbeforethefullhouseholdsurveycomesintoplace.AsimilarcasecouldbeenvisionedforexampleforaUNICEF
17 Cummings(2009)suggestedasimilarargumentforsubjectivewellbeingdata.
18 ChiefofthedemographicandsocialstatisticsbranchoftheUN’sStatisticsDivisionandleadauthorofthe2014UNMDGreport.
19 http://dhsprogram.com/data/available-datasets.cfm
20 Theuseofnewtechnologiesmayhelptoreducecostsandtime,althoughthisdiscussionfallsbeyondthescopeofthispaper.
Asking people what they think 11
projectmonitoringsocialmediaandpublicblogpoststotrackparents’attitudestowardsvaccination,particularlyvaccinehesitancyinEasternEurope.High-frequencydataontheseattitudeswouldbeveryvaluableinplanning
interventionbeforetheprogrammecomestoanend,whenverylittlecouldbedonetochangeadverseperceptionsofchildvaccination.
12 ODI Report
Filling measurement gaps
Thissectionconcentratesonthepotentialofperceptionsdatatoguidepolicyandmonitorprogressforparticularpost-2015targetswhereexistingdatagapsaresizeable.Basedonananalysisoftheexistingproposalsforpost-2015,21whereuseofperceptionsdatahasbeensuggestedaspartofthemeasurementstrategy,andtheassessmentoftheUNtaskteamonindicators(UNTaskTeam,2013:ix),weidentifiedthreeareaswherethistypeofindicatorcanbeofparticularrelevancetosupplementobjectiveindicatorsandprovideafullerpicture.Theseare:a)genderandsocialnorms,b)governanceandservicedelivery,andc)violenceandsecurity.Thissectionthisfocusesonhowperceptionsdatacanexpandthepossibilitiesformeasurement.
a) Gender and social norms Goal3oftheMDGisto‘Promotegenderequalityandempowerwomen’.Themaintargetfocusesoneliminatinggenderdisparitiesineducation,withcomplementarytargetsongapsinwageemploymentinthenon-agriculturalsectorandonpoliticalrepresentation.Traditionalgenderindicatorsmeasureprogressonoutcomesofparticularrelevanceforwomenoroutcomegapsbetweenmenandwomen.Examplesarematernalmortality,earlymarriageandfemalegenitalmutilation(FGM)rates.Thesecondtypeofindicatorinvolvesdisaggregatingexistingindicatorsbygender.MDGgenderindicators,forexample,areofthissecondtype:theshareofwomeninnationalparliaments,genderparityineducation,andtheshareofwomeninnon-agriculturalwageemployment.Athirdtypeofindicatorrelatestothecommitmentofnationalgovernmentstoensuringgenderequality.TheInter-AgencyandExpertGrouponGenderStatistics(IAEG-GS)measures11ofthosecommitmentsthroughtheexistenceoftherelevantlegislationorratificationofinternationalconventionsandcommitments.
Mosttraditionalgenderindicators,whetheroutcome-basedorprocedural,arebasedonobjectivedata.ThesetofminimumgenderindicatorsrecommendedbytheIAEG-GSincludes52indicators;22noneisbasedonperceptionsdata.Nevertheless,inthecontextofthenewsetofdevelopmentgoals,achievinggenderequalityandempowermentmayrequirenotonlytrackinggendergapsinoutcomesin2030,
butalsomonitoringtheevolutionofsocialnorms.Socialnormsaredefinedas‘informalandformallaws,beliefsandpracticesthathelptodeterminecollectiveunderstandingofwhatareacceptableattitudesandbehaviours’(Harperetal.,2014:2),andareakeycomponentofprogresstowardsgenderequalityandempowerment.
Inthepost-2015discussions,genderhasfeaturedstronglyingoalsproposedbyboththeHLPandtheOWG.IntheOWGproposal,thegoalisstatedasto‘Endallformsofdiscriminationagainstallwomenandgirlseverywhere’,andtheHLPreportstatesthat‘thesebarriers(toopportunitiesforwomenandgirls)canonlyberemovedwhenthereiszerotoleranceofviolenceagainstandexploitationofwomenandgirls’(HLP,2013:17,italicsadded).Someoftheseformsofdiscriminationwillbereflectedinobjectiveindicators,andratiosbetweenmenandwomenareusefultocapturethem.Butsomeotherformsofdiscrimination,andthezerotolerancedescribedbytheHLP,arehardertocapture.
Perceptionsdatacanhelptomeasuretheevolutionofsocialnormsthatdriveunequaloutcomesforwomenthroughtheirlife-cycleindomainssuchaseducation,health,politicalrepresentationandlabourmarkets(Branisaetal.,2009andOECD,2010(Atlas of Gender and Development)citedinHarperetal.,2012).Forexample,studiesbyHussein(2010)intheUK,orYirgaetal.(2012)anddeCaoandLutz(2014)inEthiopia, trytounderstandthebeliefsbehindthepracticeofFGMratherthanitsprevalence.TheyfoundthatFGMcanoftenbeunquestionedbywomenthemselves,andperceivedasatraditionalrequirement,despitethepaininvolved,toavoidsocialisolation.Thistypeofinformationmaybehelpfultounderstandhow,beyondlegalframeworks,informalsocialnormsunderpinunequalordiscriminatoryoutcomesforwomen.
Indicatorstomeasuresocialnormsusingperceptionsdatahavestartedtoappear.TheOECD’sSIGI(SocialInstitutionsandGenderIndex)23launchedin2009isthefirstcross-countrymeasureofsocialnorms,lookingatthede jure(legal)aswellasthede facto(actual)situationofdiscriminatorysocialinstitutionsinfivecategories:familycode,civilliberties,physicalintegrity,sonpreference,andownershiprights.InSIGI,bothattitudinalandprevalencedataareessentialtocapturethede factosituationof
21 Seehttp://tracker.post2015.org/
22 Seehttp://genderstats.org/
23 http://genderindex.org/
Asking people what they think 13
women.BasedonSIGI,Harperetal.(2014)proposepost-2015targetsmeasuredbychangesinsocialnormsassignalsforgrowingempowermentofwomenandgirls.TheWomen’sEmpowermentinAgricultureIndex(WEIA),24developedbytheInternationalFoodPolicyResearchInstitute (IPFRI),isanotherpioneeringgender-relatedindexthatincludesperceptionsdataatthecoreofthemeasurementofseveraloftheindicatorscomprisingthe‘Fivedomainsofempowerment(5DE)’index.25AsinSIGI,alargenumberofindicatorsthatcomprisetheindexarebasedonperceptionsdata(seeAlkireetal.,2013).
Indicatorsbasedonsocialperceptions(heldbybothmenandwomen)areatthebaseofvariousproposalstomeasureagendergoalpost-2015.Table1presentsselectedexamplesshowinghowbothtypesofindicatorscouldbeusedinvarioussub-categoriesofwomen’sempowerment.Forinstance,measuringatargetonsexualandreproductivehealthcanbedonethroughprevalenceratesofunmetneedforcontraception,butalsothroughresponsestoquestionsaboutperceptionstowardssexualattitudes(inparticularabouttherighttorefusesexwithapartner).TheseareasarenotnecessarilycoveredintheMDGframeworkorallfeaturedintheSDG.However,mostoftheseexampleshavebeenproposedformonitoringthepost-2015goals.Afewadditionalindicatorscurrentlyusedformeasurementwereaddedtothetableforindicativepurpose.Thereisnotnecessarilyaone-to-onecorrespondencebetweentheperceptionandtheobjectiveindicators,butthecontrastisusefultonotewiththedifferentinformationthatperceptionsorobjectiveindicatorsprovide.
b) Governance GovernancewasakeyarealeftoutoftheMDG.Incontrast,theimportanceofgoodgovernanceinitselfandtowardsthefulfilmentofgoalsforothersectorshasbeenrecognisedinproposalsforpost-2015targets.TheHLP’sreport(HLP,2013)hasoneofits13goalsas‘Ensuregoodgovernanceandeffectiveinstitutions’andproposesthefollowingfivetargets:
1) Providefreeanduniversallegalidentity,suchasbirthregistrations.
2) Ensurethatpeopleenjoyfreedomofspeech,association,peacefulprotestandaccesstoindependentmediaandinformation.
3) Increasepublicparticipationinpoliticalprocessesandcivicengagementatalllevels.
4) Guaranteethepublic’srighttoinformationandaccesstogovernmentdata.
5) Reducebriberyandcorruptionandensureofficialscanbeheldaccountable.
TheOpenWorkingGroup’slatestproposal(2014)hasanarrowerfocus,withgoal16includingsomeaspectssuchasbuilding‘effectiveaccountableandinclusiveinstitutionsatalllevels’.Althougheachofthesegoalsisaccompaniedbymorevarioustargets,thedetailofthespecificindicatorsandmeasuresthatwouldbeusedtotrackthesetargetshasnotbeenannounced.Measuringtheseissuesappropriatelywillbekeytounderstandingtheextenttowhichcountriesareprogressing(orregressing)relativetotheirtargets.
Todate,measuresofgovernanceoftenfocusontheformofgovernmentratherthanitsperformance,forexample,ontheexistenceofparticularlegislationorpolicyandtheratificationofconventions(Forestietal.,2014).Governanceisalsocloselyconnectedwithservicedelivery.Whilethequantityaspectofservicedeliveryiswellsuitedtoobjectivemeasures,issuestodowiththequalityandaccesstotheseservicesareoftenbettersuitedtosubjectivemeasures.Thepost-2015discussionhasfocusedonrefiningandexpandingthesemeasuresofservicedeliverybutstillalongobjectivelines.Anoptimumsituationwouldbetheuseofbothobjectiveandsubjectivemeasurestounderstandboththeformandfunctionofgovernance,andwhetheritisdeliveringwhatpeoplewant.Table2showssomeofthedifferentpossibleperceptionmeasuresthatcouldbeused,comparedtoobjectivemeasuresinsimilarareas.
Therearehowever,afewconsiderationswhenusingperceptionsdataonservicedelivery.Perceptionmeasuresaroundeducation,forexample,maybeunreliableifwearedealingwithfirst-generationlearners.Parentswhoarethemselvesnon-literateorwhohavenotbeentoschoolwillfinditdifficulttopassqualitativejudgementonthekindofschoolingbeingprovidedtotheirchildren.Themediacanplayanimportantroleinshapingperceptions.Ifmassmediaarerelativelyfreefromstatemanipulation,theycanactasanaccountabilitysystembothtoinformthepublicofimportantissuesandformembersofthepublictoraisetheseissuesthemselves.Wherethestatehascontrolofthemedia,theycanbeusedtomanipulatepeople’sperceptionsofthequalityofgovernance.Itshouldbenotedthatobjectivemeasuresareasmuchatriskofgovernmentmanipulation,forexamplebyadoptingcertain‘forms’thatleadto‘better’scoresonanindicator,butwithoutrealchangeintheunderlyingaspectsofgovernance(alsoknownasisomorphicmimicry)(Forestiatal.,2014).
Framingeffectsmaybeimportantinevaluatingperceptionsofservicedelivery.Thephrasingofsinglequestionscanhaveasignificantimpactonpeople’sresponses.ThisisdemonstratedinperceptionsdataaroundtheUK’sNationalHealthService(NHS),orinuseofwordssuchas‘assistancetothepoor’versus‘welfare’(Bartels(2003);asdiscussedinSection4bbelow).IntheUK,whenaskedin2011abouttheirlocalNHSprovider,
A
24 http://www.ifpri.org/publication/womens-empowerment-agriculture-index
25 Thefivedomainsare:production,resources,income,leadershipandtime.
14 ODI Report
Asking people what they think 15
Table 1: Selected gender and social norms indicators
Target: ‘End all forms of discrimination against all women and girls everywhere’
Area Perception indicators Objective indicators/self-reported objective proxy
Sexual and reproductive integrity/autonomy Percentage of women and men who think a woman can refuse sex with her husband/partner under any circumstance.U
Percentage of married women with an unmet need*** for family planning.H
Freedom from violence Percentage of women and men who think that a husband/partner is never justified in beating his wife/partner.H,U
Lifetime prevalence of domestic violence.H,S,U
Percentage of women who have undergone female genital mutilation.S,U
Decision-making ability about land, assets and resources
Extent to which the individual feels he or she can make his or her own personal decisions regarding the aspects of household life* if he or she wanted to.W
Relative Autonomy Indicator.** W
Percentage of people who think important decisions in the household should be made by both men and women, by sex.U
Self-reported questions on decision-making and women’s participation in decision-making and control over earnings in households. For example: who decides how to spend money (the wife? the husband? both?)H Self-reported questions on decision-making and woman’s participation in decision making regarding economic activities. For example: If an individual participated in the activity,**** how much input did the individual have in making decisions?W
Proportion of adult population owning land, by sex.U
Proportion of population with access to institutional credit (other than microfinance), by sex.U
Participation in political and civic life Proportion of those of voting age who agree or strongly agree that, on the whole, men make better political leaders than women.H
Score based on legal restrictions or discriminatory practices affecting women’s access to public space.*****
Percentage of women in national parliament.U
Equal value to boys and girls Percentage of respondents who agree that ‘education is more relevant for boys’.H
Missing Women indicator (gender bias in mortality due to sex-selective abortions, female infanticide or insufficient care given to baby girls).H,S
Time Rank their level of satisfaction from 1 = not satisfied to 10 = very satisfied, with the time available for leisure activities such as visiting neighbours, watching TV, listening to the radio, seeing movies, or participating in sports.W
Female-to-male ratio of average time devoted to household activities/unpaid domestic work.H,U
The female-to-male ratio of total workload (both paid and unpaid work).H
Support services and justice for women and girls Proportion of the population who feel safe walking alone at night in the area where they live, by sex.U
Proportion of women over 15 years old subjected to physical or sexual violence in the past 12 months who reported it to the justice system.U
Proportion of law enforcement professionals who are women (including judges and the police).U
Sources: H - Harper et al (2014), W - WEIA, S - SIGI, U - UNWomen (2013)* Activity areas: (a) which inputs to buy, (b) which types of crops to grow for agricultural production, (c) when to take or who should take crops
to market, and (d) whether to engage in livestock-raising.** Formed of three questions: 1. My actions in [activity area] are partly because I will get in trouble with someone if I act differently; 2.
Regarding [activity area] I do what I do so others don’t think poorly of me; and 3. Regarding [activity area] I do what I do because I personally
think it is the right thing to do. *** Women who are fecund and sexually active but are not using any method of contraception, and report not wanting any more children. This
definition points to the gap between women’s reproductive intentions and their contraceptive behaviour and thus it incorporates a perceptions
component (‘not wanting more children’). We place this under the self-reported proxy category.**** About food crop farming, cash crop farming, livestock raising, and fish culture.***** For example, the restrictions on women’s choice of domicile, restricted ability to visit family and friends, requirements for husband’s
approval to apply for a passport or widespread threats of political violence.
77%ofrespondentsagreedthattheywerebeingprovidedwithagoodservice.WhenaskedaboutwhethertheNHSdeliversagoodservicenationally,only63%ofthesamerespondentsagreedthattheservicewasgood.Whenaskedaboutthegovernment’sNHSpolicies,approvallevelsdroppedto28%.Thistrendhasbeenmaintainedsincedatawerefirstcollectedin2003whenthefigureswere76%,48%and30%,respectively(IpsosMori,2012).
c) Violence and security TheMDGdidnotincludeanygoalsortargetsaroundviolenceandinsecurity,althoughmomentumisbuildingfortheirinclusioninthepost-2015agenda.Freedomfromfear,conflictandviolencewasoneofthefivetransformativeshiftstodrivetheuniversaldevelopmentagendaoutlinedbytheHLP(2013).Itisalsoabasichumanrightandacoreelementofwellbeing.TheOWG’sproposal(UN,2014a)includestargetsoneliminatingallformsofviolenceagainstgirlsandwomen,significantlyreducingallformsofviolenceandrelateddeathrates,andendingallformsofviolenceandtortureagainstchildren.
Clearly,someoftheseissuescanbemeasuredwithobjectiveindicators.Traditionalindicatorsincludehomicideandcrimeratesandclashesbetweenarmedgroups(legalorillegal).Thesearebroadlycomparableacrosscountriesandofferinsightsintothelevelofsecurityriskposedtothegeneralpopulation.However,people’sperceptionsofviolenceandinsecuritydonotalwayscorrelatewithobjectivemeasures.Stiglitzetal.(2009)reportthattherecanbesignificantdifferencesbetweenactualandperceivedratesofviolenceandinsecurity,withthemediapotentiallyplayingasignificantroleindistortingpeople’sviews.WhenanalysingthevictimisationratecomparedwiththefearofcrimeinOECDcountriesin2005,nocorrelationwasfoundbetweenthetwo.Similarly,GarrettandAhmed(2004)foundthatinthecityofDinajpurinBangladesh,
despitethehighcrimerate,only10%ofmenandwomenreportedthattheydidnotfeelsafe.
Insituationsinwhichsocialnormsdictatethatcertaintypesofviolenceareacceptable,peoplemayunderreportit.AstrongexampleofthisisaroundFGM.Incountrieswherethispracticeisacceptedandevendesired,thistypeofviolencewillbeunderreportedbygirls(Hussein,2010;Yirgaetal.,2012).Similarly,somepeoplemaybelievethatcertainlevelsofdomesticviolencearetobeexpectedandthereforedonotperceivethemasharmful.Thissuggeststhatweshouldnotrelysolelyonperceptionsdatabutshouldusetheminconjunctionwithobjectivedata.
Large-scaleobjectivemeasureswillgiveonlyasurface-levelunderstandingofthevolumeofviolenceatbest,whereassubjectivemeasurescanhelpelucidatetheunderlyingcausesofthisviolencebyaskingpeoplemorespecificquestionsaroundhowthisviolenceisexperienced.Subjectiveindicatorsaremorenuancedandofferadeeperunderstandingofhowviolenceandinsecurityareperceivedbypeopleandcanhelptounderstandunderlyingdynamicsinfragileandconflict-affectedsituations(Mallett,2012).IntheBangladeshcaseabove,theroleofmastaan(localstrongmen)whoofferpeople‘protection’,wasapossiblecauseforthemismatchbetweenobjectiveandsubjectivemeasures.Nevertheless,thiscameatthecostofthosemostmarginalised,ornotinthefavourofthesestrongmenfigures,pointingtosomeinequalitiesintheprovisionofsecurity.Significantmismatchbetweenperceptionsandobjectivemeasurescanhighlightsignificantproblemsinasociety,forexamplelackofaccesstojusticeortotheinstitutionsinchargeofprotectingciviliansanddeliveringsecurity.
Akeycomponentoftheconceptofsecurityishowsafepeopleperceivetheirenvironmenttobe,andhowtheirwellbeinginteractswiththisperception.Forexample,Møller(2005)findsthattheperceivedlikelihoodofvictimisationandconcernaboutpersonalsafetyhasastrongernegativeimpactonlifesatisfactionthanactuallevelsofvictimisation.
16 ODI Report
Table 2: Selected governance and effective institutions indicators
Target: ‘Ensure good governance and effective institutions’
Area Perception indicators Objective indicators/self-reported objective proxy
Government effectiveness Satisfaction with education system Quality of public schools Increase in tax revenue as a proportion of GDP
Control of corruption How many government officials do you think are involved in corruption? Proportion of public who believe they can receive timely services without paying a bribe
Government efforts to tackle corruption Reduction in number of people who report paying a bribe
Voice and accountability Satisfaction with democracy Freedom of the press Open Budget Index score
Sources: Kaufmann et al. (2009) based on various sources; Foresti et al. (2014) based on various sources
Thereareavarietyofproposalsforsubjectiveindicatorsthatcouldbeintroducedinthepost-2015framework.Saferworld(2013),whichhasoneofthemostdetailedproposalsconcerningviolenceandsecurity,givesseveralexamplesofexistingsubjectiveindicatorsthatcouldbeincorporatedtotracktwoproposedtargets:ageneraltargetaroundviolenceandinsecurity(Table3),andspecificallygender-basedviolence(Table4).TheSecureLivelihoodsResearchConsortium(SLRC)26surveysinconflict-affectedcountriesalsopresentssomevaluableexamples.Again,thereisnotaone-to-onecorrespondencebetweentheobjectiveandperceptionsindicatorsinthetables,whichisindicativeofthedifferentaspectscapturedbyeachtypeofdata.
Insum,thereviewofthesethreeareasshowsthatperceptionsdatacouldserveasausefulcomplementto
objectiveindicatorsofprogress.Objectiveindicatorsareusefulbuttheyhavetheirlimits,andsometimesadditionalperceptionsindicatorsarerequired(Veenhoven,2004;GoughandMcGregor,2009).Forexample,inthecaseofgender,thereductioningapsbetweenmenandwomenmanifestsonlypartlyimprovementstogenderempowermentandequalopportunities.Datareflectingsocialbeliefsongenderrolesandacceptedbehaviourcanalsopointtoimportantadvancementsinachievinggenderequality.Inthissense,subjectiveperceptionsareveryusefultoassesspolicysuccess(Veenhoven,2004)butlesseffectiveforassessingwhyaspecificpolicyworkedandhowtogeneratesuchchanges(Hulmeetal.,2014).
26 http://www.securelivelihoods.org/
Asking people what they think 17
Table 3: Selected indicators to measure violence and insecurity
Target 1: ‘All social groups are free from violence and insecurity’
Area Perception indicators Objective indicators/self-reported objective proxy
Equal justice Do the police treat people equally? (piloted by the Vera Institute of Justice)
Percentage of police, prosecutors and judges that are women
Safety Do you feel safe walking alone at night in the city or area where you live?How safe do you feel now moving to other places e.g. markets or town?
Political stability and absence of violence score
Trust in police How much do you trust the police?
Police capabilities How easy or difficult is it to get help from the police? Number of convictions over number of police
Sources: Saferworld (2013) based on various sources; SLRC surveys
Table 4: Selected indicators to measure progress on eliminating violence against women and girls
Target 2: ‘Violence against women and girls is eliminated’
Area Perception indicators Objective indicators/self-reported objective proxy
Violence against women Percentage of population believing that a husband is justified in hitting or beating his wife/partnerPercentage of women vs. men who believe that the police would respond if they reported a crime (piloted by Vera Institute of Justice)
Number of recorded rapes per 100,000 women and girlsHomicides of females per 100,000 females
Source: Saferworld (2013) based on various sources
Why not perceptions data? Common but differentiated challenges of perception surveys
Interestisincreasingfromgovernmentsinunderstanding,collectingandusingperceptionsdata.Awiderrangeofresearchhelpstounderstandthemainconcernsandissuestoconsiderhere.Thedataalsohavesomelimitations,whichwediscussinthissection.Thus,itmaybeessentialtocomplementperceptionsdatawithobjectivemeasuresoradministrativerecordsfromothersources.
Theissuesdiscussedinthissectionarenotspecifictothepost-2015context,butratherrelevantingeneraltothecollectionofperceptions(andothersurvey)data.Becauseperceptionsdataarecollectedthroughindividualorhouseholdsurveys,keyissuesinusingthistypeofdatarelatetosurveymethodsandtheirinherentlimitations.Errormayarisefromwell-knownflawsaffectingsurveys.Thisdoesnotimplymistakes,inthecolloquialsenseoftheword,butratherdeviationsfromwhatisdesiredinthesurveyprocess(Grovesetal.,2009).Inbroadterms,surveyerrorscanbedividedintotwogroups:errorsofobservationormeasurement;anderrorsofnon-observation,whichrelatetotheprocessofobtainingstatisticsfromasampleofthepopulation(Figure1).Wetouchonsomeoftheissuesofrepresentationfirst,thenfocusonaselectionofmeasurementerrorsourcesthatcouldbeimportantinthecontextofcollectingperceptionsdata,particularlybecauseofthenatureofaskingpeopletoself-reporttheiropinions.Thisreviewisnotintendedtobe
exhaustivebutshouldindicatethemainissuesofreliabilityofperceptionsdata.27
a) Representation: whose responses are represented?First,asinanysurvey,possibleconcernsinclude:therepresentativenessofthesample,samplenumbers,lowresponseratesandaccessibilityissueswheninfrastructureislimited.28Adoptingarigoroussamplingmechanismisanecessarystepforanysurvey,asisreportingonsamplingerrors,tohaveasenseofhowclosethesampleistothepopulationofreference.29Nevertheless,incomparisonwithsomelarge-scalemulti-topichouseholdsurveyscurrentlyusedinthemonitoringoftheMDG,existingperceptionsurveystendtohavemuchsmallersamplesizes.Forexample,thesamplesizeintheGallupWorldPollisaround1,000peopleandupto2,000inlargecountriessuchasChinaandRussia.30Thesurveysalsohavesamplesizesofabout1,000peopleineachcountry,althoughinsomeexceptionalcaseslargersampleshavebeendrawn.31Incontrast,astandardDHShasasampleofbetween5,000and30,000households.32MultipleIndicatorClusterSurveys(MICS),asusedbyUnicef,alsotendtohavelargesamples;themostrecentMICSsurveyforVietnamforexample,interviewedcloseto10,000householdsand10,000women.33Ifperceptionssurveysbecamepartofthe
27 Foradetailedreview,seeGrovesetal.(2009)orKrosnikandPresser(2009),amongothers.
28 See,forexample,Traugott(2003)forsomeoftheissuesconcerningelectoralpollsintheUS,andHerbert(2013)onperceptionsurveysinconflict-affectedcountries.
29 Seehttp://www.aapor.org/Best_Practices1.htm#.U9Z1_PldU9Yforbestpracticesinperceptionsurveys.
30 http://www.gallup.com/poll/105226/world-poll-methodology.aspx
31 http://www.hks.harvard.edu/fs/pnorris/Acrobat/Global%20Communications/Technical%20Appendix%20%20B%20List%20of%20Countries.pdf
32 http://dhsprogram.com/What-We-Do/Survey-Types/DHS.cfm
33 http://www.childinfo.org/files/Viet_Nam_2013-14_MICS_KFR.pdf
18 ODI Report
SDGmonitoring,orperceptionsmoduleswereconsistentlyaddedtoexistinglargerhouseholdsurveys,perceptionsdatamightbecomemuchmorerepresentative.
Evenwiththebiggersamplesizesoftraditionalhouseholdsurveys,theremaybeissueswhentryingtogatherdataforspecificpopulationgroupslikelytobeunrepresented.Moreover,thereisabiggerpointrelatedtoakeyfocusoftheSDGinfinishingtheMDGtaskbyreachingthehard-to-reachand‘leavingnoonebehind’.Aresurveysreallygoingtobethebestwaytogathertheexperiencesofthesegroups?Thishasbeenhighlightedasproblematicforinstanceinthecaseofthoselivingininformalsettlementsofurbanareas,olderpeopleandpeoplewithdisabilities(Carr-Hill,2013;LucciandBhatkal,2014;SammanandRodriguezTakeuchi,2014).Moreover,thepossibilityofproducingrepresentativeresultsforpopulationsubgroupsbeyondgender,placeofresidence(rural/urban),andsometimesageandnationalsub-region,getssmallerwithlowersamplesizes.Surveysneedtobespecificwhenreportingresultsabouttherepresentativenessoftheirdata,sothatitisclearwhatlevelofdisaggregationispossible.Itmaybenecessarytoincreasesamplesizes
tobeabletoprovidesufficientinformationthatdoesnotmasktheheterogeneityofpointsofview.
Insomecases,actingonaggregatepreferencesmaynotbedesiredforequityreasons.Theanalysisofsubjectivedatamaycontributetopost-2015equityobjectivesbyhighlightinghowpopulationsubgroupsmaybenefitfromthedifferentgoalsandmaybeunevenlyaffectedbydifferentpolicies.However,thevariableeffectofdifferentprioritiesandtrade-offsondifferentsubgroupsofpopulation,accordingtoage,gender,socioeconomicposition,culture,andmanyothercharacteristics(Bellanietal.,2013),34mayalsoposeanimportantchallengeinusingthesedata.Inparticular,therecouldbesystematicbiasesunderlyingperceptionsdata.Forexample,thewell-known‘U’relationshipbetweenlifesatisfactionandage(BlanchflowerandOswald,2011),thegenderdifferencesinreportedlifeevaluations(e.g.Boarinietal.,2012),orthehigherdissatisfactionwithdemocraticinstitutionsreportedbyyoungpeople(Berghetal.,2014)maypointtoanimportantpolicyissuetobesolved.Further,unlessweknowmoreaboutthereasonsforthesystematicdifferences,thereistheriskofinformingpolicyonthebasisofincorrectinformation.
34 Inprinciple,‘theycanvarybetweenindividualsalmostcontinuously,thoughforpolicypurposestheidentificationofdiscretetargetgroupsmaybeeasiertointerpretandmoreusefulforthedevelopmentofpolicy’(Bellanietal.,2013).
Asking people what they think 19
Figure 1: Errors in the process of obtaining estimates from surveys
Measurement/ observation
Design
Measurement
Response
Adjustments to data
Representation/ sample selection
Target population
Sampling frame
Sample
Respondents
Coverage error
Sampling error
Non-response error
Validity
Measurement error
Processing error
Source: Adapted from Groves et al. (2009)
Aseparateissueconcernsgatheringperceptionsinareaswheredatacollectionisdifficultbecauseittouchesonsensitiveissues.Manyofthequestionsproposedtodatefortheareasofgendernorms,violenceandsecurityandgovernancemaybedifficulttogatherforthisreason.Responseratesappearhighforsomeperception-basedquestions.IntheGWPandtheWVSforexample,(objective)incomequestionshadbetween10and100timeshighernon-responseratesthandidsubjectivewellbeingquestions,dependingonthecountry(Smith,2013,citedinOECD,2013).Forothertypesofperceptions,itemnon-responsemaybeabiggerproblem,althoughtheevidenceislimited.Forexample,non-responsecouldbehigherforquestionsoritemsthataredifficulttocomprehend,offensivetoparticipants,orconcernsensitiveissues,andalsoinsurveysthatrequirelongerinterviewtimes.Non-responseinself-reportedquestionsaboutlivingconditionshasbeenanalysed(forexample,RiphahnandSerflinf(2002),forincome),anddirectsurveyquestionsonsensitiveissuesarelikelytoresultinlowparticipationandmisreportingofanswers,particularlyforthosewithsomethingconsidered‘embarrassing’toreport(TourangeauandYan,2007;Tourangeauetal.,2010).However,thatevidenceislimitedtoself-reportedproxiesofobjectiveoutcomes(forexampledruguse,voting,income),whichfallbeyondourdefinitionofperceptionsdata.Theextentofitemnon-responseforotherperceptionquestionsneedstobeexaminedfurther.
Guidanceandbestprocedurestominimisequestionandsurveynon-responsehavebeendevelopedforsectionsofhouseholdsurveystouchingonsensitiveissues,andcouldbeadaptedtosurveysthatareexclusivelydedicatedtoperceptions.Forinstance,since1998-99,DHSsetthetaskofdevelopingastandardisedapproachtomeasuredomesticviolence,accompaniedbyguidelinesontheethicalimplementationofthequestionnaire.35Someofthecriteriaare,forexample,toensureprivacyandanonymityoftheresponses,havinganinterviewerofthesamesexastherespondent,askingsensitivequestionstowardstheendofthesurveywhenrapporthasbeenbuiltup,ormakingavailableinformationonorganisationsthatprovideservicesorreferralstovictimsofdomesticviolence.Thesemeasuresarealsohelpfultosolvesomeofreliabilityissuesdiscussedbelow.
b) Reliability: how accurate are people’s responses?Thenextissueistoensurethattheresponsesproducereliableinformation,indispensablefortrackingthepost-2015targets.Pollsandpollstersoftenfacealackoftrustandareespeciallycriticisedwhenopinionmovesinthe‘wrong’direction(DionneandMann,2003),
whichcouldbecontrarytoeitherindividualbeliefsorthepredominantsociallyacceptedview.Questionsabouttheindependenceofthepollstersareoftenraised,butamorefundamentalquestionconcernsthereliabilityofself-reportedinformationandtheunderlyingquestionnaireusedtoobtainsuchinformation.Peoplemayunder-orover-statetheirresponsesorreportfalseinformation,eitherintentionallyorunintentionally,andsuchmisreportingmaybiassubjectiveindicators.Unlikewithobjectiveindicators,cross-validationisnotpossiblebysimpleobservationoflivingconditionsoradministrativerecords.Thewayinwhichquestionsareasked,andthequestionnaireitself,areperhapsthemostimportantconsiderationsinreducingmeasurementerror(KrosnikandPresser,2009).
Wehighlightfourkeyproblemsthatcanleadtoinaccurateinformationonperceptions:
• Recall bias:thisoccurswhenindividualsmakemistakesinrecallingpastevents.Forexample,whenaskedabouthowmanytimesduringthepastweek/monththeyexperiencedacertainfeeling(happiness,anxiety,etc.).Thismaylimitreliabilityofanswersrequiringparticipantstocomparecurrentandpastperceptions.
• Social desirability bias:thisoccurswhenparticipantsrespondwithanswersthoughttobemoresociallyacceptableordesirable,orwhentheyrestricttheiranswersforfearofrepercussions.Forexample,inconflictsituations,peoplemayfearreportingtheiractualperceptionsofthestate(Herbert,2013).Socialdesirabilitymayalsobepresentwhendiscussingillegaltopics(e.g.relatedtosexualbehaviourordruguse).Thistypeofbiasmaybeparticularlyrelevantinhighlypoliticisedsocietiesorwherefreedomofexpressionislimited.
• Framing effects:thisoccurswhenthewayinwhichquestionsareaskedaffectstheanswersgiven.Thistypeofbiascaninteractwithsocialdesirabilitybias.TakeforinstancetheUSGeneralSocialSurvey(1984,1985and1986):respondentswereaskedwhethertheythoughttheircountrywasspending‘toomuch,toolittle,orabouttherightamount’oneachofavarietyofgovernmentprogrammes.Whileonlyaquarteroftherespondentseachyearsaidthattoolittlewasbeingspentonwelfare,closeto65%saidthattoolittlewasbeingspentonassistance to the poor.Thesedifferencesareattributedtothedifferentconnotationsoftheterms‘welfare’and‘assistancetothepoor’intheUS(Bartels,2003).Questionsthat‘lead’theanswermaysufferfrombothframingeffectsandsocialdesirabilityissues.Forinstance,questionssuchas‘Doyouthinkthat...?’and‘Don’tyouthinkthat...?’pull‘yes’forananswer(Barkeretal.,2002).Ontheotherhand,insomecasesitmaybeusefultoincludesuchimplicitpremisesofacceptedbehaviourtoencouragetherespondent
35 Giventhattheyarepartofthesamesurvey,asisoftenthecasewithquestionsonexperienceofdomesticviolenceandperceptionsofdomesticviolence,thisguidanceisalsoappliedtotheperception-questionssectionofthesurvey.
20 ODI Report
tospeakhonestly.Forexample,studiesonsexualbehaviouroftenask‘Howoldwhereyouwhen…?’ratherthan‘Didyouever…?’(Barkeretal.,2002).Theorderofquestionsorevenwhethertherespondentisaloneorincompanymayleadtodifferentresults.Sensitivequestionstendtobeleftforlastwhentherehasbeentimetobuildrapportbetweenandtheintervieweefeelsmorepositiveaboutexpressinghertrueopinion.
• Proxy respondent:whenaquestionisaskednotdirectlytothepersoninvolved,buttoanominatedhouseholdmember,forexampletheheadofhousehold,themother(ofachild),oranyavailableadultinthehouseholdatthetimeofinterview.Bardasietal.(2011)showthatthiscangreatlyaffectestimatesderivedfromhouseholdsurveys.
Measurementerrormaybeminimisedthroughgoodevidence-basedquestionnairedesign.Forexample,thiscaninclude:usingrecordingsheets–formswhererespondentswritedowntheinformationoverthecourseofaperiod–toreducerecallbias;havinganinterviewerofthesamegendertobuildrapport;usinganonymousresponses,asforexampleinMyWorld,toencouragehonestanswersaboutsensitiveissues;introducingprobingquestionsandusingplainlanguagetoavoidconfusionandmisreporting.
Thereareparticularproblemswithself-reportedproxiesofobjectiveindicators,likelytoresultinunderestimatesofprevalenceofsensitiveissuessuchasFGM,domesticviolenceordruguse.Inamoredetailedreview,AlkireandSamman(2014)concludethatitisproblematictouseperceptualdataasproxyforobjectivedeprivations.
Perceptionquestions,asdefinedinthisreport,arenotconcernedwiththeindividualexperienceorconditionbutratherwiththeperception.Assuch,framingquestionsinsuchawaythattheydonotimplydirectdisclosureofasociallyundesirablebehaviourmayreducethereliabilitybias,althoughitmaynotcompletelyeliminateit.Forexample,inastudyonperceptionsofFGMinEthiopia,deCaoandLutz(2014)usealistexperiment36asanalternativefordirectquestioningtoobtainperceptionsaboutFGM.Thisindirectmethodresultedinhigherdisclosureofacceptanceofthepractice,comparedtodirectquestioning.Thismethodhasalsobeenusedtoaskaboutracialattitudes,voterbehaviour,thesocialacceptabilityofsexualattitudesandillegalmigration.
Thepotentialtoreducebiasbyusingindirectquestioningispromising,butadeeperissueisthatperceptionsfluctuateacrossgroupsofsocietyandovertime,andresponsesshowsystematicdifferences.Thesefluctuationsneedtobeseparatedfromreliabilityissues.37
Distinguishingbetweentruechangesinperceptions,variabilityacrosstypesofrespondentsorsometypeofbiasremainsacontestedissue,anditisasyetunclearwhatthegoalpostshouldbeforcorrectingthepotentialbiases.Forexample,inthestudyofFGMattitudesinEthiopiaforexample,theauthorsattempttofindbiasesbycomparingtheanswersfromdirectandindirectquestions;findingthatuneducatedwomenwhohadpreviouscontactwithalocalNGOweremorelikelyshowasocialdesirabilitymisreportingbias(deCaoandLutz,2014).Incomparison,IvanynaandShah(2009)comparecountryrankingsofstatecapacityusingcitizenorexpertperceptions.Theyadjustthecitizen-basedanswersforthepossibilitythatrespondentsareafraidtotellthetruthabouttheirgovernments(measuredbythefreedomofexpressionintheircountry),areindoctrinated(whenrespondentsthinkwelloftheirgovernmentsbecauseofexposuretofavourablemass-mediacoverage)orareexcessivelyoptimisticorcritical.Thesecorrectionsareveryinteresting,yetdepartfromthepremisethatsuchbiasesaredrivingthedifferencesbetweentheexpert-basedandcitizen-basedperceptionsofthestate.Inthiscase,theexistenceofreliabilitybiasmaybehardertoprovethantocorrect.
c) Adaptation: have poor people come to terms with living in poverty?Anotheraspectthataffectsthereliabilityofperceptiondataisadaptation,particularlyrelevantwhenconsideringthat,unliketheMDG,manyifnotalloftheSDGtargetsarelikelytobeappliedtoalltypesofcountriesandpeoples–poorandrich,andfromavarietyofbackgroundsandcultures.Unlikethereliabilityissues,theadaptationproblemdoesnotimplythatpeoplemisreportanswers.Infact,peoplereporttheiractualperception,buttheinterpretationofresultsbecomesdifficult.Whilethisdoesnotunderminethevalueofperceptionsdata,itaffectshowtheyshouldbeusedbypolicy-makers–forexample,tounderstandhowtoapproachpeoplewithideasandpotentialpoliciestomakeimprovementstotheirlives,ratherthantodecidewhetherornottomakesuchimprovements.
Therearetwotypesofadaptation.Thefirstresultsfromhabituationtoone’slivingconditionsandtheconsequentabandonmentofwants,aspirationsandexpectations(Neff,2012).Ifadaptationhastakenplace,theassessmentindividualsmakeoftheirwellbeing(orotheraspectsoftheirlives)maybelimitedbytheirlifecircumstances.Sen(1999)forexample,arguedthat:
36 Thisisanindirectwaytoaskaquestion.Themethodpresentsrespondentswithalistofstatementsandasksthemindicatethetotalnumberofstatementswithwhichtheyagree.Therespondentsarerandomlydividedintoacontrolandatreatmentgroup.Thecontrol-grouprespondentsreceivealistofnon-sensitiveitems.Thetreatment-grouprespondentsreceivethesamelistofnon-sensitiveitemsplusonesensitiveitem,forexampleaboutFGM(e.g.,‘Agirlshouldbecircumcised’).Thedifferenceinthetotalnumberofitemsbetweencontrolandtreatmentgroupidentifiestheproportionofpeopleinthepopulationthatagreeswiththesensitiveitem.
37 Forstandardsurveys,acommonwaytoassessreliabilityistoapplyquestionnairesvariousseveraltimestosimilarsubjectsandinsimilarconditionsandcomparetheresponses.Reliablequestionswouldresultinthesameanswersreportedeachtimethequestionnaireisapplied.
Asking people what they think 21
‘The deprived people tend to come in terms with their deprivation because of their sheer necessity of survival, and they may, as a result, lack the courage to desire any radical change, and may even adjust their desires and expectations to what they unambitiously see as feasible.’ (Sen, 1999: 63)
Adaptationmaychangetheplaceinthereferencescalewhereindividualsplacethemselveswhenansweringperceptionquestions.Forexample,inthequestionsaboutincomesufficiencypresentedinBox1,whatconstitutes‘sufficient’maybedeterminedbyone’sactuallivingconditions.Inthisscenario,peoplelivinginpovertymaysaythattheirincomeissufficient,whilesomeonewithhigherincomesmaysayitisnot.Inconsequence,ifsuchadaptationexists,itlimitsthecomparabilityoftheanswersforthesetwotypesofpersons.Thisisalsoproblematicintermsofcomparabilityacrossgroupsbecauseitmakesithardtodeterminewhetherthereasonsbehindthevariabilityofresponsesaretruedifferencesinperceptionsoraretheproductofadaptation.
Thesecondtypeofadaptationoccurswhenextraordinaryeventstemporarilychangepeople’sperceptionsandbiassubjectiveresponses,particularlyaroundsubjectivewellbeing.However,researchhasshownthatsuchbiasestendtobeonlyshort-livedandsubjectivewellbeingreturnstoastablelevelaftertheevents.Thisiscalledhedonicadaptation.Forexample,thechangeinreportedsatisfactionthathappensafterimportantlifeeventssuchasgettingmarried,havingchildrenorwinningthelotteryisoftentemporaryandtheoveralleffectonlifesatisfactionorlong-termhappinessmeasuresisminor(KahnemanandKrueger,2006;Gilbert,2004).However,theevidenceseemsmixedonthedegreeofhedonicadaptation.Somehavefoundthatpeopleadaptmoretopositivethantonegativeshocks(Burchardt,2005);othershavefoundthatforhealthshockssuchasdisabilityonset,fullhedonicadaptationdoesnottakeplace(FafchampsandKebede,2007;OswaldandPowdthavee,2008).Largeandrandomlychosensampleswouldhelptocontrolforunobservedeffectswhenthoseareunsystematicallypresent(ArnesenandTrommald,2005);usingasampledrawnfromagroupwithenoughvariation,onecanexpectthat,inaggregate,indicatorswouldbemorereliable(RodriguezTakeuchi,2014).
Itisimportanttohighlightthatempiricalevidencesuggeststhattheissueofadaptation,atleastinthecaseofincomepovertyandsubjectivewellbeing,islessstrongthaninitiallysupposed(Box2).Infact,recentresearchshowsthatdeprivationdoesnotneedtotranslateintoadaptation.Findingssuggestthatsubjectivewellbeingisdependentonthefulfilmentofbasicneeds(Veenhoven,1988),butthatincomeandbasicneedsfulfilmentarenot
thesoledeterminantsofhighersubjectivewellbeing.Forexample,Burchardt(2005)findsevidenceofadaptationamongthetopincomegrouponly,suggestingthatthoseonconstanthighincomeshaveadaptedtothatlevel.Similarly,thefamousEasterlinparadox(1974)alsoshowsthatwhilecountryincomehasapositivecorrelationwithsubjectivewellbeing,thereseemstobeathresholdabovewhichthereturnsfromincomearelower.ThiswascorroboratedbyCamfieldetal.(2009)atthemicrolevel.
Adaptationcanbeseeninmanyotherareas.Forexample,researchonwomen’sdemandsforgenderequalityshowthatingender-unequalcountries(measuredbyobjectiveindicators),thedemandforgenderequalityisstronger(Bhatkal,2014).Paradoxically,whilethesewomendemandequality,theyalsocontinuetoholdviewswhichreinforcegenderinequality.Forexample,withrespecttotherighttoeducationandworktheymaystillexpressagreementwiththestatementthatboyshavemorerighttoeducationthangirls.Thishintsatthepresenceofadaptationbecauseitsuggeststhatwomeninmoreunequalsocietiestendtoundervaluewomen’srights.Inthissense,peoples’framesofreferenceareaffectednotjustbytheirown(absolute)livingconditions,butalsobycomparisontothosearoundthem.Thissituationmaystillpointtothepresenceofverystrongnormsonacceptedsocialattitudes,andthuswouldremainagoodindicatoroftheevolution(orlackofit)ofinformalgendernormsinagivencountry,asdiscussedinSection3a.
However,iftheadaptationbiasissystematicallypresentforonesubgroup(i.e.women,thepoororthosemoreexposedtomassmedia,asinthecasepresentedbyIvanyaandShah(2009),thiscouldbecorrectedforusingeconometrictechniques,forexample(seeDecancqetal.,2009).Asdiscussed,themainissueisthatidentifyingthepresenceofabiasisnotstraightforward,asthereisnogoldstandardorcleargoalpostofcomparison.Moreover,discardingsystematicvariationsinresponsesacrossgroupsisnotdesirable.Rather,oneadvantageidentifiedforperceptionsdataistohighlightthosedifferences,aswellastheopinionsofgroupsthatotherwisecouldbemadeinvisiblebyusingaverages.Cross-validationofindicatorswithothermethodologiesisalwaysdesirable,andnecessarywhenusinganykindofindicatorsfornationalpolicydiagnosis.
d) Objective and subjective mismatchInlinewithStiglitzetal.(2009),wearguethatobjectiveandsubjectiveindicatorscouldbecomplementaryratherthanalternatives,astheymightbemeasuringdifferentaspectsofaphenomenon.Inthatcase,thereisnoreasontoexpectperfectcoincidencebetweenthetwotypesofdata.Ratherthanbeingaproblemwithperceptionsdata,amismatchbetweenthetwotypesshowsthattheyeachcontributedifferentinformationthatisusefulandnecessary.Ifthetwoweregivingthesameinformation,
22 ODI Report
therewouldnotbeaneedforusingperceptionsdata,butrathereffortswouldneedtobefocusedonperfectingtheexistingindicatorsandmethodsofdatacollection.
Stiglitzetal.(2009)pointoutthatareasonbehindthemismatchbetweensubjectivewellbeingandtraditionaleconomicindicatorsofcountryprogress(i.e.GDP)istheincompletenessofthelatterintermsofcapturingsomephenomenawhichhavealargeimpactonwellbeing.WhileGDPpercapitamaybeagoodindicatorofeconomicactivity,itisinsufficienttomeasureacountry’sprogressinwellbeing,aconceptthatencompassesmuchmorethannationaleconomicprogress.Moreover,therelationshipbetweenobjectiveandsubjectivewellbeingmaybenon-linear,thatis,itchangesdependingonthespecificlevelsoftheindicators.AsdiscussedinthecontextofadaptationandSWBmeasures(seeBox2),materiallivingconditions,includingincome,areimportantdeterminantsofsubjectivewellbeingatlowlevels.Butoncethemostbasicmaterialneedsarecovered,therelationshipmaybecomemoreindirect,andotheraspects–suchasmentalhealth,communityandfamilyrelationshipsorenvironmentalconditions–canplayagreaterroleindrivingwellbeing.
Themismatchmayalsoariseinothertypesofindicators.HeadeyandEcker(2013)findaverylowcorrelationbetweenself-reportedandothertypesofmeasuresoffoodsecurity.Thisispartlybecausethedifferentindicatorsaremeasuringdifferentaspectsoffoodsecurity.Forinstance,the2010FAOStateofFoodInsecurityreportuseddatamainlyonfoodavailabilityandaveragecaloricrequirements.Thisplacedalargeemphasisonfoodimporting,andunderestimatedtheroleofnationalfoodproductionanddistributionwithincountries.ThefoodinsecurityexperiencescalefortheVoicesoftheHungryproject38doesnotprovidespecificinformationonactualfoodavailability,butisusefultomeasurethepsychologicaldimensionoffoodsecurity(HeadeyandEcker,2013),focusingonthesocio-culturalaspectsoftheexperienceofhunger(Ballardetal.,2013).39Nevertheless,becauseoftheissuesaroundadaptationandcross-culturalunderstandingsofhungerandfoodadequacy,thesetypesofindicatorsmaypresentproblemsintheidentificationofneeds,forexampletotargetprogrammesoffoodrelieforfoodtransfer.
Anotherinterestingcaseisthemeasurementofgovernance.Forestietal.(2014)andBerghetal.(2014)
38 http://www.fao.org/fileadmin/templates/ess/voh/FIES_062014.pdf
39 Thereisalsoanissuearoundcapturinginequality.Whilehungerismoreanissueofaccessthanofavailability,foodavailabilityandcaloricconsumptioncanindicateonlytheupperboundofchronicnutritioninacountry(forexample,thesituationwhereallfoodavailableinthecountrywasequallydistributed,andallindividualshadsimilarcaloricrequirements).
Asking people what they think 23
Box 2: Adaptation, income poverty and subjective wellbeing
Someresearchonsubjectivewellbeinghasfoundapossibleimpactofadaptationintheinterpersonalcomparisonofsubjectivewellbeingresponses.Inparticular,theconcernrelatestopoorerpeoplereportinghighlevelsofsubjectivewellbeing,despitegrimlivingconditions.Contrarytopopularbelief,thereisnowidespreadevidenceofadaptationamongthepoorest(Clark,2012).Forexample:
• Neff(2012)showsthatintwovillagesinAndhraPradesh,themajorityofthepoorarelesssatisfiedwiththeirlivesonaveragecomparedtotheirnon-poorcounterparts,asarelowercastesandMuslimscomparedtohighercastes.Healsofindshighlysignificantstatisticaldifferencesregardingthemeansatisfactionlevelbetweenexpenditurequintilegroups.Thelowestexpenditurequintilegrouphasthelowestmeansatisfaction.
• InBangladesh,Camfieldetal.(2009)findthat,althoughthemajorityofpeoplearesatisfiedwiththeirlives,therearesignificantdifferencesbetweenricherandpoorergroups.Oftherich,31.5%ofpeoplereportbeing‘veryhappy’,comparedto4.7%ofthepoor.
• Mexico’sfirstnationalsubjectivewellbeingsurvey(BIARE),conductedin2013,showsthatindividualsfromthelowestincomedecilearelesssatisfiedwiththeirlivesthanindividualsfromthehighestincomequintiles(Chávez,forthcoming).
• InSouthAfrica,astudyonsocialperceptionsofmaterialnecessities(Wright,2008)foundthatpeoplelackingessentialitemsdefinedbythestudy(suchashavingabathorshowerinthehouse)areawareoftheirshortagesandlistedthemasessentialneeds.Thestudyalsofindsconsensusbetweendifferentgroupsaboutwhattheydefineasessentialneedsandwhatastandardoflivingcomprises.
ThesefindingsinIndia,Bangladesh,MexicoandSouthAfricaindicatethatpeopleinlowestincomegroupsreportlowerlevelsofsubjectivewellbeingandlifesatisfactionthanthoseinhigherincomegroups.Atthecountrylevel,thisalsoseemstohold.TheLegatumCommission(O’Donnelletal.,2014)analysedGWPdataandfindthatmoredevelopedNordiccountriesregularlyleadtheworldintermsoflifeevaluationorsatisfaction,whilelesseconomicallydevelopedcountriessuchasTogo,SierraLeone,andZimbabwe,areconsistentlyatthebottom.
havepointedtothedifferencebetweenindicatorslookingatforms ofgovernanceversusindicatorslookingatways ofgovernance,adiscussionthathasbeenprominentinthepost-2015governancedebates.Forexample,indicatorsmeasuringregimetypeortheexistenceofcertainlegislationorinstitutions(e.g.ananticorruptioncommission)maybeproneto‘isomorphicmimicry’ortheadoptionofformstomeetthetargetswithoutrealchangeinperformance(Forestietal.,2014).Thus,thegapbetweenperceptionsandobjectiveindictorsofgovernancemaysuggestagapbetweenanidealformandthefunctioningofgovernanceinstitutionsandcansignalperceivedcitizendissatisfactionsandgrievances.Berghetal.(2014)argueforexamplethatcitizendissatisfactionwiththelackofperceivedresponsivenessandfunctioningofdemocraticgovernmentsmayhavebeenbehindmanyoftherecentpopularuprisingsaroundtheworld.
Intheareaofgovernance,populationsurveyshavetheadvantageofcapturingtheviewsofthosedirectlyinvolvedintheinstitutionsofthecountry,butcouldbelessaptforcross-countrycomparisonsthanothersourcesofdatasuchasexperts’perceptionsorobjectivedata(Hulmeetal.,2014).However,theseauthorsarguethatperceptionsdataarealsolesssuitablefordistinguishingbetweenthedifferentattributesofgovernanceandthushardertouseforidentifyingrelevantpolicyinterventions.Forexample,itwouldnotbeclearwhatisbehindachangeinreporteddissatisfactionwiththepolice:anincreaseincrime,oranincreaseinpoliceeffectivenessincapturingcriminalsandthusthevisibilityofcrime.ThissuggeststhatperceptionindicatorsofgovernancewouldnotmeetForestietal.’scriterionofshowingaclear‘theoryofchange’,thatis,a‘clearpathwayofchangeunderpinningtheproposed
indicator,specificallyconsideringhowinpracticeitcouldleadtobettermonitoringandmeasurement,andhencegreateractionandresourcemobilisationtoimprovegovernanceoutcomes’(Forestietal.,2014:18).
Tosummarise,inthissectionwehaveshowedthat,whilebothtypesofdataoftensufferfromsimilarmethodologicaldifficultiesrelatedtothedesignandexecutionofsurveys,perceptionsdatahaveadditionalproblemswhichstillrequiremoreresearch.Inparticular,intermsofensuringthereliabilityofthedataandtheindicatorsandensuringcross-countryandcross-groupcomparability,thereisnotyetaclearconsensusonhowtoidentifyandcontrolforpotentialbiases.
Finally,moreresearchisalsoneededtoclarifythesourceofdifferencesbetweenobjectiveandperceptionsindicatorsandwhatthegapsbetweenthetwocantellus.Theymaybothcapturedifferentaspectsofasinglephenomenon(Stiglitzetal.,2009)–forexample,foodsecurityorgovernance–andthusbotharestilluseful,dependingonthepurposesofmeasurement.Inparticular,perceptionindicatorsappearusefulinidentifyinggapsandcriticalpointsforintervention,forexamplebeforemalnutritionbecomesmanifestorwhencitizendissatisfactionsignalsafailureofdemocraticgovernance.However,thesegapshighlightthedifficultiesofelicitingtheappropriatetheoryofchangebehindtheindicators.Determininghowtoreacttotheinformationpresentedbyperceptionsdata,oridentifyingtheexactpolicyinterventionsbehindachangeinperceptions,isahardertask.Thischallengealsooccurswithothertypesofdataandmethods;mixed-methodsapproacheshaveemergedasanattempttocapturethecomplexityofhowpoliciesandprogrammesoperate.40
40 Seehttp://betterevaluation.org/resources/guides/intro_mixed-methods_impact-evaluationandhttp://betterevaluation.org/blog/mixed_methods_part1
24 ODI Report
Perceptions data in monitoring the post-2015 agenda: implementation issues
Howcanperceptionsdatabeincorporatedinthemonitoringofkeyissuesinthepost-2015agenda?Thissectionoutlinessomepracticalissuestoconsiderforimplementation.
Thefirstissuetoconsideriswhether,inthecontextofthepost-2015targets,perceptionsdatawouldbemoreusefulfornationalorinternationalmonitoringandtrackingofthetargets.Whileperceptionsindicatorscouldbepotentiallygoodmarkersofprogressincertainareas,therearelimitstotheircomparability,especiallywhenconsideringresponsesgivenbypeoplelivingincountriesatverydifferentlevelsofdevelopment(e.g.becauseoftheadaptationproblem).Internationalrankingshaveemergedinnumerousareasandarebecominganincreasinglyacceptedpolicytoolintheinternationalarena,butnationalreactionstorankingsarenotyetwidelyunderstood(Cooley,forthcoming).
Rankingsinherentlycompareonecountrywithothers,andthuscanconferstatustocountriesrankedhighlyandstigmatisethoserankedlower(Cooley,forthcoming).WhiletheMDGandtheirSDGsuccessorsdonotestablishexplicitcountryrankings,theydoinvolvealargedegreeofcomparisonofprogressacrosscountries.Moreover,whilethemeasurementandcomparisonofcountriesmayhaveincentivisedcountriestryingtoreachtheMDGtargets(UNDP,2010),ithasbeenshownthatthismeasurementwasunfair,especiallyforthepoorestcountriesforwhichsomegoalswereunattainable(e.g.Easterly,2009).Thelessonhereisthattakingintoaccountcountries’heterogeneityandstartingpositionswasimportant.Rankingorcomparingcountriesbasedontheirperformanceonperceptionindicatorswouldlikelybeproblematicforthepost-2015targets.Moreover,asdiscussedinprevioussections,oneofthegreatadvantagesofperceptionsdataaretheirpotentialtomonitorchangingconditionsandtoactaspointersofchangesintrendsovertime.Becauseofthedifficultyofdiscerningwhatisbehindachangeinaperceptionindicator,triangulationwithothersourcesofdataisnecessaryto
determinetheappropriatepolicyresponse.Thus,wesuggestthatperceptionsdatawouldbemoreusefultomonitorchangingsituationsovertimewithincountries,ratherthantoestablishcomparisonsacrossthem.
Asecond,relatedissueistheproductionofperceptionsdata.Despitetheincreaseinavailabilityofperceptionsdata,fewNSOswereconvinceduntilrecentlyoftheirpotentialtoinformpolicyandthusfewofthemconsidereditworthinvestinginmeasuringsuchindicators.Thus,particularlyindevelopingcountries,surveysofpopulationattitudes,expectationsandsatisfactionarenotyetwellestablished(UNTaskTeam,2013).
Ontheotherhand,privatecompaniesusingperceptionsdataformarketresearchhavelongrealisedtheirpotential.Furthermore,privatepollingcompaniessuchasGallupandIpsosMoriarestillamongtheleadingcollectorsofsubjectivewellbeingdata.ResearchconsortiasuchastheEuropeanValueSurveysandtheWorldValuesSurveysarealsoleadingonthecollectionofperceptionsdata.Thefactthat,currently,perceptionsdataarestronglyassociatedwiththeseinstitutionsposesadditionalquestionsofdatagovernanceandaccountability.Theuseofprivatesourcesasofficialmeasuresfortarget-settingraisesquestionsofdataquality,ownershipandaccountability.
Dataqualitymaynotdependonwhethertheorganisationispublic,privateorresearch-based.Infact,whenprivatecompaniesarebeingcommissionedorworkinpartnershipwithpublicinstitutions,includinginternationalones,tocollectperceptionsdata,theboundariesofdataownershiparehardtodraw.Furthermore,inthecaseofresearchconsortia,thepresenceofacademicswithstrongresearchprofilesmayhelptoimprovethequalityofthedatacollected.However,itmaybemoredifficulttoassessthequalityofthedataiftheyarenotsubjecttoappropriateaccountability.
Forexample,GallupWorldPolldatahavebeenusedininternationalprojectssuchastheWorldBankGlobal
Asking people what they think 25
Findex,41UNDP’sHumanDevelopmentReportsince2010,42theOECD’sBetterLifeindex,43ILO’sSocialUnrestindex44andWorldofWorkReport,2013andtheLegatumProsperityIndex.45
Whilesomedataaremadepublicwhentheyarepartofsuchprojects,themicrodatabehindtheaggregatenumbersarenotreleased,andsometimesnoteventoapartnerinstitution.Conductingfurtheranalysis,verifyingthesamplingproceduresandrepresentativenessofthesurveys,checkingforomissions,recordingmistakesandotherdataqualityissuesisnotpossiblewithoutaccesstothemicrodata.Itisalsonotpossibletodisaggregatethedatabeyondwhatthepartnerinstitutionpresents,forexample,iftheyreleaseresultsatonlythenationallevel.Whilethepartnerinstitutionscouldensuredataqualityandaccountability,thismaynotalwaysbethecase.Beyondthat,thisalsoimpliesachangeinthewaydataaretraditionallyreportedtotrackprogressintheMDG.ItwillnotbenationalgovernmentsreportingtotheUN-systeminstitutions,butsuchorganisationscollectingdatafromprivateorganisationsandthenreportingbacktocountriesontheirprogress.
ExperiencessuchasthatofthePhilippinesSWSdemonstratethatthereisvalueinprivateinstitutionsgeneratingsocialstatisticsforpublicuse(GuerreroandMangahas,461989,citedinGuerrero,2004),especiallyinacontextofarestricteddemocracywheregovernmentsmaytrytoconcealinformation.TheSWSwasbannedin1982bytheauthoritarianrulingregimeforbeing‘toorisky’topublishonlywhentheSWSbecameindependent;theresultswerewidelydisseminatedacrossthegeneralpopulation(Guerrero,2004).Counterexperiencescanalsobefound,especiallyinthecaseofpoliticalormarketingpollscommissionedbyinterestgroupstochangepublicopinion,camouflagecontroversialpolicies,orgather‘proof’tosustaincertaindemandsorconcessionsfromthegovernment(DionneandMann,2003).
AnincreasingnumberofNSOsarestartingtocollectandreleaseperception-baseddata,particularlyonsubjectivewellbeing.Totheextentthatthesedatabecomesuseful,thereiscertainlyconsensusthatSWBmeasuresshouldbeviewedasonesetinthemuchbroaderarrayofindicatorsthroughwhichpopulationsaremonitoredandpoliciesinformed(seeOECD,2013).Thekeyissueseemsto
bewhetherthedataaremadeavailableandcommunicatedwidely,ratherthanwhethertheyarepublicorprivate.Availabilityalsoenablestheimprovementofdataquality,ascheckscanbeconductedbydatausers,andthestrengthsandlimitsofparticularsurveyscanbebetterunderstood.
Further,astheOECDhasdoneforindicatorsofSWB,acommonsetofstandardswillneedtobederivedforotherperception-basedindicators.Whileawiderrangeofperceptionsdataiscollectedongendernorms,violenceandsecurity,andgovernance,differentsurveysoftenaskdifferentquestionsandacompilationandreviewofbestpracticesisnecessary.Thereviewpresentedhere,onindicatorsofgendersocialnorms,violenceandgovernmentaccountabilityshouldserveasastartingpoint.Projectsfocusedonperceptionsdataarealsogrowingandincreasinglylookingtostandardisetheirprocedures.TheWVS,anditsprecursor,theEuropeanValueSurveys,aswellastheregionalbarometernetworkareexampleswherejointeffortshaveincreasinglyledtoacommonsetofprocedures.However,questionsremainaboutthevalueaddedofaglobalperceptionsurveythatisfullycomparableacrosscountries,especiallygiventhat,fortheSDG,perceptionsdatawouldbemostusefulatthenationallevel.
Whilesomeconsensusisstartingtoemergeoncommonstandardsfordataqualityandforcollectingandanalysingperceptionsdata,somedegreeofflexibilityinhowquestionsareaskedisneededtobeabletoaccountforthecountry-specificityofperceptions.Forexample,toensurethattheframingofquestiondoesnotresultinbiasedestimatesforsensitiveissues,thespecifictechniquesandtermsusedforthequestionsmayneedtobeadapteddependingonthecountry,asasimpletranslatingofthequestionnairesmaynotbesufficient.
Finally,itisworthnotingthattheapplicationofstandardsofgood-practiceprocedures,aswellasthedesiredincreaseinthesamplesizetotrytoaddressrepresentativenessandinclusionofmarginalisedgroups,mayalsoincreasethecostofperceptionsurveys.Thismayriskoneoftheirkeysellingpoints–frequencyandcountrycoverage.Althoughadetailedreviewisbeyondthescopeofthispaper,exploringthemixofnewmethodsandnewtechnologiesinthecollectionofperceptionsdatacouldbeawaytobringthosecostsdown.Moreexplorationofthisaspectisrequired.
41 WorldBankarticle:http://bit.ly/OGopsB
42 AlthoughperceptionsdataarenotincludedintheHDI
43 http://www.oecdbetterlifeindex.org/
44 http://www.ilo.org/newyork/voices-at-work/WCMS_217280/lang--en/index.htm
45 http://www.prosperity.com/
46 NotethatMangahasisthefounderoftheSWS.
26 ODI Report
ConclusionsThepost-2015negotiationsandinparticularthecallfora‘datarevolution’havespurreddebatearoundtheinclusionofnewindicatorsofprogress.Inthiscontext,indicatorsbasedonperceptionsdatacouldshiftthemeasurementdebateandhelptofocusonwhatisimportanttoimprovewellbeingofcitizensaroundtheworld,inbothobjectiveandsubjectivedimensions.
Themainadvantageofperceptionsdataisthattheycanpresenttherespondents’ownviewsdirectly.Thisisimportanttostimulatepublicdebatearoundthetrade-offsofcompetingpoliciesandtoholdpolicy-makerstoaccountforwhatmattersmosttopeople.Toselecttargetsandindicatorstomeasurethepost-2015goals,thisprocesscanhelptovoicetocitizen’sperspectivesonwhattheyperceiveastheconstituentelementsofthegoals.Moreover,someperceptionssurveyssuchasMyWorldcanbeusedtoprioritiseimplementationamongtheplethoraofgoalsandtargetsandtocomparethepreferencesofpeopleindifferentcountriesorbelongingtospecificsocialgroups.
Theincreaseintheavailabilityofperceptionsdataandtheresearcharoundsubjectivedimensionsofwellbeingpresentagreatopportunity.Inparticular,comparedtohouseholdsurveysandothersourcesofinformationsuchascensuses,perceptionsurveyshavegreatpotentialtoproducetimelyandfrequentdataformonitoringsocial
conditions,andtospotchangesintrendsthatcouldendangertheachievementofthepost-2015targets.
WehaveidentifiedthreeareaswhereperceptionsdatahavefeaturedstronglyinproposalsformonitoringtheSDG:genderandsocialnorms,violenceandsecurity,andgovernance.Ineachcase,perceptionsdatacouldprovidecomplementaryinformationinconjunctionwithotherdata.Whileperceptionsurveyscouldbedifficulttouseontheirowntomeasuretheachievementofatargetinanyofthesethreeareas,differentconsiderationsneedtobeincorporatedinthedecision-makingprocesswithperceptionsinformationasaconsistentinput.Moreover,adisaggregatedanalysisofperceptionsmayhelptomakeexplicitthevarietyofpreferencesinagivensociety,whichneedstobeunderstoodinthecontextofthepost-2015targetstoaddressinequalitiesinorderto‘leavenoonebehind’.
However,perceptionsdataarelessusefulindeterminingtheappropriatepolicychangerequiredwhensuchanalarmisraised.Themainlimitationsare:1)representativenessatthesub-nationallevel,particularlygivenlowsamplesizes;2)reliabilityoftheanswers;and3)theissuesaroundadaptationandhowtoseparatethisfromheterogeneityinpointsofview.Thesechallengesarestillunresolved,andmaylimitthecomparabilityofanswers,especiallyofthoselivinginextremedeprivationorwithrestrictedfreedomsandrights.
Asking people what they think 27
References Alkire,S.andSamman,E.(2014)Mobilizing the household data required to progress toward the SDGs.Sustainable
DevelopmentSolutionsNetwork.(http://bit.ly/1DpxLQq)Alkire,S.,Malapit,H.,Meinzen-Dick,R.,Peterman,A.,Quisumbing,A.,Seymour,G.andA.Vaz.(2013)‘Instructional
guideontheWomen’sEmpowermentinAgricultureIndex’.Arnesen,T.andTrommald,M.(2005)‘AreQALYsbasedontimetrade-offcomparable?AsystematicreviewofTTO
methodologies’,Health Economics14(1):39-53.Ballard,T.,Kepple,A.andCafiero,C.(2013)TheFoodInsecurityExperienceScale.Developmentofaglobalstandardfor
monitoringhungerworldwide.Rome:FAO.Bardasi,E.,Beegle,K.,Dillon,A.andSerneels,P.(2011)‘Dolaborstatisticsdependonhowandtowhomthequestions
areasked?ResultsfromasurveyexperimentinTanzania’,World Bank Economic Review25(3):418-447.Barker,C.,Pistrang,N.andElliott,R.(2002)‘Self-reportedmethods’,inResearch methods in clinical psychology. An
introduction for students and practitioners(2ndedn).Wiley.Bartels,L.M.(2003)‘Is“popularrule”possible?Polls,politicalpsychology,anddemocracy’,The Brookings Review,Summer.Bellani,L.,Hunter,G.andAnand,P.(2013)Multidimensionalwelfare:dogroupsvaryintheirprioritiesandbehaviours?
Fiscal Studies34(3):333-354.Bergh,G.,RochaMenocal,A.andRodriguezTakeuchi,L.(2014)What’s behind the demand for governance? An
assessment of people’s views.London:ODI.Bhatkal,T.(2014)Whatdowomenwant?Gender,perceptionsdataanddevelopmentpriorities.London:ODI.Blanchflower,D.andOswald,A.(2011),‘Internationalhappiness’.NBERWorkingPaper16668.NationalBureauof
EconomicResearch.Boarini,R.,Comola,M.,Smith,C.,Manchin,R.andDeKeulenaer,F.(2012)Whatmakesforabetterlife?The
determinantsofsubjectivewell-beinginOECDcountries:evidencefromtheGallupWorldPoll.Paris:OECD.Burchardt,T.(2005)‘Areoneman’sragsanotherman’sriches?Identifyingadaptiveexpectationsusingpaneldata’,Social
Indicators Research74:57-102.Camfield,L.,Choudhury,K.andDevine,J.(2009)‘Well-being,happinessandwhyrelationshipsmatter’,Journal of
Happiness Studies10:71-91.deCao,E.andLutz,C.(2014)Sensitive survey questions: Measuring attitudes regarding female circumcision through a
list experiment (http://bit.ly/145E2S5).Carr-Hill,R.(2013)‘Missingmillionsandmeasuringdevelopmentprogress’, World Development46:30-44.Chandy,L.(2013)‘Countingthepoor.Methods,problemsandsolutionsbehindthe$1.25adayglobalpoverty
estimates’. InvestmentstoEndPovertyWorkingPaper. DevelopmentInitiatives.Chávez,C.(forthcoming)‘YouthDelinquencyinMexico:Anexplorationofcontributingfactors’.PhDdiss.,UniversityofCambridge.Clark,D.(2012)‘Adaptationanddevelopment–issues,evidenceandpolicyrelevance’,inD.Clark(ed.),Adaptation,
poverty and development: the dynamics of subjective well-being.London:PalgraveMacmillan.Clemens,M.,Kenny,C.andMoss,T.(2007)‘ThetroublewiththeMDG:confrontingexpectationsofaidand
developmentsuccess’,World Development35(5):735-751.Cooley,A.(forthcoming)‘Theemergingpoliticsofinternationalrankingsandratings:aframeworkforanalysis’,inA.Cooley
andJ.Syder(eds),Ranking the world: the politics of international rankings and ratings.CambridgeUniversityPress.Coulthard,S.Camfield,L.andMcGregor,A.(2014)‘Valuingdevelopmentandsocialprogress:practicalapproachesto
measurewhatmattersindevelopment’.London:ODI(mimeo).Cummings,R.(2009)‘Measuringpopulationhappinesstoinformpublicpolicy’.The3rdOECDWorldForumon‘Statistics,
KnowledgeandPolicy’ChartingProgress,BuildingVisions,ImprovingLife,Busan,Korea,27–30October2009.Decancq,K.,Fleurbaey,M.andSchokkaert,E.(2009)‘Whatgoodishappiness?’Ecorediscussionpaper(http://www.
ecore.be/DPs/dp_1238577455.pdf).Dionne,E.J.andMann,T.E.(2003)‘Polling&publicopinion:thegood,thebad,andtheugly‘,The Brookings Review.O’Donnell,G.,Deaton,A.,Durand,M.,Halpern,D.andLayard,R.(2014)Report of the Commission on Wellbeing and
Policy.London:LegatumInstitute.Easterlin,R.(1974)‘Doeseconomicgrowthimprovethehumanlot?Someempiricalevidence’,inPaulA.DavidandMelvinW.
Reder(eds),Nations and households in economic growth: essays in honor of Moses Abramovitz.NewYork:AcademicPress.Easterly,W.(2009)‘HowtheMillenniumDevelopmentGoalsareunfairtoAfrica’,World Development37(1):26-35.Fafchamps,M.andKebede,B.(2008)‘Subjective wellbeing, disability and adaptation: A case study from rural Ethiopia’.
WorkingPaper283.Oxford:TheCentrefortheStudyofAfricanEconomies(http://www.bepress.com/csae/paper283).
28 ODI Report
Foresti,M.Wild,L.withNorton,A.andRodriguezTakeuchi,L.(2014)Governance targets and indicators for post 2015. An initial assessment.London:ODI.
Gallup(2012)Globalstatesofmind.Newmetricsforworldleaders.WashingtonD.C.:Gallup.Garrett,J.andAhmed,A.(2004)‘Incorporatingcrimeinhouseholdsurveys:aresearchnote’, Environment and
Urbanization16:139-151.Gilbert,D.(2004)‘Thesurprisingscienceofhappiness’. TEDTalks(http://www.ted.com/talks/dan_gilbert_asks_why_
are_we_happy.html).Gough,I.andMcGregor,J.A.(2009)‘After2015:“3Dhumanwellbeing”.’IDSinFocus,PolicyBriefIssue09(http://
www.ids.ac.uk/files/dmfile/IF9.2.pdf).Groves,R.M.etal.(2009)Survey methodology (2ndedn).NewJersey:Wiley.Guerrero,L.L.B.(2004)Social weather stations: Asia’s oldest barometer.WSWOccasionalPaper(June).Harper,C.,Nowacka,K.,Alder,H.andFerrant,G.(2014)Measuringwomen’sempowermentandsocialtransformation
inthepost-2015agenda.London:ODI.Harper,C.,Jones,N.andWatson,C.(2012)Genderjusticeforadolescentgirls:tacklingdiscriminatorysocialnorms.
Towardsaconceptualframework.London:ODI.Headey,D.andEcker,O.(2013)‘Rethinkingthemeasurementoffoodsecurity:fromfirstprinciplestobestpractice’,
FoodSecurity5(3):327-343.Herbert,S.(2013)Perceptionsurveysinfragileandconflict-affectedstates(GSDRCHelpdeskResearchReport910).
Birmingham,UK:GSDRC,UniversityofBirmingham(http://www.gsdrc.org/docs/open/HDQ910.pdf).HLP(HighLevelPanelofEminentPersonsonthePost-2015DevelopmentAgenda)(2013)Anewglobalpartnership:
eradicatepovertyandtransformeconomiesthroughsustainabledevelopment.NewYork:UN.Hulme,D.,Savoia,A.andSen,K.(2014)Governanceasaglobaldevelopmentgoal?Setting,measuringandmonitoringthe
Post-2015DevelopmentAgenda.ESIDWorkingPaper32.Manchester:EffectiveStatesandInclusiveDevelopment(ESID).Hussein,E.(2010)Women’sexperiences,perceptionsandattitudesoffemalegenitalmutilation.TheBristolPEERStudy.
FoundationforWomen’sHealthResearchandDevelopment(FORWARD).IEAG(IndependentExpertAdvisoryGrouponaDataRevolutionforSustainableDevelopment)(2014)Aworldthat
counts(http://bit.ly/1wApBjP).IpsosMori(2012)PublicperceptionsoftheNHSandsocialcare:anongoingtrackingstudyconductedforthe
DepartmentofHealth,December2011Wave.London:IpsosMori.Ivanyna,M.andShah,A.(2009)Citizen-centricgovernanceindicators:measuringandmonitoringgovernanceby
listeningtothepeopleandnottheinterestgroups.EconomicsDiscussionPaper2009-27,KielInstitutefortheWorldEconomy(http://bit.ly/1wwvGLh).
Jerven,M.(2103)Poornumbers:howwearemisledbyAfricandevelopmentstatisticsandwhattodoaboutit.IthacaandLondon:CornellUniversityPress.
Kahneman,D.andKrueger,A.B.(2006)‘Developmentsinthemeasurementofsubjectivewell-being‘,JournalofEconomicPerspectives20(1):3-24.
Kaufmann,D.,Kraay,A.andMastruzzi,M.(2009)GovernancemattersVIIIaggregateandindividualgovernanceindicators1996-2008.PolicyResearchWorkingPaper4978.WashingtonDC:TheWorldBank.
Kenny,C.andDykstra,S.(2013)TheGlobalPartnershipforDevelopment:areviewofMDG8andproposalsforthepost-2015developmentagenda.WashingtonDC:CDG.
Kroll,C.(2013)Globaldevelopmentandhappiness:howcandataonsubjectivewellbeinginformdevelopmenttheoryandpractice?IDSworkingpaper432.Brighton:InstituteofDevelopmentStudies.
Krosnik,J.A.andPresser,S.(2009)‘Questionandquestionnairedesign’,inJ.D.WrightandP.V.Marsden(eds),Handbookofsurveyresearch(2ndedn).SanDiego,CA:Elsevier.
Leo,B.(2013)Isanyonelistening?DoesUSforeignassistancetargetpeople’stoppriorities?CDGWorkingPaper348(http://bit.ly/19KhMtx).
Lucci,P.andBhatkal,T.(2014)Monitoringprogressonurbanpoverty:arecurrentdataandindicatorsfitforpurpose?London:ODI.Mallett,R.(2012)Howfardoperceptionsurveystakeusinfragileandconflict-affectedsituations?TheAsiaFoundation
(http://bit.ly/1tQGAA0),accessed28November2014.Mangahas,M.(2013)Thevalueofstatisticalcompetition.12thNationalConventiononStatistics,MandaluyongCity,
October1-2,2013.McGee,R.andGaventa,J.(2011)Synthesisreport:reviewofimpactandeffectivenessoftransparencyandaccountability
initiatives.London:TransparencyandAccountabilityInitiative.Melamed,C.(2012)Doesdevelopmentgivepoorpeoplewhattheywant?London:ODI.Møller,V.(2005)‘Resilientorresigned?CriminalvictimisationandqualityoflifeinSouthAfrica’,SocialIndicators
Research72(3):263-317.KluwerAcademicPublishers.
Asking people what they think 29
Neff,D.(2012)‘Adaptation,povertyandsubjectivewell-being:evidencefromSouthIndia’,inD.Clark(ed.),Adaptation,povertyanddevelopment:thedynamicsofsubjectivewell-being.London:PalgraveMacmillan.
OECD(2013)OECDguidelinesonmeasuringsubjectivewell-being.Paris:OECD.Oswald,A.J.andPowdthavee,N.(2008)‘Doeshappinessadapt?Alongitudinalstudyofdisabilitywithimplicationsfor
economistsandjudges’,JournalofPublicEconomics92(5-6):1061-1077.Prizzon,A.(2014)Whatdocountrieswantfromaid?Somereflectionsfromthefield.ValuingProgressblogseries(http://
bit.ly/1HUa1nj).Riphahn,R.T.andSerfling,O.(2002)Itemnon-responseonincomeandwealthquestions.IZAdiscussionpaperN576.Bonn:IZA.RodriguezTakeuchi,L.(2014)Incorporatingpeople’svaluesindevelopment:weightingalternatives.London:ODI.Saferworld(2013)Addressingconflictandviolencefrom2015:avisionofgoals,targetsandindicators.February2013
Briefing.London:Saferworld.Samman,E.andRodriguezTakeuchi,L.(2014)Oldage,disabilityandmentalhealth:dataissuesforapost-2015
framework.London:ODI.SciDev(2014)‘Frustration’overoldormissingdatainMDGreport,8July2014(http://bit.ly/1xISWtj).Seaford,C.(2013)‘Themultipleusesofsubjectivewellbeingindicators’,SocialIndicatorsResearch114:29-43.Sen,A.(1999)Developmentasfreedom.Oxford:OxfordUniversityPress.Stiglitz,J.E.,Sen,A.andFitoussi,J.-P.(2009)ReportbytheCommissionontheMeasurementofEconomicPerformance
andSocialProgress(http://www.stiglitz-sen-fitoussi.fr/documents/rapport_anglais.pdf).Thomas,M.A.(2009)‘WhatdotheWorldwideGovernanceIndicatorsmeasure?’, EuropeanJournalofDevelopment
Research,16July.Tortora,R.D.(2009)Sub-SaharanAfricansranktheMillenniumDevelopmentGoals(MDG).WashingtonDC:Gallup.Tourangeau,R.andYan,T.(2007)‘Sensitivequestionsinsurveyss’,PsychologicalBulletin133(5):859-883.Tourangeau,R.,Groves,R.andRedline,C.(2010)‘Sensitivetopicsandreluctantrespondents.Demonstratingalink
betweennonresponsebiasandmeasurementerror’,PublicOpinionQuarterly74(3):413-432.Traugott,M.W.(2003)‘Canwetrustthepolls?Italldepends’,TheBrookingsReview,Summer.UN(2014a)IntroductiontotheProposaloftheOpenWorkingGroupforSustainableDevelopmentGoals.NewYork:UN.UN(2014b)Theroadtodignityby2030:endingpoverty,transformingalllivesandprotectingtheplanet.Synthesis
ReportoftheSecretary-GeneralonthePost-2015Agenda.NewYork:UN.UNDP(2014)Buildingtransparency,accountabilityandanti-corruptionintothepost-2015developmentframework.
UNDPDiscussionPaper,September2014.NewYork:UN.UNDP(2010)Beyondthemidpoint:achievingtheMillenniumDevelopmentGoals.NewYork:UNDP.UNTaskTeam(onthePost-2015DevelopmentAgenda)(2013)Statisticsandindicatorsforthepost-2015development
agenda.NewYork:UN(http://bit.ly/12xWbHF).UNWomen(2013)Atransformativestand-alonegoalonachievinggenderequality,women’srightsandwomen’sempowerment:imperativesandkeycomponents.NewYork:UNWomen.
Veenhoven,R.(1988)‘Theutilityofhappiness?’,SocialIndicatorsResearch20:333-354.Veenhoven,R.(2004)Subjectivemeasuresofwell-being.UNU-WIDERDiscussionPaper2004/07(http://bit.ly/1KoTkmh).WorldBankandSecondMuse(2014)Bigdatainactionfordevelopment(http://bit.ly/1DEafwG).Wright,E.(2008)FindingsfromtheIndicatorsofPovertyandSocialExclusionproject:aprofileofpovertyusingthe
sociallyperceivednecessitiesapproach(http://bit.ly/1AKnmMY).Yirga,W.etal.(2012)‘Femalegenitalmutilation:prevalence,perceptionsandeffectonwomen’shealthinKersadistrict
ofEthiopia’,InternationalJournalofWomen’sHealth4:45-54.
This material has been funded by UK aid from the UK Government, however the views expressed do not necessarily re�ect the UK Government’s of�cial policies.
ODI is the UK’s leading independent think tank on international development and humanitarian issues. Readers are encouraged to reproduce material for their own publications, as long as they are not being sold commercially. As copyright holder, ODI requests due acknowledgement and a copy of the publication. For online use, we ask readers to link to the original resource on the ODI website. The views presented in this paper are those of the author(s) and do not necessarily represent the views of ODI or our partners.© Overseas Development Institute 2015. This work is licensed under a Creative Commons Attribution-NonCommercial Licence (CC BY-NC 3.0). ISSN: 2052-7209
Overseas Development Institute 203 Blackfriars Road London SE1 8NJTel +44 (0)20 7922 0300 Fax +44 (0)20 7922 [email protected]
Top Related