ESAR PMR Evaluation Synthesis Report FINAL 29 JUL 17 · PDF fileThe PMR Initiative’s...

InternationalSolutionsGroup www.theisg.com

SYNTHESISREPORTEvaluationofESARInstitutional

StrengtheningSupportInitiativeonDecentralizedProgrammeMonitoringand

Response

July2017PREPAREDBY:

InternationalSolutionsGroup

PREPAREDFOR:

UNICEF

ContactUs

StephenLadek,[email protected]

EvaluationofUNICEFESARPMRinitiative:SynthesisReport

InternationalSolutionsGroup www.theisg.com 2

TableofContentsTableofContents.................................................................................................................................................................................2AbbreviationsandInitialisms.......................................................................................................................................................3ExecutiveSummary...........................................................................................................................................................................4Introduction.......................................................................................................................................................................................10TheEvaluation..................................................................................................................................................................................12Limitations..........................................................................................................................................................................................17Howtousethisreport...................................................................................................................................................................17Findings................................................................................................................................................................................................18LessonsLearned...............................................................................................................................................................................34Conclusions........................................................................................................................................................................................35Recommendations...........................................................................................................................................................................37Annexes................................................................................................................................................................................................42



AbbreviationsandInitialismsBNA BottleNeckAnalysisCA CausalAnalysisiCHIS IntegratedCommunityHealthInformationSystemeCMIS ElectronicClientManagementInformationSystemCO CountryOfficeDFID UnitedKingdomDevelopmentAgencyDFNC DistrictFoodandNutritionCommitteeESAR EastandSouthernAfricaRegionESARO EastandSouthernAfricaRegionalOfficeFNC FoodandNutritionCommitteeHMIS HealthManagementInformationSystemHQ HeadquartersICT InformationandCommunicationTechnologiesISG InternationalSolutionsGroupIT InformationTechnologyKE KnowledgeExchangeMOH MinistryofHealthMSC MostSignificantChangeMSCBARS MultisectoralCommunity–basedApproachforReductionofStuntingNRT NearRealTimeNRTM NearRealTimeMonitoringNRTMS NearRealTimeMonitoringSystemODK OpenDataKitPMR ProgrammeMonitoringandResponseRHMT RegionalHealthManagementTeamRMNCAH Reproductive,maternal,newborn,childandadolescenthealthSMS ShortMessageServiceTOR TermsofReferenceUN UnitedNationsUNEG UnitedNationsEvaluationGroupUNICEF UnitedNationsChildren’sFundVFNC VillageFoodandNutritionCommitteeWASH WaterandSanitation



ExecutiveSummaryIn 2014, UNICEF headquarters, in collaboration with its East and Southern Africa Regional Office(ESARO), took the decision to address certain issues related to the health ofwomen and childrenbyimplementing a regional institutional strengthening support initiative to improve the ProgrammeMonitoring andResponse (PMR) capacity of the centralized government decision-making bodies andcommunitiesinfourcountries(Kenya,Swaziland,UgandaandZimbabwe).TheESARPMRwasdesignedasalearninginitiativewiththefollowinggoals:

• Improveserviceprovisionandprogramperformanceduringimplementationthroughimprovedmonitoring of intermediate results, establishment of citizen feedback loops, and the use ofmonitoringandfeedbackdatatoadjuststrategiesasneededinatimelymanner;and

• Identifyandpromotethosepromisingpracticesofprogrammemonitoringinresponse,testedandproven by early adopter districts, through the development of national scale-up planswithinparticipatingcountriesandtheestablishmentofknowledgeexchangeplatforms.

Four countrieswere selected for participation in the PMR Initiative (Malawi, Swaziland, Uganda andZimbabwe).Astheprogrammebegan,MalawielectedtodropoutbecauseofanationalfloodsemergencyandwasreplacedbyKenya.WhilethePMRInitiativewasdesignedwithanumbrellaresultsframeworkfor theregion,each implementingcountrycustomizedand localized theactual implementationof theInitiative.Theevaluationwascarriedoutbetween27MAR17and31MAY17.Fieldworkwascompletedbetween15April17and10May17.Thisreportisasynthesisoffindings,conclusionsandrecommendationsinthefourcountryreportsthataccompanyit.Assuch,ingeneral,forgreaterdetailaboutcountry-specificfindings,pleaserefertothecountryreports.

Findings

ThePMRInitiativehasitsgenesisinanunderstandingofthecontinuedevolutionofICT4DinnovationsinESARandhowtheseinnovationshavecontributedtobuildingacultureofmanagementusingdatainthe health sector. Because of this, the PMR Initiative has been a response that can be localized andcontextualizedtospecificcountryneeds,priorities,andcapacity.ThePMRInitiativehasestablishedorenhancedplatformsthatproducereliable,qualityandmorerealtimedatainallfourcountries.However,theemphasisof the initiativehasbeenondatause.Theevaluationcannotdetermine ifUNICEFhasacomparativeadvantage in theproductionof technology-basedsolutions.However,UNICEFappears tocurrentlyenjoyasignificantcompetitiveadvantage,butonethatneedstobecarefullyarticulatedanddefended.The PMR Initiative has not resulted in a process for incorporating citizen feedback that rivals oreliminates traditional, or established, community and government structures. The evaluation cannotdetermineifUNICEFhasacomparativeadvantageincommunityengagement.However,UNICEF’scitizenengagementtoolsofferanopportunityforcompetitiveadvantage,butonethatneedstobeestablishedandvalidated.AsignificantachievementofthePMRInitiativeisitscatalyticcontributiontoa“managementusingdata”cultureinthefourimplementationcountries.Byemphasizingdatause,evenwithincompletedatasets,thePMRInitiativehascontributedtoavirtuouscycleofincreaseddemandformoredataandadesireforimproveddataquality.UNICEFactivelypursuedtheexchangeofknowledgegeneratedbythePMRInitiativethroughavarietyofplatforms andopportunities,withmixed results.At the country level,UNICEF staff andpartners relymainlyontraditionalorestablishedmethodsofknowledgesharing.



The PMR Initiative has increased the potential for long term sustainability through theinstitutionalizationofprocessesandprotocols,suchastheS/B/Aprocess,butthisincreasedpotentialismixedacrossthefourimplementingcountries.Thepotentialforbothsustainabilityandscalabilityareincreased when UNICEF’s support focuses on enhancing existing national platforms and, to a lesserdegree, developing online platforms. The PMR Initiative has seen the beginnings of a critical shift inorganizationalandprofessionalcultureintheimplementingcountries.Thisshiftenhancessustainabilitybycreatingtheprofessionalexpectationofusingdata formanagementplanninganddecisionmaking.The sustainability and scalability of gains realized by the PMR Initiative could be further enhancedthroughgreatergovernmentpartnercontributionandbroadeningthedirectionalityofscale.ThePMRInitiativehascreatedinformationproducts(i.e.dashboards)andprocesses(i.e.S/B/Aprocess)outputsthatcanbereplicatedinothercontextwithsimilarfoundations.TheevaluationcannotdetermineifUNICEFhas a comparative advantage in replicating technology-based solutions.However,UNICEF’sprogrammingacrossmanysectorssuggestsacompetitiveadvantageexiststhatcouldbeexploited.

Conclusions

Theconclusionsbelowflowdirectlyfromthefindingsabove:WhiletheavailabilityofmorerealtimedatahasplayedakeyroleinthePMRInitiative,amorecriticaloutcomeistheInitiative’scontributiontothedevelopmentofa“managementusingdata”cultureinESAR.UNICEF’s competitive advantage lies in its strategic and long-term relationships with governmentpartners and its ability tomanage IT enabled programming. UNICEF comparative advantages, if any,relatedtoimplementedtechnologyenabledprogrammingremainunknown.ThePMRInitiative’sattemptstoengagecitizenfeedbackhaveproducedpositiveresults,buthavenotyetinspiredanewnormforhowcitizenfeedbackisgathered,utilizedandmanaged.ThePMRInitiativehasrealizedtypicalresultsintermsofknowledgeexchangeandcontributedtothedevelopmentofnewtoolsandchannelsforthesame.However,thereisstillsomedistancetogotoachieveatrueparadigmshiftwithinUNICEFforKnowledgeExchange.Thescalabilityandsustainabilityofprogramminggainsremainsahigh-qualitygoal.ThePMRInitiativeoffers three unique paths that can be used as models for both large scale releases and continuedrefinementofthistopicwithinUNICEF.ThePMRInitiativehashighlightedcriticalopportunitiesforinnovationatthebeginningandendofthefeedbackloopinhealthsystemstrengthening.

Recommendations

Therecommendationsbelowflowdirectlyfromtheconclusionsaboveandareprioritizedaspresented:

Recommendation1:Continuetoinvestinthe“ManagementUsingData”cultureinESARbyleveraging

UNICEF’suniquerelationshipwithgovernmentpartnersforlong-terminvestmentplansandreplicating

gainswithinhealthcareandinothersectors.

Undoubtedly,theclearestgainsinallfourcountriesmadebythePMRInitiativearefoundinmanagersusingcollecteddatatounderstandhowthey,andtheirfacilities,areperforming.UNICEF,anditspartners,should continue to support the building of a robustmanagement culturewhere individuals not onlydemand data for their management decisions, but also one where trust in data is the norm andunderstandingofdataisexpectedatalllevels.



Potentialdirectionsthatcouldbetakeninsubsequentprogramminginclude:• Build capacity around short, medium and long-term planning and strategy. For managers to

continuetobeexcitedaboutdata,tangibleresultsfromusingdatawillneedtohappenintheshortterm.To thatend,UNICEFshould seek toworkwithmanagers todevelopshort,mediumandlonger-termplanning and strategies that allowmanagers to have quickwins that roll up intogreatergainsoverthelongterm.

• Seektoincentivizeindividualsorteamsarounddatauseandanalysis.SimilartothescorecardapproachforRMNCAH,theremaybeotheropportunitiesforacompetitivesystemthatcomparesfacilityordistrictperformancetocontinuefeedintoemotionalattachmenttodata.

• Providesimpleandtransparentmeansformanagerstoidentifyandreportondataqualityissues.Makesurethissystemalsoincludesclearownerswithauthoritytotakeaction,accountability,andafeedbackmechanismthatmakesiteasytoseewhatactionshavebeentakentoremedyissues.

• Establishaprofessionalenvironmentwherefailureistoleratedandexpected,aslongaslearningandadaptationalsohappens–noteverydecisionbasedondatawillresultinpositiveoutcomes,oranytangibleoutcomes.

• Encourage customization and creative solutions frommanagers based on existing and futureautomatedanalytical tools.As theseuniquesolutionsbegin toemerge, facilitateaplatformormethodforsharinginthecountry,regionallyandglobally.

• Seek ways to make digital analytical tools available to all levels of the health care system,especiallynon-technicalmanagers.

• Provideopportunitiesformembersofthehealthsectortodemonstrategainsandoutcomestoothersectorsandtootherareasofthehealthcaresystem.

Recommendation2:BolsterUNICEF’scompetitiveadvantageasamanagerofenablingICTsolutions

byinvestinginternallyinskilledinformationmanagers.

UNICEFhasestablisheditselfasaleaderinmanagingprogramsthatrelyuponICTfunctionalitytoenabletherealizationofexpectedoutcomesforchildren.However,thisisnotauniqueadvantageandonethatiscovetedbymanyotherorganizationsacrossthesocialsector.UNICEFshouldcontinuetodifferentiateitself by investing in individuals who are skilled informationmanagers. These individuals should betaskedwithanswering thequestion, “Howcould [instertprogrammename]beenhanced,mademoreefficient,scaleorbemoresustainablethroughbetterdataandsubsequentanalysis?”Whileanunderstandingoftechnicalareas(e.g.Education,Protection,Health,WASH)areimportant,theexistingskillsetsoftheseindividualsshouldfirstemphasize:

• Apassionaboutdata,analyticsand theirultimateusefulness inachievingbetteroutcomes forchildren;

• An intrinsic understanding of ICTs and the willingness to play, bend, break and fix ICTs tounderstandtheirusefulness;

• A clear understanding of information management, including storage, accessibility,interoperability,systemsintegrationandmetadata;

• Atleastanappreciationfordatascienceandhowdatacanbepooled,dissectedandusedinuniqueways;

• Aprovenabilitytothinkacrosssectorsandsilos;• Aproventrackrecordinchartinglong-termvisionsassociatedwithclear,shorttermmilestones

thatproducetangiblebenefit;• Apersonalitythatiswillingandabletoconnectandpartnerwiththeprivatesector;• A personality who is able to forge new relationships, managing existing ones and build

communityarounddatasharing;and• Anindividualthat“gets”theopensourcemovement.

Importantly,whileICTsareoftenapartofthedata-enablingconversation,therealkeyhereisinformationmanagementandtheuseofinformationformanagementdecisionmaking.CreatingorincludinganICT



componentisnotalwaysnecessaryandshouldbeapproachedwiththesamecareasanyotheraspectduringprogrammedesign.

Recommendation3:InvestinestablishingUNICEF’scomparativeadvantagetomakethecasethatICT

investmentsarethebestuseofdonorfundsforachievingbetterhealthoutcomesforchildren.

Asnotedinthisreportandthecountryreports,theevaluationteamwasunabletodetermineifUNICEFhasacomparativeadvantagerelated toenabling ICTsolutions, improving the incorporationofcitizenfeedbackandscalingorreplicatingprogramming.Whilethis isnotaneasyquestiontoanswerbyanyestimation,answeringitismostlyafunctionofavailabletimeanddata.UNICEFshouldinvestinansweringthecomparativeadvantagequestionfortwoimportantreasons:first,understanding the optimalway to spenddonor funds to achieve better health outcomes for childrendirectlysupportsUNICEF’smandate.Second, if it ispossible toestablish that investing$1USD in ICTenabled programming delivered by UNICEF results in better, more or broader health outcomes forchildren than other alternatives, knowing this would be a powerful competitive advantage for theorganization.

Recommendation4:Continuetoinvestincollectingandrespondingtocitizenvoicebyiteratingcurrent

solutionsandseekingnewsolutionsfromalargercommunityofpractice.

UNICEFhasbeenaleadingadvocateandpractitionerforincludingcitizenvoiceinprogrammedesignanddeliveryformanyyears.However,theattemptstoincludecitizenvoiceviaICTenabledsolutionsinthePMRinitiativegenerallyfellshortofexpectations,orinthecaseofKenya,wasnotused.Citizen feedback isessential forsuccess indevelopmentprogramming.To thatend,asnotedabove inRecommendation1,resultsfromthePMRInitiativeshouldbeseenasamomentofreflectionforUNICEFinordertolearnanditerate,sothatthenextprogrammingcyclemightattemptsomethingnew.Inthespiritofworkingtogethertocreategreaterefficienciescollectingandrespondingtocitizenvoice,UNICEFshouldshyawayfromgoingitalone.OneimmediateopportunitywouldbetoengageFeedbackLabs,1inoneoftheirLabStorms,andusethiscommunityofmorethan300organizationsasacatalystforcrackingthecodetogatheringandengagingwithfeedbackfromruralcommunities.Asanaside,atthetimeofthisevaluation, thecitizenfeedbackcomponent inZimbabwewasonly justbeginningtogaintraction.UNICEFshouldmonitorthisprocess(basedonUReport)asitisdifferentthanthoseusedintheotherthreecountries.Theremaybeadditionalsuccessesandchallengesthatcanbelearned.

Recommendation 5: Continue to evolve knowledge exchange at UNICEF through changes in

organizationalstructure,professionalcultureandleadershipbuyin.

TheevaluationfoundthatthePMRInitiativefellshortintermsoffindingnew,uniqueorscalablemeansforsharingknowledge.However,gainsweremadeand,similartocitizenvoiceabove,thisshouldbeseenasanopportunityforreflection,iterationandnewdesign.ThreeareasUNICEFcanconcentrateonintheneartermtoimproveitsKnowledgeExchangepracticeare:Organizationalstructure:UNICEF’sdecentralizedorganizationalstructurecreatesnaturalboundariesbetweenCOs,withinregionsandglobally.ThesephysicalboundariesarefurthercompoundedthroughthedivisionofUNICEF’sworkintoseparatesectors.Whileitisbeyondthescopeofthisevaluation,itisclearthatthisstructureprovideslittletonoincentiveforUNICEFstafforteamstoregularlyandreliablyshareinformation.Whenfundingstreamsareaddedtothisequation,incentivesarereducedfurther.UNICEFshouldseektofindincentiveoptionstobreakthesebarriers.

1Seehttp://www.feedbacklabs.org



Professionalculture:UNICEF’sprofessionalculturedoesnotcurrentlyreliablyrewardrisktaking.UNICEFshouldpursuemessagingandaclearincentivestructurethatrewardsrisktakingwithregardtoinformationsharinginordertoestablishanewnormregardingexpectationsaboutwhathappenswiththelessonedlearnedineachCO.Leadership:Underlyingbothoftheaboveistheneedforclear,strong,consistentandsustainedleadershiponknowledgeexchange.IftheorganizationanditsstaffdonotperceivethatKEisahighpriority,withincentivesforgoodperformanceandlimitationsforpoorperformance,changewillnothappen.Thisleadershippositionshouldcomefromthehighestlevelpossibleandbetiedtooneortwo,simple,cleargoals.Communicationaboutleadershipexpectationsshouldbecommunicatedateveryopportunityandoverasustainedperiod(2-3years)inorderforthemessagetobecomeapartoftheUNICEFculture.

Recommendation 6: Seek to shift UNICEF staff and partner mindsets about transferability of ICT

enabledprogramming,andknowledgeexchange,from“plugandplay”to“customization.”

ThereareclearopportunitiesforgainsmadeinthePMRInitiativetobereplicatedinothercontext,inboththehealthsectorandothersectors.UNICEFshouldensurethatthefollowingtwocriteriaarepresentinanynewcontextforreplicationtobesuccessful.First,ensureanyreceivingcontexthasaminimumenablingenvironment.Thisenvironmentincludes:

1. Nationalownershipandleadershipofthesystem(s)thatwillreceivetheintervention;2. Nationalsystemsthatareownedandoperatedbythegovernmentthatcanbeenhanced;3. Theabilitytopartnerwithgovernmentandcivilsociety;4. Theabilitytofosterdevelopmentandimprovementofinfrastructureandpolicy;5. Theabilitytoalignprogrammaticgoalswithaccountabilityframeworks,incentivesandactions;6. Expertisethatcanbesystematicaboutadoptingandscalinginnovationsthatareeffective;7. TheabilitytoinvestinadequateICTcapacity;8. Along-termvisionthatallowssufficienttimeandinvestmentforactionandimpact;and9. Theabilitytofocusonshorttermmilestonesalongalonger-termroadmap.

Second,ensurethatUNICEFstaffandpartnersunderstandthatreplicationofanyICTenabledprocessorsystem will, by its nature, require customization. As noted in the Uganda country report, there is atendency for non-technical UNICEF staff and partners to desire systems and processes that aretransferrable as “plug andplay.”However, even systems that are built on awidely used, open sourceplatform such asDHIS2will require customization. As an example, consider theRMNCAH scorecard.Whiletheunderlyingmethodologyforproducingthisscorecardremainsthesameacrossallcontext,ataminimum,theindicatorschosenandtheirrespectivecriteria(e.g.denominator)willneedtobechanged.And,therefore,thefunctionalityoftheICTsystemwillneedadaptation.DevelopingthecapacitytoestablishanenablingenvironmentandthenofferthecustomizedICTenabledsolutionforthatenvironmentofferanotheropportunityforgreatercompetitivenessatUNICEF.AdoptingacustomizationmindsetalsoappliestotheKnowledgeExchangeapproachatUNICEF.Forexample,whilesettingasinglestandardcapforthegroupsizeswouldbeunrealistic,UNICEFshouldconsidersmallergroupsizesinYammerthattargetmorespecificpopulations.2Bysodoing,UNICEFcouldcreategroupsthataremoremanageableintermsofengagementbyfocusonaparticularresultareaorcompetency.Slackisoneexampleofasuccessfulapplicationofthisparadigmwhereusershavetheabilitytosetupchannelsasnecessaryforalmostanyparticulartopicandinviteusersasnecessary.

2Forexample,asof28/05/2017,thesizeoftheRTMCoPwas144.



FurthercustomizationforKEcouldbeemployedwithSharepoint.Thisrepositoryshouldhavearequirementforallinformationtohaveassociatedmetadatathatenablesfasterandmoreaccuratesearching.

Recommendation7:Use ICTenabledprogrammingasabridge formultisectoralprogrammingand

scalingvertically.

TheimplementationofthePMRInitiativeinZimbabwewasclearlyunique,ascomparedtoKenya,SwazilandandUganda,andthishasledtoitsownsetofoutcomes.OneclearlypositivegainfromtheZimbabweexperienceisthemultisectoralnatureoftheimplementation,whichisincontrasttothehealthsectoronlyimplementationoftheotherthreecountries.UNICEFshouldcontinuetopursuesimilaropportunitieswhereICTenabledprogrammingcanovercomebarriers,breaksilosorbridgegapsbetweensectorsforamoreholisticandintegratedapproachtoachievingbetteroutcomesforchildren.Inasimilarvein,UNICEFshouldseektoleverageitssuccessinverticallyscalingtheprogrammingofthePMRInitiativetoachievethesamehorizontally.

Recommendation8:CapitalizeonopportunitiesexposedbythesuccessofthePMRInitiativeinclosing

the feedback loop toaddress the“noavailable resource”challengeandensuringdatacollection is

sustainable.

UNICEFshouldimmediatelyseektocapitalizeonthetwomajorchallengesexposedbythesuccessofthePMRInitiative:Closingthefeedbacklooptoaddressthe“noavailableresource”challengeandensuringdatacollectionissustainable.Aswithotherareasmentionedabove,“solving”theseproblemscouldcreatesignificantcompetitiveadvantagesforUNICEF.Oneoptionforclosingthefeedbackloopcouldbethecreationofabasketoffundsthatcanbeusedtotargetidentifiedproblemareasinpartnershipwithlocalgovernments.Toensuresustainabilityofdatacollection, UNICEF could work with partners to establish processes for data collection that offers asustainableandworth-whilesystemofincentives.



Introduction

BackgroundandContext3

The twenty-one countries ofUNICEF’s Eastern and SouthernAfrica region (ESAR) collectively bear asignificantproportionofglobalunder-fivemortality,malnutritionandHIVdespitenotableimprovementsmadeinrecentyears.Tofurtheraccelerateimprovementsinchildhealthandnutritionoutcomes,thereisanincreasingtrendwithinESARcountriestodecentralizeresponsibilityforplanninganddeliveryofessentialchildsurvivalanddevelopmentservicesasameanstobringpublicdecision-makersclosertolocalcommunitiesandneeds,buildlocalcapacityandimproveaccountability.Concurrently,thehasbeenanobservationfrompastUNICEFandotherUNagencies’workthat,formostservicedeliverysystemsatdecentralized levels, the traditionalmodelof input-outputmonitoringandoccasionalimpactmonitoring(attheendofamulti-yearperiod)isnotaddressingdatagapsrelatedtotheachievementof intermediateresults.Thismeans there isa lackof ‘earlywarning’data,aswellasfeedbackloops,tosignalwhetherservicesarelikelytobeeffectiveornot;implementationbottlenecksand barriers need to be removed; and mid-course programme adjustments need to be made. Thedetrimentaleffectsofanabsenceof intermediateresultsmonitoringandmorereal-timeresponsearemany.Theseincludeineffectiveplanningandbudgeting,inefficientuseofmaterialandhumanresources,andcontinuedexistenceoflargeunderservedpopulationsofchildren.Further,althoughhigh-impact,evidence-basedandaffordablechildsurvivalinterventionsareknown,anumberofglobalstudieshavehighlightedthepoorinstitutionalcapacitywithindecentralizedhealthandhealth-relatedsystemsindevelopingcountriesandtheconstraintsfacedbydistrictmanagementteamsintheimplementationofhealthandhealth-relatedservices.Theseincludealackofreliable,timelyandstrategically-selectedandpresentedlocaldata;insufficientemphasisonresults;lackofidentificationofbottlenecksimpedingeffectivecoverage;failuretoinvolvecommunitiesasactiveproponentsinhelpingovercome obstacles to high coverage; and a lack of analytic and prioritization capacity which isparticularlyneededwithinthecontextofchronicresourceconstraintsfacedbysub-nationalmanagementteams.Asaresult,innovativeandsustainablestrategiesareneededtostrengthendistrictandothersub-national monitoring systems, including citizen/community monitoring and social accountabilityfeedbackmechanisms,anddatausepracticesinthehealthand/orhealthrelatedsectorsasameanstoachieveimprovedandmoreequitabledeliveryofqualityservicesandoutcomesforchildrenandwomen.

Overview:thePMRInitiative4

In 2014, UNICEF headquarters, in collaborationwith ESARO, took the decision to address the issuesenumeratedabove,relatedtothehealthofwomenandchildrenbyimplementingaregionalinstitutionalstrengtheningsupportinitiativetoimprovetheProgrammeMonitoringandResponse(PMR)capacityofthe centralized government decision-making bodies and communities in four countries (Kenya,Swaziland,UgandaandZimbabwe).TheESARPMRwasdesignedasalearninginitiativewiththefollowinggoals:

• Improve service provision and programme performance during implementation throughimprovedmonitoringofintermediateresults,establishmentofcitizenfeedbackloops,andtheuseofmonitoringandfeedbackdatatoadjuststrategiesasneededinatimelymanner;and

• Identifyandpromotethosepromisingpracticesofprogrammemonitoringinresponse,testedandproven by early adopter districts, through the development of national scale-up planswithinparticipatingcountriesandtheestablishmentofknowledgeexchangeplatforms.

3ThetextinthissectionhasbeentakenfromtheTORofthisevaluationassignment.4ThissectionwasadaptedfromtheTORforthisevaluationassignment.ForadetailedoverviewofthePMRInitiativesee:CaseStudy:UNICEFharnessesuseofnearreal-timedatatosupportmaternal,newbornandchildhealthinEasternandSouthernAfrica(2017)



ThePMRInitiativeTheoryofChange

Figure1-ThePMRInitiativeTheoryofChange

The hypothesis underlying the PMR initiative is that, if decentralizedmanagement and/or front linehealthworkerdecisionsandresponsesaretimelierandevidence-based,andifcitizenfeedbackloopsandsocialaccountabilitystructuresareestablished,thenovertimeimprovementswillbeseenin:

• Sectorserviceprovision/programperformance(i.e.coverageandqualityofservices);• Communities’commitmenttoinvestintheirownhealthandwell-being;and• The health and well-being of vulnerable and previously underserved children and women,

therebyimprovingequityofhealthoutcomes.TooperationalizethisTheoryofChange,UNICEFdesignedtheInitiative’sinstitutionalsupportacrossfiveprogrammaticpillars.TheInitiative’sactivitieshavesupported:

1. Thedevelopmentof reliable andquality (more) real-timedata availability at the sub-nationallevel for identifying and addressing barriers and bottlenecks to the delivery of health and/orhealthrelatedservices(Nutrition,HIV,WASH);

2. The provision of feedback by citizens/youth groups on the delivery of health and/or healthrelatedservices;

3. Thereviewofperformancebydecentralizeddecision-makingbodiesusingdatafromreal-timemonitoringandcitizenfeedbackonaquarterlybasisandthedevelopmentofactionplans;

4. ThedisseminationofknowledgeamongUNICEFandrelevantgovernmentstaffinESARcountrieson the use of real-timemonitoring data for decision-making in the delivery of health and/orhealth-relatedservices;and

5. The development of evidence-informed national plans of actionwithin the four countries forscalingupmodelsfordecentralizedPMR.

BecauseoftheshorttimeframeofthePMRInitiativeandthetimingofthisevaluation(i.e.performedwhilemanyoutputswerestillinprogress),theevaluationisunabletoproperlytestthelogicofthedesignandwhetherit“worked.”However,therearepromisingpreliminaryfindings(seebelow)thatindicatethesupportprovidedbythePMRInitiativeisresultinginimprovedsectorperformance,ataminimum.

CountrySpecificImplementations

Four countrieswere selected for participation in the PMR Initiative (Malawi, Swaziland, Uganda andZimbabwe).Astheprogrammebegan,MalawielectedtodropoutbecauseofanationalfloodsemergencyandwasreplacedbyKenya.WhilethePMRInitiativewasdesignedwithanumbrellaresultsframework

… thenthiswillensue… … resultinginthis.

Improvedsector

performance(coverage/quality)

Communitycommitment

toselfimprovement

More,timelierevidence

Localizeddecisionmaking

Inclusionofcitizenvoice

MoreEquitableHealth

OutcomesforChildren

Ifthishappens…



fortheregion,5eachimplementingcountrycustomizedandlocalizedtheactualimplementationoftheInitiativeinthefollowingways:6

Kenya

UNICEFKenya,inpartnershipwiththeSchoolofComputingandInformaticsattheUniversityofNairobi,hasdevelopednewcounty-andsub- county-levelDHIS2-based IntegratedCountyHealth InformationMesh(iCHIM)dashboardswhichareusedatroutinequarterlyperformancereviewmeetingsalongsidethe recently rolled out Reproductive, Maternal, New born, Child and Adolescent Health (RMNCAH)scorecard. This is combinedwith qualitative community feedbackmade available through the newlydeveloped Information and Communications Technology for Social Accountability, a digital means ofreceiving feedback from community dialogue to be entered into DHIS2 and shared with healthmanagementteams.UNICEFKenyahasalsoadvancedmobilephoneusebycommunityhealthworkersforfasterentryofstandardcommunityhealthinformationdirectlyintoDHIS2.

Swaziland

UNICEF Swaziland has developed the country’s first healthmanagement information system (HMIS)healthdashboardsforuseatthesubnationallevel.ThekeyfocusoftheinitiativeisensuringthatdatafromthenewlypilotednationalelectronicClientManagementInformationSystem(eCMIS)areusedatdecentralized levels by the Regional Health Management Teams and facilities. UNICEF’s SMS- basedcitizenengagementplatform,U-Report,isalsousedtogeneratefeedbackfromhealthfacilityclientsandtopromotepublichealthmessages.

Uganda

UNICEFUgandaisdemonstratingthevalueoflinkingindicatorstobottleneckanalysisbyfullyautomatingthenationalRMNCAHscorecardwithinDHIS2alongsideabottleneckanalysisdashboardandtwootherNearRealTimeMonitoring(NRTM)dashboardsondataqualityassuranceandanactiontracker.Thenewdashboard suite is complemented by U-Report citizen feedback, linked to a complaints hotline andfacility-based community dialogues, to concurrently understand the demand–side of health servicedelivery.

Zimbabwe

UNICEFZimbabwehasdevelopeda community-basednear real timemonitoring system(NRTMS) tostrengthentheevidence-baseoftheMultisectoralCommunity–basedApproachforReductionofStunting(MSCBARS) Programme and to involve the community in creating solutions to nutrition challengesaffectingtheirchildren.Datafromhouseholdswithpregnantwomenandchildrenundertwoyearsofageare linked to ward-level action microplans and complemented by U-Report poll data to capturecommunityfeedbackonnutritionandassociatedinterventions.

PMRInitiativeTimeline

ThePMRInitiativebeganinOctober2014andwasoriginallysetforcompletioninSeptember2016.Atthetimeoftheevaluation,theInitiativehadreceivedano-costextensionthroughJuly2017.

TheEvaluation

TimeframeandTeam

Theevaluationwascarriedoutbetween27MAR17and31MAY17.Fieldworkwascompletedbetween15April17and10May17.Theevaluationwascarriedoutby:

5TheumbrellaPMRInitiativeresultsframeworkisincludedasanAnnex.6Thetextforthissectionwastakendirectlyfrom:CaseStudy:UNICEFharnessesuseofnearreal-timedatatosupportmaternal,newbornandchildhealthinEasternandSouthernAfrica(2017)



TeamLeader–StephenLadek

Stephen Ladek, Principal of ISG, is a seasoned international development and humanitarian aidprofessionalwithmorethan15yearsofexperiencehelpingindividualsandorganisationsimprovetheperformance of how they deliver development and humanitarian aid programming throughorganisational development, results based management and independent evaluation. Mr. Ladek hasoverseen and directly implemented evaluations related to funding from numerous governmentsintergovernmental agencies international organisations, NGOs, private enterprises and community-based organisations inmore than35 countries.Heholds aBS in FarmandRanchManagement fromColoradoStateUniversity,aMAinInternationalPeaceandConflictResolutionfromAmericanUniversityinWashington,DCandanMBAfromCentralEuropeanUniversity.

Co-Evaluator–MarkIrura

Mark Irura is a Kenyan national and Nairobi-based technical manager. His expertise is in designingsystemsandmanagingtechnicalprojects,withatrackrecordofmorethan10yearsonsuccessfulprojectswith theGovernment of Kenya, the private sector andmultinational organisations.He has developedseveralprocess-focusedsoftwaresystemsandcustomer-facingapplications, includingaretail internetbankingtool.Hedeeplyunderstandswhatusersneedintechnicaltoolsforadoptionintodailyprocesses.HeholdsanMScinInformationTechnologyfromStrathmoreUniversity,aBScinComputerSciencefromUniversityofCapeTownand iscompletingaPhDinComputerScience fromtheUniversityofEasternFinland.

Purpose

Thepurposeoftheevaluationisto:• Supportthefurthernationalscale-upofdocumented,effectivePMRinstitutionalstrengthening

interventionsinthefourparticipatingcountries;• GuideanticipatedreplicationinothercountriesinwhichUNICEFoperates;• Addtotheregionalandglobalknowledgebasebygeneratingevidenceonprocessesandoutcomes

relatedtostrengtheningdecentralizedPMRsystemsinthehealthandnutritionsectors;and• Provide information on the extent to which the work undertaken in the four countries is

applicabletoothersectorsbeyondhealthandnutrition.

EvaluationObjectives

Theobjectivesoftheevaluationareto:• Assess achievement towards the programme’s expected results (milestones, outputs,

intermediateoutcomes,andprimaryoutcomes)aspertheinitiative’sResultsFramework;• Assesstheextenttowhichtheprogrammeactivitieshavechangedhow(more)realtimedatais

collected, analysed, packaged, disseminated, and used (i.e. the extent of uptake by targeteddecentralizeddecisionmakingbodies);

• Assesswhether, and how,mutual and ongoing learning on PMR is taking place across targetdecentralizedgovernmentunitsandcountries,aswellaswithintargetUNICEFoffices;

• Understand and document the range of contextual/environmental factors that served asenablers/disablers to the results achieved, including programme design, implementation,managementandcoordinationprocesses;

• Assess the programme’s relevance, effectiveness, sustainability, scalability, and replicability todetermine if the package of support interventions and tools employed by UNICEF’s fourparticipatingCOscanserveasapplicablemodelsforotherUNICEFcountriesandstakeholdersinsimilarcontextandsectors,andothersocialsectors;

• DetermineifthefourUNICEFCOscanserveas‘centresofexcellence’withinUNICEFgloballyondecentralizedPMRorspecificaspectsofdecentralizedPMR;

• Documentbestpracticesandlessonslearnedrelatedto:o EffectivePMRamongdecentralizedpublicdecision-makingteams



o InstitutionalsupportaimedatstrengtheningdecentralizedPMRmechanisms,processesandpractices,and

o Adaptiveprogrammingduringimplementationthroughtheavailabilityanduseofreal-timemonitoringandcitizen-feedback;and

• RecommendwhetherthistypeofinstitutionalstrengtheninginitiativefocusedonPMRshouldbecontinuedwithinESARand/orotherUNICEFregionsasaprogrammingstrategyregardlessofsector,andifyes,howitwouldbestbeplacedwithinUNICEF’sorganisationalstructure.

Scope

TheevaluationcoverstheentireimplementationperiodfromNovember2014tothepresent.Therearenocomparators forassessment.Theevaluation includesallactivitiescarriedout inKenya,Swaziland,UgandaandZimbabwe,ESAROandUNICEF’sNewYorkHeadquarters.Ineachcountryspecifically:

• InKeyna,HomabayandSiayacounties.• InSwaziland,allfourregions(Manzini,Hhohho,LubomboandShiselweni)• InUganda, thePMRinitiativewas initiallypilotedMoyo,MukonoandButambaladistricts.Full

initiativesupportwasscaledtoincludeAdjumani,Isngiro,Nebbi,RiubiriziandYumbeDistrictsandtheunderlyingscorecard,bottleneckanalysisandactionplanningwasscalednationally.

• InZimbabwe,Chipinge,Chiredzi,MutasaandMwenezidistricts

Methodology

GuidingPrinciples

InadditiontotheevaluationbeinginaccordancewiththeUNEGNormsandStandardsforReviewandtheUNEGEthicalGuidelinesforReview,ISGusedarangeofparticipatorymethodstoensurethatkeystakeholdersandpartnerswerecentrallyinvolvedinreflectiveandforwardthinkingprocesses.Asitwascarriedout,theevaluationadheredtothefollowingprinciples:

• Consultationwith,andparticipationby,keystakeholderstoensuretheevaluationisrelevanttoitsusersandstakeholders,andthattheevidenceandanalysisaresoundandfactuallyaccurate.Consultationwasiterative,informingeachstageoftheassignmentprogressively.

• Methodologicalrigortoensurethatthemostappropriatesourcesofevidenceforansweringtheanalyticalframeworkwereusedinatechnicallyappropriatemanner.Theevaluationteamuseddifferentdatasourcesandmethodsthroughouttheprocessofthestudytocorroboratingfindingstoandensureconsistency.

• Technicalexpertiseandexpertknowledgetoensuretheevaluationbenefitsfromknowledgeandexperience relevant to effective decentralized programme monitoring and technologicalinnovationsindevelopmentandthatitcontributestobuildingthebodyofevidencearoundwhatworks,whatdoesnotwork,andwhy.

• Independencetoensurethatthefindingsstandsolelyonanimpartialandobjectiveanalysisoftheevidence,withoutundueinfluencebyanystakeholdergroup.

Inthiscontext,ourapproachincorporatedbestpracticeevaluationcriteriaandprinciplesforeffectivedevelopmentassistanceaswellasnormsandstandardsoftheOECD/DACframework.

Datacollectiontoolsandpurposes

Theevaluationreliedonasuiteofactionresearchmethodologiesthatcollectqualitativedata.Specifically:

Documentreview

AdocumentreviewformedtheinitialassessmentofthePMRInitiativeattheglobal,regionalandnationallevels,identifyingoutputs,successandchallengesrelatedtotheprogram.Theprojectteamusedinsightsgainedfromthedocumentreviewtoinformthestudy,ingeneral,andrefineothertools.



Keyinformantinterviews(Individualandgroup)

Semi-structuredinterviewswithaselectionofstakeholdersareanidealmethodforobtainingin-depth,qualitativeinformation.Aselectionofkeyinformantstobeinterviewed(eitherindividuallyorinagroupdiscussionformat)at theglobalandcountry levelswasdevelopedinconsultationwiththeevaluationmanager(whoalsoservesasthePMRInitiativeCoordinator)andUNICEFPMRNationalOfficersatthecountry-level.ThislistincludedUNICEFstaffatglobal,regionalandcountrylevels,primarystakeholdersin each country (e.g. staff from government partners), and external stakeholders (e.g. donors,implementingpartners,otherCSOs,iNGOs).InterviewguidesandalistofintervieweesarelocatedintheAnnex.

DirectObservation(Fieldvisits)

ISGundertookfourfieldvisitstoKenya,Swaziland,UgandaandZimbabweinAprilandMayof2017.7Inaddition to administering interviews, ISG directly observed implementation of each country’scontextualizedapproachwithinthePMRInitiative.

MostSignificantChange

ISGemployedtheMostSignificantChange(MSC)methodtocollectstoriesfromstakeholdersabouttheirperceptionsofhowthePMRInitiativehaschangedsystemsandprocessesineachofthefourcountries.WhiletheoriginalintentionwastousethePMRprogrammaticpillarsasthedomainsofchange,becauseofthelimitedtimefortheassignment,onlyone,generalizeddomainwasusedwithkeyinformants.Inaddition,andagainbecauseoflimitedtime,theMSCdatawasnotfilteredthroughmultiplestakeholderlevelstoarriveatasinglestoryforthePMRInitiative.Becauseofthis,theevaluationteamchosetousetheentiretyofcollectedMSCdataasfurtherevidenceforthefindingsandconclusionsbelow.

EthicalPrinciplesandPractices

ISG adheres to international normson research standards and ethics. For all interviews, ISG ensuredexplicitinformedconsentfromindividualsthroughverbalacceptanceatthebeginningofinterviews.Allindividualsinterviewedinthisevaluation,tothebestknowledgeoftheevaluationteam,wereofadultage.

Evaluationcriteria

TheevaluationwascarriedoutusingalimitedselectionofstandardOECD/DACCriteriathatincluded:relevance, effectiveness, and sustainability. Efficiency and Impact were not included, as agreed inconsultation with UNICEF, because of the intended foci of the evaluation and overall timing of theInitiative.

7Completedetailsofthefieldvisitscanbefoundintheaccompanyingcountryreports.



AnalyticalFramework

Ananalyticalframeworkwascreated,inconsultationwithUNICEF,toguidethecontentoftheevaluation;thatis,tofocuswhattheevaluationwillspecificallyinvestigate,andhowISGwoulddothis.Insummary,theevaluationsoughttoanswerthefollowingquestions:

RELEVANCE• Is UNICEF’s decentralized PMR institutional strengthening support an appropriate response to documented

country needs, priorities, policy context, capacity; and within UNICEF’s purview (e.g. non-duplicative, withinUNICEF’smandateandcapacity?)

EFFECTIVENESS/ACHIEVEMENTOFINTENDEDRESULTS• ForOutcome1.1

o Is reliable andquality real-timedata available todistricts for identifying and addressingbarriers andbottleneckstothedeliveryofhealth,nutrition,HIVand/orWASHservices?

o Was a monitoring platform(s) that produces reliable, quality and (more) real-time supply-side dataestablishedorenhancedthroughtheinitiative?

o Arethenew,improvedorpreviouslyunutilizedreportsproducedandpresentedfromthisplatformusefultotargetdecision-makers?

o To what extent does UNICEF have a comparative advantage in this outcome area? How would thecomparativeadvantagebedefined?

• ForOutcome1.2o Wasa citizen/ community feedbackplatform(s) thatproducesrelevant,credibleand(more) real-time

demand-sideinformationestablishedorenhancedthroughtheinitiative?o Were new or improved citizen/ community feedback data reports/ tools/ processes/ practices

introduced?o To what extent does UNICEF have a competitive advantage in this outcome area? How would the

competitiveadvantagebedefined?• ForOutcome1.3

o Are quarterly performance reviews conducted and acted upon by districts using data from real timemonitoringandcitizenfeedback?

o Whatisthevalue-addedofthedigital/non-digitalplatform(s)anddatareportsthatweredevelopedorenhancedthroughthisprogram?

o Is data being demanded and used to a greater extent for decision-making by targeted decentralizedbodies?

o Hastheuseofdataleadtoevidence-basedactionbytargetdecision-makersandgreateraccountabilityonthepartof‘dutybearers’?

• ForOutcome2.1o DostaffofgovernmentsandUNICEFuseplatformsforcross-countrylearningaboutrealtimeprogramme

monitoringandresponseinESAR?o Haveknowledgeplatforms, tools,productsand/or learningopportunitiesonPMRbeen generatedand

exchangedthroughtheinitiative?o What is theuse,and feedbackonthevalue,of thevarious in-countryandcross-countryKEplatforms,

tools,productsandlearningopportunitiesdevelopedandrolled-outthroughtheInitiative?o WhatcouldbedoneinfuturebyUNICEFtostrengthenknowledgecaptureandsharingwithinasimilar

programme,withintheregionandmorebroadly?• Outcome2.2

o Have evidence-informedNational Plans ofAction for scaling-upmodels for decentralizedprogrammemonitoringbeendeveloped?

o Is there evidence to suggest that gains made through the Initiative will be sustained and/or scalednationally?

o Does UNICEF have a comparative advantage relative to other stakeholders to replicate this type ofinstitutionalsupportinothercountries?Ifyes,doesitexistequallyforallfiveoftheprogrammaticpillars?

SUSTAINABILITYANDSCALABILITY• WeretheinterventionpackagesselectedbytheCOsdesignedwithsustainabilityandscalabilityinmind?• Donationalscale-upplansexistfortheintegratedPMRmodelstested,orthosedemonstrated-effectivecomponents

ofthemodelstested,withineachofthefourparticipatingcountries?Arethesescale-upplansrealisticanddotheyincluderealisticsustainabilityplans?



LimitationsThis evaluation is focused on the processes of implementation. It is an exercise in independentlyvalidatingtheachievementofintermediateoutcomesbythePMRInitiativeineachofthefourcountries.Assuch,theevaluationteamdidnotconsiderdatawithintheHMISsystemsofthefourcountries,nordidtheyseektointerviewendbeneficiariesinthetargetedcommunities.Inotherwords,longtermresultsorPrimaryOutcomes8intendedthroughthePMRworkwerenotconsideredbytheevaluation.Thetimingoftheevaluationhasalsobeenalimitingfactorintwoimportantways:

• ThetimeframeoftheoverallPMRInitiativewasshort,coveringonly2years.Atthetimeoftheevaluation, there were still outputs from each country pending delivery and, as such, theevaluationwasnotabletofullyconsiderthemduringfieldvisitsandsubsequentanalysis.

• The timeallotted for theevaluationexercisewasminimal.Originallyplanned for90days, theevaluationwasultimatelyconductedoveronlya60-dayperiodduetoadministrativedelays.This,inturnlimitedtheamountofdatacollectedandthedepthandbreadthofsubsequentanalysis.

HowtousethisreportThis report is intended to be a synthesis of findings, conclusions and recommendations in the fourcountryreportsthataccompanyit.Assuch,ingeneral,forgreaterdetailaboutcountry-specificfindings,pleaserefertotheseintheAnnex.

8SeetheumbrellaPMRInitiativeresultsframework,includedasanAnnex.



Findings

Relevance

IsthePMRinitiativerelevanttothelocalcontextsinwhichitwasimplemented?

Finding1:ThePMRInitiativehasitsgenesisinanunderstandingofthecontinuedevolutionofICT4D

innovationsinESARandhowtheseinnovationshavecontributedtobuildingacultureofmanagement

usingdatainthehealthsector.Becauseofthis,thePMRInitiativehasbeenaresponsethatcanbe

localizedandcontextualizedtospecificcountryneeds,priorities,andcapacity.

ThefocusandstructureofthePMRInitiativedidnotoriginatefromaspecificdemandinanyoneofthefourcountriesofimplementation.Rather,asdescribedbyintervieweesatUNICEF,theoverallconceptfortheinitiativewasconceivedbyleadershipinbothUNICEFHQandESARO.However,thisisnottosaythattheinitiativeisprescriptive.Infact,theevaluationfindsquitetheoppositeistrue.ThePMRInitiativeisarecognitionthat, inESAR,thedigitizationofhealthsystemshasmaturedtoapointwheretherewasacriticalneedtomovebeyonddatacollectionandaggregationtodataanalysisandapplication–especiallyatthedistrictmanagementlevel.Saidanotherway,throughitsfocusondatause,thePMRInitiative captures the intangible need for managers tobecome“emotional”aboutthedataavailabletothem.Thepowerofthisemotionalresponsewasseenseveraltimesduringtheevaluation.Astwobriefexamples:

• The overwhelming response and immediatedemand for training fromstaff inUganda fromanon-interventiondistrictafterdemonstratingtheautomated RMNCAH scorecard to them for thefirsttime;

• Theimmediategroupresponseandfocusrelatedtoauniquescorecardproducedbythisevaluation(basedoncountryimplementationoftheinitiative– see below in the Effectiveness section) aspresentedattheFinalPartnerMeetinginUganda.

Insum,thePMRInitiativehasfoundawayfordatatohaveimmediate meaning to non-technical staff at all levels in a way that allows them to identify poorlyperformingindicatorsanddevisespecificplanstoaddressthesame.ThePMRInitiativehasanumbrellaresultsframeworkwithfivepillars.Paraphrased,thesepillarsare:

1. Developingtheavailabilityofmore,timelierdata;2. Theincorporationofmore,timeliercitizenfeedbackintomanagementprocesses;3. Instigatingregularmanagementreviewsofreporteddata;4. Greatersharingofknowledgelocally,regionallyandglobally;and5. Determininghowscaling,sustainingandreplicatingofexistingsystemscanbeaccomplished.

Whileeachcountryworksunderthesefivepillars,eachcountrymadeindividualchoicesabouthowtoimplementtheinitiativelocally.Specifically:

Sub-questionsconsideredforthisevaluationquestion:• IsUNICEF’sdecentralizedPMRinstitutionalstrengtheningsupportanappropriateresponse

to documented country needs, priorities, policy context, capacity; and within UNICEF’spurview(e.g.non-duplicative,withinUNICEF’smandateandcapacity?)

GettingEmotionalaboutData:“You can make a very good and complexsystem,andifitcannotbeeasilyandquicklyunderstood,itisuseless.Thereisathree-stepprocessaround the scorecard. You justneedthree steps to create a scorecard. You onlyneedthreecolourstoseehowyouaredoing.So, youareableto lookat themostcomplexhealth informatics problem, in just threesteps.That’sallyouneed.Also, itsmoreof alocalizedconcept.Ithasbeenacombinationofpeople and ideas. Putting it in the DHIS2meansitsnotjustUgandans,butTanzanians,Malawians.Ithinkithasthepowertochangelives.Ithasthepowertochangeprogrammes.”–Ugandanrespondent



Kenya

The Kenya health sector strategic plan 2014 – 2018 prioritizes capacity building in monitoring andevaluation(M&E)atdifferentlevelsofthehealthcaresystem.Also,theKenyaHealthDataCollaborative9roadmapsingledoutdataanalyticsatbothnationalandsubnationallevelasvitalactivitiesthatwouldaccelerateachievingexpectedresultsinthehealthsector.Assuch,thePMRInitiativewaswellalignedtoexistingnationalplansalreadyestablishedbytheMOH.Morespecifically, likemanycountries inAfrica,DHIS2is theprimarycentralizedrepositoryofdata inKenyaandallinitiativesinhealthmustbealignedtothistoolandframework.ThePMRinitiativefocusedonthedevelopmentofannualworkplans,intheimplementationcounties,thataretrulydatadrivenbysupportingtheRMNCAHscorecardanddigitizationofpreviouslypaperbasedtools(i.e.MOH513,514,515and51610).

Uganda

UNICEFhaspartneredwithandsupported theMOH the foralmostadecade inUganda.Muchof thissupport has been focused on strengthening MOH systems through digitization and technologyimplementation.DHIS2isthefoundationforthecountry’sHMIS,andthisprovidesaccesshealthdataacrossthecountryfordistricts.However,dataavailabilityremainsconstrictedbecauseofaccessbottlenecks(e.g.inpracticeonlythedistrictbiostatisticianhastruedataaccess)andanalysisofdatarequiresdataextractionandhandcodedanalysis.ThePMRInitiativewasusedasanopportunitytocontinuetosupporttheMOHinitsevolutionoftheHMISsystemby implementinganautomatedRMNCAHscorecardandsupportingworkplanningwithcapacitybuildingaroundbottleneckanalysisandcausalanalysis.

Swaziland

Swaziland is currently transitioning its existing HMIS to a Microsoft-based Client ManagementInformation System (CMIS). This transition includes the upgrade of health facilities infrastructure tosupport automated recording of data throughout the range of services delivered from point to pointwithin the facility.Theupgrade initiative fallswithin thegoalsof theMoH thataid to fulfil the largergovernment2022vision11wheretheCMISisaknownprojectthatwillrideonthegainsoftheNationalICTInfrastructurePolicy.12Assuch,whilethePMRInitiativewasnotabletomapontoaDHIS2basedsystem,asinothercountries,ithasfocusedonbuildingcapacitytousedashboardsthatarecreatedfromtheexistingHMIS,andnewCMIS.Becauseofthefacilitytransitions,dataownershipisveryhighbecausetheCMISisbecomingthestandardwayofdoingbusinessinthesepilotfacilities.DashboardscreatedunderthePMRinitiative,andtheirassociatedtraining,arebeingrolledoutineveryfacilityintandemwithCMIS.

Zimbabwe

Zimbabwe is anoutlier in thePMR Initiative, in that, theUNICEFCO took thedecision to support anexistingcountryprogrammefocusedonaddressingstunting(MSCBARS)throughtheprovisionofmore, 9SeeoneoftheprioritiesoftheKenyaHealthDataCollaborativeasoutlinedintheroadmap–GoalA,1-https://www.healthdatacollaborative.org/fileadmin/uploads/hdc/Documents/KENYA_THE_ROADMAP__Kenya__Health_Data_Collaborative__K_H_D__C_.pdf10ReadmoreintheKenyaCountryReport11ThisisoutlinedintheGoSnationalvision2022thatseekstohttp://www.gov.sz/12TheNational ICTPolicy states thatGoS isbroadlycommitted to implementationof ICTwhichwill facilitate…sectoraldevelopmentplansandprojectswhichhaveanICTcomponentincludingtheformulationofappropriatepolicies,strategiesandplansfortheimplementationofe-applicationse.g.electronicgovernmentandgovernance,e-health,electroniccommerce.(http://www.ellipsis.co.za/wp-content/uploads/2016/07/Swaziland.pdf)



realtime,crosssectordata.ThePMRInitiativewasusedtobuildabespokeonlineplatformtoreceive,analyse,reportandofferadmintools.

Effectiveness

The evaluation team created the “scorecard” below to quickly communicate the findings for theeffectiveness of the PMR Initiative in a format that was immediately familiar to all programmeparticipants. The format mimics the RMNCAH scorecard used in three of the four implementationcountries. The colour codingwas used in all four countries. Green indicates “good,” yellow indicates“improvementsareneeded”andredindicates“criticalattentionneeded.”Country Outcome1.1.

More Real TimeDataAvailable

Outcome1.2

Communityfeedback

Outcome1.3

Quarterlyperformancereviews

Outcome2.1

Knowledgeexchange

Outcome2.2

National PlansforScale

Kenya Achieved Inprocess Achieved Inprocess PartiallyAchieved

Uganda Partiallyachieved

Inprocess Achieved PartiallyAchieved

Achieved

Swaziland Achieved Inprocess Achieved Inprocess Achieved

Zimbabwe Achieved Inprocess Achieved PartiallyAchieved

PartiallyAchieved

Eachcolumnofthescorecardisdiscussedindetailinthefiveintermediateoutcomesectionsbelow.

IntermediateOutcome1.1:Isreliableandqualitymorereal-timedataavailabletodistrictsfor

identifyingandaddressingbarriersandbottlenecks to thedeliveryofhealth,nutrition,HIV

and/orWASHservices?

Asdescribedinmoredetailbelow,theevaluationfindsthat,whileoneofthepillarsofthePMRInitiativeistheincreaseofdataavailability,therealfocusoftheinitiativewasonusingdataflowsthathadalreadybeen established. The exception to this is the case of Zimbabwe,which created an entirely newdatastream,andonethatcapturesdataacrossmanysectors.

Finding2:ThePMRInitiativehasestablishedorenhancedplatformsthatproducereliable,qualityand

morerealtimedatainallfourcountries.However,theemphasisoftheinitiativehasbeenondatause.

Inallfourcountries,fundsfromthePMRInitiativewereusedtobuildorenhanceanexistingdatadrivenplatform.Describedindetailineachofthecountryreportsassociatedwiththeevaluation,insummary,thePMRinitiative:

• Kenya: Development of Community Health Information System (CHIS) targeting communityhealth; automation of RMNCAH scorecard, community score cards as well as MoH tools (forexample,theMoH516).

Sub-questionsconsideredforthisevaluationquestion:• Wasamonitoringplatform(s)thatproducesreliable,qualityand(more)real-timesupply-

sidedataestablishedorenhancedthroughtheinitiative?• Arethenew,improvedorpreviouslyunutilizedreportsproducedandpresentedfromthis

platformusefultotargetdecision-makers?• TowhatextentdoesUNICEFhaveacomparativeadvantageinthisoutcomearea?Howwould

thecomparativeadvantagebedefined?



• Uganda: Development of automated RMNCAHscorecard,bottleneckanalysis(forthcoming)andactiontracker(forthcoming).

• Swaziland: Support to build dashboards thatleveragenewCMIS real-timedata (and, to someextent,theoldHMISdatawheredataisrecordedafteramonth)andtherequisitetraining.

• Zimbabwe: Design, development andimplementation of a bespoke multi sectoralplatform and associated analytics (i.e.dashboards,trendreporting).

However,theevaluationfindsthetechnologicalaspectsofthePMRInitiativeinallfourcountriesisclearlysecondarytotheuseofdatabymanagers.First,inthreeofthefourcountries, the initiative was used as an opportunity toenhance existing HMIS systems. These enhancements,while in some cases increasing the availability of data,werefocusedonthedevelopmentofanalyticaltools(i.e.RMNCAH dashboards). Second, parallel to thedevelopment of the analytical tools, the PMR initiativefocused itssupportonhelpingdistrictand facility levelmanagersusingavailableanalytics to identifyproblemareasandgenerateplansforaction.Third,eveninZimbabwe,whereabespokeplatformwascreatedfromscratch, interviewswithrespondentsatall levelsunderscoredaprogrammaticemphasiscentredontheuseofdatabyalllevelsofFNCstoidentifypoorlyperformingindicatorsandthecreationofplansforactiontoaddressthesame.Finally,theevaluationfoundthat,eveninnon-interventionareas,mangerscoulddescribehowdatathatwasalreadyavailabletothem(viaexistingHMIS)shouldbeusedforunderstandingpriorityissuesanddevelopingplansforaction.

Finding3:TheevaluationcannotdetermineifUNICEFhasacomparativeadvantageintheproduction

oftechnology-basedsolutions.However,UNICEFappearstocurrentlyenjoyasignificantcompetitive

advantage,butonethatneedstobecarefullyarticulatedanddefended.

Thelimitationsoftheevaluationpreventtheabilitytodetermine,amongthemanyoptionsavailabletoUNICEF, what is themost optimal use of UNICEF resources for the highest return (i.e. better healthoutcomesforwomenandchildren).The evaluation finds, across the four countries a part of the study, thatUNICEF enjoys a competitive

advantageinthepublichealthsector.Thisadvantagestemsfrom UNICEF’s association with the United Nations, ingeneral, and UNICEF’s non-agenda partnership model,specifically.Ineachcountry,theevaluationteamrecordedappreciationforthefactthatUNICEFoperatesdifferentlythan other implementing partners – ensuring thatprogrammingalignswiththegoalsofthelocalgovernmentandnotwiththegoalsofthepartners(orfundingstream).Thissentimentwasexpressedateverylevel,perhapsmostpotently at district or county levels where respondentsprovided specific examples of how other implementingpartnerswork,whilehelpful,doesnotaddresstheneedsofthelocalgovernmentstructures.Importantly,emphasisonaligningUNICEF’ssupportwithgovernmentneedsunderscoresthefactthatthetoolsand

CommunitiesUsingDataForDecisions“Previously, at community level, there wassome planning taking place, but it wasarbitrary from higher levels. With NRTMScommunitiesarenowplanningontheirownand taking action. This program is nowbeginningtoinfluencethesupplyofserviceson theground. If you check thedata trends,youcanseethenumberofpeopleaccessingservicesinhealth,agriculture,andmore.Andwearealsoseeingsomebehaviourchangesinhandwashingandothersanitationservices.”–Zimbabwerespondent“IdidnotexpectthatVHWswouldbeabletoappreciatewhatcomputerscandoforthem.NowthattheycanseetheirowninformationissomethingIthinkthatismakingachangein the appreciation of how IT can directlybenefitthem.”–Zimbabwerespondent

MoreEfficientManagementDecisions“Dashboards. Because it’s anoperationalization of an idea that peoplewouldtalkabout,butvery fewexamplestoshow. The dashboards show the reality,rather thanwhat people just ‘think.’ It hasalso provided access. I can sit behind mydesk[inthe capital],openupmycomputerand see what’s happening in Chirezdi. ThealternativewouldhavebeenthatIgetinmycar, drive to that area, systematicallycombine and talking to many differentpeople,thennumbercrunching,etc.beforeIcouldeverseeanything.Thisispowerful.”–Zimbabwerespondent



processescreatedasapartofthePMRInitiativearenotauniqueoffering.Indeed, ineverycase, localcontractorsareusedtoimplementthetechnologysolution.Rather,theemphasisonshiftinggovernmentcultureandbehaviourstowards“managementusingdata”throughtheprovisionofanalyticaltoolsandsubsequentplanningprocessesaretheareasofsignificance.

IntermediateOutcome1.2:Isfeedbackfromcitizens/communitygroupsaboutthedeliveryof

healthand/orhealth-relatedservicescollectedandconsidered?

There is an understanding in all four countries that listening and responding to citizen feedback isessential to theultimate successof thehealth sector.Theevaluation teamwaspresentedavarietyofmethods inwhich this happens – fromSMS applications onmobile devices to traditional communitydialoguemeetings.What is clear is that traditional structures for communityengagement remain thenorm,andthereforethedefault,formanagerstointeractwithcommunityvoiceandthatanopportunityremainstofindamoreefficientalternative.

Finding4:ThePMRInitiativehasnotresultedinaprocessforincorporatingcitizenfeedbackthatrivals

oreliminatestraditional,orestablished,communityandgovernmentstructures.

As described in detail in the associated country reports, the PMR Initiative has sought to implementtechnology based solutions for gathering and incorporating citizen feedback into data flows andmanagementprocessesinthreeofthefourcountries.Inbrief:

• Kenya:KenyahasusedthePMRInitiativeasanopportunitytoenhanceestablishedcommunitydialogues.

• Uganda: U Reportwas used to gather data about citizen needs andmTRAC is used to collect“hotline”informationaboutindividualcomplaintsandcompliments.

• Swaziland:UReporthasbeenimplementedasendusermonitoringsystem.• Zimbabwe: U Report has been implemented to survey communities, via the VFNC, about

programmeperformance.Assuch,bydesign,intwooffourcases(KenyaandSwaziland),mechanismsforfeedbackfromthegreatercommunityremainuntouchedbythePMRInitiative.InKenya,communitydialoguesremainthenorm,andareconsideredacriticalfeatureoftheoverallhealthsystemstructure.13InSwaziland,UReportisusedtopollindividualsabouttheirsatisfactionwithservicesrecentlyreceivedatahealthfacility,andassuch,ismoreakingotendusermonitoringanddoesnotconstitutecitizenfeedbackaboutbroaderhealthcareissues.Intheothertwocases(UgandaandZimbabwe),atechnologicalsolutionforreceivingcitizenfeedbackhasbeenrolledout,buttheresultshavebeenmixed.InUganda,intervieweesinUNICEFandatthedistrictlevelstatedthatusingUReporthadbeenineffectiveincapturinglocalneeds.NotonlywasUReportseenasa“nationallevel”system,butanydatathatwascollectedanddisaggregatedforlocalusetookmonthstobereceivedanddidnotalignwithdatafromDHIS2ortheRMNCAHscorecard.InZimbabwe,UReportis used to poll VFNCs about community engagement in the community action planning process,communityaccesstomultisectorsupportviaimplementationofmicroplansandcommunitysatisfaction

13Source:PresentationbyKenyaTeamatFinalPartnerMeetinginUganda,May2017.

Sub-questionsconsideredforthisevaluationquestion:• Wasacitizen/communityfeedbackplatform(s)thatproducesrelevant,credibleand(more)

real-timedemand-sideinformationestablishedorenhancedthroughtheinitiative?• Wereneworimprovedcitizen/communityfeedbackdatareports/tools/processes/

practicesintroduced?• TowhatextentdoesUNICEFhaveacompetitiveadvantageinthisoutcomearea?Howwould

thecompetitiveadvantagebedefined?



withoverallmultisectoralsupport.Asnotedinthe2016annualreport,theZimbabweCOhasyettoseetractionoftheUReportdatainthegreaterprogramme.In contrast, in each of the four countries, the evaluation clearly finds that establish community andgovernmentstructuresremainthenormforsolicitingandreceivingcommunityfeedback.Asnotedabove,inKenyathisisintentional.However,intheremainingthreecountriestechnologiesintroducedhavenotestablishedanewnormforcitizenengagement.InUganda,intervieweesatthedistrictlevelsregularlymentioned community dialogues, Barazas14 and other structures as methods for engagement. InSwazilandtherewerequarterlydatareviewsinall4regions(thereare3facilitiesineachoftheregions)bytheRegionalHealthManagementTeams.InZimbabwe,theintensivestructureoftheFNCsareanaturalfeedbackmechanism/structurethatreachfromnationaltovillagelevels.

Finding5: Theevaluation cannotdetermine ifUNICEFhas a comparativeadvantage in community

engagement. However, UNICEF’s citizen engagement tools offer an opportunity for competitive

advantage,butonethatneedstobeestablishedandvalidated.

As stated above, the limitations of the evaluation prevent the ability to determine, among themanyoptionsavailabletoUNICEF,whatisthemostoptimaluseofUNICEFresourcesforthehighestreturn(i.e.betterhealthoutcomesforwomenandchildren).Intermsofcompetitiveadvantage,UNICEFhasnotyetbeenabletoshowenoughvalueattheindividuallevelfortechnologicaloptionstoreplacetraditionalorestablishedgovernmentfeedbackstructures.Thisisnot tosay thatUReportormTRAChavenotbeenvaluable innovations.Forexample,asnoted inaprevious evaluation,15 in Uganda, U Report has contributed significantly to political and economicprocessesandmTRAChasbeenakeydriverofaccountabilitythroughits‘hotline’feature.However,atthe timeof thisevaluation, citizensdonot look to technologyas their firstmeans for communicatingneedsorwantswiththehealthsystem.

IntermediateOutcome1.3:Arequarterlyperformancereviewsconductedandacteduponby

districtsusingdatafromrealtimemonitoringandcitizenfeedback?

Across all four countries, the evaluation team was presented with overwhelming evidence thatmanagementstructureswereunderstood,regularmeetingsheldandthatthesemeetingswereusedtoreview data from HMIS (Kenya, Swaziland and Uganda) or the NRTMS (Zimbabwe), identify poorlyperformingindicatorsandcreatespecificplansofactiontoaddressthem.

Finding6:AsignificantachievementofthePMRInitiativeisitscatalyticcontributiontoa“management

usingdata”cultureinthefourimplementationcountries.

AcrossthefourcountriesinvolvedinthePMRInitiative,themostsignificantoutcomerecordedbytheevaluation team is recognitionbymanagersatall levels that “managingusingdata”mustbecome thenorm. As expressed by interviewees, this desire is driven by the introduction of analytical tools (e.g.RMNCAH scorecard, NRTM dashboards, BNA and CA) that are accessible to non-technicalmanagers.Thesetoolsprovidethebasisformanagementconversationsaboutpoorlyperformingindicatorsandthe 14FormoreonBarazas,seehttp://www.monitor.co.ug/artsculture/Reviews/What-happens-at-a-Baraza-/691232-2009610-snj2h8/index.html15See,Multi-CountryIndependentReviewofTechnologicalInnovationsinESAR,April2016(UNICEF)

Sub-questionsconsideredforthisevaluationquestion:• Whatisthevalue-addedofthedigital/non-digitalplatform(s)anddatareportsthatwere

developedorenhancedthroughthisprogram?• Isdatabeingdemandedandusedtoagreaterextentfordecision-makingbytargeted

decentralizedbodies?• Has the use of data lead to evidence-based action by target decision-makers and greater

accountabilityonthepartof‘dutybearers’?



subsequentdevelopmentofactionplanning.InthecaseofZimbabwe,thenatureoftheNRTMSallowsthisprocesstoextend beyond the health sector into education, WASH,agricultureandmore.Specificexamplesfromeachcountryinclude:

Kenya

ThePMRInitiativehascontributedtotheoutcomeofdatafromtheiCHISbecomingakeyagendaitemforcommunitydialoguedays,whichareheldquarterly.Thisdataisthenusedtocreateactionplansatvariouslevels,becausenotallresponsibilityforactionrestswiththecountygovernments.Challengesidentifiedhaveinterventionsplannedbythemostappropriatelevelofleadership.Asexamples,intervieweessharedthatlowreportingratesledSiayaandHomabaycountygovernmentstohiremorevolunteersandthefactthatthecleanlinessofalatrinewouldbeactionedtothehouseholdlevel.

Uganda

Theautomaticscorecardhasprovidedanoptionformanagersandnon-technicalstafftodigintoanalysisanduseS/B/Aprocessdowntothefacility level.Thishasledtoevidenceof localcommunitiestakingspecific, individualizedaction.Asonespecificexample,ata facility inMokonoDistrict, theevaluationteamwas shown aBNA that indicated poor performance around antenatal drugs – pregnantwomenweren’t returning for the complete set of medications before birth. The facility took the decision toaddressthisissueandcreatedanactionplanaroundit.Overasix-monthperiod,thefacilitymovedtheresultsonthisindicatorlocallytoachievetheirexpectedtarget.

Swaziland

Observationsduringin-countryvisitsclearlyindicatedthatdashboards,andtheirassociatedanalytics,areusedbytheregionalhealthmanagementteamsinthefourpilotregions.Itisclearfromthein-countryvisitthatnotonlydidthesemeetingshappen,butthatattendeesfindvalueinthemandareprioritizingindicators foradditionaldashboards tobedeveloped. Below isanexampleof indicatoranddatausesummarypresentedbyManziniRegionalHealthTeaminaquarter1reviewmeetingin2017.Table:ManziniRegionHealthManagementTeamPresentation–Indicatortrackinganddatause.Indicator Baseline Whatmightbethecause Whatpossibleactionsthatcanbetaken

1.Numberofpregnantwomenwhotestedforthefirsttimeandreceivedresults

100% • Datacapturingissues• Humanresource• Laboratorynotcapturing

resultsintosystem• Departmentaldifferences

• Sistertotalktolaboratorytomakesuretheyenterallresults

• Nursestocheckifallclientsresultsarecapturedintosystemandassist

2.Numberofpregnantwomenre-testingforHIVforthefirsttimeduringthispregnancythisperiod

80% • Datacapturingissues• Womennotretested

• Nursestocheckifallclientsresultsarecapturedintosystemandassist

• Checkifalleligiblewomenareretested

3.NumberofexposedinfantswhotestedforHIVusingDNAPCRat6-8weeksandreceivedresults

100% • NationalLabtakesalongtimetobringbackresultsandthemonthlyreportissubmittedwithouttheresults.

• Toengagethenationallabtosubmitbackresultsontimesothattheyarereportedinthecurrentmonth

ChangesinManagementCulture“Whatcapturesitforme,isthatfor10years,we felt no need to revise our planningguidelines.Andwhenwegotthescorecardandbottleneckanalysis,wefoundareasontorevise.Someaning,it’slikewehadgivenup,saying‘leaveitthatway.’Whenwefirstsaw [the S/B/A process], it was beingpiloted with reproductive health, but wesaid,‘thisdirectlyappliestooursystem.’So,the 2017 version [of the planningguidelines], is revising the 2007 version.Evenweourselves [as theplanningoffice]have found a justification for planning.” –Ugandanrespondent



The use of dashboards for decision making has beenenabledbythecapacitythathasbeenbuiltaswellastheemphasisonthedatausepart–evidentintherootcauseanalysisandactionstobetaken.Thisisalsocapturedbetterby this response from an interview at PiggspeakGovernmentHospital(Hhohhoregion),

Zimbabwe

Local communities are taking specific, individualizedactions, when possible (i.e. when actions do not requireexternal resources). For example, interviewed VHW’srelated they have seen improvements in facility use bypregnantwomen,theconsistentmeasuringofchildheightand weight and improved exclusive breastfeedingpractices. Responses from interviewees at all levels

indicate there is clear leadership buy in for both the system and the culture of using data which itpromotes.

Finding7:Byemphasizingdatause,evenwithincompletedatasets,thePMRInitiativehascontributed

toavirtuouscycleofincreaseddemandformoredataandadesireforimproveddataquality.

Whileitcouldbearguedthatdataavailabilitywasincreasedineachofthefourcountries,onlytheNRTMSinZimbabwehascreatedatrulynewstreamofdata(andarich,multi-sectoraloneatthat). InKenya,SwazilandandUgandathePMRInitiativefocusedonusingexistingdataflowsvianationalHMISsystemstobuildoutanalyticalcapabilities.Thesesystemshavebeenbuiltwiththeknowledgethattheyarelikelyrelyinguponincompletedatasets.Assuch,theevaluationteamrecordedaconsistentrefrainfromintervieweesacrossthefourcountriesforbothmoredataandhigherdataquality.Theevaluationdidnotfindthatthesedesireshaveledtoamaterialchange.However,itisimportanttonotethatmanagersrecognizethatbothdataavailabilityanddataqualitydirectlyaffecttheirabilitytohaveaccurateanalysisthatwill feedintotheirplanninganddecisionmakingprocesses.Severalintervieweesalsoexpressedtheviewthatemphasizingdatausealsoautomaticallyleadstoanimprovement indataquality in a (near) real-time fashion.Twoexamples from interviews focusedonrecognizingobviousdataerrors:

1. Ifanenumeratorreportedvisiting600householdsinavillageoveradaybutthevillageonlyhas300households,thereisclearlyanerror;and

2. Ifareportondiseasedetectionshowsasuddenspike,itcanbequicklydeterminedifthisisanerrororatrueissuethatneedsimmediateattention.

Morenuanceddiscussionsaboutdataqualityhaveresultedfromtheintensiveconversationssurroundingagreement on the indicators within the scorecard/dashboards themselves. As stakeholders haveparticipated intheseconversations, intervieweessuggestedthatthishasalsocontributedtoagreaterawarenessaboutthedatacurrentlyavailableandhowitcouldbeimprovedgoingforward.A final example relates to the transformation of data from the enumerator or facility level throughdigitizationintotheHMISsystem.Intwoofthefourcountries(UgandaandZimbabwe)dataoriginatesinpaper form and is then physically transported to a central office for digitization. Interviewees at thevillage/facilitylevelanddistrictlevelsinbothcountriesexpressedinterestinfindingwaystoeliminate

UseofDashboards“Previouslywehadachallengeonreportingsinceeveryquarterwehadtoreportandwehad to request for the report from theRegion; but now we are happy thateverythingisinfrontofusonthedashboard.Formyself,thisweekIwasdoingquarterlyreports and I have been checking reportsfrom the nurses and the ones which areconverted(sic)tothedashboards.Andthatmakes our life easy… for the facility todiscuss these reports they are moreaccurate.”–Swazilandrespondent



errorsalongthischain.KenyaandSwazilandmaybegoodcasestudiesforthisgoingforward,asbothcountrieshavechosentopursueoptionsfordatacollectionthatisdigitalatthesource.16

Finding8:ThePMRInitiativehashighlightedtheneedtoensureresourcemobilizationforidentified

healthsectorissuesasparamount.

The PMR Initiative has successfully catalysed theconversationaroundmanagingusingdatainitsareasofimplementation.Thisshifthas, in turn, leadtoenergyand excitement on the part of both managers andcitizensaboutthepossibilityofchangebeingrealizedintheir communities. More specifically, the analyticaltools developed by the PMR Initiative have allowedmanagerstoidentifypoorlyperformingindicatorsandthenplanandprioritizeactionsaccordingly.Theevaluationteamfindsthat,acrossallfourcountries,thisprocess isquickly leading to apotential situationthat could have important consequences. Initially,managers have been focusing planning and action onareasofimprovementthatrequireno,orverylittle,newresource inputs (e.g. improvement in breastfeedingpractices or better inventory management skills).

However,thesetypesofinterventions,bytheirnature,willbeaddressedrelativelyrapidly.Thiswill,inturn, leave identified problem areas that require financial or other input to be addressed. Whileinteractingwithinterviewees,especiallyatthevillageorcommunitylevel,theevaluationteamheardaneedfortheseresourceinputs.Insomecases,theaskwassmall(e.g.materialsforenumeratorstohelppromotetheirwork)butthereareother,largerissues(e.g.therepairorconstructionofafacility).Asthehealthsectorcontinuestoevolveineachofthesecountries,closingthislooptobeabletoaddressproblemareaswillbecomemoreandmorecritical.

IntermediateOutcome2.1:DothestaffofgovernmentsandUNICEFuseplatformsforcross-

countrylearningaboutrealtimeprogrammemonitoringandresponseinESAR?

Finding 9: UNICEF actively pursued the exchange of knowledge generated by the PMR Initiative

throughavarietyofplatformsandopportunities,withmixedresults.

ThePMRInitiativehasengaged inavarietyofactivitiestopromotetheexchangeofknowledgeabouthealth systems strengthening across the PMR implementation countries and globallywithin UNICEF.Theseactivitiesinclude:

16InthecaseofKenya,theyhaveautomateddatacollectiontoolsatcommunityhealthlevel–withCHAsgoingoutwithmobiledevicestodothiswork.InSwaziland,theCMISenablesautomatedcollectionofdataatdifferentservicepointsinthehealthfacility

Sub-questionsconsideredforthisevaluationquestion:• Haveknowledgeplatforms,tools,productsand/orlearningopportunitiesonPMRbeen

generatedandexchangedthroughtheinitiative?• Whatistheuse,andfeedbackonthevalue,ofthevariousin-countryandcross-countryKE

platforms,tools,productsandlearningopportunitiesdevelopedandrolled-outthroughtheInitiative?

• WhatcouldbedoneinfuturebyUNICEFtostrengthenknowledgecaptureandsharingwithinasimilarprogramme,withintheregionandmorebroadly?

MobilizingResources“Thebiggest transformation forme is thatyouare having similar conversations [aboutstunting]takingplaceindifferentareas.And,itshappeningconsistently.And,wecansithere[atUNICEFmainoffice]andhaveinsightintowhatthose conversations were about. This is a bigchange. Previously, I would not have knownwhat theactionplanwould be for aparticularlocation and how they are coming together tohaveasetof commondemandsacrosssectors.Thishasledtotheunderstandingthatweneedto figure out how to mobilize resources inresponse.TheITsideissolved,nowhowdowerespond to these requests” – Zimbabwerespondent



Knowledge Exchange Toolbox: A Knowledge ExchangeToolbox17 was produced by UNICEF to provide staff andpartnerswithoptions forhow to share their learning.Thistoolbox is available online for free to the public. Theevaluationdidnotcollectdataabout its specificusewithinthePMRInitiative.Yammer Social Network Platform: UNICEF has adoptedYammer18asitsenterprisesocialnetworkingplatform.Aspecific“channel”withinYammerwascreatedforthePMRInitiativetofacilitateandcaptureconversationsbetweenstaffandpartners.Globally, thisplatform has seen increased usage, with approximately 1000 users/month in early 2015 growing toapproximately 3000 users/month by the end of 2016.19 While the Yammer platform has not seenextensiveuseforthePMRInitiative(seebelow),globally,UNICEFmaintainsatleasteightchannelsthatarehighlyactive.Sharepoint: UNICEF has utilized Microsoft Sharepoint as a content management and collaborationplatformforthePMRInitiative.SimilartoYammer,theuseofSharepointforthePMRInitiativehasbeenlimited,withmoststaffandpartnerschoosingtocollaboratethroughother,establishedplatforms,suchasemail.e-Discussions:ThePMRInitiativeheldfour,facilitatede-Discussions.Webinars: The PMR Initiative hosted several webinars for the exchange of information and generalknowledgesharing.Inatleastfourcases,eachcountryinvolvedinthePMRhostedawebinar.Thiseventwasusedasanopportunitytonotonlyshareinformationaboutthecountry’simplementationofPMR,but also to receive feedback from attendees. Across interviews with UNICEF country level staff,respondentsperceivedthesewebinarstobeahighvaluesharingopportunity.QuarterlyMeetings:OverthetwoyearsofthePMRInitiative,fourfacetofacemeetingswereconvened,oneineachoftheimplementingcountries.Eachofthesemeetingswereopportunitiesforcountryteamsand partners to share their experiences to date, learn from others’ implementation and discusschallenges. Because of their face to face nature, interviewees fromUNICEF and partners found thesemeetingshighlyvaluable.Asnotedbyonerespondent,peerreviewsatregionalmeetingswherecountriescomparedimplementationswereseentobeespeciallyeffectivebecause,notonlydidcountryteamssharenew ideas, but they also shared also lessons learned about what does and doesn’t work from pastexperiences.DedicatedKnowledgeExchangeFacilitator:ThePMRInitiativebenefitedfromtheattentionofaparttimededicateknowledgeexchangefacilitator,basedinNairobi.OneofthekeyrolesoftheKnowledgeExchangeFacilitatorwastoconducttrainingonusageofthedocumentrepository(Sharepoint)tosharematerials,webinarrecordings,updatesannouncementsanddiscussionsforthereal-timecommunityofpracticethroughtheYammerPlatform(fordiscussions).Howevermore investments in training is still required (within CO’s and fromESARO) to continue todemonstratetodifferentcommunitiesthebenefitsofthesecommunitiesofpracticeandthebenefitsofspecifictools.Importantly,acrossallknowledgeexchangeplatformsandopportunities,UNICEF intentionally invitedand coached government partners to participate. As related by interviewees at UNICEF, in general,

17Seethefulltoolboxat:https://www.unicef.org/knowledge-exchange/index_82053.html18FormoreinformationonYammersee:https://products.office.com/en-us/yammer/yammer-overview19TheseapproximationswereprovidedduringaninterviewwithUNICEFstaff.

InvestmentsInTraining

“If Iwereto investmoneynow… Iwouldalso invest money in moderating thiscontextual experience and providesustained technical assistance in helpingpeopleutilizethesedashboardsforthemtoseevalue.”–Swazilandrespondent



partnersdidnotactivelyengageintheknowledgesharingplatforms,butwereactiveinmoretraditionalfacetofaceopportunities.IntervieweesnotedthefollowingchallengesrelatedtoUNICEF’sknowledgeexchangeeffortsasapartofthePMRInitiative.Theseinclude:

- Theknowledgeexchangefacilitatorwasonlydedicatedparttime,whichlimitedtheirengagementandpromotionoftheplatforms;

- UNICEFintroducednewtechnologyandplatforms(i.e.Yammer,Sharepoint)andadoptionbystaffandpartnerstakestime;

- ThePMRInitiativehasaveryshortduration,whichleftlittletimeforreflectionand/oriterationrelatedtowhatworks(ornot)relatedtoknowledgeexchange;

- Newsystemsnaturallycompetedwithexistingsystemsthatstaffweremorecomfortablewithforknowledgesharing/storage.Thiscouldalsobestatedinanotherway–users(bothinternalandexternal)haveexistingsourcesofinformationforsuchknowledgeanddonotseeenoughvalueinthecurrentplatformssetuptoreplacethesesources.

The evaluation team noted that several interviewees specifically noted UNICEF’sprofessional/organizational culture as a key limiter to the participation in new platforms, such asYammer.Summarized,theseintervieweesdescribedageneralreticencetoparticipateinanenterprisesocialnetworkbecauseofafearof“lookingstupid”totheirpeers.Thiscontrastswithinteractingonotherplatforms, such asFacebookorWhatsApp,where rapid, frequent andunguarded interactions are thenorm.Another limiting factor for adoption, expressed by interviewees from UNICEF, were internal UNICEFnetworks-aknownphenomenontoallstaffofanyseniority.Stafftendtorelyonthesenetworksforawidevarietyof informationandknowledgesharingand,especially for thosewhouse thesenetworkssuccessfully,thereislittleincentivetoseekinformationelsewhere.

Finding10:Atthecountry level,UNICEFstaffandpartnersrelymainlyontraditionalorestablished

methodsofknowledgesharing.

Atthecountrylevel,theevaluationteamfoundthatintervieweesfromUNICEFhadmixedawarenessofthenewplatformsestablishedforknowledgesharingrelatedtothePMRInitiative.Whenpromptedbytheinterviewer,allstaffcouldidentifytheYammerandSharepointplatforms.However,universally,thesewere not seen as high value sharing resources, but rather low level professional responsibilitiesassociated with the project.When interviewing partners, none of the intervieweesmentioned theseonlineplatformsasknowledgesharingopportunities.Thisdoesnotmeanthatknowledgeexchangedidnothappenintheproject.Indeed,intervieweesfromboth UNICEF staff and partners expressed their appreciation for opportunities to share with othersduring webinars, meetings and other, establish channels like email. This held true within theimplementationoftheinitiativeacrossthefourcountries.Astheevaluationteaminterviewedindividualsat the district/county and village levels, it was clear that knowledge sharing took place, but thattraditionalorestablishmeanswerethemediumbywhichthishappens.AclearexampleofthiscomesfromZimbabwe.TheNRTMSwasbuiltasadatasourcecomplimenttotheMSCBARSprogramme.TheMSCBARSprogrammeestablishedfoodandnutritioncommittees(FNCs)atthe National, Provincial, District, Ward and Village levels. These councils were tasked with meetingregularlytoreviewcollecteddataandassociateddashboardandthenplanandcreate ‘microplans’ forexecution.Thecouncilsalsoweretaskedwithreportingresultsandactionplansupward,whichprovidedampleopportunitiesforfacetofaceinteractionforknowledgesharing.



IntermediateOutcome2.2:Haveevidence-informedNationalPlansofAction for scaling-up

modelsfordecentralizedprogrammemonitoringbeendeveloped?

Finding11:Interventionsthatfocusedonenhancingtheexistingnationalhealthsystemdataplatform

(i.e.HMIS)areimmediatelyscalableandhaveahigherpotentialforsustainability.Nationaladoption

remainsmixed.

InKenya,SwazilandandUganda,thePMRInitiativefocusedonenhancingtheexistingHMISinsomeway.Specifically:

Kenya

TheS/B/Aprocesshasbeenincorporatedintocountyintegratedperformancereviewsandtheannualworkplanprocessofthecountygovernments.BecausetheprocessisnowbakedintotheHMIS,anyothercountyhasaccesstotheassociateddashboards,andseveralcounties(suchasGarissa,Lodwar,TurkanaandKisumu) have requested training to take the initiative forward. The S/B/Aprocess has not beenofficiallyadoptednationallyatthetimeofthisevaluation.

Uganda

Intermsofthesustainabilityofdevelopmentinterventions,thePMRinitiativeinUgandahasachievedatextbookdevelopmentgoal:incorporationoftheS/B/AprocessintonationalplanningguidelineswithassociatedtechnologyenhancementsincorporatedintotheHMIS.Atthetimeoftheevaluation,theS/B/Aprocesshadbeenrolledouttoalldistrictsinthecountry.

Swaziland

SimilartoKenya,inSwaziland,theanalyticaltoolscreatedaspartofthePMRInitiativearehousedintheCMISthat isbeingrolledouttoallhealthfacilities.Assuch,nationalscaleupisonlyamatteroftime,especially given the fact that the government is including training on the dashboardswith each newimplementationoftheCMISinthecountry.Further,datafromthedashboardsfeaturesprominentlyinthequarterlyregionalreviewmeetingsbytheRHMT’s.

Finding12:ThescalabilityoftheNRTMSinZimbabwecouldbeachievedimmediately,howeverthe

sustainabilityoftheMSCBARSmanagementstructureitsupportsisunlikely.

TheNRTMSinZimbabweisanonlineplatformbuildonODKtechnology.Assuch,technically,itcouldbeimmediatelyusedbyanydistrict,wardorvillageacrossZimbabweimmediately,providedtheyhadaninternetconnection.Tothatend,theonlylimitingfactorforscalingtheNRTMSisitsconnectiontotheMSCBARSprogramme.Atthetimeofthisevaluation,MSCBARShasreceivedsupporttoscalefromfourto19districts.The NRTMS, as a technology, is lightweight – requiring only one server and limited additionalinfrastructure,suchastabletsandfeaturephones.However,themanagementstructurethatitsupports,viatheMSCBARS,isheavyinitstraditionalfacetofacereportingandmeetingsfromvillagetonationalFNCs. The unsustainability of this management/meeting process was noted in the Zimbabwe 2016AnnualReportfortheprogramme.

Sub-questionsconsideredforthisevaluationquestion:• WeretheinterventionpackagesselectedbytheCOsdesignedwithsustainabilityand

scalabilityinmind?• Donationalscale-upplansexistfortheintegratedPMRmodelstested,orthosedemonstrated-

effectivecomponentsofthemodelstested,withineachofthefourparticipatingcountries?Arethesescale-upplansrealisticanddotheyincluderealisticsustainabilityplans?



SustainabilityandScalability

Finding13:ThePMR Initiativehas increased thepotential for long termsustainability through the

institutionalizationofprocessesandprotocols,suchastheS/B/Aprocess,butthisincreasedpotential

ismixedacrossthefourimplementingcountries.

The sustainability of any change is underscored by the true adoption and long term commitment ofindividuals and organizational structures affected by the change. In the case of health systemsstrengthening,thiscanbemeasuredintermsofgovernmentpolicyandresourceallocation.Saidanotherway,throughinstitutionalizationofthechanges.Inallfourcountries,tovaryingdegrees,theevaluationfindsthattheseedsforlongtermsustainabilityhavebeensown.Specifically:

Kenya

Asnotedabove,thesuccessesseenthroughthePMRInitiativehavestimulatedhiringofmoreCHA’stoimprovereportingrates,andnon-implementingcountieshavebeguntorequestaccesstotrainingforthedashboardsdevelopedbytheInitiative.However,intervieweesattheMOHindicatedit isimportanttocraft proper incremental plans to detail which components developed through the Initiative thegovernmentGoKwilltakeup,sothatresourcescanbemobilized.Furthermore,inKenyatheenhancedsystem,ascurrentlyimplemented,requiressmartphonestosupportdataentryandviewing.

Swaziland

ThegainsrealizedbythePMRInitiativehavebecomeapartofde-factopolicyinSwaziland,asallnewimplementationsoftheCMISwillalsoincludethedashboardsandtrainingrelatedtotheS/B/Aprocess.Thechallengehereisthatthisrolloutrequiresresourcestobemobilized(andwhicharenotyetfullysecured)asofthisevaluation,theMoHisstillengagingwithdifferentpartnersforsustainedrolloutofthePMRinitiativealongsideCMISroll-out.

Uganda

Asnotedabove,gainsfromthePMRInitiativehavebeeninstitutionalizedindistrictplanningguidelinesasof2016.Giventhatthelastchangestotheseguidelineswas10yearsearlier,thereisahighlikelihoodthattheS/B/Aprocesswillbecomethe“norm”forplanningandbudgetingintheUgandanhealthsector.TheevaluationalsorecordedevidenceoftheS/B/Aprocessresultinginfacilitylevelresultsandbeingused for targeted (and therefore more effective) resource mobilization at the district level, furtherpromotingandembeddingtheprocess.

Zimbabwe

WhilerecentlyreceivingsupportfromDFIDtoexpand from four to 19 districts, thesustainability of the NRTMS in Zimbabwe istentative, especially given UNICEF’s internalacknowledgement that the managementstructure/processitsupportsistooheavyandexpensive to maintain for the long term.However, as related by one UNICEF staffmember, theNRTMS is generatingmore locallevelpolicyleveldecisions.Asexamples:

• In ChiredziDistrict, a policy has beendrafted that states all interventions

Sub-questionsconsideredforthisevaluationquestion:• IsthereevidencetosuggestthatgainsmadethroughtheInitiativewillbesustainedand/or

scalednationally?

MultisectorSupport“Whenwe started, therewas a resistance from sectors,saying‘whystunting’?Asifitsbeingdumpedonthem;itwasn’tontheiragendaandwhyshouldtheycare.Now,weareatastagewherestuntinghasbecomeglamourousandeveryonewantstojumponthebandwagon,whetheritisFAO or WFP or Ministry of Agriculture, or others. And,now, being led by [Office of the President and Cabinet].Now,everyonewantstoshowwhattheyhavedoneinthisarea.Thistellsastoryabouthowwe’vestartedtomakeadifferencewithNRTMS.”–Zimbabwerespondent



mustuseNRTMSgenerateddataastheirbaseline(i.e.nootherbaselineworkwillbeapproved).• Atthevillagelevel,asrelatedbyseveralinterviewees,theNRTMSimplementationhasspurred

policymakingbyChiefsrelatedtogivingbirthathealthfacilities.Thosethatchoosetogivebirthathomearefined(e.g.goatorchicken).

• AstoldbyaUNICEFinterviewee,aplanningandreviewmeetingbytheOfficeofthePresidentandCabinetgeneratedacommitmentthattheFNCs,whichwerecreatedbytheMSCBARSprogramme,shouldbecomeapartofgovernmentmandatedprovincialdevelopmentcommittees.This,inturn,createsade-factomandatefortheNRTMSdatatobeusedindecisionmaking.

Finding14:ThepotentialforbothsustainabilityandscalabilityareincreasedwhenUNICEF’ssupport

focusesonenhancingexistingnationalplatformsand,toalesserdegree,developingonlineplatforms.

AkeyfocusofthePMRInitiativeistomovefrom“piloting”innovationsforhealthsystemsstrengtheningtothatoftruescaleandsustainability.UNICEFhasmadegoodonthiscommitmentbychoosingtofocusoninterventionsthatenhancenationalplatformsandaninterventionthatcreatedanonlineplatform.EnhancingnationalplatformsIn Kenya, Swaziland and Uganda, the PMR Initiative designed and developed enhancements that areembeddedineachcountry’sestablishednationalHMIS.Bysodoing,allgamesrealizedbytheinitiativecanbeimmediatelyscalednationallytoallusersoftheHMIS.Inthebestcase,thisisalreadybeingseeninUganda,wheretheS/B/Aprocesshasbeenrolledoutnationally.AsimilarpotentialexistsinbothKenyaandSwaziland.OnlineplatformsHey similarpotential for immediate scalability canbe found inZimbabwe.TheNRTMS, as anonlineplatform,isavailableacrosstheentirecountrytoanyonewithanInternetconnection.Whilethesystemis now becoming of the de facto information system for the FNC, it is the establishment of a newinformationsystemwithinthegovernment.Assuch,theinstitutionalizationandlongtermsupportfortheNRTMS–atthegovernmentlevel–remainstentativeatbest.

Finding 15: The PMR Initiative has seen the beginnings of a critical shift in organizational and

professionalcultureintheimplementingcountries.Thisshiftenhancessustainabilitybycreatingthe

professionalexpectationofusingdataformanagementplanninganddecisionmaking.

Asnotedabove,anddetailedinthefouraccompanyingcountryreports,theevaluationfindsthatthePMRInitiative’smost important gains are found in its contribution to the organizational and professionalcultureshift ineachofthefourcountries.Thisshiftenhancessustainabilitybycreatingaprofessionalexpectationinmanagersofbeingabletousehighquality,reliabledataforplanninganddecisionmaking.Thisexpectationexistsatevery level in thehealthsystem, the furtherstrengthening thepotential forpolicycreationandresourcemobilizationwhereitiscurrentlylacking(Kenya,SwazilandandZimbabwe),andpolicyenforcementwhereitalreadyexists(Uganda).

Finding16:ThesustainabilityandscalabilityofgainsrealizedbythePMRInitiativecouldbefurther

enhancedthroughgreatergovernmentpartnercontributionandbroadeningthedirectionalityofscale.

ThePMRinitiativehasintentionallyfocuseditseffortsondecentralizedbodieswithinthehealthsystemof the four implementing countries. By broadening the directionality of scale and increasing thecontribution of government partners, UNICEF could further enhance the gains of a similar, futureinitiative.Greatergovernmentpartnercontribution.UNICEFisnotanimplementingorganization,ratheritworksthroughpartnersforallprogramminglikethePMRInitiative.Themostcriticalpartneristhegovernmentitselfatalllevels.Asconceptshavebeenproven and rolled out successfully, further sustainability and scale could be achieved through thegovernment’swillingnesstotakeonmoreriskforcontinuedenhancement,developmentanddepth.



ItisalsoimportantthatresourcescanbesecuredforthecurrentgainsmadebythePMRinitiativetoscale–within the existing pilot counties – but moreimportantlyhorizontallytoothercounties.Atthetimeof writing, there are counties which have showninterestintheprogram(butadmittedlywhosestateoffundingisnotfullyknown).InKenyaPMRinitiativeitiscurrentlyworkingupin2out of 47 counties. In Swaziland, the initiative took

placein4regions(12outof287facilitiescountrywide).Theabovequotethereforepointstothefactthatdespitethegainsmade,theresultsarewouldneedmorecoverage.ThisstillrequirestheexpertiseandskillsofanorganizationlikeUNICEFtostillprovideleadershipinatransitionperiodduringthescaleupwhereitisabletopartnerwithMoHparticularlytodemonstrateresultstotheMinistryofFinance.Thekeytosuccessasindicatedaboveishavingaclearlydocumentedandphasedapproachintransitioningcomponentsoftheprogramtothegovernment.Directionalityofscale.UNICEF’sfocusonscale,asitrelatestothePMRInitiative,isessentiallyhorizontal.Thatis,whenaconceptorenhancementisprovenitsvalue,thefocusisonhowthatgain canbemadeavailablenationally toother regions (oreven countries) through the organizational bodies andindividualswhoneedaccess(e.g. thedashboardsinHMIS)forsuccess.Butthisevaluationfindsanotherdimensionofscalewhichalso needs to be addressed for future endeavours to besuccessful;howtoalsoscaleenhancementsverticallywithinthehealthsystem.

Finding17:ThedataforHMISsystemsinESARoriginatefromvolunteerand/orlowpaidworkers,which

presentsarisktotheoveralldatachain.

Whilenotadefinedportionofthisassignment’sanalyticalframework,theevaluationteamnotedthatinallfourcountries,datathatwasinputintoHMISsystemsalmostexclusivelyoriginatedfromtheworkofvolunteerorlow-paidworkers.Specifically:

• Kenya:CHVsandCHAsaretheoriginationpointforHMISdata.• Swaziland:FacilitiesaretheoriginationpointforHMISdata.• Uganda:FacilitiesaretheoriginationpointforHMISdata.• Zimbabwe:VHW’saretheoriginationpointforthemajorityofNRTMSdata.

It can be argued (as it was during the KIIs with respondents from the University of Nairobi) that PMR in Siaya and Homa bay has addressed totally the first component of data availability especially for the health worker in the frontline. They are not paid properly and these are tools motivate them because they lessen reporting burden and make their work easier.However, still, typically volunteer and low-wage workers have high turn-over rates and competingintereststodatacollectionactivities.Anotherproblemwitnessed(inHomabayandSiaya,Kenya)isthatprivatefacilities(bothprivatelyownedorfaith-basedhealthinstitutions)oftenhavehigherstaffturn-overthaningovernmentownedfacilities,20andwhichimpactstheflowofdataintothenationalHMIS.

20TheHomabaycountygovernmentinKenyasecondedMoHstafftothe2largestfaithbasedinstitutionsinthecountytoaddressproblemofcontinuousdatareporting.

ScalingDimensions

“At county level,whoowns thedata?…. Ithinkthisisclearoncommunityhealth.Butat the national level, there hasn’t beenengagementthatcreatesownership,thatis,which unit owns the review process, thedata and promotion of its use.” – Kenyarespondent

OnScaling

“We know that resources are inadequate, but itwill not be enough to just have one countyworkingwell…forever…doingthesamething.Ifitisdoingwell,thenlet’smoveandscaleitupsothateverybodyelsebenefits…as longas resultsaretoogoodtokeepdoingit…butthenweneedtohaveameetingwith thehigher leadership toindicate where we started, where we havereached,theresults…”–Kenyarespondent



Whilethisassignmentdidnotspecificallystudythispoint,intervieweesatthecommunity/village/facilitylevelconsistentlyusedtheinterviewopportunitytorequestmoresupportandresourcesfortheirwork.ItalsopointstootherchallengesinthecountryhealthsystemswhichmightberequiringurgentattentionfromtheMoH(andpossiblycompetingforattentionandinterventionatthecentralMoHlevelbecausetheyarebetterarticulatedintermsofcostsorotherwise).

Replicability

Finding18:ThePMRInitiativehascreatedinformationproducts(i.e.dashboards)andprocesses(i.e.

S/B/Aprocess)outputsthatcanbereplicatedinothercontextwithsimilarfoundations.

An important part of the PMR Initiative seeks to generate solutions that are applicable beyond thecountriesofimplementationanddeterminehowUNICEFmightreplicatesimilargainsinothercountriesand in other sectors. The evaluation team finds three outputs from the PMR Initiative that could betransferredwithvaryingdegreesofdifficulty.

DashboardsbuiltinDHIS2

DHIS2iscurrentlyusedastheprimaryhealthinformationsystemin47countriesacrossfourcontinents(30oftheseareinAfrica).21Becauseofitsopensourcenature,anyimprovementstotheDHIS2systemmadebycontributingmemberofthedevelopmentcommunitycanbemadeavailabletotheentireuserbase.ThisholdstrueforthedevelopmentoftheRMNCAH(andtheforthcomingBNAandActiontracker)dashboardsfromthePMRInitiative.Customizationofthedashboardswouldneedtobeperformedonacasebycasebasisrelatedtoindicators,useraccessandother,minor,functionalities.Otherconsiderationsthatwouldneedtobemadeinothercountriesrelatetothecollectionandinputofdatanecessarytoruntheanalyticaltools.Forexample,inUganda,apaperbasedsystemofprimarydatacollectionisstillused.ThisdataisenteredintotheDHIS2systembyhandatthedistrictlevel.InKenya,however, a decisionwas taken to use phones to capture data digitally, eliminated the need for paperregisters and physical transport, but adding in other considerations of physical devices and dataconnectioncosts.InthecaseofSwaziland,dashboardsdevelopedfortheInitiativearebuiltintoaCMISthatisbasedonaproprietaryMicrosofttechnologyplatform.Assuch,thesedashboardscanonlybetransferredtoanothercontextthatareusingthisplatform.

S/B/AProcess

The scorecard, bottleneck analysis and action planning process UNICEF has developed and used intandemwithtechnologicalenhancements is immediatelytransferrabletoanothercontext. Indeed, theS/B/Aprocessisnottechnology,orsystem,dependentand,assuch,canbetransferredtoanycontextwithavailabledata.Thisisnotlimitedtootherhealthsystemsandcouldbeconsideredforothersectors,especiallythosethataredatarich.

OnlineNRTMS

Asstatedearlier,theNRTMSisanonlinesystembuildonopensourcetechnologies.Assuch,itnotonlycan scalewithin Zimbabwe to anyonewith an internet connection, but the system – as is - could bereplicatedwithoutterribledifficultyinothercountriesthatareexperiencingstuntingasamajorissue. 21Forthelatestdistributionfiguressee:https://www.dhis2.org/inaction

Sub-questionsconsideredforthisevaluationquestion:• DoesUNICEFhaveacomparativeadvantagerelativetootherstakeholderstoreplicatethistype

ofinstitutionalsupportinothercountries?• Ifyes,doesitexistequallyforallfiveoftheprogrammaticpillars?



However,tobecompletelyrealistic,theNRTMSwouldneedtobelocalizedandcontextualizedforeverynewinstallationtoaccountfororganizationalandpoliticalstructures,sectors,indicatorsandmore.Morelikelythattransferringthetechnologyitselfisthepossibilityoftransferringtheknow-howofbuildingsuchaplatformtootherUNICEFCOs.

Finding19:Theevaluationcannotdetermine ifUNICEFhasacomparativeadvantage in replicating

technology-based solutions. However, UNICEF’s programming across many sectors suggests a

competitiveadvantageexiststhatcouldbeexploited.

Thelimitationsoftheevaluationpreventtheabilitytodetermine,amongthemanyoptionsavailabletoUNICEF, what is themost optimal use of UNICEF resources for the highest return (i.e. better healthoutcomesforwomenandchildren).TheaccompanyingcountryreportsprovidecasebycasedetailaboutthecompetitiveadvantageUNICEFcurrentlyenjoysinpublichealthcaretechnologydevelopmentspace.TheresponsesofoneintervieweecapturewellthegeneralsentimentofgovernmentpartnersstatingthatUNICEF,asapartofthebroaderUNfamily,isapreferredpartnerbecausethey,“areadevelopmentpartnerwithwhomwespeakthesamelanguage,”they,“understandthegovernmentwayofdoingthings,”and,“theyarenotsorigid,notfocusedon a particular organizational/programme outcome.” The way UNICEF worked in partnership withgovernmenthasbeencontrastedwithotherdevelopmentpartners,whousuallyimplementprogrammingthatistiedtoaspecificgoaloragenda(e.g.HIV/AIDS,Malaria).RegardingtheS/B/Aprocess,whileUNICEF’ssupporthasbeenwellreceivedacrossKenya,SwazilandandUganda,itisfarfromauniqueoffering.TheevaluationteamwerepresentedwithsimilarscorecardstotheRMNCAHduringfieldvisitsanddistrictlevelstafffromnon-interventionareaswereverseintheiruse for analysis and planning. This indicates that other implementing partner use and teach similarmethods.

LessonsLearnedThefollowinglessonstobelearnedfromthePMRInitiativewereidentifiedduringtheevaluationprocess:Leveragingexistingnationalprogrammes,processesandsystems(i.e.HIMS)improvesbuyin,acceleratesdevelopmentandadoptionandenhancessustainability.Softwareandtechnologyarelivingprocesses.Thereisoftenanunderlyingorunspokenassumptionthatnewlycreatedoradoptedtechnologieswillofferasolutiontoaproblem.Thetruthofthematteristhattechnologyisonlyanenablerforsolvinganissueinadifferent,andoftenmoreefficientmanner.Assuchtechnologyshouldbeadoptedwiththeunderstandingthatitisalong-terminvestmentthatwillrequireconstantcareandfeeding.Usingtechnologyforfeedbackisalaudablegoalandonethatshouldbepursuedgivenglobaltrendsinconnectivityandaffordability.Atthesametime,inregionssuchasESAR,acultureofcommunicationthatis technology focused remains a significant hurdle to overcome and programming support for targetpopulationsshouldreflectthis.ThePMRInitiativeisalreadyshowingthatbetterdataandbettermanagementanalysisofthatdataleadstoidentificationofproblemsatthelocallevel.However,thisprocessmustalsoincludemechanismsforclosingthefeedbackloopwitharticulatedsolutionsfromdecisionmakers.Themanagementbydata culture, encouragedexplicitlyby thePMR Initiative, is infectiousbecause itempowers local managers in real ways and allows for individualized and nuanced funding needsproposals.However, future initiatives that adopt this typeof support should ensure themanagementreview/analysis process of data always results in clear, reasonable action items that are ultimately“owned”byoneindividual.



UNICEF’s existing corporate culture is deeply ingrained across a decentralized organization. This ispowerfulandrequirestimeandclear,consistentleadershipmessagingtoevolve,especiallywithregardstothesharingofinformation.Whendevelopingnationalplansofactionforadoptingatechnology-enabledsolution,scaledoesn’thavetomeannationalorglobal.Technologyischeapenoughthatitcanbelocalizedforspecificneedsandstillprovidethereturnandefficienciesdesired.ThebiggerhurdleforICT4Dinitiativesisuseradoptionandaprofessionalculturethatexpecttechnologytobeatthecentreofindividualskills,rolesandpractices.

ConclusionsTheconclusionsbelowflowdirectlyfromthefindingsstatedabove:

Conclusion1:WhiletheavailabilityofmorerealtimedatahasplayedakeyroleinthePMRInitiative,

amorecriticaloutcomeistheInitiative’scontributiontothedevelopmentofa“managementusing

data”cultureinESAR.(Basedonfindings1,2,6,7,9,14and15)

AtthecoreofthePMRInitiativewasthedesiretoensurethatmanagersindecentralizedstructuresdataavailabletothem.WiththeexceptionofZimbabwe,whichbuiltaninformationmanagementsystemfromscratch,thisappearstohavealreadybeentruepriortotheInitiativeviaexistingHMIS/CIMS.Assuch,theevaluation concludes that UNICEF COs rightly emphasized data use in their contextualizedimplementations. This is notable especially given the fact, in all cases,managerswereworkingwithincomplete data sets. Said another way, UNICEF and its partners were willing to embrace a morecommercial/classically entrepreneurial position of taking decisions, monitoring the results of thosedecisionsandadjustingaccordingly.

Conclusion2:UNICEF’s competitive advantage lies in its strategic and long-term relationshipswith

governmentpartnersanditsabilitytomanageITenabledprogramming(Basedonfindings3,5,and

19)

ThecompetitiveadvantagesidentifiedinthefindingsabovesuggestthatUNICEFhasthepotentialtocraftauniquepositioninthemarketforhealthsystemsstrengthening.Thisadvantagecouldalsopotentiallybe used in other sectors that exhibit similar characteristics and are data rich (e.g. education,environment).Inparticular,UNICEF’sreputationasamanagerofprogrammingthatcontains,orreliesupon,enablingtechnologyhasbeenestablishedandcanbeactedupon.

Conclusion 3:UNICEF comparative advantages, if any, related to implemented technology enabled

programmingremainunknown.(3,5and19)

ThefindingsfromtheevaluationindicatethatanycomparativeadvantageUNICEFenjoysinthehealthsystemsstrengtheningspaceremainunknownintermsofhow$1USDultimatelyresultsinsomeresultofbetterchildhealthoutcomes.

Conclusion4:ThePMRInitiative’sattemptstoengagecitizenfeedbackhaveproducedpositiveresults,

buthavenotyet inspiredanewnormforhowcitizen feedback isgathered,utilizedandmanaged.

(BasedonFindings4,5,7,8and10)

Theevaluationfindsthatcitizensareinterestedinparticipatinginhealthsystemsstrengtheningandthatmanagersareinterestedinbothcollectingandusingthisfeedbacktotheiradvantage.However,feedbackisstillreceivedviamostlytraditionalchannelsandUNICEF’sattemptstointroducetechnology-enabledfeedbackpatternshasnotyetgainedtractioninawaythatinspiresmanagersorcitizenstochange.



Conclusion 5: The PMR Initiative has realized typical results in terms of knowledge exchange and

contributedtothedevelopmentofnewtoolsandchannelsforthesame.However,thereisstillsome

distance togo toachievea trueparadigmshiftwithinUNICEF forKnowledgeExchange. (Basedon

Findings9and10)

ThePMR Initiative emphasizedKEas oneof its intermediate outcomepillars and theprogramme, ingeneral,sawextensivesharingofinformationhappenwithincountriesandacrosstheregion.However,the methods and channels used for KE were traditional in nature: within countries, face to facecommunicationwasemphasizedand longestablishplatforms, suchasemailor in-personworkshops,werepreferredinUNICEF.WhilethenewKEplatformsdevelopedbyUNICEFasapartofthisinitiative(i.e.onYammerandSharepoint)didnotgaintractionwithcountrystafforgovernmentpartners,otheroutlets,suchaswebinarsandSkypemeetingswereseenfavourably.

Conclusion6:Thescalabilityandsustainabilityofprogramminggainsremainsahigh-qualitygoal.The

PMRInitiativeoffersthreeuniquepathsthatcanbeusedasmodelsforbothlargescalereleasesand

continuedrefinementofthistopicwithinUNICEF.(BasedonFindings11,12,13,and14)

ExceptforZimbabwe,thefocusofthePMRInitiativewasinvestinginexisting,governmentowned,HMIStocreateorenhancereportingandanalysis.Assuch,thegainsrealizedfromtheInitiativewithinthesesystemshavethepotentialtobescaledorreplicatedinother,similarsettings.InthecaseofZimbabwe,becausetheNRTMSiswebbasedandopensource, ittoocanbescalednationally,butreplicatingit inothermarketsorforothercountrieswouldrequirestart-from-scratchtypesofcustomization.

Conclusion7:ThePMRInitiativehashighlightedcriticalopportunitiesforinnovationatthebeginning

andendofthefeedbackloopinhealthsystemstrengthening.(BasedonFindings8and19)

UNICEFhasshownthatinvestingatthemanageriallevelforhealthsystemstrengtheninghasimmediateand high value through results such as efficiency gains and access to analysis for non-technical staff.However,theevaluationhasalsohighlightedaclearneedatboththebeginningofthedatachain–wherePMRprogrammesarereliantonvolunteeror lowpaidstaff fordatacollection–andattheendofthefeedbackloop–whenpoorlyperformingindicatorsareidentified,bymobilizingnecessaryresourcesforasolution.



RecommendationsTherecommendationsbelowflowdirectlyfromtheconclusionsstatedabove.Theevaluationteamsuggestprioritisingrecommendationsalongtheordinalstructurebelow:

Recommendation1:Continuetoinvestinthe“ManagementUsingData”cultureinESARby

leveragingUNICEF’suniquerelationshipwithgovernmentpartnersforlong-terminvestment

plansandreplicatinggainswithinhealthcareandinothersectors.(BasedonConclusion1and

2)

Undoubtedly,theclearestgainsinallfourcountriesmadebythePMRInitiativearefoundinmanagersusingcollecteddatatounderstandhowthey,andtheirfacilities,areperforming.UNICEF,anditspartners,should continue to support the building of a robustmanagement culturewhere individuals not onlydemand data for their management decisions, but also one where trust in data is the norm andunderstandingofdataisexpectedatalllevels.Potentialdirectionsthatcouldbetakeninsubsequentprogramminginclude:

• Build capacity around short, medium and long-term planning and strategy. For managers tocontinuetobeexcitedaboutdata,tangibleresultsfromusingdatawillneedtohappenintheshortterm.To thatend,UNICEFshould seek toworkwithmanagers todevelopshort,mediumandlonger-termplanning and strategies that allowmanagers to have quickwins that roll up intogreatergainsoverthelongterm.

• Seektoincentivizeindividualsorteamsarounddatauseandanalysis.SimilartothescorecardapproachforRMNCAH,theremaybeotheropportunitiesforacompetitivesystemthatcomparesfacilityordistrictperformancetocontinuefeedintoemotionalattachmenttodata.

• Providesimpleandtransparentmeansformanagerstoidentifyandreportondataqualityissues.Makesurethissystemalsoincludesclearownerswithauthoritytotakeaction,accountability,andafeedbackmechanismthatmakesiteasytoseewhatactionshavebeentakentoremedyissues.

• Establishaprofessionalenvironmentwherefailureistoleratedandexpected,aslongaslearningandadaptationalsohappens–noteverydecisionbasedondatawillresultinpositiveoutcomes,oranytangibleoutcomes.

• Encourage customization and creative solutions frommanagers based on existing and futureautomatedanalytical tools.As theseuniquesolutionsbegin toemerge, facilitateaplatformormethodforsharinginthecountry,regionallyandglobally.

• Seek ways to make digital analytical tools available to all levels of the health care system,especiallynon-technicalmanagers.

• Provideopportunitiesformembersofthehealthsectortodemonstrategainsandoutcomestoothersectorsandtootherareasofthehealthcaresystem.

Recommendation 2: BolsterUNICEF’s competitive advantage as amanager of enabling ICT

solutionsbyinvestinginternallyinskilledinformationmanagers.(BasedonConclusion2)

UNICEFhasestablisheditselfasaleaderinmanagingprogramsthatrelyuponICTfunctionalitytoenabletherealizationofexpectedoutcomesforchildren.However,thisisnotauniqueadvantageandonethatiscovetedbymanyotherorganizationsacrossthesocialsector.UNICEFshouldcontinuetodifferentiateitself by investing in individuals who are skilled informationmanagers. These individuals should betaskedwith answering thequestion, “How could [insert programmename]be enhanced,mademoreefficient,scaleorbemoresustainablethroughbetterdataandsubsequentanalysis?”Whileanunderstandingoftechnicalareas(e.g.Education,Protection,Health,WASH)areimportant,theexistingskillsetsoftheseindividualsshouldfirstemphasize:

• Apassionaboutdata,analyticsand theirultimateusefulness inachievingbetteroutcomes forchildren;



• An intrinsic understanding of ICTs and the willingness to play, bend, break and fix ICTs tounderstandtheirusefulness;

• A clear understanding of information management, including storage, accessibility,interoperability,systemsintegrationandmetadata;

• Atleastanappreciationfordatascienceandhowdatacanbepooled,dissectedandusedinuniqueways;

• Aprovenabilitytothinkacrosssectorsandsilos;• Aproventrackrecordinchartinglong-termvisionsassociatedwithclear,shorttermmilestones

thatproducetangiblebenefit;• Apersonalitythatiswillingandabletoconnectandpartnerwiththeprivatesector;• A personality who is able to forge new relationships, managing existing ones and build

communityarounddatasharing;and• Anindividualthat“gets”theopensourcemovement.

Importantly,whileICTsareoftenapartofthedata-enablingconversation,therealkeyhereisinformationmanagementandtheuseofinformationformanagementdecisionmaking.CreatingorincludinganICTcomponentisnotalwaysnecessaryandshouldbeapproachedwiththesamecareasanyotheraspectduringprogrammedesign.

Recommendation3:InvestinestablishingUNICEF’scomparativeadvantagetomakethecase

thatICTinvestmentsarethebestuseofdonorfundsforachievingbetterhealthoutcomesfor

children.(BasedonConclusion3)

Asnotedinthisreportandthecountryreports,theevaluationteamwasunabletodetermineifUNICEFhasacomparativeadvantagerelated toenabling ICTsolutions, improving the incorporationofcitizenfeedbackandscalingorreplicatingprogramming.Whilethis isnotaneasyquestiontoanswerbyanyestimation,answeringitismostlyafunctionofavailabletimeanddata.UNICEFshouldinvestinansweringthecomparativeadvantagequestionfortwoimportantreasons:first,understanding the optimalway to spenddonor funds to achieve better health outcomes for childrendirectlysupportsUNICEF’smandate.Second, if it ispossible toestablish that investing$1USD in ICTenabled programming delivered by UNICEF results in better, more or broader health outcomes forchildren than other alternatives, knowing this would be a powerful competitive advantage for theorganization.

Recommendation4:Continuetoinvestincollectingandrespondingtocitizenvoicebyiterating

currentsolutionsandseekingnewsolutionsfromalargercommunityofpractice.(Basedon

Conclusion4)

UNICEFhasbeenaleadingadvocateandpractitionerforincludingcitizenvoiceinprogrammedesignanddeliveryformanyyears.However,theattemptstoincludecitizenvoiceviaICTenabledsolutionsinthePMRinitiativegenerallyfellshortofexpectations,orinthecaseofKenya,wasnotused.Citizen feedback isessential forsuccess indevelopmentprogramming.To thatend,asnotedabove inRecommendation1,resultsfromthePMRInitiativeshouldbeseenasamomentofreflectionforUNICEFinordertolearnanditerate,sothatthenextprogrammingcyclemightattemptsomethingnew.Inthespiritofworkingtogethertocreategreaterefficienciescollectingandrespondingtocitizenvoice,UNICEFshouldshyawayfromgoingitalone.OneimmediateopportunitywouldbetoengageFeedbackLabs,22in

22Citizenvoicesremainanunderutilizedengineforglobalchange.Globally,thereisadisconnectbetweencitizens,serviceproviders,andgovernments.Asaresult,opportunitiestomakeadjustments,targetservicestoneedsontheground, and adapt to changing conditions are unrealized. Feedback Labs aim to set the stage for a improvedecosystem that links citizen feedback with service providers and governments to create a force multiplier—rewardingwhatworks, adaptingand learning fromwhatdoesn’t, andensuring themost targeted, creative, and



oneoftheirLabStorms,andusethiscommunityofmorethan300organizationsasacatalystforcrackingthecodetogatheringandengagingwithfeedbackfromruralcommunities.Asanaside,atthetimeofthisevaluation, thecitizenfeedbackcomponent inZimbabwewasonly justbeginningtogaintraction.UNICEFshouldmonitorthisprocess(basedonUReport)asitisdifferentthanthoseusedintheotherthreecountries.Theremaybeadditionalsuccessesandchallengesthatcanbelearned.

Recommendation5:ContinuetoevolveknowledgeexchangeatUNICEFthroughchangesin

organizationalstructure,professionalcultureandleadershipbuyin.(BasedonConclusion5)

TheevaluationfoundthatthePMRInitiativefellshortintermsoffindingnew,uniqueorscalablemeansforsharingknowledge.However,gainsweremadeand,similartocitizenvoiceabove,thisshouldbeseenasanopportunityforreflection,iterationandnewdesign.ThreeareasUNICEFcanconcentrateonintheneartermtoimproveitsKnowledgeExchangepracticeare:Organizationalstructure:UNICEF’sdecentralizedorganizationalstructurecreatesnaturalboundariesbetweenCOs,withinregionsandglobally.ThesephysicalboundariesarefurthercompoundedthroughthedivisionofUNICEF’sworkintoseparatesectors.Whileitisbeyondthescopeofthisevaluation,itisclearthatthisstructureprovideslittletonoincentiveforUNICEFstafforteamstoregularlyandreliablyshareinformation.Whenfundingstreamsareaddedtothisequation,incentivesarereducedfurther.UNICEFshouldseektofindincentiveoptionstobreakthesebarriers.Professionalculture:UNICEF’sprofessionalculturedoesnotcurrentlyreliablyrewardrisktaking.UNICEFshouldpursuemessagingandaclearincentivestructurethatrewardsrisktakingwithregardtoinformationsharinginordertoestablishanewnormregardingexpectationsaboutwhathappenswiththelessonedlearnedineachCO.Leadership:Underlyingbothoftheaboveistheneedforclear,strong,consistentandsustainedleadershiponknowledgeexchange.IftheorganizationanditsstaffdonotperceivethatKEisahighpriority,withincentivesforgoodperformanceandlimitationsforpoorperformance,changewillnothappen.Thisleadershippositionshouldcomefromthehighestlevelpossibleandbetiedtooneortwo,simple,cleargoals.Communicationaboutleadershipexpectationsshouldbecommunicatedateveryopportunityandoverasustainedperiod(2-3years)inorderforthemessagetobecomeapartoftheUNICEFculture.

Recommendation6:SeektoshiftUNICEFstaffandpartnermindsetsabouttransferabilityof

ICTenabledprogramming,andknowledgeexchange,from“plugandplay”to“customization.”

(BasedonConclusions5and6)

ThereareclearopportunitiesforgainsmadeinthePMRInitiativetobereplicatedinothercontext,inboththehealthsectorandothersectors.UNICEFshouldensurethatthefollowingtwocriteriaarepresentinanynewcontextforreplicationtobesuccessful.First,ensureanyreceivingcontexthasaminimumenablingenvironment.Thisenvironmentincludes:

10. Nationalownershipandleadershipofthesystem(s)thatwillreceivetheintervention;11. Nationalsystemsthatareownedandoperatedbythegovernmentthatcanbeenhanced;12. Theabilitytopartnerwithgovernmentandcivilsociety;13. Theabilitytofosterdevelopmentandimprovementofinfrastructureandpolicy;14. Theabilitytoalignprogrammaticgoalswithaccountabilityframeworks,incentivesandactions;

lastingsolutions.FeedbackLabsaimstochangethenormsindevelopment,aid,andphilanthropicpolicytobemoreresponsivetothepeoplethatthosepoliciesaimtohelp;bycatalysingthewidespreadadoptionofclosedfeedbackloops in philanthropy, international aid, and governance with respect to the voice of citizens. Seehttp://www.feedbacklabs.org



15. Expertisethatcanbesystematicaboutadoptingandscalinginnovationsthatareeffective;16. TheabilitytoinvestinadequateICTcapacity;17. Along-termvisionthatallowssufficienttimeandinvestmentforactionandimpact;and18. Theabilitytofocusonshorttermmilestonesalongalonger-termroadmap.

Second,ensurethatUNICEFstaffandpartnersunderstandthatreplicationofanyICTenabledprocessorsystem will, by its nature, require customization. As noted in the Uganda country report, there is atendency for non-technical UNICEF staff and partners to desire systems and processes that aretransferrable as “plug andplay.”However, even systems that are built on awidely used, open sourceplatform such asDHIS2will require customization. As an example, consider theRMNCAH scorecard.Whiletheunderlyingmethodologyforproducingthisscorecardremainsthesameacrossallcontext,ataminimum,theindicatorschosenandtheirrespectivecriteria(e.g.denominator)willneedtobechanged.And,therefore,thefunctionalityoftheICTsystemwillneedadaptation.DevelopingthecapacitytoestablishanenablingenvironmentandthenofferthecustomizedICTenabledsolutionforthatenvironmentofferanotheropportunityforgreatercompetitivenessatUNICEF.AdoptingacustomizationmindsetalsoappliestotheKnowledgeExchangeapproachatUNICEF.Forexample,whilesettingasinglestandardcapforthegroupsizeswouldbeunrealistic,UNICEFshouldconsidersmallergroupsizesinYammerthattargetmorespecificpopulations.23Bysodoing,UNICEFcouldcreategroupsthataremoremanageableintermsofengagementbyfocusonaparticularresultareaorcompetency.Slackisoneexampleofasuccessfulapplicationofthisparadigmwhereusershavetheabilitytosetupchannelsasnecessaryforalmostanyparticulartopicandinviteusersasnecessary.FurthercustomizationforKEcouldbeemployedwithSharepoint.Thisrepositoryshouldhavearequirementforallinformationtohaveassociatedmetadatathatenablesfasterandmoreaccuratesearching.

Recommendation7:UseICTenabledprogrammingasabridgeformultisectoralprogramming

andscalingvertically.(BasedonConclusion6)

TheimplementationofthePMRInitiativeinZimbabwewasclearlyunique,ascomparedtoKenya,SwazilandandUganda,andthishasledtoitsownsetofoutcomes.OneclearlypositivegainfromtheZimbabweexperienceisthemultisectoralnatureoftheimplementation,whichisincontrasttothehealthsectoronlyimplementationoftheotherthreecountries.UNICEFshouldcontinuetopursuesimilaropportunitieswhereICTenabledprogrammingcanovercomebarriers,breaksilosorbridgegapsbetweensectorsforamoreholisticandintegratedapproachtoachievingbetteroutcomesforchildren.Inasimilarvein,UNICEFshouldseektoleverageitssuccessinverticallyscalingtheprogrammingofthePMRInitiativetoachievethesamehorizontally.

Recommendation8:CapitalizeonopportunitiesexposedbythesuccessofthePMRInitiative

inclosingthefeedbacklooptoaddressthe“noavailableresource”challengeandensuringdata

collectionissustainable.(BasedonConclusion7)

UNICEFshouldimmediatelyseektocapitalizeonthetwomajorchallengesexposedbythesuccessofthePMRInitiative:Closingthefeedbacklooptoaddressthe“noavailableresource”challengeandensuringdatacollectionissustainable.Aswithotherareasmentionedabove,“solving”theseproblemscouldcreatesignificantcompetitiveadvantagesforUNICEF.

23Forexample,asof28/05/2017,thesizeoftheRTMCoPwas144.



Oneoptionforclosingthefeedbackloopcouldbethecreationofabasketoffundsthatcanbeusedtotargetidentifiedproblemareasinpartnershipwithlocalgovernments.Toensuresustainabilityofdatacollection, UNICEF could work with partners to establish processes for data collection that offers asustainableandworth-whilesystemofincentives.



Annexes

Listofpeopleinterviewed

Kenya

Givenname Familyname Title OrganizationVictor Achieng M&ESpecialist, UNICEFEunice Ndungu Maternal-NewbornHealth

PartnershipsandM&EOfficeratUNICEF

UNICEF

Elijah Asadhi MonitoringOfficer UNICEFDr.David Soti Head.Dept.Preventiveand

PromotiveHealthServicesMOH

Dr.Hellen Kiarie ActingHead-M&EUnit,Div.MEHRDHI

MOH

Dr.Peter Cherutich Head.Div.MEHRDHI MOHNancy Amayo TechnicalOfficer,Health

InformationUnit,Div.MEHRDHI(DrivingRMNCAHScore-cardAgendaforHealthSector)

MOH

Dr.Dan Orwa TeamLead UniversityofNairobi-SchoolofComputingandInformatics

Raphael Pundo ProgramDirector AuraSafiraConsulting

Dr.Elijah Abade SystemsArchitect UniversityofNairobi-SchoolofComputingandInformatics

Dr.Samuel Omondi CountyDirectorofHealth SiayaCountyHealthManagementteam

Dr.Gordon Okomo CountyDirectorofHealth Homa-BayCountyHealthManagementteam

Swaziland

Givenname Familyname Title OrganizationNelisiwe Dlamini M&ESpecialist UNICEFNontobeko Mbuyane Innovationsconsultant UNICEFSamkelo Zulu DataAnalystConsultant UNICEF

Zanela Simelane HMISManager MinistryofHealth

Fortune Mhlanga HMISSeniorStatistician(Gatesinitiativeprojectmanager)

MinistryofHealth



Thabang Masangane QualityAssuranceCoordinator MinistryofHealth

Sikhumbuzo Dlamini NetworkEngineer MinistryofHealth

Kevin Sikwebele CEO IHMAfricaBheki Mamba ProgramManager IHMAfricaSicelo Kunene SoftwareDeveloper IHMAfrica

YouthGroups

BoyScoutsAssociationofSwaziland

SiteVisits

PiggsPeakGovernmentHospital(HhohhoRegion)

RegionalHealthManagementTeam MinistryofHealth

CanaAllianceClinic(ManziniRegion).

HealthManagementTeamFacilitylevels MinistryofHealth

Uganda

Givenname Familyname Title Organization

Jennifer Ajiambo HealthcareworkerSetaNazigoHealthCentreIIIMukono

Prosper Behumbiize DHIS2Implementation HISP

Viorica BerdagaChiefChildSurvival&Development UNICEF

Sean Blaschke HealthSpecialist UNICEFDr.Sarah Byakika CommissionerPlanning MinistryofHealthEunice Esule RTMProjectManager UNICEFBaryarama Fulgentius DHIS2Implementation HISPDr.Sam Kamba SeniorPlanner MinistryofHealth

Hadija Katongole HealthcareworkerGombeHospitalButambalaDistrict

Sylvia Kissakye HealthcareworkerGombeHospitalButambalaDistrict

Faridaj Magyanzi HealthcareworkerGombeHospitalButambalaDistrict

Zakiiru Matovu HealthcareworkerGombeHospitalButambalaDistrict

Moses Mbuusi VolunteerDataentry MukonoDistrictHealthCentreCary McCormick ProgrammeSpecialist UNICEFFlavia Mpanga HealthSpecialist UNICEFMoses Mukundane ResearchAssociate ACODE



EricMunyambabazi DHIS2Implementation HISP

Sulaiman Mutamba HealthcareworkerGombeHospitalButambalaDistrict

Siraj Muyomba Biostatistian ButambalaDistrictHealthOfficeIvan Mwesigwa Biostatistian MukonoDistrictHealthCentreIvan Mwesigwa Biostatistian BuikweDistrictHealthCentreSusan Najjuko SeniorEconomist MinistryofHealthNoreen Prendiville Dep.Representative,Uganda UNICEFFatoumatta Sabally

ChiefPlanning,Monitoring&Evaluation UNICEF

Swaleh Sebina Economist MinistryofHealth

Patrick Ssenfuka HealthcareworkerKyabadazaHealthCentreIIIButambala

Elly Tumushabe DistrictHealthOfficer BuikweDistrictHealthCentre

Zimbabwe

Givenname Familyname Title OrganizationMiriam Barda MSCBMOfficer FNCMarinyame Bongai DFNSC WAGCDFreedom Chaike PlanInternationalPikitami Chanayiwa DFNSC DVSKunwana Chiezda YouthOfficer Ward22Rerai Chigayo CommitteeMember Ward22Raymond Chikomba DFNSC ZivaEnsureMiriam Chiliyo VHW Ward22Elizabeth Chithkiman CommitteeMember Ward22Tsikeyi Chokera VHW Ward22I. Gono MechanicalEngineer MechanisationEugenia Hanyan VHW Ward22Sabhuku Hasani Ward22F.P Jeke LivestockExtension LivestockProductionMishock Komundela Councillor Ward22Leonard Kungami LCCW Ward22Chuba Lavemore HOD ZimstatJuliet Lisongo VHW Ward22Moline Mabihani ViceChairperson Ward22T. Madzivanyika DFNSC CRDCC. Mahandar DFNSC AgritexHonest Mahlatini Nutiritionist MOHCCL. Mamungaire BCDO SMECDInnocent Mangwiro Analyst FNCNgoni Maniyika ChiefPolicyandPlanningOfficer MOPSLSW



Mildred Mapani PrincipalPolicyPlanningOfficer MOPSLSWJimmy Mapepa DFNSC EducationIsheunesu Matimbira Nutiritionist MOHCCYvonne Mavhunga FoodSecurityOfficer FNCC. Mazibne DFNSC DAOfficesJesca Mbnda DIHO MOHCCElizabeth Mhango VHW Ward22Loreen Mharapura AEW Ward22Ruramai Mpande ResearchOfficer MOPSEC. Muchayi DFNSC DAOfficesTatenda Mudiwa ProgrammeAssistant FNCE. Mugari AgriculturalExtensionSpecialist AgritexPullima Mukaro EHO CRDCKudzai Mukudoka-mudukuti Nutrition UNICEFKudzai Musasi ViceSecretary Ward22Alfa Ndlovu ProgrammeAssistant FNCD. Nechikwika DFNSC Vitalis Nhimba PlanInternationalC. Ntadzivanyika DFNSC MIEEPerpetual Nyaderga ProgrammeOfficer FNCD.H. Nzarayerani DFNSC AgritexMaria Salani Secretary Ward22Judith Shayamano EHO Ward22Chigora Siboniso ProgrammeOfficer FNCEsnath Siclhuna Nurse Ward22Mufudza Tafadzwa EHO MOHCCMativenga Tariro AES Ward22Christine Taswirei VillageChairperson Ward22Phillip Tizai AgriculturalExtensionSupoervisor AgritgpDhobha Tongai AEW Ward22Leonard Turugari NationalOfficer UNICEFHerbert Zvirenre Analyst FNC



InterviewGuides

KeyInformantInterviewGuideKeyinformantinterviewsaresemi-structured.Thefollowingisaguide,notaquestionnaire.Itisexpectedtobe more of a discussion than an ‘interview’. Some interviewees will know much more about the PMRprogrammethanothers.Someintervieweeswillhavemoreusefulinsightsthanothers.Theinterviewwillbeledbyan international ISGproject teammember.Each interviewwill lastapproximatelyonehour, thoughgroupinterviewsmayrequiremoretime.NOTE:Theinterview/discussionquestionswillbeprioritizedoradjustedtothespecificrespondentstobeinterviewedandwillbelocalizedtothecountrycontext.

ForeachKII theproject teammemberwillbeginwith introductionsandprovideabriefexplanationof thepurpose of the meeting/discussion, its emphasis on a participatory and learning oriented approach, andexpressappreciationfortherespondent’sparticipationintheinterview/discussion.

Relevance:

1) IsthePMRinitiativeanappropriateresponsetothiscountry’sneeds?

2) IsUNICEFtherightorganizationtoimplementthistypeofwork?Whyorwhynot?

3) DoesUNICEFhavethecapacity(staff,knowhow,funding,etc.)todothiswork?

Effectiveness:

1) WhathastrulychangedintermsofbeingabletoeffecthealthoutcomesinthiscountrybecauseofthePMRinitiative?

a. Askforspecificexamples

SustainabilityandScalability:

1) WhathappensnowthattheUNICEFPMRinitiativehasendedinthiscountry?

2) Havebudgetsbeenallocatedfortheinitiative(s)inthiscountrytocarryitforward?

3) Doesaworkplanexistfortheinitiativesinthiscountrytocarryitforward?

4) Arestaffandothernecessaryresourcesavailableinthiscountrytocarryitforward?

Replicability:

1) WhatisUNICEF’s“core”business?

2) WasthereabetterwaytoallocatethePMRinitiativefundingtoeffecthealthoutcomesforwomenandchildren?How?

3) DoesUNICEFhaveoneormoreparticularadvantagesoversimilarorganizations(IOs,iNGOs,Privatecompanies)intermsofbeingabletodeliverserviceslikethePMRinitiative?

a. Monitoringplatformsforrealtimedata?

b. Citizen/communityfeedbackplatforms?

c. Knowledgesharingplatforms?



Outcome1.1.–MoreRealTimeDataAvailable

1) Whatmonitoringplatform(s)wereestablishedorenhancedforthePMRinitiative?

2) Dothese(this)platformprovidereliable,relevantandmorerealtimedata?

3) What specific types of data have now been made available which weren’t reliably and routinelyavailablepriortotheinitiative?

4) Aredecisionsbeingregularlymadeusingthedata?(Getspecificexamples)

a. Whatistheprocessthatleadstodecisionmaking?

Outcome1.2–Citizen/Youth/Communityfeedback

1) Doesacitizen/communityfeedbackplatformexistorwasenhanced?Whichone?

2) Doesthisplatformprovidereliable,relevantandmorerealtimeinformation?

3) What specific types of data have now been made available which weren’t reliably and routinelyavailable prior to the initiative? (where possible, provide supporting documentation, pressreleases/interviews,etc.)

4) Whousestheplatformandwithwhatfrequency?(wherepossible,providesupportingdocumentation,pressreleases/interviews,etc.)

5) Aredecisionsbeingregularlymadeusingthedata?(Getspecificexamples)

a. Whatistheprocessthatleadstodecisionmaking?

Outcome1.3–Quarterlyperformancereviews

1) In which districts/sub-counties were activities ongoing? (where possible, provide supportingdocumentation,pressreleases/interviews,minutesofmeetingsetc.)

2) WhichdecentralizedbodiesweretargetedbythePMRinitiativeinthiscountry?

3) Whowerethetargeteddecisionmakers/bodies?

4) How,specifically,dotheneworenhancedplatforms(andtheirresultingreports)addvaluetotheirrespectivesystems,processesanddecisionmakers?(do

5) Doestheirexistenceresultinbettercare/protection/healthofwomenandchildren?(wherepossible,providesupportingdocumentation,pressreleases/interviews,etc.)

6) Whatdatawasavailablepriortotheinitiative?

7) What new data is now available? (where possible, provide supporting documentation, pressreleases/interviews,etc.)

8) Whatnewdatademandsarependingrightnow?

9) Canyouprovideexamplesofrecentandfuturedecisionsthatwillbeinfluencedbythenewlyavailabledata?

Outcome2.1–Knowledgeexchange

1) DoyouhaveaccesstoknowledgeplatformsforexchanginglearningaboutthePMRinitiative?

2) WhichKEplatformsexist,orhowwasanexistingoneenhanced?

3) HowhaveyouusedtheKEplatformduringthePMRinitiative?



4) WhatistheplanfortheKEplatformnowthatthePMRinitiativehasconcluded?

5) WhatcouldbedoneinthefuturebyUNICEFtostrengthenknowledgecaptureandsharingwithinasimilarprogramme,withintheregionandmorebroadly?

Outcome2.2–NationalPlansfordecentralizedmonitoring

1) WhatisthecurrentplantosustainandultimatelyscalethePMRinitiativeinthiscountry?

2) DonationalscaleupplansexistfortestedPMRmodels?

a. Ifyes,whatisthebudgetallocation?

b. Whatotherresourceshavebeenallocated?

MostSignificantChangeGuideEvaluatorswillexplaintoparticipants:

TheMostSignificantChangetechniqueisanalternativeevaluationmethodology,specificallysuitedforcomplexprogramming,thatseekstouncoverresultsandimpactthatwouldotherwisenotberevealedthroughothermethods.

IwouldlikeforyoutoconsiderthefollowingquestionasitrelatestoyourexperiencewiththePMRinitiativeandhowithascreatedchangeinthiscountry.Wewillrecordyourstoriesandtheywillbesharedwithothers ina two-stepprocess todeterminehowvaluehasbeencreatedviathePMRinitiative.

Question:LookingbackovertheentirePMRinitiative,ifyouhadtosummarizewhatyouthinkwasthemost significant change in this countrybecauseof theprogram ina single story…whatwouldthatstorybe?



TORforassignment

Title:End-of-programmeevaluationofUNICEF'sregionalInstitutionalStrengtheningSupportInitiativeondecentralizedProgrammeMonitoringandResponse(PMR)Location:Deskreviewwithpre-fieldinterviewsinNairobiandfieldvisitstoKenya,Swaziland,UgandaandZimbabweDuration:3monthsBackgroundandJustificationThe twenty-one countries ofUNICEF's Eastern and SouthernAfrica region (ESAR) collectively bear asignificantproportionofglobalunder-fivemortality,malnutritionandHIVdespitenotableimprovementsmadeinrecentyears.Tofurtheraccelerateimprovementsinchildhealthandnutritionoutcomes,thereisanincreasingtrendwithinESARcountriestodecentralizeresponsibilityforplanninganddeliveryofessentialchildsurvivalanddevelopmentservicesasameanstobringpublicdecision-makersclosertolocalcommunitiesandneeds,buildlocalcapacity,andimproveaccountability.ConcurrentlytherehasbeenanobservationfrompastUNICEFandotherUNagencies'workthat,formostservicedeliverysystemsatdecentralised levels, the traditionalmodelof input-outputmonitoringandoccasionalimpactmonitoring(attheendofamulti-yearperiod)isnotaddressingdatagapsrelatedtotheachievementof intermediate results.Thismeans there isa lackof "earlywarningdata, aswellasfeedbackloops,tosignalwhetherservicesarelikelytobeeffectiveornot;implementationbottlenecksand barriers need to be removed; and mid-course programme adjustments need to be made. Thedetrimentaleffectsofanabsenceof intermediateresultsmonitoringandmorereal-timeresponsearemany.Theseincludeineffectiveplanningandbudgeting,inefficientuseofmaterialandhumanresources,andcontinuedexistenceoflargeunderservedpopulationsofchildren.Further,althoughhigh-impact,evidence-basedandaffordablechildsurvivalinterventionsareknown,anumberofglobalstudieshavehighlightedthepoorinstitutionalcapacitywithindecentralizedhealthandhealth-relatedsystemsindevelopingcountriesandtheconstraintsfacedbydistrictmanagementteamsintheimplementationofhealthandhealth-relatedservices'.Theseincludealackofreliable,timely,andstrategically-selectedandpresentedlocaldata;insufficientemphasisonresults;lackofidentificationofbottlenecksimpedingeffectivecoverage;failuretoinvolvecommunitiesasactiveproponentsinhelpingovercome obstacles to high Coverage; and a lack of analytic and prioritization capacity which isparticularlyneededwithinthecontextofchronicresourceconstraintsfacedbysub-nationalmanagementteams.Asaresult,innovativeandsustainablestrategiesareneededtostrengthendistrictandothersub-national monitoring systems, including citizens community monitoring and social accountabilityfeedbackmechanisms,anddatausepracticesinthehealthand/orhealth-relatedsectorsasameanstoachieveimprovedandmoreequitabledeliveryofqualityservicesandoutcomesforchildrenandwomen.Toaddresstheabove,UNICEFHeadquarters,incollaborationwithitsEastandSouthernAfricaRegionalOffice(ESARO),decidedtoaddresstheabovehighlightedissuesbyimplementingaregionalInstitutionalStrengtheningSupportinitiativetoimprovetheProgrammeMonitoringandResponse(PMR)capacityofdecentralizedgovernmentdecision-makingbodiesandcommunitiesinfourcountries(Kenya,Swaziland,Uganda&Zimbabwe).Atthistime,UNICEFESAROisseekingtheservicesofawell-establishedandexperiencedconsultingfirmtoConductanend-of-programevaluationwhichwillsystematicallyandobjectivelyassessthisregionalinitiative in relation to its design, implementation and results taking into account agreed evaluationcriteria.TheevaluationwillassessbothquantitativeandqualitativeresultsachievedinrelationtotheInitiative'sResultsFramework(seeAnnexA-ResultsFramework).



Atthistime,UNICEFESAROisseekingtheservicesofawell-establishedandexperiencedconsultingfirmtoconductanend-of-programevaluationwhichwillsystematicallyandobjectivelyassessthisregionalinitiative in relation to its design, implementation and results taking into account agreed evaluationcriteria.Theevaluationwillbecomprisedofanend-lineassessmentofquantitativeresultsachievedinrelation to the initiative's Results Framework and a sequenced, field-based process and outcomeevaluation.ProgrammeDescriptionIn2014,UNICEFHeadquarters,incollaborationwithitsEastandSouthernAfricaRegionalOffice (ESARO),decided toaddress theabovehighlighted issuesby implementinga regionalInstitutional Strengthening Support initiative to improve the Programme Monitoring and Response(PMR)capacityofdecentralizedgovernmentdecision-makingbodiesandcommunitiesinfourcountries(Kenya,Swaziland,Uganda&Zimbabwe).The ESAR Institutional Strengthening Support in Program Monitoring and Response initiative wasdesignedasalearninginitiativewiththefollowinggoals:

• improve service provision/programme performance during the course of implementationthrough improvedmonitoringof intermediateresults,establishmentofcitizen feedback loops,andtheuseofmonitoringandfeedbackdatatoadjuststrategiesasneededinatimelymanner;and

• identifyandpromotethosepromisingpracticesofprogrammemonitoringandresponse,testedandprovenbyearlyadopterdistricts,throughthedevelopmentofnationalscale-upplanswithinparticipatingcountriesandtheestablishmentofknowledgeexchangeplatforms.

The initiative is takingplace in selectdistricts in the four countriesofKenya, Swaziland,UgandaandZimbabwe and within the context of existing national and sub-national government monitoring andmanagement informationsystems.UNICEF'sdesignforthis institutionalsupport initiative isbasedonfiveprogrammaticpillars:

1. areliable&quality(more)real-timedataisavailableatthesub-nationallevelforidentifyingandaddressing barriers and bottlenecks to the delivery of health and/or health-related services(Nutrition,HIV,WASH)

2. citizens/youthgroupsprovidefeedbackonthedeliveryofhealthand/orhealth-relatedservices3. quarterlyperformancereviewsareConductedandacteduponbydecentralizeddecision-making

bodiesusingdatafromrealtimemonitoringandcitizenfeedback4. knowledgeisincreasedamongUNICEFandrelevantgovernmentstaffinESARcountriesonthe

use of real timemonitoring data for decision-making in the delivery of health andlor health-relatedServices;and

5. evidence-informedNationalPlansofActionaredevelopedwithinthefourcountriesforscaling-upmodelsfordecentralizedPMR.

Thesepillars also comprise theexpected intermediateoutcomesof the initiativeasper theapprovedResultsFramework.UNICEF'sregionallearninginitiativeendeavorstotestwhethersub-national,decision-makingbodiesinthe health and/or health-related sectors (Nutrition, HIV, WASH) could improve the timeliness andeffectivenessoftheirdecisionsandconsequentactionsifmonitoringdatawereprovidedandusedonamore timely, reliable and higher quality basis; and if associated citizen engagement and socialaccountability mechanisms were established or strengthened so that feedback from citizens andcommunitieswas also routinely used to informpriorities, decisions and actions by both institutionaldecisionmakersandcitizensorcommunitiesthemselves.Inotherwords,UNICEF'shypothesisisthatifdecentralizedmanagementand/orfront-linehealthworkerdecisionsandresponsesaremore timelyandevidence-based,and ifcitizen feedback loopsandsocialaccountabilitystructuresareestablished,thenovertimeimprovementswillbeseenin:



a sector service provision/programme performance (i.e. coverage and quality of services); ocommunities'commitmenttoinvestintheirownhealthandwell-being;andothehealthandwell-beingof vulnerable and previously under-served children andwomen, thereby improving equity of healthoutcomes.ScopeofWorkPurpose of the Evaluation This evaluation is being done to support the further national scale-up ofdocumented effective PMR institutional strengthening interventions in the four initial participatingcountries; guide anticipated replication in other countries inwhichUNICEFoperates; and add to theregional and global knowledge base by generating evidence on processes and outcomes related tostrengtheningdecentralizedPMRsystemsinthehealthandnutritionsectors.Inaddition,theevaluation'sfindings and recommendations will provide valuable information on the extent to which the workundertakenwithinthesefourcountriesisapplicabletoothersectorsbeyondhealthandnutrition.The primary audiences for the evaluation will be UNICEF, UNICEF's partners directly involved inimplementationoftheinitiative(e.g.governmentbodies,otherimplementingpartnersandcommunityorganizations),andthecurrentdonorsoftheprogramme(theBillandMelindaGatesFoundationandtheUS Fund for UNICEF). Secondary audiences will be other donors, development stakeholders, andprofessionals interested in improvedprogrammemonitoringandresponse,citizenengagement/socialaccountability,anddatademand&use.EvaluationObjectivesTheobjectivesoftheevaluationareto:

• Assess achievement towards the programme’s expected results (milestones, outputs,intermediateoutcomes,andprimaryoutcomes)aspertheinitiative’sResultsFramework;

• Assesstheextenttowhichtheprogrammeactivitieshavechangedhow(more)realtimedataiscollected, analysed, packaged, disseminated, and used (i.e. the extent of uptake by targeteddecentralizeddecisionmakingbodies);

• Assesswhether, and how,mutual and ongoing learning on PMR is taking place across targetdecentralizedgovernmentunitsandcountries,aswellaswithintargetUNICEFoffices;

• Understand and document the range of contextual/environmental factors that served asenablers/disablers to the results achieved, including programme design, implementation,managementandcoordinationprocesses;

• Assess the programme’s relevance, effectiveness, sustainability, scalability, and replicability todetermine if the package of support interventions and tools employed by UNICEF’s fourparticipatingCOscanserveasapplicablemodelsforotherUNICEFcountriesandstakeholdersinsimilarcontextandsectors,andothersocialsectors;

• DetermineifthefourUNICEFCOscanserveas‘centresofexcellence’withinUNICEFgloballyondecentralizedPMRorspecificaspectsofdecentralizedPMR;

• Documentbestpracticesandlessonslearnedrelatedto:o EffectivePMRamongdecentralizedpublicdecision-makingteamso InstitutionalsupportaimedatstrengtheningdecentralizedPMRmechanisms,processes

andpractices,ando Adaptiveprogrammingduringimplementationthroughtheavailabilityanduseofreal-

timemonitoringandcitizen-feedback;and• RecommendwhetherthistypeofinstitutionalstrengtheninginitiativefocusedonPMRshouldbe

continuedwithinESARand/orotherUNICEFregionsasaprogrammingstrategyregardlessofsector,andifyes,howitwouldbestbeplacedwithinUNICEF’sorganisationalstructure.

EvaluationScope



TheevaluationwillcovertheentireprogrammeperiodoftheESARPMRinitiativefromNovember2014untilpresent(withtheofficialendingdateoftheprogramme30June2016).Therearenocomparatorsforassessment.UNICEFofficestobeincludedintheevaluationprocesswillbethefourparticipatingCOs(Kenya,Swaziland,Uganda,Zimbabwe),ESAROandNYHCR.Itislikelytheconsultingfirmwillcarryouttheworkinfivephases:Phase 1 - Desk Review and Inception Planning (approximately 2.5 weeks): Thoroughly review keyreference documents related to the overall initiative, as well as reports and other documentationsubmitted by the countries over the course of implementation. Prepare a full design for the endlineassessmentandend-of-programmeevaluation,asaseparateannexeddocumenttotheInceptionReportandproposedWorkPlansTimeline.Theevaluationdesignistoincludeananalyticalframeworkanddatacollectiontools.relevantNYHCRstaff.Thisinformationwillprovidethebasisforthepreliminaryevaluationdraftreportandinformcountry-leveldatacollectionprocesses.Country-levelfieldvisitlogisticswillalsobeorganizedduringthisperiodincollaborationwithESAROandtheCOs.Phase3-VerificationFieldVisits(to4countriesoverapprox4weeks):Undertakefieldvisitstotheparticipatingcountriestoverifyexistingend-lineassessmentandevaluationdata(quantitativeandqualitative).Phase4-Analysis,ReportingandValidation(approx4weeks):SubmitWell-supportedpreliminary findings, conclusions and recommendations for discussion and comment at a remoteValidation Meeting with key internal UNICEF stakeholders. Prepare full versions of draft evaluationreportsfollowingValidationMeeting.Phase5-Dissemination(approx2week):FollowingValidationMeeting,presentConclusionsandrecommendationsathigh-levelreviewandforward-planningPartnersForumtobeheldinUgandainmid-May2017.Incorporatefeedbackfromfinalreviewforumandsubmitfinalevaluationreports.Additionally,theconsultingfirmwilltakeintoaccountthebelowoutlineofprocesseswhendevelopingits proposed workplan (subject thereafter to agreements made during the Inception Phase of thecontract):1.Inconsultationwiththe4COsandESARO,decideonthefinalsetofdatasourcesandspecificattributesof thedata systems thatwillbeassessed in theCountriesat thevariousdecentralizedadministrativelevels targetedby the Initiative, aswell aswithinUNICEF.TheEvaluationTeammayalsobeasked toanalyzeexistingdatatoproduceabaselineforsomeoftheevaluationquestionswherepossibleandasneeded.2.DevelopinacollaborativemannerwithUNICEF(ESARPMRinitiativeSteeringCommitteeandthe4COs)andobtainwrittenemail approval for thekeyevaluationquestionsandanalytical framework todrive the evaluation (see Annex B: List of indicative Evaluation Questions). In addition to questionsfocused on the evaluation of outputs and outcomes, there is also need to understand the range ofcontextual/environmental factors that served as enablers 1 disablers to the outputs and outcomesachieved,includingprogrammemanagementandimplementationprocesses.3.Developananalysisplan.Mixedformsofanalysisareencouragedtofacilitatetriangulationofdataandstrengthenthebasisforevaluationfindings,conclusionsandrecommendations.4.Develop,shareandobtainWrittenemailapprovalforthedatacollectionandassessmenttoolstobeusedfor(i)datasystems/monitoringsystems;(ii)citizenengagementmechanisms/processes;(iii)datauseprocesses/practices;(iv)knowledgeexchangeplatforms/processes,and(v)sustainabilityandscale-upplanningprocesses.Thetoolswillneedtobecontextualizedbytypeofdatasource.



5.DevelopaSamplingPlanforthoseaspectsoftheevaluationrequiringit(e.g.inrelationtothelargenumberoffacilitiesandcommunitiesthatcouldpotentiallybeinvolvedintheevaluation).The bidder will build upon the baseline and evaluability assessment work done at the beginning ofprogrammeimplementationbyanexternalconsultingfirm,andtakeintoaccountsecondrydatagatheredOvertheCourseofprogrammeimplementation inaccordancewiththe initiative'sResultsFrameworkandassociatedM&EPlan,toproposeoutputandoutcomemeasuresthatmaygobeyondthoseforwhichdataiscurrentlybeinggathered.ThesewillbedevelopedinrelationtothecoreresearchquestionsagreedwiththecontractorduringtheInceptionPhase.PleaserefertoAnnexBforanascent,thoughindicative,listofpotentialevaluationquestionstobeconsideredandfurtherdevelopedbytheconsultingteam.MethodologyTheoverallmethodologicalapproachanddesignofthisevaluationwillbeparticipatory,involvinganappropriaterangeofindividualsdirectlyinvolvedinimplementationandoversightoftheinitiativeaswellasotherrelevantstakeholders.Theapproachwillbeboththeory-andcase-basedandwillemploymixedmethods,bothquantitativeandqualitative,tosupporttriangulationofdatagathered.Emphasiswillbeplacedongatheringdisaggregateddatabygeographicarea,programmaticpillar,implementingpartner,andperhapsonadditionalvariablesyettobedecided.Whileitisclearfromtheobjectivesthattheevaluationwillhaveafocusonoutputsandoutcomes(notimpact),therearealsoanumberofprocessissuesofrelevance.Bidderswillincludeintheirproposalsafull evaluation design, including the approach to be taken to complete the endline assessment ofquantitativeresultsasComparedtothebaseline.Thedesignwillincludeevaluationcriteria,keyresearchquestions,samplingplan,anddatacollectionandanalysismethods.Thedatacollectionandanalysismethodsproposedbythebidderwillbesufficientlyrigoroustoensureanunbiasedassessmentofboththeprocesses(howtheinitiativewasimplemented)andtheoutputsandoutcomes (what results were achieved through the initiative - either through direct contribution orthroughleveragingofotherrelevantactorsandresources).Theconsultingteamisfurtherencouragedtoutilise a range of data collection methods including desk reviews, key informant interviews fquestionnaires, focus group discussions, technical reviews of data monitoring and managementinformation systems and protocols, direct observation (particularly of performance 1 data reviewmeetingsandcommunityengagementprocesses),etc.The consulting team, with support from each county (UNICEF staff and relevant national and localgovernmentstaff),will identify thespecificsystems ineachdistrict tobeassessed.Thesystemstobeassessedneedtobeagreedwitheachcountry.Keyinformantsforeachsystemcouldincludedevelopers,managers,usersandoversightbodies.He/shewillalsohaveconsultationswithpartnersthatfundsuchsystems.9UNICEFESAROincoordinationwiththe4participatingCOswillstrivetoassessthereliabilityofthedisaggregateddatapriortothestartoftheevaluation.VerificationofUNICEF'sassessmentwillberequiredoftheconsultingfirm.Themethodologyproposedbythebidderwillbeevaluatedforitslogicalandexplicitlinksbetweentheevaluationquestions/criteria,datasources,datacollectionmethodsandanalysismethods.Primarydatasourceswillinclude,thoughnotlimitedto:

• Health/NutritionManagementinformationSystemdatareports;• citizenengagementmechanismdatareports;and• meeting minutes and action plans from targeted decentralized government decision-making

bodiesandcommunitygroups;• Individuals, Committees/teams and implementing partners involved in the direct

implementation, technical support, oversight and funding of the programme at the country,regionalandgloballevel;



• Targetdecision-makingbodiesintendedtobenefitfromthecountry-specificinstitutionalsupportinterventions implementedwithineachof thecountries(health/nutritionmanagementteams;facilitymanagementteams;wardandvillageleadershipcommittees;citizengroups)

Secondarydatasourceswillinclude,thoughnotbelimitedto:

• Baseline&evaluabilityassessmentreportspreparedbyanexternalconsultingfirm(seeAnnexC• forasummaryoftheseassessmentfindings).• UNICEFprogrammedocumentationasavailableintheRTMCommunityofPracticedocumentand

video libraries (e.g. data submitted by the COs inmonthlymonitoring reports including thatgathered through periodic country-level spot checks, monthly coordination/managementmeeting minutes, ESARO TAQA trip reports, annual programme review reports, stakeholdermapping)

• KnowledgeproductsproducedasoutputsoftheinitiativeTheproposedmethodologicalapproachshouldbeclearonhowtheevaluationwillbesequencedoverthethree-monthperiodfromMarch-May2017,includingwhichquestionswillbeaddressedatwhichtimepoints.Itisenvisagedthatforsomequestionstrenddatawillneedtobegatheredandanalyzedwhileforothersanend-linepositionmaybesufficient.Informationshouldalsobeprovidedonwheredatatoaddresseachquestionareexpected tocome from,howanysamplingneededwillbeundertaken,andwheredatawillbequantitativeorqualitativeinnature.TheevaluationistobeconductedinaccordancewithUNICEF'sevaluationguidelinesandUNEGqualityguidelinesandethicsprinciplesforevaluationandresearch.Assumptionsandlimitationsofthechosenevaluationmethodswillalsobeincludedinthebidder'sproposalandlaterinthemorerefinedInceptionReport. Quality assurance to ensure a high standard of design and reporting is expected within theEvaluationTeam,butwillalsobesupportedatcertainagreedstagesthroughESARO'sPPMEOfficeandtheESARPMRinitiative'sSteeringCommittee.Inaddition,UNICEF'sESARO'sexternalqualityassuranceservice (provided through an organization known as Universalia) will provide inputs andrecommendationsontheevaluationTOR,designdocumentsandreports.UltimatelythemethodologywillbemutuallyagreedbetweenUNICEFESAROandtheexternalEvaluationTeam.TheEvaluationTeamwillberesponsibleforputtingforwardtheirapproachtoansweringthekeyquestionsandpresentingthisintheirproposaldocumentandlateragainintheInceptionReport.UNICEFdoes not wish to prescribe any particularmethodology but expects that Evaluation Teams' previousexperienceandexpertisewillbeutilizedtodevelopthemostefficientandeffectiveevaluationapproach.LimitationstotheEvaluationThefourcountriesparticipatinginthelearninginitiativebeganatconsiderablydifferentdevelopmentallevels in relation to the existence and/or functionality of decentralized programme monitoring andresponseinstitutionalizedsystemsandpractices.Therefore,theirbeginningandendingpointsinrelationto any of the integrated programmatic pillars is expected to vary from country-to-country. Othersignificantlimitationstotheevaluationarenotexpectedthoughthequalityofinformationprovidedbysomesourcesisanticipatedtobeanissue.Withintheevaluationtimeframeandbudget,itwillalsonotbepossibletogatherprimaryinformationfrom all related sources and therefore the evaluator will be encouraged to utilize both primary andsecondary data sources-particularly data gathered by the UNICEF programme managers andimplementation teams as part of the programme's monitoring plan. Citizen and community groups,representing vulnerable populations, will be available and included in the evaluation as focus groupdiscussants,intervieweesorparticipantsinotherformsofevaluationdatacollection.


PMRINITIATIVELOGFRAME

IntermediateOutcomes,Outputs,Milestones&Activitiesfor4CountryOfficesIntermediateOutcome

Output Kenya Swaziland Uganda ZimbabweOutput Milestones Activities Output Milestones Activities Output Milestones Activities Output Milestones Activities

1.1 Reliable& qualityreal-timedata is usedby districtsforidentifyingandaddressingbarriers andbottlenecksto thedelivery ofhealth,nutrition,HIV and/orWASHservices

1.1.1Platformfor collectionof real timemonitoringdataestablished orupdated

1. DHIS2platform atfacility-levelin 4 sub-counties(districts)

1. DHIS2platformestablished atfacility-levelin 4 sub-counties(districts)2. DigitalDHIS2reportingoperational atfacility &communitylevels (movedfrom paper-basedsystem)

1. Facilities’readinessassessed2. Facilities’infrastructureupgraded3. Uploadingrights forfacility HWsobtained4. CHWsreportroutineDHIS2 datausingmobiles

1. HMISplatformcountry-wide

1. HMISplatformstrengthenedinallfacilities

1. Vehicleprocured fortransferringdatafromhealthfacilities toregions2. HMIS officerstrained on datamanagementand qualityassurance

1. HMIS,DHIS2 andIHRISplatformsupdatedcountry-wide

1.Strengthennationalaccountability systemslinked toperformance2.Addclientsatisfactionandcommunity/demand-sidedata toHMIS&DHIS2

1. HMIS,DHIS2 &IHRISplatformsupdated togenerateBottleneckAnalysisMaps2. HMIS &DHIS2platformsupdated tohaveCommunityScorecard(CSC) dataintegrated

1. ODKplatformforFood &NutritionCouncil’s(FNC’s)NutritionReal TimeMonitoring(NRTM)DataSystem

1. ODKplatformestablishedfor NRTMDataSystem

1.Datasystemrequirementsdetermined2. FNC server&connectivityassessed3. TOR forDataSystems/DashboardConsultantdeveloped4. DataSystemsConsultantrecruited5. Multi-sectoral,nutrition-related dataentry modulerefined6. Systemtested (ODKdata&alerts)7. Equipmentprocured(computers,smartphones,featurephones,printers)8. Systemperformancereviewed

1.1.2 Data usereportsdevelopedmonthly

1.DHIS2RoutineSystemReports

1.All levelsoftrainingscomplete onDHIS2 &Quality DataAudits (TOTs,facilities,CHWs,CHEWs)

1. TOTtraining onDHIS2 &Quality DataAuditscompleted2. FacilityHWs trainedonDHIS23.CHWs/CHEWstrained onMOVE-IT andmobileroutinereporting

1.DashboardsforRHMTs2.Dashboardsforfacilities

1.Dashboardsfor RHMTs &facilitiesoperational

1. Performanceindicatorsfinalized2. Dashboarddevelopmentconsultantrecruited3. Dashboardsdeveloped forfacilities4. Dashboardsdeveloped forRHMTs

1. DHIS2Dashboardswith CSCdataintegrated2. RMNCHDashboards3.BottleneckAnalysisMaps4.U-Reports

1. UpdatedHMIStools/reports operationalin 112districts2. UpdatedHMIStools/reports operationalat facilitylevelwithin3districts

1.BottleneckAnalysisMapsdeveloped2. HWstrained onrevisedHMIStools3. RevisedHMIS toolsprinted &disseminated

1. NRTMSystemReports atthree levels:District,Ward &Village

1. DistrictNRTMSystemReportsoperationalin4districts2. WardNRTMSystemReportsoperationalin20wards3. VillageNRTM

1.Developlistofindicators2. Defineindicatorthresholds3. Developdatatools4. Developmodel fordataflows/mgmt(collection &handling)5. Test datatools&model


IntermediateOutcome

Output Kenya Swaziland Uganda ZimbabweOutput Milestones Activities Output Milestones Activities Output Milestones Activities Output Milestones Activities2. RMNCHScorecards3.HouseholdRegisterDashboards4. MOVE-ITreports

2. RMNCHScorecardsgeneratedmonthly byDHIS2System3. HouseholdRegistrationDashboardsgeneratedmonthly byDHIS2 Systemin4districts4.Communitiesreportingmonthlyusingmobiles toMOVE-ITPlatform

4. RMNCHScorecardsdeveloped5. HouseholdRegistrationDashboardsdeveloped6. MOVE-ITapplicationinstalled inCHWs’mobiles7. CHWsreport toMOVE-ITusingmobiles

3.DataQualityAssessmentReports

2.DataQualityAssessmentsavailable

5. Data QualityAssessmentsconducted

5. DataQualityReports

3. UpdatedHMIStools/reports operationalatcommunitylevelwithin3districts

2. MonthlyNutritonDashboardsfor4levels:National,Provincial,District,Ward

SystemReportsoperationalin60villages2. NutritionDashboardsfor 4 levelsavailablemonthly

for dataflows/mgmt6. Refine datatools & datamgmtmodel7.Map datamgmt model&timelines8. TOT datacollection &managementtraining9. Districttrainers trainwards/villages10.Initialdatacollection bywards/villages11. Dataqualitychecksof initialsystemreports12. Finalrefinementsto system,tools, datamanagementprocesses13. FNC datasystem roll-out to 4districts14. NutritionDashboardsdeveloped

1.2 Citizen/youthgroupsprovidefeedback onthe deliveryof health,nutrition,HIV and/orWASHservices

1.2.1Platformfor collectionof citizenfeedbackinformationestablished

2. SMS-basedcitizenfeedbacksystemplatform(may beRapid-ProSMSPlatform)

2. SMS-basedcitizenfeedbacksystemplatformoperational in4 sub-counties

1. Determinewhich type ofSMS-basedsystem toestablish andthen developadditionalactivities

1. U-ReportRapid-ProSMSPlatform

1. U-ReportPlatformoperational

TBD 1.Rapid-ProSMSPlatformcountrywide

1. U-ReportRapid-ProSMSPlatformoperational& inter-operable

1. Leadefforts toupgradeMTrac toRapid-Pro2. MakeRapid-Prointer-operablewith DHIS2&IHRIS

1.Rapid-ProSMS-Platformoperationalcountry-wide

1. Rapid-ProSMSPlatform(interoperable with ODK-based FNCNRTM DataSystem)

1.Adaption/adoption ofRapid-Procitizenengagementplatform2. SocialaccountabilitySMS datamoduledevelopmentforFNC(ODK)system3. Citizenengagementinformationincluded inNutritionDashboards&


IntermediateOutcome


NRTM SystemReports

1.2.2 Citizenfeedbackreportsdevelopedeachmonth

1.CommunityScorecards2. Citizen-feedbackreportsfromSMS-basedsystem withCSC dataintegrated

1. CSCsoperational inall241CUs inHoma BayCounty(later CUs inSiaya Countytobeadded)2. SMS-basedcitizenfeedbacksystemoperational3.SMS-systemintegratedwith CSCsoperational

1. CSCadapted/developed2. CommunityorientationonCSC & socialaccountabilityconducted3. CommunitySocialAccountability Teamsestablished4. CSC rolled-out5. ISPProviderconsultationsconducted6. SMS-basedcitizenfeedbacksystemdeveloped7. CommunityorientationonSMS-basedsystemconducted8.SMSsystemintegratedwithCSCs9. IntegratedSMS/CSCCitizenFeedbackSystem rolledout

1.U-Reports U-Reportsoperational

1. U-Reportdeveloped2. Opinionpolling surveysontopicalhealthor health-related issuesdeveloped3. Opinionpolling surveysconducted

1.U-Reports2. CommtyScorecards

U-Reportsavailablewithexpandedcitizenfeedback2.CommunityScorecardsoperational

1. RecruitadditionalU-Reporters2. Monthlypolls to U-Reporterssent3. FeedbackStrategyDeveloped4. CommtyScorecardsdeveloped

1.U-Reports U-Reportsavailable

1. Orientationworkshop forcitizengroupson citizenengagement2. Tool forvillage-levelreview ofcitizenfeedbackinformationdeveloped3. Tool forward-levelreview ofcitizenfeedbackinformationdeveloped

1.3Quarterlyperformancereviews areconductedand actedupon bydistrictsusing datafrom realtimemonitoringand citizenfeedback

1.3.1Quarterlyperformancereviewsconducted

1. QuarterlySub-County(SCHMT)PerformanceReviews2. QuarterlyCounty(CHMT)PerformanceReviews

1. QuarterlySHMTPerformanceReviewMeetingsconsistentlyheld2. QuarterlyCHMTPerformanceReviewMeetingsconsistentlyheld

1. Monthly &QuarterlyRHMTPerformanceReviews2. Monthly &QuarterlyFacilities’PerformanceReviews

1. Monthly &QuarterlyRHMTPerformanceReviewMeetingsheld2. Monthly &Quarterlyfacilities’PerformanceReviewMeetingsheld

1. Monthly&QuarterlyDHMTPerformanceReviews2. Monthly& QuarterlyFacilities’PerformanceReviews

1.Monthly&QuarterlyDHMTPerformanceReviewMeetingsheld usingBNA/CA2.Monthly&QuarterlyFacilities’PerformanceReviewMeetingsheld usingBNA/CA

1. TAContract forBNA/CAtrainingdone2. Trainingon BNA/CAforDHMTs/Facilities’HWsconducted3.Barriers&bottlenecksidentifiedbyDHMTs&Facilities’HWs

1. MonthlyPerformance Reviewsby threelevels ofFNCCommittees(District,Ward &Villages)

1. FNCCommittees’arefunctional2. Villagesand Wardswithin 4Districtsselected to

1. Ensure allFNCCommittees’TORs haveclear datamanagementresponsibilities2. DeveloptoolstoassessFNCCommittees’functionality3. Select 3functionalVillage FNCCommittee’s


IntermediateOutcome


3. QuarterlyCommunityDialogues

3. QuarterlyCommunityDialoguesconvened incommunitieswithinfacilities’catchmentareas andbottlenecksidentified

usingBNA/CA

2. QuarterlyInter-DistrictPerformanceReviews

participate intheproject3. MonthlyPerformanceReviewmeetingsconductedusingBNAbyFNCCommitteesin 4 districts,20wardsand60villages2. QuarterlyInter-DistrictPerformanceReviewsamong 4districts

per Ward toparticipate inproject4. Buildcapacity of allselectedWards toensurefunctionality5. MonthlyPerformanceReviewMeetingssupported byUNICEF &FNC in whichbottlenecks &barriers areidentified6. QuarterlyInter-DistrictPerformanceReviewsrotatedbetween 4districts

1.3.2Quarterlyperformancereviews resultin updatedAction Plansusinginformationfrom real-time data usereports(includingcitizenfeedbackreports)

1. Sub-County(District)Action Plansupdated(using datausereports)2. CountyAction Plansupdated(using datausereports)

1.Informationfromreal-timedata usereportsused:i. DHIS2routinesystemreports (withCHWs routinemobile dataincluded)ii. RMNCHScorecardsiii.HouseholdRegistrationScorecardsiv. QuarterlyFacilityQuality AuditReportsv. CitizenFeedbackReports withCSC dataintegrated

1.Informationuse fromreal-time data usereportsverified2. Facilityquality dataauditsconductedquarterly

1. RHMTAction Plansupdated(using datausereports)2. Facilities’Action Plansupdated(using datausereports)

1.Informationfromreal-timedata usereportspresented toQA Program,RHMTs &Facilities andused:i. RHMT &Facilities’Dashboardsii.DataQualityReportsiii.U-Reportsiv. HMISRoutineSystemReports

1. RHMTstrained on datause(dashboards, U-Reports, HMISreportingtemplates)2. QA Programtrained on theuseofU-Reports3. Facilities’Managers/HWs trained onthe use ofdashboards,U-Reports,HMISreportingtemplates3. RHMT &Facilities ActionPlansdeveloped4. Info use fromreal-time datause reportsverified

1. Monthly& QuarterlyDistrictActionPlansupdated(using datausereports)2. Monthly& QuarterlyFacilityActionPlansupdated(using datausereports)

1. Info fromreal-timedata usereportsused,as perFeedbackStrategy &SOP:i. DHIS2routinesystemdashboards(with CSCdataintegrated)ii. RMNCHScorecardsiii.BottleneckAnalysisMapsiv.U-Reportsv. DataQualityReports

1. SOPdeveloped&used formanagement solutionsbyDHMTs2.Informationuse fromreal-timedata usereportsverified

1. DistrictActionPlansdevelopedorupdated2. WardActionPlansdevelopedorupdated

1.Informationfrom real-timedatausereports usedby DistrictFNCCommittees:i. NRTMRoutineSystemReportsii. NutritionDashboardsiii. QualityAuditReportsiv.U-Reports2.Informationfrom real-timedatausereports usedbyWard FNCCommittees:

1.Informationusefromreal-time data usereportsverified


IntermediateOutcome


3. UpdatedMonthly &QuarterlyCommunityActionPlans(usingdata)

3. CommtyDialoguemeetingsheld byHealth UnitCommitteesto developCommunityActionPlans

3. VillageActionPlansdevelopedorupdated

i. Ward-levelReview Toolfor CitizenFeedbackii. NRTMRoutineSystemReportsiii. NutritionDashboards3.Informationfrom real-timedatausereports usedby VillageFNCCommittees:i.Village-levelReview Toolfor citizenfeedback

1.3.3 Actionstakenbasedonquarterlyperformancereviewswithintimeframespecified inDistrict ActionPlans

1. Actionstaken byCHMTs/Sub-Countieswithintimeframespecified inrespectiveActionPlans

1. Actionstaken byRHMTs,FacilityManagers, &QA Programwithintimeframespecified inrespectiveActionPlans

1. Actionstaken byDHMTs,Facilities &Communitiestimeframespecified inrespectiveActionPlans

1. Actionstaken byDistricts,Wards, &Villageswithintimeframespecified inrespectiveActionPlans

2.1 Staff ofgovernments andUNICEF usee-Platformsfor cross-countrylearningabout realtimeprogrammemonitoringandresponse inESAR

2.1.1 e-Platforms forcross-countryknowledgeexchange &learningestablished

N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

2.1.2 Non-electronicPlatforms forcross-countryknowledgeexchange &

1. Nationaldisseminatnat NationalHealthCongress to

1. Nationalreport onintegratedSMS/CSCsystem ready

1. Nationalreport onintegratedSMS/CSCCitizenFeedback

Inter-districtlearningplatform


IntermediateOutcome


learningestablished

all 47Counties2. Inter-County KEForum3. County-Level KEPlatform4. Sub-County KEForums

fordissemination

Systemproduced2. Documentfacility-levellearnings &bestpractices3. Documentcommunity-levellearnings& bestpractices

2.1.3 Toolboxfor cross-countryknowledgeexchange &learningdeveloped

N/A N/A N/A N/A N/A N/A N/A N/A N/A ToolsDevelopedincluding:1. DataCollectionTrainingMaterials2.IndicatorsManual3. SocialAccountabilityManual4.Guidelinesat all levels=3

N/A N/A

2.1.4 CaseStudiesdocumentinglessonslearnedproduced

2 CaseStudies

2CaseStudiesproduced

2CaseStudiesdrafted

2CaseStudies 2CaseStudiesproduced

2 Case Studiesdrafted

2 CaseStudies

2 CaseStudiesproduced

2 CaseStudiesdrafted

2 CaseStudies

2 CaseStudiesproduced

2CaseStudiesdrafted

2.1.5Recommenations Paper onscaling use ofinnovativetechnologiesfor real-timemonitoringproduced

N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

2.2Evidence-informedNationalPlans ofAction forscaling-upmodels fordecentralizedprogrammemonitoringdeveloped

2.2.1 Baseline& EvaluabilityAssessmentsconducted

KenyaBaselineReport

KenyaBaselineReportCompleted

TBD SwazilandBaselineReport

SwazilandBaselineReportCompleted

TBD UgandaBaselineReport

UgandaBaselineReportCompleted

TBD ZimbabweBaselineReport

ZimbabweBaselineReportCompleted

TBD


IntermediateOutcome


2.2.2 End-of-ProjectEvaluationReportPrepared

N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

ESAR PMR Evaluation Synthesis Report FINAL 29 JUL 17 · PDF fileThe PMR Initiative’s...

Documents

Transcript of ESAR PMR Evaluation Synthesis Report FINAL 29 JUL 17 · PDF fileThe PMR Initiative’s...