From policy to impact - Serious Social Investing 2013
-
Upload
tshikululu-social-investments -
Category
Education
-
view
559 -
download
0
description
Transcript of From policy to impact - Serious Social Investing 2013
Presentation to Tshikululu CSI Conference13 March 2013
Dr Ian GoldmanHead of Evaluation and Research
The Presidency Department of Performance Monitoring and Evaluation
Why is M&E important?What is government doing?
The Presidency: Department of Performance Monitoring and Evaluation 2
Summary
Approach – M&E as a system – not an ad-hoc donor approach
Why is M&E important? Evidence for policy- and decision-making Helping to get a results culture – transforming the
public serviceThe problemHow is government approaching it?Challenges
• Helping policy makers to make better decisions and achieve better outcomes
• Providing better services (public and private)
By using: • Existing evidence more effectively• New research/evaluation to fill the gaps in the
evidence baseAnd: • Integrating sound evidence with decision
makers’ knowledge, skills, experience, expertise and judgement
What Is Evidence-Based Policy?
Source: Oxford Evidentia
The Presidency: Department of Performance Monitoring and Evaluation
Evidence for
Policy
What is already known about the problem/policy?
HarnessExistingEvidence
Research Synthesis
What is the nature, size and dynamics
of the problem?Descriptive and
Experiential
Evidence
StatisticsSurveys
Qualitative Research
What has been shown to work
elsewhere?
Evidence of Proven
Effectiveness
Experimental and Quasi-
Experimental Evidence
Source: Oxford Evidentia
How do we make The policy work?
Implementation
Evidence/
Case StudiesInterviews
Focus GroupsEthnographyOperations Research
What are the ethical implications of
the policy?
Ethical Evidenc
e
Social EthicsPublic
Consultation
What are the costs and benefits of the
policy]
Cost-Benefit/Effectiveness/Utility Analysis
Economic and
Econometric Evidence
How is the policy supposed to work?
Logic Model
Theories of
Change
The Presidency: Department of Performance Monitoring and Evaluation 5
Views of senior managers Scientific and objective, enabling reliable predication based on
facts that speak for themselves, collected by objective and independent specialists, derived through replicable methods and constituting objectively verifiable proof; or
Probabilistic, emergent and contested, an iterative search for explanations and understanding of how to achieve politically derived values in which the choice of facts and sources is influenced by existing ideas, ideology, mind-set, values and interests and subject to specific and changing contextual factors.
A third group straddled these views, indicating that the choice should be dictated by the type of policy to be developed and the type of research methodology appropriate to that type of policy decision.
The Presidency: Department of Performance Monitoring and Evaluation 8
Use of M&E as change strategy - WPTPS A mission statement for service delivery, together with service guarantees; The services to be provided, to which groups, and at which service charges; in
line with RDP priorities, the principle of affordability, and the principle of redirecting resources to areas and groups previously under-resourced;
Service standards, defined outputs and targets, and performance indicators, benchmarked against comparable international standards;
Monitoring and evaluation mechanisms and structures, designed to measure progress and introduce corrective action, where appropriate;
Plans for staffing, human resource development and organisational capacity building, tailored to service delivery needs;
The redirection of human and other resources from administrative tasks to service provision, particularly for disadvantaged groups and areas;
Financial plans that link budgets directly to service needs and personnel plans; Potential partnerships with the private sector, NGOs and community
organisations to provide more effective forms of service delivery;
The Presidency: Department of Performance Monitoring and Evaluation 9
Measuring results
Having a clear direction Having targets (you know what you want to achieve) (Having a theory of change - logical link between what
you do and what you achieve) Linking resources to plans, monitoring progress against
plans Challenge of target approach – does a good headmaster
do what she does because of targets? Be careful of taking private sector models too far.
The Presidency: Department of Performance Monitoring and Evaluation 10
So
How can we strengthen and formalise the use of evidence
How can we formalise the need for effective theories of change Through strengthening planning Through the evaluation process
How can we use M&E as part of an organisational change strategy
The Presidency: Department of Performance Monitoring and Evaluation 11
But we have a problem…
12
1.3 Performance Area: Monitoring and Evaluation
1.3.1 Indicator name: Use of monitoring and evaluation outputs
Indicator definition: Extent to which the department uses monitoring and evaluation information.
Secondary Data: AGSA findings on pre determined objectives – Reported information not reliable.
Question: Which set of statements best reflects the department’s use of M&E outputs?
Statement Evidence Performance level
Department does not have an M&E Policy/Framework or does not have capacity to generate information.
Not required Level 1
Monitoring reports are available but are not used regularly by top management and programme managers to track progress and inform improvement.
Quarterly monitoring reports
Minutes of top management meetings or programme meetings to assess use of reports
Level 2
Monitoring reports are regularly used by top management and programme managers to track progress and inform improvement.
Quarterly monitoring reports
Minutes of top management meetings or programme meetings to assess use of reports
Level 3
All above in Level 3 plus:
Evaluations of major programmes are conducted periodically and the results are used to inform changes to programme plans, business processes, APP and strategic plan.
All above in Level 3 plus: Evaluation Reports Changes to programmes
and plans
Level 4
Score in M&E(based on self-assessments by 103 national and provincial departments)
13
The Presidency: Department of Performance Monitoring and Evaluation 14
Problem Evidence and analysis not used sufficiently in decision-making, planning,
or budgeting ,particularly of programmes 44% of national and provincial departments not regularly using
monitoring reports to improve performance Monitoring undertaken as compliance, not as part of culture of
continuous improvement Evaluation applied sporadically and not informing planning, policy-
making and budgeting sufficiently - missing the opportunity to improve Government’s effectiveness, efficiency, impact and sustainability.
Parliament relatively weak compared to executive, so oversight limited (on this trip to US and Canada to improve understanding of committee that oversees DPME)
The Presidency: Department of Performance Monitoring and Evaluation 15
Government’s approach
The Presidency: Department of Performance Monitoring and Evaluation 16
Roles and Responsibilities for Planning and M&E in SA
Auditor General
• Independent monitoring of compliance
• Auditing of performance information
• Reporting to Parliament
Public Service Commission
• Independent monitoring and evaluation of public service
• Focus on adherence to public service principles in Constitution
• Reporting to Parliament
National Treasury
• Regulate departmental 5 year and annual plans and reporting
• Receive quarterly performance information
• Expenditure reviews
Public Service Dept (DPSA)
• Monitor national and provincial public service
• Regulate service delivery improvement
Presidency
• National Planning Commission (NPC):
o Produce long-term plan (20 years)
• Department of Performance Monitoring and Evaluation (DPME)
o Produce government-wide M&E frameworks
o Facilitate production of whole of government 5 year plans for priorities
o Monitor and evaluate plans for priorities as well as performance of individual departments and municipalities
Cooperative Governance Dept (DCOG)
• Regulate local government planning
• Monitor performance of local government
• Intervention powers over local government
Constitutional power
Legal power
Executive power
The Presidency: Department of Performance Monitoring and Evaluation 17
Focus of DPME to date
M&E of national priorities
Management performance M&E
M&E of front-line service delivery
Government-Wide M&E System
• Plans for the 12 priority outcomes (delivery agreements)• Monitoring (ie tracking) progress against the plans• Evaluating to see how to improve programmes, policies,
plans (2012-13 8 evaluations, then 15, then 20)
• Monitoring of experience of citizens when obtaining services (joint with states)
• Presidential Hotline – analysing responses and follow-up
• Assessing quality of management practices in individual departments (MPAT) at national/state level
• Moderated self assessment and continuous improvement
• M&E platforms across gov – nationally, provincially• Data quality issues• Structures of M&E units/Capacity development • Emerging focus on (implementation) programmes• National Evaluation System (initially NEP-focused)
The Presidency: Department of Performance Monitoring and Evaluation 18
Why evaluate? Improving policy or programme performance (evaluation for
continuous improvement): this aims to provide feedback to programme managers.
Evaluation for improving accountability: where is public spending going? Is this spending making a difference?
Improving decision-making: Should the intervention be continued? Should how it is implemented be
changed? Should increased budget be allocated?
Evaluation for generating knowledge (for learning): increasing knowledge about what works and what does not with regards to a
public policy, programme, function or organization.
The Presidency: Department of Performance Monitoring and Evaluation
Impact evaluation Has the intervention had impact at outcome and impact level, and why
DESIGN
Design evaluationDoes the theory of
change seem strong?
Economic EvaluationWhat are the cost-benefits?
Diagnostic what is the underlying situation and root causes of the problem
Implementation evaluation
- what is happening and
why
Different types of evaluations related to questions around the outcome model
19
The Presidency: Department of Performance Monitoring and Evaluation 20
Following-up of the evaluations
Evaluation report 1page policy summary, 3p exec summary, 25p report
Management response Each department responds formally, and also put on website
Improvement plan Developed with the departments involved after report approved Monitored
Communication Development of customised communication materials for different
audiences Evaluation report, management response and improvement plan
put on dept and DPME website
The Presidency: Department of Performance Monitoring and Evaluation 21
Challenges emerging
Overall the system is working but some challenges are emerging. These include: Poor communication channels from some DGs and programme managers
often not aware of the possibility Some senior managers wary and don’t see it as an opportunity to improve
their performance. Not getting right people to briefing sessions so senior managers don’t understand the system and haven’t bought in
Making sure the evaluations proposed are the strategic ones Sometimes departments not budgeting for evaluations and expecting DPME
to provide all the money Departments not planning ahead – very important for impact evaluations in
particular where need to plan 3+ years ahead Some avoidance strategies happening – eg parallel evaluations, not providing
information to evaluators.
The Presidency: Department of Performance Monitoring and Evaluation 25
So we are developing a corpus of evaluations 7 underway 16 being scoped 93 from 2006 that will go on website in May 15 for 2014/15……..
We are on the journey
The Presidency: Department of Performance Monitoring and Evaluation 26
Background slides
The Presidency: Department of Performance Monitoring and Evaluation
8 evaluations in National Evaluation Plan 2012-13 (1)
1. Impact Evaluation of the National School Nutrition Programme (NSNP). (DBE)
2. Impact Evaluation of Grade R. (DBE)3. Implementation Evaluation of the Integrated Nutrition Programme. (Health)4. Implementation Evaluation of the Land Reform Recapitalisation and
Development Programme. (Department of Rural Development and Land Reform)
5. Implementation Evaluation of the Comprehensive Rural Development Programme. (Department of Rural Development and Land Reform)
6. Implementation/design evaluation of the Business Process Services Incentives Scheme. (Department of Trade and Industry)
7. Implementation Evaluation of the Integrated Residential Development Programme (IRDP).. (Department of Human Settlements)
8. Implementation Evaluation of the Urban Settlements Development Grant (USDG). (Department of Human Settlements)
27
The Presidency: Department of Performance Monitoring and Evaluation 28
Evaluations recommended for 2013/14
1. Evaluation of Export Marketing Investment Assistance incentive programme (DTI).
2. Evaluation of Support Programme for Industrial Innovation (DTI).3. Impact evaluation of Technology and Human Resources for Industry
programme (DTI).4. Evaluation of Military Veterans Economic Empowerment Programme
(Military Veterans).5. Impact evaluation on Tax Compliance Cost of Small Businesses (SARS).6. Impact evaluation of the Comprehensive Agriculture Support Programme
(DAFF).7. Evaluation of the Socio-Economic Impact of Restitution programme
(DRDLR).8. Evaluation of the Quality of the Senior Certificate (DBE).
The Presidency: Department of Performance Monitoring and Evaluation 29
2013/14 continued
9. Setting the Baseline for Impact Evaluation of the Informal Settlements targeted for upgrading (DHS).
10. Evaluating interventions by the Department of Human Settlements to facilitate access to the city (DHS).
11. Provision of state subsidised housing and asset poverty for households and local municipalities (DHS).
12. Impact evaluation of the Community Works Programme. (DCOG).13. Evaluation of the National Advanced Manufacturing Technology Strategy
(DST).14. Impact Evaluation of the Outcomes Approach (DPME).15. Impact/implementation evaluation of national coordination structures
including the cluster system (Presidency).
10 Steps
30