Realizing More Benefits From Logic Models · 2017-05-23 · • Logic Models are useful for...
Transcript of Realizing More Benefits From Logic Models · 2017-05-23 · • Logic Models are useful for...
Draft
1
Realizing More BenefitsFrom Logic Models:
What Logic Modeling Can Learnfrom Benefits Realization
Presentation to Canadian Evaluation Society –
National Capital Region ChapterJune 19, 2007
June 19, 2007 1
Background
• Evaluation is intended to inform Deputies, senior governmentmanagers and taxpayers– Are outcomes/benefits being realized
– ‘Value for money’
• Logic Models are a useful Evaluation planning tool:– display program theory,
– desired outcomes or benefits that program is trying to realize, and
– identifying issues
• But Logic Models do not tell senior managers what needs to be doneto realize benefits or who should be doing it (i.e., accountability)
How can Logic Models overcome these weaknessesand be even more beneficial to senior managers?
Draft
2
June 19, 2007 2
Outline
1. What is a Logic Model?
2. Weakness of Traditional Logic Models
3. What is Benefits Realization?
4. Benefits Realization within Government of Canada (GC)
5. Comparison of Evaluation Logic Model and Benefits Realization
6. What Evaluators can Learn from Benefits Realization Results Chain
7. Conclusions
June 19, 2007 3
1. Logic Model provides ...
• Schematic of the program theory, i.e., how program issupposed to work
– to achieve intended changes and outcomes, and
– how it will make a difference for the affected stakeholders
• Visual of high level ‘road map’ linking program activities,outputs, affected stakeholders and what you want toachieve
• Way to facilitate thinking, planning, communication andshared understanding about program and its intendedoutcomes
Draft
3
June 19, 2007 4
1. Logic Models are used ...
• As the basis for:
– identifying outputs/outcomes,
– coming up with indicators and data sources for ongoing performancemeasurement, and
– developing evaluation issues and methods
• In GC context, integral part of planning:
– government program Business Case, and its performance measurement,and
– evaluation of program, policy or initiative as part of Evaluation Frameworkor Results-based Management Accountability Framework (RMAF)
• Policy Responsibility – Centre of Excellence for Evaluation, TBS
June 19, 2007 5
1. A Logic Model often looks like this ...
Objective
Activities
Outputs
ImmediateOutcomes
IntermediateOutcomes
UltimateOutcomes
Some (or all) Canadians benefit or are better off along one or more dimensions(health, wealth, literacy, security, safety, understanding who we are)
Draft
4
June 19, 2007 6
2. But Traditional Logic Models do haveweaknesses ...
Logic Models are good for identifying outputs/outcomes and evaluationissues at high level, but do not clearly identify:
• Exactly what has to be done to achieve intended outcomes
• Accountability, i.e., who is responsible for achieving outcomes
• Attribution and external factors
• Outcome box size does not reflect effort, time or resources required
• Assumptions and risks associated with achieving outcomes
Developing Logic Model may be a strategic process, but not really usedas management tool once evaluation issues are determined
June 19, 2007 7
2. The Miracles of Logic Models …
Draft
5
June 19, 2007 8
Benefits Realization can help overcomeLogic Model prescriptive and accountabilityweaknesses
1. What is Benefits Realization?
2. Where does it come from?
3. Can Benefits Realization be applied toprograms and help Logic Models realizebenefits?
Over last year or two, TBS ChiefInformation Officer Branch (CIOB) pilotedBenefits Realization re-labelled as“Outcome Management”
2. Overcoming Logic Model Weaknesses …
June 19, 2007 9
Outline
1. What is a Logic Model?
2. Weakness of Traditional Logic Models
3. What is Benefits Realization?
4. Benefits Realization within Government of Canada (GC)
5. Comparison of Evaluation Logic Model and Benefits Realization
6. What Evaluators can Learn from Benefits Realization Results Chain
7. Conclusions
Draft
6
June 19, 2007 10
3. Benefits Realization – what it is …• Traditional IT Project Management - deliver project on time, within
budget and quality targets, but business benefits often not realized
• Benefits Realization is set of activities, methods, processes, andtools for understanding, planning, tracking, managing andrealizing desired outcomes from IT initiatives or projects
• Benefits Realization methodology
– Created in Canada in 1990s
– Published in The Information Paradox
– Realization of intended outcomes or benefits includes business functionbeyond the IT project
– Logic of process to realize benefits is represented by ‘Results Chain’
– Helps manage stakeholder expectations
• Benefits Realization developed with no knowledge of ‘ProgramEvaluation’– But trying to solve similar problems
June 19, 2007 11
Assumptions / Risks
Project Initiatives
Stakeholder Initiatives
OGD Initiatives
GC Outcomes
Project Outcomes
Key Outcomes
Legend
A Key Deliverable: Results Chain links project initiatives to outputs & outcomes
3. Benefits Realization – What it can look like …
Draft
7
June 19, 2007 12
3. Benefits Realization – A closer look …Results Chain model linking new on-line tracking tools that track data on OGDcompliance to new policy and integrate with data from external stakeholders
SimplifiedExample:
Assumptions / Risks
Project Initiatives
Stakeholder Initiatives
OGD Initiatives
GC Outcomes
Project Outcomes
Key Outcomes
O-2On-line
monitoring tools
created
O-3Number of
trained employees increased
O-4Use of
tracking tools increased
O-6Tracking
costs reduced
O-7Reallocation of
resources to higher
government priorities
O-8Standardization
of data and reports
increased
O-9Compliance
to policy increased
O-10Achieve greater
compliance to policy
I-1OGDs authorize skilled resources to attend training
I-2Develop online tools to track policy outcomes
I-3Develop training
protocol and plans
I-4OGDs shift
resources from administration
to analysis
O-5Time and effort
required to produce reports
decreased
A-3Staff can
be re-focused
I-5Stakeholders modernize
their systems
A-4TBS
modernizes policies
A-1OGD senior
management commitment
A-2Data from third party sources
will be available on a timely basis
June 19, 2007 13
4. Benefits Realization withinGovernment of Canada (GC) …
• GC IT/IM projects: deliver on time, on budget to required specifications– but still challenge of achieving intended business outcomes
• CIOB exploring how Benefits Realization could increase projectsuccess and help achieve intended business outcomes
• In June 2005, ADM-level TBS Service Management Board (SMB)mandated GOL Initiative to:– … promote the Benefits Realization methodology endorsed by the TBS …
• CIOB has piloted use of Benefits Realization, e.g., Shared TravelService Initiative (STSI)
• BR viewed as supporting complex horizontal or businesstransformation initiatives by breaking down silos and crossingtraditional organizational boundaries to achieve intended outcomes
Draft
8
June 19, 2007 14
4. Benefits Realization Potential …
Benefits Realization Results Chain has potential to helpovercome weakness of Logic Model:
1. Lays out how downstream business benefits or outcomes canbe achieved
2. Supports project, program and initiative levels
3. Supports an enterprise and horizontal approach
4. Details and assigns accountabilities for realizing outcomes
5. Facilitates on-going and ex post evaluation of “hard” and “soft”benefits
June 19, 2007 15
4. But even Benefits Realization haslimitations …
Limitations of BR Results Chain• A lot of information to collect and to convey in a set of diagrams
• Too complex to present in a single diagram without extensiveexplanation
• Few senior managers are prepared to deal with this level of detail
• ‘Senior Executive’ version not always developed for each project
Draft
9
June 19, 2007 16
Outline
1. What is a Logic Model?
2. Weakness of Traditional Logic Models
3. What is Benefits Realization?
4. Benefits Realization within Government of Canada (GC)
5. Comparison of Evaluation Logic Model and Benefits Realization
6. What Evaluators can Learn from Benefits Realization Results Chain
7. Conclusions
June 19, 2007 17
5. Comparing Evaluation Logic Modeland Benefits Realization (1 of 4) …
Planning tool - can help IT project managerand stakeholders understand logic of howdesired business outcomes can beachieved and prescribe what has to bedone to achieve outcomes
Planning tool - can help manager andstakeholders understand logic of programand theory of how outcomes can beachieved
IT Project driven – Recommended byproject manager and stakeholders
Government Program driven – Mandatedby TBS and Departmental evaluationcommittee
TBS CIOB silo – Outcome ManagementTBS CEE
Benefits Realization / Results ChainEvaluation Logic Model
Draft
10
June 19, 2007 18
5. Comparing Evaluation Logic Modeland Benefits Realization (2 of 4 cont’d) …
Chaos and accountability – clearaccountability of what has to be done toachieve intended Outcomes and who hasto do it
Chaos and hope – no clear accountabilityfor achieving Outcomes
Prescribes Business Transformation byPublic Servants and stakeholders toachieve desired outcomes
Does not address scope of change requiredby Public Servants and stakeholders toachieve outcomes
Linked to Management Improvementculture – help determine what to do toachieve intended outcomes
Linked to Evaluation culture – helpdetermine if outcomes have been achieved
Benefits Realization / Results ChainEvaluation Logic Model
June 19, 2007 19
5. Comparing Evaluation Logic Modeland Benefits Realization (3 of 4 cont’d) …
Updated and maintained as environmentchanges and phases of project arecompleted
Once developed for Evaluation Framework,not likely to change during Evaluation cycle
Useful as planning and ongoingmanagement tool, and for monitoring andmeasuring achievement
Useful in planning Evaluation (Framework)but also maybe in Evaluation
Finite time horizon – Desired outcomesachieved
Ongoing without time limit toward UltimateOutcome
Benefits Realization / Results ChainEvaluation Logic Model
Draft
11
June 19, 2007 20
5. Comparing Evaluation Logic Modeland Benefits Realization (4 of 4 cont’d) …
Deputies and senior managers can benefitfrom the application of the BenefitsRealization methodology
Deputies and senior managers can benefitfrom an Evaluation
Visual display is complex; not so easy tofollow
Visual display is relatively easy to follow
Explicitly addresses assumptions and risks– conveys sense of urgency and what hasto be done and who has to do it
Does not explicitly address assumptionsand risks – no sense of urgency
- Although RMAF text can address these
Benefits Realization / Results ChainEvaluation Logic Model
June 19, 2007 21
Outline
1. What is a Logic Model?
2. Weakness of Traditional Logic Models
3. What is Benefits Realization?
4. Benefits Realization within Government of Canada (GC)
5. Comparison of Evaluation Logic Model and Benefits Realization
6. What Evaluators can Learn from Benefits Realization Results Chain
7. Conclusions
Draft
12
June 19, 2007 22
6.What Evaluators / Logic Modellers CanLearn (1 of 2) …
1. Benefits Realization Results Chain includes different stakeholders,initiatives, and assumptions and risks impacting achievement ofdesired outcomes or benefits
2. BR Results Chain lays out the complex sequence of initiatives andoutputs/outcomes and lays out the accountabilities
3. BR Results Chain is more detailed and comprehensive than a LogicModel – it tells a more complete story consistent with programtheory
4. BR Results Chain is powerful tool for dealing with:
• horizontal initiatives, projects or programs, and
• new initiatives and ‘change’ initiatives
June 19, 2007 23
6.What Evaluators / Logic Modellers CanLearn (2 of 2 cont’d) …
5. Explicitly identify what project/program, department and externalstakeholders are expected to do to help achieve outcomes, i.e.,realize benefits
6. Explicitly identify underlying assumptions about how programactivities yield outputs and how outputs are transformed intooutcomes
7. Explicitly identify external factors (i.e., risks and barriers) that mayinhibit or accelerate program achieving outcomes and how to dealwith them
8. Colour code related initiatives to convey sense of urgency
Draft
13
June 19, 2007 24
7.Conclusions …
• Current Logic Models were developed in a ‘simpler age’
• Logic Models are useful for understanding link between programand outcomes, but today face real challenges – accountability,external factors, risks, horizontality
• BR Results Chains, coming at these challenges from a differentdirection, has developed techniques and processes for addressingthese challenges
• BR Results Chains are useful for understanding how to achievedesired outcomes
• Opportunity exists to modernize Logic Models and to increase theirvalue by incorporating aspects of BR Results Chains
June 19, 2007 25
Arthur Rabinovitch, CMCMaxar Consulting Inc.
(formerly withGovernment Consulting Services)
For further information on the BR methodology, please contact:
David HeathPrincipal
Fujitsu Consulting
613-238-2697
During June and July 2007
James McGibbonPrincipal
Fujitsu Consulting
613-238-2697 ext 4401