Www.easyoutcomes.org Creative Commons © 2007 Presentations in this powerpoint file This file...

www.easyoutcomes.org Creative Commons © 2007

Presentations in this powerpoint file

This file contains three presentations on Easy Outcomes1. Easy Outcomes: Introductory Presentation (P1)

2. Easy Outcomes: Summary for Decision Makers (P2)

3. Easy Outcomes: Guidelines for Drawing Outcomes Models (P3)


Presentation 1

Easy Outcomes Introductory Presentation (P1)


Easy outcomes

Introductory Presentation

www.easyoutcomes.org

This presentation is available from www.easyoutcomes.org P1 V1-0

http://www.easyoutcomes.org/



Origin of Easy Outcomes• Developed by an expert evaluator & strategist – Dr Paul Duignan• User friendly version of Systematic Outcomes Analysis – a

comprehensive applied outcomes, evaluation & strategy approach (www.systematicoutcomesanalysis.org)

• Implemented using DoView – outcomes visualization software (www.doview.com)

• We’re interested in hearing where Easy Outcomes is being applied and answering any questions you have [email protected]

• This material is covered by Creative Commons copyright – you can use it for most commercial or non-commercial purposes just

with acknowledgement (check web site for further information)

http://www.systematicoutcomesanalysis.org/

http://www.doview.com/


Easy outcomes lets you

• Work out your outcomes, the steps needed to get to them & to prioritize your activity

• Find out which outcomes you can and can’t measure and track them

• Figure out which evaluation questions to ask & if you can or can’t answer them

• Decide who should be accountable for what in delegation or contracting


How is Easy Outcomes done?• A template in DoView outcomes software is filled out *• It’s based around drawing an outcomes model (also called a program

logic, results chain, strategy map, ways-means diagram) using a set of guidelines

• Once the outcomes model has been drawn, you put onto it indicators, evaluation questions etc.

• You the analyse what’s appropriate, feasible and affordable (e.g. for outcomes evaluation you use the 7 possible types of outcome evaluation identified in Easy Outcomes)

• Once the work is done you are easily able to quickly overview the whole monitoring and evaluation planning

* If one wanted to, Easy Outcomes can be implemented using standard office software but it would take longer to do so.


The process1. Getting started

2. Draw an outcomes model

3. Set strategic priorities for action (if doing strategic planning)

4. Put indicators onto your model

5. List indicator projects

6. Put evaluation questions onto your model

7. Decide which evaluation questions to try to answer

8. Work out what outcome evaluation is possible

9. List evaluation projects

10. Work out what economic evaluation to do

11. Work out overall evaluation scheme

12. Work out delegated or contracting accountabilities


1. Getting started

• There’s only one step absolutely required in Easy Outcomes – drawing an outcomes model

• You can do all or some of the other steps and sometimes people do them in a different order

• A DoView template file for doing the process is available at www.easyoutcomes.org

• You can get a trial version of DoView from www.doview.com




2. Drawing an outcomes model

• Draw your model using the Easy Outcomes guidelines for outcomes models (www.easyoutcomes.org/guidelines/outcomesquidelines.html)

• At this stage don’t worry if you can’t measure some of the steps or outcomes

• Also, don’t worry if you can’t prove that you changed some of the steps or outcomes

• Example: an outcomes model for a social event (a party) (usually there would be more than one diagram in an outcomes model)


3. Set strategic priorities for action

• The top levels of you model will show your vision and mission• Work out the priority areas you’ll attempt to change in the next

planning period• If you’re just doing monitoring and evaluation, you may not do this

step at the start• Example: The strategic focus has been identified for organizing

future parties (the darker the blue, the higher the priority)


4. Put indicators onto your model

• Indicators are routinely collected measures of outcomes or steps• You may or may not be collecting these indicators at the moment• Use the picture you now have of which indicators you are already

(or are planning to) collect to think about your indicator coverage• Example: Potential indicators mapped onto the outcomes model


5. List indicator projects

• List any indicator projects you want to undertake• These may be projects only you work on or ones you do with other

stakeholders who also want to measure the same indicators• Example: Under each indicator project there is a list of the indicators

which will be developed


6. Put evaluation questions onto your model

• Put possible evaluation questions next to the outcomes or steps they apply to

• Put both outcome evaluation questions (proving that high level outcomes have been changed by the program and non-outcome evaluation questions (asking about how to improve the program or about how the process works)

• Don’t worry whether or not you can answer these evaluation questions at this stage

• Example: Evaluation questions mapped onto the outcomes model


7. Decide which evaluation questions to try to answer

• Don’t just assume that you have to, or will be able to, answer high level outcome evaluation questions. If you can that’s good, but often it would be better for your funding or controlling organization to undertake such outcome evaluation (e.g. by intensively evaluating a pilot program rather than demanding all programs have high level outcome evaluation)

• Example: Evaluation question analysis looking at which evaluation questions you’re going to attempt to answer. The evaluation question ‘Did the party make the guests happy?’ will be examined in more detail in the next step.


8a. Work out what outcome evaluation is possible

• There’s only a limited range of possible outcome evaluation designs and Easy Outcomes currently divides these into seven possible types

• Work out whether any of these are possible for your high level evaluation question(s) (if you don’t know how to do this, involve someone else who does) (more information: www.systematicoutcomesanalysis.org/oiiwa/model/designs.html)

• Example: Analysis of the appropriateness, feasibility and affordability of the first three possible high level outcome evaluation designs to answer the evaluation question ‘Did the party make the guests happy?’


8b. Work out what outcome evaluation is possible (cont.)

• Note that only some of the possible high level outcome evaluation designs will give you information needed in later stages of Easy Outcomes. In particular, certain types of economic analysis can only be done if there is a quantitative measures of the effect of your activity on high level outcomes (only the first four designs have the potential to provide this)

• Example: Appropriateness, feasibility and affordability of last four high level outcome evaluation designs


9. List evaluation projects

• Set out your evaluation projects and list under them the evaluation questions they’ll be answering

• This helps you to be very clear about exactly which evaluation question(s) are being answered by a particular evaluation project (very helpful in large scale evaluations)

• Example: Evaluation projects and the evaluation questions they will attempt to answer


10. Work out what economic evaluation to do

• There are only a limited range of possible economic evaluation designs and Easy Outcomes currently divides these into ten possible types

• Work out whether any of these are possible (if you don’t know how to do this, involve someone else who does) (more information: www.systematicoutcomesanalysis.org/oiiwa/model/economic.html)

• Example: Two examples of types of economic analysis which could

be done in this case.


11. Work out overall evaluation scheme

You can chose between two major overall evaluation schemes:

1. Doing outcome evaluation on the whole program roll-out

2. Just doing outcome evaluation on a pilot and then making sure that the full program roll-out is applying best practice learnt in the pilot (but not doing outcome evaluation on the full roll-out)

Example: Shows the two possible major overall evaluation schemes for the party example


12. Work out delegated or contracted accountabilities

There are three possible types of contracting in Easy Outcomes:

1. Contracting for outputs2. Contracting for outputs and ‘Managing for outcomes’ *3. Contracting for not fully controllable outcomes *It’s very important to be clear about which kind of contracting you’re doing.

Example: The waiting staff are contracted to be accountable for indicators related to the food and drink being adequately distributed (outputs) but not further up the outcomes model

* Type 2. does not involve being accountable for high level outcomes. Type 3 occurs, for example, in the private sector when a CEOs gets a bonus related to the organization’s share price.


Conclusion: Easy Outcomes lets you

• See what evaluation and monitoring is planned (outcome, non-outcome & economic evaluation) and make decisions about what should be done

• See if monitoring indicators are measuring the important (not just the measurable)

• See what’s appropriate, feasible and affordable for high level outcome evaluation

• Link monitoring and evaluation to strategic planning• Be clear about who is accountable for what in delegation or

contracting


Further informationFurther information available from:


More detailed information available from

www.systematicoutcomesanalysis.org

Information on DoView outcomes software from

www.doview.com

Comments to: [email protected]





Presentation 2

Easy Outcomes: Summary for Decision Makers (P2)


Easy outcomes

Summaryfor decision makers

(Representatives, Parliamentarians & Boards, CEOs & Managers)






The decision maker’s problem

• Having to make decisions in a world of uncertainty• Wanting more and more robust evaluation and monitoring information

for guidance• Knowing that evaluation and monitoring are costly and when linked to

accountability may have unintended consequences• No quick way of making decisions about which evaluation and

monitoring to do• Being unclear about how much uncertainty there is in evaluation &

monitoring information they’re given• Lack of effective organizational linkages between evaluation &

monitoring, strategy development and delegation & contracting


The risk

• Decision makers demand more and more evaluation and monitoring• They’re given evaluation and monitoring information but have no real

idea of the actual level of uncertainty in that information• They end up doing the easily measurable (but often badly measured)

rather than the strategically important• And they have no way of deciding on the trade-off between spending

more and more money & effort (including organizational preoccupation) on monitoring and evaluation rather than on program delivery


Examples of mistakes

• Demanding only certain types of evaluation design be used (e.g. experimental outcome evaluation) regardless of the appropriateness, feasibility or affordability of these designs in particular cases

• Demanding all programs be evaluated and then getting weak evaluations of many programs rather than robust evaluations of a selected few which could be done if evaluation resources and effort were more focused

• Insisting that players ‘contract for outcomes’ or be ‘accountable for outcomes’ and exposing both contractors and contractees to risks in situations where it is difficult to actually attribute change to a

particular player


A solution – Easy Outcomes lets decision makers

• See what evaluation and monitoring is planned (outcome, non-outcome & economic evaluation) and make decisions about what should be done

• See if monitoring indicators are measuring the important (not just the measurable)

• See what’s appropriate, feasible and affordable for high level outcome evaluation

• Link monitoring and evaluation to strategic planning• Be clear about who is accountable for what in delegation or

contracting


It can also show that providers are ‘Managing for outcomes”

• Easy Outcomes also lets decision makers quickly assess whether a provider is ‘Managing for outcomes’

• A peer reviewed Easy Outcomes analysis gives a decision maker some assurance that the provider is attempting to Manage for outcomes*

* Managing for outcomes is different from being held accountable for achieving not fully controllable outcomes


It provides a standard format

• Think of Easy Outcomes as a standard format for setting out outcomes, monitoring and evaluation planning

• This is particularly useful where a number of programs are reporting to a decision maker and means the decision maker or his or her staff do not have to grapple with many different reporting formats

• Not having a standard format for reporting on outcomes and evaluation planning is a bit like not having a standard format for reporting accounting information – likely to lead to a great deal of additional work as you have to come to terms with different frameworks


How is Easy Outcomes done?• A template in DoView outcomes software is filled out *• It’s based around drawing an outcomes model (also called a program

logic, results chain, strategy map, ways-means diagram) using a set of guidelines

• Once the outcomes model has been drawn, you put onto it indicators, evaluation questions etc.

• You do some analysis of what’s appropriate, feasible and affordable (e.g. for outcomes evaluation you use the 7 possible types of outcome evaluation identified in Easy Outcomes)

• Once the work is done you are easily able to quickly overview the whole monitoring and evaluation planning

* If one wanted to, Easy Outcomes can be implemented using standard office software but it would take longer to do so.


Since strategic advantage often lies in difficult to measure areas…

• Strategic advantage in the public, private and community sectors often lies in ‘hard to measure’ areas

• Well measured areas where causation has been established through monitoring and evaluation are usually full of players and activity

• In the private sector this makes them commodity markets, in the public & community sectors they’re the places where much activity is already being focused


…It’s important that you get better control of evaluation, monitoring and

outcomes planning

• An unsophisticated ‘go forth and evaluate’ or ‘just do outcomes evaluation’ approach will not work (you’ll end up doing the measurable, not necessarily the strategic)

• The Easy Outcomes approach means that you and your staff will know what is known, what is not yet known and what can be known to aid decision making

• If you’re going to get optimal strategic intelligence for decision making you need to make the decisions about how to deploy your limited monitoring and evaluation resources


Put the outcomes models at the heart of your organization

• The same Easy Outcomes model that has been built for monitoring and evaluation can also be used for:

– Defining your vision and mission– Priority setting in strategic planning– Identifying the outcomes of your organizational units– Organizing the evidence base around your interventions– Talking to outside stakeholders about what you’re doing– Delegating and contracting


Example of an Easy Outcomes analysis

• The following diagrams show a simple Easy Outcomes analysis using the example of a social function (party)

• This example is discussed in more detail in the Easy Outcomes Introductory Presentation* and on the Easy Outcomes website

* Available from the Easy Outcomes website


The process• Draw an outcomes model• Set strategic priorities for action• Put indicators onto your model• List indicator projects• Put evaluation questions onto your model• Decide which evaluation questions to try to answer• Work out what outcome evaluation is possible• List evaluation projects• Work out what economic evaluation to do• Work out overall evaluation scheme• Work out delegated or contracting accountabilities


Example: An Easy Outcomes Analysis for a social event (party)


Origin of Easy Outcomes• Developed by an expert evaluator & strategist – Dr Paul

Duignan• User friendly version of Systematic Outcomes Analysis – a

comprehensive applied outcomes, evaluation & strategy approach (www.systematicoutcomesanalysis.org)

• Implemented using DoView – outcomes visualization software (www.doview.com)

• We’re interested in hearing where Easy Outcomes is being applied and answering any questions you have

• This material is covered by Creative Commons copyright – you can use it for most commercial or non-commercial purposes just

with acknowledgement (check web site for further information)









www.doview.com






Presentation 3

Easy Outcomes: Guidelines for Drawing Outcomes Models for Easy Outcomes (P3)


Easy outcomes

Guidelines for drawing outcomes models for Easy Outcomes






Guidelines for drawing outcomes models

• Easy Outcomes provides guidelines for drawing outcomes models (http://www.easyoutcomes.org/guidelines/outcomesguidelines.html)

• Using the guidelines means that the outcomes models will be able to be used for all steps in the Easy Outcomes process

• In addition, external stakeholders usually find Easy Outcomes models very accessible and useful


The guidelines1. Use outcomes not activities2. Draw the ‘cascading set of causes in the real world’3. Don’t force models into horizontal levels4. Don’t make ‘siloed’ outcomes models5. Singular not composite steps and outcomes6. Keep outcomes short7. Put outcomes into an hierarchical order8. Include all relevant steps needed to cause an outcome9. Keep measurement separate from steps and outcomes10. Put a ‘value’ in front of steps and outcomes11. Make just the right number of ‘slices’ 12. An organization does not necessarily need a single high level

outcome13. Include both current and future priorities


1. Outcomes not activities

Activities can be changed into outcomes just by changing the way they are written.


2. Cascading set of causes in the real worldDraw all of the steps (causes) leading to high level outcomes in the outside world. Drill down as far as you like. Draw the full outcomes model as on the left, don’t limit yourself just to measurable outcomes (middle model) or those that it can be proved have been changed by a player (attributable outcomes – as in the right hand model where only outcomes which it can be proved were caused by a player have been included in the model).


3. Don’t force models into horizontal levelsDifferent outcomes models will have more or less steps at a particular level, don’t force steps into levels or restrict the number at a particular level (as has been done in these diagrams). Outputs can be identified on your model but they don’t have to lie at a particular level.


4. Don’t make ‘siloed’ outcomes models

In the ‘siloed’ model on the left, lower level steps are only allowed to cause a single high level outcome. In the real world, a lower level step may well cause more than one high level outcome (as in the model on the right shows where the blue step causes both the blue and the yellow higher level outcome).


5. Singular not composite outcomes

Make sure that outcomes are not ‘composite’ like the one on the left hand side which contains both an ‘end’ (improve father-son relationships) and a ‘means’ (teaching them to fish together). Composite outcomes lock you permanently into using one strategy to get an outcome. If by, through, to, in order to or similar terms appear in an outcome they show that it is a composite outcome. Divide the composite outcome into two or more outcomes and open up the possibility that there is another way (strategy) of getting to the outcome (e.g. the blue box in the diagram on the right).


6. Keep outcomes and steps short

Keep outcomes and steps short, as in the one on the right rather than the one on the left. If you need to, keep an additional longer description of the outcome separately. For every additional word you use in an outcome name, you will lose or confuse some readers.


7. Put outcomes into an hierarchical order

The outcomes on the left are not in an hierarchical order. The ones on the right are. You can tell that there is an hierarchical order to the outcomes on the right because if you could magically achieve the top one, you would not bother intervening to do the lower ones.


8. Include all relevant steps needed to cause an outcome or step

The model on the left is incomplete because it does not include an essential step (the red one in the model on the right) which is needed to cause the outcome above to occur. Include all notable steps needed. You need to use your common sense, because there will be steps which are needed but which can be taken for granted in some situations (e.g. electricity, telephone systems) you don’t need to include such ‘taken for granted’steps where it can be assumed they will be present.


9. Keep measurement separate from steps & outcomesFirst draw your outcomes model without worrying about whether you can measure the steps and outcomes (as in the model on the left). Then insert measurements as indicators (as in the model on the right). If you put measurements into your actual outcomes model you cannot deal with the fact that particular ways of measuring an outcome may be inaccurate. In a few cases, where measurement causes something to happen (e.g. audit processes), they may also appear as steps in their own right.


10. Put a ‘value’ in front of steps & outcomes

Put a ‘value’ such as successful, sufficient, effective or accurate in front of your outcomes and steps. You don't need to define this when you are first building your outcomes model. By putting this value in you often limit long discussions about the exact amount of the step or outcome that is needed. If you take too much time up with such discussions when first drawing your model in a workshop situation you’re likely to run out of time). The question of how much is sufficient etc can then become the subject of an evaluation project if necessary.


11. Make just the right number of ‘slices’

You model should be made up of just the right number of ‘slices’ needed to represent the world of outcomes and steps in which you are intervening. The actual world of causes is a multi-dimensional world which you need to somehow represent in a finite set of dimensions. Your challenge is to ‘slice’ that multi-dimensional world up into a set of slices which make sense to you and your stakeholders. For instance below you can see a set of three slices (there would be a set of outcomes below each of these). Each one is based on a different conceptual level: national, locality and individual level. It may take some time to work out the best set of slices to use in a particular model. Just try developing different slices as they occur to you and then as you progress in drawing your model, get rid of duplicate steps and outcomes and join or separate them as you think appropriate.


12. An organization does not necessarily need a single high level outcome

Where an organization has been delegated or contracted to intervene in more than one area, it’s appropriate for it to have separate outcomes models for each area. On the left below is an artificial attempt at a single outcomes model. Having separate outcomes models for each area (as on the right) means that the organization is in a better position to collaborate with outside organizations working in the same area. For instance, the two green outcomes on the extreme right (which are the focus of another organization) can be much more easily integrated into the outcomes model on this topic when it has been set up as in the right hand rather than the left hand model.


13. Include both current and future prioritiesYour outcomes models should include all relevant causes in the real world and should not be limited just to your current priorities (determined by resourcing or other constraints). The model on the right is limited to only priorities and so does not give a full strategic view of the world in which you’re intervening. Highlighting priorities onto a full outcomes model (as in the middle example) is a better approach and works well in the public sector where public sector employees can give ‘free and frank’ advice about the world as it is and elected officials can then determine current priorities on this model.


Conclusion

• Drawing outcomes models using the guidelines makes sure that they will work with all of the steps in Easy Outcomes.

• More information on the outcomes model standards is available http://www.easyoutcomes.org/guidelines/outcomesguidelines.html

• If you have any comment on these standards please let us know: [email protected]

http://www.easyoutcomes.org/guidelines/outcomesguidelines.html







www.doview.com





Www.easyoutcomes.org Creative Commons © 2007 Presentations in this powerpoint file This file...

Documents

Transcript of Www.easyoutcomes.org Creative Commons © 2007 Presentations in this powerpoint file This file...