1

Thai Health Promotion Foundation

Outcome Mapping Seminar

Bangkok, Thailand

March 25, 2007

Terry Smutylotsmutylo@magma.ca

2

seminar objectives

Introduce Outcome Mapping

Help you decide if & when Outcome Mapping would be useful

Exchange experiences & ideas on applying the method

3

overview of presentation:

1. Evaluation in a complex world

2. Origins of Outcome Mapping

3. Core concepts

4. The 7 planning steps

5. OM in Monitoring & Evaluation

4

1.How do you experience planning, monitoring & evaluation?

2.How could M&E be made more useful and practical in your work environment?

CHECKING IN:

5

Accountability & Learning: A Balancing Act

6

Accountability & Learning: A Balancing Act

7

a shift from:

“Did you do what you promised to do?”

to…

“Are we on the right track?”

8

Evaluation to support innovation:Traditional Evaluations:

Judge success or failure

Measure against fixed goals

External for objectivity

Linear cause/effect models

Accountability to external

Accountability for control, blame

Evaluator controls evaluation

Engender fear of failure

Developmental Evaluations:

Provide feedback for improvement

New measures as goals evolve

Internal, integrated, interpretive

Seek to capture system dynamics

Accountability to values, commitments

Understand & respond strategically

Evaluator matches process to context

Feed hunger for learning

Adapted from: Patton, Michael Q., 2006, “Evaluation for the Way We Work”, The Nonprofit Quarterly, Spring.

9

Connecting Research To Well-Being

BeneficiariesNARO

RESEARCH MANAGERS

POLITICS•New Knowledge/ Ideas

POLICIES• Motivations

FUNDERS•Policies•Motivations•Money + Inputs

RESEARCHERS

USERS (Farmers & Families)

PRIVATE SECTORNational & InternationalRURAL SOCIOLOGISTS

Identification of opportunities & constraints

MINISTRY•Agricultural•Facilitation•Regulations

NGOs•Facilitate Adoption of Technology

NGOs•Environmental•Women Groups•Advocacy

DISSEMINATION•New Knowledge•Extension•Inputs

GOV/ MINISTRIES•Incentives to Facilitate Adoption of Technology

EXTENSION GROUPS•Government•NGOs, •Advisory services

FARMER ORGANIZATIONS•Identify problems•Dissemination LOCAL ORGs

•Leadership•Mobilization

PRIVATE SECTOR•Seed suppliers•Technical Assistance

POST PRODUCTION•Marketing•Transportation/ Shipping

FARMERS/ PRODUCERS•New Knowledge•Sharing •Motivation

RESEARCH INSTITUTES•Outside Community

NARO •Support Staff•Research Support

REG~L/ INTERN~L CENTRES•Information Technology

NGOs•Farmer training•Transfer of Technology

FARMER ASSOCIATIONS•Extension Services

OTHER RESEARCHERS•Universities•Technologies•Information FARMERS

•Farmer Orgs•Advocacy for research

ENVIRONMENT•National/ Regional•Money/ policy/ services

DONORS•Financial resources•Human resources

8-15 Years

10

the problem with « impact »

Impact Implies:

Cause & effect

Positive, intended results

Focus on ultimate effects

Credits a single contributor

Story ends when program obtains success

Development Implies:

Open system

Unexpected positive & negative results occur

Upstream effects are important

Multiple actors create results & need credit

Change process never ends

12

                                                                                                                               

©TOM Jochen Enterprises, Möckernstr.78 10965 Berlin

focus of outcome mapping

Inputs Activities Outputs Outcomes Impacts

Behavioural Changes

community capacity & ownership increases

program influence decreases

14

what is outcome mapping?

• A methodology for planning and assessing the social effects & internal performance of projects, programs, & organizations

15

a flexible, multiple-use tool

• Planning

• Monitoring

• Evaluation

16

What are we trying to

accomplish and how?

What do we want to learn?

What do we want to know?

17

key messages in

18

look at the bigger picture

• Seeing yourself as a part of a interconnected web of relationships and systems

19

recognizing that change is…

• Continuous

• Complex

• Non-linear

• Multidirectional

• Not controllable

20

embrace constant change

“It is not possible to see the same river twice.”

21

keeping your eyes wide open

• Being attentive along the journey is as important as the destination

22

Relying on Indicators can be dangerous:

1. lulling: create a false sense of security

2.corrupting: they can become our objectives

3.biased: are not use, user, context or value neutral; subject to interpretation

4.static: do not show complex, incremental change

5.misleading: may be chosen for ease of access or need to aggregate rather than relevance

23

Key concepts in Outcome Mapping:

behavior change as results

direct partners

sphere of influence

24

why behaviour changes?

• development is done by and for people

• a program can influence the achievement of outcomes - it cannot control them

• ultimately responsibility rests with the people affected

25

focus on direct partners

• Key concept is « boundary partners »

(the individuals, groups, and organizations you work with directly and try to influence)

26

spheres of influence

project / program

The rest of the world

= boundary partners

27

contribution not attribution

• your influence towards better world

• you can influence but not control change in your partners

28

CHECKING IN:

1. How would these ideas resonate in your work environment?

2. Where could a focus on behavior change be useful?

29

uses of outcome mapping

30

• PLANNING: articulate goals & define

activities

• MONITORING:track program performance & partners’ progress

• EVALUATION:design & conduct a use-oriented evaluation

primary uses

31

step 1: vision

Intentional DesignIntentional Design

32

I have a dream!

Martin Luther King, Jr.August 28, 1963

33

vision is a guide

Compass

Light house

Distant star

34

Vision describes the improved human, social, & environmental wellbeing to which the program is committed and contributing.

You are NOT accountable for achieving your vision!

35

vision in graphic form, Nagaland (India)

36

Dream boldly, Evaluate modestly

within the broad development

context

within the program’s

sphere of influence

design

evaluate

37

step 2: mission

Intentional DesignIntentional Design

38

The mission is that “bite” of the vision statement on which the

program is going to focus.

39

your mission is your “business”

• What do you do?

• Who do you do it with?

• Why do you work with them?

40

MISSION VISION

Beneficiary=

ChildrenThai

Health

Child MediaProgram

Government/ Policy maker

Educational Institute

Corporate/ Ad. Agencies

Board of TV/Radio Station

Academics/Psychologist on children

Child media Network

TV/Radio Producers Network

Book for children Network

= BP

strategies

outcomes

41

step 3: boundary partners

Intentional DesignIntentional Design

42

boundary partners

Those individuals, groups, and organizations with whom the program:

• interacts directly to effect change

• anticipates opportunities for influence

• engages in mutual learning

43

boundary partners have boundary partners

program program’s bp bp’s bp

44

CIDA

IDRC

BAIF

StateNGO

StateNGO

StateNGO

StateNGO

StateNGO

StateNGO

SHG Police CommunityLeaders

Families Banks PHCs

Swayamsiddha

45

moving from stakeholders...

46

...to boundary partners

project

47

Example: a network as a boundary

Researchers

Service providers

NGOs

Policy makers

Project intends to get them working together to solve a common problem

Expect: meet, listen to from each other

Like: organize an event together

Love: implement a joint initiative

48

step 5: progress markers

Intentional DesignIntentional Design

49

Being clear & specific:

Greater awareness…Better access to…Participate actively…More effective management…Empowered women…Reduced conflict…Increased collaboration…Gender sensitivity…

50

progress markers

✓ A graduated set of statements describing a progression of changed behaviours in the boundary partner

✓ Describe changes in actions, activities and relationships leading to the ideal outcome

✓ Articulate the complexity of the change process

51

progress markers (ladder of change)

High level of transformation

Actively engaged, learning, commitment

Early encouraging response to program, initial engagement

Love to see

Like to see

Expect to see

52

progress markers are graduated

move from initial to more profound changes in behaviour

show transformation in a single boundary partner

more informative than a single indicator

53

✓ Do not require consultant to write quantifiable indicators

✓ Show directionality of change

✓ Support learning function

✓ Can be monitored & observed from the start of the program

✓ Permit on-going assessment of partner’s progress (including unintended results)

✓ Provide basis for dialogue with partners

Some advantages of progress markers

54

Taken together, progress markers:

facilitate mid-course corrections and improvements

articulate the complexity of change

stimulate the program to consider how it can contribute to the most profound transformation possible

suggest the logic model of change

are NOT a checklist of accomplishments!

55

Example:

1. Attending self help group meetings2. Participating when abuse is discussed

Like to see women victims of domestic violence:

3. Educating self about combating domestic violence4. Setting an agenda for taking collective action 5. Raising funds for victim support services

Love to see women victims of domestic violence:

6. Taking action to protect self7. Lobbying police to change enforcement practices

Expect to see women victims of domestic violence:

56

OUTCOME CHALLENGE: The Program intends to see farmers and farmers’ committees which are fully engaged in the research process. They are participating in the design, management and monitoring of field trials; they regularly give researchers full and frank feed back on the technologies being tested; and they share their learning and experiences with extension agents and other farmers.

EXPECT TO SEE FARMERS AND FARMERS’ COMMITTEES:

1 Participating in the research in accordance with researchers’ guidance

2 Initiating contact with researchers

3 Continuously monitoring and reporting on their field trials

Example: AHI Outcome Journal

57

LIKE TO SEE FARMERS AND FARMERS’ COMMITTEES:

4 Frequently raising problems and questions with researchers

5 Keeping complete records on trials

6 Negotiating trial designs with researchers

LOVE TO SEE FARMERS AND FARMERS’ COMMITTEES:

7 Promoting the feedback process among other farmers

8 Carrying out jointly planned trials and constantly feeding back assessments of the results to researchers and extension agents

Observations – Sources of evidence

Contributing factors & actors

Planned use of, or response to, the above monitoring information

58

Checking In:

How would the organizations you work with react to the concepts of “Boundary Partners” and “Progress Markers”?

59

step 6: strategy maps

stage 1

Intentional DesignIntentional Design

60

6 types of strategies

causal persuasive supportive

Iaimed at individual

boundary partner

directinfluence

arouse new thinking;

build skills, capacity

on-going support

Eaimed at boundary partner’s

environment

alter the physical,

regulatory or information environment

broad information

dissemination; access to new

info

create / strengthen

peer networks

61

causal persuasive supportive

Iaimed at

individual boundary partner

Eaimed at boundary

partner’s environment

62

Value of strategy maps

• Clarify what the project is doing, with whom and why

• Anticipate the program’s influence on the BP

• Articulate the range, mix and logic of the strategies

• Detect strategy gaps and over extension of resources

• Encourage multiple interventions to increase effectiveness

•Suggest appropriate evaluation methods

63

step 7: organizational practices

Intentional DesignIntentional Design

64

organizational practices

keep learning

foster creativity & innovation

seek better ways to assist your partners

maintain your niche

maintain higher level support

build relationships

What you do as an organization to:

65

8 organizational practices

1. Prospecting for new ideas, opportunities, & resources

2. Seeking feedback from key informants

3. Obtaining the support of your next highest power

4. Assessing & (re)designing products, services, systems, and procedures

66

8 organizational practices

5. Checking up on those already served to add value

6. Sharing your best wisdom with the world

7. Experimenting to remain innovative

8. Engaging in organizational reflection

67

planning and assessment possibilities in OM

program partner

outcomes(behaviour changes in the

partners)implementation(interventions by the program)

relevance & viability(of the program)

contextual information

situational data

68

ongoing OM applicationshttp://www.outcomemapping.ca

Ghana Korea Switzerland

Kenya Ecuador United Kingdom

Madagascar Mexico Brussels

Mali Egypt Netherlands

Namibia Bhutan Australia

Uganda Honduras Sri Lanka

Zimbabwe Guatemala India

69

a few examples…

• Nagaland (India)

• Latin American Trade Network

• International Model Forests (Mexico, Russia, Chile)

• SchoolNet Namibia

• ADRA (Cambodia)

• ST2EEP (Zimbabwe)

70

The International Model Forest Network(Mexico, Russia, Chile)

71

Progress Markers for local communities

1. Participating in regular model forest meetings

2. Establishing a structure for cooperation

3. Acquiring new skills for managing model forests

4. Contributing resources to get the MF operational

5. Articulating a locally relevant vision for the MF

6. Promoting their MF nationally

7. Expanding the partnership

8. Calling upon external experts for advice

9. Requesting new opportunities for training

10. Publishing examples of benefits achieved through MF

11. Seeking out new partners for the MF

12. Obtaining funding from different national sources

13. Helping other communities establish MFs

14. Sharing lessons learned internationally

15. Influencing national policy debates on resource use

72

Comparing Results in 3 MF countries

ProgressMarker

Chile -Chiloe

Russia -Gassinsky

Mexico -Chihuahua

Mexico -Calakmul

Mexico -Monarch

Expect to see local communities:

1 ! ! ! ! !

2 ! ! ! ! !

3 ! ! ! ! !

4 ! ! ! ! !

Like to see local communities:

5 ! ! ! ! !

6 ! ! ! ! !

7 ! ! ! ! !

8 !

9 ! ! ! !

10 Potential !

11

12

Love to see local communities:

13

14 ! !

15 !

73

Another example…

SchoolNet Namibia Research

Conducted by the Council for Scientific and Industrial

Research (CSIR)

http://www.idrc.ca/en/ev-81099-201-1-DO_TOPIC.html

74

principles of use

Flexible: modular to be adapted to use & context

Complementary: use with other methodologies

Participatory: seeks dialogue and collaboration with partners

Evaluative thinking: culture of reflection, results oriented thinking, and promotes social & organizational learning

75

http://web.idrc.ca/en/ev-27705-201-1-DO_TOPIC.html

examples of OM use:

“Not everything that matters can be counted,and not everything that can be counted matters”

Albert Einstein