Outcome Harvesting Ricardo Wilson-Grau 9 February 2012 Beirut, Lebanon.

Outcome Harvesting

Ricardo Wilson-Grau

9 February 2012Beirut, Lebanon

[email protected]

The next two hours

Purpose: Introduce you to the principles of Outcome Harvesting as a monitoring and evaluation tool for Outcome Mapping.

Intended results: You will understand the essential principles of Outcome Harvesting and be able to decide when and if it could be useful in your Outcome Mapping work.

[email protected]

Outcome Harvesting

A tool for monitoring and evaluating the results of development interventionsDeveloped since 2003 by me and my colleagues Claudia Fontes, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, and Wolfgang Richert, and currently Barbara Klugman and Natalia Ortiz.

[email protected]

1. International social change networks

2. Funding agencies

http://www.pso.nl/

Outcome Harvesting

0. Focus on utilisation1.Design the harvest2.Review documentation and draft outcomes3.Engage with informants4.Substantiate 5.Analyse, interpret6.Support use of findings

Outcome Harvesting to meet needs

Monitoring Evaluation

“I can honestly say that not a day goes by when we

don’t use those evaluations in one way or

another.”

[email protected]

Outcome Harvesting


[email protected]

0. Focus on usefulness

Primary intended users

Principal intended uses

Useful M&E questions

[email protected]

Examples

Management team requires information about programme effectiveness in order to make funding decisions for the next three years.

[email protected]

Examples

Management team requires information about programme effectiveness in order to make funding decisions for the next three years.To what extent did the outcomes we influenced in 2009-2011 represent patterns of progress towards our strategic objectives?Was our investment in the activities and outputs that contributed to our 2009-2011 outcomes cost-effective?

What? So what?

Focus on usefulness compared to OM

INTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission

STEP 3: Boundary Partners

STEP 4: Outcome Challenges

STEP 5: Progress Markers

STEP 6: Strategy Maps

STEP 7: Organizational Practices

STEP 8: Monitoring Priorities

STEP 9: Outcome Journals

STEP 10: Strategy Journal

STEP 11: Performance Journal

OUTCOME &PERFORMANCE MONITORING

INTENTIONAL DESIGN

STEP 12: Evaluation Plan

EVALUATION PLANNING

My comparison

INTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING

[email protected]

Outcome Harvesting

0. Focus on utilisation1.Design the harvest

The M&E balancing act

The OM concept of “outcome” as changes

in social actors

[email protected]

Useful Evaluation QuestionTo what extent did the outcomes we influenced in 2009-2011 represent patterns of progress towards our strategic objectives?

Data required to answer In the light of monitoring and evaluation questions, what data is required and how and from whom will it be obtained?

SOCIAL ACTOR

Individual

OrganisationInstitution

Behaviour Relationships

Actions, activities

Policies and

practices

CHANGE

Group or community

To be harvested

[email protected]

Outcome defined

A. An observable and significant change in a social actor’s behaviour, relationships, activities, actions, policies or practice that has been achieved. ..

B. … and that has been influenced by the change agent.

[email protected]

Outcome example

A. On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals.

B. The Government Accountability Project got the incriminating information to the media, leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign.

Source: Evaluation of the Ford Foundation’s Global Governance Programme (2009)

Design of Outcome Harvest compared to OM

INTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING

My comparisonINTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING

[email protected]

0. Focus on utilisation1.Design the harvest2.Review documentation and draft outcomes

Outcome Harvesting

When?

Who?

What?

Where?

[email protected]

Exercise

Please take five minutes to read this short, one-page case study from AWID, the Association for Women’s Rights in Development. Then, with the person seated next to you, identify one outcome:A.Who changed what, when and where?

[email protected]

MGD3 Fund

Description: During 2011, the Dutch government decided to allocate an additional €12 million to the MGD3 Fund and to launch a new phase of support for the MGD3 with €70 million for the Fund for Leadership Opportunities for Women (FLOW) to strengthen the rights and opportunities for women and girls.The formulation of an outcome is as specific, verifiable and as detailed as makes sense for the primary intended users and their principal uses.

[email protected]

Exercise

Now please identify how AWID contributed to it.

A.Who changed what, when and where? B.What activities and outputs plausibly contributed to the change in the social actor, however partially, indirectly and even unintentionally?

[email protected]

MGD3 Fund

Contribution of AWID: AWID compiled the document "Mid-term Summary of the MDG3 Fund Value Added and Outcomes", which was used by the Dutch Ministry in its internal advocacy for the replenishment of the MDG3 Fund. This material along with AWID’s coordination efforts among MDG3 grantees also supported the Dutch gender lobby platform with a strong advocacy tool that they used to lobby the Dutch government to commit these additional resources for women’s rights organizations around the world.

[email protected]

Other information?

Significance of the outcome Collaboration with other social actors Contribution of other actors and factors History Context Evidence of impact on people’s lives And so forth – it all depends on the information

required to answer the M&E questions.

[email protected]

For example

Outcome’s significance: In the light of the impact to which you wish to contribute, why is this outcome important?

[email protected]

Outcome example


B. The Government Accountability Project got the incriminating information to the media, leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign.

C. The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself and it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal.

Reviewing documentation and drafting outcomes compared to OM

INTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING


STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING

[email protected]

Outcome Harvesting

0. Focus on utilisation1.Design the harvest2.Review documentation and draft outcomes3.Engage with informants

3. Engage with informants Email Skype or telephone In person

[email protected]

MGD3 Fund

Comment [RW-G1]: Can you be more specific? Who in government? Which agency?

Comment [RW-G1]: Can you be more specific? Who in government? Which agency?

Description: During 2011, the Dutch government decided to allocate an additional €12 million to the MGD3 Fund and to launch a new phase of support for the MGD3 with €70 million for the Fund for Leadership Opportunities for Women (FLOW) to strengthen the rights and opportunities for women and girls.

[email protected]

Rita FundComment [RW-G1]: Who created this fund?When was it created? Specifically where was it created?

Comment [RW-G1]: Who created this fund?When was it created? Specifically where was it created?

Comment [RW-G2]: Is this an appropriate characterisation of “funding gap”?

Comment [RW-G2]: Is this an appropriate characterisation of “funding gap”?

Comment [RW-G3]: Did AWID do something more active to influence the creation of the Fund?

Comment [RW-G3]: Did AWID do something more active to influence the creation of the Fund?

Description: The Rita Fund is created in the United States. The Rita Fund is a woman’s fund which strives to respond to the “funding gap” between donors’ interest and their actual funding by creating a reliable non restrictive funding source for women’s funds operating worldwide.Contribution of AWID: AWID’s report Where is the Money for Women’s Rights?, published in 2008, was the source of information and inspiration from which came the idea to create the Rita Fund.

Outcome Harvesting to date

[email protected]

How much or little detail?

Depends on the users, uses and M&E questions.Can be one sentence as in the example or a page or more….

[email protected]

Outcome example


B. The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself and it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal.

C. The Government Accountability Project got the incriminating information to the media, leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign.

Title: On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals.Outcome: On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of wide-ranging scandals. Among them: Wolfowitz’s companion, Shaha Riza, received salary raises far in excess of those allowable under Bank rules; Riza received a questionable consulting position with a U.S. defense contractor in 2003 at his direction that resulted in State and Defense Department inquiries; Juan Jose Daboub, Bank Managing Director and Wolfowitz-hire, attempted to remove reference to “family planning” from the Bank’s new health sector strategy; Wolfowitz’s office was responsible for weakening a “climate change” strategy document; Bank Senior Management delayed reporting to Bank staff that a fellow staffer had been seriously wounded in a shooting in Iraq; Bank lending to Africa was deficient and delayed and could only meet lending targets if new loans were brought to the Board for approval without the proper formulation procedures; and that Wolfowitz was trying to broaden the Bank’s portfolio in Iraq over Board opposition, among other disclosures. Within a year, four controversial Wolfowitz appointees had left the Bank, including special advisors Robin Cleveland and Kevin Kellems, General Counsel Ana Palacio and Suzanne Folsom, the director of the Bank’s Department of Institutional Integrity (INT).Contribution of the grantee: The Government Accountability Project was the recipient of anonymously sent internal World Bank memos and tips relevant to many of the disclosures leading to the Wolfowitz resignation, among them, information that Wolfowitz was negotiating a contract with the resident Iraq Country director, in violation of the Bank’s Articles of Agreement; an email about the shooting of a Bank employee in Iraq and the Bank’s failure to follow its protocol for informing staff about these injuries; payroll records of Shaha Riza; internal documents showing that Daboub, the Wolfowitz-appointed Managing Director, had instructed a team of Bank specialists to delete all references to “family planning” from the proposed Country Assistance Strategy for Madagascar; internal documents indicating that Daboub had ordered the toning down of references to “climate change” in Bank environmental strategy papers; etc. GAP was directly responsible for getting this information to the media – through press releases, leaking documents or corroborating information – leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign. As The Economist wrote on April 12, 2007: “As he prepares to welcome ministers to the bank's spring meetings in Washington, DC, on April 14th and 15th, Mr. Wolfowitz's own toes … may feel a bit toasty. For that, he can thank the Government Accountability Project (GAP), a vibrant participant in America's civil society.”Significance: The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself, even as its senior officials demand good governance to borrowing countries. In the wake of the scandal, it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal.

[email protected]

A storyOn May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of wide-ranging scandals. As The Economist wrote on April 12, 2007: “As he prepares to welcome ministers to the bank's spring meetings in Washington, DC, on April 14th and 15th, Mr. Wolfowitz's own toes … may feel a bit toasty. For that, he can thank the Government Accountability Project (GAP), a vibrant participant in America's civil society.”Wolfowitz’s companion, Shaha Riza, received salary raises far in excess of those allowable under Bank rules; Riza received a questionable consulting position with a U.S. defense contractor in 2003 at his direction that resulted in State and Defense Department inquiries; Juan Jose Daboub, Bank Managing Director and Wolfowitz-hire, attempted to remove reference to “family planning” from the Bank’s new health sector strategy; Wolfowitz’s office was responsible for weakening a “climate change” strategy document; Bank Senior Management delayed reporting to Bank staff that a fellow staffer had been seriously wounded in a shooting in Iraq; Bank lending to Africa was deficient and delayed and could only meet lending targets if new loans were brought to the Board for approval without the proper formulation procedures; and that Wolfowitz was trying to broaden the Bank’s portfolio in Iraq over Board opposition, among other disclosures. The Government Accountability Project was the recipient of anonymously sent internal World Bank memos and tips relevant to many of the disclosures leading to the Wolfowitz resignation, among them, information that Wolfowitz was negotiating a contract with the resident Iraq Country director, in violation of the Bank’s Articles of Agreement; an email about the shooting of a Bank employee in Iraq and the Bank’s failure to follow its protocol for informing staff about these injuries; payroll records of Shaha Riza; internal documents showing that Daboub, the Wolfowitz-appointed Managing Director, had instructed a team of Bank specialists to delete all references to “family planning” from the proposed Country Assistance Strategy for Madagascar; internal documents indicating that Daboub had ordered the toning down of references to “climate change” in Bank environmental strategy papers; etc. GAP was directly responsible for getting this information to the media – through press releases, leaking documents or corroborating information – leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign. The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself, even as its senior officials demand good governance to borrowing countries. In the wake of the scandal, it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal.Within a year, four controversial Wolfowitz appointees had left the Bank, including special advisors Robin Cleveland and Kevin Kellems, General Counsel Ana Palacio and Suzanne Folsom, the director of the Bank’s Department of Institutional Integrity (INT).

[email protected]

One liners

1. In December 2005, the United Nations issues a groundbreaking whistleblower protection policy.

2. In August, 2006, GAP, in tandem with the World Bank Staff Association, stopped an attempt by then Bank President Paul Wolfowitz’s office to hurry through a flawed draft whistleblower protection policy.

3. In January, 2007, the Board of Executive Directors of the African Development Bank (AFDB) approves a whistleblower protection policy.

4. On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals.

5. The World Bank adopts recommendations for improving the Bank’s investigative unit INT and the Director of INT resigns.

6. In June 2008, the World Bank passes an improved whistleblower protection policy.

Engaging with informants compared to OM

INTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING


STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING

[email protected]

Outcome Harvesting

0. Focus on utilisation1.Design the harvest2.Review documentation and draft outcomes3.Engage with informants4.Substantiate

Present the final outcome formulation to one or more credible (independent, knowledgeable) person(s) and ask them to go on record with their opinion:

1. To what degree you are in agreement with the description of the Dutch government’s decision to allocate an additional €12 million to the MGD3 Fund and to launch a new phase of support for the MGD3 with €70 million for the Fund for Leadership Opportunities for Women (FLOW)

[ ] Fully agree[ ] Partially agree[ ] DisagreeComments if you like:

2. How much do you agree with the description of how AWID influenced the Dutch government’s decision? [ ] Fully agree[ ] Partially agree[ ] DisagreeComments if you like:

Case studiesTo enhance the understanding and credibility of the findings.

Substantiation compared to OMINTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING


STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN

STEP 12: Evaluation Plan??

EVALUATION PLANNING

[email protected]

Outcome Harvesting

0. Focus on utilisation1.Design the harvest2.Review documentation and draft outcomes3.Engage with informants4.Substantiate 5.Analyse, interpret

[email protected]

Organise the outcomes so that they can be interpreted in ways that will enable you to answer the Outcome Harvesting questions.

To what extent did the outcomes we influenced in 2009-2011 represent patterns of progress towards our strategic objectives?

Classify the outcomes according to strategic objectives, country or region, year…

Excel Database

Access databaseAccess Database

Policy

change

Legislature passes women’s right to

land inheritanc

e law

Over the past 5

years, in 90% of the cases the

Land Commissi

on has given to widows the titles of their

late husbands’

land

Practice

change

Time

Women’s group

launches campaign

+

Widow takes late husband’s family to court

+Court decides against widow

Late husbands’

families murder three

widowsFor the first time, press

runs story on women’s land

inheritance

+Legislator drafts law

+

Finance committee

discusses draft law

+

President declares

women's land inheritance a

right

+

Agriculture committee

discusses draft law

+

Ten widows take late

husbands’ family to court

+

Land Commission

approves regulations for

women’s inheritance of

deceased husband’s land

+

Religious leader

declares women's land inheritance a

right

+

Women’s group initiates legal aid for

widows

+

Court decides landmark case

in favour of widow

+

Agricultural union admits widows as members

+

Seeking processes, patterns, trends

Theory of change

RESULTS

SOCIAL ACTORS

ASSUMPTIONS

MISSION

STRATEGIES

IMPACT

[email protected]

Systems approach

Analyse, interpret compared to OM

INTENTIONAL DESIGN

STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN


EVALUATION PLANNING


PLANNING


STEP 1: Vision

STEP 2: Mission











INTENTIONAL DESIGN

EVALUATION

[email protected]

Outcome Harvesting


6. Support for use of findings

My summary comparison between

Outcome Harvesting and

Outcome Mapping

Outcome Harvesting

0. Focus on utilisation STEP 8: Monitoring Priorities1. Design the harvest STEP 9: Outcome Journals 2. Review documentation and draft outcomes

STEP 9: Outcome Journals STEP 10: Strategy Journal?STEP 11: Performance Journal?

3. Engage with informants STEP 9: Outcome Journals 4. Substantiate STEP 9: Outcome Journals?5. Analyse, interpret INTENTIONAL DESIGN,

EVALUATION6. Support use of findings

Many thanks! Do you have questions?

ricardo wilson-grau consultingRua Marechal Marques Porto 2/402, Tijuca, Rio de Janeiro, CEP 20270-260,

BrasilTelephone: 55 21 2284 6889; Skype: ricardowilsongrau

Direct via VOIP, dialing locally from: The Netherlands (or Europe): 31 71 302 0429

USA or Canada: 1 347 404 5379

S.M.A.R.T. outcomes

Specific ExampleEach outcome is formulated in sufficient detail so that a primary intended user without specialised subject or contextual knowledge will be able to understand and appreciate what changed. Who changed what, when and where?

In 2008-2009, the Norwegian Ministry of Foreign Affairs replenished its Budget Line on Gender Equality to fund the implementation of their Women’s Rights Program, which included significant funding for women’s organizations.

SMART outcomes

Measurable ExampleThe description of the outcome provides objective, verifiable quantitative and qualitative information, independent of who is collecting data. How much? How many? When and where did the change happen?

In 2008-2009, the Norwegian Ministry of Foreign Affairs replenished its Budget Line on Gender Equality to fund the implementation of their Women’s Rights Program, which included significant funding for women’s organizations.

SMART outcomes

Achieved ExampleBy AWID while perhaps not fully attributable to AWID. A plausible relationship, a logical link between the outcome and what AWID did that contributed to it. What did AWID do, when and where that contributed — wholly but probably partially, indirectly or indirectly, intentionally or unexpectedly?

WITM had a session on funding at CSW that generated important discussions and sparked conversations with various partners in the movement as well as bi-laterals and provided evidence for the steady or expanding funding support for women’s organizations .

SMART outcomes

Relevant ExampleThe outcome represents noteworthy progress towards the impact you desire.

Shifts in donor and development policy and practices for a greater financial and political commitment to gender equality and women’s rights, including women’s organizations.

SMART outcomes

Timely ExampleThe outcome occurred within 2006-2011, although AWID’s contribution may have been months or even years before.

In 2008-2009, the Norwegian Ministry of Foreign Affairs replenished its Budget Line on Gender Equality…WITM had a session on funding… and provided evidence for the steady or expanding funding support for women’s organizations .

[email protected]

Outcome substantiation

One or more people independent of AWID and knowledgeable about a mutually agreed percentage of the outcomes, or of a representative group of outcomes, agrees (or not) with the formulation of the outcome and AWID’s contribution. The credible “authority” goes on public record with her or his opinion about the outcome and AWID’s contribution.

Outcome Harvesting Ricardo Wilson-Grau 9 February 2012 Beirut, Lebanon.

Documents

Transcript of Outcome Harvesting Ricardo Wilson-Grau 9 February 2012 Beirut, Lebanon.