Contribution Analysis

A different approach to measurement of impact

The challenge

Justify your

programme’s impact

Typical metrics

Number of students

Length of courses

Cost comparisons

Benchmark against similar institutions

…

…

But do these show impact?But do these show impact?

Options

Call for an evaluation– Usually time consuming and one off– Often struggles to get to level 4 of the

Kirkpatrick Evaluation

Possible approach

Set up regular measuring to:

Paint a picture about the attribution / contribution to corporate goal

routinely provide an indication on efficiency

Contribution Analysis

“Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly”

John Mayne

Office of the Auditor General of Canada

June 1999

WMO and Contribution Analysis2007 UN commissioned a consultant to

recommend a process to examine impact of fellowships

2008 UN agreed to use the “Contribution Analysis” approach

2009 WMO commissioned a study on the impact of the WMO Fellowship Programme

Contribution Analysis

Acknowledge the problem

Present the logic of the program.

Identify and document behavioural changes.

Use discriminating indicators.

Track performance over time.

Discuss, and test alternative explanations.

Gather additional relevant evidence.

Gather multiple lines of evidence.

When required, defer to the need for an evaluation.

Financial Resources (WMO Regular Budget, VCP, Trust

Fund)

WMO Secretariat Staff (Education and Fellowship

Division, BO, Finance, travel…)

NMHS Human Resource development needs

1Pre-screening of incoming requests2Consideration by FELCOM3Approval by SG4Fellowship administration prior to, during and after the fellowship (payments, contact with host training institutions, PRs and fellows, travel arrangements,…)

Efficiency and effectiveness of existing services is

improved

New services are being delivered by NMHSs

New technologies/ methods/practices/

procedures are being introduced in NMHSs

Fellows’ career within the services is enhanced

Fellows use their acquired knowledge upon return to

their services

Successful completion of studies

Fellows are satisfied with their studies

Fellows have acquired new skills/knowledge in relevant

fields

Programme Inputs

Programme Activities

Programme Outputs

Programme Outcomes 1

Programme Outcomes 2

Programme Impact(at national level)

PRs are satisfied with the trained fellows (meeting

NMHSs’ HR development needs)

Assumptions: Fellows continue their employment within the NMHS (retention); PRs facilitate fellows’ reintegration into the services and make use of their new knowledge and skills

Assumptions: Fellowship administration procedures and practices are efficient; selection criteria are clear and are applied consistently; only qualified candidates are selected; only those requests are approved which satisfy a genuine HR development need within the NMHSs

Assumptions: Fellows continue their employment within the NMHS (retention); PRs facilitate fellows’ reintegration into the services and make use of their new knowledge and skills

Assumptions: Requests for fellowships are made by PRs based on the HR development needs of their services; suitable candidates are suggested for fellowships; sufficient WMO secretariat staff resources are available to deal with fellowship administration; sufficient funds are available to offer WMO fellowships.

Summary

WMO using contribution analysis for assessing the impact of the Fellowship Programme

The logic map has assisted us to identify the key steps in the fellowship approach and the things we need to monitor and watch

We need to devise and test the discriminating indicators

Look for an update in 12 months