1

ANNUAL PROGRAMME REPORTING

WEBINAR

Results and Evaluation Unit, FMOBrussels, 21 January 2016

2

After this webinar…

You should:

1. know how to register results correctly

2. have a better understanding of how to analyse and present

programme results

3. Be able to submit your report through DoRIS

by 15 February 2016

Click on the “Contact” link at the top of the webinar page OR send your question to doris-support@efta.int

3

Annual Programme Report (APR)

Has 3 main components in DoRIS:

The indicators: show progress and results

The calls: show calls organised and projects contracted

The narrative APR: assesses and summarises programme progress and achievements during 2015

4

This webinar…

Outlines the ‘standard’ APR template.…but the main messages also apply to the

• Combined Strategic Report and APR

• The APR for NGO programmes

5

Introduction • Purpose of the APR • The reporting cycle

Session 1

Registering results• Overview of the workflow, create APR• Registering progress and achievement values

Session 2

Narrative Annual Report• Overview of content• How to present achievements• Examples of good practice

Session 3

Conclusion• Upload and submitting your report• Summary

Session 4

Questions and Answers

6

Session 1Introduction

• Purpose of the APR• The reporting cycle

7

Purpose of APR

1. progress in implementing the programme and in achieving results

2. challenges to implementation and plans to overcome such challenges

3. summary of irregularities and measures taken to remedy these.

Adapted from the Regulation FM09-14 (Art 5.11. para 1)

Also, see the APR template in Annex 9 of the Regulation

8

February 15Annual

Programme Report

March/AprilStrategic Report

Sector ReportFMC Status Report

JuneDonor’s

Annual Report

OctoberNMFA Annual

Report to Parliament

NovemberDonor’s

Report to EFTA

The results and the reporting and communication cycles START with the APR

Programmes achieve results

POs highlight and interpret these results

YEAR

9

10

11

What are your results for 2015?

12

Session 2Register results

Create APR in DoRIS

13

Create APR in DoRIS

13

Create APR in DoRIS

13

Create APR in DoRIS

13

Create APR in DoRIS

13

Create APR in DoRIS

13

Create APR in DoRIS

13

Create APR in DoRIS

13

Open APR task

21

Open APR task

21

Open APR task

21

24

Annual Programme Report task consists of 3 steps:

Register indicator achievements

Register calls progress information related to the programme

Upload the annual programme report based on the annotated template

Register indicator achievement values

16

Register indicator achievement values

• Register 2015 achievement only

• Consistent with the narrative

• Correct values → used for aggregate reporting on the results across years

16

27

Register indicator achievement valuesAttention to potential issues!

Target is 1 → do not repeat the same achievement across years

Indicators measured in % → insert only the difference attributed to 2015

Indicators that aim to achieve a reduction → the achievement values are input negatively (use ‘-’ sign)

28

Target not defined (set to zero)→ make sure the achievement is included

Potential double counting: Outcome/output indicator linked to an outcome split between the two mechanisms → achievements are distributed proportionately with the allocations between FM in the programme

Register indicator achievement valuesAttention to potential issues!

Register calls values

20

Register calls values

20

• Register values attributed to 2015 only

• Do not repeat the number of applications received or projects contracted already reported in the last Annual Programme Report

• Call closed last year but projects contracted this year => this year report only the number of projects contracted

• Register values attributed to 2015 only

• Do not repeat the number of applications received or projects contracted already reported in the last Annual Programme Report

• Call closed last year but projects contracted this year this year report only the number of projects contracted

• Identify incorrect amounts in the previous Annual Programme Report => contact DoRIS support

31

In summary… INSERT ONLY THE ACHIEVEMENTS

THAT TOOK PLACE IN 2015

VALUES ARE CORRECT AND CONSISTENT WITH THE NARRATIVE DOCUMENT

ATTENTION TO POTENTIAL ISSUES (INDICATORS MEASURED IN % OR NEGATIVE ACHIEVEMENTS)

… AND TO POTENTIAL DOUBLE COUNTING(SAME INDICATOR CONTRIBUTING TO TWO MECHANISMS)   

32

Session 3Narrative annual programme report

33

Narrative annual programme report:What we are looking for?

WHAT ARE WE LOOKING FOR?

ANALYTICAL REPORTS THAT CLEARLY DEMONSTRATE RESULTS AND PROGRESS TOWARDS OBJECTIVES - USE THE EVIDENCE E.G.

INDICATORS- FOCUS ON RESULTS: WHAT

DIFFERENCE IS THE PROGRAMME MAKING? FOR WHOM? HOW?

- USE SPECIFIC EXAMPLES TO SUPPORT YOUR ANALYSIS

- KEEP IT SHORT

Structure of the Annual Programme Report

1. Executive summary2. Programme area specific developments3. Reporting on Outputs4. Reporting on Programme Outcomes5. Project selection

6. Progress of bilateral relations7. Monitoring8. Need for adjustments9. Risk management10. Information and publicity11. Cross cutting issues12. Reporting on sustainability

34

APR Section 2: Programme area specific developmentsDescribe important developments in the Programme area (including policy, financial or administrative changes).

The aim of this section is to provide Context for the achievements in the next sections e.g.

- What are the main trends in the country, region, sector?- Has a changing context made it easier or more difficult to achieve results?

35

APR Section 2: Programme area specific developments

• Refer to the justification in programme proposal. Has anything changed that may have an effect on the planned programme results?

• Include statistics and analyse trends in the sector and/or the programme area where possible.

• If conditions have changed since the programme proposal, do the changes present risks? How might these might be handled?

36

Example: Context data

Suicide rates are important context data for programmes with a strong mental health component.Where does your country stand and what are the trends? (Macro Indicators)

EU - 28

Bulgaria

CroatiaCyp

rus

Czech Republic

Estonia

Greece

HungaryLa

tvia

Lithuania

MaltaPoland

Portugal

Romania

Slova

kia

Slove

niaSp

ain0

5

10

15

20

25

30

35

40Suicide rates per 100,000 people by country

2011 2012

37

• In order to answer this question, we need to place an emphasis on Outcomes

• But we also need to focus on Outputs to monitor implementation

Are we on track?

38

APR Section 3: Programme outputsDefinition. An output is the product or service directly financed by the programme.

The basic question for Section 3 is…

Are you achieving your programme outputs?

Examples. - Equipment purchased for pollution control; - NGOs directly supported; - renovated buildings, or - number of people trained; research projects supported

39

Analyse how projects have contributed/are contributing to Programme outputs.

• Back up your analysis with data and assess progress towards targets. • If you are exceeding targets, let us know! • Or, if there are achievements but no indicators established in your programme

reporting template, please describe these in the narrative.• If no/few results, also explain why. For example this may be due to project

selection recently completed: In this case, consider how the selected projects are expected to contribute to programme outputs?

• What worked, what didn’t work, and what could be done better?

APR Section 3: Reporting on Outputs

40

APR Section 4: Programme outcomesDefinition. An outcome is related to a behaviour change, implementation of

innovative thinking, or change in quality of life.

The basic question for section 4 of the APR is…

What difference are the programmes making?

Examples: - Reduction in Green House Gas emissions; - And related to the above: reduction in health problems related to pollution; - changes in attitudes (resulting from NGO actions);- People are using news skills that have been gained via training; - publishing or applying results from research

41

APR Section 4: Reporting on outcomes

• Use evidence: programme indicators, monitoring, review or evaluation results• Include charts if appropriate and compare against baseline/last year’s results• Look at the contribution of the programmes, keeping in mind

the context (section 2).• Use a project example for each outcome to demonstrate results. • Consider relevant horizontal concerns• Summarise main risks and mitigating actions• What worked, what didn’t work, and what could be done better?

42

Key sources of information for your Annual Programme Report

These Reports, together with monitoring and evaluation findings, show progress towards results

Annual Programme Reports: create the ‘story’ of progress towards results and follow it through each level of analysis

Project

information

Report 41

Bilateral engagementReport 35

Analysing progress and results

43

Contexte.g. trends in mental health service provision

Awareness of mental health issues

Annual Programme Reports

Describe progress, challenges and opportunities in improving mental health services, contribution of bilateral partners and resulting achievements

Bilateral engagementDPP/dpp

Programme outcome Improved mental health services

Project(s)e.g. centres, access, rehabilitation systems, screening, services

Results example: Mental health

44

Aggregate indicators show significant results

395,737 people benefiting from improved health services

Source: preliminary data provided by POs of Health programmes, November 2015.

Programme summaries

Project information

45

Improved mental health is one of the main priorities

Other Outcomes

Improved prevention and treatment of communicable diseases (including HIV/AIDS and TB)

Improved access to and quality of health services including reproductive and preventive child health care

Improved mental health services

Life-style related diseases prevented or reduced

Improved access to and quality of health services including reproductive and preventive child health care as well as

health care related to ageing society

17%

7%

13%

17%

18%

27%

Committed grant by outcome PA13 and PA27

98 projects of a total of 280.

29 projects with a dpp

46

Programme outcome/outputs show…

• Concentration in the mental health area

• Selected results:

9,237 youth who benefited from improved mental health services

3,000 children and youth get counselling in mental and reproductive

health

82% satisfaction of users of integrated mental health services

Add project examples that contribute to this story.

Programme summaries

Project information

47

APR Section 6: Progress of bilateral relationsAssess progress towards strengthening bilateral relations: most important achievements and highlights.

Again, the basic question is…

What difference are the programmes making?

48

• Follow the 4 bilateral outcome areas (Bilateral Guideline)• Assess progress in relation to bilateral ambitions

• Go beyond the indicators and use specific examples to demonstrate achievements

• Use evidence: programme indicators, monitoring, review or evaluation results

• What worked, what didn’t work, and what could be done better?

Bilateral thinking (Outcomes)

• Extent of cooperation

• Shared results• Improved

knowledge and mutual understanding

• Wider effects

APR Section 6: Progress of bilateral relations

# of partnerships established, events organised, and use of the bilateral fund indicate the extent of cooperation

…but what is the result?

49

Results Example: Bilateral

Bilateral cooperation is highest in the Research and Scholarships sector…but there is significant cooperation across other sectors. See Report 35 in DORIS

Bilateral Cooperation

Data: April 2015.

Map of co-operation

Programme summaries

Project

information

51

• Click to add text.

Results Example

Outputs = the school, equipment, trained teacherOutcomes = better educated children, social integrationDescribe the whole picture!

52

Country level results – Bulgarian justice sector

• Training - of judges, magistrates, and probation staff

• Secondments to the European Court of Human Rights.

• Involvement of the Council of Europe• Study visits, exchanges, joint activities with Norway

• The overall outcome is increasing competence within the Bulgarian judiciary, strengthening the capacity of the justice sector.

• The APR narrative could also highlight other evidence of increased capacity.

Results Example: Capacity Building

53

Session 4Conclusions

Upload and submit APR

Annual Programme Report task consists of 3 steps:

Register indicator achievements

Register calls progress information related to the programme

Upload the annual programme report based on the annotated template

Upload annual report

Upload annual report

Upload annual report

Upload annual report

View information on submitted APRs

50

View information on submitted APRs

50

View information on submitted APRs

50

View information on submitted APRs

50

View information on submitted APRs

50

View information on submitted APRs

50

65

Summary: registering results

1. REPORT ONLY THE ACHIEVEMENTS THAT TOOK PLACE IN 2015

2. ENSURE VALUES ARE CORRECT AND CONSISTENT WITH THE NARRATIVE ANNUAL PROGRAMME REPORT

3. PAY ATTENTION TO THE SOME OF THE COMMONS MISTAKES

66

1. IT’S ABOUT RESULTS: COHESION, BILATERAL, HORIZONTAL CONCERNS

2. ASK THE RIGHT QUESTIONSACHIEVEMENTS? FOR WHOM? HOW?

3. ANALYSE AND REPORT ON RESULTS RATHER THAN GIVING A DETAILED RECORD

4. USE EVIDENCE AND SPECIFIC EXAMPLES

5. INCLUDE RISKS AND MITIGATING ACTIONS

6. KEEP IT SHORT

Summary: Narrative APR

67

Further help…

Further questions? email: doris-support@efta.int

• APR submission user manual• APR training video• APR annotated template

DoRIS reports:• Aggregated results -

Report 44• Individual programme

summaries - Report 15• Programme results –

Report 21• Project information -

Report 41• Bilateral engagement –

Report 35

68

69

Thank you!

www.eeagrants.orgwww.facebook.com/eeanorwaygrants