MAFAP Lessons Learned

Lessons learned feedback document

Contents

Introduction .................................................................................................................................................. 3

Part 1. Processing of the lessons learned questionnaires .............................................................................. 4

A. Producing and sending out questionnaires ..................................................................................... 4

B. Extracting and clustering information – first step.......................................................................... 4

C. Clustering the information – second step ...................................................................................... 4

Part 2. Lessons learned information ............................................................................................................. 5

A. Quantitative information ................................................................................................................ 5

B. Qualitative information .................................................................................................................. 8

1. SWOT matrices .......................................................................................................................... 8

2. Recommendations for improvement ........................................................................................ 18

Part 3. Lessons learned workshop .............................................................................................................. 21

Introduction

After three years of implementation, from 2010 to 2013, FAO’s Monitoring African Food and Agricultural Policies (MAFAP) is coming towards the end of its first phase. In order to improve MAFAP during phase II, a lessons learned exercise is being held. Questionnaires have been sent to persons involved in implementing MAFAP during phase I : the MAFAP Secretariat (FAO Rome), the MAFAP country teams, and external partners. In total, 48 questionnaires were treated, and answers were clustered to generate information that will be discussed during a lessons learned workshop in Addis-Ababa on the 27th to 29th of May. The process to produce the information (1), the information itself (2) and its intended use during the workshop are presented below (3).

Part 1. Processing of the lessons learned questionnaires

The lessons learned information was produced following a step-by-step process.

A. Producing and sending out questionnaires The MAFAP Secretariat produced lessons learned questionnaires. The questionnaires were designed to cover all the aspects of MAFAP, with eight categories : process, administration, management, team structure, technical outputs, capacity-building, policy dialogue and communication.

The questionnaires were then sent to country-partners, Secretariat and external partners that have been involved in MAFAP during the last three years. They were filled in and sent back to the Secretariat before the 24th of April.

B. Extracting and clustering information – first step In order to extract the information from the questionnaires, the MAFAP Secretariat created a lessons learned database on Excel, with the eight categories mentioned above. All answers from the questionnaires (country-partners, Secretariat, external) were summarized and copied in the corresponding categories in the database. The recommendations were separated from the lessons learned and copied in a different sheet of the database. Each answer was attributed a color to reflect the satisfaction level : red for dissatisfaction, orange for ‘’could be improved’’(suggesting improvement) and green for ‘’good’’ (no change suggested).

C. Clustering the information – second step Each category with its corresponding answers was transformed into a matrix divided into four categories : Strength, Weakness, Opportunity, Threat (named SWOT matrix). This allowed to cluster the answers by category and not by respondent. The “Strength” cell of the matrix presents the strong points that have been mentioned by country partners and/or Secretariat. The “Weakness’’ cell of the matrix presents the weak points. The ‘’Opportunity’’ cell presents the opportunities for improvement. The ‘’Threat’’ cell presents the threats that can go against improvement or deteriorate the component.

Part 2. Lessons learned information

A. Quantitative information

• Feedback rate In total, 61 questionnaires were sent, with 48 sent back, resulting in a 78 percent feedback rate. For country partners, 33 questionnaires were sent , and 29 were sent back, an 82 percent feedback rate. For the Secretariat, 17 questionnaires were sent and 14 were sent back, an 82 percent feedback rate as well. For external partners, 11 questionnaires were sent with 5 questionnaires sent back, a 45 percent feedback rate.

Even when sending back the questionnaire, no respondent gave feedback on all questions. This was expected as questions were designed to be as comprehensive as possible, in order to capture as many lessons learned as possible. Out of all respondents, feedback was highest on the outputs and management categories, and was the lowest on the policy dialogue and administration categories.

0 20 40 60 80 100

Overall

Secretariat

Country partners

External

% feedback

Sour

ce

The Secretariat provided less feedback than the country partners in all categories except communication and administration. The difference of feedback was particularly striking for process (82 percent versus 63 percent) and capacity-building (97 percent versus 64 percent). The external partner sample being too small (5), feedback per category has not been analyzed.

0 10 20 30 40 50 60 70 80 90

Pdialogue

Admin

Process

Comm

Capacity-B

Mgmt

Outputs

% feedback

Cate

gorie

s

0 20 40 60 80 100

Process

Admin

Mgmt

Capacity-B

Outputs

Pdialogue

Comm

% feedback

Secretariat

Country

• Satisfaction versus improvement

The Secretariat and the country partners had a different approach to the lessons learned exercise. Whereas the Secretariat team tended to systematically focus on what could be improved for phase II, the country partners also used the questionnaire to provide satisfaction feedback. Country partners tended to express more often their judgment on the quality of a component of the project.

Overall, the country partners’ satisfaction rate was higher than that of the Secretariat, with 50 percent of ‘’good’’, 43 percent of ‘’could be improved’’ and 7 percent of ‘’wrong’’. Secretariat satisfaction rate was of 79 percent of ‘’could be improved’’, 11 percent of ‘’wrong’’ and 10 percent of good. The external partners satisfaction rate was 67 percent of could be improved, 18 percent of wrong and 15 percent of good.

Good35%

Could be improved56%

Wrong9%

Overall satisfaction rate

B. Qualitative information

1. SWOT matrices This section provides the SWOT matrices for all seven categories of the lessons learned exercise.

1. a. Process: MAFAP approach to country partners

Strength Weakness

General • Participatory and sustainable approach • Good choice of national counterparts overall • Good partnerships and involvement of a lot of people in some

countries (i.e Burkina Faso) Scoping/starting phase • Successful scoping missions, according to Secretariat and most

country partners • Worked better when there was prior success of FAO initiatives

General • Assessment of complementarities between MAFAP and other initiatives (ReSAKSS/PER

notably) was not strong enough. • No clear road map on the phase in/phase out approach, no medium-term assessment,

and some outputs added on an ad hod basis (not discussed in the scoping phase/LOA). • Preparatory versus intensive was not good for the project. • Could strengthen synergy between MAFAP countries

Scoping/starting phase

• For some countries, no real feasibility study (data availability)/mapping of partners/weak assessment of country partners’ interest & capacity.

• Have a more demand-driven approach, with longer scoping missions involving more partners, to ensure full buy-in/commitment

• Some outputs requests added on an ad hoc basis by Secretariat, not discussed in the scoping phase and not in the LOA.

• Not enough buy-in from local institutions other than partners, and of FAO offices. • No systematic approach to identify partners : 1,2,3, research vs ministry ?

Opportunity Threat

Scoping • Phase II will allow longer scoping phases and more clarity on

required outputs • MAFAP is better defined and outputs have already been produced General • One technical partner rather than two is more efficient • Need technical partners also influential on policy dialogue

General • Need to be careful about the phasing-out • No one-size fits all recipe for success, depends on the country • Restriction in staff time and capacities for some partners, even when they are

committed. • Contrasted views on whether research/ministry partners are the best approach Scoping • Difficult to assess commitment and interest of partners from the beginning, not much

choice for partners in some countries

1. b. Process: MAFAP partnerships with other stakeholders -

Strength Weakness

• MAFAP has already worked with a great number of partners in some countries (i.e Burkina Faso).

• No clear partnership strategy: who should we partner with, and not ? • Not enough interaction with FOs, NGOs, private sector, commodity boards • Not enough involvement of local FAO offices, should be made aware of all

missions

Opportunity Threat

• Consider having someone responsible for partnerships/networking • Possibility to design a partnership strategy • More external partners (private sector, FOs…) could help for data

collection • Relationship with new partners can only grow in phase II • MAFAP can help fill the information gaps for new partners (ministries,

regional organizations, non-state actors)

• Country partners, not just the Secretariat, can work on building partnerships.

• If working with FOs on data collection, they will need training • Private sector can be reluctant to partner and ‘’give away data’’ (Tea Board,

CMDT…)

• Globally, partnerships involve transaction costs, to build long-term trust. Need to be carefully thought.

• Partnerships should not only be based on data collection

1. c. MAFAP Secretariat/Countries relationship

Strength Weakness • Good feedback on relationship between Secretariat and country-level

overall • Good follow-up, numerous emails/phone calls, respectful tone • Lot of coaching from Secretariat to HQ partner (emails, phone)

• Not enough demand-driven interaction between country partners and Secretariat

• Not enough presentations, outputs etc. prepared on a proactive basis by country partners

• Difficult to understand why country partners do not keep deadlines and vice versa

• Not enough involvement of Secretariat in some countries. Need to keep partners in the loop and attribute due credit to them.

• Too many emails, not enough missions • Too many different people to interact with from the Secretariat • Not enough consultations w. country partners before traveling

Opportunity Threat • More country-based staff will increase the level of HQ/country

relationship • Trust has been built over time in most MAFAP countries

• Difficult for Secretariat to communicate and follow-up on a frequent basis with some counterparts.

• Difficult to hold individual team members at country level accountable for their portion of work

• Lot of reliance on interpersonal relationship : staff leaving/changing in Secretariat and countries means that the relationship has to be rebuilt

2. Administration

Strength Weakness

Finance/ contract/ Travel : • Good administration of contracts, travels and payments

Finance/ contract/ Travel : • Not enough visibility of contracts, done at the last minute • Lack of tracking payment • Issue on frequency and timing for payment • DSA before travel • Slow disbursement, too much delay • No funding for travel cost and data collection at country-level • Lack of transparency of payments

Sharing doc : • Poor sharing • Random and disorganized system

Opportunity Threat

Finance/ contract/ Travel : • Need more Adm. capacities • Improve travel planning • Prepare contract more in advance

• Improve relation with regional offices Need closer follow up at country

level

Finance/ contract/Travel : • Contract visibility

Sharing doc • Better archive system and more frequent update

Sharing doc • Poor sharing of doc: what happens if someone leaves the team?

3. Management and coordination – Group B

Strength Weakness

MGMT style : Outstanding- good vision -effective - open- transparent-stimulating- open to new ideas-participatory-flexible- good reporting

Work/deliverables: clear attribution of roles & responsibilities

Deadlines: clear deadlines - good follow-up

Planning: good overall planning

Self development: support for personal training; good career opportunities

Meetings: well organized - clear objectives- participatory

Team: good mix of skills- young and more experience staff

MGMT style : Say “Yes” to everything- purpose of work unclear - ideas proposed but not implemented - no possibility to propose ideas- no clear assignment of roles & responsibilities -working process unclear/inconsistent - allows for too many opinions/disagreement

Work/deliverables: lack of time - unpolished products - low quality products - uneven quality distributed - unclear structure process to finalize documents - final clearance unclear

Deadlines: Blurry- missed- loose- moving- ad hoc- changing- unrealistic- too tight- tough- unexpected- no pressure to deliver- overall project timeline unrealistic

Planning: poor planning of activities- more planning needed- too much work- poor organization of work- work plan by country needed

Country level work: no M&E mechanism- monitoring of country partners work/achievements/deadlines - Clarify roles & responsibilities – Improve coordination- follow-up/status report- need more milestone meetings with country partners

Career: No professional improvement of young people- No support to career growth- disappointing contracts – limited space for growth

Meetings: Document/report/minutes of meetings- More explanations, more time, more control & follow-up on Monday morning- Monday mtgs too long- not focused

Opportunity Threat

Deadlines : Request for more follow-up and tougher supervision

Team: Lack of negotiation skills in the team

Country level: Insufficient involvement of FAO Reps - No involvement of country partners in planning- Authority of country coordinator - Coordination/alignment between technical and institutional partners

4. MAFAP team structure

Strength Weakness • Talented team with a diverse background at Secretariat level. • Very good young professionals at Secretariat level. • The experience of having a technical assistant was positive in Mali/BF • Good balance between young and senior researchers (country-level). • Very good to have communication staff (Secretariat).

• Not enough staff at Secretariat and even less at country level : lower quality for work and sustainability at risk.

• Not enough Secretariat staff based in countries. • Not enough economists/ag metrics specialists in the Secretariat. • Shortage of senior staff for policy dialogue. • Not enough English native speaker staff.

Opportunity Threat • Hire a country-based junior staff for 6 month with skills in

communication and mgmt/data collection + regional coordinator. Need prior country buy-in for this.

• Consider hiring an editor, statistician, senior economist, native English speaker.

• Could have clearer focal points for each country for HQ/Country communication

• Possibility to send Secretariat staff for 1 month missions rather than several short missions

• Financial limitations + transaction costs in recruiting more staff

5. Capacity-building

Strength Weakness Material • Helpful, comprehensive • Direct guidance • Simple language Audience • Improve local team knowledge • Did build capacity • High self development Approach • Suitable tool: meeting and discussion • Rigorous and good support from Rome • Good training session • Part of the institutionalization

Material • Methodology not yet final when training was delivered • Not enough interactive • No use of the materials

Audience • Audience not ready to absorb such a high-level training • Not targeted enough training sessions

Approach • Should be integrated at an early stage • Not designed from the start • Not enough training before starting the analysis • Secretariat did not spend enough time at country level • Only one workshop, not enough • Too short • Too intensive at the beginning

Opportunity Threat Material

• Need to improve or develop new material (external expert) + tool kit • Need high quality CB material • Focus on how to understand the result/interpretation • More training on concepts • Learning by doing/coaching

Audience • More collaboration between the team • Good opportunity for self development • Train more researchers/ more team members

Approach • Need strategy • More meetings workshops • On site coach+ distance • Need more travel or spend more time with the country partner • Need long run training

Material • Key challenge: interpretation of the results • PE classification : difficult

Audience • Need good capacity • Team to be trained: restricted or large audience?

Approach • Coaching from distance: limited impact • Need time to understand the methodology and how to apply it • Not allow local perspective

6. Technical outputs

Strength Weakness

• All the technical outputs are of good quality, relevant and useful. • Correct outputs, just need to refine them • Technical notes are the strongest technical output • Policy Briefs are a good communication tool • Database is good at global level and for comparisons

Process • Slow process to produce outputs (especially technical notes) • Methodology not consolidated and need of more clear guidelines and tools • Weak language editing and harmonization of language • Difficulty to produce final outputs • Work for some TO still heavily done by FAO, mainly for country reports • Deficient data mgmt, not always clear how the data was collected or treated Content • Technical notes and country reports are too long • Audience is not clearly targeted • Difficulty to create narratives or understand the implications of the results • Unbalanced focus towards I-D over PE and Policy Coherence

Opportunity Threat Process • Increase the importance of policy briefs, deal with new topics, not only

commodities incentives/disincentives • MAFAP-FAPDA synergies in the production of TO still to be fully

exploited • More MAFAP-related outputs can be produced, i.e. academic articles • Database and Data Management System to streamline TO production • Database can be used as an interface with partners • The synthesis report can have a high impact at global and regional level Content • More focus can be put on the drivers of I-D • Policy coherence still to be fully exploited • More focus can be put on consumers and on the input side for producers • Stronger messages and recommendations can be put forward

Process • The review process is not always followed • A monitoring system should be more dynamic. • Low updateability of the TO • Distribution of recognition between partners and Secretariat can be difficult • All the hard work and information contained on the TO end up being read

only by a few people • The importance of the synthesis report is underestimated • Excessive number of outputs may undermine quality Content • Weak recommendations in the PBs

7. Communication – Group D

Strength Weakness

• Strong improvement over the last months. • Good website overall. • Good and impactful brochures. • Good success of the policy briefs. • Good success of the Twitter account.

Process • No strategy, no targeting of audience. • Not enough peer review. • No communication culture in the team, writing to be read, and by non-tech

audience • Not keeping track of communication impacts. Country level • Not enough country-level communication and awareness of MAFAP • Not enough MAFAP presence in workshops/symposiums and radio/TV • Documents not enough shared with country partners, not translated in

Swahili for East Africa. Website & other media • Restricted access to outputs on the website, slow and difficult to dl outputs • Website possibilities too limited on FAO platform. • Need a link to the FAPDA website • Not enough communication media used. • Overall appearance of communication products still too weak.

Opportunity Threat • 2 communication officers have joined the team, 1 of them graphic designer • Preparation of communication strategy, identifying the target audience, etc. • Phase 1 was output-focused, phase 2 can be more focused on comm. and

outreach, especially in the first MAFAP countries. • Some communication outputs have been designed and can now be updated • The database will soon be on the website and can attract a lot of visitors. • The outputs will soon be unrestricted. • In intensive countries, possibility to organize a large event to disseminate

results, and to increase communication at decentralized level.

• Difficult to develop a communication culture in the team • Financial and transaction cost to work with a private partner to design the

communication strategy. • Transaction costs to coordinate country-level and HQ communication. • Hard to move away from FAO website platform. • Costly and difficult to translate the MAFAP communication outputs in

Swahili • If outputs are unrestricted on the website, they need to be validated by

country partners and of highest quality.

8. Policy dialogue – Group D

Strength Weakness

• Good approach on PD • Many stakeholders involved brings visibility • Good targeting of PD partners-Targeting MINAG is key • Good to involve PD partners from the start • Good High Profile Policy Fora (ex. Kenya/Uganda) • Good to build on technical workshop (Nat & Reg) • Keep focus on messages for policy makers not academia

• Needs time to build trust/relation, need to involve PD partner from the beginning

• PD comes at a late stage of MAFAP implementation • MAFAP to facilitate PD not lead • Need to monitor outcomes of PD-Keeping track of PD impact • Insufficient partnership with WB/ReSAKSS • Insufficient visibility of MAFAP to initiate PD well-Ownership by Ministry

required

Opportunity Threat

• Have outlets identified CAADP events FAO regional conference REC ministerial meetings

• Have a clear PD strategy • Propose PD events along the implementation phase

• Other tools or methods than MAFAP used by decision makers • No policy change/reform if no follow-up after PD • Political uncertainties/deadlocks/sensitivity • Poor responsiveness from high level decision makers • Lack of sustained support by decision makers if not kept in the loop

2. Recommendations for improvement

This part presents the recommendations that were extracted from the lessons learned questionnaires.

1. Process a) MAFAP approach to country partners

• Better and longer scoping phase to ensure more ownership and demand. • Feasibility studies to assess data availability and mapping of actors • Discuss at Secretariat level to have an harmonized approach for country partners : upgrade, new

countries, 1 or 2 partners… • More accountability on their work from country-partners, and vice versa • More integration of MAFAP in policy planning of partner institutions (Ministries) • More discussions between MAFAP partner countries

b) MAFAP partnerships with other stakeholders

• More ownership by various stakeholders, not just the partners : ministries, NGOs, FOs.. • Design a partnership strategy and assess who to partner with and who not to.

c) Team structure

• More country-based staff from Secretariat • 1 or 2 economists/ag metrics specialists • Add one senior staff at least, ensure editing functions are sustained (English/French) • Limit recruitment of consultants at country-level • More consultation between technicians and coordinators/senior staff at country-level

d) Relationship Secretariat/Countries

• Longer duty travel missions • Regular phone calls/Skype meetings w. country partners • Clarify focal point in Secretariat for each country and vice versa • More country-based staff to improve relationship

e) Working arrangements

• Move away from PSAs for junior staff at Secretariat level • Use of LOAs, but with provision for 1 or 2 full-time MAFAP posts at partner level • Better contracts at country-level and more financial means for institutions to recruit • Limit recruitment of consultants, if so recruit them within the group of people already involved in

MAFAP • More transparency on payments and contracts at country-level • Develop the relationship with regional offices • Provide more funds for travel and data collection

f) FAO role towards MAFAP

• Integrate MAFAP in strategic framework (SO1, SO4, others ?) • More clarity on senior management over MAFAP, especially in EST • More financial commitment : in-kind contribution to recruit staff

g) FAO and FAPDA

• More informal meetings between the two teams • Strengthened collaboration

2. Administration

Finance /Contract/Travel

• Regarding the contract extension, give more visibility, increase predictability and transparency and process contract more quickly;

• Ensure that all payments are done in time (wages and DSA); • Increase predictability of travel /mission to facilitate the travel arrangements • With the country partners, ensure better tracking of payment and follow up and reduce delay for

disbursement; • Sharing of documents • Archiving should be more systematic and frequent • Better organize the archive system

Sharing of documents

• Archiving should be more systematic and frequent • Better organize the archive system

3. Management and coordination • Develop a stronger M&E and planning framework, a strategy with clear roles, responsibilities for

outputs, communication, partnerships. • Open more career opportunities and allocate more time for self-development

4. Capacity-building

Approach

• Develop a comprehensive capacity building strategy • Target ownership and sustainability • Revise the duration and the frequency of the training session • Spend more time within countries with partners to work on the methodology • Better define the window of opportunity to start the training session

Material/ content

• Develop high quality material with the help of external specialists, it would include in-site coaching and distance coaching

• Focus the training session on the understanding and interpretation of the results • Learning by doing

Audience

• Develop relationship between countries • Better define the audience to train (number and capacities)

5. Technical outputs

• More planning ahead for realistic deadlines. More scheduling to deliver final outputs. • Improve the review process (shorter, realistic, timely and enforced) • Improve data collection: increase statistical accuracy and primary data collection, and dedicated

teams. • Streamline technical outputs. Better data management system (templates, guidelines) to make

data processing easier, dedicate more time to analysis, and increase updateability. • Increase number of staff working on the synthesis report. • Focus on quality outputs rather than quantity, especially for technical notes • Reduce the length of technical outputs and focus on the message and impact, especially for the

technical notes and country reports. • Increase focus on policy coherence

6. Communication

• Design a communication strategy with external assistance if needed. Define MAFAP’s identity,

target audience, agree on a calendar and on specific communication outputs. • Develop a communication culture in the team (Secretariat and Country partners), by sharing the

communication strategy and methodology, with guidelines and templates, and by training the team on communication practices.

• Improve global communication, on the content with clearer messages, more editing (possibly with a full-time editor), and on the process with more media outlets (YouTube, Twitter), more communication outputs and updates, and a tune-up of the MAFAP products design (website, brochure, briefs…).

• Strengthen communication at country-level, by having MAFAP participating in more workshops/symposiums, radio and TV debates, by putting MAFAP outputs on local websites, by translating some outputs and by communicating at a decentralized and University level. Local FAO offices could also provide more support. In intensive countries, a large MAFAP event could be organized, and more regional workshops could be held.

• Focus the communication on outputs, by making all outputs unrestricted on the website and easy to find and download, and by producing technical outputs that can be easily disseminated.

7. Policy dialogue

• Involve policy-makers from the beginning and update them on the project. • Target high-profile policy makers • Involve the policy dialogue partner more often

Part 3. Lessons learned workshop The objectives of the workshop are threefold :

1. To share the results of the MAFAP team (Secretariat and Country partners) and to have a more complete feedback on MAFAP Phase I activities

2. To engage MAFAP country partners and Secretariat in building recommendations for MAFAP Phase II, and agreeing on the way forward

3. To strengthen the working relationships/trust and communications among all participants.

To achieve these objectives, the qualitative and quantitative data presented above will be used. Questionnaire results for each of the eight categories (see Part 2.) will be shortly presented to participants by the Secretariat and discussed in plenary. Participants will then be divided into four groups, which will discuss key points related to each category. The groups will put recommendations forward, which will be debated. Recommendations endorsed, will be turned into commitment by the MAFAP team, both Secretariat and Country partners.

Participants of the workshop will be divided into four group, with an equal amount of topics clusters. The groups will be divided as such :

Group Categories Group A Process

a) MAFAP approach to country partners b) MAFAP relation with other stakeholders c) MAFAP Secretariat/Country relationship

Group B Administration Management and coordination Team structure

Group C Technical outputs Capacity-building

Group D Communication Policy dialogue

Each group will mix Secretariat and Country partners.