MAPPING A JOURNEY; ADJUSTING THE COURSE

A summary of the interim evaluation of the Aflateen Program Pilot

October 2013

Approach: UFDE

• Utilization focused evaluation – focus on users, uses, key evaluation questions, and facilitating use

• Developmental evaluation – documenting a model in the making and learning from it

2 components

• Secretariat level: assessment of the Aflateen curriculum development process to help the Secretariat review and improve its rollout and support of the program

• Implementation level: generating lessons for current and future partners.

Somewhat like … videotaping someone walking inside a moving train- at times from the inside, at times from outside. [Happily the train is still on track and heading in the same direction.]

5 topics

1. The curriculum development process2. E-learning3. Partners / dissemination4. Training5. Evaluation

5 uses

1. Define and document our desired curriculum development process by the end of 2013 for future revisions.

2. Create a plan to scale up and broaden the user base of the e-learning.

3. To align Aflateen partner support practices with strategic goals and document this for future PM staff

4. To differentiate training process for Aflateen vs. Aflatoun as needed

5. To improve a methodology for research calls that enhances learning of partners for the purpose of improving their program and the quality of their monitoring and evaluation.

Key Evaluation Questions

1. To what extent are partners / Secretariat satisfied with the process and the final product?

2. Did the Implementation match with Strategic Objectives, in other words did the outputs meet the plan for reaching 50,000 participants?

3. What are the factors used by PMs to prioritize support provision to Aflateen partners?

4. To what extent do/should we differentiate Aflateen and Aflatoun trainings?

5. To what extent did the research call create a learning opportunity for partners?

Review of program evaluations

i. Balancing implementation fidelity with local contextualization

ii. Documenting proximal outcomes from the five core components of the curriculum

iii. Describing causal pathways and potential barriers towards long-term impacts, and

iv. Describing causal pathways towards reaching scale at the country level

v. Assess the quality of the evidence provided in the reports

Who they are, why they join

• 65% of partners are NGOs• The join because of the attractiveness of the

participatory teaching style; the interest related to the content of the curriculum; the non-formal education approach used to deliver the Aflateen program; and youth’s interest in social and financial enterprises.

What attracts youth to Aflateen

• the participatory teaching style (24.3%)• The content of the curriculum (18.9%)• the non-formal education delivery (16.2%)• youth’s interest in social and financial

enterprises (10.8%).

Partners’ opinions

• There were positive responses to the curriculum process as a whole.

• 76.9% of the Aflateen partners that were consulted during the consultation process were “Very satisfied” with the consultation process, and the rest (23.1%) were “Somehow satisfied”.

• 44% implement Aflateen together with other programs

Country reports

• China• Nepal• Georgia• DRC• Kenya• Zimbabwe• Bolivia• Peru

Benefits for youth (Nepal example)

• i) learning about self-exploration, • ii) social factors,• iii) savings, and • iv) budgeting and money management.

… as the result of the program, youth got organized and took responsibilities to help other

youth to learn about social issues and financial factors (change agents).

Context matters - readiness• the nature of the program & follow-up support • the level of training by facilitator• whether the setting is urban or rural• the nature or existence of other assets and services, • the personal and family histories of the youth.

Pathways for scale-up

• formalization into the official curriculum• wider coverage (geography, schools, NGOs)• further training of trainers, of teachers• partnering with other organizations that can

enhance readiness

Challenges & responses

• Social franchise model – important to communicate it• Target numbers – explain decisions around ‘what counts’• Theory of change – outcomes matter (inherent good)• Contribution to impact – longitudinal study opportunity• Digital platform – learning by doing, embrace surprise

Curriculum- key findings & recommendations

• Partners requested more support for country level curriculum integration/ advocacy

• Partners were in general satisfied with the process but still require some adjustments

• Partners are either not aware or not making use of the current Contextualization Manual; yet they still ask PMs for support

1. Develop a Secretariat plan to support country-level curriculum integration events, which will include a Guide for partners

2. Modify and simplify the curriculum Contextualization Manual through the creation of simple checklists, a list of most frequent changes, and FAQs, to increase use

3. Document process for the development of new curriculum based on experience from Aflateen, Aflatoun and Aflatot as part of the Quality Assurance process

E-learning - key findings & recommendations

• Connectivity and language challenges were confirmed • 30% of partners were aware of the platform, even though it had not been

launched • Original rationale remains valid: Disseminate Aflateen digital through

partners, youth networks and youth organizations to reach over 50,000 youth by 2015

1. Disseminate Aflateen Digital learning to existing Aflateen partners2. Disseminate Aflateen Digital learning to youth organizations and other

stakeholders who are not currently partners 3. Identify ‘Aflateen Digital Champions’ from amongst the youth who are

actively using and promoting the platform4. Optimize the platform on an ongoing basis via usability testing

Program/partner support - key findings & recommendations

• Partners rely on the Secretariat and don’t make use of each others’ expertise

• Secretariat strategy relies on the idea of a network while partners’ sense of a network is weak

• Implementation may be strengthened by ‘readiness’ (e.g. partner organizations with complementary services)

• The Secretariat is prioritizing national coalitions as a path to scale, especially in priority countries

• Current PM’s decision-making process is more clearly documented (e.g. differentiated approach with nascent, network, or government partners)

1. Strengthen national networks and partners that function as country level secretariats for scale-up of Aflateen

2. Strengthen direct linkages between partners regionally and globally by leveraging partner skills and experience more strategically

3. Ensure that Aflateen partners get more differentiated support and Aflateen specific partner communication

Training- key findings & recommendations

• Aflateen partners asked for facilitation skills• Secretariat team is integrating TTI and RMT training that also emphasizes

facilitation • Training is key to scale-up• Training is very dependent on one person -there is no redundancy

1. Update the Aflateen training to strengthen content related to facilitation skills for training youth and then conduct new regional master trainings in 2014

2. Expand and strengthen the pool of Aflateen specific trainers3. Revise monitoring and evaluation tools for both Secretariat and Partner

led trainings to facilitate learning

Evaluation - key findings & recommendations

• Findings confirmed that partners have varied implementation approaches and they need clarity about how and what to monitor

• There is a trade off between programme fidelity and the ability to adapt the curriculum to local contexts• Partners in the RFPs reported awareness gains in evaluation but still wanted more handholding from the

Secretariat• The Secretariat findings and the varying quality of the country studies underlined the need to

differentiate when to focus more on building evaluation capacity and when to focus more on research quality

1. To prepare and implement a research plan that privileges proof of concept, scaling and performance while linking partners to external capacity development opportunities

2. Complete the creation of the Implementation & Monitoring Guide for monitoring programme numbers. This should also capture variability of programme implementation

3. Produce user-friendly guides and integrate them into programmatic support for conducting research of various types

Big picture issues

• Self-organizing youth; peer-to-peer

• Direct interaction with youth

• Potential for more scale than initially thought

• Linking with vocational, agricultural

• Implications for growth/out-sourcing

• What can be delegated to partners….?

• Prioritization of strategies (training, evaluation, partner support)

• Communication dimension – be more specific on functions (fund raising, public relation & corporate communication, networking, listening to partners, reporting to funders)

Reflecting on the process

• UFDE did encourage learning as the main driver• Developmental side of evaluation cannot be 100%• Evaluator’s role as facilitator is not 100%• Tension between DE (learning) & country reports (outcomes) • Difficult to disaggregate ‘moving train effect’: how findings

affect organizational change; how organizational change affects findings

• Unused USES and KEQs• Staff – pressure to pause & reflect (work loads, learning to say

no)• Staff – have data to confirm what they know