Monitoring, Evaluation and Learning (MEL) FrameworkThis document presents BRAC‟s Monitoring,...

1

Monitoring, Evaluation and Learning (MEL) Framework

Community Based Education for Marginalised Girls in Afghanistan, BRAC Afghanistan

Girls Education Challenge – Transition Window (GEC-T): 5085

2

Table of Contents 1. Introduction ............................................................................................................................................. 4 2. Learning from GEC 1 ............................................................................................................................. 8 3. Monitoring ............................................................................................................................................ 11 4. Key evaluation questions ...................................................................................................................... 17 5. Evaluation approach ............................................................................................................................. 18

5.1 Research design .............................................................................................................................. 18 5.2 Measuring outcomes ....................................................................................................................... 19

5.2.1 Sustainability ............................................................................................................................. 32 5.3 Ethical protocols ............................................................................................................................. 35

5.3.1 Ethics ......................................................................................................................................... 36 5.3.2 Child protection ......................................................................................................................... 41

6. Sampling framework ............................................................................................................................ 45 6.1 Target groups .................................................................................................................................. 45 6.2 Control groups / Counterfactual scenario .................................................................................... 47 6.3 Cohort tracking .............................................................................................................................. 49

6.3.1 Learning cohort ......................................................................................................................... 49 6.3.2 Transition cohort ....................................................................................................................... 51 6.3.3 Replacement strategy ................................................................................................................ 52

6.4 Power calculations and sample sizes ............................................................................................. 53 6.5 Benchmarking ................................................................................................................................. 57

7. Baseline study ........................................................................................................................................ 59 8. Evaluation governance ......................................................................................................................... 60

8.1 Evaluation steering group ............................................................................................................. 60 8.2 External evaluator .......................................................................................................................... 60 8.3 Data validation ................................................................................................................................ 61

9. Data quality assurance ......................................................................................................................... 61 9.1 Training ........................................................................................................................................... 61 9.2 Piloting ............................................................................................................................................. 63 9.3 Data cleaning and editing............................................................................................................... 63

10. Risks and risk management ............................................................................................................... 64 11. Learning ............................................................................................................................................... 67

11.1 Learning strategy .......................................................................................................................... 67 11.2 Stakeholder engagement, dissemination and influencing ......................................................... 69

12. Evaluation work plan ......................................................................................................................... 70 12.1 Timetable ....................................................................................................................................... 70 12.2 Responsibilities ............................................................................................................................. 71

Annex A: Logframe .................................................................................................................................. 72 Annex B: Draft evaluation tools .............................................................................................................. 72 Annex C: Draft sampling framework ..................................................................................................... 72 Annex D: ToR for evaluators ................................................................................................................... 73

4

1. Introduction

This document presents BRAC‟s Monitoring, Evaluation and Learning (MEL) framework for the

Community Based Education for Marginalised Girls in Afghanistan project, an eight year project funded

by UK Aid under the Girls Education Challenge – Transition (GEC-T) window.

The overall aim of BRAC‟s project is to improve the life chances for marginalized girls in Afghanistan,

with the following three objectives being central to achieving this aim:

1. Supporting marginalized girls to learn the skills they need to be empowered, valued and

productive members of their communities,

2. Supporting marginalized girls to transition effectively from BRAC community-based education

(CBE) primary schools into government schools, CBE secondary schools or TVET training,

3. Ensuring the sustainability of marginalized girls‟ educational outcomes.

BRAC‟s project has been designed based on a number of key contextual factors. Afghanistan is one of the

poorest and most conflict-affected countries in the world, with four decades of political and civil conflict

and war leading to poor development outcomes. One of these outcomes is a large deprivation in

educational opportunities for children, particularly for girls. There is a range of supply and demand

factors that continue to pose challenges to girls‟ access to education in Afghanistan. For instance, supply

factors include: lack of accessible government schools, particularly for those girls living in remote areas

who must travel long distances to reach the nearest school; lack of female teachers; poor quality of

teaching; inadequate learning resources, school facilities and infrastructure; and a lack of safe

environments for girls in schools, including prevalence of corporal punishment by teachers. Demand

factors include: poverty, which is linked to requirements for child labour (particularly domestic labour for

girls); violence and insecurity; conservative cultural norms that restrict women‟s and girls‟ mobility,

linked to concerns about honour and chastity; negative perceptions of girls‟ education and participation in

other activities such as employment; and early marriage and subsequent pregnancy of girls.

Supply and demand barriers to girls‟ education in Afghanistan intersect in complex ways, and have

shifting impacts on girlsas they get older and transition through different stages of schooling, particularly

from primary to secondary school. As girls are preparing to transition into secondary school, they enter

adolescence and puberty, leading to family concerns about girls‟ chastity and honour (exacerbated by lack

of female teachers), more focus on marrying girls (linked to poverty and the need to secure bride price or

offset household expenditure), and increasing requirements for girls‟ domestic labour (including caring

for younger siblings and older or sick family members). A shortage of secondary schools, particularly for

girls, also exacerbates concerns about girls‟ safety when traveling long distances, with distances to

5

secondary schools from remote areas often greater than for primary schools.

In light of the barriers to girls‟ education outlined above, BRAC‟s GEC-T project includes a range of

activities that sit within four broad packages.

1. Provision of primary and secondary education and TVET vocational training for girls. BRAC

will support girls‟ primary, secondary and vocational education through three types of schools: CBE

primary schools (established in GEC 1) that will continue into secondary school grades; government hub

schools (primary and secondary) to which BRAC CBE students have transitioned since GEC 1; and

newly established TVET centres aimed at increasing girls‟ vocational skills. This provision of education

will include the development of gender sensitive curricula, extra-curricular and life skills activities,

distribution of stipends to highly marginalized girls (including poor, disabled and orphaned girls)

attending government hub schools, provision of chaperones (khalas) for girls attending government hub

schools, transport assistance to girls attending TVET centres, and a girls‟ mentoring and co-curricular

program implemented in government hub schools.

2. Teacher training. BRAC will deliver a package of teacher training at all three school levels, including

subject-based, gender awareness and child protection/safeguarding training. This will include refresher

and grade changing teacher training.

3.Community engagement and stakeholder advocacy. BRAC will engage and raise the awareness of a

range of stakeholders at the community level, including parents and other community members, in

relation to the importance of girls‟ educational and employment outcomes. This will include the

implementation of advocacy meetings with community members and school management shuras (SMSs),

community training orientations (e.g. on sanitation, corporal punishment and gender awareness) and

mother‟s forums.

4. School management strengthening. BRAC will strengthen school management through a range of

activities, including: conducting advocacy workshops with school and government stakeholders;

implementing training on gender awareness, corporal punishment and school management for SMSs,

school staff, community development councils (CDCs) and other relevant actors; supporting the linkage

between school child safeguarding standards and codes of conduct, and community and local authority

child protection mechanisms and systems; piloting sustainable CBE models; and sharing learning

products with school, government actors and other stakeholders.

BRAC‟s Theory of Change (see Figure 1) stipulates that the activities outlined above will contribute to

five intermediate outcomes: attendance, teaching quality, girls‟ life skills, improved attitudes and

perceptions, and improved school governance. By ensuring that all girls can access government schools,

CBE schools or TVET centres with qualified teachers that provide supportive and learning environments

and co-curricular activities, BRAC can provide high quality education for girls. By raising awareness of

6

and promoting girls‟ and women‟s rights, BRAC will improve attitudes and perceptions of girls‟ and

women‟s participation in education, employment and income earning activities, and, by providing girls

with important life skills, BRAC will ensure that girls are prepared and empowered to engage in these

activities. Supportive schools, communities and effective school management in accessible locations will

ensure attendance. Working with the community and government to improve school governance and

ensure adherence to international child safeguarding standards will boost attendance, retention and

completion and also work towards creating safe, efficient and sustainable school infrastructure and

knowledge in communities. Overall, an holistic, supportive and quality education, accessible to all, will

lead to good learning outcomes and ensure transition,while effective community engagement, the

strengthening of school governance and strengthened links between government bodies, government

schools and CBE schools will improve sustainability of infrastructure and project results. The Theory of

Change is based on several key assumptions: community, government and other relevant stakeholders

continue to support girls‟ education programming and participate in project activities;no external factors

such as conflict, natural disasters or political unrest impact negatively on project operations; community

members are receptive to gender awareness and engagement; and school stakeholders at all levels

collaborate to strengthen child protection and safeguarding.

BRAC will work in 10 provinces: Baghlan, Balkh, Hirat, Jawzjan, Kabul, Kapisa, Kunduz, Nangarhar,

Parwan and Samangan. The main beneficiaries of the project are girls attending three different kinds of

educational institutions: CBE schools at the community level (girls continuing their CBE from GEC-1),

government hub schools (girls who have transitioned from CBE in GEC-1 to government schools), and

TVET centres (girls from CBE communities in GEC-1 who dropped out of BRAC classes or did not

transition from CBE to hub schools, and will attend vocational training at the district level).The project

will work with other beneficiaries and stakeholders at the local, regional and national levels. At the local

level BRAC will work with parents, community members, community leaders (community shura leaders

and hub school shura leaders), Community Development Councils (CDCs), teachers, student organizers

(khalas) and education volunteers through school committees, mother‟s forums and parent meetings.

BRAC will also work closely with Child Protection Action Network (CPAN) groups and local

government officials to ensure cooperation and collaboration in relation to child protection. At the

regional levels, the project will engage with provincial and district level MoE officials, vocational skills

associations (e.g. ICT and midwifery) and partner NGOs for improving the overall performance of

schools and TVET centres. At the national level, BRAC will engage national bodies such as the MoE,

Ministry of Women‟s Affairs (MoWA), Ministry of Labor, Social Affairs, Martyrs and Disabled

(MoLSAMD), the Directorate of TVET, the Afghan Midwifery Association and national and international

development and research partners.

7

Figure 1: BRAC Theory of Change: GEC-T

8

The primary aim of this document is to provide a framework and guiding structure for the monitoring,

evaluation and learning of BRAC‟s GEC-T project. It should be used by: BRAC programming staff as a

cross-referencing tool to ensure that MEL objectives and activities are aligned with programming across

the life of the project; BRAC‟s monitoring and research staff, who will be responsible for conducting high

quality monitoring, overseeing the quality and ethical implementation of external evaluations, and

conducting issue-based research projects linked to GEC-T; and external evaluators, who will find the

document useful in guiding and designing evaluation activities, methods, processes and outcomes based

on BRAC‟s expectations. Across all these stakeholders, the MEL framework should be used as a learning

tool that can be updated accordingly over the life of the project based on how monitoring and evaluation

results are fed back into programming.

2. Learning from GEC 1

The BRAC GEC-1 endline evaluation identified a number of key challenges in the MEL approach that

will be addressed and mitigated in GEC-T. These are outlined below.

Monitoring

Several GEC-1 output indicators were not easily measurable in terms of number or percentages, with

strong reliance on descriptive qualitative data in evaluation methods and unclear pre-planning on how

outputs should be quantified. In GEC-T, the logframe will be designed to measure output indicators with

both quantitative and qualitative data (where relevant) with clearly labeled data sources.

The endline evaluation suggested that there might be some problems ensuring that stipends were

distributed to the most marginalized government school girls. In GEC-T, stipend recipients will be highly

marginalized CBE school girls who have transitioned to government hub schools. Given some evidence

to suggest that transition of girls from CBE to government hub schools may be more common in higher

income households, BRAC will ensure that stipend recipients are genuinely marginalized, with low

household income being an important measure of marginalisation in this context. Other measures of

marginalization include disabled and orphaned girls. BRAC will continue to appoint a stipend selection

committee as in GEC-1; however, committee members will additionally comprise community

representatives from BRAC CBE communities where girls reside, including parents (e.g. from the

Mothers Forum) and school and community shura members.

Evaluation

Tracking and recontacting longitudinal cohorts of girls was a challenge in all GEC-1 evaluation phases,

particularly for out-of-school girls. There were a number of reasons for these challenges, primarily, the

lack of a comprehensive tracking system with inconsistent ID numbers used at different evaluation waves,

and limited tracking data recorded for girls, their family members, alternative community contacts and

9

household addresses/locations. The lack of a comprehensive tracking system was complicated by the

sampling design, which drew from school-based sampling for CBE girls and subsequent household

sampling for girls‟ household members, and household-based sampling for out-of-school girls.

Furthermore, the tracking system was not effectively calibrated to track in-school girls who had dropped

out between evaluation waves. In GEC-T, these challenges will be mitigated by ensuring that effective

tracking, recontacting and sampling systems and approaches are designed (see section six of the MEL

framework). Furthermore, unlike in GEC-1, the GEC-T evaluation tracking system will be linked to

BRAC‟s beneficiary tracking database for all three groups of intervention girls (CBE, government and

TVET).

In GEC-1, BRAC‟s independent evaluator only conducted household surveys with girls‟ mothers or

female primary caregivers, and not with male household members (e.g. fathers, or older brothers or uncles

in single-mother households). This meant that KAP survey questions (i.e. on attitudes and perceptions

towards girls‟ and women‟s education, employment and participation in shuras) appended to the

household survey, were only answered by women, limiting the ability to quantify male household

members‟ changes in attitudes over time. In GEC-T, any KAP survey questions included in the household

survey will be asked of both male and female caregivers of girls.

The endline evaluation found some limitations in the methods for collecting attendance data. These

included the inability to conductunannounced attendance spotchecks, some gaps in attendance data

collection tools (i.e. no checklist to record actual same-day attendance recorded by teachers on days when

spotchecks were conducted), and oversampling attendance data in government schools (potentially

leading to a reduction in quality of attendance and spotcheck data where enumerators are requested to

record attendance for ALL classes in a school, but with insufficient budget or time allocated to ensuring

this can be realistically done). Challenges conducting unannounced spotchecks are difficult to mitigate

given the security context and BRAC‟s requirements to facilitate enumerators‟ contact with schools and

communities. Although unannounced spotchecks are not possible, BRAC field staff will support a semi-

announced approach, whereby school staff and community members will be advised of a one week

window in which data collectors will visit schools, with the actual day not to be announced. Gaps in the

attendance data collection tool will be corrected such that checklist options are included for the teachers

record of attendance on the day of the spotcheck and enumerators‟ spotcheck head counts. Finally, unlike

in GEC-1, in which government hub school attendance was recorded for all classes from grades four to

nine, in GEC-T attendance will only be recorded for the classes within BRAC‟s cohort grades (i.e. grades

five and six at baseline, and subsequent cohort grades at midline and endline evaluation waves).

The endline evaluation found that there were a number of data entry errors that were not corrected at

baseline or midline. These included EGRA EGMA time being included as a fraction rather than an integer

reflecting whole seconds left on the clock. There were also data entry errors at baseline and midline in

relation to incorrectly recording Persian or English numerals that may be different from one another but

10

look similar or the same. This impacted mostly on recording of age, which made it difficult to match girls

across longitudinal cohorts. BRAC will brief the external evaluator on these errors early in the evaluation

process to ensure that: tools are properly designed to avoid data problems; enumerators are trained in

avoiding data errors; and data entry staff are aware of possible sources of error and ways to avoid them. If

possible, and depending on the available resources of the external evaluator, BRAC will encourage digital

data collection to sideline data entry based on paper surveys.

A number of problems with EGRA and EGMA tools provided by the fund manager were found at

baseline, midline and endline. These included: formatting problems when using Dari and Pashto

(particularly the latter) in Microsoft Word; incorrect reading orientation of subtask tables; translation

errors; omission of mathematics equations; and structural problems related to the calibration of tool use in

Afghanistan (particularly in relation to ensuring that Dari and Pashto tools were equally calibrated, with

Pashto learners disadvantaged in a number of subtasks). Given that the external evaluator is expected to

design EGRA and EGMA tools for GEC-T, with the new addition of SeGRA and SeGMA tools for

secondary school grades, a number of procedures must be put in place when designing such tools to avoid

the errors found in GEC-1. The external evaluator should ideally be familiar with literacy and numeracy

tool design and proper calibration according to language, particularly in Afghanistan. Such an evaluator

may be difficult to find, in which case BRAC will seek additional support from the fund manager in the

design of tools. All literacy and numeracy tools will also be piloted prior to baseline data collection.

In GEC-1, only half of BRAC‟s target provinces were sampled for qualitative data collection, limiting the

ability of the external evaluator to draw from qualitative findings to contextualize and expand learning

based on the results of the quantitative data, particularly for learning, attendance and teaching quality

data. In GEC-T, qualitative data will be collected in all 10 provinces receiving BRAC interventions.

In GEC-1, qualitative interviews and focus groups with girls drew from the same methods and approaches

used with adults (a question and answer approach with follow up, and facilitated discussion in focus

groups). In GEC-T, the external evaluator will be encouraged to design and adopt more child friendly and

interactive qualitative methods, which may include playing games, drawing/painting, or role playing etc.

For example, rather than simply asking girls about teacher quality and classroom practices, girls could be

asked to do a role play where they enact a typical day in the classroom. BRAC also encourages the use of

methods such as auto-ethnography and life history mapping to examine qualitative data longitudinally.

Learning

In GEC-1, BRAC made few programming modifications as a result of evaluation findings, suggesting

that there was a lack of learning systems in place to ensure that evaluation and monitoring findings fed

back meaningfully into project design.Challenges are always shifting and changing in an unpredictable

context such as Afghanistan, and these challenges should be closely monitored and relevant changes to

11

MEL design and implementation made accordingly. In GEC-T, BRAC will ensure that learning feedback

mechanisms are in place, as outlined in section 11 of the MEL framework.

3. Monitoring

BRAC‟s monitoring department will monitor key project outputs and activities across the life of the

project. A Real Time Monitoring (RTM) system will be implemented in order to deliver timely feedback

to programming staff, and will be administrated via a web application and mobile devices. The project

will use three technology-based methods and tools for monitoring outputs and activities: (1) monthly

progress tracking ICT-based MIS tools; (2) monthly program quality monitoring using event-based

checklists and assessment forms; and (3) quarterly output monitoring.

The independent monitoring unit will appoint 10 Monitoring and MIS (MMIS) officers in 10 provinces to

ensure timely, appropriate and accurate data collection and broader monitoring tasks. The independent

monitoring team will randomly visit different intervention locations on a weekly basis to monitor the

quality of program activities. Following weekly visits, the MMIS officer will meet with the project team

members and develop an action plan based on the findings. A quarterly report will be created centrally

using the accumulated quality monitoring data and shared in the quarterly progress review meetings.

Outputs will be tracked on a quarterly basis following the logframe and both quantitative and qualitative

data will be collected to assess progress against targets, summarized in target versus achievement reports

as per BRAC‟s monitoring policy. A variety of tools and methods will be used to track progress against

each output (see Table 1). Analysis will be done using software such as Excel, SPSS and STATA, with

qualitative data to be analysed manually through thematic coding. All data will be disaggregated

accordingly, including by province, gender and other demographic factors. Quarterly reports will be

generated and shared with management, headquarters and donors, and will include a summary of findings

and recommendations.

Project monitoring will also involve a number of stakeholders external to BRAC. As per government

policy, the MoE will conduct joint monitoring of the project through coordinated and routine visits to

beneficiary schools by PED and DED monitoring officers in collaboration with BRAC. Joint monitoring

will also involve community members, including school shura members, parents and other local

stakeholders. The monitoring department will maintain close links with community stakeholders

throughout the project in order to track and quickly respond to any local challenges in project

implementation.

Monitoring data will be primarily used to inform project management. The head of the monitoring

department will be responsible for checking the validity of all reports compiled by monitoring staff and

giving feedback to the project implementation team to strengthen project performance and achieve

12

milestones and targets. Feedback will take place through regular monitoring results meetings with the

GEC-T project manager.

Monitoring results will also be used to inform the project evaluation. Although the independent evaluator

will not report against outputs, BRAC will provide summaries of monitoring data to enable the

independent evaluator to explore how effectively activities are feeding into outputs, intermediate

outcomes, and primary outcomes.

Table 1 contains a summary of each output indicator included in the project logframe, with corresponding

information on how outputs will be measured, relevant data collection tools and methods, rationale behind

BRAC‟s approach and the frequency of data collection for each output indicator.

Table 1: Outputs for measurement

Output Level at

which

measurement

will take place

Tool and mode of data

collection

Rationale Frequency of

data collection

Output 1.1: # learners

received the mentoring

and lifeskills training

(disaggregated by

mentor and mentee)

School

Monitor site visits, mentor and

mentee profiles,

activity/training observation,

life skills training enrolment

and attendance register, mentee

and mentor satisfaction and

program evaluation surveys,

interviews with mentors and

mentees, life skills and

knowledge survey

Two groups of beneficiary are

defined in the monitoring

approach, mentors and mentees.

Mentors will mentor large

groups of girls (mentees) who

comprise a cascaded knowledge

group. Both groups can be

monitored using the same

general methods (site visits and

observations of training and co-

curricular activities, satisfaction

surveys), however an additional

survey with knowledge- and life

skills-related questions should

take place with mentees to

measure how much knowledge

has effectively cascaded from

mentors to mentees.

Quarterly

Output 1.2: Number of

learners received co-

curricular activities

(disaggregated by

categories i.e. story

writing, wall magazine

etc.)

School

Co-curricular activity

enrolment and attendance

register, learner satisfaction and

program evaluation surveys

This output will focus only on

the measurement of enrolment

and participation in co-curricular

activities, and perceived quality

of these activities. Life skills

developed as a result of the

activity will be measured at the

IO level.

Quarterly


extremely

marginalized girls who

received incentives to

stay in school

(stipends etc)

School,

community

Minutes of stipend selection

committee meetings, stipend

recipient lists, interviews with

girls receiving stipends and

parents

Endline concerns about whether

all girls receiving stipends were

genuinely marginalized mean

that community members and

parents will be more involved in

the stipend selection committee

meetings. Minutes of these

Annually

13

meetings are a key way of

ensuring that stipend selection is

transparent, and can be

complemented with qualitative

interviews with girls and parents

Output 2.1: # of Govt.

school teachers, CBGS

teachers and MTs

received training on

basic teaching and

subject base learning

and active pedagogy

School

Teacher and MT training

enrolment and attendance lists,

teacher and MT training

satisfaction and evaluation

forms, qualitative interviews

with teachers and MTs

Teacher and MT training should

take place annually and thus

should be monitored annually.

Although several tools and

approaches can be useful to

measure this output, teacher

observations are not necessary as

they are being conducted at the

IO level for teacher quality.

Annually


CBGS teachers and

TVET instructors

received refresher and

grade changing

training

School

Teacher training enrolment and

attendance lists, teacher

training satisfaction and

evaluation forms, qualitative

interviews with teachers

Teacher refresher and grade

changing training should take

place annually and thus should

be monitored annually.

Annually


POs, MTs and SO

trained and involve

them in their role

School

Training enrolment and

attendance lists, training


forms, staff performance

reviews, qualitative interviews

with POs, MTs and SOs, and

the stakeholders they support

(i.e. teachers and girls)

The output indicator must

measure two levels: (1) whether

stakeholders are being trained

effectively and (2) whether they

are fulfilling their roles. Thus,

monitoring will require more

general monitoring tools used

during training (attendance lists,

training satisfaction forms) but

these should be complemented

with external evidence,

including annual staff

performance appraisals, and

qualitative interviews with the

stakeholders that staff will

support (e.g. girls supported by

khalas, and teachers supported

by master trainers)

Annually

Output 3.1:# of

teachers, staff and

education stakeholders

received training on

SRGBV and using the

knowledge to make

school environments

safe

School





with teachers and school staff,

gender, CP and GBV KAP

survey

Although monitoring will

require more general monitoring

tools used during training

(attendance lists, satisfaction

forms), it should also include the

measurement of change in

knowledge, attitudes and

practices at the school level.

BRAC will design a mini-KAP

survey to this end, with

questions related to CP,

harassment of girls (at school

and on the way to school) and

school practices to mitigate

violence and harassment.

Annually

Output 3.2:# of

mothers of GEC

learners are

School

Shura/council memberships and

meeting participation lists,

minutes of shura meetings,

This output will be measured

through tracking women‟s

membership in shuras and their

Quarterly

14

participating in

community

representative

structures (i.e. school

Shuras, local

education council,

district education

council etc.) and

supporting to enhance

girls education

attendance in meetings, and

measuring change over time. It

is known that women‟s

membership and participation

can often be tokenistic and

symbolic in nature, rather than

meaningful and engaged

participation leading to decision-

making. This aspect of

meaningful participation will

measured at the IO level (school

governance) – see further below.

Output 3.3: Men and

boys involved in "X "

project event, i.e.

community events,

school management,

reporting plan and

local level advocacy

School,

community

Project event and activity

registration and attendance lists

(disaggregated by gender, age

and type of event/activity),

KAP survey with men and boys

It is important to monitor how

many events have participation

from boys and men; however, it

is also important to understand

which events and activities they

are participating in, and if this

participation is leading to change

in knowledge, attitudes and

practices (e.g. if boys and men

are participating in school

management but not in

discussion groups about girls

harassment, this would not be

likely to change the status quo).

BRAC will conduct the same

KAP survey included in the

household survey with a

proportion of men and boys

participating in events and

activities and will track change.

Quarterly

Output 4.1:# of

mothers/parents

forums organized and

raising cumulative

voice to claiming the

rights of girls

education

Community

Mothers forum attendance

records and minutes (if

available), monitor observation

of mother‟s forum, case study

interviews with mothers

Mothers may not be literate and

may not be able to take meeting

minutes, however, if there are

literate/numerate or partially

literate/numerate members, they

will be asked to record at the

very minimum the number of

forum meetings per month and

the number of participants per

forum. This will be crosschecked

by monitor site visits to observe

forum meetings. Mothers will

also be interviewed to gauge any

examples of decision making or

change resulting from the forum

and these will be captured in

case studies.

Monthly forums

Quarterly

monitoring

visits

Output 4.2: # of

community

sensitization events

(i.e. meetings,

community dialogues,

open forums etc)

organized on gender,

Community

Community sensitization event

attendance records and minutes

(if available), monitor

observation of community

sensitization events, interviews

with community stakeholders

Given that community events

related to children‟s and girls‟

rightsoften have school staff

present (e.g. teachers,

principals), there should be

literate members who can take

minutes of meetings. Minutes,

Quarterly

15

children‟s rights&

violence, girls

education and

empowerment of girls

monitor observations of

meetings and interviews with

stakeholders will all contribute

to tracking stakeholder support

for girls‟ education and

empowerment and any obstacles

to this support that need to be

addressed in the quarterly

monitoring cycle.

Output 4.3: # of joint

monitoring conducted

involving government,

other agencies and

donors

Community

Government monitoring

reports, minutes of joint

monitoring meetings

GEC-T will see the continuation

of joint BRAC and government

monitoring of BRAC activities,

but with a strengthened

monitoring presence from the

FM country team. Joint

monitoring reports and meeting

minutes will provide an

objective measure of any

problems arising in joint

monitoring.

Quarterly

Output 5.1: # of

schools identify and

establish effective

reporting mechanisms

for mitigating GBV in

schools or on the way

to school

School

Hard copy referral reports,

interviews with teachers and

SMSs, CPAN meeting minutes,

CPAN referral closure reports

A key issue to track is whether

cases are being successfully

resolved after reporting and

referral. This will be done

primarily through CPAN‟s

closure reports (indicating

successful or unsuccessful

closure of a report)

Quarterly

Output 5.2: # of issues

identified in CP audit

and take measure to

address the issue in

time and appropriate

standard

School

Safeguarding audit reports

reviewed by BRAC

safeguarding committee, list of

recipients of reports (e.g. email

lists), minutes of CPAN

meetings

Although the production of

audits is important,

dissemination needs to be

tracked to ensure that audits are

accountable and useful to

stakeholders. It is important to

understand whether audit reports

are leading to building of

commitment in relation to CP

and safeguarding. This can be

measured through CPAN

minutes of meetings, by tracking

reference to audit reports and

actions.

Annually

Output 5.3: # of staffs,

teachers and others

affiliated to GEC-T

project endorsed and

following the child

protection principals

and standards

School

Standard CP assessment toolkit,

class observations, staff

performance reviews,

interviews with students

Monitoring should not focus on

staff and teachers‟ own

statements of project

endorsement, but rather any

evidence of change in behaviour

or knowledge. These

measurements require the use of

tools such as CP assessment

toolkits and interviews with

students to capture behaviour

and knowledge change.

Annually

Output 6.1: # School

Shuras trained and

made functional to

School,

community




Standard tools and approaches,

including training enrolment

lists, participant satisfaction

Annually (SMS

capacities

checklists to be

16

effectively school

management and

governance


with SMS members and

government officials, school

shura capacities checklist

surveys etc, can tell us about

how many SMSs or other

stakeholders are trained and

what they think of the training.

However, the same SMS

capacities checklist designed for

the evaluation will be

implemented through BRAC

monitoring to ensure timely

response to obstacles to capacity

building.

administered

every two

years, not in an

evaluation year)

Output 6.2: # Number

of advocacy events

conducted at district,

province and national

level to reduce the

girls education

challenges

Community,

government

Enrolment and attendance lists

of advocacy events, monitor

observation of advocacy events,

interviews with event

participants, advocacy event

reports

Although the output should be

measured quantitatively through

enrolment and attendance lists

for events, events should also be

observed by monitors who may

detect continuing barriers to

addressing girls‟ education

challenges, and who may need to

follow up with formal or

informal interviews with

advocacy stakeholders

Quarterly

Output 6.3: # of

communities that have

successfully mobilised

resources to support

schools

Government

Financial and time records for

in-kind and monetary

contributions, interviews and

focus groups with school

shuras, school staff, teachers

and PED/DED officials

It is expected that self-financing

will not occur suddenly and will

likely be preceded by smaller

indications of in-kind or

monetary contributions.

Tracking these contributions will

be an important part of

establishing the scale of self-

financing of CBE(see output

7.1). However, it is also

important to track successful

resource mobilization at hub

school levels. Qualitative data

from school shuras and school

staff will enable a deeper

analysis of enablers of and

challenges to successful

mobilization, with special

attention to examples of success

and what factors contributed to

this success.

Quarterly

Output 7.1: # Self

financing CBGS

demonstrated and

lessons documented

Community

Financial and time records for

in-kind and monetary

contributions, interviews and

focus groups with community

members (e.g. parents), SMSs

and other community

stakeholders

Evidence of successful

mobilization of resources is

captured under output 6.3.

Output 7.1 is more related to the

community ability to take over

CBE classes. This output will

draw from financial and time

record for contributions, but will

largely be measured qualitatively

through interviews and focus

groups with community

members. These interviews will

track key issues annually,

Annually

17

including enablers and barriers

to self-financing, entry points to

mobilizing self-finance, and the

efficacy of community and SMS

mobilization in generating

support for community financing

models.

Output 7.2: # of ICT

based learning hubs

established for

providing distance

support to learners

Community,

ICT hub

Log of ICT learning

correspondence, interviews

with ICT support teachers,

interviews with girls receiving

ICT support

By virtue of its virtual platform,

number of correspondence

communication can be tracked

digitally. However, the quality

of the correspondence and

support should additionally be

measured through interviews

with ICT support teachers and

girls.

Quarterly

Output 7.3: # of

studies conducted,

lessons and best

practices documented

and disseminated

BRAC office

ToRs for research studies, list

of recipients of disseminated

reports and best practices (e.g.

email lists)

Key to measuring this output is

tracking dissemination, for

instance through email lists of

recipients. This will further

allow BRAC to follow up with

recipients to see if any research

uptake has occurred.

Annually

4. Key evaluation questions

In line with guidance from the GEC Fund Manager (FM), BRAC‟s MEL framework is based on a set of

key evaluation questions at the program level, and a set of sub-questions at the project level. Program-

level questions, rooted in the key GEC primary outcomes (transition, learning and sustainability), are:

1. Was the GEC successfully designed and implemented? Was the GEC good Value for Money?

2. What impact did the GEC funding have on the transition of marginalized girls through education

stages and their learning?

3. What works to facilitate transition of marginalized girls through education stages and increase

their learning?

4. How sustainable were the activities funded by the GEC and was the program successful in

leveraging additional interest and investment?

Project evaluation questions, based on the intermediate outcomes (attendance, teaching quality, life skills,

attitudes and perceptions, and school governance) and broader evidence of efficacy, are:

5. To what extent did the intervention contribute to a change in girls‟ attendance?

6. What impact does BRAC teacher training have on improving teachers‟ pedagogical skills,

gender- sensitive teaching practices, and safe and protective learning environments for girls?

7. What is the impact of mentoring and co-curricular activities on developing girls‟ life skills, and

18

how do these skills interact with primary outcomes (learning and transition)?

8. What impact has the project had on community members‟ attitudes towards and perceptions of

girls‟ safety, and girls‟ and women‟s education, employment and participation in civic activities?

9. How effective are school governance activities and capacities in creating safer environments and

more sustainable education for girls?

10. Was the project‟s theory of change supported? What necessary adjustments (if any) should be

made to the theory of change to ensure more effective programming?

11. What external contextual factors explain/mitigate outcome results?

5. Evaluation approach

This section of the MEL framework outlines the overall evaluation approach, including the research

design, strategies for measuring primary and intermediate outcomes, and the ethical protocols that must

be in place in order to conduct a rigorous and ethical evaluation, including ensuring child protection and

safe guarding procedures are in place.

5.1 Research design

A randomised control trial (RCT) is not feasible in Afghanistan for a number of reasons. Continuing and

deepening insecurity, and local disillusionment and distrust related to inadequate reach of development

projects, pose serious ethical challenges to an RCT approach, which would require explicitly randomizing

communities to receive interventions or not. Given key timelines and transition objectives of the project

(e.g. to transition girls through primary into secondary school and complete full cycles of education over

eight years), more ethical RCT approaches such as staggering entry of intervention groups (i.e. where

control groups receive interventions at later stages of the project cycle) are not feasible. Consequently, the

research design will be two pronged: (1) a quasi-experimental design tracking two cohorts of girls in

intervention groups (CBE and government hub school), and corresponding cohorts of girls in control

schools, and (2) pre-post test design for girls enrolled in TVET vocational education classes. Further

details of control groups and establishing a counterfactual, including the rationale for not including a

control group for TVET girls,are included in section6.2 of the MEL framework.

The project evaluation will take place over four waves: baseline, first midline, second midline, and

endline. The justification for two midlines is due to the length of the intervention (eight years) and

requirements to track progress and impact consistently and regularly across the intervention.

The research design will include four approaches to ensuring the rigour of the evaluation and establishing

causality of project impacts. First, the evaluation will compare outcomes for treatment and comparison

groups with a difference-in-difference approach, with change occurring in the comparison group

19

providing the counterfactual scenario to interventions. Second, the project will use longitudinal cohort

tracking to follow cohorts of girls receiving interventions across the life of the project. Longitudinal

tracking will be used primarily to track cohorts across two primary outcomes, learning and transition, and

these cohorts will be linked. Longitudinal cohorts will not be the same as in GEC-1 and will be newly

sampled at GEC-T baseline. Third, the evaluation will draw from both quantitative and qualitative

methods, comprising a mixed-methods evaluation that will allow the triangulation of various data sources,

and a more nuanced interpretation of what project change has occurred and why. Fourth, the evaluation

will integrate primary and intermediate outcomes. This will be done by collecting data at the school,

community and household levels in the same sampling clusters, matching different levels through school

and community numerical identifiers, and merging quantitative and qualitative findings through

qualitative and quantitative summaries on key outcomes and intermediate outcomes for each sampling

cluster.

5.2 Measuring outcomes

This section outlines key information on the primary and intermediate outcomes that will be evaluated

and the corresponding approaches and methods proposed to measure these outcomes. This information is

summarized in Table 2 and described in detail further below.Primary outcomes will be measured for both

treatment and control groups; however, intermediate outcomes will only be measured for treatment

groups.

Table 2: Outcomes for measurement

Outcome

Level at which

measurement will take

place

Tool and mode of data

collection Rationale

Frequency of

data collection

Primary Outcomes

Literacy School

EGRA/SeGRA,

qualitative interviews

and focus groups with

girls, teachersand

school shuras

EGRA/SeGRA will be tested at the

school level for CBE and government

school girls receiving BRAC

interventions. This will take place to

ensure that tested girls are enrolled in

BRAC supported schools. Literacy

tests will be complemented by

qualitative interviews and focus

groups with girls, teachers and school

shuras in order to contextualize

learning results. These interviews will

also take place at the school level.

Per evaluation

point

Numeracy School

EGMA/SeGMA,



girls, parents, and

teachers

EGMA/SeGMA will be tested at the

school level for CBE and government

school girls receiving BRAC

interventions. This will take place to

ensure that tested girls are enrolled in

BRAC supported schools. Numeracy

tests will be complemented by

qualitative interviews and focus

Per evaluation

point

20

groups with girls, teachers and school

shuras in order to contextualize

learning results. These interviews will

also take place at the school level.

Vocational and

life skills for

TVET learners

School (TVET centre),

household

Vocational

proficiencies and

knowledge survey,

Rosenberg Self-Esteem

(RSE) Scale and

General Self-Efficacy

Scale (GSE), qualitative

interviews with teachers


girls

The vocational proficiencies and

knowledgeof girls participating in

TVET training will be tested at the

school level (TVET centre) at baseline

and second midline (selection of new

cohorts of girls), and at the household

level at first midline and endline

(follow up of cohort). School-based

sampling is necessary when first

sampling girls in order to ensure that

enrolled girls are participating in the

evaluation. However, household

sampling is necessary for follow up as

TVET girls will no longer be studying

and will not be trackable at the school

level.The EE will need to create a

vocational survey tool based on the

core competencies and knowledge

expected for each type of vocational

course. Self-esteem and self-efficacy

measures (Rosenberg Self-Esteem

Scale and General Self Efficacy scale)

will be added to the survey to test

whether TVET training increases

girls‟ self esteem, confidence and

efficacy. The rationale for using the

above scales is in order to use

standardized measures and attempt to

apply these in the Afghan context.

Qualitative interviews with TVET

teachers and focus groups with TVET

girls will be conducted to complement

the survey data.

Per evaluation

point

Transition Household

Household

survey,qualitative

interviews and focus

groups with girls,

parents, teachers,

school shuras

Measuring transition will need to take

place at the household level as girls

may have dropped out of school or

completed vocational training between

evaluation waves and may not be

trackable at the school level.

Transition data from the household

survey will be complemented with

qualitative data collection in order to

understand pathways and barriers to

transition, and contextualize reasons

for successful and unsuccessful

transition.

Per evaluation

point

Intermediate Outcomes

Attendance School, household

School register, spot

checks, checklists

appended to

EGRA/SeGRA and

EGMA/SeGMA tools,

Attendance data from school registers

and spot checks will need to take

place at the school level. However,

additional data should be collected at

other levels in order to triangulate the

Per evaluation

point

BRAC will

additionally

21

household survey,



girls, parents, teachers

and school shuras

core attendance data. Questions about

girls‟ reported attendance and

perceptions about attendance

(including barriers and enablers) will

be appended to EGRA/SeGRA and

EGMA/SeGMA tools. Questions

about parental perceptions of girls‟

attendance (including barriers and

enablers to attendance) will be

included in the household survey,

which will enable triangulating

different data sources and

disaggregating girls‟ attendance by

household and demographic factors.

Perceptions of attendance will also be

collected through qualitative

interviews and focus groups to enable

a more nuanced understanding of how

the project activities have impacted on

attendance, and enablers and barriers

to attendance.

conduct

spotchecks and

record school

register

attendance

biannually

Teaching

quality School, household

Classroom

observations, Active

Pedagogy Inventory

(API) tools, Teacher

Value-Added (TVA)

modeling, checklists

appended to

EGRA/SeGRA and

EGMA/SeGMA tools,

household survey,



girls, parents, teachers

and School shura

members

Teaching quality should be measured

primarily at the school level through

classroom observations, and questions

appended to girls‟ EGRA/SeGRA and

EGMA/SeGMA tools. BRAC

encourages the EE to adopt innovative

methods such as API and TVA. At the

household level, questions will be

included in the household

questionnaire to capture parental

perceptions of change in teaching

quality. Inclusion of teacher quality

questions in literacy/numeracy tools

and household surveys will enable the

integration of outcomes and IOs,

particularly how teaching quality

intersects with learning and transition.

Qualitative data will assist in

collecting and interpreting

stakeholders‟ perception of change in

teaching quality over time.

Per evaluation

point

Life skills School, household

Rosenberg Self-Esteem

(RSE) Scale and

General Self-Efficacy

Scale (GSE) appended

to EGRA/EGMA and

SeGRA/SeGMA tools,



girls (mentors and

mentees), teachers, and

parents

Life skills will be primarily measured

through a survey appended to

EGRA/EGMA and SeGRA/SeGMA

tools. BRAC recommends usingRSE

and GSE scales, as per the learning

outcome for TVET girls. Qualitative

data will assist in interpreting how life

skills interact with other outcomes

(including learning and transition).

Per evaluation

point

Attitudes and

perceptions Household, community

KAP survey questions

included in the

household survey (for

A mini KAP survey included in or

appended to existing tools (i.e.

household survey, learning tests or

Per evaluation

point

22

parents) or appended to

learning test tools (for

girls),or appended to

school shura

management checklist

(for shura members),



teachers, parents, girls,

school shuras, and

khalas

school shura management checklist) is

an efficient way of quantifying shifts

in attitudes and perceptions over time,

particularly if using an ordinal scale

such as a Likert scale. However, it

needs to be complemented with

qualitative data to contextualize shifts

in attitudes and perceptions,

particularly since there are a range of

activities contributing to this IO.

School

governance School, community

School shura

management checklist,

focus groups with shura

members, interviews

with PED and DED

officials

School shura members will complete a

checklist with results aggregated

across the shura and then

disaggregated by gender. The

checklist will thus be based on self-

reported perceptions of capacity and

skills, and knowledge related to

BRAC training. The results of the

checklist will be triangulated with

focus group data with shura members.

Central to the GEC-T‟s GESI

approach is understanding how gender

inclusive school shuras are and how

BRAC‟s activities are promoting

gender equality. These questions will

be answered through cross-referencing

the results of the shura checklist with

qualitative data. Interviews with PED

and DED officials will provide

additional information on how

government has supported school

governance, including through

funding and resourcing, and what

remaining challenges there are to CBE

hand over or self-sustainability.

Per evaluation

point

Learning

The outcome indicator for learning is: Number of marginalized girls supported by GEC with improved

learning outcomes. For BRAC‟s project, learning outcomes will be measured at three levels: literacy,

numeracy and vocational skills.

Literacy and numeracy will be measured for two BRAC intervention cohorts, girls attending CBE schools

and government hub schools, and will not be measured for TVET girls as vocational education will not

include literacy and numeracy skills training. Literacy and numeracy will be measured through EGRA

and EGMA tools (for primary grade students) and SeGRA and SeGMA tools (for secondary grade

students), and this will be done at the school level through random sampling of girls‟ enrolment lists.

This is to ensure that all girls sampled are in fact enrolled in a BRAC intervention, with household

23

sampling running the risk of selecting girls enrolled in a non-BRAC school. While this will be more

straightforward at the CBE level (as we know that all girls enrolled in CBE classes are BRAC

beneficiaries), care must be taken at the government hub school level to ensure that selected girls were

previously enrolled in a BRAC CBE intervention in GEC-1 (rather than always enrolled in a government

hub school or other type of non-BRAC school intervention). Consequently, enrollment lists used for

random sampling will be provided by BRAC, not by the government hub school. The primary

measurement for learning will be whether girls have achieved a 0.25 SD increase above the benchmarked

mean at each evaluation wave.

Given that literacy and numeracy tests may be administered to disabled girls (one of BRAC‟s

marginalization target groups), efforts must be made to ensure that such tools are accessible and do not

further marginalize girls by excluding them from participation in the evaluation. According to BRAC‟s

recent assessment of special needs of girls, a large proportion of girls with disabilities have learning

disabilities (including autism) and physical disabilities (including hearing problems, speaking problems,

difficulties walking and difficulties seeing). The external evaluator will be expected to incorporate

inclusive approaches to collecting learning data from disabled children. These may include printing

literacy and numeracy stimuli in large fonts, speaking loudly, slowly and clearly, ensuring that tests are

taking place in a location that physically disabled children can access, and considering applying more

time in timed subtasks for girls who have difficulties speaking.

Literacy and numeracy test results will be complemented by qualitative interviews with teachers and

focus groups with girls, parents and school shura members. Qualitative interviews and focus groups with

teachers and girls will be conducted at the school level, focus groups with parents will be conducted at the

community level, and focus groups with school shura members will be conducted at the school level in

government schools and at the community level in CBE communities. Qualitative data will assist in

contexualising the results of learning tests. Key questions to be addressed through the qualitative data

may include:

How do girls‟ learning experiences and perceptions of teaching quality intersect with learning

results?

How do teachers‟ perceptions of their own classroom practices intersect with girls‟ learning

results?

How do household attitudes and practices (e.g. support for girls‟ education, pressure for girls‟ to

conduct household labour, parental literacy, and support of girls‟ learning and homework etc.)

impact on learning?

How do school shura practices (e.g. classroom monitoring, parent engagement etc.) impact on

learning?

What are the barriers to girls‟ learning and how can BRAC address these barriers through

programming modifications?

24

The third learning outcome, vocational and life skills, will only be measured for girls who are enrolled in

TVET training (life skills as an intermediate outcome will be measured for government hub school girls

engaging in mentoring and co-curricular activities). This learning outcome will be measured by

administering a survey testing girls in key competencies, skills and knowledge related to the specific

vocational courses they are enrolled in. Self-efficacy and self-esteem scales (Rosenberg Self-Esteem Scale

and General Self-Efficacy Scale) will be included in the survey. As noted elsewhere in this document, two

different cohorts of TVET girls will be tracked for the evaluation: cohort one, which will be sampled at

baseline and followed only up until first midline; and cohort two, which will be sampled at second

midline and followed up at endline. Vocational skills will be tested at the school level (TVET centre) at

baseline and second midline, and at the household level at first midline and endline. This is due to initial

cohort sampling being more efficient and straightforward at the school level, but with follow up

evaluation waves having to occur at the household level due to TVET courses only running for six

months, with girls unable to be tracked longitudinally at the school level. The EE will be expected to

design a short survey with knowledge- and skills-based questioning related to each of the core

competencies expected for each type of vocational course. Girls will only answer the questions related to

the core competencies of the training they have participated in (i.e. girls doing beauty parlour training

would not be asked questions about carpet weaving).

The vocational and life skills survey will be complemented by qualitative interviews with TVET teachers

and focus groups with TVET girls. Qualitative data will be primarily directed towards exploring girls‟

perceptions of the quality of TVET curriculum and teaching, their own confidence in their new

knowledge and proficiencies, and perceptions about future prospects based on newly acquired skills.

Transition

The outcome indicator for transition is: Number of marginalized girls who have transitioned through key

stages of education, training or employment (primary to lower secondary, lower secondary to upper

secondary, training, employment or other).

Successful transition is defined in different ways across the three different types of cohorts.

For primary school girls, completing primary school and entering into and completing lower-

secondary school

For lower-secondary school girls, completing lower-secondary school, and entering into and

completing upper-secondary school, or entry into safe, fairly paid employment (if age

appropriate)

For upper-secondary school girls, completing upper-secondary, and enrolling in post-secondary

education (e.g. tertiary), TVET, or entry into safe, fairly paid employment

25

For TVET girls, completing TVET training, sufficient acquisition of vocational skills, or entry

into safe, fairly paid employment (if age appropriate).

For cohort girls in primary CBE and government hub schools, successful transition is defined solely as

entry into and completion of lower-secondary school. Employment is not thought to be successful

transition as employment among primary school children would usually be an indication of child labour

rather than the employment expected after successful transition through secondary school. It should be

noted that girls enrolled in CBE or government hub schools who drop out of primary or lower-secondary

school will not be eligible for BRAC TVET training.

For lower-secondary school girls, successful transition involves completing lower-secondary school, with

two possible transition pathways, entry into and completion of upper-secondary school, enrollment in

TVET, or entry to safe and fairly paid employment. BRAC will not provide TVET training for girls in

this category, hence, entry into TVET would not be linked to the intervention (i.e. girls may enter into

TVET implemented by another governmental or non-governmental organisation). Entry into employment

is in part dependent on girls‟ age. Although employment before the age of 18 is classified as illegal in

Afghanistan, it is not realistic to think that girls cannot enter into safe and fair employment before the age

of 18. For instance, girls completing lower-secondary school may be recruited to teach CBE classes,

particularly in remote areas where accessing female teachers is challenging. Hence, although age is a

defining feature of whether girls should be employed, there should be some flexibility around determining

if this is successful transition (or child labour). This will be done by adding appropriate questions in the

household survey tool (see below).

For upper-secondary school girls, successful transition is defined as completion of upper-secondary and

entry into post-secondary education (e.g. tertiary), TVET, or safe and fairly paid employment. As per

lower-secondary school girls, BRAC will not provide TVET training for girls who have dropped out of or

completed upper-secondary school, and so any entry into TVET would not be linked to the intervention. It

is assumed that on completion of upper-secondary school, girls are at an eligible age for employment.

For all girls in primary and secondary schools, if they remain enrolled but fail a grade and repeat it, this is

classed as unsuccessful transition, although the same girls may transition in future evaluation waves by

progressing on to the next grade.

In relation to TVET girls, it should be noted that BRAC is not conducting any direct support activities

that would lead to TVET girls finding suitable employment; however, BRAC will conduct indirect

activities, such as sharing information about TVET graduates with district and province level skills

associations and the Directorate of TVET with the aim of establishing links between vocationally trained

girls and government and non-government groups that may assist with their future employment. Although

employment in safe and fairly paid employment may be one type of transition for TVET, many girls will

26

be too young to legally work. Consequently, BRAC additionally defines successful transition where girls

complete TVET and acquire sufficient vocational skills, with the view that girls may be able to use these

skills in paid and safe employment when reaching a suitable age.

Measuring transition will take place at the household level. For CBE and government school girls,

learning and transition cohorts will be the same at baseline. At subsequent follow up evaluation waves,

girls in the learning sample must be replaced if they have dropped out of school. However, girls in the

transition sample (sampled at baseline) must be followed up at all evaluation waves, regardless of their

enrollment status, in order to capture an accurate understanding of their different transition pathways over

time. Due to the unpredictability of drop out rates, an appropriate attrition buffer will need to be

established to ensure that a sufficient number of girls in the transition sample are retained throughout the

project (see section 6.4 of the MEL framework). As TVET girls enroll in vocational training for six

months, and those sampled at baseline will not be enrolled in TVET at midline or endline evaluation

waves, transition of TVET girls must also be measured at the household level. TVET girls are not as

susceptible to drop out due to the short course implementation time; however, attrition buffers must still

be established due to other possible reasons for attrition (see section 6.4 of the MEL framework). In

contrast to the transition sample for CBE and government school girls, which will be tracked and

followed from baseline to endline, TVET girls will only be tracked for one evaluation wave.

Consequently, TVET girls will be sampled at baseline and tracked and surveyed at the first midline

evaluation point. At the second midline evaluation point, a new cohort of TVET girls will be sampled and

then tracked and followed up on at endline.

Transition for all three cohorts of girls will be measured through a household survey tool. Questions will

include: current enrolment status and type of school, past year enrolment status and type of school, grade

(for CBE and government hub school girs), current employment status, past year employment status and

type of employment. Questions should also be asked about the quality of current employment and income

earning activities, whether girls are remunerated for their work, and whether they feel safe and respected

in their employment environment. For all three cohorts of girls, transition will be measured quantitatively

through a nominal variable (yes or no) indicating successful transition or not. The transition figure entered

into the logframe will be the total beneficiary number (total N) multiplied by the transition rate (% of

sampled beneficiaries who have transitioned). The independent evaluator will also include sub-reporting

on the frequency of different types of transition pathways.

Measuring transition will also require adopting qualitative methods in order to build a deeper

understanding of mediating pathways and barriers to girls‟ transition. Qualitative interviews and focus

groups will be conducted with girls (both those who have successfully transitioned and those who have

not), parents (mothers and fathers) of girls who have transitioned or not, teachers (for all three types of

school/training), and school shura members (for CBE and government hub schools). Qualitative data will

be collected from in-school girls, teachers and government hub school shuras at the school level, and from

27

dropped-out/non-transitioned girls, parents and CBE shuras at the community level. Key questions to be

addressed through the qualitative data may include:

Where successful transition has taken place, which project activities account for different types of

transition?

How have project activities reduced barriers to girls‟ transition?

What are the reasons for girls not succeeding in transition and how can BRAC address these

barriers through programming modifications?

What are the intersections of different categories of marginalization with transition?

Are there other types of successful transition not expected or included in the transition pathway

plan (for instance, re-enrollment of TVET girls into madrasas or formal schools)?

For girls who have entered employment, is this considered child labour, or are girls working in

safe, fairly paid employment? What does a safe working environment look like?

Attendance

The outcome indicator for attendance is: percentage improvement in girls' attendance.

Attendance will be measured for two BRAC intervention cohorts, girls enrolled in CBE schools and

government hub schools, and will not be measured for TVET girls as vocational education courses are

only six months in length. Attendance will be measured through attendance data collection forms

implemented at two levels: attendance recorded in school registers, and spot check attendance. For school

register attendance, enumerators will collect: total class enrolment, number of students present on the six

days in the week immediately preceding the data collection (i.e. six attendance figures collected). For

spotcheck attendance, enumerators will do a head count of sampled schools on the same day as collecting

school attendance register data. In addition to doing a head count, enumerators will record data for total

class enrolment, and the same day attendance number recorded by the teacher. BRAC will also do two

annual spot checks of attendance using the same method. In CBE schools, independent evaluator

attendance and spot check data will be collected for all sampled schools. In government hub schools,

attendance and spot check data will be collected for all grade 5 and 6 classes at baseline and subsequent

cohort grades at midline and endline.

There are a number of possible challenges in collecting attendance data. As noted in section 2 of the MEL

framework, conducting announced spotchecks is highly challenging in Afghanistan, where security risks

to enumerators must be mitigated by BRAC facilitating their entry into target communities and schools.

However, BRAC field staff will support a semi-announced approach, whereby school staff and shuras

will be advised of a one week window in which data collectors will visit schools, with the actual day not

to be announced. Another challenge is in assuming that the attendance registers kept by teachers are

accurate, particularly if they are aware that enumerators will be visiting to collect attendance data. This

28

challenge will be partially mitigated by collecting 6-day attendance in the week prior to data collection

and doing semi-announced spotchecks in the following week. Data will also be collected in classroom

observation checklists on whether teachers correctly call the attendance roll in order to triangulate

attendance data and determine its likely accuracy.

Self-reported quantitative attendance data will also be collected from girls and carers. Questions about

girls‟ perceptions of attendance, including self-reported attendance and reasons for non-attendance, will

be added to surveys and checklists appended to EGRA/EGMA and SeGRA/SeGMA tools. Questions

about carer reports of girls‟ attendance will also be included in the household survey. Although girl-

reported and carer-reported attendance are not the ideal measures of attendance, they can complement the

primary attendance measures by allowing more complex disaggregation of data (e.g. according to key

demographic characteristics).

Quantitative data will be complemented by qualitative data, including in-depth interviews with teachers,

and focus groups with girls, parents, and school shura members. Qualitative interviews and focus groups

with teachers, girls and government hub school shuras will be conducted at the school level, and focus

groups with parents and CBE school shuras will be conducted at the community level. Qualitative data

will assist in contexualising the results of the attendance data. Key questions to be addressed through the

qualitative data may include:

What are the enablers of girls‟ school attendance and to what extent have BRAC project activities

(e.g. stipends, transport incentives, mothers‟ forum, parents‟ meetings, mentors, khalas, school

shura management, community engagement) supported these?

What are the persisting barriers to girls‟ school attendance and how can BRAC strengthen its

programming to respond to these barriers.

Teaching quality

There are two outcome indicators for teaching quality:(1) percentage of teachers demonstrating improved

capacity in subject-based learning, gender-sensitive pedagogy and child-centred learning, and (2)

percentage of schools that have established child-friendly learning environments and following updated

teaching-learning approaches.

Teacher quality will be measured for teachers in all three treatment groups (CBE schools, government

hub schools and TVET centres), primarily through classroom observations and checklists. When

designing classroom observation checklist tools, the independent evaluator should be familiar with

BRAC‟s teacher training packages and ensure that tools are properly calibrated to capture the key teacher

skills, pedagogy and practices that are included in training.Observation checklists will also measure

teacher corporal punishment and respectful, child friendly behaviours in the classroom. At each

29

evaluation wave, the teaching quality indicator achievement will comprise a % of checklist items

observed in classrooms for child-centred and gender-sensitive pedagogical practices, and appropriate

child protection practices (i.e. non-violent disciplinary methods).In addition to standard classroom

observation methods, BRAC encourages the independent evaluator to include more innovative methods,

such as Active Pedagogy Inventory tools, and Teacher Value-Added (TVA) modeling.

Improved capacity in subject-based learning cannot be reliably measured through a classroom observation

as it will be challenging to control which class subject will be observed, and all subjects would ideally

need to be observed, which is not feasible from a time and budget perspective. Furthermore, enumerators

would theoretically be required to understand the key skills and observed behaviours that would comprise

adequate subject-based teaching, which is beyond the scope of the evaluation. Hence, subject-based

learning will be measured through qualitative methods (see further below).

Teaching quality will also be measured through surveys appended to the EGRA/EGMA and

SeGRA/SeGMA tools in order to capture girls‟ perceptions of teaching quality (and improvement in

teacher quality), and questions in the household survey for carer perceptions of teaching quality (and

improvement in teacher quality). Questions asked of both girls and carers will also address teacher

absence and corporal punishment.

Qualitative interviews with teachers, and focus groups with girls and school shuras (all at the school

level), will be conducted to assist in triangulating and contextualising the data from classroom

observations. Key questions to be addressed through the qualitative data may include:

What are teacher perceptions of the quality of their own teaching and their confidence in teaching

and how have these changed over time?

What are girls‟, parents and school shura members‟ perceptions of teaching quality and how have

these perceptions changed over time?

What other factors may explain teaching quality?

Has subject-based training (e.g. maths) impacted on teacher confidence and girls‟ learning? How?

Has grade change training adequately equipped teachers to deliver quality education at higher

grade levels?

Life skills

There outcome indicator for life skills is: percentage of learners with improved level of self esteem or self

confidence.

This outcome is linked to BRAC‟s mentoring program and co-curricular activities (e.g. debating, story

telling, drama, reading competitions and wall magazines). All the activities are implemented in

government hub schools only,and are hypothesized to lead to girls‟ improved life skills, defined as

30

enhanced self-esteem, self-confidence and self-efficacy. Therefore, the outcome is measured only at the

government hub school level. Life skills will be measured through a survey appended to EGRA/EGMA

and SeGRA/SeGMA tools, with appropriate measures to be determined by the independent evaluator. For

self-esteem, self-confidence and self-efficacy, BRAC suggests using scales that have been standardized in

other contexts such as the Rosenberg Self-Esteem Scale and General Self-Efficacy Scale (GSE), with

attention paid in piloting to the cultural and linguistic appropriateness of these scales in Afghanistan.

Qualitative methods will include focus groups with girls (mentors and mentees), and teachers, all at the

school level, and focus groups with parents. Qualitative research questions may include:

How effective has the mentoring program been in increasing girls‟ life skills, learning and

attendance? Why/why not?

How do girls‟ life skills contribute to other key outcomes, including learning, attendance and

transition?

Have girls‟ life skills had any impact on family attitudes and perceptions about girls‟ education,

future employment and broader civic participation?

Attitudes and perceptions

There are two outcome indicators for attitudes and perceptions:(1) percentage of parents reporting

improved attitudes towards girls education and employment, and (2) percentage of girls reporting that

they feel confident and safe while attending and travelling to and from schools and TVET centres.

This outcome is linked to a variety of BRAC activities, and can be broadly separated into two key areas:

(1) supportive attitudes towards girls‟ and women‟s education, employment and civic participation

(linked to advocacy meetings with community members and school management shuras, community

training orientations covering gender awareness, mother‟s forums, and gender awareness training with

school shuras); and (2) perceptions of girls‟ safety and comfort traveling to school or TVET (linked to the

provision of stipends and travel incentives, and school chaperoning by khalas). Due to the various

stakeholders whose attitudes and perceptions may change, BRAC recommends measuring this

intermediate outcome with parents and girls (both for perceptions of safety and supportive gender

attitudes), and with school shuras (for supportive gender attitudes). This will be done by appending the

same KAP survey to the household survey (for parents), learning tests (for girls) and school shura

management checklists (for school shuras).

Although KAP survey questions will allow the primary measure of a percentage of stakeholders with

positive attitudes and perceptions, qualitative data will allow for a more nuanced understanding of how

and why change has happened, and how much this change can be linked to BRAC‟s activities or broader

contextual issues.For instance, girls‟ and parental perceptions of girls‟ safety declining could be linked to

31

broader insecurity, or linked to BRAC‟s activities not being implemented sufficiently (e.g. khalas not

accompanying girls to school anymore). Furthermore, given the range of activities linked to this IO,

qualitative data will assist to identify why change is happening and how it can (or cannot) be attributed to

BRAC activities. Qualitative interviews will be conducted with teachers at the school level, and focus

groups will be conducted with girls and (gender segregated) school shuras at the school level and parents

at the community level. Qualitative research questions may include:

What is the strongest driver of change in attitudes and perceptions?

Which BRAC activity has contributed most to changing attitudes and perceptions?

School governance

The two outcome indicators for school governance are: (1) Responsiveness and accountability of school

shura members improved in “X” percentage from the baseline, and (2) number of schools with further

funding/government support/other and serving the education demands of “X” number of students.

School shuras will be established and/or supported in CBE and government hub schools. TVET centres

will not have shuras and, thus, this outcome indicator is linked only to CBE and government hub schools.

This outcome indicator will be primarily measured through a school shura management checklist to be

filled out by school shura members. The checklist will be developed by the independent evaluator, and

will include items related to BRAC‟s shura capacity and knowledge training, such as shura advocacy

skills, school monitoring and auditing skills, knowledge and observation of child protection standards and

codes of conduct, and knowledge of gender-sensitive school development plans. Responses to checklist

items will be aggregated across all responding shura members to form a % score indicating shura skills

and capacities. This score will be disaggregated by gender in order to track capacity for both male and

female members over time.

The second indicator, schools obtaining further funding or other types of contributions, is included as

output 6.3 in the monitoring plan, and will be primarily measured by BRAC through financial and time

records for in-kind and monetary contributions (this will require BRAC to develop a monetised value

system for in-kind contributions). Although this data will not be collected at outcome level, the EE will

draw from monitoring output data and complement it with qualitative data aimed at exploring the

opportunities and challenges for self-financing or advocating for school funding (see below).

Qualitative data will be collected through (preferably gender segregated) focus groups with school shura

members, and in-depth interviews with PED and DED officials. Qualitative research questions may

include:

How effective has shura training been in preparing shuras to effectively manage schools? What

more could be done to support shuras?

32

Has the shura been able to mobilise in-kind or monetary contributions for the school? What are

the barriers and enablers of doing this?

Has the shura been able to mobilise support for community take-over of CBE classes? What are

the barriers and enablers of doing this?

To what extent is school shura membership gender inclusive? Do women participate

meaningfully in oversight and accountability processes? What are the barriers to their

participation?

What is the relationship between shura attitudes towards girls‟ and women‟s participation (see

previous IO) and women‟s meaningful participation in school governance?

How can BRAC programming promote more meaningful participation of women in school shuras

and reduce barriers to their participation?

What kind of support has the PED and DED provided for the continuity of schools, and has this

support changed over the life of the project? Why/why not?

5.2.1 Sustainability

The GEC-T outcome for sustainability is that the project can demonstrate that the changes it has brought

about which increase learning and transition through education cycles are sustainable. In line with GEC

guidance, sustainability will be measured at three levels (school, community and system), against a

Sustainability Scorecard with ratings between 0 and 4 for each level. The ratings are defined as below:

Score of 0: Negligible sustainability (null or negative change)

Score of 1: Latent sustainability (knowledge development and change in attitudes)

Score of 2: Emerging sustainability (changes in behaviour)

Score of 3: Sustainability becoming established (critical mass of behaviour change)

Score of 4: Established sustainability (changes are institutionalized)

The independent evaluator will determine ratings at each evaluation wave based on progress on key

indicators.

BRAC has developed a set of indicators for each of the three sustainability levels.

School

Indicator 1: Percentage of schools with established child-friendly learning environments and following

updated teaching-learning approaches.

Indicator 2: Percentage of trained teachers and school management shura members continuing their

services.

33

Indicator 3: Percentage of CBE schools operating with community financing.

Community

Indicator 1: Percentage of communities actively involved in government schools, CBE secondary schools

and vocational schools.

Indicator 2: Percentage of communities actively involved in negotiation with the government and

mobilizing resources to continue girls’ education.

Indicator 3: Parents’ perceptions and attitudes changed X% from baseline.

System

Indicator 1: Percentage of CBE students and teachers absorbed by the government education department.

Indicator 2: Policy makers are responsive and providing necessary support to improve girls’ education.

Indicator 3: Responsiveness and accountability of SMS members improved in “X”% from the baseline.

Table 3 indicates the types of measurement, sources, methods, rationale and frequency of data collection

required to measure indicators across the three key sustainability levels.

Table 3: Sustainability outcome for measurement

Sustainability

Level

Where will

measurement take

place?

What source of

measurement/ verification

will you use?

Rationale – clarify how you

will use your qualitative

analysis to support your

chosen indicators.

Frequency

of data

collection

School School

School

School, community

1. School curriculum checklist

item, school CP and child

safeguarding policy checklist

item, qualitative interviews

with school staff, teachers and

SMSs

2. Teacher and SMS

registration lists (BRAC

monitoring), qualitative

interviews and focus groups

with teachers and shuras

3. Financial and time records

for in-kind and monetary

contributions (available from

BRAC monitoring), interviews

and focus groups with school

staff, SMSs, community

members (e.g. parents) and

There is a combination of

quantitative and qualitative

methods proposed to measure

sustainability at the school level.

School curriculum and

CP/safeguarding checklist items

can be appended to a school

survey to be implemented along

side teacher observations.

Teacher and SMS registration

lists and financial and in-kind

contributions will be obtained

from BRAC monitoring data.

Qualitative interviews and focus

groups at the school and

community levels will be central

to measuring enablers of and

barriers to sustainability and

Per

evaluation

point

34

other community stakeholders

measuring the scale of

sustainable outcomes according

to the scorecard ratings.

Community Community

Community,

government

Household,

community

1. Qualitative interviews and


members (parents, community

leaders, religious leaders)

2. Financial and time records

for in-kind and monetary

contributions (disaggregated

by type of contributor),

qualitative interviews and


members (parents, community

leaders) and government

officials

3. KAP survey on gender

attitudes and perceptions

(appended to household

survey), qualitative interviews

and focus groups with parents

Records of financial and time

contributions that will be used to

measure other outcomes and

outputs will be key for

understanding how communities

and community members are

specifically contributing,

requiring disaggregation of data

by type of contributor.

Qualitative data will be central

to measuring sustainability at the

community level. While KAP

surveys can measure change

over time, qualitative data will

assist in understanding how

sustainable these changes are

and the enablers of and barriers

to sustainability (e.g. perceptions

may improve over time but

qualitative data may suggest that

these improvements are linked to

schools remaining open or

employment availability).

Qualitative methods will also

explore the types of community

engagement with schools, and

how this engagement could

continue in the absence of the

school (e.g. if TVET centres

close at the end of the project).

Interviews with government

officials will assist to triangulate

data on community engagement,

including government

perceptions of the extent and

efficacy of this engagement.

System School,

government

Policy level

School

1. BRAC student tracking

database, CBE teacher

tracking database, qualitative

interviews with teachers,

school staff, SMSs, and

PED/DED officials

2. Policy activity (e.g.

workshop) reports, and policy

outputs

3. School shura management

and capacity checklist, focus

groups with school shuras

Although BRAC‟s student

tracking database will contain

information on CBE student

transfers to government hub

schools, BRAC will also require

a CBE teacher tracking database

to understand the trajectory of

CBE teachers after project

completion. The extent to which

students and teachers have been

absorbed into the hub school

system, and the enablers of and

35

challenges to this occurring will

be measured through qualitative

methods. At the policy level,

measuring sustainable systems

will require a review of policy

documents, which may include

reports from events and activities

which BRAC has contributed to

(e.g. donor or MoE advocacy or

policy workshops) or policies

BRAC has contributed to (e.g.

CBE policy). Central to

understanding sustainability is

whether clear actionable points

from event reports or policy

documents have been produced

and followed up on. At the SMS

school level, while the shura

checklist will measure change in

capacities and competencies

over time, focus groups will

measure shura perceptions of

how much these capacities and

competencies have been

institutionalized and whether

internal processes have been put

in place for the continuity of

capacity and management

support.

5.3 Ethical protocols

The basis for BRAC‟s ethics framework is a set of four internationally recognized ethical principles for

research conduct (respect, beneficence, justice, research integrity), outlined below.

Respect for persons

Researchers must respect the dignity of all individuals involved in research and recognize their rights to

autonomy and self-determination, such that they can make their own choices and decisions to participate

in research (or not) based on informed consent and with no coercion. Respect for persons also involves

respect for their privacy, which requires ensuring confidentiality or anonymity, and guaranteeing

protection of data.

Beneficence

36

Beneficence in research ethics refers to an obligation to maximize benefits and minimize harm, by

anticipating any potential negative consequences of conducting the research and ensuring that procedures

for mitigating harm are in place.

Justice

Justice in research means to distribute the risks and benefits of research fairly and refrain from research

practices that reproduce injustice, or take advantage of or exclude marginalized populations.

Research integrity

The principle of research integrity is related to ensuring the quality, rigour, professionalism, transparency

and validity of research. Conducting research that is of poor quality should be seen as unethical and,

furthermore, an ineffective and inefficient use of resources.

5.3.1 Ethics

This section describes the ethical protocols that should be put in place to ensure that all BRAC GEC-T

evaluation activities are conducted according to high ethical standards. We present a broad approach that

links key ethical principles, standards and risk mitigating procedures at each evaluation stage (pre-data

collection and planning, data collection, data analysis, storage and report writing, and dissemination of

results). Some of these risk mitigation procedures are outlined in more detail in section 10 on risk

management. In the following section, we focus on a specific sub-component of BRAC‟s ethical protocol,

child protection, in order to outline how children‟s safeguarding standards will be ensured.

While this section outlines key standards and mitigating procedures, the external evaluator will be

expected to develop a comprehensive risk register detailing full details of research risks and mitigations in

line with their operational procedures. The external evaluator should also ensure there is an individual on

the evaluation team who is responsible for ensuring ethics protocols are systematized and implemented

throughout all evaluation waves.

In relation to national research ethics protocols in Afghanistan, there are no compulsory norms or policies

for government or private regulation of research or evaluation. The Ministry of Public Health does have a

recently established Institutional Review Board (IRB) that governs research ethics for health related

projects (primarily medical research with humans); however, there is no corresponding IRB in the

Ministry of Education or other ministries currently in place. BRAC‟s Research and Evaluation Unit will

thus takeresponsibility for ethics oversight and ensure that risk mitigation procedures are being

implemented by the external evaluator. The independent evaluator will be asked to provide BRAC with

regular (e.g. monthly) ethics reports detailing any ethical challenges and how these were addressed. In

37

relation to child safeguarding, and evaluator staff observing or participating in any type of violence or

harm, specific protocols will be in place (see further details in the following section on child protection).

Pre-data collection and planning

Prior to conducting data collection, the external evaluator in consultation with BRAC must ensure that all

research design and planning is conducted with a clear set of risk mitigation procedures in place.A key

risk mitigation procedure in the starting phase of the project is for BRAC to ensure the selection of an

adequately skilled and experienced external evaluator. Another vital risk mitigation procedure that cuts

across multiple ethical standards is the implementation of comprehensive training of evaluation staff,

including enumerators and interviewers (see Table 4, and section 9.1 of the MEL framework on training).

The design of tools and methodological approaches should also be ethically implemented through a range

of standard practices, including by piloting to ensure that tools and methods are appropriate and sensitive

to various forms of diversity and cultural context, and making any necessary adjustments.

In the pre-data collection and planning stage, the principle of justice is particularly important.

Marginalised populations should be included in the evaluation where relevant, and adequate

methodological and sampling procedures designed to ensure they are not excluded. For instance, in a

context where community leaders and other powerful persons often dominate discussions in focus groups,

sampling and methodological procedures should be mindful of ensuring that less powerful voices are

heard. Furthermore, where required, focus groups should be gender segregated due to conservative norms

in Afghanistan and to ensure that women‟s voices are heard. Given that BRAC‟s intervention in GEC-T is

targeting disabled children, procedures must be in place for the meaningful inclusion of these children, for

instance, by locating adequately accessible locations for interviews with physically disabled children, and

ensuring that adequate pace and clarity of interview processes are put in place for children with learning

or attention disabilities. Although the participation of marginalized and vulnerable groups in the

evaluation should be facilitated, this should always be balanced with an understanding of how

participation may burden these groups. When designing tools, the external evaluator should consider

which methods are essential and with whom, and avoid oversampling, collecting unnecessary data or

burdening participants with time intensive and obtrusive methods, particularly with the knowledge that

such methods may remove participants from important daily activities, including income-earning, school-

related and household duties and activities. For example, ethical research at the school level should avoid

taking children out of a learning context (i.e. taking children out of an active classroom) or taking

teachers away from their duties (i.e. conducting an interview during class times, thus leading to teacher

absence or the cancelation of class).

Table 4: Ethical standards and mitigating procedures in pre-data collection and planning procedures

Ethical

principle

Ethical standard Mitigating procedures

38

Respect

Respect the rights and dignity of all

people involved in the research,

regardless of age, ethnicity, gender,

religion, disability or other forms of

diversity

Train enumerators in non-discriminatory, culturally

sensitive and respectful behaviours when conducting

research.

Recruit female enumerators to interview women and

girls, and male enumerators to interview men and boys.

Train enumerators in how to obtain informed and

voluntary consent with no coercion, and how to respect

confidentiality and privacy of participants at all times.

The external evaluator should ensure that research tools

and methods have been piloted and are appropriate and

sensitive to age, gender, and other forms of diversity.

Beneficence

Do no harm and avoid any practices that

may harm, abuse, discriminate or exploit

participants.

Ensure that duty of care and any incident protocols and

reporting procedures are in place prior to data collection

and that field staff are properly trained in how to use

them.

Protect the wellbeing and security of

researchers.

The external evaluator should present adequate safety

and security plans for field staff, including any necessary

security training

Justice

Ensure that marginalized populations are

included in the research, where relevant.

Ensure that clear categories of marginalization are

identified, and that sampling approaches and methods

are inclusive of marginalized groups where relevant.

Avoid disproportionately burdening

marginalized populations.

Consider which methods are essential and with whom,

and avoid oversamplingor burdening participants.

When conducting focus groups or other time and energy

intensive data collection procedures, offer refreshments.

Research

integrity

Ensure that researchers have the

necessary skill and experience to conduct

the research

Implement robust selection procedures when choosing

an independent evaluator.

Implement comprehensive training of enumerators and

research staff.

Data collection

During data collection, the principle of respect is paramount to ensuring that research is conducted

consensually, privately and confidentially. Obtaining informed consent first involves ensuring that

participants are provided with sufficient information about the evaluation and their participation in it, and

that their consent is given with no coercion or pressure to participate. In Afghanistan, given very low

literacy rates, particularly among the marginalized populations targeted by BRAC interventions,

information should be provided and consent obtained verbally rather than through written information

sheets and signed consent forms.

Ensuring confidentiality and privacy of participants during data collection can be complex, particularly in

school settings, where there may be insufficient private settings, particularly in BRAC CBE schools

where there may only be one room where a class is taking place. Finding a private space in households

may also be difficult where other family members may insist on being present during data collection

procedures. Such challenges should be considered in the pre-data collection and planning stages, and

necessary mitigating strategies developed. For instance, enumerators should be trained in how to politely

request bystanders for privacy, and ways to conduct interviews (or terminate them) if privacy cannot be

39

established. During qualitative focus groups, ensuring complete individual privacy and confidentiality is

clearly not possible as there are multiple participants present. The limits of confidentiality should be

stated clearly in these circumstances.

Research integrity is also vital to ensuring that data collection is implemented rigorously and to a high

quality.External evaluators must have comprehensive monitoring systems in place to ensure that data has

been collected honestly and without fabrication, sampling and data collection procedures have been

conducted without bias, and data outputs are of a sufficiently high standard. In cases where data outputs

are poor or where there are doubts about whether data has been fabricated, the external evaluator must

have clear follow up procedures, including additional enumerator training and re-fielding of data

collection if necessary. In cases where enumerators are unable to implement sampling or data collection

protocols due to unavoidable or difficult to mitigate challenges (e.g. pressures from community members

to sample certain groups of people), these cases should be reported to the external evaluator and BRAC

and discussed in any evaluation outputs.

Table 5: Ethical standards and mitigating procedures in data collection

Ethical

principle


Respect Obtain informed and voluntary consent

without coercion

Provide sufficient information to participants before

obtaining consent, including details of the evaluation,

what participation entails, any benefits or risks of

participation, how the data will be used, and the limits of

confidentiality.

Participants should at no time be coerced or put under

pressure to consent to participate.

All participants should be allowed to withdraw their

consent at any time.

Consent should be obtained before recording

participants with any audio or video devices, or before

taking photos.

Ensure privacy and confidentiality Enumerators should respect the privacy of individuals,

including respecting participants‟ refusal to answer any

sensitive questions.

Enumerators must respect the confidentiality of

participants by not sharing any information obtained in

interviews with others.

Research

integrity

Ensure that research is conducted with

transparency, honesty and accountability

Enumerators must conduct all data collection

transparently and honestly, and must not falsify or

fabricate data.

The external evaluator must ensure that adequate field

monitoring systems are in place to monitor the quality

and legitimacy of data collection processes and outputs.

Data analysis, storage and report writing

40

Ethical procedures should be put in place for all post-data collection processes, including data cleaning

and analysis, storage and report writing. There are two key ethical principles that are central to ethical

conduct in these post-data collection processes. Respect and ensuring privacy and confidentiality of

participants requires following appropriate protection procedures related to storage, transfer and

destruction of data.For instance, external evaluators and BRAC will store contact details and other

identifying features of participants in a separate file, linked to the main dataset through a unique

numerical identifier. External evaluators and BRAC will also establish clear timelines and processes for

the transfer, retention or destruction of hard copy and/or soft copy data sources. To ensure participant

confidentiality, all quantitative and qualitative data sets will be de-identified before sharing with external

stakeholders (including the fund manager). All data included in reports will also be de-identified. If this is

challenging (e.g. when reporting data from a highly recognizable individual such as a highly ranking

government official), the external evaluator will make sure the limits of confidentiality where explained

during data collection and consider modifying or omitting identifiable details from reporting (such as

province of interview).

Research integrity is also vital during post-data collection processes. Data cleaning should be transparent

and where any associated procedures have involved the modification of raw data that has not contained

errors, justification for this by the external evaluator should be provided to BRAC.Data analysis

procedures should be as independent and objective as possible, and, where possible, subjected to inter-

rater reliability processes (e.g. independent coding of a proportion of qualitative data sets by two different

research staff) and peer review (e.g. feedback from more senior research staff in the external evaluator

team).Reporting should also be as independent and objective as possible, and external evaluators should

ensure that any limitations or biases in the findings are presented in reports and necessary caveats

included.

Table 6: Ethical standards and mitigating procedures in data analysis, storage and reporting

Ethical

principle


Respect

Ensure privacy and confidentiality Follow appropriate protection procedures related to

storage, transfer and destruction of data.

De-identify all datasets (quantitative and qualitative)

before sharing with stakeholders, including the FM.

De-identify all data included in reports.

Research

integrity

Ensure that research is conducted with

transparency, honesty and accountability

Data cleaning, analysis and reporting processes should

be transparent, and as independent and objective as

possible.Inter-rater reliability and peer review are key

processes that can support this transparency and

objectivity.

Dissemination of results

41

Sharing and disseminating the results of research and evaluation to key stakeholders is important so that

others may benefit, and should always be done unless the risks of doing so outweigh the benefits. There

are a number of ways to mitigate these potential risks. It is assumed that respectful practices such as

maintaining privacy and confidentiality through de-identifying data will already have taken place when

writing evaluation reports. However, further protocols should be in place to ensure that the dissemination

of results and reporting outputs will not harm research stakeholders, particularly marginalized and

vulnerable groups. Where necessary, BRAC will consider omitting any potentially harmful findings from

publically shared documents. BRAC will also ensure, as much as possible, that dissemination is inclusive

of participants and communities, and that marginalized groups (e.g. women, disabled persons, children)

are not excluded from dissemination activities. This will require some thinking of the best mediums,

settings and processes for dissemination, for instance: disseminating findings verbally rather than in

written form in order to be inclusive of illiterate persons; organizing separate dissemination activities for

men and women in more conservative settings where women may be discouraged from sharing spaces

with men, and ensuring that female staff are disseminating findings to women; creating accessible spaces

and dissemination mediums to encourage the participation of disabled persons; and ensuring that

dissemination activities take place with children and are child-friendly (see next section for further

details).

Table 7: Ethical standards and mitigating procedures in dissemination of results

Ethical

principle


Beneficence

Do no harm and avoid any practices that

may harm, abuse, discriminate or exploit

participants.

When disseminating reports, BRAC will be mindful of

any findings that may inadvertently harm participants

and, where necessary, omit these from any publically

shared documents.

Justice Ensure that marginalized populations are

included in the research, where relevant.

Ensure that findings are shared with research

participants and communities, and dissemination

strategies and practices are inclusive of marginalized

groups.

5.3.2 Child protection

Overall, the ethical protocols outlined in the previous section are applicable to both adults and children;

however, there are some more specific protocols that are relevant to ensuring protection and safeguarding

of children throughout the different stages of the evaluation process, particularly during pre-data

collection and planning, data collection and dissemination of results.These protocols will be overseen by

BRAC‟s Child Safeguarding Committee throughout the project.

Pre-data collection and planning

When bidding out and selecting an external evaluator, BRAC will require bid submissions to demonstrate

42

that external evaluators have extensive experience conducting research with children. If not already in

place, the external evaluator will be expected to develop a child protection policy, adhere to this policy

throughout the evaluation and ensure all research staff, including data collectors, have been briefed and

trained on the child protection policy and related practices and expectations. Due to poor or non-existent

governance structures in place at government levels to track and register child offenders, thorough

screening of data collection staff for evidence of past abusive practices towards children is unrealistic.

However, the external evaluator will be expected to do everything possible to ensure that recruited staff

do not have a history of child abuse or violence (e.g. by refusing to hire enumerators known to have acted

inappropriately with children in previous research projects).

When designing research methods, tools and sampling procedures, the external evaluator must ensure that

these are safe. For instance, questions in research tools should be sensitive and age appropriate to

minimize distress to children, and data collectors must understand how to deal with any possible distress

or trauma that may arise (e.g. by terminating an interview or providing referral services to children).

Prior to data collection, as part of the child protection policy, the external evaluator will be expected to

develop a code of conduct and incident protocols to deal with (1) any enumerator observation of violent

or inappropriate conduct, and (2) any enumerator participation in violent or inappropriate conduct. In the

first case, data collection staff may observe violence, for example, observing corporal punishment by

teachers in a classroom setting. Incident protocols should involve enumerators reporting incidents to

evaluator fieldwork supervisors or managers, who should then report them to directly to BRAC. It is also

possible that enumerators do not directly observe violence, but are told of violent practices or experiences

by children during interviews. This is a highly sensitive issue and must be treated carefully; however, if

children request assistance, research staff should be prepared with incident protocols and possible referral

contacts. Although referral services for children are limited in more remote areas, BRAC‟s collaboration

with CPAN (Child Protection Action Network) groups in target provinces could serve as an important

referral pathway.In the second case, data collection staff may breach the external evaluator child

protection policy by acting inappropriately, disrespectfully or violently towards a child. The external

evaluator‟s child protection policy must include mitigating strategies and, importantly, strategies for

monitoring enumerator behaviour and conduct with children. Any breaches found must be dealt with

accordingly, including by directly reporting such breaches to BRAC staff (who may refer such cases to

CPAN groups), immediately terminating employment of the enumerator in question (and taking any

punitive measures necessary), and ensuring the wellbeing of the child (referring to any health, justice or

other services if necessary).

Data collection

As outlined in section 5.3.1, data collectors must obtain informed consent before conducting any data

collection activities. When interviewing children, in most cases children‟s parents‟, guardians or

43

caregivers must also provide informed consent. In BRAC‟s GEC-T evaluation, this will be highly

challenging due to the proposed sampling procedures, which involve sampling children at the school level

and then visiting homes to interview parents or other household members. Obtaining individual consent

from all parents prior to doing school-based sampling would be very labour, time and cost intensive,

particularly for obtaining parental consent for girls enrolled in government schools or TVET centresthat

are far from CBE communities. A more viable alternative would be the following procedure (although

BRAC will welcome alternative solutions from the external evaluator). Prior to any data collection,

BRAC field staff will include an evaluation briefing session with parents in CBE communities (i.e.

parents of girls attending CBE schools, government schools or TVET centres) to share information about

the evaluation and obtain group consent. These briefing sessions could be linked to existing BRAC

activities, including Mothers Forum meetings, parent meetings or community advocacy meetings, and

BRAC field staff should take care not to exert undue influence or coercion when seeking parental

consent. If any parents refuse to consent, BRAC will record the details of the relevant girls (including

name, age, father‟s name etc.) and these details will be shared with the external evaluator, who will

exclude girls from the school enrolment lists before conducting random sampling. At the school level,

girls must also individually consent to participate in any evaluation data collection, and may refuse to do

so even if their parents have provided consent.

Although the above procedures are viable for obtaining the consent of girls‟ parents in CBE (i.e.

intervention) communities, they will be much more difficult to implement in government school control

communities, for a number of reasons. Girls may live in a number of different communities, near or far

from government control schools. BRAC staff would not realistically be able to find girls‟ households

and may not be able to securely access these communities. A possible alternative might be to have

government schools share information about the research with parents through evaluation information

sheets, request written consent, and only randomly sample participants from lists of students who have

parental consent. This latter option would likely have strong limitations. It may be unethical to place time

and resource burdens on already under-resourced government schools, particularly schools that are

receiving no intervention benefits from BRAC. Furthermore, sharing information and requiring consent in

written form would likely be highly inaccessible for illiterate parents and would likely lead to strong

sampling biases (i.e. students permitted to participate in the evaluation would be more likely to come

from more educated and literate households). A more realistic alternative could be to seek consent only at

the school level, from girls and teachers, assuming teachers can act as appropriate guardians.

Much like when obtaining consent, ensuring children‟s privacy can sometimes be more complex than for

adults. Although children‟s privacy should be encouraged, there are limitations to this that the external

evaluator and enumerators need to be aware of. At the school level, as noted in section 5.3.1, finding a

private setting may be challenging, particularly in smaller schools, such as BRAC CBE schools, where a

classroom may be the only available space and may be occupied with other students. External evaluator

44

experiences in GEC-1 suggest that students and even teachers may be curious about surveys and

interviews and may try to observe or interfere (for instance, by helping students when conducting literacy

and numeracy tests). At the household or community level, parents or guardians may insist on being

present during children‟s interviews, and it is within their right to do so.Having others present during an

interview may influence how children respond to some questions. It may also place children at risk when

asking sensitive questions,for instance, questions about children‟s experience of violence at school or in

the community (i.e. asking a student about school corporal punishment when a teacher is present may lead

to further corporal punishment being enacted against the child in the future). Enumerators spending long

periods of time alone with children in private interview settings, in both school and household settings,

may also lead to concerns about child safety, particularly among girls in a cultural context such as

Afghanistan where honour and chastity and concerns about sexual violence are prominent.

The potential scenarios outlined above raise a number of concerns about both data quality and integrity,

and possible risks to child safety. Safety to children should always take precedence; however, often both

data quality and child safety can be balanced given the proper preparation and considerations. For

instance, enumerators should be trained in how to ensure appropriate privacy and confidentiality of

children. The external evaluator should be mindful of possible risks when designing tools, and if

necessary minimize highly sensitive questions. The external evaluator may also consider including

questions in data collection tools about who was or how many people were present during the interview in

order to contextualize results. Ensuring that all girls are interviewed only by female enumerators is an

important way to mitigate risks related to concerns about girls‟ safety and honour; however, it is not

necessarily sufficient as female enumerators may also act disrespectfully, violently or inappropriately

with girls. It may be necessary for enumerators to shift traditional notions of privacy from physical

privacy to auditory privacy, and attempt to conduct interviews out of earshot of others but where the

interview can always be observed and seen by others.

One anticipated challenge in BRAC‟s GEC-T evaluation is how to manage child protection when

transferring sampling from the school to the household level (only relevant for girls in the evaluation

intervention group, not the control group which will not include household-based sampling). According to

the proposed sampling procedures, girls living in CBE communities will be sampled first at the school

level (either in a CBE school, government school or TVET centre) and then enumerators teams will travel

to girls‟ households to interview household members. The ways in which enumerators‟ movement

between schools and households will be facilitated could have some ethical consequences for children if

they are traveling with enumerators between the school and household. If girls are attending CBE schools

in their own community, it is possible for enumerators to enlist the help of teachers, BRAC staff and

community leaders to help find girls‟ households and minimize or eliminate enumerator time spent alone

with girls. However, this process is much more complex for those girls attending a government hub

school located far away from their community. The external evaluator should design an approach to

45

overcoming this challenge, which may include collecting girls‟ information (name, age, father‟s name,

mother‟s name, phone number, and household address or information about any landmarks to assist in

finding households), then traveling alone to communities on a separate day and enlisting the help of

community leaders and BRAC staff in findings girls‟ households. Any approach proposed by the external

evaluator should be both realistic and ethical, and should take into consideration potential time and

budgeting implications.

Dissemination of results

Dissemination of results to children should be inclusive and child-friendly. Different categories of

children, including younger and older children, literate and illiterate children, abled and disabled children,

married and non-married children, and poor and less poor children, may require different dissemination

practices and mediums. BRAC may not have the financial and human resources to target different

dissemination activities to multiple categories of children (as well as adults as outlined in section 5.3.1);

however, BRAC will attempt to balance any requirements based on child diversity with overall project

requirements based on budgeting and resources. BRAC will also ensure that the dissemination of results

will be child friendly. This may require avoiding written outputs given literacy rates among some girls

may be limited, and should lean towards verbal presentation of findings in creative and interactive ways

(e.g. through role play). BRAC may also consider linking dissemination with, and thus strengthening,

project activities, for instance: supporting teachers to disseminate research results with their students

through the use of classroom techniques included in BRAC teacher training (e.g. role play, group

discussion); or supporting mentors and mentees in government hub schools to incorporate the

dissemination of key research findings into extra-curricular activities such as public speaking or debating.

6. Sampling framework

BRAC‟s sampling framework is included as Annex C to the MEL framework and includes information on

sampling clusters (treatment schools and communities), matched control schools and communities, and

how target groups are represented within clusters.

6.1 Target groups

In Afghanistan, girls are marginalized from educational outcomes due to a range of factors, including

their location (a large proportion of the country‟s population live in remote or rural locations), poverty,

conservative attitudes about girls‟ education, and conflict- and disaster-induced displacement. BRAC

identifies three levels of marginalization among its girl beneficiaries: (1) girls are marginalized due to

contextual, economic and social factors, including distance from schools, poverty, and conservative

communities that discourage girls‟ education, (2) girls are marginalized due to poverty and embedded

gender norms, live in remote areas affected by conflict and are likely to face harassment traveling to

schools, and (3) girls are highly marginalized and live in the most remote and conflict-affected areas and

46

face the highest levels of poverty and gender discrimination. According to BRAC program learning,

approximately 60% of its beneficiaries face level one marginalization, 13% face level two marginalization

and 27% face level three marginalization. Across all three levels of marginalization, there are beneficiary

girls in additional categories of marginalization, including girls with disabilities, girls who are orphaned,

girls from ethnic minorities, and girls who experience violence or abuse, who face additional challenges

such as lack of mobility, discrimination and social stigma. Furthermore, once girls enter adolescence, they

often become increasingly marginalized from educational opportunities due to concerns about honour and

chastity, and family preferences for marriage or work over education, frequently restricting their

transition to secondary school. All BRAC girls will fit into this category of marginalization.

Consequently, BRAC defines its primary marginalization group as girls living in remote, rural or un-

served urban areas. Sub-categories of marginalization include:

Girls living in extreme poverty

Girls with disabilities

Orphaned girls

Girls who experience violence or abuse

BRAC is aware that some girls face multiple layers of marginalization and is currently undertaking

analysis of data on its target beneficiaries to identify those girls who fall into marginalisation subgroups.

As noted elsewhere in the MEL framework, there are three key treatment groups in GEC-T: (1) girls

enrolled in CBE schools, (2) girls who completed BRAC GEC-1 CBE education and transitioned to

government hub schools, and (3) girls who dropped out of BRAC GEC-1 CBE schools (or did not

transition to hub schools) enrolling in TVET. All girls across the three treatment groups can be classified

as marginalized according to the primary category of marginalisation (living in remote, rural or un-served

areas); however, treatment groups 2 and 3 may experience heightened barriers to learning and transition

due to the need to travel to hub schools or TVET centres (the latter located in district centres).Girls

belonging to the sub-categories of marginalization will be found in all three target groups; however, these

marginalized girls may also face distinct barriers to learning and transition across the key target groups.

Poor, disabled, orphaned and adolescent girls may face particular barriers due to traveling distance and

lack of mobility (for physically disabled girls), lack of safe, accessible and affordable transport (for poor,

disabled and orphaned girls), requirements for child labour (for poor girls), and family concerns about

honour and chastity (for adolescent girls).

Under each treatment group, it is possible to define target groups that will benefit from specific BRAC

intervention activities. These are listed below.

Treatment group 1:

47

Target group 1.1: Disabled girls receiving special physical needs assistance (e.g. ramps, seating

and large text books)

Target group 1.2: Girls who have experienced violence or abuse

Target group 1.3: All other in-school girls

Treatment group 2:

Target group 2.1: Poor, disabled and orphaned girls receiving stipends

Target group 2.2: Girls receiving mentoring and participating in co-curricular activities

Target group 2.3: Poor, disabled and orphaned girls receiving transportation assistance from

school organizers (khalas)

Target group 2.4: All other in-school girls

Treatment group 3:

Target group 3.1: Girls receiving transportation incentives/stipends

Target group 3.2: All other girls in TVET

A community mapping exercise has not been completed at the time of drafting the MEL framework.

However, BRAC does not expect any significant sampling bias for transition of girls enrolled in CBE

schoolsas they are all located in girls‟ communities. There may, however, be a possible sampling bias for

girls in GEC-1 CBE communities enrolled in government hub schools as the distance of CBE

communities from hub schools varies. Distance in Km between girls‟ communities and hub schools, and

the location of government hub schools (rural, semi-urban and urban), are included in hub school lists,

allowing for a stratified sampling of schools. Data is not currently available for distance between GEC-1

CBE communities and district centres for TVET girls, and this will be difficult to attain prior to baseline

data collection given that girls will be traveling from a large number of catchment areas to district centres.

However, BRAC will provide general information on average distances, allowing for stratified sampling

that will include, at the province level, a balance of remote and less remote district-level TVET centres.

6.2 Control groups / Counterfactual scenario

The research design is two pronged, including a pre/post design for TVET girls, and a quasi-experimental

design for CBE and hub school girls. One comparison group consisting of non-intervention government

school girls will serve as the control group for both CBE and government hub school girls receiving

interventions (see Table 8). The reason for omitting a control group for TVET girls is due to the two

primary outcomes for TVET girls being specific to the intervention and linked to one another. TVET girls

are not receiving any literacy and numeracy training and so will not conduct learning tests, but they will

sit learning tests related to vocational skills development and knowledge. Transition for TVET girls is

also quite specific as employment is not necessarily a realistic transition pathway given BRAC‟s

48

programming does not directly assist girls to find employment. Consequently, one transition pathway is

the acquisition of vocational skills and knowledge, with the understanding that these skills may lead to

girls to find safe and fairly paid employment when they reach an acceptable age for employment.

Controlling for vocational skills and knowledge is not realistic and thus a pre-post test approach is more

appropriate.

Table 8: Treatment and corresponding control groups

Type of intervention Intervention cohort Control cohort

CBE education Girls enrolled in BRAC CBE classes

Girls enrolled in government schools NOT receiving

BRAC interventions Government hub

school education

Girls who have transitioned from BRAC CBE

classes in GEC-1 to government hub schools

TVET education Girls who have dropped out of BRAC CBE

classes in GEC-1 and enrolled in BRAC

vocational training

No control group (pre-post test design)

The comparison group will be determined through the identification of non-intervention government

schools matched to BRAC intervention CBE and hub school communities. BRAC will consult with PED

and DED officials to identify appropriate matched schools. Matching will be done primarily at the district

level to increase the likelihood of cultural, ethnic and linguistic homogeneity. Data for additional

matching criteria may be difficult to obtain at the community level (such as number of households and

population size); however, key data will be accessible at the school level, including rural/urban location,

gender composition (mixed-gender or girl-only), and whether existing interventions (non-BRAC) are

currently being implemented. Attempts will be made to match control schools where no existing

education intervention is taking place in order to minimize contamination of the control sample.

Access to government control schools will be facilitated through permissions with the corresponding

PEDs and DEDs. Although BRAC does not anticipate any challenges in obtaining PED and DED

permissions, it is possible that school staff may be resistant to having data collected in schools due to lack

of knowledge of BRAC and due to schools not currently receiving any BRAC interventions. Given that

BRAC staff do not work in control schools, their ability to facilitate independent evaluator access will be

more limited than for treatment schools. BRAC will coordinate closely with government officials to

facilitate independent evaluator access, including formal introductions to government school staff. At the

community level, there may be some challenges in collecting household data to measuretransition.

Community members may be suspicious of unknown outsiders entering the community, particularly when

asking questions about girls in the absence of any intervention. Thus, BRAC will seek additional

assistance from the PED and DED in contacting community shuras, CDCs or community leaders in order

to facilitate community access.

49

Additionality will be estimated through the difference-in-difference approach, with change occurring in

the comparison group providing the counterfactual scenario to CBE and hub school interventions. The

first assumption of a difference-in-difference approach, „the outcome trajectories of the intervention and

comparison groups would be parallel in the absence of the intervention‟, will be partially tested by

populating the sampling framework with secondary data on geographical and educational characteristics.

As noted above, it will likely be difficult to obtain more complex secondary data such as population and

demographic data; however, such data can be collected ex-post throughout the evaluation waves in order

to verify the first assumption. The second assumption of a difference-in-difference approach, „all

individuals in the target group are reached by the same intervention or interventions at the same time and

no individual of the comparison group received any of the interventions‟, will be partially verified ex-post

by analyzing treatment and control group exposure to interventions. Further, as noted above, selection of

control schools and communities will be based on a prior mapping of any non-BRAC interventions being

implemented, with updated mapping of possible contaminating interventions being conducted throughout

the evaluation waves.

6.3 Cohort tracking

This section provides information on how girls in treatment groups and the control group should be

tracked longitudinally for both learning and transition. All girls in learning cohorts (CBE and government

hub school treatment groups and government control group) should be tracked at the school for the entire

length of the project and should be replaced if unsuccessfully tracked at any evaluation point. In contrast,

the transition samples for all treatment and control groups need to be tracked at the household level.

Furthermore, in order to successfully track transition over time, CBE, government hub schools girls and

control government school girls must not be replaced if they have been found to have dropped out of

school (i.e. when attempting to track learning cohorts at the school level). Consequently, all CBE and

government hub schools girls, and corresponding government control group girls sampled for the learning

and transition cohorts at baseline must be followed up throughout the project until endline. They should

not be replaced at midline or endline, as this would jeopardise the project‟s ability to learn about

unsuccessful transition pathways and thus remaining barriers to transition. For TVET girls, tracking

transition over time will differ. TVET girls will only be tracked for one evaluation wave. They will be

sampled at baseline and tracked and surveyed at the first midline evaluation point. At the second midline

evaluation point, a new cohort of TVET girls will be sampled and then tracked and followed up on at

endline.

6.3.1 Learning cohort

The learning cohort of CBE girls and government school girls will be sampled and tracked at the school

level. Random sampling will be conducted through BRAC enrollment lists in order to ensure that girls are

BRAC beneficiaries. These girls will be tracked through the design of a comprehensive tracking system,

50

with information about girls‟ names, age, grade, father‟sname,mother‟s names, name of community in

which she lives, and location of her household in the community (if known). Community and location of

household will facilitate visits to girls‟ household to conduct household surveys (see section on transition

below). This tracking system will be linked to BRAC‟s programming tracking system through a unique

identifier number. This will enable effective school-based tracking at midline and endline waves, as

crosschecks with BRAC‟s tracking system will provide information on girls‟ grade and enrollment status.

BRAC does not anticipate having any problems tracking girls moving from primary to secondary school

grade at the CBE level, as these schools will be in the same location. At the government hub school level,

tracking girls who are moving between primary and secondary may be challenging in cases where

primary and secondary, or lower-secondary and upper-secondary, schools are in different locations.

According to BRAC‟s field assessment of government hub schools, only three schools have education up

to 6th grade only, suggesting that tracking learning samples between primary and secondary school

transition will not be a large challenge. A total of 81 secondary schools in the hub school sample provide

education up until grade nine only (all remaining beneficiary hub schools provide education until 12th

grade). Presumably, when girls graduate from schools that do not provide 12th grade education, they will

have the opportunity to transition to lower-secondary or upper-secondary schools in accessible locations

in the same communities or districts. These types of shifts between different schools should be trackable

through the BRAC programming tracking system, and BRAC will share details about the location of

corresponding secondary schools.

The process for tracking the learning cohort (for both CBE and government hub school girls) will thus

involve the independent evaluator: (1) reviewing cohort girls listed in BRAC‟s tracking system; (2)

identifying whether girls have failed/repeated a grade or whether they have officially dropped out; (3)

approaching the school and providing cohort lists to school administrations who will assist in identifying

girls‟ classes (in government hub schools); (4) and sampling girls.

Although the linkage of the learning cohort tracking system to BRAC‟s tracking system (through unique

identifiers) is feasible for treatment groups, this will clearly not be possible for the government school

control group. Theoretically, the processes outlined in the previous paragraphs would still apply to control

groups, minus the linkage to BRAC‟s program tracking system. Girls will thus be tracked through the

design of a comprehensive tracking system, with information about girls‟ names, age, grade, father‟s

name and mother‟s names, as for treatment groups. They will then be tracked at midline and endline

follow up evaluation points by drawing from school assistance in identifying girls‟ classes and sampling

girls. Control girls may move from primary to lower-secondary or lower-secondary to upper-secondary

schools between evaluation waves. There are a number of ways in which BRAC and the independent

evaluator can handle this situation. First, when completing the sampling framework before baseline,

attempts should be made to record the names of other schools in the area to which girls may transition.

Second, enumerators will approach control schools sampled in the previous evaluation wave and request

51

assistance in identifying if tracked girls have transitioned to other schools (government schools should

record this information).

Girls participating in TVET training will not be tested for literacy and numeracy as their vocational

courses will not involve any learning related to literacy and numeracy. Hence, we would not expect to see

any significant improvements in reading and maths over time within this cohort. Instead, they will be

tested for vocational skills acquisition. As noted previously, TVET girls will be sampled in two separate

cohorts with each cohort only being followed up for one evaluation wave. Although the first learning data

collection wave for each cohort (baseline or second midline) will be implemented at the school level (i.e.

TVET centres), follow up (first midline or endline) will be done at the household level as TVET girls will

have completed their vocational training. In the first learning data collection wave, information for a

tracking system will be obtained, including girls‟ name and age, father‟s and mother‟s name, community

name, household location/address and telephone number (if available). Enumerators will then go to girls‟

community to obtain tracking data (see below). At follow up evaluation waves, girls in both the learning

and transition cohorts will be tracked at the household level.

6.3.2 Transition cohort

For CBE and government school girls, and government school control girls, at baseline the transition

cohort will be the same as the learning cohort. After conducting learning tests at the school level and

collecting girl details (including community name and household location, if the latter is known), on a

separate day enumerators will go to girls‟ communities (if different to the community in which the school

is located, which will be the case for girls attending government hub school) and track girls‟ households.

Given that girls‟ reports of household location may not be sufficient to find households, BRAC staff and

community leaders and shuras will assist enumerators to find girls. Enlisting the help of community

leaders and shuras will be particularly important in government control schools, where BRAC staff will

be unfamiliar with the community and will be unlikely to be able to assist.

At the first midline follow-up, CBE, government hub school and government control transition cohorts

will be tracked at the household level. Enumerators will first attempt to track girls at the school level for

learning tests. All girls sampled at baseline will then be tracked at the household level as part of the

transition sample, regardless of whether they were present in school on the day of the test or not, or

whether they have dropped out. In some cases, making this distinction will not be clear as girls may not

have formally dropped out but may not have attended school for many days or months. If girls who are

absent on the day of learning tests are tracked at the household level and found to be still enrolled in

school, learning tests should be administered to them at the household level. At second midline and

endline, girls in the transition sample will be tracked directly at the household level with no visits to

schools required.

52

For TVET girls, at baseline and second midline, transition cohorts will be sampled at the household level

from the same cohorts sampled at the school level for learning. Tracking from the school to the

community will take place according to the same processes outlined above for CBE and government

school girls, i.e. collecting information for the tracking system and enlisting the help of BRAC field staff

and community leaders and shuras to find girls‟ households. At first midline and endline (the two follow

up points for the two different cohorts of TVET girls sampled at baseline and second midline), TVET

girls will be directly sampled at the household level according to the same standardized procedures

outlined in this section for other cohorts.

Learning and transition cohorts will be matched through unique identifiers attributed to the same girls in

the cohort tracking system. However, it should be noted that any girls in the learning cohort that have

been replaced at midline or endline will not be included in the original transition cohort and thus cannot

be matched to this cohort. Throughout evaluation waves, attrition of girls from the learning cohort due to

drop out or other reasons is expected, as is attrition of the transition cohort. Consequently, as the project

progresses, we would expect to see a reduction in the collection of data from learning and transition girls

matched at baseline.

6.3.3 Replacement strategy

The independent evaluator will design a detailed replacement strategy for girls who cannot be sampled at

follow up evaluation points. These strategies will be different for different cohorts of girls. A

recommended approach is outlined below.

Learning cohort CBE, government hub school, and control government school girls will all be replaced if

unable to be tracked at follow up evaluation points. Due to original sampling of learning cohorts in the

treatment groups happening directly from BRAC enrolment lists, any replacements will be done by

conducting a random sample of the same class lists, including only beneficiary girls who are enrolled in a

corresponding grade to the girl who has been replaced. This would be done prior to post-baseline

evaluation data collection waves with assistance from BRAC so that enumerators are prepared with

possible girl replacements prior to data collection. In the case of government school control girls, this

process of random selection of enrollment lists will need to happen at the school level on the day/s of data

collection. Tracking data must be collected for all replaced girls.

As noted previously, if girls are not in school on the day of learning tests and are still enrolled, household

survey data collection will reveal this when enumerators collect transition data. In these cases, girls

should be administered learning tests at the household level. This means that in some cases, although

likely few, we will have a substitute girl at the school level but with the girl who has been replaced

retained (as they are still enrolled in school). In such cases, substitute girls should be eliminated from the

sample.

53

Transition cohort CBE, government hub school and control government school girls will not be replaced

and will be followed up across all evaluation waves. TVET girls will not be replaced as they will only be

tracked for one evaluation wave.

Due to unpredictable security across many of the provinces in which BRAC works, it is possible that

schools and communities will have to be replaced, for instance, if BRAC schools are closed or if

enumerators cannot securely access schools or communities. In such cases, whole communities will need

to be replaced. This will be done by randomly selecting a new cluster (school or community) from the

same province, with additional stratification to maintain a similar cluster composition as the cluster that

was replaced.

6.4 Power calculations and sample sizes

All sample size calculations have been done assuming a national level minimal detectable effect (MDE)

of 0.25 SD for learning and 10% for transition. The MDE would need to be higher for province-level

estimates. The below parameters are currently used for treatment 1 and treatment 2 for literacy and

numeracy outcomes and transition outcomes (the significance level and power defined are the default

values and therefore are not listed in the lines of code in this section).

Literacy and numeracy

Minimum detectable effect (MDE) = 0.25 standard deviations

Sig level = 5%

Power = 80%

Attrition = 30%

Intra-cluster correlation (ICC) = 0.1

Transition

Minimum detectable effect (MDE) = 10%

Sig level = 5%

Power = 80%

Attrition = 30%

Intra-cluster correlation (ICC) = 0.1

Power calculations and sample sizes were conducted using STATA 13. To assist with determining sample

sizes, the add-on Stata command “clustersampsi” was installed. Options within this procedure which are

used in this analysis are as follows:

clustersampsi, <options>

rho = intraclass correlation coefficient

alpha = the significance level of the test; default is alpha(0.05)

beta = the power of the test; default is beta(0.8)

54

k = number of clusters in each arm

m = average cluster size

mu1 = mean for sample 1

mu2 = mean for sample 2

sd1 = standard deviation of sample 1; default is sd1(1)

sd2 = standard deviation of sample 2; default is sd2(1)

Clusters, also referred to as primary sampling units (PSUs), are used in the analysis. At the CBE level,

the PSU is a community given that some communities have more than one CBE class and more than one

grade. At the government school level, the PSU is a school. At the TVET level, the PSU is a TVET

centre. A two-stage cluster sampling technique is used to first choose the PSU and then the sampling units

within the PSU. Sampling units within a PSU are the individual girls who will be assessed for learning

and transition.

Control group 1 is the same for treatment 1 and treatment 2. This is similar to a three-armed study, as it

will be of interest to the project to examine change within the two treatment groups against change in the

control group, but also examine any differences in change between CBE and hub school treatment groups.

Hence, it is recommended to use the same number of clusters in each arm; however, the size of the

control group may be reduced after analysing the control schools and communities in the sampling

framework (see further below).

Literacy and numeracy

The base-cases (see Table 9) have been run using the specified parameters and estimated clusters for an

initial view of the command. A sample of 20 clusters in this case is not allowed, and a number of clusters

below 40 is not feasible from a budget perspective due to large corresponding sample sizes, and an

insufficient mean number of students in CBE clusters to achieve the target sample (i.e. there is a mean of

35 students in each CBE cluster, and a mean of 27 students in each class).Given that clusters will be

further stratified at several levels, including province (10 provinces), district (average of four districts per

province for both CBE and hub schools) and grade (grades 5 and 6), it is recommended to select 70

clusters per group to allow for more representative sampling while retaining the lowest possible sample.

This would allow for a sample size of 490 girls doing learning tests in CBE (treatment group 1) and 490

in government hub schools (treatment group 2) after attrition buffers are applied (see Table 10).

Table 9: Base-case ‘clustersampsi’

clustersampsi, Number of

clusters/PSUs

Cluster/PSU

size

Sample

Size

k(20) rho(0.1) mu1(0) mu2(0.25) Infeasible with cluster randomisation

k(27) rho(0.1) mu1(0) mu2(0.25) 27 256 6,912

k(30) rho(0.1) mu1(0) mu2(0.25) 30 59 1,770

55

k(40) rho(0.1) mu1(0) mu2(0.25) 40 17 680

k(50) rho(0.1) mu1(0) mu2(0.25) 50 10 500

k(60) rho(0.1) mu1(0) mu2(0.25) 60 7 420

k(70) rho(0.1) mu1(0) mu2(0.25) 70 5 375

k(80) rho(0.1) mu1(0) mu2(0.25) 80 5 400

Table 10: Estimated sample with 30% attrition buffer

Number of

clusters

Cluster size

with attrition

(rounded)

Sample size

with attrition

27 333 8991

30 77 2310

40 22 880

50 13 650

60 9 540

70 7 490

80 7 560

A list of control government schools was not available at the time of developing the MEL framework, so

it is not possible to examine the number of available clusters, their location (province and district) and the

population of students. Consequently, it is recommended to sample the same number of clusters for

control government schools as for treatment clusters, and then the feasibility of number of clusters and

sampling sizes can be verified when control cluster lists are available. This would mean that 70 CBE

clusters, 70 government hub clusters and 70 control clusters would be sampled, with seven girls per

cluster receiving learning tests.

Transition

The current sample size of learning outcomes will satisfy the desired parameters for transition outcomes

at the national level, for both CBE and government treatment groups, where the mean size of CBE

clusters is 35 and the mean size of government hub clusters is 119 (although note that the figure 119 is the

mean number of GEC-1 CBE girls per cluster who should be transferred to the hub school, not the actual

mean enrolment number, which is not available). This is inclusive of 30% attrition. For both CBE and

hub school treatment groups, the learning outcome sample sizes will detect a difference of 10percent (the

MDE for transition). The first line of STATA code below is for CBE and the second is for hub schools.

The minimum sample will more than suffice to statistically cover the required precision at the national

level.

clustersampsi, binomial detectabledifference k(70) m(27) rho(0.1) p1(0.7)

clustersampsi, binomialdetectabledifference k(70) m(119) rho(0.1) p1(0.7)

TVET girls

56

Developing sample estimates for treatment 3 (TVET girls) through power calculations is not advisable

given that TVET girls will not have a MDE applied for either learning or transition. Given there are 40

clusters (TVET centres), four per province (at the district level), it is feasible to sample 20 clusters, two

per province. It is recommended to sample one central and one more remote district in order to capture

barriers and impact of BRAC activities in different kinds of locations. It is difficult to estimate how many

girls should be sampled in each cluster until more information is provided about how many girls will

enrol in each cluster; however, a minimal sample of 10 girls per cluster will be established until more

information is available.

Proposed sample

Table 11 contains details of the baseline sample for three treatment groups and one control group, with

the same sample to be followed up on at both midlines and endline. Table 12 contains further details of

the sample required for each type of primary data collection tool.

Table 11: Estimated total sample

Clusters in

Population

Sampled clusters Sampled units/PSU

(girls/school)

Sample Size

(# of girls)

Treatment 1 204 70 7 490

Treatment 2 300 70 7 490

Treatment 3 40 20 10 200

Control 1 Unknown 70 7 490

Total > 544 230 1670

Table 12: Sample of primary data collection types

Learning tests Household

survey

Attendance Classroom

observations

Treatment 1 490 490 70 70

Treatment 2 490 490 70 70

Treatment 3 N/A 200 N/A 20

Control 1 490 490 N/A N/A

Total 1470 1670 140 160

Qualitative sample

In GEC-T, qualitative data will be collected in all 10 provinces in one community per treatment group per

province, except where certain types of FGDs or IDIs are not required (for instance, mentor and mentee

FGDs are only required in hub schools) (see Table 13 and 14). A total of 120 FGDs and 65 IDIs will be

done at baseline and subsequent follow up evaluation waves.

57

Table 13: FGD qualitative sample

FGD

mothers

FGD

fathers

FGD SMSs FGD girls FGD

mentor and

mentees

Treatment 1 10 10 10 10 ---

Treatment 2 10 10 10 10 10

Treatment 3 10 10 --- 10 ---

Total 30 30 20 30 10

Table 14: IDI qualitative sample

IDI teachers IDI Stipend

recipients

PED official DED

official

Khala

Treatment 1 10 ---

10 10

---

Treatment 2 10 5 5

Treatment 3 10 5 ---

Total 30 10 10 10 5

6.5 Benchmarking

Learning

Learning benchmarking for CBE and government hub school girls will take place in BRAC government

hub schoolsincluded in the baseline sample. Benchmarking will take place with girls in grades 7, 8, 9, 10,

11 and 12 (grade six girls don‟t have to be benchmarked separately as they already comprise a treatment

group that will receive learning tests). Learning will be benchmarked for 50% of clusters sampled at

baseline in BRAC government hub schools. Two girls from each benchmark grade in each cluster will

conduct SeGRA and SeGMA learning tests, comprising a total benchmark sample of 480 girls (2*6*40).

Girls in each grade will be randomly selected from class enrolment lists. As girls included in

benchmarking will not comprise BRAC beneficiaries, school enrolment lists rather than BRAC cohort

lists will be used for sampling. Targets will be calculated for treatment cohorts based on a 0.25 standard

deviation increase in learning scores above the benchmark for the corresponding grade.

Transition

Transition will be benchmarked at the household level in communities where BRAC beneficiaries live. It

should be noted that all BRAC girl beneficiaries live in CBE communities. These are communities in

which CBE classes were located in GEC-1, some classes that have continued and others that have closed.

For those CBE classes that have closed, some beneficiaries are traveling to attend government hub

schools and others are attending TVET centres. However, transition benchmarking will not take place in

communities in which government hub schools are located, nor within communities in which TVET girls

58

reside. In the first case, this is because the girls in government school communities are not theoretically

receiving direct BRAC interventions and are thus not comparable to girls living in CBE communities.In

the second case, given that TVET transition is primarily defined as learning vocational skills,

benchmarking is not required.

The benchmarking sample of CBE communities in which CBE schools are active will be 50% of baseline

learning and transition sampling (n=40). Determining the total sample of CBE communities in which girls

are living but attending government hub schools is difficult given that each government hub school may

have taken over girls from multiple different CBE communities in a catchment area. According to BRAC

data, each treatment government hub schools has received transitioned girls from a mean of 5.36 CBE

communities. Given this complexity, the number of these communities to be sampled could be up to 80*5

(400), although this is not possible to establish prior to baseline data collection. It is recommended that

transition benchmarking take place in 40 communities in which government school treatment girls

sampled for the baseline reside. The process for selecting these communities will be managed by the

external evaluator, but will likely involve a combination of sampling strategies (e.g. stratified and random

sampling of communities) that will be possible once BRAC has more information on the communities in

which hub school girls reside.

Hence, transition benchmarking will take place in 80 communities. Transition benchmarking will take

place among 10- to 20-year-old girls/young women, with 5 girls sampled per community (total of 400

girls). All girls/women sampled for benchmarking must not be enrolled in a BRAC school intervention

(they cannot be attending a CBE class or grades 5 or 6 in a government hub school that BRAC is

supporting). The benchmarking exercise will involve administering a very small survey that will include

demographic questions on a household girl, and her current enrollment status, past year enrollment status,

and type of school and grade (if the girl is in school). BRAC recommends also including questions related

to employment and quality of employment, with details of age-appropriate work transition described

previously in section 5.2. It is envisaged that the transition benchmarking would only take approximately

10 minutes to conduct per household.

Sampling at the household level for transition benchmarking will be random (i.e. random walk), with a

screening strategy developed to (1) identify if there is any household girl not enrolled in a BRAC

intervention, and (2) if there are any girls in the household aged 10 to 20 (the estimated age range for

BRAC beneficiaries across the eight years of the project). Households must meet both criteria to be

sampled, with an additional random sample of eligible girls then taking place (e.g. through a kish grid).

BRAC anticipates some challenges with this process. In communities in which CBE classes are active,

screening should be straightforward by asking if any girls are enrolled in the local CBE class (under the

assumption that BRAC is not engaging in any project activities in these communities to transition girls to

government hub schools). For communities in which previous CBE girls have transitioned to government

hub schools, the process is slightly more complex as there may be some girls attending the government

59

school who are not direct beneficiaries (i.e. in grades 5 or 6). Furthermore, some parents may not know

whether BRAC is supporting the government school where girls are enrolled (for instance, there may be

more than one hub school in the area). Consequently, screening at the household level needs to occur

according to the same screening criteria listed above; however, additional information might need to be

shared with household members, including the name of the hub school that BRAC supports.

Total sample for benchmarking

The total benchmarking samples for learning and transition are listed in Table 15.

Table 15: Learning and transition benchmarking samples

Type of cluster Sampled clusters Sampled units per

cluster

Sample Size

Learning School (Gov hub) 40 12 480

Transition Community (CBE) 80 5 400

Total 120 880

7. Baseline study

Information about requirements for the baseline study and how these differ to requirements for midline

and endline evaluation waves is presented throughout the MEL framework. However, for clarity, key

requirements and expectations are briefly listed here.

The baseline will include piloting of all tools and benchmarking of learning scores and transition rates;

however, these processes are not expected to occur in future evaluation waves. One exception for piloting

is if completely new tools are designed in future evaluation waves, or if tools used at baseline require

substantial modification (for instance learning tools).

At baseline, all treatment CBE and hub school girls, and control government school girls, will be in

grades 5 and 6 grade. At future evaluation waves, cohort girls will be tracked at their corresponding

higher grade levels, except in cases where cohort girls have repeated a grade, in which case they will be

tracked at their current grade level rather than the expected grade level.

New cohorts of girls will be sampled at baseline (the evaluation will not track cohorts from GEC-1) and

these must be tracked across the project. New cohort girls should only be selected in future evaluation

waves for learning cohorts in the case where girls cannot be found or followed up on from previous

evaluation waves, or where whole communities must be substituted and completely new learning and

transition cohorts must be selected. One exception is for TVET girls, who will be sampled in two groups:

baseline cohort (tracked only until first midline), and a newly selected cohort at second midline (tracked

until endline).

60

Cohorts of girls selected at baseline will be tracked throughout the eight years of the project (for treatment

CBE and government hub school girls, and control government school girls), and learning and transition

cohorts will be matched. However, throughout evaluation waves, attrition of girls from the learning

cohort due to drop out or other reasons is expected, as is attrition of the transition cohort. Consequently,

as the project progresses, BRAC expects an overall reduction in the amount of data that can be collected

from the same learning and transition cohort girls matched at baseline.

8. Evaluation governance

8.1 Evaluation steering group

BRAC‟s Research and Evaluation Unit has primary oversight of the evaluation and is responsible for

ensuring that all evaluation waves are completed at a high quality. This will involve review of all

evaluation outputs, including any inception outputs, research and tool design, datasets (including

quantitative and qualitative datasets), and final reports. BRAC‟s Monitoring Department will also

contribute to independent monitoring of all fieldwork, enlisting the assistance of the provincial office

heads and monitoring officers to conduct random monitoring of fieldwork.

A Research Steering Committee will be formed for providing policy guidance and expert advice on major

M&E activities. The Committee will be formed by BRAC GEC-T Project Manager, GEC M&E Focal

person (BRAC AFG Research Unit), Representative from External Evaluator, Representative from GEC

project co-implementing organizations/M&E Focal point and an Education Research Expert from BRAC

International HQ. The committee will be chaired by BRAC AFG Country Representative.

The committee meeting will be held once in a year. The committee will review M&E Plan, take account

of M&E activities, and endorse all research reports.

8.2 External evaluator

BRAC will engage in a competitive tender for independent evaluators. A ToR for the tender is included in

Annex D of the MEL framework. The ToR will be announced publically and additionally shared with

known research organisations in Afghanistan, including the GEC-1 independent evaluator. Interested

organizations will be requested to submit technical and financial proposals to BRAC. A committee,

consisting of the GEC-T project manager, the head of the Research and Evaluation Unit, the procurement

and accounts department, under the guidance of the Country Representative, will review all proposals on

the basis of BRAC‟s selection criteria and due diligence processes. These criteria include: capacity of the

organization to plan and conduct the evaluation, fulfil scope of work requirements, relevant experience

and expertise, approach to planning and implementing the evaluation, risk management strategies, quality

61

assurance, research ethics, and proposed budget. Each proposal will be assessed against the criteria and a

composite score allocated. The highest technical scoring and best value for money evaluator will be pre-

selected by the committee. The BRAC country representative will interview the evaluator, and BRAC‟s

team will meet the principal researcher in order to gauge the organisation‟s suitability in undertaking a

complex quasi-experimental evaluation and its experience in both qualitative and quantitative research

design and data analysis (i.e. using SPSS/STATA and Nvivo software). Once a selection is made for the

baseline evaluation, BRAC reserves the right to retain the same evaluator in future evaluation waves

subject to high quality outputs, or re-tender the evaluation in future evaluation waves.

8.3 Data validation

BRAC will require the independent evaluator to establish a rigorous monitoring system of all evaluation

processes, including training, data collection, data entry, translation, data cleaning and data analysis (see

section 9 for more information). Details of this system (including any established monitoring systems and

how these will be adjusted or enhanced for the GEC-T evaluation) will be requested in the ToR for

external evaluator proposals and will be further developed in consultation between BRAC and the

selected independent evaluator. At a minimum, BRAC expects clear monitoring strategies to be

implemented at all key phases of the evaluation, including pre-data collection, during data collection, and

post data collection. Pre-data collection must include comprehensive training of research staff. During

data collection, monitoring and governance could include unannounced monitoring field visits to data

collection sites, back checks of previously sampled communities and data collection observations. At the

data processing level, data entry and translation should be checked by the EE (e.g. through double entry

or a % of re-entry, duplicate analysis of databases, and other checking procedures). BRAC staff will also

conduct monitoring of EE data collection, data entry and data products (e.g. transcripts). BRAC will

request weekly monitoring updates during the fieldwork and data entry, and will request all syntax for

data analysis in order to ensure accurate and rigorous analytical techniques have been employed.

9. Data quality assurance

9.1 Training

Experience from GEC-1suggests that comprehensive training of research and evaluation staff is required

to successfully implement the range and complexity of monitoring and evaluation data collection tools

and approaches. The external evaluatorwill be asked to provide recommendations on requirements for

training based on the existing skills and experience of enumerators. However, BRAC has the following

expectations for training on the evaluation background, literacy and numeracydata collection, qualitative

research, sampling, and ethics and safety.

Evaluation background. All enumerators should be fully briefed on BRAC‟s intervention and the

evaluation, including a full description of all BRAC project activities, evaluation objectives and research

62

questions.

Literacy and numeracy.EGRA/EGMA and SeGRA/SeGMA training should last for a minimum of five

full days and could include the following components: review of tool subtasks; overview of possible

challenges to collecting reliable data (e.g. working in noisy and possibly disruptive settings, providing

students with too much time on timed subtasks, wanting to assist students with subtasks) and how to

mitigate these challenges; practice using the tools; role plays and inter-rater reliability exercises; and

quizzes/tests. The external evaluator should ensure that all trainers have been sufficiently trained in the

tools and issues to be covered in the enumerator training, and will be responsible for checking the

progress of enumerators and correcting any errors that arise in the training.

Qualitative research. The quality of qualitative data in Afghanistan can be limited unless proper training

procedures are implemented. Qualitative training for GEC-1 varied in length between five and seven days

depending on the number of qualitative tools to be used (up to 10 different in-depth and focus groups

interview tools). Training should include:overview of qualitative research, review of tools, practice and

role plays, and qualitative tips (e.g. probing, follow up, and establishing rapport).

Sampling, recontacting and replacement.GEC-T sampling, recontacting and replacement procedures

are complex and operate at multiple levels. These procedures should be covered in detail in enumerator

and interviewer training. BRAC also recommends that the external evaluator produce an

enumerator/interviewer guidance manual to provide clear instructions on sampling, recontacting and

replacement for intervention and control group girls, with linked instructions on how to deal with

complexities and challenges that may arise.

Ethics and safety. The external evaluator is expected to conduct ethics and safety training with all

enumerators and qualitative interviewers collecting data for the GEC-T. Ethics and safety training should

include the following topics: how to obtain informed consent, including respecting the rights of

participants to refuse to participate in any data collection procedures; how to deal with sensitive issues

and questions, and cultural sensitivities (e.g. male enumerators asking men about their female family

members); how to maintain confidentiality; how to conduct ethical research with children; how to

mitigate ethical challenges; and child safeguarding procedures, including when and how to report the

observation of a breach in child safety (e.g. when observing corporal punishment in schools, or any type

of violence perpetrated against a child).

Additional training will be required for other types of research tools not listed above (e.g. household

surveys, classroom observation tools, attendance data collection tools, etc.). The length and content of

these training components is at the discretion of the external evaluator, but should include tool review,

practice, challenges in data collection, and ways to mitigate challenges.

63

Although training will be an essential part of ensuring that enumerators and other research staff are fully

prepared to collect high quality data, it will also provide an opportunity for the external evaluator to

recognize and correct any potential challenges or problems with tools, samplingdesign and data collection

approaches not identified in the piloting (see below).

9.2 Piloting

Some tools used in GEC-1 will be reproduced in GEC-T with some minor or major modifications (e.g.

household survey, attendance checklists). Consequently, the external evaluator will be required to pilot

quantitative and qualitative evaluation tools at baseline to ensure that questioning, translations and length

of tools are functional and adequate for the research objectives. EGRA/EGMA and SeGRA/SeGMA and

any other school-based tools (e.g. attendance checklists, classroom observations, qualitative interviews

with teachers) should be piloted in BRAC intervention schools not sampled for the evaluation, with

corresponding household-based tools (e.g. household survey, qualitative interviews and focus groups with

girls and parents) being piloted at the household level. Piloting could take place in one or more of

BRAC‟s intervention schools/communities in Kabul province for ease of access; however, the language

of tools should be taken into account (i.e. if no BRAC intervention schools in Kabul have Pashto as the

primary language of instruction, piloting may also need to take place elsewhere such as Nangarhar).

The external evaluator will be responsible for identifying any errors or problems with the data collection

after debriefing with enumerators and research staff, and will make necessary adjustments in consultation

with BRAC. If the external evaluator recommends major modifications to tools, BRAC will refer these

recommendations to the FM and other GEC-T stakeholders if necessary, particularly in relation to tools

that have been mainstreamed across the GEC-T portfolio of projects (e.g. the household survey).

Piloting of tools will only need to take place before baseline data collection and will not take place before

the implementation of other evaluation waves, unless new tools are added at midline and endline waves.

In this case, all new tools should be piloted.

9.3 Data cleaning and editing

Data checking, cleaning and de-identification will primarily be the responsibility of the independent

evaluator. However, BRAC will review all quantitative and qualitative data sets, including checking for

duplicates and internal consistency, ensuring that adequate cleaning has been conducted, and checking

that data has been sufficiently de-identified. In quantitative data sets, any references to names should be

removed, and household contact information, including addresses and telephone numbers, will not be

shared with the FM or any other project stakeholders. In qualitative data sets, any identifying details of

participants should also be removed. It may be necessary to remove any sensitive information that may

cause harm to participants if shared (see further below). Decisions about if this is necessary should be

64

made by the independent evaluator (the staff member in charge of overseeing research ethics and child

safeguarding) in consultation with BRAC.

10. Risks and risk management

Table 16 outlines potential risks that may impact on BRAC‟s ability to ensure a robust and rigorous

evaluation of GEC-T, along with recommended risk mitigation strategies. Some of these risks are linked

to section 5.6 on ethics protocols.

65

Table 16: Risks and mitigations

Potential risks Probability of

risk occurring

over the course

of the project

Potential

impact on

project’s

success

Proposed actions to mitigate risks that have both significant

probability and impact/importance

Necessary permissions are

not secured

Low High BRAC has obtained permissions from the MoE to implement

GEC-T programming, including conducting project monitoring

and evaluation in government hub schools receiving BRAC

interventions. Necessary permissions must also be obtained and

maintained to conduct evaluation data collection in government

control schools. Although the probability of this risk occurring

is small due to BRAC‟s strong relationship with the MoE, if it

should occur this could jeopardise the inclusion of control

groups in the project evaluation, and thus the proposed quasi-

experimental approach. This risk will be mitigated by BRAC

securing permissions well in advance of the baseline data

collection, and ensuring timely renewals of all necessary

permissions over the course of the project.

Insecurity leads to closure

of schools in BRAC target

communities

Low Medium There was no closure of BRAC schools in GEC-1; however,

with an increasingly unstable security situation in Afghanistan,

this risk is possible. An outcome of this risk to the project

evaluation could be the loss of cohort girls from the longitudinal

study. This may lead to a sampling bias (i.e. if only girls living

in secure communities are recontacted). When producing the

sampling framework and community/school selection for the

baseline study, the EE will identify possible replacement

communities (e.g. through intra-cluster random selection). Any

resulting sampling bias will be discussed and contextualized in

evaluation reports.

Insecurity limits ability of

EE to sample target

communities

High Medium An increasingly unpredictable security context may limit

enumerator access to insecure communities, even when BRAC

schools remain open. As noted above, this may lead to the loss

of cohort girls from the longitudinal study. This risk will be

mitigated by: BRAC coordinating enumerator access to

communities; the EE identification of possible replacement

communities; and any sampling biases discussed in evaluation

reports.

The safety of evaluator

staff is compromised

Medium High Insecurity in Afghanistan poses a large risk to the safety of

evaluation staff, including evaluators, fieldwork supervisors and

field monitors. This risk can lead to the injury, kidnapping or in

extreme circumstances death of field staff. BRAC will mitigate

this risk by: facilitating enumerators‟ access to target

communities; ensuring that the EE has key security protocols for

66

data collection (including any necessary security training); and

ensuring that enumerators are recruited from the province or

district where data collection takes place.

Survey fatigue after

multiple evaluation waves

Medium Medium In GEC-1, the external evaluator reported some survey fatigue

among community members who had been sampled multiple

times over evaluation waves, potentially accounting for some

participation refusals in the endline evaluation. Such survey

fatigue may, over time, lead to the loss of cohort girls and their

households from the longitudinal study, and may impact

negatively on community relations with BRAC staff. This risk

will be mitigated by: attempting to minimize the length of

survey/interviews tools (to be tested during piloting); providing

refreshments to qualitative interview and focus group

participants; and ensuring that sampled girls and household

members fully understand that they will be recontacted in future

evaluation waves before consenting to participate in data

collection.

Data fraud High High Data fraud is an important and frequent challenge in

Afghanistan, and can occur at different levels of the evaluation

process, including during data collection, and data entry and

processing (including translation). At the field level, data fraud

may occur partly due to insecurity leading to enumerators being

frightened to access insecure areas and fabricating data, and

insecurity also limiting the ability of evaluators to monitor data

collection in the field. At the data entry and processing level,

data entry staff and translators may also fabricate data (e.g. by

copy and pasting transcripts or data entry cases) to shorten data

processing timelines, or for other reasons. Fraudulent data

would render evaluation findings invalid and could lead to

subsequent changes in project design being made based on false

information. This risk will be mitigated by ensuring that the EE

has a strong data monitoring and data governance system in

place (see section 8.3). The EE should also, to the extent

possible, recruit field staff from the provinces and districts

where data collection will take place to enhance enumerator

access.

Low capacity of research

staff

Medium High Low capacity of research staff is a common challenge in

Afghanistan and can lead to poor quality of data, subsequently

harming the quality of evaluation results. This risk will be

mitigated by BRAC carefully selecting the EE based on past

experience and existing skills, and ensuring that the EE

implements intensive enumerator training, conducts

comprehensive monitoring of data collection, and feeds

monitoring results into fieldwork practices (e.g. conducting

additional training or re-fielding data collection if necessary).

Incorrect sampling Medium Medium Sampling procedures such as random sampling are sometimes

67

procedures difficult to ensure, particularly in a complex and conflict-

affected environment such as Afghanistan where ethnic

divisions and intra-community power relations may interfere

with robust implementation of sampling protocols. A number of

challenges could arise, including: enumerators bypassing

complex sampling procedures in order to facilitate easier access

or reduce timelines; community members interfering in

sampling procedures (e.g. community leaders attempting to

influence participant recruitment); and BRAC staff interfering in

sampling procedures (e.g. teachers attempting to encourage

recruitment ofmore advanced girls in to learning tests). Incorrect

sampling procedures would likely lead to a range of possible

sampling biases. This risk will be mitigated by ensuring that:

enumerator monitoring includes verification of sampling

approaches; enumerators are sufficiently trained on sampling

protocols (including on possible challenges and ways to mitigate

them); and enumerators report any pressures to deviate from

sampling protocols to fieldwork supervisors to enable evaluator

contextualization of evaluation results (and include any relevant

discussion in evaluation reports).

11. Learning

As noted in section 2 of the MEL framework, in GEC-1 BRAC lacked a structured learning system to

ensure that M&E findings were fed back meaningfully into project design, partly due to the GEC-1 focus

on monitoring and evaluation, with less emphasis on learning feedback loops and broader GEC

learning.This section describes the structures BRAC will put in place to ensure that learning is

mainstreamed across the project in GEC-T.

11.1 Learning strategy

BRAC‟s learning strategy is based on two levels of learning: internal and external.

Internal learning

Internal learning will be primarily based on the results of monitoring and evaluation results. There are a

number of activities that were shown to be critical to the success of GEC-1. Although BRAC is interested

in learning about how the efficacy of these activities continues through GEC-T, the focus of learning will

be on how new activities impact on outcomes, intermediate outcomes and outputs, and any implications

for the theory of change. Learning issues include:

The role of TVET in contributing to girls‟ vocational and life skills

The role of mentoring and co-curricular activities in increasing girls‟ life skills

68

The efficacy of mainstreaming child safeguarding standards, auditing and policies on corporal

violence and the creation of safe learning environments

How engaging men and boys can lead to change in gender norms and support for girls‟ and

women‟s education, employment and broader participation in social and civic activities

(including school management)

The efficacy of gender-sensitive curriculum and teaching and how this may lead to change in

gender norms and teaching practices

How inclusive education activities, particularly for highly marginalized girls (e.g. disabled girls,

orphaned girls, and girls who experience violence), can improve learning and transition

Sustainable pathways in girls‟ education, particularly the efficacy of piloting sustainable CBE

models.

BRAC is also interested in learning about how effective new methodologies are in tracking project

outputs and outcomes. Such methodologies include real-time monitoring, auto-ethnography and life

history mapping, use of scales to measure life skills (Rosenberg Self-Esteem Scale and General Self-

Efficacy Scale), Active Pedagogy Inventory (API) tools, and Teacher Value-Added (TVA) modeling.

Learning issues above will be integrated into GEC-T through a range of internal mechanisms. BRAC‟s

monitoring department will engage in informal meetings with project management staff where key

learning opportunities arise. However, more structured learning mechanisms will include BRAC quarterly

meetings after the submission of quarterly reports, to review monitoring results and discuss how findings

should be fed back into project design and implementation. At the evaluation level, BRAC will request

the external evaluator to present the findings and recommendations from each evaluation wave through a

formal presentation to the BRAC GEC-T team, Research and Evaluation Unit, Monitoring department,

and other internal staff where relevant. Following these presentations, BRAC will hold an evaluation

steering committee meeting to discuss any implications for modifications in programming, and design

clear steps on how these modifications will be implemented. These evaluation meetings should also re-

examine the theory of change to interrogate whether it needs to be modified based on evidence from the

evaluation.

Internal learning from M&E will be complemented by small issue-based research projects contracted or

implemented by BRAC‟s Research and Evaluation Unit. Research will address key question related to

inequalities and extreme marginalization.

A timeline for BRAC‟s internal learning activities and deliverables is included in Table 17.

69

Table 17: Timelines for BRAC internal learning activities and deliverables

Activities and deliverables Expected dates

Quarterly monitoring meetings Quarterly

External evaluator presentation In March/April following each evaluation wave

BRAC evaluation review meetings Following each external evaluator presentation

Evaluation report March 2018, 2020, 2022 and 2024

Issue based research report 1

Issue based research report 2

External learning

External learning activities will take place primarily at the GEC-T level, through a range of activities to

promote cross-project learning at the national and global levels. At the country level, BRAC will

collaborate on learning activities with STAGES, the other GEC-T project in Afghanistan. This could take

place through quarterly learning meetings with STAGES consortium partners and the FM in-country

team. Topics of discussion could include challenges, enablers, and project experiences to share best

practices, particularly in relation to new GEC-T focus areas, including transition from primary to

secondary, and CBE sustainability models. BRAC will also participate in learning activities sponsored by

the in-country FM team, including workshops and evaluation events.

At the global level, BRAC will engage in learning activities to promote learning across the GEC-T

portfolio. BRAC will contribute to………….BRAC to list which learning clusters they would like to

contribute to and learn from, and why.What specifically can BRAC contribute to the learning themes?

BRAC will also actively participate in quarterly learning cluster webinars.

11.2 Stakeholder engagement, dissemination and influencing

BRAC‟s dissemination strategy for GEC-T will be based on three levels: accountability, learning and

advocacy.

Monitoring and evaluation outputs, including quarterly monitoring reports and evaluation reports, will be

disseminated to the donor, FM, external evaluator (EM) and other relevant stakeholders, primarily for

accountability, to demonstrate the impact of GEC-T funding.

To promote learning, monitoring quarterly reports and evaluation reports will not be disseminated in their

original forms, partly due to the need to disseminate concise and accessible M&E outputs. Instead, a

series of smaller, more concise outputs will be produced. Quarterly monitoring reports and evaluation

70

reports will be the basis for the production of case studies and summary documents (e.g. briefing notes)

on what works in girls‟ education programming in Afghanistan. The executive summaries of evaluation

reports will also be published as stand alone documents. Furthermore, BRAC‟s issue-based research

projects will be disseminated in their entirety. These outputs will be disseminated to key stakeholders,

including government ministries and intra-ministry units (e.g. the MoE and other key bodies engaged in

the project such as the Ministry of Women‟s Affairs, and the TVET Directorate), other NGOs working on

girls‟ education programming and CBE education, UN stakeholders including UNICEF and UNESCO,

and other donors participating in dialogues and funding of CBE education and girl‟ education.

As noted in section 5.3, BRAC will ensure that dissemination of learning outputs is inclusive of

beneficiaries and communities, and that marginalized groups (e.g. women, disabled persons, children)

will be included in dissemination activities. Due to high rates of illiteracy, dissemination of printed

outputs will be complemented by the verbal presentation of project results and learning. This will happen

at the community and district levels to ensure inclusion of marginalized groups. BRAC will organize

gender-segregated presentations in more conservative areas to ensure women are not excluded and will

attempt to ensure that presentations for women are delivered by women. BRAC will also consider

inclusive locations for presentations to encourage the participation of disabled persons. Dissemination of

results to children will be inclusive and child-friendly, and BRAC will link dissemination practices with

project activities. This may include, for instance, supporting teachers to disseminate research results with

their students through the use of classroom techniques included in BRAC teacher training, or supporting

mentors and mentees in government hub schools to incorporate the dissemination of key research findings

into extra-curricular activities such as public speaking or debating (see section 5.3.2 of the MEL

framework).

At the advocacy level, BRAC will use evidence and learning to support dialogue with the MoE and other

government organisations and programs (e.g. the Citizen‟s Charter), including through advocating for the

sustainability of CBE education and the sign off of the CBE policy, and other relevant strategic outputs.

12. Evaluation workplan

12.1 Timetable

Evaluation timelines are listed in Table 18for baseline, and with the same timelines projected for both

midlines and endline (albeit in subsequent years) in Table 19. The baseline evaluation will take place in

2017/2018, first midline in 2019/2020, second midline in 2021/2022, and endline in 2023/2024 (with

project closure expected in 2024). School term dates will not impact on the evaluation timelines;

however, school examinations generally begin at the end of November or early December and all data

collection should be completed prior to these dates. After submission of reports to the FM, learning

activities will take place at the end of each evaluation cycle (see section 11).

71

Table 18: Timelines for baseline evaluation

July 17 Aug 17 Sept 17 Oct 17 Nov 17 Dec 17 Jan 18 Feb 18 Mar 18

EE procurement

Research & tool design

Piloting

Baseline data collection and

benchmarking

Data processing (entry and

translation)

Data cleaning and analysis

Report writing

Submission of draft report to FM

Table 19: Timelines for midline and endline evaluations

Aug Sept Oct Nov Dec Jan Feb Mar

Research design & tool revisions

Baseline data collection

Data processing (entry and

translation)

Data cleaning and analysis

Report writing

Submission of draft report to FM

12.2 Responsibilities

This section outlines the MEL responsibilities of key actors, both within and externally to BRAC,

summarized in Table 20.

Table 20: MEL responsibilities

MEL responsibility Person/unit/organisation responsible

Procurement of external evaluator BRAC EE selection committee

Internal BRAC monitoring BRAC Monitoring Department and 10 Monitoring officers

External evaluation External evaluator

Evaluation ethics and safeguarding Key staff member selected in the external evaluator team

BRAC‟s Child Safeguarding Committee

Overall oversight of external evaluations,

including quality assurance

BRAC Research and Evaluation Unit

Evaluation Steering Committee???

Oversight and/or conduct of issue-based

research

BRAC Research and Evaluation Unit

Learning fed into project design and

implementation

BRAC Monitoring Department and BRAC Project Manager

Contribution to learning clusters BRAC GEC-T Project Manager, Research and Evaluation

Unit, and Monitoring department

72

Annex A: Logframe

The project logframe is a separate Excel document that should be read in conjunction with the MEL

framework.

Annex B: Draft evaluation tools

All evaluation tools will be designed by the external evaluator in consultation with BRAC prior to

baseline data collection, and thus are not included in the draft MEL framework.

Annex C: Draft sampling framework

A draft sampling framework with details of all community and school clusters is attached to the MEL

framework in Excel format.

73

Annex D: ToR for evaluators

BRAC to insert

Monitoring, Evaluation and Learning (MEL) FrameworkThis document presents BRAC‟s Monitoring,...

Documents

Transcript of Monitoring, Evaluation and Learning (MEL) FrameworkThis document presents BRAC‟s Monitoring,...