1

Module 3

Designs

2

Family Health Project: Exercise Review Discuss the Family Health Case and these questions.

Consider how gender issues influence the formulation and implementation of the development of a family health project?

Consider some likely questions that you might ask if you wanted to monitor and/or evaluate this family health project?

What additional questions might you ask to make sure your questions are gender-sensitive? What other actions would you take?

3

Learning Objectives

At the end of this session participants will understand:the importance of design in answering questions about impact

design options

gender issues as they relate to design

4

Type of Questions - Determine Strategy

1. For descriptive and normative question, strategy may be simple and straight forward.

2. For impact questions, how you will control for other factors so you can ascertain the effects of your program.

3. Impact questions require a different types of design.

Evaluation Research Design

5

Designs for M&E

Design = Strategy for answering M&E questions:

Did the program cause the changes you observe?

To answer this key causal question, you must be able to eliminate other explanations.

This is done through design which controls for other factors so you can measure the effect of your program.

6

Design Validity

Judge the quality of the design by its validity

Two types of validity:1. Internal validity

• Is what you observe due to the program and not other factors?

2. External validity• Is what you observe true in the larger

population?

7

Threats to Internal Validity History

Due to a particular event that took place while data was being conducted.

Maturation

Skills increase because people get older.

TestingRisk is that they “learned” how to do the test.

Instrumentation Changes in data collection instrument, procedures

or measures.

8

Threats to Internal Validity (Continued)

Regression to the Mean The next set of scores are likely to change – to fall closer to the mean– regardless of treatment.

SelectionThe group under study may be different in ways that effect the results.

AttritionDifferent rates of dropping out of the program may affect results.

9

Threats to External Validity Biased or non-representative sample

Sample is selected that is different from the larger population.

Unique programResults are not likely to be the same elsewhere because of a unique program quality.

Unique settingResults are not likely to be the same in a different location.

10

Types of Design

Experimental

Quasi-Experimental

Non-Experimental

11

Types of Design Experimental Design

Ideal Design for Measuring Impact Key elements:

• Random Assignment to groups- Program group- Control group

• Before and After Measurement

Strongest design because all other explanations (threats to internal validity) have been ruled out

May be a problem with external validity

12

Types of Design Experimental Design

R O1 X O2

R O1 O2

R indicates Random assignmentO is the Observation or measureX is the Program or the

intervention

13

Experimental design is the strongest for answering causal questions

But:• it can be flawed for examining gender

issues if the group that is randomly assigned does not include both men and women

• experimental studies of only men are not valid for conclusions about women

Experimental Design Gender Issues

in Answering Causality

14

Experimental Design is Rarely Used It is often not possible to use experimental

design.• ethics, practicality, costs

Other evaluation designs• Quasi-experimental• Non-experimental

Each successive design is weaker and more vulnerable to wrong interpretations of project impacts.

Experimental Design

15

Quasi-Experimental Design O1 X O2 Program Group

O1 O2 Control Group

Groups• No random assignment• Matched pairs • Non-equivalent comparison groups.

Types of Design

16

Collect data from a large sample or the whole population and then statistically create comparison groups

• Are widowed women more likely to control resources than married women?

• Are men more or less likely to repay loans than women?

Useful to measure program impact when it is not possible to randomly assign people to programs

Eliminates threats to external validity

Quasi-Experimental Design Using Statistical Controls

17

Before and After Design (lacks control group): O X O

Static Group Comparison (lacks pre-measure):X O

O

One Shot (lacks control group and pre-measure):X O

Types of Design Non-Experimental Design

20

Case Discussion Micro-Credit Studies

Economic Study: • Quasi-experimental design using statistical

controls to create various comparison groups Cross-sectional survey

• No pre-test (baseline) data

Social Study:• Data collected from participants only after

project was implemented • One-shot design: no comparison group, no

pre-measures.