UNWOMEN/IPEN Transformative Mixed Methods Evaluation: Day 3 Mixed Methods
description
Transcript of UNWOMEN/IPEN Transformative Mixed Methods Evaluation: Day 3 Mixed Methods
UNWOMEN/IPEN TRANSFORMATIVE MIXED METHODS EVALUATION:DAY 3 MIXED METHODS
Prof. Donna M. MertensGallaudet UniversityAlmaty, KazakhstanJuly 2011
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Three days togetherDay 1: What is the role of the evaluator?
Overview of evaluation (Terms of Reference)Transformative paradigmDiversity & social justice; privilegeCultural competence
Day 2: Quantitative & qualitative approaches
Day 3: Mixed methods approaches
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Methods Options Quantitative approaches such as experimental,
quasi-experimental, causal-comparative, correlational, survey, and single-case designs
Qualitative approaches such as group processes (e.g., focus groups or some indigenous methods), case studies, ethnographic research, phenomenological research, and PAR
Gender analysis is a mixed-methods approach that provides a framework for transformative research and evaluation that has potential for transfer to other groups that experience discrimination.
Mixed methods are most likely to be the approach of choice because of the need to integrate community perspectives into the inquiry process, thus necessitating collection of qualitative data during the research or evaluation process. (Mertens, 2009, TRE, p. 165).
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Mixed Methods Decisions Compatibility: Match the purpose, focus,
questions, and design. Timing: Determine the temporal
relationship between the quantitative and qualitative data collection.
Weight: Establish priority or emphasis of the qualitative and quantitative approaches in the study.
Mixing: Determine when quantitative and qualitative strategies/data will be mixed in the process of evaluation.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Concurrent DesignQuantitative and Qualitative occur more or less simultaneously
Qualitative Quantitative
Sequential Design: Quantitative Followed by Qualitative
OR
Sequential Design: Qualitative Followed by Quantitative
Quantitative
Qualitative
Mertens, 2009, TRE, p. 167
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Community Participation
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Example: Making VisibleBotswana youth:addressing
powerinequities in thefight against HIV/AIDS using atransformative
lens
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Transformative Mixed Methods Design
Stage 4Concurre
nt
Stage 3Sequenti
al
Stage 2Concurre
nt
Stage 1Qual
Assemble team; read
documents; engage in dialogues
Preliminary
studies: youth, older
men
Process eval
Pilot intervention: Observations,
Interviews, Surveys
Demographic
information; Surveys; Incidence
data
Pretest: Knowledge,
Attitude, Behavior;
Posttests: QuantQual;Behavior& PolicyChange;TransferTo othercontexts
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Transformative Design Components:Post-Colonial Critical Ethnography
A statement of your evaluation problem or questions (focus). A description of your data-collection methods, including
interviewing, journaling, and coding processes, and how these will be accomplished with the evaluator as a co-performer in the field or participant observer.
An explanation of your ethical methods and how the welfare of the participants will be put first by protecting their rights, interests, privacy, sensibility, and offering reports at key stages to them, including the final report.
A description of the participants in terms of population, geographic location, norms and rules, significant historical and cultural context, and expectations for key informants.
A time frame for entering the field, collecting the data, departing from the field, coding and analysis and completion of the written report, and/or public performance.
Use of a critical theoretical framework in the design, implementation, and dissemination of the study.
Madison (2005)
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
PAR and Transformative Design
1. The group decides on the focus and questions for the evaluation.
2. Evaluators and participants observe, engage in action, observe and record.
3. Evaluators and participants immerse themselves in action and elaborate and deepen their understandings.
4. Group members reassemble and share their knowledge, using this iteration as an opportunity to revise their plans for the next cycle ofevaluation.
5. This cycle might be repeated between 6 and 10 times depending on the complexity of the evaluation context.
(Heron & Reason, 2006).
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Transformative MM and PARGreenwood and Levine (2007) summarize:
“Surveys, statistical analyses, interviews, focus groups, ethnographies, and life histories are all acceptable, if the reason for deploying them has been agreed upon by the AR collaborators and if they are used in a way that does not oppress the participants” (p. 6).
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Questions for Thought: Design
What is your reaction to the idea that one group will receive an intervention and another group will not?
Under what circumstances would you accept that format as an ethical course of action?
What other alternatives are possible?(Mertens, 2009, TRE, p. 197)
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Gender Analysis and MM
Outcomes, Outputs & ImpactsInputs —> Activities —> Outputs —> Outcomes —
> Impacts Outputs are the products, services, and
capacities that result from the completion of activities Outcomes are the intended or achieved short-term
and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners.
Impacts are the long term effects or change to which the programme, through collective effort with partners, will contribute.
GE/HR evaluation: how did the programme contribute to GE and HR?
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
INDICATORS Indicators are the means by which you
determine progress towards a result or whether an expected result has been achieved.
Developed for ALL levels of results – outputs,
outcomes & impact
Indicators measure (quantitatively or qualitatively) the status of an expected result
15Module 2-1Source: UNIFEM RBM training
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
INDICATORS Quantitative: focus on numbers & counting
(percentage of women and men in parliament, male and female wage rates, school enrolment for girls and boys)
Qualitative: capture opinions, attitudes and feelings (nature of dissatisfaction, extent of increased awareness)
16Module 2-1
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
EXAMPLESQuantitative - Can be directly counted & expressed as a
number % of … # of … Frequency of … Ratio of … Amount of … Timeliness of …
Qualitative - Involves perception (can be expressed quantitatively or as narrative)
Level of Satisfaction with … Knowledge of … Ability to … Appropriateness of... Importance of … Use and usefulness of …
17Module 2-1Source: UNIFEM RBM Training.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
GENDER SENSITIVE INDICATORS Measures gender-related changes in
society over time
Includes sex-disaggregated indicators that
provide separate measures for women and men
Indicators that are gender-specific to either women or men
18Module 2-1Source: A. Moser. 2007Gender and Indicators
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
HUMAN RIGHTS INDICATORSAssess both state progress in guaranteeing
human rights and individual programme adherence and promotion of rights.
HR indicators should measure: Improvements in the capacities of rights
holders and duty bearers to realize rights
Improvements in the enjoyments of rights
19Module 2-1
GE & HR RESPONSIVE EVALUATION
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Follow-Up and Use:
Dissemination strategies should make evaluation findings accessible and barrier-free to women, including both RHs and DBs Targeting women’s organizations/networks and
knowledge networks User-friendly language Stakeholder workshops that include women and other
groups subject to discrimination (RHs &DBs) Management Response should be issued to ensure
follow-up on key gender and human rights issues
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
EXAMPLE INDICATORSOutcome indicator: Evidence of changes in
the capability of ministries of xxx to formulate and implement policies responsive to indigenous women
Output indicator: Capacity assessments (focusing on individual competencies within ministries of xxx) indicating increase in relevant knowledge, skills and attitudes within key public institutions
Examples: GE/HR Indicators proportion of target group by sex aware
of the benefits of birth registration, employment to population ratio by age
and sex, gross primary graduation ratio by sex, percentage of women in parliament.
“REALITY CHECK”
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Expense and making use of available data; are there too many indicators?
Unrelenting pull to quantitative measures - need a balance of qualitative and quantitative data
Proxy indicators, where measurement takes an more indirect path
Indicators should be disaggregated, as much as possible, by sex, race, ethnicity, age, geographic area
Indicators do not exist in a vacuum and must be tied to a result.
EXERCISE: DEVELOP INDICATORS
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Develop two indicators (one qualitative and one quantitative) for the course example.
Sample Relevance Evaluation ?’s How well do the programme objectives
target the identified rights and needs of male and female beneficiaries?
What rights does the programme advance under CEDAW, the Millennium Development Goals and other international development commitments?
Sample Evaluation Effectiveness ?’s To what extent have the objectives been
achieved, and do the intended and unintended benefits meet fairly the needs of disadvantaged women?
To what extent have the capacities of duty-bearers and rights-holders been strengthened?
Sample Evaluation Efficiency ?’s Could the activities and outputs have been
delivered with fewer resources to the target populations without reducing their quality and quantity?
How has the programme maximized partnerships in the delivery of the programme?
Have UN Woman’s organisational structure, managerial support and coordination mechanisms effectively supported the delivery of the programme?
Sample Evaluation Impact ?’s To what extent have efforts been
successful in stopping harmful and discriminatory practices against women?
What is the evidence that the programme enabled the rights-holders to claim their rights more successfully and the duty-holders to perform their duties more efficiently?
Sample Evaluation Sustainability ?’s Is the programme supported by national
and local women’s organisations? Do these organisations demonstrate
leadership commitment and technical capacity to continue to work with the programme or advocate for change?
Additional GE/HR ?’s Equality and non-discrimination: Did the programme benefits
affect equally men and women? Empowerment: Did the budget designate sufficient resources
and level of effort to address the inclusion of disadvantaged or marginalised groups?
Accountability: Were monitoring data (disaggregated according to relevant criteria such as gender, age, ethnicity, location, and income) collected and used to adjust implementation?
Social transformation: Does the intervention’s theory of change include attention to GE and HR?
Participation and inclusion: Did the implementation make systematic and appropriate efforts to include women and men, and/or reach out to disadvantaged groups? UN Women 2011
GE/HR Evaluation Methods Avoid bias: selection of data collection
methods & in sampling methods Potential biases: Gender & Power
(sources able to contribute more easily because of privacy & confidentiality issues ), Class or distance (favoring the more accessible)
Plans for how to include marginalized groups
Mixed methods: Quan & Qual
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Rigor applied to methods choices
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
In the positivist tradition, there are two important tests of knowledge claims:
1. Is the knowledge claim true in this situation? Are the changes observed in the dependent variable due to the effect of the independent variable? (Internal Validity)
2. Is the knowledge true in other situations? Generalizability? (External Validity)
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Comparative Studies: Rigor Causal relationship assumed? Competing
explanations? Comparable groups in causal comparative? Third variable cause both predictor and
criterion variables? Sub groups analysis? Correlational: ordering of variables? Predictive studies: Other screening variables?
Level of .8 or better? Reliability and range of variables
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Criteria that establish rigor in qualitative methods
Credibility: just like validity in qualitative research – asks if there is a correspondence between the way the respondents actually perceive social constructs and the way the evaluators portray their viewpoints.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
(a) persistent observation (b) peer debriefing(c) progressive subjectivity (d) member checks(e) triangulation(f) transferability (g) dependability(h) confirmability(i) authenticity and fairness
Criteria that establish rigor in qualitative methods
TRANSFORMATIVE EVALUATION: Rigor
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Emphasizes Human Rights and Social justice Analyses asymmetric power relations Advocates culturally competent relations
between the evaluator and community members
Employs culturally appropriate mixed methods tied to social action
Applies feminist theory, critical race theory, postcolonial and indigenous theories
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Authenticity (Transformative criteria)
Fairness: The evaluator presents all value differences, views, and conflicts.
Ontological Authenticity: An individual’s or groups’ conscious experience of the world became more informed and sophisticated.
Catalytic Authenticity: Action was stimulated by the inquiry process.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Power and Privilege How do we understand the dynamics of power when
participatory methods are employed by the powerful? Whose voices are raised and whose are heard? How are these voices mediated as issues of
representation become more complex with the use of participatory methods in larger-scale planning and consultation exercises? (Mertens, 2009, P. 85)
What if I am a member of the community? How does that prepare me to work in that community? What if I am not a member of a community? To what extent is it necessary to share salient characteristics of a community?
How does cultural competency come into the discussion of interactions in evaluation contexts? (Mertens & Wilson, in press)
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Criteria for Rigor: Utilization Dissemination Use Management Response Engagement with Stakeholders
Dissemination Purposes Transparency and accountability Informing and improving an organisation’s
work Sharing good practices and ‘how to’ to
advance women’s human rights Sharing lessons with partners on the ground
and building their capacity, and Generating knowledge on how gender
equality, women’s empowerment and human rights can be advanced
Dissemination Language and presentation of the report: use graphs or
pictures, and be written in a manner that is gender and culturally sensitive.
Translation: anticipate the need for translation when planning the evaluation so time and budget can be included)
Making the report public: Within the UN, there is requirement to make all evaluations public (UNEG Norms and Standards) and in UN Women, this is accomplished by having the Evaluation Unit post the evaluation report in the Evaluation Resource Center (which is publicly accessible).
Reaching target audiences in a user-friendly way: lower rates of literacy? The manager needs to be creative in using other dissemination channels, suited to different audiences.
Dissemination Strategies Brochures outlining
key evaluation lessons and recommendations
Annual reports Articles in technical or
organisational newsletters
News releases Press conferences Media appearances
Public meetings, public debates or town halls
Seminars, workshops, and informal group discussions
Electronic media (e-mail, websites, blogs etc.)
Meeting with community leaders, one on one.
Challenges to Use Lack of consensus on recommendations
& required action Those who don’t like the results attack
the process Dissemination is minimal: no funds,
interest or time No follow-up process
Formal Management Response presentation of the response, action or
non action to evaluation recommendations and lessons learned and follow-up or tracking mechanisms.
engage with stakeholders to reflect on the evaluation process, findings, recommendations and lessons learned.
Involve reference group in developing the management response, with the manager.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Agenda for Action How does the information you learned
here apply to the work that you are/will be doing in evaluation?
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Resources Mertens, D. M. & Wilson, A. (in
press). Program Evaluation. NY: Guilford.
Mertens, D. M. (2010). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 3rd ed. Thousand Oaks, CA: Sage.
Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford
Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Contact information Donna M. Mertens, Gallaudet University