Qualitative Methods for Health Program Evaluation

CHSC 433CHSC 433

Module 5/Chapter 12Module 5/Chapter 12L. Michele Issel, PhD

UIC School of Public Health

“Different kinds of

problems require

different types of

data.” Patton, 1997

Objectives

List the major qualitative designs List at least one pro and con for

each of the major qualitative designs

Provide an outline of how qualitative data analyses are done

Beyond the Paradigm Debate

History of science favored quantitative (empiricism), deductive hypothesis testing, logical postivism

Current science favors understanding based on rigorous methods

Use Qualitative when

Want to minimize research manipulation by studying natural field setting.

Program aims at individual outcomes (so when program aims at common outcomes across individuals, use quantitative methods).

Key Characteristics

The use of non-numeric data, such as narratives, pictures, music

The use of subjective, experiential, naturalistic inquiry to explain phenomena

Use of inductive, iterative analysis Holistic and contextual concerns Pays attention to individual’s

uniqueness

Functions of Qualitative Methods (Adapted from Green & Lewis ‘86)

1. Develop and delineate program elements

2. Booster power of quantitative designs

3. Broad the observational field

4. Analyze processes and cases to understand why or how the program worked

5. Generate a program or intervention theory

6. Use instead of quantitative methods

Underlying Perspectives

Phenomenology- experiences and meanings

Ethnography- cultureCritical analysis- communication and

powerGrounded Theory- discovery of

theory Content Analysis-manifest meanings

in the written word

Perspective --> Question

Phenomenology

Ethnography

Critical analysis

Grounded theory

Content analysis

What does it mean to the person?

What are the norms and values (culture)?

How has power shaped it ?

What are the relationships (theory)?

What themes are in the text?

Major Types of Qualitative Methods

Participant observation Case studies In-depth Interviews Focus groups Open-ended survey questions

Participant Observation

Acting as a member of a group, collect data

Make narrative notes and memos about processes, events, people observed

Use key informants to verify data analysis

Case Studies

Define what is a case (organization, program, person)

Use variety of types of raw data generated by or about the case: memoranda, observations, surveys, interviews, etc

Case Study

Key Benefits for use in planning and evaluation

Key Challenges for use in planning and evaluation

Allows for understanding of context as influence on program or participant

Allows for understanding of context as influence on program or participant

In-depth Interviews

Use open-ended questions with key individuals (participants, key informants)

Use probes to clarify and explore issues or topics alluded to by the respondent or earlier data analysis

Use tape recorder and transcripts of the interviews

In-depth Interviews

Key Benefits for use in planning and evaluation

Key Challenges for use in planning and evaluation

Provides rich insights into personal thoughts, values, meanings, and attributions

Provides rich insights into personal thoughts, values, meanings, and attributions

Focus Groups

Carefully selected group of individuals who participate in guided discussion about a specific topic

Use a facilitator and a recorder

Focus Groups

Key Benefits for use in planning and evaluation

Key Challenges for use in planning and evaluation

Inexpensive given the amount and type of data, get collective views rather than individual views

Inexpensive given the amount and type of data, get collective views rather than individual views

Observations

Non-participatory and Participatory techniques can be used

Need training on what will be observed and how will record the observation

Data collection methods vary:• Audio-visuals recording• Field notes• Logs

Observations

Key Benefits for use in planning and evaluation

Key Challenges for use in planning and evaluation

Can identify sequence of causes and effects, may identify new behaviors or events

Can identify sequence of causes and effects, may identify new behaviors or events

Use open-ended question placed at end of quantitative survey

Unable to use probes for clarification

Handwriting and spelling can make interpretation difficult

Open-ended survey questions

Sampling for Naturalistic Inquiry

Small purposive samples

• Select for a specific characteristic

Theoretical sampling

• Select based on what “ought” to matter

Sample for category saturation

• Select until no new information is

gained from participants

Data Analysis

Coding and interpreting the data

To count or not to count

Coding Terminology

CategoryCategory- classification of concepts in the data

DimensionDimension- implies continuum

PropertyProperty- attributes or characteristics of a category

Constant comparisonConstant comparison- process to develop categories, involves comparing new with existing categories

CodableCodable- unit of data to be categorized

Analysis Procedures

1. Identify codable units of data2. Understand the meaning3. Discover categories4. Name categories5. Discover properties and

dimensions of the categories6. Generate explanation

Scientific Rigor = Trustworthiness (Lincoln & Guba, 1985)

CredibilityCredibility ~ Internal validity

TransferabilityTransferability ~ External validity

DependabilityDependability ~ Reliability

ConfirmabilityConfirmability ~ Objectivity

Credibility

Have confidence in the truth of the findings by:

Invest sufficient time, triangulate Use outsiders for insights (peer

debriefing) Refine working hypotheses with negative

cases Check findings against raw data Use participant feedback

Transferability

Applicability to other contexts and respondents

Provide thick (detailed, comprehensive) descriptions for others to assess possibility of transferability

Dependability

Find same results if repeated the study

Leave a trail that can be followed so that others can see the findings are supported by the data

Confirmability

Findings are from the respondents not the researcher

Leave an Audit Trail (same as for Dependability)

To count or not to count

Number of participants who mentioned a category

Number of times category mentioned throughout the study

Issues…. Neither help with interpretation of meanings, both can misrepresent the scope of the sentiment

From Data to Description

Categories as typologies are rudiments of a theory

Category dimensions and properties as essential

Linkages between categories form theory

Data Presentation

Use descriptions of context to show transferability

Use tables showing category development to show dependability and confirmation

Use participants’ words to show confirmation

Use diagrams of relationships among categories

Realities of Data Analysis

Messy, confusing, repetitive • Iterative category development

Overwhelming quantities of data Conflicting interpretations of data by

peers and participants Manifest versus implied meanings

cloud data analysis Investigator biases

Cost of Data Collection

Interview time• Travel to interviewee• Reading and listening

Transcription time• 1 hour interview: 3 hours transcribing

Data analysis time

Evaluation Caveats

Integrate with quantitative data Use of participant feedback

(credibility) as key to acceptance of findings

Stories are more powerful than numbers and make the numbers more human

Qualitative MethodsAcross the Pyramid

Direct Health Care

Services____________________

Enabling Services___________________________

Population-Based Services___________________________________

Infrastructure Services

Each qualitative method has potential usefulness for programs at each level of the Pyramid.