July 20, 2020 Key Concepts
Transcript of July 20, 2020 Key Concepts
2
Learning Objectives
You will be able to state the four key analytic concepts that will be used throughout the course.
You will be able to apply the theory of change to yourindividual/team project.
You will be able to identify who is included and excluded from an entry cohort, point-in-time, and exit cohort population.
4
1. Making better decisions requires evidence. Evaluation—particularly rigorous evaluation—is an
important way to produce evidence.
2. Rigorous evaluation follows scientific principles. Shows cause and effect, when relevant
Uses best practices in measurement
3. Rigorous evaluation is worth the investment. If you don’t evaluate, you can’t know if it works.
JUNE REV IEW
5
OVERVIEW
The Four Key Evaluation Concepts We Are Going to Talk About Today:Concept 1: Have a clear theory of change.
Concept 2: Ask a good question.
Concept 3: Think longitudinally.
Concept 4: Know your population.
7
Your Theory of Change Will Guide Your Entire Evaluation A theory of change logically describes how and why
an intervention will cause a change. It is a compass, helping you stay on the path to get where
you want to go.
A logic model visually displays the theory of change. It is a map, showing the road that you plan to follow.
THEORY OF CHANGE
8
THEORY OF CHANGE
Four Statements to Complete before Starting Your Theory of Change:
I observe [some outcome I want to
improve].
I think it’s because [of this reason].
So I plan to [carry out this intervention],
Which I think will result in [an improved
outcome].
Source: L. A. Alpert, J. M. Haight, B. O. Orlebeke, and F. H. Wulczyn, EDGE: Evidence-Driven Growth and Excellence (Chicago: Chapin Hall at the University of Chicago, Center for State Child Welfare Data, no date).
9
THEORY OF CHANGE
Source: L. A. Alpert, J. M. Haight, B. O. Orlebeke, and F. H. Wulczyn, EDGE: Evidence-Driven Growth and Excellence (Chicago: Chapin Hall at the University of Chicago, Center for State Child Welfare Data, no date).
You Need Evidence That Supports Your Theory of Change:
Which I think will result in…
How will you
know?
So I plan to…
How do you
know?
I think it’s because…
How do you
know?
I observe that…
How will you
know?
10
THEORY OF CHANGE
Source: L. A. Alpert, J. M. Haight, B. O. Orlebeke, and F. H. Wulczyn, EDGE: Evidence-Driven Growth and Excellence (Chicago: Chapin Hall at the University of Chicago, Center for State Child Welfare Data, no date).
You Need Evidence, cont’d
Use an RCT to compare changes in use of corporal punishment and rates of
physical abuse substantiation after 1 year.
which I think will result in decreased use of
corporal punishment, and in turn, prevent physical
abuse.
Contact logs from our information system. Copies of parents’ action
plans for each session in case files.
So I plan to implement a
parent education & skill building program...
I did a qualitative analysis of parents’ responses to the family assessment tool and reviewed published studies
about why parents use corporal punishment.
I think it’s because parents do not have the knowledge and skills to apply age-appropriate
discipline.
I collected data from recent physical abuse investigations. I reviewed published studies on
what leads to the physical abuse of young children.
I observe that parents who physically abuse young children use
corporal punishment.
11
THEORY OF CHANGE
Source: L. A. Alpert, J. M. Haight, B. O. Orlebeke, and F. H. Wulczyn, EDGE: Evidence-Driven Growth and Excellence (Chicago: Chapin Hall at the University of Chicago, Center for State Child Welfare Data, no date).
Use the Four Statements to Develop a Theory of Change for Your Project
How will you
know?
Which I think will result in…
How will you
know?
So I plan to…
How do you
know?
I think it’s because…
How do you
know?
I observe that…
14
KNOW YOUR QUEST ION
Lery, B. Haight, J. M., Alpert, L. (2016). Four principles of big data practice for effective child welfare decision-making. Journal of Public Child Welfare, 10(4), 466-474, DOI: 10.1080/15548732.2016.1209149
A Good Question Is Key to Building Evidence for Your Program
Claim,Decision, or
Action
to support aEvidence
which may be understood as
Findingsresults in
Data
which when applied toDesign and
Analysis
which decides the
QuestionOne expresses what they care
about in the form of a
15
Four Components of a Good Research Question:
1. Make sure it’s a question.2. It’s narrow and specific.3. The answer can be seen.4. It’s appropriate for the intervention: Aligned with the theory of change;
Matches its current phase of development.
KNOW YOUR QUEST ION
17
Two Aspects to Thinking Longitudinally:
1. Time itself is a factor that can impact outcomes.2. The impact of time can be analyzed using longitudinal
data.
THINK LONGITUDINALLY
18
Time Itself Is a Factor That Can Impact Outcomes
Time-dependent means the amount of time that has passed influences whether or not something happens.
Examples of time-dependent outcomes are: Placement moves while in foster care;
Likelihood of reentry after reunification.
THINK LONGITUDINALLY
19
THINK LONGITUDINALLY
Longitudinal data collects information more than once about the same group of people over a period of time.
Measuring something year to year, but from different people, is not longitudinal data.
The Impact of Time Can Be Analyzed Using Longitudinal Data
Time Start Year 1 Year 2 Year 3
Annual Measures
Longitudinal Data
20
Longitudinal Data Can Answer Questions about Whether and When Something Happened
Did our program reduce the time to reunification? Did our program reduce the likelihood of abuse? Did our program increase placement stability?
What are some examples of longitudinal questions from your evaluation projects?
THINK LONGITUDINALLY
22
Your Theory of Change Will Guide the Selection of Your Target Population The target population is the group of people who the
intervention is created for. The study population is the group of people from the
target population who will be in your evaluation. A sample is when the study population includes some, but
not all, of the target population.
KNOW YOUR POPULAT ION
23
Three Common Ways to Get Your Study Population in Child Welfare1. Point-in-time: only people in the program at a specific time,
or the active caseload.
2. Entry cohort: all people who entered the program during a period of time.
3. Exit cohort: only people who left the program over a period of time.
These are not “three views of the same population.”
They are three very different populations.
KNOW YOUR POPULAT ION
24
KNOW YOUR POPULAT ION
Who gets counted Who is ignored
Point-in-time Only those in the program that day Everyone who already left the program
Exit cohort Only those who exited during the period
Everyone who is still in the program and everyone who left early
Entry cohort Everyone starting program during the period
Nobody
Only Entry Cohorts Include Everyone You Intend to Treat
25
KNOW YOUR POPULAT ION
Let’s Practice! What Sample Was Used for Each Analysis? 1. 60% of parents who completed the program were reunified with
their children.2. Of those assigned to the treatment group, 15% had an overdose
one year later compared with 25% of the comparison group.
Exit
Entry
3. More children were living with family in April 2020 compared with April 2019, the month before family meetings were implemented.
Point-in-time
4. At one-year follow-up, young people from the treatment group were more likely to have completed school than those from the comparison group.
Entry
5. On the last day of the fiscal year, 70% of young people were housed.
Point-in-time
6. Parents who graduated from the program reported less use of corporal punishment six months later.
Exit
26
OVERVIEW
Review: The Four Key Evaluation Concepts You Will Use Throughout This Course Are:Concept 1: Have a clear theory of change.
Concept 2: Ask a good question.
Concept 3: Think longitudinally.
Concept 4: Know your population.
Questions?
This report is in the public domain. Permission to reproduce is not necessary. Suggested citation: Urban Institute et al. (2021). Slide Deck Session 2: Key Concepts - Child Welfare Evidence-Building Academy. OPRE Report 2021-108, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.