Post on 13-Dec-2015
1National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Welcome!
• We will begin our webinar at 9:00 am PT
• As you log on, do not be surprised if you don’t hear anyone else; participants are placed on mute in order to limit background noise
National CASA Performance Measurement
Lori Delale-O’ConnorSarah Catherine Williams
Child Trends
www.childtrends.org Twitter/childtrends www.facebook.com/childtrends:
April/May 2014
3National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Presenter Information
Lori Delale-O’ConnorScientist
Child Trends Having difficulty with the sound? Please send us a message in the chat box.
4National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Roadmap
Part I: Introductions and Overview of the National CASA project In-depth interviews Survey development, testing, and fielding
Part 2: Survey Results Survey completion demographics Survey data Potential “priority areas” identified
Part 3: Process of Becoming Evidenced-Based What it means to be evidence-based “Arrow” and process National CASA’s work to becoming evidence-based
5National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
What you will walk away with…….
• Knowledge about the current data collection activity of CASA/GAL programs across the country
• Understanding of what it means to be “Evidence Based” – and why it matters for your CASA/GAL program
• Insight into how National CASA is moving forward with performance measurement and working to build an evidence base
6National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Part I: Introductions and Overview
Child Trends’ Mission
To improve outcomes for children by providing research and analysis to the people and institutions whose decisions and actions affect children.
http://www.childtrends.org/
7
8National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Research Areas
8
FATHERHOOD AND PARENTING
EARLY CHILDHOOD
CHILD WELFAREINDICATORS
EDUCATION (K-12)
DATA DEVELOPMENT
YOUTH DEVELOPMENT FERTILITY AND FAMILY STRUCTURE
9National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Develop and track data on children and youth Help to design, implement, and evaluate child
and youth-serving programs Analyze and synthesize data and evidence Provide training and technical assistance
A key focus of our work is identifying evidence-based programs, practices, and policies
Child Trends’ Work
9
10National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
National CASA/Child Trends Performance Measurement
Goal: Develop key measures to help National CASA better understand performance and move toward developing an evidence base.
Phase 1: Scan of current data collection and use (Sept-March) In-depth interviews Survey development & testing Survey fielding
Phase 2: Presentation of findings (April-May)Phase 3: Recommendation of performance measures (May-June)
11National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Part II: Survey Findings
12National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Survey Completion OverviewCategory % Category %Level Urbanicity
local 89 rural 50state 9 suburban/mixed 36state – direct advocacy 2 urban 14
Age (in years) Program Model
10 or fewer 20 friend of the Court 5611 – 20 36 guardian ad litem 2821 or more 44 friend of the Court/Attorney team 10
Size (# of children served) Type of Agency
50 or less 24 independent non-profit 6251 – 300 59 part of a government agency 21301 or more 18 part of a non-profit umbrella agency 16
13National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Topics Addressed in the Survey
• Data collection responsibility and burden • Data collection systems
• Systems used• Meeting needs
• Data use • Priorities for future data collection
14National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Data collection and level of burdenType of staff % Hours per month
CASA supervisors/coordinators 70 20
CASA administrators 56 21
Volunteer 16 4
Assistant or support staff 9 31
Other 5 19
CASA supervisors/coordinators
CASA administrators
volunteers
assistant/support staff
2
2
14
7
15
14
14
12
54
48
49
38
21
28
18
31
8
9
6
12
Don't know Very burdensome Somewhat burdensom Not very burdensome Not burdensome at all
15National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Data collection systems and satisfaction Current Data System % Current Data System %
CASA Manager 39 Other 4
Microsoft Excel 36 Optima 4
COMET 16 CASA Tracker 4
State system 12 Own system 2
Efforts to Outcomes (ETO) 6 Microsoft Access 2
CASA Manager
COMET
Excel
State System
1
2
10
27
20
24
55
59
60
56
35
15
18
20
Not at all Not very well Somewhat well Very well
16National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Data use Activity %
Reporting requirements 90
Demonstrate effectiveness 73
Make program improvements 62
Fundraising 58
Service improvements 43
17National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Data support needs
Area of desired support %
Training on how to analyze data 53
Guidance on collecting data 46
Funding for data collection/analysis software 44
Guidance in identifying outcomes 44
Funding for personnel 40
18National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Data priorities
Domain %
Permanency 60
Well-being 44
Placements 44
Education 39
Mental health 31
19National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Participant Feedback
20National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Participation
What do you think about the priority areas identified in terms of their alignment with CASA?
• I think they are an accurate reflection of CASA’s mission and the reach of a volunteer.
• I think one or more of them are outside of CASA’s mission and the reach of a volunteer.
• I think one or more of them are outside of CASA’s mission, but a volunteer may impact these areas anyway.
21National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Participation
Do you/local programs you work with already collect data on measures in the priority areas?
• Yes, I/they collect data on ALL of these measures.• Yes, I/they collect data on SOME of these measures.• No, I/they do not collect data on ANY of these measures.
22National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Participation
What potential challenges do you see to getting data on measures in the priority areas?
• My program/the programs I work with does not have access to the data needed for these measures.
• My program/the programs I work with would need to administer additional assessments (i.e., mental health assessments/screenings) in order to capture data on these measures.
• I am not clear on what type of data are needed for these measures.• Programs may interpret the meaning of these measures differently,
and therefore data quality would be inconsistent.
23National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Participation
What potential challenges do you see to reporting on measures in the identified priority areas?
• It is a challenge to get the data for these measures, and therefore my program/the programs I work with would most likely be unable to report on these measures.
• I/the programs I work with can get the data on these measures, but I am not confident I/they would be able to report on them.
• There is a danger that programs could interpret the measures differently, and therefore the data would not be as useful.
24National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Part III: Becoming Evidence Based
25National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Evidence-based programs: – Interventions that have been found to have positive effects
or impacts on targeted outcomes Evidence-based practices:– Intervention strategies or program practices that have
been found to improve outcomes for children and youth Evidence-based policies:– Regulations and laws that have been found to have
positive effects on child and youth outcomes
What We Mean by “Evidence-Based”?
25
26National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
High
Low
Quasi-experimental
Experimental
Replicated Experimental
Evidence-informed, non-experimental
Levels of Evidence
26
27National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Collect Data on Performance & Outcome Measures
Conduct Needs
Assessment
Identify Your Population
Becoming Evidence-Based
Targeting
Conduct a Quasi-Experimental Outcomes Evaluation
Conduct an Implementation Evaluation
Select Intervention, Develop Logic Model &
Identify Indicators
Implement Program/Approach & Conduct Ongoing Performance
Management
Conduct a Randomized-Controlled Impact Evaluation
27
28National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Participant Feedback
29National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Participation
Where is your CASA/GAL program/the programs that you work with on the arrow?
• We/They are not on the arrow at all.• We/They are in the preliminary stages of performing a needs
assessment and identifying target population.• We/They are in the more secondary stage of developing a logic model
and identifying indicators.• We/They are in the conducting ongoing performance management
phase.• We/They are in the more advanced stage of conducting an
implementation evaluation, quasi-experimental evaluation, or a randomized control trial (RCT).
30National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Food for thought—post-webinar survey question
Where do you see your CASA/GAL program/the programs you work with being able to move on the arrow?
• Not any further than we/they currently are.• Performing a needs assessment and identifying target
population.• Developing a logic model and identifying indicators.• Conducting ongoing performance management.• Conducting an implementation evaluation, quasi-
experimental evaluation, or a randomized control trial (RCT).
31National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Building the Evidence Base
32National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
• Set up a clear path to success or a roadmap for achieving program goals
• Provide a visual test of whether inputs, and activities align with outputs and outcomes
• Help to get everyone “on the same page”
• Facilitate performance management and evaluation
Benefits of Logic Models
32
33
OutcomesOutputsActivities
Inputs Outputs Outcomes
Program Component
Inputs Outputs Outcomes
Program Mission
Program Component
Logic Model Example
Inputs
Activities Activities
34
Logic Model Example
Outcomes Outcomes Outcomes
Outputs
Inputs
Activities
Inputs
ActivitiesActivities
OutputsOutputs
35National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Outcomes and Outputs
Outcomes: The observed changes in attitudes, knowledge, or behavior among program
participants that result from a program / service / initiative.
Output: Measures of evidence of program implementation or quality.
36National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Outputs are sometimes confused with outcomes
Outputs Child Outcomes
Numbers of volunteersNumber of children served
Changes in healthChanges in behaviorChanges in knowledgeChanges in placementChanges in permanency
37National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
What exactly is impact? Researcher definition of impact: Linking (through study/research) the
observed outcomes to a program / service / initiative. Does the program lead to outcomes? Can we control for as many other explanations as possible?
Impact could also mean the very long term changes attributable to a program / service / initiative. Child welfare advocates – do children who return home to their
parents STAY with their parents? Or do they re-enter foster care at a later date or repeatedly?
Impact could also mean the collective outcomes across multiple programs / services / initiatives. Does the same program offered in multiple locations have outcomes
that can be aggregated across the locations?
38National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
How does measuring impact differ?
Court Appointed Special Advocates
Program Outcome(s)
Decrease in length of stay in
foster care
Decrease in trauma
symptoms
Change in social worker at public child
welfare agency
Initiative at local
child welfare agency
Alternative programming
Are other conditions or programs causing these changes?
39National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
National CASA’s next steps for consideration…
• Developing a logic model• Identifying a small set of “key measures” for all CASA/GAL
programs• Addressing the evaluation challenges of networks structured
like National CASA• Conveying potential benefits and collaborating across the
network to build an evidence base
40National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams
Contact Information
Lori Delale-O’ConnorE-mail: ldelale-oconnor@childtrends.org
Sarah Catherine WilliamsE-mail: swilliams@childtrends.org