1 YOUTHBUILD EVALUATION Building evidence about the effect of YouthBuild on the young people it...
-
Upload
adrian-austen-patrick -
Category
Documents
-
view
217 -
download
0
description
Transcript of 1 YOUTHBUILD EVALUATION Building evidence about the effect of YouthBuild on the young people it...
1
YOUTHBUILD EVALUATIONBuilding evidence about the effect
of YouthBuild on the young people it serves
August 17, 2011Washington, D.C.
2
Overview• Project overview and rationale for
study• Research questions and study
design• Roles and responsibilities of
organizations participating in the study
• Timetable for study activities• Questions
2
3
What is the YouthBuild Evaluation and who is
involved?• Large scale, random assignment study
evaluating the effects of program participation on youth enrolled in 77 DOL and CNCS funded programs nationwide
• Sponsored by the U.S. Department of Labor and the Corporation for National and Community Service
• Research team:– MDRC– Social Policy Research Associates (SPR)– Mathematica Policy Research (MPR)
3
4
Why a study of YouthBuild?• Reengaging young high school
dropouts is an important national policy challenge
• Rigorous studies of second chance programs like YouthBuild can help shape their services and policies for youth
• Previous YouthBuild studies very useful, but not designed to measure program effects
4
5
What are the study benefits?
• Distinguishes YouthBuild as a model youth program to policymakers and local program operators
• Provides evidence that supports national policy and public funding
• Opens up the possibility of enrollment for some applicants who might otherwise be screened out
5
6
What questions will the study answer?
• What are the effects or impacts of YouthBuild participation?
• How are YouthBuild services delivered in different settings and what are the characteristics of the youth who participate?
• What is the cost effectiveness of YouthBuild programs?
6
7
What will we learn about program effects?
How do the outcomes for youth enrolled in YouthBuild programs compare to similar youth?– On educational attainment, employment,
earnings, civic engagement, criminal justice involvement?
– Do the effects change over time?– Do some youth benefit more?
Data sources: public records data, enrollment forms, and surveys at 12, 30 and 48 months
7
8
What will we learn aboutprogram implementation?
What services are provided and to whom?– How are services delivered?–What kinds of organizations provide
services?–What are the characteristics of the youth?–What challenges are encountered and how
are they addressed?–What alternative programs are available?
Data sources: visits to programs, focus groups with participants, and program service data
8
9
What will we learn aboutcost effectiveness?
What is the cost of serving the average YouthBuild participant?
How does the cost compare to the impacts generated by the program?
Data sources: analysis of programs costs and participation
9
10
What is random assignment?
• Random assignment is a research method that determines the benefit a program has for the individuals it serves
• Uses a lottery-like process to place individuals into a program group and a comparison group
• Assures that groups have similar characteristics, so the differences over time between the groups is the impact of the program
10
11
Why use random assignment?
• Most reliable way to measure program effects
• Widely used in social service settings• Endorsed by OMB, DOL, DOE and
other federal and private agencies• Fair and equitable way to determine
who receives program services
11
12
80
70
0
20
40
60
80
100
Ready to Work
Per
cent
Pla
ced
In A
Jo
b
I Want A Job
Program Group
Comparison Group
40
0
20
40
60
80
100
Per
cent
Plac
ed In
A
Job
20
General example of random assignment results
12
13
Example from National Guard Youth ChalleNGe
ProgramEducational Attainment
Outcome (%) ChalleNGe Comparison Difference
Has High School diploma or GED 71.8 55.5 16.2***
HS Diploma 30.3 26.6 3.7
GED 56.9 34.5 22.4***
Earned any college credit 34.9 18.8 16.1***
Employment in the Past 12 Months
Employed 88.4 84.5 3.9*
Earnings ($) 13,515 11,248 2,266***
Number of months employed 8.1 7.2 0.9***
13
14
Summary of random assignment steps
Comparison group referred to other
services in the community
Youth receive study welcome letter and $10 gift card 14
15
What is the program’s role?• Designate a contact person to liaise with study
team on progress and challenges• Participate in training on the research procedures• Recruit a sufficient pool of eligible applicants to
make random assignment possible• Administer the research procedures– Ensure applicants are informed about the study
and consent to participate– Record enrollment data into research database
• Participate in survey of grantees and host the study team during the process study
15
16
What is the research team’s role?
• Schedule meetings and site visits• Tailor random assignment to your program• Train program staff on research procedures• Monitor applicant pool to ensure that there
are no crossovers between YB and Non-YB comparison groups
• Collect and analyze research data• Provide assistance throughout research• Report findings to funder• Disseminate findings to practitioners, policy-
makers and others16
17
What is the coach’s role?• Provide technical assistance and
clarification to programs as needed.
• Potential issues include:– Challenges regarding recruitment– Comparison group – Data entry requirements– DOL performance standards
17
18
When will programs be involved?
• Select 77 sites from pool of DOL and CNCS funded YouthBuild programs (2011)
• Visit all sites and begin to develop study enrollment plan (2011)
• Enroll approx. 4,600 youth in the study (2011-2012)
• Administer the Grantee survey (2012)• Study program implementation (2012-
2013)
18
19
QUESTIONS?
19
20
Need more information?Please contactSharon Rowser
[email protected] Cynthia Miller
Eileen [email protected]
or visit www.mdrc.org