APHSA Members WebinarSeptember 26, 2013
The Innovative Strategies for Increasing Self-Sufficiency (ISIS) Evaluation
Introduction
Purpose of webinar Introduction of speakers Review of agenda
– ACF Perspective– Evaluation Summary– Evaluation Partner: Workforce Development Council
of Seattle-King County– Q & A
2
What Is ISIS? ISIS – the Innovative Strategies for Increasing Self-Sufficiency
evaluation – is an evaluation of promising strategies for increasing education, training, employment and self-sufficiency among low-income families.– Impact Study using Random Assignment– Implementation Study– Cost-Benefit Analysis
Evaluation by the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF)
Led by Abt Associates, in partnership with MEF Associates, APHSA, NCSL, NGA, consultants, Public Strategies, Inc. and the University of Michigan.
Support for programs provided by the Open Society Foundations, The Joyce Foundation, The Kresge Foundation, The Meadows Foundation and ACF/OFA Health Professions Opportunity Grants.
10-year project, currently in the random assignment phase (year 6)3
Why is ACF and OPRE evaluating career pathways? OPRE is the principal research and evaluation office of ACF.
– OPRE is to serve as the “principal advisor” to the Asst Secretary on research and evaluation matters.
– Our research covers the entirety of ACF’s program areas (though not evenly).
Our research and evaluation activities are guided by the principles of rigor, relevance, transparency, independence and ethics.
We have a long history of conducting experimental research, particularly in welfare to work and employment and training programs (see new book by Gueron and Rolston).
Our approach to research and evaluation has also taken on greater prominence and importance in the current Obama Administration and OMB’s emphasis on “evidence-based policy-making.”
4
Poverty and Pathways:The Motivation for ISIS 3 Key “Facts”
– Previous research has demonstrated effectiveness in moving families off cash assistance and into employment. (NEWWS, GAIN, New Hope, MFIP, SSP, state waiver experiments, etc.)
– Economic mobility is increasingly “skill-biased.”– Individuals with lower skills/lower income have fewer financial and
human capital resources from which to draw on and make productivity investments.
• Money is a major constraint, but time is also an important constraint. 2 Important Limitations of Existing Research:
– Education and training has not been as promising or financially beneficial for poor families as it has for the population as a whole.
– Families were not substantially better off even in the successful interventions (large numbers remain poor.)
1 Major Challenge– How can we integrate education, workforce development and human
services to create pathways out of poverty and increase self-sufficiency for a large and diverse set of American families? 5
Emerging Evidence: Career Pathways and Sector-Based Strategies
Year Up (Economic Mobility Corp, 2010)
• National org provides 6 months customized training + 6 month paid internship in IT and finance sectors w/local colleges, completers earn 14+ credits
• Results from small initial experiment (n=164)
6
Two key questions:1) Do future earnings recoup those foregone while in training?2) Do the benefits exceed the costs?
Emerging Evidence: Career Pathways and Sector-Based Strategies
Sectoral Training Strategies (Maguire et al. 2010)
3 experienced CBOs provide customized short-term training in varied high-demand fields
Disadvantaged adults with HS+ education
Careful screening Total sample n=1,014
7
The ISIS Evaluation
ISIS aims to learn about promising strategies for increasing employment and self-sufficiency among low-income individuals and families
Core principles– Collaborate with stakeholders and partners to Identify most relevant
interventions and strategies • Semi-structured discussions with over 250 individuals (key APHSA role)
Test a few things well– Experimental design with random assignment– Test one type of intervention in nine sites, rather than nine disparate
interventions Aim for positive influences on social policy and practice
8
Key Career Pathways Ideas
A series of connected education and training programs and support services – Addresses the wide range of skill and other needs of low-skilled
individuals– Provide credentials valued in high demand occupations/sectors
Enables individuals to secure employment within a specific industry or occupational sector, and to advance to successively higher levels of education and employment within that sector
Each step is designed to prepare the participant for the next level of employment and education
Build effective partnerships– Education and training providers, social service providers,
workforce 9
V. BA+ Programs Upper-Skilled Jobs V. BA+ Programs Upper-Skilled Jobs
IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs
IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs
III. Short-Term Certificate Programs Entry-Level Skilled Jobs
III. Short-Term Certificate Programs Entry-Level Skilled Jobs
II. Sectoral Bridge Programs Semi-Skilled Jobs
II. Sectoral Bridge Programs Semi-Skilled Jobs
I. Basic Bridge ProgramsI. Basic Bridge Programs
Pro
spec
ts f
or
go
od
-pay
ing
, st
ab
le e
mp
loym
en
t
Occupational, academic, and life skills
The Basic Career Pathways Model
Key Career Pathways Services
Comprehensive assessment– Academic and non-academic skills
Basic and technical skills instruction– Modularization, contextualization, acceleration, flexible delivery,
active learning
Supports – Proactive advising and guidance, supplemental instruction,
social supports, supportive services, financial assistance
Employment connections– During and after training
11
Foundational Academic SkillsFoundational Academic Skills
Intermediate Outcomes
Increase Performance & Persistence in Training
• Certificate/Diploma• 2- year, 4- year Degree
Improve Performance & Advancement in Jobs
• Earnings• Benefits• Job security
Improve Other Outcomes
• Income & assets• Child & adult well being• Local economic growth
Participant Characteristics
•Demographic•Educational•Economic
Program Inputs Primary Outcomes
TO
NE
XT
ST
EP
Occupational SkillsOccupational Skills
Psycho-social Factors Psycho-social Factors
Career Orientation and Knowledge Career Orientation and Knowledge
Resource Constraints Resource Constraints
Other Personal and Family ChallengesOther Personal and Family Challenges
Contextual Factors: Institutional, Economic, SocialContextual Factors: Institutional, Economic, Social
Initial Targeting & Placement Decisions
Initial Targeting & Placement Decisions
Theory of Change for Career Pathways
Take First/ Next Step In
Career Pathway/Latti
ce
Comprehensive Assessment
Supports
Employment Connections
Core Curriculum
The ISIS Sites
13
Year Up National Capital Region
Year Up Atlanta
Year Up Boston
Year Up Chicago
Year Up New York City
Year Up Providence
Year Up Puget Sound
Year Up San Francisco Bay
Area
San Diego Workforce Partnership Bridge to Employment
Program
Des Moines Area Community College
Prepared Learner Program
Valley Initiative for Development and
Advancement (VIDA)
Instituto del Progreso Latino Carreras en Salud
Madison Area Technical College Patient Care Pathways
Bellingham Technical College I-BEST
Pima Community College Pathways to Healthcare Program
ISIS HPOG site
ISIS site
Workforce Development Council of Seattle-King County Health Careers for All
Everett Community College I-BEST
Whatcom Community College I-BEST
Health Professions Opportunity Grants (HPOG) In 2010, the HPOG program was created as part of
the Affordable Care Act in order to provide TANF recipients and other low-income individuals with access to jobs that pay well and address workforce needs in health professions.
$67 million a year to 32 grantees across 23 states, including 5 tribal grantees.
Program is administered by the Office of Family Assistance (OFA).
Research and evaluation overseen by OPRE.
14
Health Professions Opportunity Grants (HPOG) To evaluate HPOG ACF has implemented a multi-
pronged research strategy that includes:– Implementation, systems and outcomes evaluation for
HPOG TANF/low income grantees– Evaluation of Tribal HPOG – Performance tracking: HPOG PRS collects uniform
data across all grantees– Impact studies in a subset of grantees
• HPOG Impact Studies• Innovative Strategies for Increasing Self-Sufficiency (ISIS)
– University-based field initiated research15
The Sample of ISIS Sites
Career pathways is both a framework and a model for particular programs – many variations incorporating basic ideas possible
9 ISIS programs exemplify central career pathways ideas, but vary greatly in:– Basic elements such as target populations (e.g.,
TANF), occupations and training steps, provider partnerships
– Lead organization—CBOs (3), Community/Technical Colleges (4) and WIBs (2)
– Key strategies included; their design, intensity, and duration
16
Who is participating in ISIS?
17
BTC DMACC Instituto Madison Pima SDWP VIDA WDC Whatcom Year Up NCR
Total Sample
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
48.3%
26.8%
6.9%18.4% 14.8% 13.0%
28.0%16.7% 14.3%
52.2%
22.0%
51.7%
73.2%
93.1%81.6% 85.2% 87.0%
72.1%83.3% 85.7%
47.8%
78.1%
Sex
Male Female
Who is participating in ISIS?
18
BTC
DMACC
Inst
ituto
Mad
ison
Pima
SDWP
VIDA
WDC
What
com
Year U
p NCR
Total S
ample
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
55.2%
19.6%28.7%
39.8%
19.0%29.5%
36.7%
14.6% 14.3%
98.6%
32.8%
31.0%
38.4%
55.4%
45.8%
44.6%
42.5%
53.3%
58.3%
1.4%1.4%
44.9%
13.8%
42.0%
15.9% 14.4%
36.4%28.0%
10.0%
27.1%
85.7%
22.3%
Age at Baseline
Under 25 25 to 39 40 and older
Who is participating in ISIS?
19
BTC
DMACC
Inst
ituto
Mad
ison
Pima
SDWP
VIDA
WDC
What
com
Year U
p NCR
Total S
ample
0%
10%
20%
30%
40%
50%
60%
70%
80%
44.8%
75.4%
54.4% 53.2%
68.1% 66.5%
53.5%
75.0%
28.6%
50.7%
60.4%
37.9%
9.4%
19.8%27.4%
7.1%
13.5% 12.7%
6.3%
28.6% 31.9%
15.0%
Total Family Income in Prior Year
Below $15,000 $30,000 or Above
ISIS Program Diversity
Health Office/Business
IT Welding/Electrical
Customer Service
Building Maint..
DMACC X X X X
I-BEST X X X
Instituto X
MATC X
PCC X
SDWP X
VIDA X X X X
WDC X
Year Up X X
20
ISIS Research Questions
What is the impact of the program on persistence in education and achievement of credentials and degrees?
What is the impact of the program on career-track employment and earnings?
What is the impact on well-being?
21
Study Components
ISIS will include site-specific: Impact study
– Using baseline data, administrative data (UI, college records), surveys
Implementation study– Interviews, staff surveys and site-level administrative data
Cost-benefit study
Study parameters: Randomly assign a minimum of 1,000 eligible
individuals to treatment and control groups– Control group can access other service in the community
22
Study Deliverables
Evaluation design Program profiles (2013) Cross-program implementation report Program-specific 15 month reports
– Implementation – Impact– Cost
23
ISIS Program
Health Careers for All Workforce Development Council of Seattle-King
County
24
What is HCA?
Health Careers for All is funded by a 5-year HHS HPOG grant
Healthcare training via cohorts or individual training support
Career and education “navigation” support Efforts to better align systems that serve the target
population (workforce, education, social service)
Outcome Targets for HPOG
Over the 5 years of the grant period: Enroll 920 TANF recipients and other low-income
adults and youth Achieve a training completion rate of 70%
(benchmark = 55%) Achieve a training-related placement rate of 60%
(benchmark = 25%) Achieve a return-to-training rate of 25% (benchmark
= 11%)
Outcomes to Date
Over life of the HPOG grant Nearly 600 customers enrolled Approximately 300 customers completed healthcare
training More than 150 customers employed Approximately 60 customers continued/returned for
next level training
TANF Participation
Project Goal to have TANF recipients comprise approximately 1/3 of enrollments over the HPOG project period
TANF participation has averaged 40-45% over the first 3 years of the project
Project designed in close partnership with TANF agency
TANF Partnership
Partners in developing project model and proposal Members of project steering committee overseeing
implementation of project model Clearly articulated outreach and referral protocol Regular check-in meetings in addition to steering
committee
HCA in ISIS
Implemented ISIS in summer 2012 (end of Year 2 of HPOG grant)
ISIS Goal: Randomly assign 700 individuals Outcomes to Date (end of Aug 2013):
– 267 individuals RA’d (51% TANF)– 134 assigned to treatment group (51% TANF)– 110 enrolled in HCA– 94 entered training
30
TANF Partnership Post ISIS
Re-doubled coordination efforts during implementation of ISIS study
Refined referral protocols Generated materials to support TANF staff Efforts to safeguard against over-enrollment in
control group during random assignment (ensure 50-50 treatment/control split among TANF referrals in addition to overall RA pool)
Customer Story Kim is a single mom of three boys who came to HCA in 2011 after nearly two years of training and job hunting in medical administration produced no employment. She had been trying to enter the healthcare field since 2008, was nearing the end of her lifetime TANF benefits, and had nearly exhausted her allotted training time through DSHS. Kim had 9 years of steady employment as a journeyman Industrial Maintenance Mechanic but in 2008 was terminated from employment because of excessive time off related to domestic violence issues. Due to the downturn in the economy, she was unable to find alternate employment in her profession. After a period of homelessness, she found transitional housing and subsequently entered subsidized housing. In addition to employment and housing challenges, Kim was busy managing 3 young boys all with behavioral issues and special needs. After some counseling to figure out the best path for her to enter a health career as quickly as possible, and to best utilize skills gained along the way, she enrolled in the HCA Phlebotomy cohort, which she completed with the highest scores in the class. Aside from excellent grades, Kim demonstrated patience, bilingual skills, a wry sense of humor and excellent ability work calmly with diverse patients. Following the phlebotomy training, Kim received job search assistance through WorkFirst and the HCA job developer. In September 2012, she was offered a position at LabCorp at $14.54 per hour, 25 hours per week to start. Three months later, Kim called to say that she had received praise for the work she was doing and was offered a full-time position. Kim is very thankful for the ongoing services received from the HCA program and now wants to start nursing prerequisites so she can eventually become an RN.
Lessons Learned to Date
Importance of understanding the TANF program and how it is implemented in your area
Importance of cultivating relationships at both the local and regional/state level
Benefits and challenges of grappling together with barriers (e.g., participation requirements and 12-month limit on training as a “countable” activity)
Questions?
34
Additional Information
www.projectisis.orghttp://www.acf.hhs.gov/programs/opre
35
Karen GardinerAbt [email protected]
Brendan KellyOPRE/[email protected]
Seanna RuvkunWorkforce Development Council of Seattle-King [email protected]
Cynthia [email protected]
Top Related