A Methodology for Statistical Adjustment under the ... · A Methodology for Statistical Adjustment...
Transcript of A Methodology for Statistical Adjustment under the ... · A Methodology for Statistical Adjustment...
2016
Ryan Sutter
Chief Evaluation Office
6/24/2016
A Methodology for Statistical Adjustment under the Workforce Innovation and Opportunity Act (WIOA)
i
A Methodology for Statistical Adjustment under the Workforce Innovation and
Opportunity Act (WIOA)
Ryan Sutter
Chief Evaluation Office (CEO)
U.S. Department of Labor
Summary
This document presents CEO’s proposed methodology for setting WIOA performance targets. It begins
with background information on WIOA and its requirements on the statistical adjustment model,
followed by a brief description of previous methodologies used by The Department of Labor for
establishing performance outcome goals. A description of the data, limitations and constraints on
modeling follows. Next, the framework for identifying the methodology is presented. Model
specification results are then presented for the WIA Dislocated Worker employment rate 2nd quarter
after exit measure in order to motivate the final model specification as well as to assess the value of
conducting variable selection for all measures and programs. The final recommendation with respect to
the general approach is then made. Following the recommendation, regression results for the resulting
model specification for all programs and measures are presented using that approach. Simulations are
then presented for Program Years 2011 and 2012 (for those measures for which proxy data is available)
in order to demonstrate what implementation would have looked like using historic data. Finally,
summarized pass/fail results are presented along with some alternative methods for making this
determination.
ii
Contents
1. Introduction .............................................................................................................................................. 1
1.1. Summary of WIOA Requirements for Performance Goals ........................................................... 1
2. Historical Statistical Adjustment of Department of Labor (DOL) Program Performance ......................... 1
2.1 Previous Approaches to Statistical Adjustment .................................................................................. 3
2.1.1. Previous Approaches to Estimating the Weights used in the Statistical Adjustment ................ 3
3. Data on Department of Labor (DOL) Programs ........................................................................................ 5
3.1. Required WIOA Performance Outcomes ........................................................................................... 6
3.2. Cross Agency Data Limitations ........................................................................................................... 8
3.3. Description of DOL and Economic Data Used in the Analysis ............................................................ 8
4. Method for Statistical Adjustment of DOL Program Performance under WIOA .................................... 12
4.1. Methods Considered ........................................................................................................................ 12
4.2. Cross Validation ............................................................................................................................... 15
4.3. Variable Selection ............................................................................................................................ 15
5. Results – A WIA Dislocated Worker Program Employment Rate 2nd Quarter Example ......................... 17
5.1. Variable Selection – A Simulation .................................................................................................... 17
5.2. An Application to the WIA Dislocated Worker Employment Rate Data .......................................... 19
5.3. Cross Validation Results – Model Specification ............................................................................... 21
5.4. Stability of the Unit Effects .............................................................................................................. 22
5.5. Methodology Conclusions................................................................................................................ 23
6. Regression Results .................................................................................................................................. 24
7. Program Year 2011 and 2012 Simulations .............................................................................................. 28
7.1. Program Year 2011 and 2012 Detailed State Results ...................................................................... 43
7.2 Program Year 2011 and 2012 State Summary Results ..................................................................... 44
7.3. Program Year 2011 and 2012 Results – Alternative Thresholds ...................................................... 45
Appendix A. Example of Setting D to the National Average ....................................................................... 47
Appendix C. Detailed State Results ............................................................................................................ 50
Appendix D. Summarized State Results ................................................................................................... 103
iii
Tables
Table 1. WIOA Performance Outcomes ........................................................................................................ 6
Table 2. Explanatory Variables on Participant Characteristics ..................................................................... 9
Table 3. Explanatory Variables on Economic Conditions ............................................................................ 11
Table 4. RMSE for Each Model Specification .............................................................................................. 22
Table 5. Adult Program Employment Rate 2nd Quarter Regression Results.............................................. 25
Table 6. Adult Program Employment Rate 2nd Quarter Unit Effects ......................................................... 26
Table 7. Detailed State Results - Alabama .................................................................................................. 43
Table 8. Summarized State Results ............................................................................................................. 44
Table 9. Results for Alternative Thresholds ................................................................................................ 45
iv
Figures
Figure 1. Variable Selection Results – Test ................................................................................................. 18
Figure 2. Percentage of Appearance of the Variables in Models within One Standard Deviation of the Top
Model ........................................................................................................................................... 19
Figure 3. Dislocated Worker Employment Rate Variable Selection Results ............................................... 20
Figure 4. Percentage of Appearance of the Variables in Models within One Standard Deviation of the Top
Model ........................................................................................................................................... 21
Figure 5. Stability of the Unit Effects .......................................................................................................... 23
Figure 6. Adult Program Employment Rate 2nd Quarter ........................................................................... 29
Figure 7. Adult Program Employment Rate 4th Quarter ............................................................................ 30
Figure 8. Adult Program Median Earnings 2nd Quarter ............................................................................. 31
Figure 9. Adult Program Credential Rate 4th Quarter ................................................................................ 32
Figure 10. Dislocated Worker Program Employment Rate 2nd Quarter .................................................... 33
Figure 11. Dislocated Worker Program Employment Rate 4th Quarter .................................................... 34
Figure 12. Dislocated Worker Program Median Earnings 2nd Quarter ...................................................... 35
Figure 13. Dislocated Worker Program Credential Rate 4th Quarter ......................................................... 36
Figure 14. Wagner-Peyser Program Employment Rate 2nd Quarter ......................................................... 37
Figure 15. Wagner-Peyser Program Employment Rate 4th Quarter .......................................................... 38
Figure 16. Wagner-Peyser Program Median Earnings 2nd Quarter ........................................................... 39
Figure 17. Youth Program Employment or Education Rate 2nd Quarter ................................................... 40
Figure 18. Youth Program Employment or Education Rate 4th Quarter .................................................... 41
Figure 19. Youth Program Credential Rate 4th Quarter ............................................................................. 42
1
1. Introduction
WIOA requires that performance outcome targets be set at the state level. State-level performance
outcomes are a function of the characteristics of the participants being served as well as the labor
market conditions in which those participants are being served, and WIOA specifically requires that
these factors be accounted for.
The use of a statistical model when setting performance outcome targets is intended to level the playing
field by accounting for variation in the characteristics of the participants being served as well as for
differences in the economies they are being served in. A properly specified statistical model will
appropriately adjust performance goals for states serving hard-to-serve populations and/or in
economies facing more difficult labor market conditions.
The statistical model objectively quantifies how, and to what extent, each of these factors affects
program performance outcomes. This information can be used to set appropriate performance
outcome targets in an effort to avoid penalizing sponsored programs for variability in participant
characteristics and economic conditions. The goal of the statistical approach is to account for these
factors and separate them from those factors that program administrators are able to control and
should be held accountable for.
1.1. Summary of WIOA Requirements for Performance Goals
In order to promote enhanced performance outcomes and to facilitate the process of reaching
agreements with States, the Secretary of Labor, in conjunction with the Secretary of Education, must
establish performance goals for the pertinent core programs in accordance with WIOA Sec. 116 as well
as with the Government Performance and Results Act (GPRA) of 1993, and the amendments made by
that Act. Such goals shall be long-term performance goals on each of the primary indicators to be
achieved by each of the programs. The goals shall be revised to reflect the actual economic conditions
and characteristics of participants in that state program, each year. Revisions to the performance goals
are to be based on an objective statistical model, which must be developed and disseminated, that
accounts for actual economic conditions and participant characteristics. WIOA requires the model to
account for differences in unemployment rates and job gains/losses in particular industries in addition
to the characteristics of participants when they entered the programs. Specific participant
characteristics to be adjusted for include: prior work experience, educational/occupational skills
attainment, dislocation for high-wage and high-benefit jobs, low levels of literacy or English proficiency,
disability status, homelessness, ex-offender status, and welfare dependency.
2. Historical Statistical Adjustment of Department of Labor (DOL)
Program Performance
2
The DOL has been using statistical methods in various ways and at various intensities for developing
program performance outcome targets for several decades under the Job Training Partnership Act of
1982 (JTPA) and the Workforce Investment Act (WIA) of 1998.
Under Job Training Partnership Act (JTPA) of 1982, DOL began establishing sets of national standards
using historical data. These standards were to be used to establish performance goals for 600+ local
state development areas (SDAs) receiving federal money through state formula grants. However, given
a desire to hold SDAs harmless for differences in local conditions, DOL developed regression models to
adjust national performance standards for variation in local economic conditions and the characteristics
of program participants. In the early period of JTPA, state administrators had considerable latitude in
how they used the national standards and regression adjustments to set performance measures for
SDAs receiving grants on their behalf1. They could employ national standards without any adjustment,
they could use the DOL regression adjustments, they could use the regression adjusted targets with
further discretionary adjustments based on state specific conditions, or they could develop their own
adjustment methodologies2. Amendments to JTPA in 1992 eliminated this flexibility and required
statistical adjustments based on specific considerations3.
WIA replaced JPTA in 1998. Under WIA, performance standards were required at both the state and
local level but were negotiated beforehand on a program year (July – June) basis based on expected
economic conditions, participant characteristics and the services to be provided. Concerns over
subjectivity in the negotiated performance standards as well as the question of how to accommodate
the continuous improvement requirements of GPRA during the great recession lead to the resuscitation
of statistical adjustment methodologies for establishing performance standards. Beginning with a pilot
in nine states in program year 2010, DOL reintroduced the use of regression models. Under WIA,
however, regression-based statistical adjustment models were used to inform the negotiations between
states and the DOL and to provide local area estimates as a service to states for informing the
establishment of performance standards for local areas under WIA (denoted as Workforce Investment
Boards - WIBs). DOL expanded the pilot to all states for program year 2011 negotiations on
performance goals and has used various regression-based statistical adjustment methodologies up until
the passage of the Workforce Innovation and Opportunity Act in 20144.
1 King, C. and J. Siedlicki, Approaches to Adjusting Workforce Development Performance Measures, Occasional
Brief Series, Vol. 1 No. 2, (2005). 2 Barnow, B. and J. Smith, “Performance Management of U.S. Job Training Programs” In Job Training Policy in the
United States, C. O’Leary, R. Straits, Wandner, eds. (2005), Kalamazoo, MI: W.E. Upjohn Institute, pp. 21-56. 3 Social Policy Research Associates, “Guide to JTPA Performance Standards for Program Years 1998 and 1999”,
Menlo Park, CA, (1999). 4 See for example: Training and Employment Guidance Letter (TEGL) No. 25-13, “Negotiating Performance Goals
for the Workforce Investment Act (WIA) Title 1B Programs and Wagner-Peyser Act Funded Activities for Program Year (PY) 2014”, U.S. Department of Labor, (2014).
3
2.1 Previous Approaches to Statistical Adjustment
Historically, DOL has used statistical adjustment methodologies for establishing performance goals that
were based on predictions of performance outcomes (�̂� in statistical descriptions and models). The
predictions were used as reference points in comparison to actual outcomes. Those entities that met or
exceeded �̂� passed, while those significantly below it failed. Unit specific effects were not considered
when constructing performance goals and so were not separated from model error and were essentially
treated as residuals. Positive residuals indicated higher quality performance while negative residuals
indicated lower quality performance. Although this was the basic logic underlying the various JTPA and
WIA adjustment methodologies, there were important differences with respect to the approaches taken
for parameter estimation, as well as definitional aspects involving minimally acceptable performance (or
the threshold for failure) when setting (or negotiating in the case of WIA) the performance goals.
The DOL adjustment models under JTPA and WIA were based on the following equation:
𝑇 = 𝐷 + 𝑤1(𝑋1 − 𝑥1̅̅ ̅) + ⋯ + 𝑤𝑛(𝑋𝑛 − 𝑥𝑛̅̅ ̅)
Where T was the model adjusted performance goal, D was the departure point, 𝑋1 … 𝑋𝑛 were the values
for the n explanatory variables included in the model for each entity, 𝑥1̅̅ ̅ … 𝑥𝑛̅̅ ̅ were the national
averages of the n explanatory variables included in the models, and 𝑤1 … 𝑤𝑛 were the coefficients (or
what were described as weights) for each of the n explanatory variables.
Setting D to the national average actual outcome results in a T that is equivalent to �̂�. This is shown
visually using a one variable case, for the purposes of visualization, in the Appendix A. This was the
approach taken under the WIA pilot in program year 2010. JTPA set the departure point arbitrarily to a
level that could be exceeded by approximately 75% of SDAs5. Under WIA program years 2011 – 2014, D
was set to the most recent observable past performance of each entity while 𝑥1̅̅ ̅ … 𝑥𝑛̅̅ ̅ represented the
most recent observable levels for each of the n variables6. In all cases, the w’s were estimated using
various types of regression methodologies.
2.1.1. Previous Approaches to Estimating the Weights used in the Statistical Adjustment
Under JPTA, model coefficients (w) were estimated using pooled OLS methods at the SDA level using
annual observations. Under WIA two different methods were used.
In program years 2010 and 2011, coefficients in the estimation process were estimated in two stages:
one that included individual personal characteristics, X, and the other that included local labor market
conditions, L. Formally, the estimation equation can be expressed as follows.
5 Social Policy Research Associates, “Guide to JTPA Performance Standards for Program Years 1998 and 1999”,
Menlo Park, CA, (1999) 6 Eberts, R., H. Wei-Jang, and C. Jing, “A Methodology for Setting State and Local Regression-Adjusted Performance
Targets for Workforce Investment Act Programs”, Upjohn Institute working paper 13-189, (2012).
4
isqqisqoisq QXY 21
Where isqY is the outcome variable for individual i in WIB s in year-quarter q, isqX are the individual
attributes for person i in area s in year-quarter q, Q denotes a quarter-year dummy variable and the 𝛽′𝑠
represents the coefficients7.
As indicated, model estimation was conducted in two stages8. The first stage used data from individuals
to estimate the relationship between the personal characteristics X and performance outcomes. The
second stage took the residuals from that estimation (that portion of the variation in the performance
outcome that was not explained by the personal characteristics) and aggregated them to the WIB level.
These aggregated residuals were then regressed WIB unemployment rates. A separate model was
estimated for each performance measure in each program. The observation of analysis for the first
stage was the individual program exiter9. For outcome measures other than six months average
earnings, the dependent variable (Y) was a dichotomous variable that took on the values of 1 if an
individual achieved the outcome and 0 if not. For example, entered employment was defined as having
positive earnings in the first quarter after exit for those individuals that were not employed at initial
program participation.
For simplicity and speed and because of the large number of models estimated, the models were
estimated using linear probability models, even when the dependent variable was binary10. Logit and
probit estimation techniques are generally recommended for estimating equations with zero-one
dependent variables. However, the authors of the methodology reported that using logit or probit
made it more difficult to interpret the results and created some complexities in calculating adjustments.
For example, they stated that because logit and probit are non-linear models, the adjustment factor
could not be calculated using sample means but rather required calculating probabilities for all
observations using the full set of data. Further, the argument was made that econometricians had
shown that the drawbacks of using linear probability models, compared with logit and probit
techniques, were minimal11. In order to test the sensitivity of the estimates to this simplification, both
techniques for entered employment and retention performance measures for the WIA Adult program
7 Eberts, R., H. Wei-Jang, and C. Jing, “A Methodology for Setting State and Local Regression-Adjusted Performance
Targets for Workforce Investment Act Programs”, Upjohn Institute working paper 13-189, (2012). 8 Note that dividing the estimation process into two stages assumed that personal characteristics were orthogonal
to local labor market conditions. 9 The term exiter denotes participants that have completed their program participation, which is defined as not
having received a service for 90 consecutive days. See: Training and Employment Guidance Letter (TEGL) No. 17-05, “Common Measures Policy for the Employment and Training Administration’s (ETA) Performance Accountability System and Related Performance Issues”, U.S. Department of Labor, (2006). 10
Eberts, R., H. Wei-Jang, and C. Jing, “A Methodology for Setting State and Local Regression-Adjusted Performance Targets for Workforce Investment Act Programs”, Upjohn Institute working paper 13-189, (2012). 11
For example: Jeffrey M. Wooldridge, Econometric Analysis of Cross Section and Panel Data, Cambridge, MA: MIT Press, (2002).
5
were estimated. The coefficients estimates were found to be quite similar if not virtually identical in
most cases12.
Beginning in program year 2012 and through 2014, estimation was conducted at the WIB level by
aggregating individuals and computing the outcome measures as rates (note that earnings outcomes
were computed as averages consistent with the definition of the measure). Mathematically, the
estimation equation is summarized as:
lqlqolq XY 1
Where lqY is the outcome variable for local area l in year-quarter q, lqX are the attributes (including
the WIB unemployment rate) for each LWIA l in year-quarter q, 𝛽 are the coefficients, and lq is an IID
error term. This methodological modification was done for two reasons. One was to simply estimation. Two, since
the outcomes were officially reported at the on an aggregated basis (at the local and state levels), it
seemed more straight forward to model the impacts of the given variables at that level. Why introduce
a multi-level modeling problem as well as rely on individual-level data when the outcomes were not
reported at the individual-level13?
Under all previous methods, variable selection was achieved by estimating a set of saturated models
containing all possible variables and excluding those that were not statistically significant. Further,
predictive performance was not directly addressed, rather, model fit was presented for the final models
in the form of the amount of variance explained (𝑟2).
3. Data on Department of Labor (DOL) Programs U.S. DOL has been collecting individual data on program participants for many years in various forms.
Under WIA, DOL has been receiving individual record data in the Workforce Investment Act
Standardized Record Data (WIASRD). Wagner-Peyser program data was collected at the state level but
was not submitted to the DOL at the individual level through the Labor Exchange Reporting System
(LERS) until program year 2013.
The WIASRD contains detailed information about each participant’s characteristics, program activities,
and outcomes. These standardized records are maintained by state workforce investment agencies for
all individuals that receive services or benefits from programs funded by WIA Title IB. Thus, this dataset
contains information on the several million participants receiving services funded full or in part by the
WIA Adult, WIA Dislocated Worker (including services financially assisted by National Emergency Grants
12
Per the authors of that report, Wooldridge in Econometrics: A Modern Approach, (2009) and Angrist and Pischke in Mostly Harmless Econometrics, (2009) reported very similar marginal effects using linear probability models, logit, and probit even for values of explanatory variables that are not close to the mean. 13
Sutter, R., “The Methodology for Setting Program Year 2012 Performance Targets”, Internal Analysis for the Office of Policy Development and Research, U.S. Department of Labor, (2012).
6
- NEGs), and WIA Youth programs14. The information contained in state workforce agency databases
was submitted to the DOL on an annual basis from program 2001 – 200915. Beginning in the 3rd quarter
of program year 2009, WIASRD records began to be submitted on a quarterly basis. Each file contained
the ten most recent quarters of information (in order to have enough information to compute the
lagged outcome measures) on all individuals that had received funded services during that time span.
Wagner-Peyser (WP) participant information is also maintained by state workforce investment agencies
for all individuals that receive services or benefits from programs funded by this program. Only
aggregated information was submitted to the DOL prior to program year 2013. Individual record data
on WP participants is more limited but does include information on tens of millions of participants,
including their characteristics, program activities, and outcomes. This information is submitted to DOL
through the Labor Exchange Reporting System (LERS). For more information, see the ET Handbook
Number 40616.
3.1. Required WIOA Performance Outcomes
Table 1 provides information on the WIOA performance outcomes specifically required under WIOA,
their definition, their detailed formulation, the programs they apply to, existing DOL data that is
available to compute them, and some notes on whether or not this data currently exists17.
The first column presents each required WIOA performance measure. The second column contains their
definition, as defined in the statute, while the third column presents more specific information
regarding how the measures have been computed using existing data. The fourth column provides
information on which of the DOL and Department of Education (DOE) programs that the measures apply
to. The fifth column indicates the programs on which DOL data currently exists to compute the
measures now, in spite of the new definitions. The last column contains summary information
indicating the extent to which the new WIOA measures can be computed with existing data. This
indication is also noted by the row shadings. The white rows can be fully computed with existing data,
the light grey rows can be partially computed with existing data, and the dark grey rows cannot be
computed with existing data.
Table 1. WIOA Performance Outcomes
WIOA Performance Measures
Definition Formula Relevant Programs Existing DOL
Data Available
DOL Data Notes
14
Detailed documentation and user guides on the WIASRD record layout, including reporting specifications and instructions and data files are available at: http://www.doleta.gov/performance/reporting/wia.cfm 15
Note that while data was collected beginning in 2001, it wasn’t until about program year 2004 that system growing pains were sorted out and the data became sufficiently reliable. 16
http://www.doleta.gov/performance/guidance/wia/et-406-handbook-expiration-022809.pdf 17
WIOA Sec. 116 (b)(2)(A).
7
Employment rate 2nd quarter after
exit
The percent of exiters who are employed in the 2nd quarter after
program exit
(count of unique exiters where 'earnings 2nd quarter after exit'
> $0 and ~=999999.99 within reporting cohort) / (Total
exiters in the reporting period)
WIOA Adult WIOA Dislocated Worker
Wagner-Peyser Adult Education
Voc. Rehabilitation
WIA, Wagner-Peyser
Can be done with existing
labor data
Employment rate 4th quarter after
exit
The percent of exiters who are employed in the 4th quarter after
program exit
(count of unique exiters where 'earnings 4th quarter after exit' > $0 and ~=999999.99 within
reporting cohort) / (Total exiters in reporting cohort)
WIOA Adult WIOA Dislocated Worker
Wagner-Peyser Adult Education
Voc. Rehabilitation
WIA, Wagner-Peyser
Can be done with existing
WIA data
WP data through the third quarter (proxy for the
4th quarter measure)
Median earnings 2nd quarter after
exit
The median earnings of all employed exiters
in the 2nd quarter after program exit
(median earnings of all individuals where earnings 2nd quarter after exit' > $0 within
reporting cohort) / (Total exiters in the reporting period)
WIOA Adult WIOA Dislocated Worker
Wagner-Peyser Adult Education
Voc. Rehabilitation
WIA, Wagner-Peyser
Can be done with existing
labor data
Postsecondary/diploma credential Rate within 1 year of exit
The percentage of program exiters who obtain a recognized
postsecondary credential or a
secondary school diploma or its
recognized postsecondary credential or
secondary school diploma or it's
recognized equivalent during participation or
one year after.
Adults and DW = (count of unique exiters that earned a
credential within 3 quarters of program exit / (total exiters that received training in the
reporting period)
Youth = (count of unique exiters that earned a credential or a diploma within 3 quarters of program exit/ (total exiters that received training or were
enrolled in education in the reporting period)
WIOA Adult WIOA Dislocated Worker
WIOA Youth Adult Education
Voc. Rehabilitation
WIA
Can be done with existing labor data for Adults, DW, and Youth programs through 3
quarters after exit (proxy for
4th quarter)
Measureable skill gain rate
the percentage of program exiters who,
during a program year, are in an education or training program that leads to a recognized
postsecondary credential or
employment and who are achieving
measurable skill gains toward such a credential or employment
(Total exiters who obtained documented progression on a
career pathway) / (total exiters enrolled in an education or
training program)
WIOA Adult WIOA Dislocated Worker
WIOA Youth Adult Education
Voc. Rehabilitation
None Not possible with existing
data
Employment or education rate 2nd quarter after exit
The percent of exiters who are employed or enrolled in education
in the 2nd quarter after program exit
(Count of unique exiters
employed in the 1st quarter
after exit or placed in post-
secondary education, advanced
training, military service, or a
qualified apprenticeship in the
1st quarter after exit / (Total
exiters in the reporting period)
WIOA Youth
WIA
Can be done with existing
WIA data for 3 quarters
quarter after exit (proxied by
1st quarter)
8
Employment or education rate 4th quarter after exit
The percent of exiters who are employed or enrolled in education
in the 4th quarter after program exit
(Count of unique exiters employed in the 3rd quarter after exit or placed in post-
secondary education, advanced training, military service, or a
qualified apprenticeship in the 3rd quarter after exit / (Total
exiters in the reporting period)
WIOA Youth
WIA
Can be done with existing
WIA data for 3 quarters
quarter after exit (proxied by
3rd quarter)
Employer services indicator
The primary indicators of serving employers
Not Yet Defined
WIOA Adult WIOA Dislocated Worker
WIOA Youth Wagner-Peyser Adult Education
Voc. Rehabilitation
None Not possible with existing
data
3.2. Cross Agency Data Limitations
One of the primary limitations on the types of models that can be used for establishing performance
goals is data availability across agencies. While the primary focus of this document is on DOL data, it is
important to stress that the WIOA requires the use of “an” objective statistical model18. As of the
writing of this document, that has been interpreted to mean using the same class of models and at the
same level of aggregation, although the explanatory variables included in the models may differ19. Due
to the fact the DOE does not have individual record data for all of its programs; the methodology is
currently limited to aggregate models until adequate individual record data are available for all
pertinent programs.
3.3. Description of DOL and Economic Data Used in the Analysis
The current methodology document is based on an analysis of the WIA and WP program data collected
by the DOL. As a result, the description of the data that follows pertains to the WIA and WP program
data available in the WIASRD and the individual-level WP program data contained in LERS.
Data in the WIASRD and the WP individual-level records is submitted by state workforce agencies on a
quarterly basis (as of program year 2009 3rd quarter for WIA and program year 2013 1st quarter for WP).
The individual-level participant information was aggregated to the state-level and aligned to the time
period associated with each outcome measure on quarterly basis. Each performance outcome is
associated with a particular “lag” due to the time it takes to receive wage record information as well as
the length of time defined by the particular performance measure. To properly relate the explanatory
variables to the outcome variables (dependent variables) for each quarter, the data must be processed
so that the outcomes directly correspond to the characteristics and economic conditions that occurred
in that quarter at the point when the participants exited the program and entered the labor market.
This was accomplished by creating a quarterly dataset that reflected the characteristics of the
participants that exited the program in each quarter, the outcomes that set of participants eventually
18
WIOA Sec. 116 (b)(3)(A)(viii). 19
It is not yet clear how firm this interpretation is.
9
obtained per the definition of the particular outcome measure, and the economic conditions occurring
in the quarter at which they exited and entered the labor market. For example, consider the
employment 2nd quarter after exit measures provided in Table 1. This measure relates the
characteristics of participants and the economic conditions they faced at exit to an employment rate
two quarters later.
Construction of the dataset began with back computing all of the new WIOA measures presented in
Table 1 for each program. These outcomes were then aligned to an aggregated set of explanatory
variables that were expressed in percentage terms (with three exceptions described below). Each
observation in the dataset represents a state-level quarterly observation relating outcomes to
percentages of individuals with each of the given personal characteristics presented in Table 2. All of
these variables are expressed as percentages of total exiters except for the youth educational
functioning levels and youth pre and post-test scores, which are expressed as averages. Columns 3 - 6 of
Table 2 indicate which variables are available for each of the specific programs, as the various programs
have different data collection requirements.
For specific information on how each variable was coded at collection, refer to the WIASRD and WP
individual record layouts for full details20. For example, the RecOtherGov variable reflects the
percentage of individuals that had received other public assistance. The WIASRD reporting
specifications define other public assistance recipients as having received cash assistance or other
support services from one of the following sources in the last six months prior to participation in the
program: General Assistance (from state or local government), Refugee Cash Assistance, Food Stamp
Assistance, and Supplemental Security Income (SSI-SSA Title XVI) but does include foster child payments,
temporary assistance for needy families (TANF) or needs-related payments provided by WIA title IB for
the purpose of enabling the individual to participate in approved training funded under WIA Title IB.
Many of these excluded sources of assistance are collected separately by other explanatory variables.
Table 2. Explanatory Variables on Participant Characteristics
Variable Names Variables Included
Program Adult DW Youth WP
GenderF Female x x x x
AGE1415 14<=Age<=55 x
AGE1617 16<=Age<=17 x
AGE18 Age=18 x
AGE1920 19<=Age<=20 x
AGE2635 26<=Age<=35 x x x
AGE3645 36<=Age<=45 x x x
AGE4655 46<=Age<=55 x x x
20
For detailed information on the specific coding instructions, see the WIASRD record layout documentation available here: http://www.doleta.gov/performance/reporting/wia.cfm.
10
AGE5665 56<=Age<=65 x x x
AGE66 66<=Age x x x
RACEHISP Hispanic ethnicity x x x x
RACEASIAN Race: Asian (not Hispanic) x x x x
RACEBLACK Race: Black (not Hispanic) x x x x
RACEHPI Race: Hawaiian/Pacific Islander (not Hispanic) x x x x
RACEAI Race: American Indian or Native Alaskan(not Hispanic) x x x x
RaceMulti Race: More than one (not Hispanic) x x x x
HsDropOut Highest grade completed: Less than High School graduate x x x x
HsGrad Highest grade completed: High school equivalency x x x x
CollegeDropOut Highest grade completed: Some college x x x x
Cert&OtherPs Highest grade completed: Certificate or Other Post-Secondary Degree x x x x
Assoc Highest grade completed: Associate degree x x x
Ba Highest grade completed: Bachelor degree x x x
EmpParticipation Employed at participation x x x
DIS Individual with a disability x x x
VETERAN Veteran x x
WageP2P3 Had earnings in 2nd and 3rd preprogram quarters x x x
WageP3 Had earnings in 3rd preprogram quarter x x x
WageP2 Had earnings in 2nd preprogram quarter x x x
WP Received services financially assisted under the Wagner-Peyser Act x x x
LIMENG Limited English-language proficiency x x x
SINGLEPAR Single parent x x
LowInc Low income x x x
RecTanf TANF recipient x x x
RecOtherGov Other public assistance recipient x x x
Homeless Homeless x x x
Offender Offender x x x
UIClaimant Unemployment insurance claimant, non-exhaustee x x x
UI Exhaustee Unemployment insurance claimant, exhaustee x x x
RecSuppServ Received supportive services x x
RecNeeds Received needs-related payments x x
RecInt Received intensive services x x
RecTrain Received training services x x
RecITA Established Individual Training Account (ITA) x x
RecPell Pell grant recipient x x x
RecPreVoc Received pre-vocational activity services x x
YouthParent Pregnant or parenting youth x
11
YouthNAA Youth who needs additional assistance x
EdStat Youth enrolled in education at or during program participation x
EdStatExit Youth enrolled in education at exit x
YouthEnrollEd Youth enrolled in education at participation x
YouthBSD Youth with basic literacy skills deficiency (at or below 8th grade) x
YouthFoster Youth that is or was in foster care x
YouthEdServ Youth that received educational achievement services x
YouthEmpServ Youth that received employment opportunities x
YouthAS Youth participated in an alternative school x
AvgEdLvl Average educational functioning level for Youth participants x
AvgPreTest Average standardized pre-test score x
AvgPostTest Average standardized post-test score x
Table 3 contains the information on the economic variables included to address the requirements of the
WIOA that the models explain variation in local labor market conditions, including unemployment and
changes in industrial structures (job gains and losses). All models contained the economic variables as
explanatory variables. The data described in Table 3 were obtained from the Bureau of Labor
Statistics21. One important note is that the unemployment rate was measured without seasonal
adjustment. This was done because the outcome measures derived from the WIA data were not
seasonally adjusted and, hence, seasonal variation should be equivalently modeled.
Table 3. Explanatory Variables on Economic Conditions
Economic Variables Definitions
Unemp Rate Not seasonally adjusted quarterly unemployment rate
NatResEmp
Percentage of total employment in NAICS 1133-Logging, or Sector 21-Mining
ConstEmp Percentage of total employment in Sector 23-Construction
ManfEmp Percentage of total employment in Sectors 31, 32, 33-Manufacturing
TechEmp
Percentage of total employment in Sector 51-Information, Sector 52-Finance and Insurance, Sector 53-Real Estate and Rental and Leasing, Sector 54-Professional, Scientific, and Technical Services, Sector 55-Management of Companies and Enterprises, or Sector 56-Administrative and Waste Services
EdHealthEmp
Percentage of total employment in Sector 61-Eductaional Services, or Sector 62-Health Care and Social Assistance
LeisHospEmp
Percentage of total employment in Sector 71-Arts, Entertainment, or Recreation, and Sector 71-Accommodations and Food Services
OtherServEmp Percentage of total employment in Sector 81-Other Services
21
Unemployment rate: http://www.bls.gov/lau; Employment: http://www.bls.gov/cew/datatoc.htm.
12
PublicAdminEmp Percentage of total employment in Federal, State, or Local Government
4. Method for Statistical Adjustment of DOL Program Performance under
WIOA
The general strategy for selecting a methodology for setting performance goals taken by CEO can be
summarized in the following way. Work began by back computing the WIOA measures (to the extent
possible) and building and assessing the available data (including data available to all of the agency
partners). Next, the intended use of the model within a performance management framework was
considered: was the goal to produce a forecast of what future outcomes will be; was it to identify, ex
post facto, what outcomes should have been attained; or was it to identify which particular places are
performing well given the characteristics of the participants and economic conditions in which they are
being served?
In some ways, the primary purpose was a bit of all three, however, with the latter goal dominating the
other two. The fundamental logic underlying the current work was that the statistical adjustment
process should level the playing field with respect to variation in performance outcomes by adjusting
outcome goals for participant characteristics and local economic conditions in such a way as to separate
them from program specific effects under the control of program administrators, to the extent this was
possible. Out of sample forecasting performance was computed using cross validation and was used to
select an appropriate model specification and to assess the significance of data driven variable selection.
4.1. Methods Considered
The identification of potential methods began with review of the previous target setting methodologies
discussed in Section 2, where predicted outcomes (�̂�) were used to set performance goals. In addition
to this type of approach, an alternative was to obtain an estimate of state specific effects that could be
separated from the impacts of measured participant characteristics and economic conditions when
establishing performance goals. This type of approach would produce an expected performance
outcome (�̂�′) that adjusts only for the participant characteristics and economic conditions while directly
parsing out state specific effects not explained by the included participant characteristics and economic
conditions.
To demonstrate, consider the linear model below, where observations are grouped into states j = 1 … j,
for each quarterly time period t = 1…t:
𝑦𝑗𝑡 = 𝛼𝑗 + 𝛽𝑥𝑗𝑡 + 𝜀𝑗𝑡; 𝜀𝑖𝑗𝑡~ 𝑁(0, 𝜎𝑦2).
Here, the effect of x on y, denoted 𝛽, represents the relationship of measured participant characteristics
and economic conditions (x) on performance outcomes (y). However, after accounting for the effect of
x, there is still additional variation in the overall level of y across units. The unit effect 𝛼𝑗 captures the
13
additional variation by which predictions of y in unit j must be adjusted upward or downward, given only
observations of x.
The interpretation of 𝛼𝑗 is that it represents unmeasurable or omitted factors that affect y, beyond
those included in x. If these factors were measureable and of interest, they could be included as
additional explanatory variables in the matrix x, eliminating the variation captured by 𝛼𝑗. However, in
situations where these factors are not practicably measurable, they must be captured by 𝛼𝑗. If the unit
effects are equivalent, the model reduces to the pooled model with 𝛼𝑗 = 𝛼 for all units.
There are two commonly applied methods for estimating variation in 𝛼𝑗: fixed effects and random
effects models. The fixed effects model is equivalent to a linear regression of y on x with a series of
dummy variables included to account for unit to unit variation in the outcome variable. The coefficients,
𝛼�̂�, computed for each unit are estimates of the true unit effects 𝛼𝑗. In the random effects model, the 𝛼𝑗
are not estimated directly, but are assumed to follow a probability distribution with a mean 𝜇𝛼 and
variance 𝜎𝛼2. The mean unit effect is estimated by 𝜇𝛼 and 𝜎𝛼
2 denotes how much the unit effects vary
around 𝜇𝛼.
Both models have their strengths and weakness. As noted by several researchers, the random effects
estimator is equivalent to the fixed effects estimator when it is assumed that 𝛼𝑗~𝑁(𝜇𝛼 , ∞) rather than
𝛼𝑗~𝑁(𝜇𝛼 , 𝜎𝛼2)22. That is to say, the random effects approach models 𝛼𝑗’s as arising from a finite
variance 𝜎𝛼2 that can be estimated whereas the fixed effects approach models them as being distributed
with an infinite variance (note that the pooled model assumes 0 variance). By estimating the variance,
𝜎𝛼2, the random effects model is a compromise between the fixed effects and pooled models that
accepts some bias in 𝛽 in exchange for a decrease in the variance of 𝜎𝛼2. As a result, the fixed effects
model will produce unbiased estimates of 𝛽 but those estimates can be subject to high sample to
sample variability. The random effects model, by partially pooling information across units, will accept
some amount of bias in 𝛽 in exchange for a considerable reduction in the variance of the estimates,
resulting in estimates that are closer to the true values across different samples23. In this application,
the selection of final methodology was based on minimizing out-of-sample prediction error using cross
validation methods (summarized below). As a result, historic sample data is used to determine which
approach fits the data better out-of-sample.
In addition to the traditional fixed and random effects variants, spatial econometric extensions were
explored to investigate the importance of spatial dependence in this sample data24. The primary
motivation was that spatial units (states in this context) can differ in their background variables, which
22
See for example: A. Gelman and J. Hill, “Data Analysis Using Regression and Multilevel/Hierarchical Models”, (2007), Cambridge University Press. J. Bafumi and A. Gelman, “Fitting Multilevel Models When Predictors and Group Effects Correlate”, (2007), Available at SSRN: http://ssrn.com/abstract=1010095. 23
T. Clark and D. Linzer, “Should I Use Fixed or Random Effects?”, (2012), Emory University. Available at http://polmeth.wustl.edu/mediaDetail.php?docId51315. 24
See for example: J. Elhorst, “Specification and estimation of spatial panel data models”, (2003), International Regional Science Review 26(3). O. Parent and J. LeSage, “A spatial dynamic panel model with random effects applied to commuting times”, (2010), Transportation Research Part B 44(5).
14
tend to be space-specific time-invariant variables that affect the dependent variable but that are difficult
or impossible to directly measure25. For example, some spatial units are located in coastal tourist
locations while others are not. Some units are primarily rural areas with higher concentrations of
periphery industries and transportation infrastructures while others are urban with higher
concentrations of urban industries and transportation networks. In addition, norms and values
regarding factors such as education, religion, criminal behavior, labor/leisure decisions, land use
patterns, etc., can differ rather dramatically from place to place. Failing to account for the spatial
distribution of these factors can lead to biased estimation results if spatial correlation is substantial26.
To address these concerns, spatial extensions of the fixed and random effects models were estimated in
the context of the spatial autoregressive framework. Comparisons of out-of-sample predictive
performance were again used to assess model performance. Spatial dependence was incorporated by
adding an additional term to the models, a spatially lagged dependent variable of the form 𝜌𝑊𝑦.
Where, W was a row normalized n x n spatial weight matrix that represented the spatial connectivity
among the various locations, 𝜌 was the spatial dependence parameter representing the strength of the
spatial dependence between neighboring observations and y was the dependent variable27.
Under all of the above modeling strategies, the unit effect 𝛼𝑗 can be interpreted as unmeasurable
variation resulting from differences in program design, a primary factor that cannot be directly
measured. It is important to note, however, that the unit effects likely also include additional omitted
variation in the outcomes that is not captured in the explanatory variables. In other words, the unit
effects are not exclusively program design effects, particularly in programs with fewer numbers of
explanatory variables (such as the WP program). However, it is certain that the estimated unit effects
represent some mixture of program design effects and omitted population characteristics.
To summarize, several alternative approaches have been considered: the pooled OLS model, Bayesian
variants of the pooled OLS model, fixed and random effects panel models, spatial extensions of the fixed
and random effects panel models, and quantile regression models that rely on estimating the
conditional median (as opposed to the mean) of the response variable y. The relative performance of
forecasts produced by these alternative methodologies were compared by cross validation to identify an
appropriate methodology for target setting that is based on the model’s real world ability to forecast the
outcome of interest in unobserved data. The model that minimizes out-of-sample prediction error is
ultimately used to set performance goals that adjust for participant characteristics and economic
conditions.
25
J. Elhorst, “Spatial panel data models”, (2010), In: M. Fischer and A. Getis (eds) “Handbook of applied spatial analysis”, Springer, Berlin, Heidelberg and New York. 26
J. LeSage and R. Pace, “Introduction to Spatial Econometrics”, (2009), CRC Press Taylor & Francis Group, Boca Raton. 27
For more detail, see: J. LeSage, “Regression analysis of spatial data”, (1997), Journal of Regional Analysis and Policy 27(2) or R. Sutter, “The Psychology of Entrepreneurship and the Technological Frontier-A Spatial Econometric Analysis of Regional Entrepreneurship in the United States”, (2010), Ph.D. thesis, George Mason University, http://ebot.gmu.edu/bitstream/handle/1920/5807/Ryan%20Sutter%20Dissertation%20Final%20CD%20Copy.pdf?sequence=1&isAllowed=y.
15
4.2. Cross Validation
Cross-validation is a model validation technique for assessing how a statistical model will generalize to
an independent (i.e. unobserved) dataset28. It is most often used when one wants to estimate
how accurately a predictive model will perform in applied practice. To accomplish this task, a model is
fed a dataset of known data on which it is trained (training dataset) and its predictive ability is assessed
against a set of withheld data (testing dataset). The goal of cross validation is test the model out-of-
sample, in order to limit problems like overfitting and provide insight into how the model will generalize
to an independent dataset (i.e. future program data in this case)29.
In this application, cross validation for selecting the best target setting model was implemented by
dividing the dataset into 36 folds. These folds corresponded to each quarter of program data that was
available and were sequentially used to test model performance out-of-sample. Under the approach
used here, each of the 36 quarters were excluded one by one. The model was then estimated using the
35 included quarters while the withheld quarter was predicted out-of-sample. This was done for each
quarter.
Once that has been done for all quarters, root mean squared errors (RMSEs) for each quarter was
averaged to identify which model predicted the out-of-sample quarters best. In this manner, model
performance can be considered with respect to complexity in order to determine a framework for target
setting purposes.
4.3. Variable Selection
The Longitudinal modelling techniques discussed above are commonly used for studying data collected
on units repeatedly through time. A variety of modelling approaches were investigated for handling
such data. However, variable selection, which is critical in many statistical applications, is not always
appropriately emphasized. This has been true for prior DOL target setting models, even though there
were many potential explanatory variables (and, hence, a huge number of candidate models), with little
to no theoretical basis for choosing between the alternative potential variables.
Although the final choice of model(s) must take into account subject matter and other non-statistical
aspects, as well as legislative requirements to include specific data elements, data-based statistical
methods are a useful tool for informing variable selection in cases where considerable uncertainty exists
regarding tradeoffs among alternative sets of explanatory variables. There are a number of available
routines, such as stepwise regression or Bayesian model comparison among others, however, many
have limited success and are conditional on the initial set of starting variables (stepwise regression) or
are very complex (Bayesian model comparison). An alternative method that was relatively straight
forward (although computationally intensive) for choosing between the candidate models based on
28
R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection", (1995), Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence 2(12). 29
E. Bradley and R. Tibshirani, “Improvements on Cross-Validation: The .632+ Bootstrap Method, (1997), Journal of the American Statistical Association 92(438).
16
alternative variable specifications was cross-validation (discussed above in the context of alternative
estimation strategies rather than for variable selection)30.
After considerable work, we utilized cross-validation to help inform variable selection. Implementation
was as follows. Given a dataset, split the data into a construction sample and a validation sample. We
used the construction sample to fit the model, and the validation sample to evaluate the prediction
error of the particular set of explanatory variables. The process is repeated for four splits (for the
purposes of speed). With few explanatory variables, estimation can proceed directly. However, with the
large number of variables, we were not able to compute the prediction error for all possible models
(note there 2𝑘 − 1 possible models, where k is the number of candidate variables). Therefore cross-
validation was conducted using a Markov Chain Monte Carlo (MCMC) random search procedure that
allowed us to systematically sample the model space in a strategic manner. This approach was based on
ideas originally developed for Bayesian comparison (and averaging) and was used to move efficiently
through the model space by turning the cross-validation procedure into one of random sample
generation from a finite population (the set of candidate models based on alternative sets of
explanatory variables)31.
Essentially, a probability distribution for the various candidate models was defined based on minimizing
the RMSE of the alternative sets of variables. Under the approach, the Metropolis-Hastings algorithm
was used to compare samples from this probability distribution. Convergence of the sampling scheme
ensures that variable selection from the random sample generated was consistent with that from all
candidate models. The main benefit of conducting this analysis was that it provided inferences regarding
how predictive accuracy of the models varied across the alternative specifications, including the full
saturated model.
Depending on the goal(s) and subsequent results, one can choose among many options. For example,
one could implement the most parsimonious model or choose to average predictions over set of top
models (for example those within one standard deviation of the best model based on error minimization
or some other metric) or simply choose all variables that appear in a model within one standard
deviation of the top model32. Based on considerable analysis and experimentation, we have determined
that the full saturated models can be utilized with confidence, as the saturated models were among the
top models and so we determined that the complexity of implementing this approach in a production
setting was not worth the benefit.
The approach was based on the following steps:
Step 1. Choose a starting model with a given set of explanatory variables.
30
See for example: E. Cantoni, C. Field, J. Flemming, and E. Ronchetti, “Longitudinal variable selection by cross-validation in the case of many covariates”, (2007), Statistics in Medicine 26 or S. Arlot and A. Celisse, “A survey of cross-validation procedures for model selection”, (2010), Statistics Surveys 4. 31
See for example: J. Hoeting, Madigan, D., Raftery, A., and C. Volinsky, “Bayesian Model Averaging: A Tutorial”, Statistical Science 14(4). 32
E. Cantoni, C. Field, J. Flemming, and E. Ronchetti, “Longitudinal variable selection by cross-validation in the case of many covariates”, (2007), Statistics in Medicine 26.
17
Step 2. Chose an alternative model that differs from the starting model by one explanatory variable.
The alternative model is selected using the starting model and randomly (with a 1/3rd chance):
o Adding an explanatory variable, not already included, into the model – “Birth Step”;
o Deleting an included explanatory variable from the model – “Death Step”;
o Switching and included explanatory variable with an excluded explanatory variable –
“Move Step”.
Step 3. Evaluate the RMSE of the starting model and the alternative model using an M fold cross
validation procedure.
Step 4. Compute the ratio, RMSE starting model / RMSE alternative model.
If the ratio > 1, the alternative model becomes the starting model (because it has better out
sample predictive performance, or rather, a smaller RMSE).
Else, draw a sample from a binomial distribution with the probability of success set equal to the
ratio from 4. If that equals 0, keep the starting model, otherwise that alternative model
becomes the starting model.
Step 5. Repeat the procedure until convergence is assured.
5. Results – A WIA Dislocated Worker Program Employment Rate 2nd
Quarter Example
Results are presented below for the case of the WIA Dislocated Worker program employment rate
second quarter after exit outcome measure. The presentation begins with a demonstration that the
variable selection routine worked with the application to the real dataset following. Cross validation
results with respect to model specification follows, relying on the same Dislocated Worker program
example. Finally, the stability of the unit effects is presented and some preliminary conclusions are
drawn.
5.1. Variable Selection – A Simulation
To conduct a simulation of the variable selection routine, a 15 x 500 explanatory variables matrix was
constructed using pseudorandom draws from the standard normal distribution. Five of these variables
were included in the generation of a simulated dependent variable, along with random perturbations
reflective of residuals. Coefficients on these variables were set to one as was the level of random noise
included in the construction of the dependent variable. The remaining 10 variables were not used to
generate the simulated dependent variable and so represent pure random noise irrelevant to the
generation of the simulated dependent variable. These 10 variables are likened to irrelevant
explanatory variables. This 15 variable dataset was subjected to the variable selection routine
presented in section 4.3.
18
The top portion of Figure 1 shows the 19,144 unique models identified by the sampling scheme, sorted
from best predictive performance to worst. Each row represents a unique model. Note that with 15
variables there are 32,767 possible unique models based on alternative combinations of explanatory
variables33. Black dashes represent cases where the given variable was included in the model whereas
blue dashes represent cases where the given variable was not included in the model. This figure shows
that the top models contain the 5 true variables much more often that do the worst models. The ten
false variables, on the other hand, were almost equally present across all of the candidate models. The
bottom portion of Figure 7 shows the RMSE for each of the sampled models. Approximately 700 models
were within one standard deviation of the RMSE of the top model.
Figure 1. Variable Selection Results – Test
33
215 − 1.
19
Figure 2 presents the percentage of the number of appearances of each variable in the 700 models
within one standard deviation of the RMSE of the top model. As can be seen from the figure, the
procedure clearly identified the true explanatory variables. Collectively, these variables were included
in the top models more than 90 percent of the time. The false variables, on the other hand, show up in
the top models around 55 percent of the time. Clearly, the procedure is able to identify the important
explanatory variables.
Figure 2. Percentage of Appearance of the Variables in Models within One Standard Deviation of the Top Model
5.2. An Application to the WIA Dislocated Worker Employment Rate Data
Figure 3 presents OLS based-results using 200,000 MCMC draws. 186,281 unique models were
identified by the sampling scheme. In the figure, black dashes once again represent cases where the
given variable was included in the model whereas blue dashes represent cases where the given variable
was not included in the model. These results suggest that specifications including many variables
tended to have smaller RMSEs. This is reflected by the presence of much more black at the top of the
figure relative to blue. Models associated with larger RMSEs have many fewer variables than those with
smaller RMSEs. Variable 45 (the unemployment rate) shows a pattern that is rather distinct among the
others. This variable is present in nearly all of the top models and almost never appears in the worst
models. This pattern is not readily present in any of the other variables.
The bottom portion of the figure shows the distribution of RMSEs across the candidate models. RMSEs
range from 0.05 to 0.1 with rapid declines (albeit small) after the best several hundred models. RMSE
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
% o
f A
pp
eara
nce
s in
th
e To
p M
od
els
The 15 Explanatory Variables
False Variables
True Variables
20
increases rather constantly for next 140,000 models with large increases in the worst performing areas
of the model space. Approximately 50 percent of the sampled models have RMSEs in the range of 0.05
and 0.065. The full saturated model, that is the model including all of the potential explanatory
variables, is among the best performing models. This an important result because it provides conclusive
evidence that estimating the models with the full set of explanatory variables does not degrade the
accuracy of the models in predicting performance outcomes out-of-sample. As a result, implementation
of the statistical adjustment model can proceed simply, using the full set of explanatory variables.
Figure 3. Dislocated Worker Employment Rate Variable Selection Results
Figure 4 presents the percentage of the number of appearances of each variable in the 8,188 models
within one-half standard deviation of the RMSE of the top model for two separate runs. Two separate
runs of the sampling scheme were conducted to check for convergence in the sampling scheme.
This figure provides a couple of important conclusions. First, while convergence appears to have
occurred, the results from the separate runs are not identical. The blue run tends to have slightly higher
21
inclusion percentages for all variables except the unemployment rate. While more runs would provide
smaller differences, the results do provide clear indication that 200,000 MCMC draws is sufficient for
convergence, as the general pattern is consistent. Second, in stark contrast to the simulated results,
only one variable clearly reduces RMSE vis a vis the others; that being the unemployment rate. This
variable appears in nearly every top model and stands out alone as the most relevant variable to
decreasing RMSEs. The other variables appear to have similar levels of importance, aside from the
white race and low income variables. These two variables are present in less than 50 percent of the top
models, a level consistent with their irrelevance based on the simulated results provided above.
Figure 4. Percentage of Appearance of the Variables in Models within One Standard Deviation of the Top Model
5.3. Cross Validation Results – Model Specification
As indicated in section 4.2 above, cross validation for model specification was conducted by dividing the
dataset into 36 quarterly folds. Each quarter was left out one by one and was used to estimate out of
sample predictive performance based on RMSE. The Male, MA+, and TradeTranUtilEmp variables were
omitted to avoid singularity of the explanatory variables matrix34. In order to assure comparable results,
34
The categories of variables pertaining to gender, education, and the economic structure sum to one, which would result in perfect collinearity between these variables if all categories are included in the explanatory
0
0.2
0.4
0.6
0.8
1
1.2
Gen
der
M
AG
E26
35
AG
E46
55
AG
E66
RA
CEA
SIA
N
RA
CEH
PI
RA
CEW
hit
e
HsD
rop
Ou
t
Co
llege
Dro
pO
ut
Ass
oc
Ma+ DIS
Wag
eP2
P3
Wag
eP2
LIM
ENG
Low
Inc
Rec
Oth
erG
ov
Off
end
er
UI E
xhau
stee
Rec
Nee
ds
Rec
Trai
n
Rec
Pel
l
Un
emp
Rat
e
Co
nst
Emp
Tech
Emp
Leis
Ho
spEm
p
Pu
blic
Ad
min
Emp
22
these variables were omitted from all specifications. The low income variable was included in spite of
the variable selection results because this variable is specifically mentioned by WIOA.
Results for OLS, a Bayesian heteroscedastic linear variant of OLS, Quantile, Fixed Effects, Random
Effects, and Spatial autoregressive variants of the Fixed and Random Effects models are provided in
Table 435. The worst performing model, based on RMSE, was the OLS model. The best was the Spatial
Fixed Effects model. However, differences in RMSE between the various approaches are less than 1
percent. The last important result is that traditional fixed effects models can be applied without spatial
extension with relatively no degradation in out of sample predictive performance. Out-of-sample
predictive performance declines by less than 2/5ths of one percent.
Table 4. RMSE for Each Model Specification
Model Specification RMSE
OLS 0.0520
Bayesian OLS 0.0512
Quantile 0.0516
Fixed Effects 0.0460
Random Effects 0.0462
Spatial Fixed Effects 0.0421
Spatial Random Effects 0.0432
5.4. Stability of the Unit Effects
Figure 5 presents the unit effects based on the traditional fixed effects model for three time periods.
The three groups are based on three equal sized groupings of the available data. The results indicate
that for the majority of states, the fixed effect has the same sign across the entire period investigated.
The magnitudes of the effects are also quite similar across the time period, albeit with some variation. It
is the case, however, that large positive or negative unit effects tend to maintain their sign across the
period. Only a few state’s have unit effects larger than +/-0.05 that flip sign during the period. The
average unit effect over the entire period is 0.06. Thus, the average unit effect is around 1/10th of the
size of the variation in employment rates across the states.
variables matrix. Perfect collinearity would render the explanatory variables matrix singular (as the determinant would be zero). 35
Weight matrices in the spatial models were specified according to first order contiguity.
23
Figure 5. Stability of the Unit Effects
5.5. Methodology Conclusions
Model specification results have been presented for the Dislocated Worker program employment rate
2nd quarter after exit measure. The results provide a couple of important concise conclusions. One,
variable selection is a non-issue in this particular application. The inclusion of the full saturated set of
explanatory variables does not degrade the accuracy of the models when predicting out of sample. As a
result, the final methodology does not require the implementation of data driven variable selection
routines. This greatly simplifies implementation of the statistical adjustment model and facilitates the
inclusion of all of the variables required under WIOA without concern. Two, simple fixed effects panel
models can be relied on for model estimation without the added complexity of spatial econometric
extension.
As a result of these conclusions, CEO is recommending that the statistical adjustment model include the
variables specified in Tables 2 and 3 with parameters estimated using a fixed effects model that includes
fixed effects for each state (or local area for a local area model). The individual-level data should be
aggregated to the state level (or local area for a local area model) on a quarterly basis.
Targets should be set according to the following expression 1, where 𝑇𝑗 denotes the target for state
j=1…53, �̂� denotes the coefficient estimates of each explanatory variable, and 𝑥𝑗 denotes the values for
each states explanatory variables. Under this approach, the unit effects, while estimated, are not
included in the targets. Expression 2 is equivalent to 1 but expresses the results in a more intuitive way.
In 2, the mean of the unit effects are added to the explanatory variables with each unit effect expressed
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53
Q4-Q14 Q15-Q25 Q26-Q36
24
as a deviation from the mean. Under this expression, the mean of the unit effects represents the
conditional mean of the outcome measure (average value of the outcome after parsing out the personal
characteristics and economic conditions). Each specific unit effect represents how much each state
(unit) deviates from the average.
𝑇𝑗 = �̂�𝑥𝑗 (1)
𝑇𝑗 = 𝑚𝑒𝑎𝑛(𝛼𝑗=1…53) + 𝛽𝑥𝑗 (2)
Initial WIOA performance targets (those targets set prior to the beginning of the program year) must be
set using the most recent available data at the time of model estimation. After the actual data is
reported and made available, the targets can be re-estimated using the actual year end program data.
Under this approach, the targets reflect the outcome the state should have achieved after adjusting only
for differences in the measured characteristics of the individuals being served in the programs and the
condition of the local economies, as measured by the economic variables. The unit effects are treated
as program specific effects that program administrators can control.
As indicated in Section 4, it is likely that the unit effects also contain some amount of unknown omitted
variation out of the direct control of program administrators, particularly in the WP program where
fewer participant characteristics are reported in the underlying program data. As a result, the true
program effects are some unknowable portion of the estimated unit effect. As a result, to the extent
that negotiations occur, it would make sense to focus them on establishing, through negotiation, that
portion of the estimated unit effects that will be treated as program effects.
6. Regression Results
The results from Section 5 indicated that simple fixed effect estimation using the full set of explanatory
variables available for each program and measure is an appropriate methodology for implementation of
the statistical adjustment model. This section of the report will present parameter estimates based on
the estimation of this approach.
Table 5 contains the parameter estimates for the Adult program employment 2nd quarter after exit
outcome measure as defined in Table 1. The results indicate that the model explains approximately 40%
of the variance of the outcome measure, not including that explained by the unit effects. The results
indicate that older workers have negative impacts on Adult program employment rates while younger
workers have positive (albeit statistically insignificant in this case) impacts. Racial composition tends to
have insignificant impacts as only the multi race category has a statistically significant negative impact at
the 90% level. Education attainment is an important variable for this measure with high school drop
outs having a statistically significant negative impact. Post-secondary education has positive impacts
with Associates degrees having the largest positive impact. Disabilities have a significant negative
impacts while limited English speaking status has positive impacts on employment rates. Larger shares
of exiters having worked in the quarter prior to participation have positive and statistically significant
impacts. Receipt of supportive services, intensive and training services have positive associations while
25
needs related payments and other government services have negative ones. The unemployment rate is
associated with a statistically significant negative impact on employment rates as does higher
concentrations of construction, leisure and hospitality, and service employment in the local economies.
Table 5. Adult Program Employment Rate 2nd Quarter Regression Results
Fixed Effects Model Dependent Variable = Dislocated Worker Employment Rate 2nd Quarter
R-squared = 0.3971
Rbar-squared = 0.3812
Time = 0.0180
Nobs, Nvars = 1904, 50
Variable Coefficient t-statistic t-probability
GenderF 0.0321 1.3937 0.1636
AGE2635 0.0163 0.4055 0.6852
AGE3645 0.0652 1.5684 0.1170
AGE4655 0.0287 0.6444 0.5194
AGE5665 -0.0977 -1.4123 0.1580
AGE66 -0.4556 -1.8760 0.0608
RACEHISP -0.0168 -0.5047 0.6139
RACEASIAN 0.1041 1.1344 0.2568
RACEBLACK -0.0334 -1.4436 0.1490
RACEHPI 0.1079 1.0472 0.2952
RACEAI -0.0295 -0.5226 0.6013
RaceMulti -0.1748 -1.8826 0.0599
HsDropOut -0.1896 -6.2392 0.0000
HsGrad 0.0108 0.6548 0.5127
CollegeDropOut 0.0820 3.2108 0.0013
Cert&OtherPs 0.1475 2.1015 0.0357
Assoc 0.2917 4.0238 0.0001
Ba 0.0824 1.3793 0.1680
EmpParticipation 0.0617 3.1741 0.0015
DIS -0.1472 -3.4946 0.0005
VETERAN -0.0705 -1.5392 0.1239
WageP3P2 -0.0893 -1.1314 0.2580
WageP3 -0.0271 -0.5106 0.6097
WageP2 0.1565 2.8307 0.0047
WP -0.0105 -1.8793 0.0604
LIMENG 0.1982 3.3633 0.0008
SINGLEPAR 0.0254 1.0039 0.3156
LowInc -0.0020 -0.1575 0.8749
RecTanf -0.0424 -1.0181 0.3088
RecOtherGov -0.0262 -2.2598 0.0239
26
Homeless -0.1124 -1.3500 0.1772
Offender -0.0330 -1.1091 0.2675
UIClaimant 0.0143 0.9661 0.3341
UI Exhaustee 0.0624 1.0961 0.2732
RecSuppServ 0.0345 3.2021 0.0014
RecNeeds -0.0931 -3.5618 0.0004
RecInt 0.0208 1.8956 0.0582
RecTrain 0.0752 5.5601 0.0000
RecITA -0.0128 -1.1771 0.2393
RecPell -0.0146 -0.6021 0.5472
RecPreVoc -0.0113 -1.2377 0.2160
Unemp Rate -0.0145 -10.9127 0.0000
NatResEmp -0.4484 -0.8937 0.3716
ConstEmp -1.0890 -2.8947 0.0038
ManfEmp 0.1559 0.3867 0.6990
TechEmp 0.4094 0.8498 0.3955
EdHealthEmp -0.3074 -1.0481 0.2948
LeisHospEmp -0.8130 -2.1491 0.0318
OtherServEmp -1.4630 -2.0215 0.0434
PublicAdminEmp 0.4310 1.0443 0.2965
Table 6 contains the estimated unit effects for the Adult program employment rate 2nd quarter after exit
measure. The average effect is 0.8461. This indicates that the conditional mean of this outcome
measure is an employment rate of 84.6. The Virgin Islands, Puerto Rico, and the District of Columbia
have the largest negative deviation from the average effect. These negative impacts mean that these
places have employment rates that are 10-20 points lower than the conditional average, after
accounting for the types of participants they are serving and the condition of the local economies.
Montana, Nevada, and Wyoming, on the other hand, tend to have employment rates that are
approximately 10 points higher than the conditional average, after accounting for their characteristics.
Table 6. Adult Program Employment Rate 2nd Quarter Unit Effects
StateFixedEffects Value
Average Effect 0.8461
AL -0.0553
AK 0.0056
AZ 0.0341
AR 0.0582
CA -0.0119
CO 0.0089
CT -0.0409
DE -0.0412
27
DC -0.1520
FL 0.0665
GA -0.0288
HI -0.0382
ID 0.0377
IL -0.0669
IN 0.0018
IA -0.0119
KS -0.0018
KY 0.0470
LA 0.0403
ME -0.0074
MD 0.0107
MA 0.0035
MI 0.0692
MN 0.0116
MS 0.0347
MO 0.0028
MT 0.1124
NE -0.0342
NV 0.1209
NH -0.0687
NJ -0.0018
NM 0.0522
NY -0.0400
NC -0.0256
ND 0.0265
OH 0.0415
OK -0.0432
OR -0.0099
PA 0.0178
RI 0.0488
SC 0.0310
SD 0.0588
TN 0.0328
TX 0.0132
UT -0.0811
VT -0.0024
VA -0.0452
WA 0.0162
WV 0.0403
WI -0.0413
WY 0.1065
28
PR -0.1167
VI -0.1852
The complete set of regression results are found in Appendix B.
7. Program Year 2011 and 2012 Simulations
Section 7 presents a simulation of the implementation of the recommended target setting approach for
PY 2011 and PY 2012. For these simulations, targets were set according to expression 1 in Section 5.5
where the targets do not include any portion of the estimated unit effects. Results for the simulations
are shown for each program and measure in Figures 6-17. In all of these figures, the pink circles denote
the targets while the blue circles denote the actual outcomes obtained by each state for that measure,
program, and year. The black lines represent the targets plus the estimated unit effects. In other
words, the black lines denote the predicted value of the outcome within the program year after
accounting for personal characteristics, economic conditions, program design effects and unmeasured
omitted variables. The green shading around the targets denote a 90% interval around the target (ETA’s
proposed failure threshold). Actual outcomes (blue circles) falling below the green shaded area
represent failures while those above within the green areas meet the targets and those above them
exceed them. The yellow and pink shaded areas show results for the imposition of an 85% interval
(yellow) and 80% interval (pink).
Figure 6. Adult Program Employment Rate 2nd Quarter
30
Figure 7. Adult Program Employment Rate 4th Quarter
31
Figure 8. Adult Program Median Earnings 2nd Quarter
32
Figure 9. Adult Program Credential Rate 4th Quarter
33
Figure 10. Dislocated Worker Program Employment Rate 2nd Quarter
34
Figure 11. Dislocated Worker Program Employment Rate 4th Quarter
35
Figure 12. Dislocated Worker Program Median Earnings 2nd Quarter
36
Figure 13. Dislocated Worker Program Credential Rate 4th Quarter
37
Figure 14. Wagner-Peyser Program Employment Rate 2nd Quarter
38
Figure 15. Wagner-Peyser Program Employment Rate 4th Quarter
39
Figure 16. Wagner-Peyser Program Median Earnings 2nd Quarter
40
Figure 17. Youth Program Employment or Education Rate 2nd Quarter
41
Figure 18. Youth Program Employment or Education Rate 4th Quarter
42
Figure 19. Youth Program Credential Rate 4th Quarter
7.1. Program Year 2011 and 2012 Detailed State Results
Section 7.1 describes how to interpret detailed breakouts of the state by state results for the PY 2011
and PY 2012 simulations. There is one important caveat with respect to these tables. In these tables,
targets for outcomes measured in rates were capped at 1. In other words, targets above 1 were set
equal to one. Minimum values for the rates were set to 0.1. For median earnings, there was no cap.
However, minimum targets were set to $1,000. These rules were put in place to render the percent of
target measures more meaningful.
Table 7 presents the results for Alabama. The rows of the tables present the targets, the actuals, and
the percent of the target that the actual outcome represents (shown in bold). The columns present the
results for each of the 4 measures for which historic data was collected and available. The greyed
numbers represent the average percent of targets by measure and by program. In PY 2011, Alabama
obtained an average of 92.7% of the target for the employment rate 2nd quarter after exit measure
(denoted ER), an average of 103.6% of the target for the employment rate 4th quarter after exit measure
(denoted ER4), an average of 92.0% of the target for the median earnings measure (denoted ME), and
an average of 70.0% of the credential rate target. By program the results were an average of 91.0% of
the Adult program targets, 85.6% of the DW targets, 89.6% of the Youth targets, and 98.7% of the WP
targets. The number shown in the bottom right hand corner of the table corresponds to the average
percent of the targets across both the programs and measures, and is computed as the average of the
averages.
Table 7. Detailed State Results - Alabama
Alabama
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.758 0.708 $5,065.50 0.856
Adult Actual 0.704 0.720 $5,068.30 0.594
Adult% 92.9% 101.7% 100.1% 69.34% 91.0%
DW Target 0.843 0.773 $6,250.00 0.866
DW Actual 0.712 0.747 $6,341.40 0.517
DW% 84.5% 96.7% 101.5% 59.72% 85.6%
Youth Target 0.638 0.608
0.615
Youth Actual 0.601 0.569
0.498
Youth% 94.3% 93.6%
80.92% 89.6%
WP Target 0.639 0.523 $5,814.10
WP Actual 0.634 0.640 $4,325.30
WP% 99.2% 122.3% 74.4%
98.7%
Avg % of Target by Measure 92.7% 103.6% 92.0% 70.0% 90.4%
44
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.782 0.729 $5,080.60 0.839
Adult Actual 0.722 0.708 $4,915.80 0.594
Adult% 92.3% 97.1% 96.8% 70.77% 89.2%
DW Target 0.849 0.776 $6,233.70 0.831
DW Actual 0.743 0.739 $6,232.50 0.470
DW% 87.5% 95.3% 100.0% 56.59% 84.9%
Youth Target 0.674 0.662
0.606
Youth Actual 0.643 0.616
0.485
Youth% 95.4% 93.0%
79.98% 89.5%
WP Target 0.650 0.534 $5,977.80
WP Actual 0.629 0.642 $4,446.70
WP% 96.8% 120.2% 74.4%
97.2%
Avg % of Target by Measure 93.0% 101.4% 90.4% 69.1% 89.3%
Defining state failure as having a target below 90% of the target on one measure or one program for
two consecutive years would results in Alabama failing this simulated two year period. Alabama would
have failed for three reasons. First, their average credential rate outcomes were approximately 70%,
which is lower than 90% for both years. In addition, the average percentage of the target for the
Dislocated Worker program was approximately 85%, once again below the 90% threshold. Lastly, the
average percentage of the Youth targets was a fraction below the 90% threshold. The complete set of
detailed state results is found in Appendix C.
7.2 Program Year 2011 and 2012 State Summary Results
Table 8 presents summarized output from the PY 2011 and PY 2012 simulations for Alabama and Alaska.
These tables present the same information that was shown in table 7 with the same conditions.
Table 8. Summarized State Results
Alabama
Measure 2011 2012 Fail=1
2Q Employment 92.7% 93.0% 0
4Q Employment 103.6% 101.4% 0
Earnings 92.0% 90.4% 0
Credential 70.0% 69.1% 1
Program 2011 2012 Fail=1
Adults 91.0% 89.2% 0
Dislocated Workers 85.6% 84.9% 1
Youth 89.6% 89.5% 1
Wagner-Peyser 98.7% 97.2% 0
Alaska
Measure 2011 2012 fail=1
2Q Employment 106.2% 115.0% 0
4Q Employment 99.2% 106.9% 0
Earnings 141.4% 149.2% 0
Credential 96.3% 106.6% 0
Program 2011 2012 fail=1
Adults 113.5% 114.5% 0
Dislocated Workers 111.4% 121.7% 0
Youth 113.1% 136.3% 0
Wagner-Peyser 98.6% 100.5% 0
45
The rows of the table present the percent of the target that the actual outcome represented by measure
and program. The results are shown for PY 2011 and PY 2012. Failures are marked by 1s in the table.
The purpose of these tables is to more efficiently present the causes of the failures by state and
program. From table 8, it is apparent that Alabama failed its performance targets due to the credential
rate and the Dislocated Worker and Youth programs. Alaska, on the other hand, passed all measures for
all programs and years.
Results for all states are presented in Appendix D.
7.3. Program Year 2011 and 2012 Results – Alternative Thresholds
Table 9 presents 4 alternative methods for determining passing and failing the performance methods. It
should be noted that these results are based on straight comparison of the actual results to the targets
without negotiating. The only rules applied were that targets were capped to a maximum value of 1.0
for all rate measures and minimum targets of 0.1 and $1,000 dollars were imposed on the rate and
earnings measures.
Column one of this table indicates states that would have failed to meet their performance targets using
a 90% threshold rule. If a state had an actual outcomes that averaged less than 90% of their targets for
any program or measure for two consecutive years, they were deemed a failure and marked with a 1 in
this column. Columns two and three relaxed the thresholds to 85% and 80% using the same procedure.
The fourth column, on the other hand, provides an alternative that consolidates the 8 individual metrics
into one consolidated metric that represents the overall average percent of the target each state
achieved. This metric corresponds to the value in the lower right hand corner of table 7 (and the tables
in Appendix C). This consolidated metric takes into account the fact that states may perform
significantly above target on some measures while they may perform slightly below target on others. In
other words, it provides a more concise metric that better represents the true percentage of the target
each state achieved across all measures and programs. Over two years, no states failed. In one year
increments, Alabama would have failed in PY 2011. This metric could be considered as an alternative to
the proposed threshold.
Table 9. Results for Alternative Thresholds
State Below 90%
Target 2 Year Failures
Below 85% Target
2 Year Failures
Below 80% Target
2 Year Failures
Overall Average 90%
2 Year Failures
Alabama 1 1 1 0 Alaska 0 0 0 0 Arizona 0 0 0 0 Arkansas 0 0 0 0 California 0 0 0 0 Colorado 1 0 0 0 Connecticut 0 0 0 0
46
Delaware 0 0 0 0 District of Columbia 1 1 1 0 Florida 0 0 0 0 Georgia 1 0 0 0 Hawaii 1 1 1 0 Idaho 0 0 0 0 Illinois 0 0 0 0 Indiana 1 0 0 0 Iowa 0 0 0 0 Kansas 0 0 0 0 Kentucky 1 1 0 0 Louisiana 1 1 0 0 Maine 0 0 0 0 Maryland 0 0 0 0 Massachusetts 0 0 0 0 Michigan 0 0 0 0 Minnesota 0 0 0 0 Mississippi 1 1 1 0 Missouri 1 0 0 0 Montana 0 0 0 0 Nebraska 1 0 0 0 Nevada 1 1 1 0 New Hampshire 0 0 0 0 New Jersey 1 0 0 0 New Mexico 1 1 0 0 New York 1 0 0 0 North Carolina 1 0 0 0 North Dakota 1 1 0 0 Ohio 0 0 0 0 Oklahoma 1 1 0 0 Oregon 0 0 0 0 Pennsylvania 0 0 0 0 Rhode Island 0 0 0 0 South Carolina 0 0 0 0 South Dakota 1 1 0 0 Tennessee 0 0 0 0 Texas 0 0 0 0 Utah 1 0 0 0 Vermont 1 1 1 0 Virginia 1 1 1 0 Washington 0 0 0 0 West Virginia 0 0 0 0 Wisconsin 0 0 0 0 Wyoming 1 1 1 0 Puerto Rico 1 0 0 0 Virgin Islands 1 1 1 0
Total # Failures 25 15 9 0
47
Appendix A. Example of Setting D to the National Average
Figure 1 shows the Dislocated Worker employment rate 2nd quarter after exit along the y-axis and the unemployment rate along the x-axis.
Figure 1.
Figure 2 shows the OLS regression line in blue fitted to this data with an intercept term and the
unemployment rate as the sole explanatory variable. The red dots are the �̂�’s estimated as �̂� = �̂� + 𝑋�̂�.
Figure 2.
48
Figure 3 plots the targets computed as 𝑇 = 𝐷 + 𝑤1(𝑋1 − 𝑥1̅̅ ̅) with D set to the average Dislocated
Worker employment rate 2nd quarter after exit, 𝑤1being the weight, �̂�, and 𝑥1̅̅ ̅ being the average unemployment rate. The graph shows that the targets are identical to those seen in Figure 2.
Figure 3.
Figure 4 adds an additional red line representing the failure threshold under WIA, which is performance outcomes that are 80% or more below the performance targets.
Figure 4.
Figure 5 adds the JTPA targets represented by black dots, which are again computed as 𝑇 = 𝐷 +𝑤1(𝑋1 − 𝑥1̅̅ ̅) . Under JTPA, however, D, was specified as minimally acceptable performance, which was generally set to a level where 75% of the entities could pass. Note that, coincidentally in this test case, JTPA and WIA method 1 produce nearly identical levels of failure.
49
Figure 5.
The demonstration above shows that the WIA approach (setting D to the average outcome) leads to
targets that are equivalent to �̂� = �̂� + 𝑋�̂� (note that failure was only triggered if actual outcomes were
more than 80% below the targets). The JTPA approach was functionally equivalent to �̂� = �̂� + 𝑋�̂� but
with �̂� replaced by an arbitrarily specified D.
50
Appendix C. Detailed State Results
Alabama
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.758 0.708 $5,065.50 0.856
Adult Actual 0.704 0.720 $5,068.30 0.594
Adult% 92.9% 101.7% 100.1% 69.34% 91.0%
DW Target 0.843 0.773 $6,250.00 0.866
DW Actual 0.712 0.747 $6,341.40 0.517
DW% 84.5% 96.7% 101.5% 59.72% 85.6%
Youth Target 0.638 0.608
0.615
Youth Actual 0.601 0.569
0.498
Youth% 94.3% 93.6%
80.92% 89.6%
WP Target 0.639 0.523 $5,814.10
WP Actual 0.634 0.640 $4,325.30
WP% 99.2% 122.3% 74.4%
98.7%
Avg % of Target by Measure 92.7% 103.6% 92.0% 70.0% 90.4%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.782 0.729 $5,080.60 0.839
Adult Actual 0.722 0.708 $4,915.80 0.594
Adult% 92.3% 97.1% 96.8% 70.77% 89.2%
DW Target 0.849 0.776 $6,233.70 0.831
DW Actual 0.743 0.739 $6,232.50 0.470
DW% 87.5% 95.3% 100.0% 56.59% 84.9%
Youth Target 0.674 0.662
0.606
Youth Actual 0.643 0.616
0.485
Youth% 95.4% 93.0%
79.98% 89.5%
WP Target 0.650 0.534 $5,977.80
WP Actual 0.629 0.642 $4,446.70
WP% 96.8% 120.2% 74.4%
97.2%
Avg % of Target by Measure 93.0% 101.4% 90.4% 69.1% 89.3%
51
Alaska
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.730 0.699 $4,823.30 0.958
Adult Actual 0.821 0.764 $7,546.50 0.725
Adult% 112.6% 109.3% 156.5% 75.67% 113.5%
DW Target 0.766 0.656 $6,419.50 0.797
DW Actual 0.767 0.741 $9,024.00 0.735
DW% 100.0% 112.9% 140.6% 92.24% 111.4%
Youth Target 0.451 0.523
0.472
Youth Actual 0.535 0.522
0.572
Youth% 118.5% 99.7%
121.06% 113.1%
WP Target 0.640 0.788 $5,328.90
WP Actual 0.600 0.590 $6,779.70
WP% 93.8% 74.9% 127.2%
98.6%
Avg % of Target by Measure 106.2% 99.2% 141.4% 96.3% 110.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.737 0.690 $5,038.00 1.070
Adult Actual 0.803 0.768 $8,130.00 0.815
Adult% 109.0% 111.4% 161.4% 76.10% 114.5%
DW Target 0.793 0.657 $7,357.40 0.766
DW Actual 0.824 0.832 $11,523.00 0.762
DW% 104.0% 126.7% 156.6% 99.47% 121.7%
Youth Target 0.407 0.505
0.408
Youth Actual 0.615 0.573
0.589
Youth% 151.0% 113.5%
144.36% 136.3%
WP Target 0.647 0.807 $5,480.10
WP Actual 0.622 0.612 $7,094.40
WP% 96.1% 75.9% 129.5%
100.5%
Avg % of Target by Measure 115.0% 106.9% 149.2% 106.6% 118.8%
52
Arizona
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.726 0.658 $5,501.30 0.533
Adult Actual 0.741 0.655 $5,301.70 0.782
Adult% 102.1% 99.6% 96.4% 146.68% 111.2%
DW Target 0.797 0.739 $6,822.40 0.627
DW Actual 0.778 0.695 $6,296.40 0.797
DW% 97.7% 94.1% 92.3% 127.20% 102.8%
Youth Target 0.676 0.661
0.687
Youth Actual 0.643 0.659
0.636
Youth% 95.2% 99.8%
92.53% 95.9%
WP Target 0.649 0.658 $4,246.40
WP Actual 0.579 0.567 $4,193.80
WP% 89.1% 86.2% 98.8%
91.3%
Avg % of Target by Measure 96.0% 94.9% 95.8% 122.1% 101.3%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.752 0.693 $5,805.90 0.499
Adult Actual 0.759 0.669 $5,600.00 0.767
Adult% 101.0% 96.4% 96.5% 153.86% 111.9%
DW Target 0.801 0.743 $6,832.50 0.580
DW Actual 0.785 0.678 $6,308.70 0.789
DW% 98.0% 91.3% 92.3% 135.93% 104.4%
Youth Target 0.687 0.673
0.700
Youth Actual 0.673 0.642
0.630
Youth% 97.9% 95.5%
89.96% 94.5%
WP Target 0.641 0.664 $4,506.40
WP Actual 0.559 0.564 $4,466.10
WP% 87.3% 84.9% 99.1%
90.4%
Avg % of Target by Measure 96.0% 92.0% 96.0% 126.6% 101.5%
53
Arkansas
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.812 0.850 $5,633.30 0.993
Adult Actual 0.879 0.849 $5,905.10 0.791
Adult% 108.2% 99.9% 104.8% 79.65% 98.2%
DW Target 0.808 0.827 $6,575.60 1.004
DW Actual 0.872 0.846 $5,921.40 0.744
DW% 107.9% 102.3% 90.1% 74.42% 93.7%
Youth Target 0.580 0.643
0.560
Youth Actual 0.839 0.806
0.812
Youth% 144.6% 125.3%
145.18% 138.4%
WP Target 0.670 0.572 $6,071.00
WP Actual 0.643 0.661 $4,456.50
WP% 96.0% 115.6% 73.4%
95.0%
Avg % of Target by Measure 114.2% 110.8% 89.4% 99.8% 104.9%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.821 0.846 $5,896.00 0.968
Adult Actual 0.870 0.851 $6,249.10 0.811
Adult% 106.0% 100.5% 106.0% 83.81% 99.1%
DW Target 0.813 0.820 $6,582.50 1.004
DW Actual 0.898 0.864 $6,897.50 0.785
DW% 110.4% 105.5% 104.8% 78.45% 99.8%
Youth Target 0.606 0.626
0.614
Youth Actual 0.826 0.807
0.847
Youth% 136.4% 128.9%
138.14% 134.5%
WP Target 0.669 0.564 $6,114.00
WP Actual 0.654 0.656 $4,588.00
WP% 97.8% 116.3% 75.0%
96.4%
Avg % of Target by Measure 112.6% 112.8% 95.3% 100.1% 106.3%
54
California
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.592 0.520 $4,320.80 0.289
Adult Actual 0.612 0.582 $4,713.00 0.569
Adult% 103.3% 111.8% 109.1% 196.60% 130.2%
DW Target 0.630 0.633 $6,869.50 0.375
DW Actual 0.687 0.669 $6,644.10 0.627
DW% 109.0% 105.8% 96.7% 167.28% 119.7%
Youth Target 0.743 0.549
0.577
Youth Actual 0.689 0.710
0.616
Youth% 92.8% 129.2%
106.71% 109.6%
WP Target 0.569 0.620 $5,120.50
WP Actual 0.534 0.560 $5,442.60
WP% 93.9% 90.5% 106.3%
96.9%
Avg % of Target by Measure 99.8% 109.3% 104.0% 156.9% 115.8%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.646 0.544 $5,136.00 0.303
Adult Actual 0.655 0.644 $4,914.00 0.561
Adult% 101.4% 118.5% 95.7% 185.42% 125.3%
DW Target 0.664 0.647 $7,890.50 0.404
DW Actual 0.708 0.710 $6,934.40 0.609
DW% 106.6% 109.9% 87.9% 150.85% 113.8%
Youth Target 0.727 0.536
0.564
Youth Actual 0.657 0.623
0.651
Youth% 90.3% 116.2%
115.28% 107.3%
WP Target 0.624 0.664 $5,075.30
WP Actual 0.585 0.600 $5,387.60
WP% 93.7% 90.4% 106.2%
96.8%
Avg % of Target by Measure 98.0% 108.7% 96.6% 150.5% 112.1%
55
Colorado
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.765 0.677 $7,531.40 0.398
Adult Actual 0.778 0.708 $6,760.10 0.701
Adult% 101.8% 104.6% 89.8% 175.96% 118.0%
DW Target 0.813 0.754 $8,662.10 0.440
DW Actual 0.787 0.710 $8,129.60 0.702
DW% 96.7% 94.1% 93.9% 159.68% 111.1%
Youth Target 0.754 0.702
0.809
Youth Actual 0.685 0.540
0.681
Youth% 90.8% 77.0%
84.18% 84.0%
WP Target 0.591 0.585 $4,640.20
WP Actual 0.530 0.495 $5,140.10
WP% 89.8% 84.6% 110.8%
95.1%
Avg % of Target by Measure 94.8% 90.1% 98.1% 139.9% 103.9%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.783 0.699 $7,730.90 0.443
Adult Actual 0.795 0.714 $7,599.30 0.429
Adult% 101.6% 102.1% 98.3% 96.77% 99.7%
DW Target 0.805 0.744 $8,854.60 0.444
DW Actual 0.807 0.734 $8,697.50 0.389
DW% 100.3% 98.6% 98.2% 87.54% 96.2%
Youth Target 0.769 0.718
0.778
Youth Actual 0.686 0.566
0.696
Youth% 89.3% 78.8%
89.52% 85.9%
WP Target 0.596 0.600 $4,728.70
WP Actual 0.559 0.525 $5,378.70
WP% 93.7% 87.4% 113.8%
98.3%
Avg % of Target by Measure 96.2% 91.7% 103.4% 91.3% 95.3%
56
Connecticut
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.738 0.700 $4,804.10 0.461
Adult Actual 0.688 0.671 $4,599.60 0.782
Adult% 93.2% 95.9% 95.7% 169.59% 113.6%
DW Target 0.791 0.838 $7,292.70 0.456
DW Actual 0.769 0.757 $7,000.00 0.776
DW% 97.2% 90.4% 96.0% 170.08% 113.4%
Youth Target 0.796 0.651
0.771
Youth Actual 0.748 0.731
0.846
Youth% 94.0% 112.3%
109.67% 105.3%
WP Target 0.601 0.503 $5,655.70
WP Actual 0.537 0.569 $5,347.90
WP% 89.4% 113.0% 94.6%
99.0%
Avg % of Target by Measure 93.5% 102.9% 95.4% 149.8% 109.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.725 0.697 $4,716.80 0.465
Adult Actual 0.714 0.664 $4,570.20 0.772
Adult% 98.4% 95.3% 96.9% 165.98% 114.1%
DW Target 0.780 0.819 $6,966.60 0.439
DW Actual 0.790 0.779 $6,960.50 0.819
DW% 101.3% 95.0% 99.9% 186.57% 120.7%
Youth Target 0.797 0.639
0.783
Youth Actual 0.784 0.754
0.883
Youth% 98.5% 118.0%
112.84% 109.8%
WP Target 0.619 0.505 $5,668.80
WP Actual 0.612 0.620 $5,651.70
WP% 99.0% 122.8% 99.7%
107.2%
Avg % of Target by Measure 99.3% 107.8% 98.8% 155.1% 114.1%
57
Delaware
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.801 0.779 $4,694.90 0.423
Adult Actual 0.797 0.775 $4,551.80 0.593
Adult% 99.5% 99.4% 97.0% 140.16% 109.0%
DW Target 0.876 0.852 $7,248.60 0.440
DW Actual 0.760 0.754 $6,019.60 0.577
DW% 86.8% 88.5% 83.1% 131.31% 97.4%
Youth Target 0.709 0.563
0.721
Youth Actual 0.738 0.579
0.889
Youth% 104.1% 102.9%
123.41% 110.2%
WP Target 0.628 0.533 $4,716.20
WP Actual 0.591 0.616 $4,480.50
WP% 94.1% 115.6% 95.0%
101.5%
Avg % of Target by Measure 96.1% 101.6% 91.7% 131.6% 104.9%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.806 0.794 $5,137.30 0.411
Adult Actual 0.784 0.818 $4,876.20 0.635
Adult% 97.2% 103.0% 94.9% 154.44% 112.4%
DW Target 0.827 0.805 $7,009.00 0.393
DW Actual 0.813 0.807 $6,041.70 0.495
DW% 98.4% 100.2% 86.2% 126.09% 102.7%
Youth Target 0.710 0.546
0.762
Youth Actual 0.664 0.650
0.848
Youth% 93.6% 119.1%
111.28% 108.0%
WP Target 0.644 0.532 $4,940.20
WP Actual 0.610 0.608 $4,619.60
WP% 94.7% 114.1% 93.5%
100.8%
Avg % of Target by Measure 96.0% 109.1% 91.5% 130.6% 106.4%
58
District of Columbia
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.725 0.652 $8,468.60 0.100
Adult Actual 0.589 0.585 $4,494.00 0.614
Adult% 81.2% 89.6% 53.1% 613.78% 209.4%
DW Target 0.991 0.645 $12,591.00 0.100
DW Actual 0.646 0.684 $6,686.00 0.626
DW% 65.2% 106.1% 53.1% 625.90% 212.6%
Youth Target 0.320 0.100
0.203
Youth Actual 0.644 0.394
0.336
Youth% 201.5% 393.7%
165.38% 253.5%
WP Target 0.636 0.562 $6,214.90
WP Actual 0.510 0.542 $4,867.30
WP% 80.2% 96.5% 78.3%
85.0%
Avg % of Target by Measure 107.0% 171.5% 61.5% 468.4% 196.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.750 0.666 $8,941.90 0.100
Adult Actual 0.607 0.625 $5,466.00 0.808
Adult% 81.0% 93.9% 61.1% 807.69% 260.9%
DW Target 0.989 0.627 $12,945.00 0.100
DW Actual 0.626 0.650 $7,522.00 0.667
DW% 63.3% 103.6% 58.1% 666.67% 222.9%
Youth Target 0.353 0.100
0.211
Youth Actual 0.398 0.338
0.333
Youth% 112.8% 338.4%
157.96% 203.0%
WP Target 0.640 0.582 $6,262.70
WP Actual 0.543 0.559 $5,183.60
WP% 84.9% 96.1% 82.8%
87.9%
Avg % of Target by Measure 85.5% 158.0% 67.3% 544.1% 203.7%
59
Florida
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.753 0.727 $6,213.30 0.466
Adult Actual 0.837 0.777 $7,670.00 0.874
Adult% 111.2% 106.8% 123.5% 187.36% 132.2%
DW Target 0.764 0.712 $5,732.60 0.525
DW Actual 0.835 0.780 $6,583.50 0.767
DW% 109.4% 109.5% 114.8% 146.21% 120.0%
Youth Target 0.662 0.639
0.699
Youth Actual 0.571 0.556
0.720
Youth% 86.2% 87.0%
102.95% 92.0%
WP Target 0.600 0.684 $3,576.50
WP Actual 0.609 0.595 $4,670.30
WP% 101.4% 87.1% 130.6%
106.4%
Avg % of Target by Measure 102.0% 97.6% 123.0% 145.5% 114.8%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.770 0.733 $6,277.60 0.476
Adult Actual 0.838 0.791 $7,842.00 0.863
Adult% 108.8% 107.9% 124.9% 181.31% 130.7%
DW Target 0.777 0.718 $5,719.10 0.543
DW Actual 0.806 0.750 $6,654.00 0.740
DW% 103.7% 104.5% 116.4% 136.38% 115.2%
Youth Target 0.691 0.668
0.691
Youth Actual 0.595 0.589
0.800
Youth% 86.1% 88.2%
115.67% 96.7%
WP Target 0.631 0.716 $3,729.30
WP Actual 0.658 0.637 $4,876.80
WP% 104.3% 88.9% 130.8%
108.0%
Avg % of Target by Measure 100.7% 97.4% 124.0% 144.5% 114.7%
60
Georgia
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.741 0.724 $4,697.20 0.860
Adult Actual 0.671 0.683 $5,164.70 0.673
Adult% 90.5% 94.4% 110.0% 78.20% 93.3%
DW Target 0.862 0.795 $6,092.20 0.875
DW Actual 0.735 0.752 $6,529.90 0.735
DW% 85.3% 94.6% 107.2% 84.02% 92.8%
Youth Target 0.611 0.600
0.686
Youth Actual 0.673 0.653
0.733
Youth% 110.1% 108.9%
106.99% 108.7%
WP Target 0.626 0.477 $4,509.60
WP Actual 0.586 0.573 $4,146.40
WP% 93.7% 120.2% 91.9%
101.9%
Avg % of Target by Measure 94.9% 104.5% 103.0% 89.7% 98.6%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.777 0.741 $4,951.90 0.837
Adult Actual 0.735 0.717 $5,045.80 0.681
Adult% 94.5% 96.8% 101.9% 81.34% 93.6%
DW Target 0.860 0.797 $5,983.80 0.864
DW Actual 0.779 0.771 $6,370.60 0.686
DW% 90.5% 96.8% 106.5% 79.39% 93.3%
Youth Target 0.624 0.607
0.665
Youth Actual 0.674 0.630
0.709
Youth% 108.0% 103.9%
106.60% 106.2%
WP Target 0.654 0.503 $4,594.30
WP Actual 0.647 0.651 $4,241.80
WP% 98.9% 129.4% 92.3%
106.9%
Avg % of Target by Measure 98.0% 106.7% 100.2% 89.1% 99.2%
61
Hawaii
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.620 0.950 $5,421.80 0.440
Adult Actual 0.583 0.587 $4,696.40 0.688
Adult% 94.2% 61.8% 86.6% 156.15% 99.7%
DW Target 0.749 0.817 $6,405.40 0.458
DW Actual 0.739 0.713 $6,051.70 0.809
DW% 98.7% 87.3% 94.5% 176.60% 114.3%
Youth Target 0.751 0.750
0.610
Youth Actual 0.546 0.430
0.511
Youth% 72.7% 57.3%
83.83% 71.3%
WP Target 0.334 0.793 $4,089.60
WP Actual 0.473 0.471 $4,977.30
WP% 141.6% 59.4% 121.7%
107.6%
Avg % of Target by Measure 101.8% 66.5% 100.9% 138.9% 100.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.748 0.961 $5,478.50 0.434
Adult Actual 0.645 0.626 $4,627.40 0.641
Adult% 86.2% 65.1% 84.5% 147.77% 95.9%
DW Target 0.731 0.803 $6,837.80 0.481
DW Actual 0.694 0.690 $5,981.50 0.645
DW% 95.0% 85.9% 87.5% 134.19% 100.6%
Youth Target 0.807 0.839
0.558
Youth Actual 0.538 0.557
0.539
Youth% 66.6% 66.4%
96.55% 76.5%
WP Target 0.342 0.816 $4,222.30
WP Actual 0.510 0.515 $5,100.40
WP% 149.0% 63.1% 120.8%
111.0%
Avg % of Target by Measure 99.2% 70.1% 97.6% 126.2% 97.1%
62
Idaho
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.743 0.681 $5,607.30 0.772
Adult Actual 0.830 0.800 $5,253.10 0.728
Adult% 111.7% 117.6% 93.7% 94.30% 104.3%
DW Target 0.792 0.783 $7,958.50 0.729
DW Actual 0.865 0.854 $6,997.50 0.824
DW% 109.3% 109.0% 87.9% 112.98% 104.8%
Youth Target 0.687 0.686
0.716
Youth Actual 0.816 0.783
0.811
Youth% 118.8% 114.2%
113.35% 115.5%
WP Target 0.633 0.660 $5,099.20
WP Actual 0.713 0.684 $4,995.10
WP% 112.7% 103.7% 98.0%
104.8%
Avg % of Target by Measure 113.1% 111.1% 93.2% 106.9% 106.7%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.799 0.690 $5,830.70 0.746
Adult Actual 0.828 0.793 $5,011.50 0.663
Adult% 103.6% 114.9% 86.0% 88.96% 98.4%
DW Target 0.780 0.737 $7,688.70 0.770
DW Actual 0.849 0.829 $6,447.00 0.763
DW% 108.9% 112.6% 83.9% 99.08% 101.1%
Youth Target 0.717 0.737
0.707
Youth Actual 0.790 0.805
0.709
Youth% 110.2% 109.3%
100.20% 106.6%
WP Target 0.635 0.690 $5,169.50
WP Actual 0.726 0.717 $5,282.80
WP% 114.4% 103.9% 102.2%
106.8%
Avg % of Target by Measure 109.3% 110.2% 90.7% 96.1% 102.4%
63
Illinois
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.772 0.763 $4,974.40 0.519
Adult Actual 0.733 0.706 $5,331.30 0.648
Adult% 94.9% 92.5% 107.2% 124.79% 104.9%
DW Target 0.787 0.808 $6,933.40 0.585
DW Actual 0.775 0.764 $7,545.60 0.651
DW% 98.4% 94.6% 108.8% 111.31% 103.3%
Youth Target 0.717 0.596
0.650
Youth Actual 0.704 0.586
0.642
Youth% 98.2% 98.3%
98.71% 98.4%
WP Target 0.721 0.579 $5,520.70
WP Actual 0.568 0.598 $5,009.70
WP% 78.8% 103.4% 90.7%
91.0%
Avg % of Target by Measure 92.6% 97.2% 102.3% 111.6% 100.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.776 0.756 $5,078.30 0.490
Adult Actual 0.731 0.723 $5,365.50 0.643
Adult% 94.2% 95.6% 105.7% 131.13% 106.7%
DW Target 0.790 0.810 $7,018.80 0.548
DW Actual 0.759 0.748 $7,318.30 0.655
DW% 96.1% 92.4% 104.3% 119.47% 103.1%
Youth Target 0.712 0.540
0.650
Youth Actual 0.678 0.560
0.663
Youth% 95.3% 103.8%
102.03% 100.4%
WP Target 0.733 0.580 $5,892.50
WP Actual 0.637 0.641 $6,154.10
WP% 86.9% 110.6% 104.4%
100.6%
Avg % of Target by Measure 93.1% 100.6% 104.8% 117.5% 103.3%
64
Indiana
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.658 0.660 $4,757.90 0.676
Adult Actual 0.693 0.703 $5,054.00 0.412
Adult% 105.4% 106.6% 106.2% 60.93% 94.8%
DW Target 0.669 0.751 $5,527.70 0.690
DW Actual 0.703 0.726 $6,022.60 0.386
DW% 105.2% 96.6% 109.0% 55.89% 91.7%
Youth Target 0.640 0.619
0.530
Youth Actual 0.674 0.620
0.622
Youth% 105.3% 100.2%
117.36% 107.6%
WP Target 0.569 0.466 $6,098.20
WP Actual 0.625 0.630 $4,576.90
WP% 109.9% 135.3% 75.1%
106.7%
Avg % of Target by Measure 106.4% 109.7% 96.7% 78.1% 99.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.693 0.679 $5,091.00 0.639
Adult Actual 0.721 0.723 $5,055.80 0.462
Adult% 104.1% 106.5% 99.3% 72.30% 95.5%
DW Target 0.686 0.756 $5,519.20 0.625
DW Actual 0.741 0.745 $5,873.00 0.453
DW% 108.1% 98.5% 106.4% 72.41% 96.3%
Youth Target 0.650 0.632
0.552
Youth Actual 0.707 0.689
0.628
Youth% 108.7% 109.0%
113.75% 110.5%
WP Target 0.584 0.476 $6,360.20
WP Actual 0.686 0.677 $5,124.50
WP% 117.4% 142.1% 80.6%
113.4%
Avg % of Target by Measure 109.6% 114.0% 95.4% 86.2% 102.6%
65
Iowa
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.675 0.693 $4,631.30 0.676
Adult Actual 0.675 0.664 $4,482.40 0.657
Adult% 99.9% 95.9% 96.8% 97.19% 97.4%
DW Target 0.805 0.769 $6,117.30 0.765
DW Actual 0.857 0.703 $6,668.00 0.698
DW% 106.5% 91.4% 109.0% 91.14% 99.5%
Youth Target 0.695 0.624
0.585
Youth Actual 0.731 0.662
0.636
Youth% 105.1% 106.1%
108.74% 106.6%
WP Target 0.549 0.471 $5,042.70
WP Actual 0.699 0.700 $5,155.90
WP% 127.3% 148.7% 102.2%
126.1%
Avg % of Target by Measure 109.7% 110.5% 102.7% 99.0% 106.4%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.684 0.689 $4,198.10 0.754
Adult Actual 0.643 0.660 $4,110.80 0.707
Adult% 94.1% 95.7% 97.9% 93.77% 95.4%
DW Target 0.632 0.730 $5,100.80 0.717
DW Actual 0.650 0.672 $5,030.50 0.776
DW% 102.9% 92.1% 98.6% 108.29% 100.5%
Youth Target 0.717 0.609
0.600
Youth Actual 0.718 0.670
0.659
Youth% 100.1% 109.9%
109.82% 106.6%
WP Target 0.559 0.483 $5,184.90
WP Actual 0.719 0.723 $5,494.70
WP% 128.7% 149.7% 106.0%
128.1%
Avg % of Target by Measure 106.5% 111.9% 100.8% 104.0% 106.7%
66
Kansas
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.734 0.770 $6,416.70 0.638
Adult Actual 0.764 0.756 $5,815.30 0.788
Adult% 104.1% 98.2% 90.6% 123.51% 104.1%
DW Target 0.798 0.834 $7,511.70 0.699
DW Actual 0.798 0.798 $7,738.60 0.766
DW% 100.1% 95.7% 103.0% 109.53% 102.1%
Youth Target 0.660 0.654
0.604
Youth Actual 0.712 0.664
0.664
Youth% 108.0% 101.6%
109.83% 106.5%
WP Target 0.682 0.593 $6,770.70
WP Actual 0.657 0.685 $5,327.10
WP% 96.2% 115.4% 78.7%
96.8%
Avg % of Target by Measure 102.1% 102.7% 90.8% 114.3% 102.4%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.788 0.804 $6,908.90 0.624
Adult Actual 0.802 0.803 $6,200.40 0.725
Adult% 101.9% 99.9% 89.8% 116.32% 102.0%
DW Target 0.831 0.850 $7,629.50 0.686
DW Actual 0.823 0.815 $7,941.20 0.759
DW% 99.1% 95.8% 104.1% 110.55% 102.4%
Youth Target 0.662 0.675
0.606
Youth Actual 0.756 0.693
0.747
Youth% 114.2% 102.7%
123.33% 113.4%
WP Target 0.678 0.574 $6,813.50
WP Actual 0.660 0.662 $5,610.80
WP% 97.3% 115.4% 82.4%
98.4%
Avg % of Target by Measure 103.1% 103.4% 92.1% 116.7% 103.9%
67
Kentucky
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.753 0.742 $4,999.30 0.705
Adult Actual 0.781 0.657 $4,942.90 0.483
Adult% 103.8% 88.5% 98.9% 68.49% 89.9%
DW Target 0.785 0.788 $6,255.60 0.795
DW Actual 0.825 0.766 $6,540.00 0.571
DW% 105.1% 97.2% 104.6% 71.84% 94.7%
Youth Target 0.702 0.674
0.701
Youth Actual 0.741 0.592
0.687
Youth% 105.5% 87.8%
97.91% 97.1%
WP Target 0.610 0.552 $5,451.90
WP Actual 0.615 0.614 $4,869.60
WP% 100.8% 111.2% 89.3%
100.5%
Avg % of Target by Measure 103.8% 96.2% 97.6% 79.4% 94.9%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.757 0.734 $5,031.30 0.653
Adult Actual 0.817 0.729 $5,024.50 0.459
Adult% 107.9% 99.3% 99.9% 70.38% 94.4%
DW Target 0.818 0.790 $6,419.50 0.776
DW Actual 0.826 0.786 $6,745.70 0.580
DW% 100.9% 99.5% 105.1% 74.67% 95.1%
Youth Target 0.723 0.653
0.709
Youth Actual 0.708 0.598
0.682
Youth% 98.1% 91.5%
96.31% 95.3%
WP Target 0.628 0.560 $5,635.00
WP Actual 0.643 0.634 $4,955.50
WP% 102.5% 113.3% 87.9%
101.2%
Avg % of Target by Measure 102.3% 100.9% 97.6% 80.5% 95.9%
68
Louisiana
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.579 0.600 $3,939.20 0.862
Adult Actual 0.641 0.643 $4,497.00 0.639
Adult% 110.7% 107.1% 114.2% 74.15% 101.5%
DW Target 0.600 0.642 $5,569.00 0.784
DW Actual 0.688 0.691 $6,252.00 0.533
DW% 114.7% 107.7% 112.3% 67.96% 100.7%
Youth Target 0.619 0.664
0.657
Youth Actual 0.661 0.639
0.609
Youth% 106.7% 96.2%
92.71% 98.6%
WP Target 0.655 0.758 $4,913.10
WP Actual 0.602 0.597 $4,859.30
WP% 92.0% 78.8% 98.9%
89.9%
Avg % of Target by Measure 106.0% 97.5% 108.4% 78.3% 97.6%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.599 0.627 $4,170.00 0.893
Adult Actual 0.654 0.661 $4,695.00 0.625
Adult% 109.2% 105.4% 112.6% 70.01% 99.3%
DW Target 0.622 0.642 $5,599.60 0.757
DW Actual 0.703 0.702 $5,408.00 0.600
DW% 112.9% 109.4% 96.6% 79.21% 99.5%
Youth Target 0.636 0.668
0.642
Youth Actual 0.689 0.678
0.627
Youth% 108.4% 101.5%
97.55% 102.5%
WP Target 0.659 0.779 $4,947.30
WP Actual 0.603 0.615 $4,740.80
WP% 91.5% 78.9% 95.8%
88.8%
Avg % of Target by Measure 105.5% 98.8% 101.7% 82.3% 97.3%
69
Maine
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.740 0.754 $4,693.80 0.716
Adult Actual 0.783 0.769 $5,105.30 0.706
Adult% 105.9% 102.0% 108.8% 98.55% 103.8%
DW Target 0.765 0.829 $5,813.50 0.746
DW Actual 0.818 0.814 $6,482.00 0.684
DW% 106.9% 98.2% 111.5% 91.74% 102.1%
Youth Target 0.738 0.706
0.715
Youth Actual 0.663 0.699
0.734
Youth% 89.9% 99.0%
102.72% 97.2%
WP Target 0.490 0.499 $4,709.90
WP Actual 0.602 0.597 $4,728.40
WP% 122.8% 119.6% 100.4%
114.3%
Avg % of Target by Measure 106.4% 104.7% 106.9% 97.7% 104.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.743 0.774 $4,597.80 0.700
Adult Actual 0.741 0.730 $5,051.00 0.689
Adult% 99.8% 94.4% 109.9% 98.31% 100.6%
DW Target 0.744 0.831 $5,838.20 0.723
DW Actual 0.803 0.804 $6,487.80 0.713
DW% 107.8% 96.7% 111.1% 98.62% 103.6%
Youth Target 0.771 0.703
0.732
Youth Actual 0.678 0.620
0.709
Youth% 87.9% 88.2%
96.85% 91.0%
WP Target 0.498 0.492 $4,814.00
WP Actual 0.620 0.606 $4,867.50
WP% 124.5% 123.1% 101.1%
116.2%
Avg % of Target by Measure 105.0% 100.6% 107.4% 97.9% 102.8%
70
Maryland
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.771 0.767 $5,963.80 0.383
Adult Actual 0.791 0.758 $6,023.00 0.548
Adult% 102.6% 98.9% 101.0% 142.83% 111.3%
DW Target 0.895 0.784 $7,711.90 0.465
DW Actual 0.831 0.801 $7,322.00 0.593
DW% 92.9% 102.2% 94.9% 127.69% 104.4%
Youth Target 0.664 0.560
0.624
Youth Actual 0.760 0.629
0.774
Youth% 114.4% 112.3%
124.17% 117.0%
WP Target 0.663 0.636 $5,016.80
WP Actual 0.581 0.604 $4,921.00
WP% 87.6% 95.1% 98.1%
93.6%
Avg % of Target by Measure 99.4% 102.1% 98.0% 131.6% 107.2%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.779 0.747 $6,443.90 0.415
Adult Actual 0.800 0.766 $6,547.00 0.587
Adult% 102.7% 102.6% 101.6% 141.61% 112.1%
DW Target 0.888 0.789 $8,476.30 0.431
DW Actual 0.833 0.785 $8,238.00 0.571
DW% 93.8% 99.4% 97.2% 132.51% 105.7%
Youth Target 0.651 0.568
0.648
Youth Actual 0.763 0.630
0.757
Youth% 117.1% 110.9%
116.80% 115.0%
WP Target 0.686 0.650 $5,195.20
WP Actual 0.618 0.630 $5,049.80
WP% 90.2% 96.8% 97.2%
94.7%
Avg % of Target by Measure 100.9% 102.4% 98.7% 130.3% 107.5%
71
Massachusetts
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.740 0.762 $4,986.80 0.229
Adult Actual 0.763 0.718 $4,764.40 0.869
Adult% 103.2% 94.2% 95.5% 379.06% 168.0%
DW Target 0.781 0.854 $7,867.60 0.316
DW Actual 0.790 0.753 $7,084.90 0.890
DW% 101.1% 88.2% 90.1% 281.84% 140.3%
Youth Target 0.800 0.555
0.768
Youth Actual 0.790 0.673
0.715
Youth% 98.7% 121.2%
93.17% 104.4%
WP Target 0.550 0.447 $5,647.40
WP Actual 0.554 0.574 $5,126.00
WP% 100.7% 128.2% 90.8%
106.6%
Avg % of Target by Measure 100.9% 108.0% 92.1% 251.4% 134.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.737 0.756 $4,569.60 0.222
Adult Actual 0.817 0.757 $4,615.50 0.821
Adult% 110.8% 100.1% 101.0% 369.44% 170.3%
DW Target 0.762 0.834 $8,003.40 0.248
DW Actual 0.822 0.771 $6,965.90 0.676
DW% 107.9% 92.5% 87.0% 271.91% 139.8%
Youth Target 0.755 0.560
0.741
Youth Actual 0.806 0.713
0.724
Youth% 106.7% 127.4%
97.71% 110.6%
WP Target 0.578 0.470 $5,839.20
WP Actual 0.609 0.618 $5,548.40
WP% 105.4% 131.4% 95.0%
110.6%
Avg % of Target by Measure 107.7% 112.8% 94.4% 246.4% 136.6%
72
Michigan
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.783 0.741 $7,085.60 0.541
Adult Actual 0.855 0.716 $6,433.00 0.845
Adult% 109.3% 96.6% 90.8% 156.39% 113.3%
DW Target 0.818 0.825 $7,323.80 0.616
DW Actual 0.895 0.755 $7,134.90 0.856
DW% 109.4% 91.6% 97.4% 138.86% 109.3%
Youth Target 0.623 0.569
0.527
Youth Actual 0.657 0.700
0.584
Youth% 105.4% 122.9%
110.86% 113.1%
WP Target 0.615 0.463 $5,846.70
WP Actual 0.614 0.628 $4,651.20
WP% 99.9% 135.6% 79.6%
105.0%
Avg % of Target by Measure 106.0% 111.7% 89.3% 135.4% 110.4%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.773 0.732 $6,900.00 0.420
Adult Actual 0.842 0.751 $6,866.90 0.882
Adult% 109.0% 102.7% 99.5% 210.04% 130.3%
DW Target 0.797 0.825 $7,369.30 0.562
DW Actual 0.895 0.790 $7,376.20 0.875
DW% 112.3% 95.7% 100.1% 155.89% 116.0%
Youth Target 0.661 0.591
0.568
Youth Actual 0.671 0.713
0.606
Youth% 101.5% 120.7%
106.54% 109.6%
WP Target 0.700 0.518 $6,267.10
WP Actual 0.686 0.689 $5,004.40
WP% 98.1% 133.2% 79.9%
103.7%
Avg % of Target by Measure 105.2% 113.1% 93.2% 157.5% 116.1%
73
Minnesota
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.804 0.779 $5,979.00 0.554
Adult Actual 0.809 0.768 $5,292.00 0.787
Adult% 100.7% 98.6% 88.5% 142.19% 107.5%
DW Target 0.812 0.835 $8,163.10 0.607
DW Actual 0.815 0.788 $7,863.50 0.766
DW% 100.4% 94.4% 96.3% 126.06% 104.3%
Youth Target 0.651 0.599
0.532
Youth Actual 0.633 0.682
0.493
Youth% 97.2% 113.9%
92.60% 101.3%
WP Target 0.657 0.538 $6,721.30
WP Actual 0.627 0.673 $5,920.00
WP% 95.4% 125.2% 88.1%
102.9%
Avg % of Target by Measure 98.4% 108.0% 91.0% 120.3% 104.2%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.787 0.796 $6,293.50 0.511
Adult Actual 0.767 0.785 $5,385.10 0.831
Adult% 97.5% 98.6% 85.6% 162.51% 111.1%
DW Target 0.794 0.835 $8,070.20 0.600
DW Actual 0.831 0.794 $8,202.90 0.778
DW% 104.6% 95.1% 101.6% 129.68% 107.8%
Youth Target 0.692 0.590
0.608
Youth Actual 0.685 0.728
0.551
Youth% 98.9% 123.4%
90.61% 104.3%
WP Target 0.689 0.557 $7,121.10
WP Actual 0.672 0.669 $6,476.10
WP% 97.5% 120.1% 90.9%
102.9%
Avg % of Target by Measure 99.7% 109.3% 92.7% 127.6% 106.9%
74
Mississippi
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.619 0.622 $3,964.10 0.875
Adult Actual 0.683 0.663 $4,072.00 0.263
Adult% 110.3% 106.5% 102.7% 30.06% 87.4%
DW Target 0.595 0.635 $5,070.00 0.811
DW Actual 0.645 0.629 $4,548.00 0.234
DW% 108.5% 99.2% 89.7% 28.82% 81.5%
Youth Target 0.486 0.565
0.576
Youth Actual 0.733 0.685
0.783
Youth% 150.7% 121.3%
135.83% 135.9%
WP Target 0.640 0.641 $5,601.50
WP Actual 0.627 0.628 $3,799.90
WP% 98.0% 98.0% 67.8%
88.0%
Avg % of Target by Measure 116.9% 106.3% 86.8% 64.9% 96.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.670 0.693 $4,486.90 0.823
Adult Actual 0.778 0.793 $4,788.00 0.284
Adult% 116.2% 114.3% 106.7% 34.52% 92.9%
DW Target 0.621 0.668 $5,023.70 0.759
DW Actual 0.665 0.704 $4,756.50 0.337
DW% 107.1% 105.4% 94.7% 44.40% 87.9%
Youth Target 0.502 0.591
0.575
Youth Actual 0.762 0.733
0.891
Youth% 151.7% 124.0%
154.90% 143.5%
WP Target 0.646 0.645 $5,580.50
WP Actual 0.667 0.660 $3,833.00
WP% 103.3% 102.4% 68.7%
91.5%
Avg % of Target by Measure 119.6% 111.5% 90.0% 77.9% 101.9%
75
Missouri
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.631 0.638 $4,331.90 0.542
Adult Actual 0.585 0.485 $4,085.20 0.391
Adult% 92.7% 76.1% 94.3% 72.17% 83.8%
DW Target 0.625 0.660 $5,380.90 0.519
DW Actual 0.637 0.527 $4,712.70 0.447
DW% 102.0% 79.7% 87.6% 86.02% 88.8%
Youth Target 0.740 0.647
0.712
Youth Actual 0.704 0.600
0.656
Youth% 95.2% 92.8%
92.09% 93.4%
WP Target 0.591 0.515 $4,540.30
WP Actual 0.585 0.573 $4,236.90
WP% 99.1% 111.1% 93.3%
101.2%
Avg % of Target by Measure 97.2% 89.9% 91.7% 83.4% 91.2%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.662 0.689 $4,359.80 0.528
Adult Actual 0.586 0.582 $4,085.10 0.472
Adult% 88.5% 84.5% 93.7% 89.27% 89.0%
DW Target 0.674 0.699 $5,506.70 0.477
DW Actual 0.626 0.622 $4,666.60 0.479
DW% 92.9% 89.0% 84.8% 100.47% 91.8%
Youth Target 0.779 0.683
0.742
Youth Actual 0.716 0.588
0.637
Youth% 91.9% 86.0%
85.91% 87.9%
WP Target 0.609 0.550 $4,521.10
WP Actual 0.634 0.626 $4,350.30
WP% 104.1% 113.9% 96.2%
104.7%
Avg % of Target by Measure 94.3% 93.3% 91.6% 91.9% 93.1%
76
Montana
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.591 0.573 $3,964.40 0.804
Adult Actual 0.700 0.707 $4,910.00 0.591
Adult% 118.5% 123.3% 123.9% 73.47% 109.8%
DW Target 0.644 0.668 $6,016.50 0.721
DW Actual 0.707 0.649 $6,863.80 0.565
DW% 109.8% 97.3% 114.1% 78.36% 99.9%
Youth Target 0.545 0.565
0.507
Youth Actual 0.682 0.654
0.609
Youth% 125.0% 115.7%
120.03% 120.2%
WP Target 0.482 0.718 $3,599.90
WP Actual 0.684 0.673 $5,091.10
WP% 141.7% 93.7% 141.4%
125.6%
Avg % of Target by Measure 123.7% 107.5% 126.5% 90.6% 113.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.587 0.592 $4,255.80 0.760
Adult Actual 0.700 0.673 $5,375.00 0.184
Adult% 119.2% 113.7% 126.3% 24.16% 95.8%
DW Target 0.696 0.713 $6,263.70 0.629
DW Actual 0.639 0.633 $6,781.90 0.193
DW% 91.8% 88.8% 108.3% 30.60% 79.9%
Youth Target 0.537 0.586
0.527
Youth Actual 0.539 0.512
0.541
Youth% 100.4% 87.3%
102.60% 96.8%
WP Target 0.487 0.737 $3,633.60
WP Actual 0.724 0.713 $5,420.80
WP% 148.7% 96.7% 149.2%
131.6%
Avg % of Target by Measure 115.1% 96.6% 127.9% 52.5% 99.5%
77
Nebraska
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.776 0.768 $4,562.30 0.769
Adult Actual 0.745 0.731 $4,287.50 0.512
Adult% 96.0% 95.2% 94.0% 66.55% 87.9%
DW Target 0.861 0.810 $6,526.50 0.784
DW Actual 0.870 0.879 $6,857.10 0.574
DW% 101.0% 108.6% 105.1% 73.25% 97.0%
Youth Target 0.789 0.748
0.682
Youth Actual 0.743 0.755
0.652
Youth% 94.2% 100.9%
95.63% 96.9%
WP Target 0.694 0.664 $5,082.00
WP Actual 0.705 0.705 $4,913.40
WP% 101.5% 106.2% 96.7%
101.5%
Avg % of Target by Measure 98.2% 102.7% 98.6% 78.5% 95.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.800 0.774 $4,973.80 0.706
Adult Actual 0.754 0.749 $4,486.00 0.549
Adult% 94.3% 96.9% 90.2% 77.79% 89.8%
DW Target 0.874 0.813 $6,844.20 0.780
DW Actual 0.884 0.845 $6,438.50 0.624
DW% 101.2% 104.0% 94.1% 80.06% 94.8%
Youth Target 0.799 0.701
0.659
Youth Actual 0.791 0.774
0.746
Youth% 99.1% 110.6%
113.18% 107.6%
WP Target 0.711 0.671 $5,328.90
WP Actual 0.734 0.732 $5,128.80
WP% 103.3% 109.1% 96.3%
102.9%
Avg % of Target by Measure 99.5% 105.1% 93.5% 90.3% 97.9%
78
Nevada
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.544 0.283 $4,116.20 0.517
Adult Actual 0.689 0.639 $5,062.10 0.385
Adult% 126.5% 226.0% 123.0% 74.58% 137.5%
DW Target 0.604 0.432 $3,489.10 0.409
DW Actual 0.729 0.703 $6,080.20 0.482
DW% 120.6% 162.6% 174.3% 117.93% 143.8%
Youth Target 0.723 0.894
0.862
Youth Actual 0.624 0.501
0.520
Youth% 86.3% 56.0%
60.26% 67.5%
WP Target 0.341 0.837 $3,317.30
WP Actual 0.471 0.470 $4,397.80
WP% 138.4% 56.1% 132.6%
109.0%
Avg % of Target by Measure 117.9% 125.2% 143.3% 84.3% 116.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.578 0.318 $4,356.20 0.484
Adult Actual 0.724 0.707 $5,374.90 0.518
Adult% 125.2% 222.4% 123.4% 106.96% 144.5%
DW Target 0.623 0.439 $3,600.00 0.366
DW Actual 0.759 0.741 $5,922.20 0.598
DW% 121.8% 168.8% 164.5% 163.65% 154.7%
Youth Target 0.768 0.920
0.887
Youth Actual 0.642 0.214
0.651
Youth% 83.6% 23.3%
73.42% 60.1%
WP Target 0.349 0.859 $3,345.40
WP Actual 0.519 0.521 $4,563.40
WP% 148.8% 60.6% 136.4%
115.3%
Avg % of Target by Measure 119.9% 118.8% 141.4% 114.7% 121.2%
79
New Hampshire
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.780 0.767 $4,823.40 0.652
Adult Actual 0.770 0.751 $4,613.50 0.768
Adult% 98.6% 97.9% 95.7% 117.80% 102.5%
DW Target 0.844 0.842 $6,104.50 0.809
DW Actual 0.863 0.838 $6,776.80 0.718
DW% 102.2% 99.6% 111.0% 88.77% 100.4%
Youth Target 0.668 0.631
0.564
Youth Actual 0.533 0.528
0.584
Youth% 79.8% 83.7%
103.54% 89.0%
WP Target 0.632 0.582 $5,385.40
WP Actual 0.589 0.584 $5,404.80
WP% 93.2% 100.3% 100.4%
97.9%
Avg % of Target by Measure 93.4% 95.4% 102.3% 103.4% 98.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.778 0.762 $5,282.90 0.637
Adult Actual 0.699 0.699 $4,870.00 0.581
Adult% 89.8% 91.7% 92.2% 91.23% 91.2%
DW Target 0.806 0.809 $6,065.50 0.782
DW Actual 0.792 0.789 $7,074.00 0.668
DW% 98.2% 97.4% 116.6% 85.36% 99.4%
Youth Target 0.738 0.664
0.629
Youth Actual 0.644 0.627
0.665
Youth% 87.2% 94.4%
105.69% 95.8%
WP Target 0.640 0.573 $5,468.10
WP Actual 0.604 0.598 $5,699.30
WP% 94.5% 104.5% 104.2%
101.1%
Avg % of Target by Measure 92.4% 97.0% 104.4% 94.1% 96.9%
80
New Jersey
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.747 0.686 $4,194.70 0.620
Adult Actual 0.739 0.674 $4,979.00 0.525
Adult% 98.9% 98.3% 118.7% 84.72% 100.2%
DW Target 0.807 0.786 $5,954.00 0.616
DW Actual 0.737 0.716 $6,695.90 0.587
DW% 91.3% 91.1% 112.5% 95.26% 97.5%
Youth Target 0.701 0.558
0.760
Youth Actual 0.658 0.525
0.709
Youth% 93.8% 94.0%
93.33% 93.7%
WP Target 0.640 0.507 $4,687.40
WP Actual 0.479 0.493 $5,195.00
WP% 74.8% 97.2% 110.8%
94.3%
Avg % of Target by Measure 89.7% 95.2% 114.0% 91.1% 97.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.774 0.711 $4,840.20 0.567
Adult Actual 0.726 0.656 $5,515.60 0.558
Adult% 93.8% 92.3% 114.0% 98.29% 99.6%
DW Target 0.818 0.804 $6,377.90 0.582
DW Actual 0.767 0.733 $6,912.60 0.636
DW% 93.7% 91.1% 108.4% 109.31% 100.6%
Youth Target 0.729 0.523
0.800
Youth Actual 0.676 0.265
0.745
Youth% 92.8% 50.6%
93.14% 78.8%
WP Target 0.649 0.508 $4,735.20
WP Actual 0.500 0.508 $5,231.90
WP% 77.0% 99.8% 110.5%
95.8%
Avg % of Target by Measure 89.3% 83.5% 110.9% 100.3% 94.8%
81
New Mexico
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.767 0.736 $9,117.20 0.606
Adult Actual 0.860 0.834 $9,844.80 0.769
Adult% 112.2% 113.4% 108.0% 126.83% 115.1%
DW Target 0.756 0.681 $8,025.60 0.644
DW Actual 0.716 0.689 $6,645.00 0.474
DW% 94.7% 101.1% 82.8% 73.59% 88.1%
Youth Target 0.589 0.633
0.716
Youth Actual 0.568 0.558
0.435
Youth% 96.6% 88.0%
60.73% 81.8%
WP Target 0.580 0.752 $4,204.80
WP Actual 0.545 0.542 $4,435.50
WP% 94.0% 72.1% 105.5%
90.5%
Avg % of Target by Measure 99.4% 93.7% 98.8% 87.1% 94.3%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.775 0.723 $8,992.40 0.644
Adult Actual 0.769 0.755 $7,940.20 0.604
Adult% 99.2% 104.5% 88.3% 93.69% 96.4%
DW Target 0.755 0.678 $8,243.30 0.660
DW Actual 0.728 0.742 $7,449.90 0.497
DW% 96.4% 109.5% 90.4% 75.27% 92.9%
Youth Target 0.566 0.639
0.689
Youth Actual 0.519 0.552
0.454
Youth% 91.7% 86.3%
65.86% 81.3%
WP Target 0.598 0.791 $4,338.50
WP Actual 0.620 0.621 $4,733.60
WP% 103.6% 78.5% 109.1%
97.1%
Avg % of Target by Measure 97.7% 94.7% 95.9% 78.3% 91.8%
82
New York
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.648 0.706 $5,066.40 0.219
Adult Actual 0.615 0.617 $4,366.00 0.338
Adult% 95.0% 87.5% 86.2% 153.82% 105.6%
DW Target 0.657 0.685 $6,858.20 0.331
DW Actual 0.545 0.577 $5,802.00 0.388
DW% 82.9% 84.2% 84.6% 117.34% 92.3%
Youth Target 0.745 0.532
0.753
Youth Actual 0.683 0.609
0.642
Youth% 91.6% 114.4%
85.29% 97.1%
WP Target 0.720 0.645 $5,434.90
WP Actual 0.567 0.576 $4,731.30
WP% 78.8% 89.4% 87.1%
85.1%
Avg % of Target by Measure 87.1% 93.9% 85.9% 118.8% 95.7%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.647 0.724 $4,847.30 0.172
Adult Actual 0.620 0.630 $4,326.00 0.339
Adult% 95.8% 87.0% 89.3% 197.50% 117.4%
DW Target 0.645 0.713 $6,811.60 0.298
DW Actual 0.537 0.587 $5,673.00 0.337
DW% 83.2% 82.2% 83.3% 113.18% 90.5%
Youth Target 0.741 0.538
0.751
Youth Actual 0.691 0.634
0.661
Youth% 93.3% 117.9%
87.93% 99.7%
WP Target 0.742 0.665 $5,555.20
WP Actual 0.616 0.614 $4,839.80
WP% 83.1% 92.4% 87.1%
87.5%
Avg % of Target by Measure 88.9% 94.9% 86.6% 132.9% 99.8%
83
North Carolina
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.737 0.720 $5,094.50 0.771
Adult Actual 0.724 0.700 $4,462.00 0.602
Adult% 98.3% 97.2% 87.6% 78.07% 90.3%
DW Target 0.817 0.775 $6,550.10 0.722
DW Actual 0.790 0.780 $6,194.00 0.658
DW% 96.6% 100.7% 94.6% 91.20% 95.8%
Youth Target 0.580 0.565
0.574
Youth Actual 0.619 0.553
0.633
Youth% 106.6% 97.8%
110.20% 104.9%
WP Target 0.643 0.540 $5,608.40
WP Actual 0.602 0.608 $4,070.20
WP% 93.6% 112.7% 72.6%
93.0%
Avg % of Target by Measure 98.8% 102.1% 84.9% 93.2% 95.3%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.737 0.726 $5,004.00 0.737
Adult Actual 0.720 0.703 $4,608.50 0.552
Adult% 97.7% 96.9% 92.1% 74.82% 90.4%
DW Target 0.827 0.779 $6,775.30 0.678
DW Actual 0.787 0.782 $6,361.50 0.574
DW% 95.2% 100.5% 93.9% 84.69% 93.6%
Youth Target 0.589 0.577
0.575
Youth Actual 0.596 0.603
0.593
Youth% 101.2% 104.5%
103.14% 103.0%
WP Target 0.672 0.568 $5,757.30
WP Actual 0.666 0.663 $4,376.50
WP% 99.1% 116.7% 76.0%
97.3%
Avg % of Target by Measure 98.3% 104.7% 87.3% 87.6% 95.2%
84
North Dakota
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.723 0.732 $4,109.30 0.952
Adult Actual 0.780 0.741 $4,772.50 0.710
Adult% 107.9% 101.3% 116.1% 74.62% 100.0%
DW Target 0.763 0.780 $7,995.40 0.993
DW Actual 0.819 0.776 $8,990.00 0.653
DW% 107.4% 99.5% 112.4% 65.71% 96.3%
Youth Target 0.679 0.665
0.632
Youth Actual 0.761 0.748
0.675
Youth% 112.1% 112.6%
106.77% 110.5%
WP Target 0.535 0.726 $4,961.90
WP Actual 0.497 0.485 $6,866.20
WP% 93.0% 66.8% 138.4%
99.4%
Avg % of Target by Measure 105.1% 95.1% 122.3% 82.4% 101.4%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.772 0.715 $4,304.80 1.117
Adult Actual 0.796 0.779 $4,736.90 0.729
Adult% 103.1% 108.9% 110.0% 65.21% 96.8%
DW Target 0.747 0.724 $7,465.20 1.098
DW Actual 0.913 0.870 $7,770.00 0.784
DW% 122.2% 120.2% 104.1% 71.37% 104.5%
Youth Target 0.694 0.670
0.640
Youth Actual 0.751 0.729
0.648
Youth% 108.2% 108.9%
101.27% 106.1%
WP Target 0.550 0.765 $4,858.70
WP Actual 0.542 0.530 $7,346.70
WP% 98.6% 69.3% 151.2%
106.4%
Avg % of Target by Measure 108.0% 101.8% 121.8% 79.3% 103.1%
85
Ohio
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.705 0.725 $5,325.30 0.523
Adult Actual 0.820 0.761 $6,171.00 0.601
Adult% 116.3% 105.0% 115.9% 114.97% 113.0%
DW Target 0.813 0.848 $7,060.80 0.611
DW Actual 0.840 0.810 $7,785.00 0.617
DW% 103.4% 95.6% 110.3% 101.10% 102.6%
Youth Target 0.627 0.568
0.596
Youth Actual 0.632 0.580
0.606
Youth% 100.8% 102.0%
101.62% 101.5%
WP Target 0.438 0.294 $5,770.60
WP Actual 0.494 0.519 $6,487.50
WP% 112.7% 176.4% 112.4%
133.8%
Avg % of Target by Measure 108.3% 119.7% 112.9% 105.9% 112.2%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.720 0.723 $5,443.10 0.504
Adult Actual 0.815 0.771 $6,139.20 0.569
Adult% 113.3% 106.5% 112.8% 112.77% 111.3%
DW Target 0.807 0.829 $6,961.70 0.595
DW Actual 0.849 0.813 $7,891.50 0.634
DW% 105.3% 98.1% 113.4% 106.46% 105.8%
Youth Target 0.690 0.623
0.623
Youth Actual 0.704 0.619
0.655
Youth% 102.0% 99.4%
105.26% 102.2%
WP Target 0.498 0.366 $6,308.10
WP Actual 0.452 0.483 $6,714.10
WP% 90.8% 132.2% 106.4%
109.8%
Avg % of Target by Measure 102.8% 109.1% 110.9% 108.2% 107.5%
86
Oklahoma
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.647 0.634 $4,863.50 0.790
Adult Actual 0.597 0.629 $4,464.30 0.591
Adult% 92.3% 99.3% 91.8% 74.84% 89.6%
DW Target 0.642 0.617 $6,289.50 0.668
DW Actual 0.628 0.676 $5,374.10 0.503
DW% 97.8% 109.7% 85.5% 75.29% 92.1%
Youth Target 0.616 0.692
0.658
Youth Actual 0.673 0.655
0.536
Youth% 109.3% 94.7%
81.43% 95.1%
WP Target 0.671 0.661 $6,003.80
WP Actual 0.606 0.625 $4,739.50
WP% 90.3% 94.6% 78.9%
87.9%
Avg % of Target by Measure 97.4% 99.6% 85.4% 77.2% 90.5%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.686 0.650 $5,295.00 0.765
Adult Actual 0.610 0.635 $4,477.00 0.616
Adult% 89.0% 97.7% 84.6% 80.57% 88.0%
DW Target 0.747 0.719 $7,151.50 0.660
DW Actual 0.747 0.750 $5,877.50 0.596
DW% 100.1% 104.4% 82.2% 90.41% 94.3%
Youth Target 0.623 0.695
0.663
Youth Actual 0.666 0.658
0.509
Youth% 106.9% 94.6%
76.73% 92.8%
WP Target 0.647 0.637 $5,955.50
WP Actual 0.602 0.606 $5,020.20
WP% 93.0% 95.1% 84.3%
90.8%
Avg % of Target by Measure 97.3% 98.0% 83.7% 82.6% 90.9%
87
Oregon
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.603 0.599 $4,751.40 0.533
Adult Actual 0.618 0.633 $4,895.40 0.290
Adult% 102.5% 105.8% 103.0% 54.37% 91.4%
DW Target 0.559 0.605 $5,930.30 0.410
DW Actual 0.625 0.637 $4,998.30 0.290
DW% 111.8% 105.4% 84.3% 70.66% 93.1%
Youth Target 0.702 0.603
0.598
Youth Actual 0.698 0.488
0.726
Youth% 99.4% 80.9%
121.41% 100.6%
WP Target 0.538 0.567 $4,836.50
WP Actual 0.596 0.601 $4,922.80
WP% 110.6% 106.1% 101.8%
106.2%
Avg % of Target by Measure 106.1% 99.5% 96.4% 82.2% 96.9%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.644 0.630 $4,877.00 0.517
Adult Actual 0.633 0.654 $5,044.00 0.425
Adult% 98.3% 103.8% 103.4% 82.22% 96.9%
DW Target 0.594 0.634 $6,228.90 0.445
DW Actual 0.638 0.657 $5,179.40 0.388
DW% 107.4% 103.6% 83.2% 87.11% 95.3%
Youth Target 0.750 0.641
0.650
Youth Actual 0.681 0.501
0.713
Youth% 90.8% 78.2%
109.75% 92.9%
WP Target 0.556 0.579 $4,953.10
WP Actual 0.621 0.617 $5,137.50
WP% 111.7% 106.5% 103.7%
107.3%
Avg % of Target by Measure 102.1% 98.0% 96.8% 93.0% 97.8%
88
Pennsylvania
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.694 0.724 $5,203.90 0.609
Adult Actual 0.726 0.721 $5,555.80 0.471
Adult% 104.6% 99.6% 106.8% 77.32% 97.1%
DW Target 0.787 0.794 $7,100.00 0.655
DW Actual 0.758 0.771 $7,036.00 0.606
DW% 96.3% 97.1% 99.1% 92.59% 96.3%
Youth Target 0.703 0.555
0.697
Youth Actual 0.637 0.522
0.826
Youth% 90.6% 93.9%
118.53% 101.0%
WP Target 0.626 0.520 $5,340.60
WP Actual 0.651 0.662 $5,375.60
WP% 103.9% 127.2% 100.7%
110.6%
Avg % of Target by Measure 98.9% 104.5% 102.2% 96.2% 100.8%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.710 0.730 $5,245.10 0.525
Adult Actual 0.737 0.733 $5,580.40 0.431
Adult% 103.8% 100.4% 106.4% 82.06% 98.2%
DW Target 0.792 0.796 $7,196.00 0.611
DW Actual 0.780 0.785 $6,931.60 0.519
DW% 98.5% 98.6% 96.3% 85.02% 94.6%
Youth Target 0.723 0.583
0.709
Youth Actual 0.628 0.527
0.835
Youth% 86.8% 90.4%
117.76% 98.3%
WP Target 0.561 0.465 $5,670.60
WP Actual 0.666 0.677 $5,792.30
WP% 118.6% 145.6% 102.2%
122.1%
Avg % of Target by Measure 101.9% 108.7% 101.6% 95.0% 102.6%
89
Rhode Island
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.662 0.657 $4,524.80 0.247
Adult Actual 0.713 0.728 $4,475.90 0.707
Adult% 107.7% 110.9% 98.9% 286.00% 150.9%
DW Target 0.699 0.734 $6,806.70 0.349
DW Actual 0.762 0.787 $6,567.70 0.771
DW% 109.1% 107.2% 96.5% 220.72% 133.4%
Youth Target 0.550 0.389
0.441
Youth Actual 0.417 0.463
0.383
Youth% 75.7% 119.2%
86.91% 93.9%
WP Target 0.546 0.523 $5,039.90
WP Actual 0.603 0.617 $4,947.30
WP% 110.3% 118.0% 98.2%
108.8%
Avg % of Target by Measure 100.7% 113.8% 97.9% 197.9% 124.7%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.701 0.675 $5,176.20 0.351
Adult Actual 0.778 0.807 $5,022.60 0.683
Adult% 111.0% 119.7% 97.0% 194.94% 130.7%
DW Target 0.743 0.761 $7,379.00 0.383
DW Actual 0.827 0.829 $6,217.50 0.712
DW% 111.3% 108.8% 84.3% 185.85% 122.6%
Youth Target 0.576 0.443
0.486
Youth Actual 0.493 0.539
0.468
Youth% 85.5% 121.6%
96.33% 101.1%
WP Target 0.590 0.557 $5,329.70
WP Actual 0.669 0.665 $5,242.70
WP% 113.4% 119.3% 98.4%
110.3%
Avg % of Target by Measure 105.3% 117.3% 93.2% 159.0% 117.4%
90
South Carolina
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.680 0.661 $4,529.40 0.539
Adult Actual 0.717 0.687 $4,307.30 0.481
Adult% 105.5% 104.1% 95.1% 89.30% 98.5%
DW Target 0.744 0.740 $5,835.40 0.562
DW Actual 0.727 0.710 $6,044.90 0.482
DW% 97.8% 95.9% 103.6% 85.79% 95.8%
Youth Target 0.609 0.606
0.617
Youth Actual 0.694 0.663
0.706
Youth% 114.0% 109.5%
114.28% 112.6%
WP Target 0.656 0.563 $5,507.90
WP Actual 0.597 0.587 $4,296.20
WP% 91.0% 104.2% 78.0%
91.1%
Avg % of Target by Measure 102.1% 103.4% 92.2% 96.5% 99.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.703 0.671 $4,585.40 0.576
Adult Actual 0.729 0.708 $4,484.40 0.518
Adult% 103.7% 105.6% 97.8% 89.83% 99.2%
DW Target 0.771 0.761 $5,899.80 0.584
DW Actual 0.770 0.751 $6,145.60 0.500
DW% 99.9% 98.7% 104.2% 85.66% 97.1%
Youth Target 0.615 0.616
0.623
Youth Actual 0.710 0.651
0.718
Youth% 115.5% 105.8%
115.23% 112.2%
WP Target 0.686 0.598 $5,734.70
WP Actual 0.634 0.632 $4,444.50
WP% 92.5% 105.8% 77.5%
91.9%
Avg % of Target by Measure 102.9% 104.0% 93.2% 96.9% 99.7%
91
South Dakota
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.712 0.712 $4,631.60 0.775
Adult Actual 0.777 0.755 $4,908.40 0.597
Adult% 109.3% 106.1% 106.0% 77.01% 99.6%
DW Target 0.826 0.791 $6,202.90 0.880
DW Actual 0.866 0.848 $6,757.80 0.675
DW% 104.9% 107.2% 108.9% 76.74% 99.5%
Youth Target 0.688 0.728
0.645
Youth Actual 0.705 0.711
0.577
Youth% 102.5% 97.7%
89.46% 96.5%
WP Target 0.600 0.647 $5,785.80
WP Actual 0.618 0.610 $5,001.70
WP% 103.0% 94.3% 86.5%
94.6%
Avg % of Target by Measure 104.9% 101.3% 100.5% 81.1% 97.2%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.713 0.710 $4,767.90 0.762
Adult Actual 0.741 0.747 $4,957.40 0.559
Adult% 103.9% 105.2% 104.0% 73.28% 96.6%
DW Target 0.775 0.776 $5,902.10 0.841
DW Actual 0.857 0.842 $6,689.80 0.612
DW% 110.6% 108.4% 113.4% 72.75% 101.3%
Youth Target 0.670 0.725
0.610
Youth Actual 0.760 0.773
0.529
Youth% 113.5% 106.5%
86.80% 102.3%
WP Target 0.614 0.643 $5,817.60
WP Actual 0.662 0.645 $5,316.00
WP% 107.9% 100.3% 91.4%
99.9%
Avg % of Target by Measure 109.0% 105.1% 102.9% 77.6% 99.3%
92
Tennessee
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.785 0.757 $5,133.20 0.727
Adult Actual 0.849 0.785 $6,803.30 0.740
Adult% 108.2% 103.7% 132.5% 101.78% 111.5%
DW Target 0.801 0.796 $5,540.60 0.838
DW Actual 0.860 0.802 $7,002.30 0.727
DW% 107.4% 100.7% 126.4% 86.69% 105.3%
Youth Target 0.691 0.664
0.683
Youth Actual 0.783 0.723
0.796
Youth% 113.2% 108.8%
116.43% 112.8%
WP Target 0.647 0.547 $5,113.30
WP Actual 0.617 0.617 $4,342.90
WP% 95.5% 112.8% 84.9%
97.7%
Avg % of Target by Measure 106.0% 106.5% 114.6% 101.6% 107.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.801 0.770 $5,025.30 0.707
Adult Actual 0.834 0.804 $6,278.40 0.728
Adult% 104.2% 104.4% 124.9% 103.06% 109.1%
DW Target 0.809 0.790 $5,420.70 0.829
DW Actual 0.849 0.789 $7,279.70 0.730
DW% 104.9% 99.8% 134.3% 88.08% 106.8%
Youth Target 0.747 0.682
0.704
Youth Actual 0.842 0.761
0.769
Youth% 112.7% 111.6%
109.21% 111.2%
WP Target 0.610 0.533 $5,024.00
WP Actual 0.613 0.627 $4,578.50
WP% 100.5% 117.6% 91.1%
103.1%
Avg % of Target by Measure 105.6% 108.4% 116.8% 100.1% 107.6%
93
Texas
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.693 0.696 $4,646.50 0.612
Adult Actual 0.698 0.700 $4,978.70 0.689
Adult% 100.8% 100.5% 107.2% 112.46% 105.2%
DW Target 0.760 0.781 $6,949.30 0.671
DW Actual 0.764 0.759 $7,364.70 0.705
DW% 100.5% 97.2% 106.0% 105.06% 102.2%
Youth Target 0.712 0.657
0.700
Youth Actual 0.671 0.615
0.593
Youth% 94.2% 93.6%
84.79% 90.9%
WP Target 0.654 0.690 $4,248.80
WP Actual 0.625 0.632 $5,024.90
WP% 95.6% 91.7% 118.3%
101.8%
Avg % of Target by Measure 97.8% 95.7% 110.5% 100.8% 100.6%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.698 0.706 $4,534.90 0.577
Adult Actual 0.695 0.688 $4,218.60 0.716
Adult% 99.5% 97.4% 93.0% 124.13% 103.5%
DW Target 0.751 0.780 $6,854.80 0.647
DW Actual 0.777 0.777 $7,129.00 0.709
DW% 103.5% 99.7% 104.0% 109.54% 104.2%
Youth Target 0.738 0.665
0.705
Youth Actual 0.706 0.625
0.615
Youth% 95.8% 93.9%
87.24% 92.3%
WP Target 0.672 0.710 $4,375.40
WP Actual 0.661 0.662 $5,325.10
WP% 98.4% 93.1% 121.7%
104.4%
Avg % of Target by Measure 99.3% 96.0% 106.3% 107.0% 101.6%
94
Utah
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.792 0.704 $5,072.90 0.431
Adult Actual 0.604 0.681 $4,092.50 0.302
Adult% 76.3% 96.7% 80.7% 70.02% 80.9%
DW Target 0.778 0.795 $6,966.00 0.518
DW Actual 0.791 0.793 $6,912.00 0.525
DW% 101.8% 99.8% 99.2% 101.34% 100.5%
Youth Target 0.670 0.671
0.623
Youth Actual 0.602 0.613
0.467
Youth% 89.8% 91.4%
74.95% 85.4%
WP Target 0.597 0.641 $5,194.30
WP Actual 0.708 0.704 $5,604.90
WP% 118.7% 109.8% 107.9%
112.1%
Avg % of Target by Measure 96.6% 99.4% 95.9% 82.1% 94.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.659 0.717 $5,577.10 0.464
Adult Actual 0.691 0.688 $5,858.00 0.364
Adult% 104.9% 96.0% 105.0% 78.50% 96.1%
DW Target 0.755 0.797 $6,988.70 0.531
DW Actual 0.775 0.783 $7,095.00 0.574
DW% 102.7% 98.2% 101.5% 108.01% 102.6%
Youth Target 0.753 0.733
0.644
Youth Actual 0.643 0.648
0.543
Youth% 85.3% 88.3%
84.25% 86.0%
WP Target 0.587 0.615 $5,177.80
WP Actual 0.532 0.527 $5,494.50
WP% 90.7% 85.7% 106.1%
94.2%
Avg % of Target by Measure 95.9% 92.1% 104.2% 90.3% 95.2%
95
Vermont
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.643 0.681 $4,016.70 0.676
Adult Actual 0.667 0.611 $4,306.80 0.603
Adult% 103.7% 89.7% 107.2% 89.14% 97.5%
DW Target 0.681 0.801 $4,962.70 0.747
DW Actual 0.908 0.769 $6,610.90 0.685
DW% 133.4% 96.1% 133.2% 91.78% 113.6%
Youth Target 0.545 0.486
0.354
Youth Actual 0.377 0.366
0.263
Youth% 69.1% 75.4%
74.28% 72.9%
WP Target 0.495 0.560 $5,359.60
WP Actual 0.622 0.646 $4,905.50
WP% 125.7% 115.4% 91.5%
110.9%
Avg % of Target by Measure 108.0% 94.2% 110.7% 85.1% 99.1%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.670 0.653 $3,699.10 0.691
Adult Actual 0.627 0.658 $4,581.60 0.608
Adult% 93.7% 100.8% 123.9% 87.99% 101.6%
DW Target 0.668 0.769 $5,673.40 0.738
DW Actual 0.741 0.790 $7,051.80 0.505
DW% 110.8% 102.8% 124.3% 68.46% 101.6%
Youth Target 0.526 0.552
0.339
Youth Actual 0.337 0.405
0.260
Youth% 64.1% 73.4%
76.88% 71.5%
WP Target 0.455 0.507 $5,644.30
WP Actual 0.535 0.540 $5,217.40
WP% 117.6% 106.5% 92.4%
105.5%
Avg % of Target by Measure 96.5% 95.9% 113.5% 77.8% 95.5%
96
Virginia
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.748 0.753 $5,570.60 0.401
Adult Actual 0.703 0.688 $4,297.60 0.695
Adult% 93.9% 91.3% 77.2% 173.41% 109.0%
DW Target 0.875 0.843 $8,589.60 0.431
DW Actual 0.769 0.763 $6,601.60 0.638
DW% 87.9% 90.5% 76.9% 148.06% 100.8%
Youth Target 0.722 0.595
0.688
Youth Actual 0.585 0.543
0.624
Youth% 80.9% 91.3%
90.83% 87.7%
WP Target 0.730 0.647 $5,707.50
WP Actual 0.665 0.675 $4,882.60
WP% 91.1% 104.4% 85.6%
93.7%
Avg % of Target by Measure 88.5% 94.4% 79.9% 137.4% 98.9%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.773 0.766 $5,725.60 0.406
Adult Actual 0.714 0.710 $4,240.30 0.724
Adult% 92.5% 92.7% 74.1% 178.27% 109.4%
DW Target 0.886 0.845 $8,532.10 0.416
DW Actual 0.727 0.731 $6,329.10 0.685
DW% 82.1% 86.6% 74.2% 164.47% 101.8%
Youth Target 0.749 0.619
0.714
Youth Actual 0.639 0.621
0.721
Youth% 85.4% 100.4%
101.05% 95.6%
WP Target 0.743 0.665 $5,857.40
WP Actual 0.699 0.702 $5,086.90
WP% 94.1% 105.6% 86.9%
95.5%
Avg % of Target by Measure 88.5% 96.3% 78.4% 147.9% 101.7%
97
Washington
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.721 0.690 $5,129.50 0.500
Adult Actual 0.746 0.733 $6,035.50 0.637
Adult% 103.5% 106.1% 117.7% 127.42% 113.7%
DW Target 0.758 0.762 $8,382.50 0.585
DW Actual 0.807 0.781 $8,138.00 0.698
DW% 106.5% 102.5% 97.1% 119.42% 106.4%
Youth Target 0.728 0.573
0.612
Youth Actual 0.674 0.619
0.733
Youth% 92.5% 108.0%
119.74% 106.7%
WP Target 0.482 0.540 $5,531.60
WP Actual 0.617 0.622 $5,233.70
WP% 127.9% 115.1% 94.6%
112.6%
Avg % of Target by Measure 107.6% 107.9% 103.1% 122.2% 110.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.749 0.712 $5,608.90 0.463
Adult Actual 0.745 0.737 $6,061.50 0.604
Adult% 99.6% 103.6% 108.1% 130.62% 110.5%
DW Target 0.767 0.765 $8,630.30 0.560
DW Actual 0.797 0.791 $8,301.10 0.656
DW% 104.0% 103.4% 96.2% 117.20% 105.2%
Youth Target 0.737 0.585
0.643
Youth Actual 0.693 0.620
0.757
Youth% 94.1% 106.1%
117.65% 106.0%
WP Target 0.495 0.546 $5,998.90
WP Actual 0.652 0.650 $5,354.80
WP% 131.7% 119.1% 89.3%
113.4%
Avg % of Target by Measure 107.3% 108.1% 97.8% 121.8% 108.7%
98
West Virginia
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.691 0.685 $4,862.90 0.993
Adult Actual 0.770 0.719 $5,062.20 0.805
Adult% 111.4% 105.0% 104.1% 81.02% 100.4%
DW Target 0.725 0.702 $8,104.50 0.918
DW Actual 0.797 0.772 $6,895.20 0.824
DW% 109.9% 110.0% 85.1% 89.79% 98.7%
Youth Target 0.558 0.528
0.720
Youth Actual 0.571 0.470
0.587
Youth% 102.3% 89.1%
81.58% 91.0%
WP Target 0.480 0.653 $4,481.80
WP Actual 0.596 0.614 $4,341.50
WP% 124.2% 94.0% 96.9%
105.0%
Avg % of Target by Measure 111.9% 99.5% 95.4% 84.1% 98.2%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.705 0.690 $5,374.10 0.949
Adult Actual 0.767 0.719 $5,226.60 0.812
Adult% 108.8% 104.2% 97.3% 85.49% 98.9%
DW Target 0.680 0.699 $7,856.40 0.871
DW Actual 0.830 0.812 $7,192.00 0.808
DW% 122.1% 116.2% 91.5% 92.77% 105.6%
Youth Target 0.606 0.567
0.723
Youth Actual 0.656 0.494
0.664
Youth% 108.2% 87.2%
91.85% 95.8%
WP Target 0.483 0.652 $4,512.20
WP Actual 0.635 0.641 $4,535.10
WP% 131.4% 98.3% 100.5%
110.1%
Avg % of Target by Measure 117.6% 101.5% 96.4% 90.0% 102.0%
99
Wisconsin
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.769 0.704 $5,154.30 0.698
Adult Actual 0.730 0.710 $4,848.20 0.514
Adult% 94.9% 100.9% 94.1% 73.75% 90.9%
DW Target 0.836 0.824 $6,618.40 0.665
DW Actual 0.822 0.809 $7,150.80 0.611
DW% 98.4% 98.2% 108.0% 91.89% 99.1%
Youth Target 0.715 0.629
0.619
Youth Actual 0.667 0.676
0.760
Youth% 93.3% 107.5%
122.65% 107.8%
WP Target 0.596 0.454 $6,325.70
WP Actual 0.700 0.690 $5,446.30
WP% 117.6% 152.1% 86.1%
118.6%
Avg % of Target by Measure 101.1% 114.7% 96.1% 96.1% 103.0%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.767 0.714 $5,325.50 0.631
Adult Actual 0.738 0.712 $4,937.80 0.607
Adult% 96.2% 99.6% 92.7% 96.31% 96.2%
DW Target 0.827 0.826 $6,810.70 0.633
DW Actual 0.800 0.785 $7,193.40 0.636
DW% 96.7% 95.0% 105.6% 100.47% 99.4%
Youth Target 0.741 0.633
0.648
Youth Actual 0.754 0.701
0.729
Youth% 101.7% 110.7%
112.57% 108.3%
WP Target 0.619 0.466 $6,519.80
WP Actual 0.736 0.736 $5,594.60
WP% 119.0% 158.1% 85.8%
121.0%
Avg % of Target by Measure 103.4% 115.8% 94.7% 103.1% 105.3%
100
Wyoming
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.695 0.539 $5,415.80 1.113
Adult Actual 0.752 0.743 $5,712.60 0.658
Adult% 108.3% 137.9% 105.5% 65.79% 104.4%
DW Target 0.629 0.480 $9,609.50 1.148
DW Actual 0.813 0.813 $7,024.50 0.740
DW% 129.2% 169.3% 73.1% 74.03% 111.4%
Youth Target 0.622 0.676
0.771
Youth Actual 0.716 0.684
0.637
Youth% 115.2% 101.2%
82.62% 99.7%
WP Target 0.467 0.899 $3,852.90
WP Actual 0.721 0.711 $5,661.10
WP% 154.5% 79.1% 146.9%
126.8%
Avg % of Target by Measure 126.8% 121.9% 108.5% 74.1% 109.2%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.703 0.567 $5,936.60 1.106
Adult Actual 0.796 0.821 $6,858.60 0.683
Adult% 113.2% 145.0% 115.5% 68.26% 110.5%
DW Target 0.622 0.530 $9,050.10 1.085
DW Actual 0.873 0.845 $7,526.50 0.725
DW% 140.4% 159.4% 83.2% 72.46% 113.9%
Youth Target 0.630 0.667
0.785
Youth Actual 0.712 0.674
0.683
Youth% 113.0% 101.1%
86.96% 100.4%
WP Target 0.471 0.900 $3,863.00
WP Actual 0.720 0.716 $5,868.20
WP% 153.0% 79.6% 151.9%
128.2%
Avg % of Target by Measure 129.9% 121.3% 116.9% 75.9% 112.1%
101
Puerto Rico
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.699 0.550 $4,049.40 0.489
Adult Actual 0.511 0.377 $2,677.50 0.592
Adult% 73.2% 68.5% 66.1% 120.98% 82.2%
DW Target 0.687 0.573 $3,677.40 0.460
DW Actual 0.504 0.372 $3,139.70 0.533
DW% 73.4% 64.8% 85.4% 115.80% 84.9%
Youth Target 0.104 0.310
0.100
Youth Actual 0.352 0.380
0.249
Youth% 337.1% 122.3%
248.47% 236.0%
WP Target 0.913 0.651 $1,431.80
WP Actual 0.460 0.456 $2,389.50
WP% 50.4% 70.1% 166.9%
95.8%
Avg % of Target by Measure 133.5% 81.4% 106.1% 161.8% 122.7%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.726 0.573 $3,969.60 0.521
Adult Actual 0.592 0.419 $2,379.00 0.722
Adult% 81.7% 73.1% 59.9% 138.56% 88.3%
DW Target 0.766 0.555 $3,673.50 0.519
DW Actual 0.612 0.459 $2,803.50 0.726
DW% 79.8% 82.7% 76.3% 139.88% 94.7%
Youth Target 0.197 0.354
0.086
Youth Actual 0.526 0.497
0.421
Youth% 267.5% 140.5%
420.89% 276.3%
WP Target 0.909 0.627 $1,467.60
WP Actual 0.443 0.439 $2,820.80
WP% 48.8% 69.9% 192.2%
103.6%
Avg % of Target by Measure 119.4% 91.6% 109.5% 233.1% 139.6%
102
Virgin Islands
2011 ER ER4 ME CR Avg % Target by Program
Adult Target 0.559 0.304 $2,499.50 0.561
Adult Actual 0.473 0.370 $3,977.30 0.364
Adult% 84.6% 121.8% 159.1% 64.86% 107.6%
DW Target 0.708 0.336 $2,175.20 0.322
DW Actual 0.528 0.431 $8,405.80 0.180
DW% 74.6% 128.1% 386.4% 55.77% 161.2%
Youth Target 0.471 0.470
0.163
Youth Actual 0.180 0.323
0.198
Youth% 38.3% 68.7%
121.27% 76.1%
WP Target 0.522 0.787 $2,335.70
WP Actual 0.329 0.318 $4,739.70
WP% 62.9% 40.4% 202.9%
102.1%
Avg % of Target by Measure 65.1% 89.8% 249.5% 80.6% 116.5%
2012 ER ER4 ME CR Avg % Target by Program
Adult Target 0.530 0.310 $2,251.30 0.790
Adult Actual 0.289 0.289 $2,600.30 0.581
Adult% 54.5% 93.3% 115.5% 73.51% 84.2%
DW Target 0.602 0.289 $1,846.80 0.674
DW Actual 0.358 0.321 $3,233.30 0.511
DW% 59.5% 111.1% 175.1% 75.84% 105.4%
Youth Target 0.354 0.515
0.237
Youth Actual 0.281 0.345
0.521
Youth% 79.4% 67.0%
219.88% 122.1%
WP Target 0.530 0.800 $2,166.20
WP Actual 0.344 0.338 $4,379.80
WP% 64.9% 42.3% 202.2%
103.1%
Avg % of Target by Measure 64.6% 78.4% 164.3% 123.1% 105.6%
103
Appendix D. Summarized State Results
Alabama
Measure 2011 2012 Fail=1
2Q Employment 92.7% 93.0% 0
4Q Employment 103.6% 101.4% 0
Earnings 92.0% 90.4% 0
Credential 70.0% 69.1% 1
Program 2011 2012 Fail=1
Adults 91.0% 89.2% 0
Dislocated Workers 85.6% 84.9% 1
Youth 89.6% 89.5% 1
Wagner-Peyser 98.7% 97.2% 0
Alaska
Measure 2011 2012 fail=1
2Q Employment 106.2% 115.0% 0
4Q Employment 99.2% 106.9% 0
Earnings 141.4% 149.2% 0
Credential 96.3% 106.6% 0
Program 2011 2012 fail=1
Adults 113.5% 114.5% 0
Dislocated Workers 111.4% 121.7% 0
Youth 113.1% 136.3% 0
Wagner-Peyser 98.6% 100.5% 0
Arizona
Measure 2011 2012 fail=1
2Q Employment 96.0% 96.0% 0
4Q Employment 94.9% 92.0% 0
Earnings 95.8% 96.0% 0
Credential 122.1% 126.6% 0
Program 2011 2012 fail=1
Adults 111.2% 111.9% 0
Dislocated Workers 102.8% 104.4% 0
Youth 95.9% 94.5% 0
Wagner-Peyser 91.3% 90.4% 0
Arkansas
Measure 2011 2012 fail=1
2Q Employment 114.2% 112.6% 0
4Q Employment 110.8% 112.8% 0
Earnings 89.4% 95.3% 0
Credential 99.8% 100.1% 0
Program 2011 2012 fail=1
Adults 98.2% 99.1% 0
Dislocated Workers 93.7% 99.8% 0
Youth 138.4% 134.5% 0
Wagner-Peyser 95.0% 96.4% 0
California
Measure 2011 2012 fail=1
2Q Employment 99.8% 98.0% 0
4Q Employment 109.3% 108.7% 0
Earnings 104.0% 96.6% 0
Credential 156.9% 150.5% 0
Program 2011 2012 fail=1
Adults 130.2% 125.3% 0
Dislocated Workers 119.7% 113.8% 0
Youth 109.6% 107.3% 0
Wagner-Peyser 96.9% 96.8% 0
Colorado
Measure 2011 2012 fail=1
2Q Employment 94.8% 96.2% 0
4Q Employment 90.1% 91.7% 0
Earnings 98.1% 103.4% 0
Credential 139.9% 91.3% 0
Program 2011 2012 fail=1
Adults 118.0% 99.7% 0
Dislocated Workers 111.1% 96.2% 0
Youth 84.0% 85.9% 1
Wagner-Peyser 95.1% 98.3% 0
104
Connecticut
Measure 2011 2012 fail=1
2Q Employment 93.5% 99.3% 0
4Q Employment 102.9% 107.8% 0
Earnings 95.4% 98.8% 0
Credential 149.8% 155.1% 0
Program 2011 2012 fail=1
Adults 113.6% 114.1% 0
Dislocated Workers 113.4% 120.7% 0
Youth 105.3% 109.8% 0
Wagner-Peyser 99.0% 107.2% 0
Delaware
Measure 2011 2012 fail=1
2Q Employment 96.1% 96.0% 0
4Q Employment 101.6% 109.1% 0
Earnings 91.7% 91.5% 0
Credential 131.6% 130.6% 0
Program 2011 2012 fail=1
Adults 109.0% 112.4% 0
Dislocated Workers 97.4% 102.7% 0
Youth 110.2% 108.0% 0
Wagner-Peyser 101.5% 100.8% 0
District of Columbia
Measure 2011 2012 fail=1
2Q Employment 107.0% 85.5% 0
4Q Employment 171.5% 158.0% 0
Earnings 61.5% 67.3% 1
Credential 468.4% 544.1% 0
Program 2011 2012 fail=1
Adults 209.4% 260.9% 0
Dislocated Workers 212.6% 222.9% 0
Youth 253.5% 203.0% 0
Wagner-Peyser 85.0% 87.9% 1
Florida
Measure 2011 2012 fail=1
2Q Employment 102.0% 100.7% 0
4Q Employment 97.6% 97.4% 0
Earnings 123.0% 124.0% 0
Credential 145.5% 144.5% 0
Program 2011 2012 fail=1
Adults 132.2% 130.7% 0
Dislocated Workers 120.0% 115.2% 0
Youth 92.0% 96.7% 0
Wagner-Peyser 106.4% 108.0% 0
Georgia
Measure 2011 2012 fail=1
2Q Employment 94.9% 98.0% 0
4Q Employment 104.5% 106.7% 0
Earnings 103.0% 100.2% 0
Credential 89.7% 89.1% 1
Program 2011 2012 fail=1
Adults 93.3% 93.6% 0
Dislocated Workers 92.8% 93.3% 0
Youth 108.7% 106.2% 0
Wagner-Peyser 101.9% 106.9% 0
Hawaii
Measure 2011 2012 fail=1
2Q Employment 101.8% 99.2% 0
4Q Employment 66.5% 70.1% 1
Earnings 100.9% 97.6% 0
Credential 138.9% 126.2% 0
Program 2011 2012 fail=1
Adults 99.7% 95.9% 0
Dislocated Workers 114.3% 100.6% 0
Youth 71.3% 76.5% 1
Wagner-Peyser 107.6% 111.0% 0
105
Idaho
Measure 2011 2012 fail=1
2Q Employment 113.1% 109.3% 0
4Q Employment 111.1% 110.2% 0
Earnings 93.2% 90.7% 0
Credential 106.9% 96.1% 0
Program 2011 2012 fail=1
Adults 104.3% 98.4% 0
Dislocated Workers 104.8% 101.1% 0
Youth 115.5% 106.6% 0
Wagner-Peyser 104.8% 106.8% 0
Illinois
Measure 2011 2012 fail=1
2Q Employment 92.6% 93.1% 0
4Q Employment 97.2% 100.6% 0
Earnings 102.3% 104.8% 0
Credential 111.6% 117.5% 0
Program 2011 2012 fail=1
Adults 104.9% 106.7% 0
Dislocated Workers 103.3% 103.1% 0
Youth 98.4% 100.4% 0
Wagner-Peyser 91.0% 100.6% 0
Indiana
Measure 2011 2012 fail=1
2Q Employment 106.4% 109.6% 0
4Q Employment 109.7% 114.0% 0
Earnings 96.7% 95.4% 0
Credential 78.1% 86.2% 1
Program 2011 2012 fail=1
Adults 94.8% 95.5% 0
Dislocated Workers 91.7% 96.3% 0
Youth 107.6% 110.5% 0
Wagner-Peyser 106.7% 113.4% 0
Iowa
Measure 2011 2012 fail=1
2Q Employment 109.7% 106.5% 0
4Q Employment 110.5% 111.9% 0
Earnings 102.7% 100.8% 0
Credential 99.0% 104.0% 0
Program 2011 2012 fail=1
Adults 97.4% 95.4% 0
Dislocated Workers 99.5% 100.5% 0
Youth 106.6% 106.6% 0
Wagner-Peyser 126.1% 128.1% 0
Kansas
Measure 2011 2012 fail=1
2Q Employment 102.1% 103.1% 0
4Q Employment 102.7% 103.4% 0
Earnings 90.8% 92.1% 0
Credential 114.3% 116.7% 0
Program 2011 2012 fail=1
Adults 104.1% 102.0% 0
Dislocated Workers 102.1% 102.4% 0
Youth 106.5% 113.4% 0
Wagner-Peyser 96.8% 98.4% 0
Kentucky
Measure 2011 2012 fail=1
2Q Employment 103.8% 102.3% 0
4Q Employment 96.2% 100.9% 0
Earnings 97.6% 97.6% 0
Credential 79.4% 80.5% 1
Program 2011 2012 fail=1
Adults 89.9% 94.4% 0
Dislocated Workers 94.7% 95.1% 0
Youth 97.1% 95.3% 0
Wagner-Peyser 100.5% 101.2% 0
106
Louisiana
Measure 2011 2012 fail=1
2Q Employment 106.0% 105.5% 0
4Q Employment 97.5% 98.8% 0
Earnings 108.4% 101.7% 0
Credential 78.3% 82.3% 1
Program 2011 2012 fail=1
Adults 101.5% 99.3% 0
Dislocated Workers 100.7% 99.5% 0
Youth 98.6% 102.5% 0
Wagner-Peyser 89.9% 88.8% 1
Maine
Measure 2011 2012 fail=1
2Q Employment 106.4% 105.0% 0
4Q Employment 104.7% 100.6% 0
Earnings 106.9% 107.4% 0
Credential 97.7% 97.9% 0
Program 2011 2012 fail=1
Adults 103.8% 100.6% 0
Dislocated Workers 102.1% 103.6% 0
Youth 97.2% 91.0% 0
Wagner-Peyser 114.3% 116.2% 0
Maryland
Measure 2011 2012 fail=1
2Q Employment 99.4% 100.9% 0
4Q Employment 102.1% 102.4% 0
Earnings 98.0% 98.7% 0
Credential 131.6% 130.3% 0
Program 2011 2012 fail=1
Adults 111.3% 112.1% 0
Dislocated Workers 104.4% 105.7% 0
Youth 117.0% 115.0% 0
Wagner-Peyser 93.6% 94.7% 0
Massachusetts
Measure 2011 2012 fail=1
2Q Employment 100.9% 107.7% 0
4Q Employment 108.0% 112.8% 0
Earnings 92.1% 94.4% 0
Credential 251.4% 246.4% 0
Program 2011 2012 fail=1
Adults 168.0% 170.3% 0
Dislocated Workers 140.3% 139.8% 0
Youth 104.4% 110.6% 0
Wagner-Peyser 106.6% 110.6% 0
Michigan
Measure 2011 2012 fail=1
2Q Employment 106.0% 105.2% 0
4Q Employment 111.7% 113.1% 0
Earnings 89.3% 93.2% 0
Credential 135.4% 157.5% 0
Program 2011 2012 fail=1
Adults 113.3% 130.3% 0
Dislocated Workers 109.3% 116.0% 0
Youth 113.1% 109.6% 0
Wagner-Peyser 105.0% 103.7% 0
Minnesota
Measure 2011 2012 fail=1
2Q Employment 98.4% 99.7% 0
4Q Employment 108.0% 109.3% 0
Earnings 91.0% 92.7% 0
Credential 120.3% 127.6% 0
Program 2011 2012 fail=1
Adults 107.5% 111.1% 0
Dislocated Workers 104.3% 107.8% 0
Youth 101.3% 104.3% 0
Wagner-Peyser 102.9% 102.9% 0
107
Mississippi
Measure 2011 2012 fail=1
2Q Employment 116.9% 119.6% 0
4Q Employment 106.3% 111.5% 0
Earnings 86.8% 90.0% 0
Credential 64.9% 77.9% 1
Program 2011 2012 fail=1
Adults 87.4% 92.9% 0
Dislocated Workers 81.5% 87.9% 1
Youth 135.9% 143.5% 0
Wagner-Peyser 88.0% 91.5% 0
Missouri
Measure 2011 2012 fail=1
2Q Employment 97.2% 94.3% 0
4Q Employment 89.9% 93.3% 0
Earnings 91.7% 91.6% 0
Credential 83.4% 91.9% 0
Program 2011 2012 fail=1
Adults 83.8% 89.0% 1
Dislocated Workers 88.8% 91.8% 0
Youth 93.4% 87.9% 0
Wagner-Peyser 101.2% 104.7% 0
Montana
Measure 2011 2012 fail=1
2Q Employment 123.7% 115.1% 0
4Q Employment 107.5% 96.6% 0
Earnings 126.5% 127.9% 0
Credential 90.6% 52.5% 0
Program 2011 2012 fail=1
Adults 109.8% 95.8% 0
Dislocated Workers 99.9% 79.9% 0
Youth 120.2% 96.8% 0
Wagner-Peyser 125.6% 131.6% 0
Nebraska
Measure 2011 2012 fail=1
2Q Employment 98.2% 99.5% 0
4Q Employment 102.7% 105.1% 0
Earnings 98.6% 93.5% 0
Credential 78.5% 90.3% 0
Program 2011 2012 fail=1
Adults 87.9% 89.8% 1
Dislocated Workers 97.0% 94.8% 0
Youth 96.9% 107.6% 0
Wagner-Peyser 101.5% 102.9% 0
Nevada
Measure 2011 2012 fail=1
2Q Employment 117.9% 119.9% 0
4Q Employment 125.2% 118.8% 0
Earnings 143.3% 141.4% 0
Credential 84.3% 114.7% 0
Program 2011 2012 fail=1
Adults 137.5% 144.5% 0
Dislocated Workers 143.8% 154.7% 0
Youth 67.5% 60.1% 1
Wagner-Peyser 109.0% 115.3% 0
New Hampshire
Measure 2011 2012 fail=1
2Q Employment 93.4% 92.4% 0
4Q Employment 95.4% 97.0% 0
Earnings 102.3% 104.4% 0
Credential 103.4% 94.1% 0
Program 2011 2012 fail=1
Adults 102.5% 91.2% 0
Dislocated Workers 100.4% 99.4% 0
Youth 89.0% 95.8% 0
Wagner-Peyser 97.9% 101.1% 0
108
New Jersey
Measure 2011 2012 fail=1
2Q Employment 89.7% 89.3% 1
4Q Employment 95.2% 83.5% 0
Earnings 114.0% 110.9% 0
Credential 91.1% 100.3% 0
Program 2011 2012 fail=1
Adults 100.2% 99.6% 0
Dislocated Workers 97.5% 100.6% 0
Youth 93.7% 78.8% 0
Wagner-Peyser 94.3% 95.8% 0
New Mexico
Measure 2011 2012 fail=1
2Q Employment 99.4% 97.7% 0
4Q Employment 93.7% 94.7% 0
Earnings 98.8% 95.9% 0
Credential 87.1% 78.3% 1
Program 2011 2012 fail=1
Adults 115.1% 96.4% 0
Dislocated Workers 88.1% 92.9% 0
Youth 81.8% 81.3% 1
Wagner-Peyser 90.5% 97.1% 0
New York
Measure 2011 2012 fail=1
2Q Employment 87.1% 88.9% 1
4Q Employment 93.9% 94.9% 0
Earnings 85.9% 86.6% 1
Credential 118.8% 132.9% 0
Program 2011 2012 fail=1
Adults 105.6% 117.4% 0
Dislocated Workers 92.3% 90.5% 0
Youth 97.1% 99.7% 0
Wagner-Peyser 85.1% 87.5% 1
North Carolina
Measure 2011 2012 fail=1
2Q Employment 98.8% 98.3% 0
4Q Employment 102.1% 104.7% 0
Earnings 84.9% 87.3% 1
Credential 93.2% 87.6% 0
Program 2011 2012 fail=1
Adults 90.3% 90.4% 0
Dislocated Workers 95.8% 93.6% 0
Youth 104.9% 103.0% 0
Wagner-Peyser 93.0% 97.3% 0
North Dakota
Measure 2011 2012 fail=1
2Q Employment 105.1% 108.0% 0
4Q Employment 95.1% 101.8% 0
Earnings 122.3% 121.8% 0
Credential 82.4% 79.3% 1
Program 2011 2012 fail=1
Adults 100.0% 96.8% 0
Dislocated Workers 96.3% 104.5% 0
Youth 110.5% 106.1% 0
Wagner-Peyser 99.4% 106.4% 0
Ohio
Measure 2011 2012 fail=1
2Q Employment 108.3% 102.8% 0
4Q Employment 119.7% 109.1% 0
Earnings 112.9% 110.9% 0
Credential 105.9% 108.2% 0
Program 2011 2012 fail=1
Adults 113.0% 111.3% 0
Dislocated Workers 102.6% 105.8% 0
Youth 101.5% 102.2% 0
Wagner-Peyser 133.8% 109.8% 0
109
Oklahoma
Measure 2011 2012 fail=1
2Q Employment 97.4% 97.3% 0
4Q Employment 99.6% 98.0% 0
Earnings 85.4% 83.7% 1
Credential 77.2% 82.6% 1
Program 2011 2012 fail=1
Adults 89.6% 88.0% 1
Dislocated Workers 92.1% 94.3% 0
Youth 95.1% 92.8% 0
Wagner-Peyser 87.9% 90.8% 0
Oregon
Measure 2011 2012 fail=1
2Q Employment 106.1% 102.1% 0
4Q Employment 99.5% 98.0% 0
Earnings 96.4% 96.8% 0
Credential 82.2% 93.0% 0
Program 2011 2012 fail=1
Adults 91.4% 96.9% 0
Dislocated Workers 93.1% 95.3% 0
Youth 100.6% 92.9% 0
Wagner-Peyser 106.2% 107.3% 0
Pennsylvania
Measure 2011 2012 fail=1
2Q Employment 98.9% 101.9% 0
4Q Employment 104.5% 108.7% 0
Earnings 102.2% 101.6% 0
Credential 96.2% 95.0% 0
Program 2011 2012 fail=1
Adults 97.1% 98.2% 0
Dislocated Workers 96.3% 94.6% 0
Youth 101.0% 98.3% 0
Wagner-Peyser 110.6% 122.1% 0
Rhode Island
Measure 2011 2012 fail=1
2Q Employment 100.7% 105.3% 0
4Q Employment 113.8% 117.3% 0
Earnings 97.9% 93.2% 0
Credential 197.9% 159.0% 0
Program 2011 2012 fail=1
Adults 150.9% 130.7% 0
Dislocated Workers 133.4% 122.6% 0
Youth 93.9% 101.1% 0
Wagner-Peyser 108.8% 110.3% 0
South Carolina
Measure 2011 2012 fail=1
2Q Employment 102.1% 102.9% 0
4Q Employment 103.4% 104.0% 0
Earnings 92.2% 93.2% 0
Credential 96.5% 96.9% 0
Program 2011 2012 fail=1
Adults 98.5% 99.2% 0
Dislocated Workers 95.8% 97.1% 0
Youth 112.6% 112.2% 0
Wagner-Peyser 91.1% 91.9% 0
South Dakota
Measure 2011 2012 fail=1
2Q Employment 104.9% 109.0% 0
4Q Employment 101.3% 105.1% 0
Earnings 100.5% 102.9% 0
Credential 81.1% 77.6% 1
Program 2011 2012 fail=1
Adults 99.6% 96.6% 0
Dislocated Workers 99.5% 101.3% 0
Youth 96.5% 102.3% 0
Wagner-Peyser 94.6% 99.9% 0
110
Tennessee
Measure 2011 2012 fail=1
2Q Employment 106.0% 105.6% 0
4Q Employment 106.5% 108.4% 0
Earnings 114.6% 116.8% 0
Credential 101.6% 100.1% 0
Program 2011 2012 fail=1
Adults 111.5% 109.1% 0
Dislocated Workers 105.3% 106.8% 0
Youth 112.8% 111.2% 0
Wagner-Peyser 97.7% 103.1% 0
Texas
Measure 2011 2012 fail=1
2Q Employment 97.8% 99.3% 0
4Q Employment 95.7% 96.0% 0
Earnings 110.5% 106.3% 0
Credential 100.8% 107.0% 0
Program 2011 2012 fail=1
Adults 105.2% 103.5% 0
Dislocated Workers 102.2% 104.2% 0
Youth 90.9% 92.3% 0
Wagner-Peyser 101.8% 104.4% 0
Utah
Measure 2011 2012 fail=1
2Q Employment 96.6% 95.9% 0
4Q Employment 99.4% 92.1% 0
Earnings 95.9% 104.2% 0
Credential 82.1% 90.3% 0
Program 2011 2012 fail=1
Adults 80.9% 96.1% 0
Dislocated Workers 100.5% 102.6% 0
Youth 85.4% 86.0% 1
Wagner-Peyser 112.1% 94.2% 0
Vermont
Measure 2011 2012 fail=1
2Q Employment 108.0% 96.5% 0
4Q Employment 94.2% 95.9% 0
Earnings 110.7% 113.5% 0
Credential 85.1% 77.8% 1
Program 2011 2012 fail=1
Adults 97.5% 101.6% 0
Dislocated Workers 113.6% 101.6% 0
Youth 72.9% 71.5% 1
Wagner-Peyser 110.9% 105.5% 0
Virginia
Measure 2011 2012 fail=1
2Q Employment 88.5% 88.5% 1
4Q Employment 94.4% 96.3% 0
Earnings 79.9% 78.4% 1
Credential 137.4% 147.9% 0
Program 2011 2012 fail=1
Adults 109.0% 109.4% 0
Dislocated Workers 100.8% 101.8% 0
Youth 87.7% 95.6% 0
Wagner-Peyser 93.7% 95.5% 0
Washington
Measure 2011 2012 fail=1
2Q Employment 107.6% 107.3% 0
4Q Employment 107.9% 108.1% 0
Earnings 103.1% 97.8% 0
Credential 122.2% 121.8% 0
Program 2011 2012 fail=1
Adults 113.7% 110.5% 0
Dislocated Workers 106.4% 105.2% 0
Youth 106.7% 106.0% 0
Wagner-Peyser 112.6% 113.4% 0
111
West Virginia
Measure 2011 2012 fail=1
2Q Employment 111.9% 117.6% 0
4Q Employment 99.5% 101.5% 0
Earnings 95.4% 96.4% 0
Credential 84.1% 90.0% 0
Program 2011 2012 fail=1
Adults 100.4% 98.9% 0
Dislocated Workers 98.7% 105.6% 0
Youth 91.0% 95.8% 0
Wagner-Peyser 105.0% 110.1% 0
Wisconsin
Measure 2011 2012 fail=1
2Q Employment 101.1% 103.4% 0
4Q Employment 114.7% 115.8% 0
Earnings 96.1% 94.7% 0
Credential 96.1% 103.1% 0
Program 2011 2012 fail=1
Adults 90.9% 96.2% 0
Dislocated Workers 99.1% 99.4% 0
Youth 107.8% 108.3% 0
Wagner-Peyser 118.6% 121.0% 0
Wyoming
Measure 2011 2012 fail=1
2Q Employment 126.8% 129.9% 0
4Q Employment 121.9% 121.3% 0
Earnings 108.5% 116.9% 0
Credential 74.1% 75.9% 1
Program 2011 2012 fail=1
Adults 104.4% 110.5% 0
Dislocated Workers 111.4% 113.9% 0
Youth 99.7% 100.4% 0
Wagner-Peyser 126.8% 128.2% 0
Puerto Rico
Measure 2011 2012 fail=1
2Q Employment 133.5% 119.4% 0
4Q Employment 81.4% 91.6% 0
Earnings 106.1% 109.5% 0
Credential 161.8% 233.1% 0
Program 2011 2012 fail=1
Adults 82.2% 88.3% 1
Dislocated Workers 84.9% 94.7% 0
Youth 236.0% 276.3% 0
Wagner-Peyser 95.8% 103.6% 0
Virgin Islands
Measure 2011 2012 fail=1
2Q Employment 65.1% 64.6% 1
4Q Employment 89.8% 78.4% 1
Earnings 249.5% 164.3% 0
Credential 80.6% 123.1% 0
Program 2011 2012 fail=1
Adults 107.6% 84.2% 0
Dislocated Workers 161.2% 105.4% 0
Youth 76.1% 122.1% 0
Wagner-Peyser 102.1% 103.1% 0