Barrier Screening and Assessment Evaluation

Barrier Screening and Assessment Evaluation

Barrier Screening and Assessment Evaluation

Wisconsin Department of Workforce Development Division of Workforce Solutions

December 2005

Barrier Screening and Assessment Evaluation 2

TABLE OF CONTENTS

INTRODUCTION........................................................................................................................1 Background.............................................................................................................................1 Purpose of the Study ..............................................................................................................2 Data Sources and Research Methodology..............................................................................2 Statistical Tests.......................................................................................................................4 Organization of Report ............................................................................................................4

COMPLETION RATES ..............................................................................................................5

Variation in Completion Rates.................................................................................................5 Completion and Decline Rates by Participant Characteristics ................................................7 Reasons for Declining to Complete the BST...........................................................................9 “Comfort” with the BST .........................................................................................................10

SCREENING AND ASSESSMENT..........................................................................................11

Potential Barriers ..................................................................................................................11 Quality of Assessments.........................................................................................................11 Assessment Providers ..........................................................................................................13

CASE OUTCOMES..................................................................................................................15

Recommendations in Assessments and EPs........................................................................15 Assessments and Follow-Through........................................................................................16 Detailed Follow-Through.......................................................................................................17 Impact on Case Planning......................................................................................................17

SUMMARY OF KEY FINDINGS ..............................................................................................21

Completion Rates .................................................................................................................21 Screening and Assessment ..................................................................................................22 Case Outcomes ....................................................................................................................22

Appendix: Research Notes...................................................................................................23

Note 1: Measuring the Accuracy of the Barrier Screening Tool ...........................................23 Note 2: Telephone Survey Questions ..................................................................................24 Note 3: Telephone survey sampling and weighting..............................................................25 Note 4: Administrative Data .................................................................................................25 Note 5: Methods for Intensive Case Review ........................................................................26 Note 6: Intensive Case Review Sampling and Weighting ....................................................27


INTRODUCTION

Background This study examines the effectiveness of the Barrier Screening Tool (BST) and assessment process in addressing the employment barriers of participants in the Wisconsin Works (W-2) program. The purpose of the BST is to provide Financial and Employment Planners (FEPs) with information that will enable them to develop suitable, individualized Employability Plans (EPs) for W-2 participants. The BST was implemented by the Division of Workforce Solutions (DWS) in May 2003. The BST identifies potential barriers to employment and participation in W-2 program activities in seven areas:

• Alcohol and other drug abuse • Traumatic brain injury • Domestic abuse • Learning disabilities • Medical or physical problems • Mental health issues • Trauma1

The BST is administered by FEPs in face-to-face interviews with W-2 participants in the FEP’s office. The FEP or other interviewer reads the questions to the participant and then enters responses on a computer screen as they proceed through the BST instrument. The instrument is automated and screeners receive prompts throughout the screening process. Screen 1 of the instrument examines functions in activities of daily living and possible domestic violence. Based on answers to the questions in Screen 1, the system may trigger follow-up questions on Screen 2. Follow-up questions examine in more detail conditions that are often unidentified but are significant barriers to the participant becoming employed, such as learning disabilities, trauma (e.g., Post Traumatic Stress Disorder), mental health, alcohol and other drug abuse and traumatic brain injuries. Local agency staff are required to offer the tool to new W-2 participants within 30 days after initial placement. The tool is also offered if the FEP has reason to believe re-administering it may identify unidentified barriers, the participant’s case has been closed for one year or more or the FEP is considering denying an extension of W-2 time limits. It is also administered if the participant requests that it be re-administered. 2 Before administering the BST, FEPs must complete a training session, which includes a review of the background and history of the tool, an overview of W-2 policy regarding its application and hands-on practice in its use. Participants are required to sign a form indicating whether or not they agree to complete the BST. There is no program penalty for those participants who decline to be screened by the tool.

1 Trauma disorders are caused by a single or series of traumatic events and may include Post Traumatic Stress Disorder or Generalized Anxiety Disorder. 2 In most instances, the participant’s FEP administers the BST. In some agencies a specialized FEP or staff from a contracted agency administer the tool. Participants in W-2 Transitions (W-2 T), Community Service Job (CSJ), Trial Job or Case Management for Unemployed Individuals (CMS) must be offered screening. Participants in Case Management for Individuals in Unsubsidized Employment (CMU), Case Management Follow-up (CMF), Case Management for Non-custodial Parents, Case Management for Pregnant Women and Case Management for Minor Parents are not offered screening.


In instances where the BST results indicate that a formal assessment is needed, the FEP must refer a participant for a formal assessment.3 The participant may decline or refuse to participate in the formal assessment. By policy, such a refusal cannot result in an adverse action to the participant. The formal assessment establishes the extent and severity of the participant’s employment barrier(s) and any alternative services or accommodations that may be required. The assessment must contain certain types of information about the participant, including a personal diagnosis, general aptitude or cognitive level (if at-risk for learning disability), areas of strengths and deficits, recommendations for appropriate services or accommodations and a summary of findings describing the rationale for the diagnoses, prognoses or recommendations. The formal assessment must be completed by a qualified assessing agency or individual with qualifications appropriate to the suspected barrier. In addition, the assessing agency or individual must demonstrate an understanding of the objectives of the assessment process.

Purpose of the Study The BST and the procedures for its use are new and innovative, and have not been evaluated.4 This study examines how the barrier screening tool and assessment procedures have been implemented and the impact of the screening and assessment process on case planning for W-2 participants.5 It addresses the following three issues:

• Consistency in use of the BST: Do completion rates vary among agencies or among FEPs within agencies? Is a participant’s decision to decline use of the tool influenced by participant characteristics, such as race, placement type or the length of time in the program?

• Follow-through on potential barriers identified by the BST: Are formal assessments conducted

when required, and are they performed within the specified time frame? Do assessing agencies/individuals have the appropriate credentials and do they understand the objectives of the assessment process?

• Impact of the BST on case planning: Do assessment recommendations appear in the case

plan? Are participants who complete the BST assigned to different patterns of activities than those who do not complete the BST?

Data Sources and Research Methodology The study uses a number of data sources and research methodologies, including an analysis of a survey of W-2 participants, analyses of administrative data and an intensive review of a sample of cases and interviews with agency staff. A descriptive summary of each of these approaches follows.

3 Division of Workforce Solutions, Wisconsin Works Screening and Assessment Policy, Wisconsin Department of Workforce Development, January 2003. 4 Many of the components of the BST were adopted from validated instruments used in other programs. The tool was developed in partnership with researchers from the UW-Milwaukee Center for Addiction and Behavioral Health Research. A work group made up of local W-2 agency representatives, disability and domestic abuse experts from the Department of Health and Family Services, a learning disability provider, Division of Vocational Rehabilitation (DVR) representatives and training and curriculum writers provided guidance and feedback throughout the development of the instrument. The BST was field tested in eight W-2 agencies. 5 The original study design proposal included analyzing whether formal assessments confirmed the existence of potential barriers identified by the BST. While the current study provides some measure of this aspect of the tool’s accuracy, directly examining this (and other) measures of the tool’s accuracy were not possible due to resource limitations. For more information on measuring the accuracy of the BST, see Appendix Note 1.


Participant Survey. To gather information about participants’ experience with the BST, questions were added to a monthly telephone sample survey of W-2 participants conducted by DWS.6 Four BST-related questions (two of which were open-ended) were included in the survey from February 2004 through April 2005. Their responses were recorded and coded into categories. Participants who had declined to complete the BST were asked why they declined. Participants who had completed the BST were asked about their experience with it.7 Administrative Data. CARES (Client Assistance for Reemployment and Economic Support) is Wisconsin’s automated case management and information system for the W-2 population. The CARES information system includes detailed historical and current case record, demographic and service information. BST information, including detailed screening results and follow-up documentation, is contained in a separate automated system. By merging selected data from CARES and the BST automated system, a single data set was produced with a wide range of participant, case and barrier information.8 Intensive Case Reviews. Intensive case reviews were conducted on a sample of W-2 participants who had potential employment barriers identified and the worker had indicated making a referral for at least one formal assessment. These reviews were used to determine whether the screening and assessment process was carried out according to policy. The sample was drawn from the population of participants who had a barrier identified and a formal assessment scheduled between January 1 and July 31, 2004. The DWS specified the universe of participants who had at least one potential barrier identified by the BST and where the worker had indicated making referral for at least formal assessment. Participant cases were included in the universe if the BST had been administered at least three months prior to the sampling, so that agencies would have had sufficient time to follow up on screening results and subsequent assessments. A random sample of participants was drawn from this universe from each of three strata: Milwaukee County, other large urban agencies and smaller agencies.9 Seventy cases were sampled from each stratum for a total of 210 cases. Local agencies were contacted by DWS Regional Office staff and asked to provide all formal assessments that were completed for the sampled case from a period one year before the BST assessment date to the date of the contact (August 10, 2004). The cases were divided among members of a review team made up of five Regional Administrators and one central office staff member. For each case, the reviewers examined the BST results; formal assessments provided

6 To be included in the population from which the sample was drawn, a participant must have been in a W-2 paid placement for at least the previous 60 days. The 6,751 respondents were those who responded to one or more questions in the main part of the survey. Analyses of participant survey data in this study are based on sub samples who received the BST questions. 7 This survey was administered in English, Spanish and Hmong. For the text of the survey, see Appendix Note 2. The survey sample was stratified into four groups: Milwaukee County, large agencies, medium sized agencies and smaller agencies. The analysis weights cases to recreate the proper representation in the statewide population of cases that meet the selection criteria. For more information on the sampling and weighting technique, see Appendix Note 3. 8 For more information on the administrative data used in this analysis, see Appendix Note 4. 9 “Milwaukee County” includes two of the three agencies operating W-2 in Milwaukee County at the time the sample was drawn: MAXIMUS, OIC and UMOS. “Large agencies” include Brown, Kenosha, Outagamie, Racine and Rock Counties and Capitol Consortium, which includes Dane County. “Balance of state” includes the remaining 19 county agencies and seven consortia.


by the agencies; case histories and backgrounds from the automated CARES system; case notes and employability plans. The results from the case reviews were entered on a formatted instrument which included space for each of the seven barrier areas.10

Agency Staff Interviews. To better interpret results obtained from the administrative data, case reviews and the participant survey, agency staff involved in the administration of the BST and the assessment process were interviewed. A “purposeful sampling technique” was used to select six agencies for site visits: two from Milwaukee County, two from large urban agencies and two from smaller agencies.11 Half of the agencies were selected because they had high BST completion rates and half because they had low completion rates. Similarly, within agencies, FEPs were selected because they had high or low completion rates. Persons to be interviewed included FEPs, supervisors and at least one administrator from each agency. Interviews were conducted by a team of DWS central and Regional Office staff in August and September 2005. Questions were open-ended and interviewees were questioned separately.

Statistical Tests Where appropriate, relationships between variables were tested using statistical tests of significance. Where data were nominal or categorical, a Chi Square (X2) test of statistical significance was used. Where data were interval or averages, a difference of means t-test was used. In tables, one asterisk (*) indicates significance at the .05 level. Two asterisks (**) indicate significance at the higher .01 level.

Organization of Report This report in divided the following four sections:

• Completion Rates • Screening and Assessment • Case Outcomes • Summary of Key Findings

The appendix provides a detailed discussion of the data sources and research methodologies used in this report.

10 Each case was reviewed by one reviewer. To promote consistency, reviewers participated in training in which they jointly reviewed a sample of cases. In addition, an automated data recording instrument was developed to promote consistency among reviewers. For more information on the case review criteria, see Appendix Note 5. The analysis weights cases to recreate the proper representation in the statewide population of cases that meet the selection criteria. For more information on the sampling and weighting method, see Appendix Note 6. 11 Agencies were selected because they had wide variations in completion rates and individual FEPs within those agencies were selected because they had wide variations in completion rates. The sampling method used here was not intended to produce a sample that is representative of the state as a whole. This approach is intended to select a small number of “information-rich cases for study in depth.” For more information on this technique, see Patton, Michael Quinn, How to Use Qualitative Methods in Evaluation, Sage, 1987, pp. 44-69.


COMPLETION RATES

Most new W-2 participants12 must be offered an opportunity to be screened by the BST within 30 days after placement.13 Every participant must sign the Wisconsin Works (W-2) Barrier Screening Tool Agreement indicating that they were offered an opportunity to be screened and chose to either complete or decline use of the tool. The Agreement form explains that completion of the tool is not mandatory, that the respondent may skip questions they are uncomfortable answering, that results are confidential and that they will not be penalized in any way if they decline the BST. The tool is offered in English, Spanish and Hmong. Other languages are accommodated through interpreters. Beyond the requirement that the BST be offered and the Agreement signed by the participant, agencies and individual FEPs have considerable latitude in how they present the BST and the Agreement form. A report by the Wisconsin Legislative Audit Bureau (LAB) indicated considerable variation in BST completion rates between May 2003 and June 2004. The LAB report concluded “significant variations in agencies’ screening rates raise concerns about whether all agency staff explain the benefits of screening and encourage participants to complete it.”14

Variation in Completion Rates There was considerable variation in the BST completion rates among agencies for W-2 participants who were eligible for screening between May 2003 and April 2005.15 Statewide, 51 percent of W-2 participants completed a screening during this time period. The completion rates among agencies ranged from 0 to 100 percent with a median completion rate of 73 percent (Table 1).16 Completion rates among agencies with few cases may be influenced by randomly occurring events. Differences among agencies with large numbers of cases, however, may reflect variations in local agency practice and caseload characteristics. Among the 28 agencies with 100 or more cases, completion rates varied from 19 percent to 99 percent.

12 Participants in W-2 Transitions (W-2 T), Community Service Job (CSJ), Trial Job or Case Management for Unemployed Individuals (CMS) must be offered screening. Participants in Case Management for Individuals in Unsubsidized Employment (CMU), Case Management Follow-up (CMF), Case Management for non-custodial Parents, Case Management for Pregnant Women and Case Management for Minor Parents are not offered screening. 13 During the phase-in period, May through July 2003, individuals who were on-going W-2 participants were offered the BST within 30 days of their next six-month review of eligibility. Local agency staff are required to offer the tool to new W-2 participants within 30 days after initial placement. The tool is also offered if the FEP has reason to believe re-administering it may identify unidentified barriers, the participant’s case has been closed for one year or more or the FEP is considering denying an extension of W-2 time limits. It is also administered if the participant requests that it be re-administered. 14 Legislative Audit Bureau, An Evaluation: Wisconsin Works (W-2) Program, Wisconsin Legislative Audit Bureau, Report 05-6, April 2005, pp. 75-78. 15 The data set included records for 25,636 individuals. In 3,631 of the cases (14.2 percent) the record had missing data in the “completed or declined” field. Here a case is considered to have completed the BST if the participant ever completed, or partially completed, a screening. 16 W-2 is administered locally by a number of counties, consortia of counties and contractors within counties. Over the period covered by this analysis the boundaries of these administrative units varied. During the period of this analysis, Milwaukee County consists of six administrative regions. These regions were administered by four agencies (MAXIMUS, OIC-GM, UMOS and YWCA) and the regions covered by those agencies varied over time. Therefore, in Table 1, for Milwaukee County the agency and administrative region are both listed.


Table 1: Barrier Screening Tool Completion Rates by Agency, May 2003 - April 2005 (n=22,005)

Contract Agency

Number of

Cases

Percent

Completed

Contract Agency

Number of

Cases

Percent

Completed Barron 20 80 Rusk 9 100 Bayfield 22 91 Shawano 21 57 Brown 165 50 Taylor 19 53 Buffalo 6 83 Trempealeau 49 47 Burnett 2 0 Vernon 7 86 Calumet 32 94 Walworth 115 44 Chippewa 26 100 Waukesha 274 85 Clark 90 80 Waupaca 100 73 Columbia 60 60 Winnebago 173 54 Crawford 12 100 Southwest Consortium 71 73 Door 40 80 Workforce Connections 128 59 Douglas 100 99 Forward Services 35 100 Dunn 123 91 Northern Consortium 117 90 Eau Claire 144 84 Capitol Consortium 831 36 Fond du Lac 211 80 PAW Consortium 290 70 Green Lake 9 100 Bay Area Consortium 27 85 Iron 2 100 OW Consortium 119 39 Jefferson 108 85 AP Consortium 34 53 Kenosha 744 38 S/W Consortium 10 20 La Crosse 118 62 OIC-GM (Regions 1, 3, 4) 4,611 52 Marathon 150 77 UMOS (Region 2) 3,971 51 Marinette 1 100 MAXIMUS 5,444 40 Oconto 8 88 Lakeshore Consortium 120 85 Outagamie 112 19 Workforce Connections 56 25 Pepin 3 0 YWCA 605 53 Polk 27 33 OIC-GM (Region 3) 417 66 Racine 477 43 UMOS (Regions 3, 4) 1,095 45 Rock 445 82 Total 22,005 51

Source: Work Programs Warehouse, WP038 extract. There was also considerable variation in completion rates among FEPs within agencies. There were eight W-2 agencies with more than one FEP with 40 or more cases.17 The completion rates among these FEPs varied from zero to 100 percent (Table 2, next page). The variation in completion rates indicates considerable discretion in how the individual FEPs may present and emphasize the use of the BST. In the Capital Consortium, for example, if participants were assigned to the FEP with the highest completion rate, they were approximately seven times more likely to be screened than if they were assigned to FEP with the lowest completion rate. In some cases, however, the variation reflects the impact of the specialized caseloads that some FEPs have (e.g., all Hmong speakers or “intake” cases), which can result in low completion rates.18

17 Twenty-four FEPs worked for more than one agency during the period covered by this analysis. In those cases, the completion rate for the FEP is assigned to the agency for which they worked the preponderance of the time. 18 In interviews, a small number of FEPs confirmed that their opinion of the value of the BST and how they present it to participants likely affects completion rates.


Table 2: Barrier Screening Tool Screening Completion Rates for FEPs with 40+ Cases by

Agency, May 2003 - June 2005

Agency

FEPs with 40+ Cases

Minimum Percent

Completed

Maximum Percent

Completed

Average Percent

Completed Brown 1 35 35 35 Capitol Consortium

10

9

64 35

Dodge 1 19 19 19 Douglas 1 100 100 100 Dunn 1 98 98 98 Eau Claire 1 37 37 37 Kenosha 4 0 0 0 La Crosse 1 98 98 98 OIC-GM 9 0 100 45 MAXIMUS 52 0 100 37 YWCA 16 3 100 39 UMOS 52 0 100 42 Outagamie 1 4 4 4 Portage 1 70 70 70 Racine 2 17 31 24 Rock 3 58 100 83 Waukesha 1 100 100 100 Winnebago 1 23 23 23 Wood 1 98 98 98

Source: Work Programs Warehouse, WP038 extract.

Completion and Decline Rates by Participant Characteristics If completion rates vary by placement type, differences in the proportion of participants assigned to a particular placement could contribute to agency variation in the overall proportion of cases with completed screenings. Thus, the participant’s placement at the time they were last offered the BST was examined. Statewide, 49 percent of CSJ and W-2 T participants had completed a screening. Those in W-2 T placements were more likely than those in CSJ placements to have completed a screening. This difference was also apparent in Milwaukee County. There was no significant difference in completion rates between the CSJ and W-2 T participants in the balance of the state (Table 3).19

Table 3: Barrier Screening Tool Screenings Completed by Placement Type and Location, May 2003 - April 2005 (n=13,463)

Statewide Milwaukee Balance of State Placement Number Percent Number Percent Number PercentCSJ 6,283 44 ** 5,955 43 ** 329 56W-2 T 7,180 53 ** 6,316 53 ** 863 56Total 13,463 49 ** 12,271 48 ** 1,192 56Source: CARES and Work Programs Warehouse, WPO38 extract ** Indicates statistically significant at .01 level.

19 Since Milwaukee County makes up nearly 87 percent of the W-2 participants in the state, one would expect the “statewide” and “Milwaukee” results to be very similar and when results for Milwaukee are statistically significant, results for the state as a whole would tend to be significant also.


The amount of time a participant was on W-2 was associated with the decision to complete or decline the BST. In Milwaukee County and statewide, participants who were on the W-2 program longer were more likely to have completed the tool. This may be explained by the fact that participants can be offered the tool multiple times. In agency staff interviews, some FEPs reported offering the BST to participants who were approaching their 24-month time limit in a placement and to participants experiencing significant non-participation problems (Table 4). Table 4: Barrier Screening Tool Screenings Completed by Time in W-2 and Location, May 2003 -

April 2005 (n=13,465)

Statewide Milwaukee Balance of State Months on W-2 Number Percent Number Percent Number Percent

1 month 2,797 41 ** 2,332 39 ** 464 502-6 months 2,299 47 ** 2,545 45 ** 455 597-20 months 4,039 49 ** 3,709 48 ** 331 5621 or more months 4,330 54 ** 4,202 54 ** 129 56Source: CARES and Work Programs Warehouse, WP038 extract. ** Indicates statistically significant at the .01 level.

Black participants statewide were less likely to have completed a screening when compared to Hispanic or white participants, but this difference was not found in Milwaukee County. In the balance of the state, 60 percent of white participants completed a screening while only 44 percent of black participants did so (Table 5).

Table 5: Barrier Screening Tool Screenings Completed by Race and Location, May 2003 - April 2005 (n=14,230)

Statewide Milwaukee Balance of State

Race Number Percent Number Percent Number PercentBlack 9,625 48 * 9,226 48 398 44 **Hispanic 1,716 48 * 1,636 48 79 54 **White 2,376 51 * 1,529 46 847 60 **Other 513 48 * 449 47 64 53 **Source: CARES and Work Programs Warehouse, WP038 extract.

*Indicates statistically significant at the .05 level. ** Indicates statistically significant at the .01 level.

There was no systematic relationship between the race of the participant and completion rates for either placement type in Milwaukee. However, black participants in both placement types were less likely than other groups to have completed a screening in the balance of the state (Table 6). Table 6: Barrier Screening Tool Screenings Completed by Race, Placement Type, and Location,

May 2003 - April 2005 (n=13,464)

Milwaukee Balance of State CSJ W-2 T CSJ W-2 T

Race Number Percent Number Percent Number Percent Number PercentBlack 4,256 44 4,647 53 141 43 ** 209 46 *Hispanic 849 44 698 55 24 63 ** 43 54 *White 606 40 791 53 145 66 ** 579 59 *Other 242 41 181 59 20 55 ** 33 52 *Source: CARES and Work Programs Warehouse, WP038 extract. * Indicates statistically significant at the .05 level. ** Indicates statistically significant at the .01 level.


Reasons for Declining to Complete the BST In the monthly W-2 Participant Survey, participants were asked three questions about their experience with the BST. The questions were preceded by a three-paragraph introduction that explained the BST, assured respondents that the interviewer did not know their screening results and informed participants that their answers to the survey would not affect their benefits in any way. For those who declined to be assessed, the first question was “Our records show that you declined to answer the questions on the Barrier Screening Tool…Is that true?” For those who responded “yes” or “don’t know,” the next question was “Why did you decline to answer these questions?”20 Table 7: Reasons Why Participants Declined to Use the Barrier Screening Tool,

February 2004 - April 2005 (n=1,836)

Reason Percent Didn’t have time 11 Didn’t need services, wouldn’t help 8 Didn’t want to, didn’t interest me 6 No need to answer, questions didn’t apply 4 Caseworker said it was optional 4 Questions too personal 4 Didn’t feel able to answer questions 3 Didn’t understand tool 2 No issue the tool would help with 2 Was ill at the time 2 Had already answered questions 1 Am already seeing someone 1 Didn’t understand, language barrier 1 I wasn’t offered questions <1 Felt caseworker not qualified <1 Difficult due to learning disabilities <1 My spouse did them <1 Other 2 Don’t remember 55

Note: Percent total exceeds 100% because respondents could identify multiple reasons. Source: W-2 Participant Survey.

A majority of respondents “didn’t remember” why they declined to use the tool.21 The other most frequently cited reasons for declining to use the tool were that the respondent felt that the BST would not be useful, or did not apply to their personal situation (Table 7). The specific responses included:

• Didn’t need services, wouldn’t help (8%) • Didn’t want to, didn’t interest me (6%) • No need to answer, questions didn’t apply (4%) • No issue the tool would help with (2%) • Am already seeing someone (1%)

20 The questions were open-ended in that the interviewer did not prompt the respondent. The respondent may have offered more than one reason for declining to complete the BST. For more information on the Participant Survey, see Appendix Note 2. 21 For cases that were newly opened in May 2003, the tool was first offered nearly two years before the survey.


Another somewhat smaller group declined to use the tool because of issues related to the administration of the BST or the tool itself. These included:

• Questions too personal (4%) • Didn’t feel able to answer questions (3%) • Don’t understand tool (2%) • Didn’t understand, language barrier (1%) • Felt caseworker not qualified (<1%) • Difficult due to learning disabilities (<1%)

A third group identified possible issues related to the timing of the administration of the tool, including:

• Didn’t have time (11%) • Was ill at the time (2%)

“Comfort” with the BST Survey respondents who completed the tool were asked “Or records show that you agreed to answer the questions on the Barrier Screening Tool…Is that true?” For those who responded “yes” or “don’t know,” the next question was about “comfort” with the BST: “Using a scale of 1 to 10, where ‘10’ means ‘very comfortable’ and ‘1’ means ‘very uncomfortable,’ how comfortable did you feel answering the questions (on the Barrier Screening Tool)?” A total of 1,960 participants answered that question and the average score was 8.14 on a 10-point scale. Black participants were less comfortable completing the tool than white participants. Participants who were in W-2 T placements at the time the screening was offered were less comfortable than those in CSJs. There was no significant difference between the other groups tested, including Milwaukee County and the balance of the state, English and Spanish speakers and the youngest and oldest age groups (Table 8).22

Table 8: Comfort Level of Participants Who Completed the Barrier Screening Tool,

February 2004 - April 2005 (n=1,960)

Groups

Average Score

Milwaukee 8.10 Balance of State 8.32 Black 8.03 * White 8.36 * W-2 T 8.01 * CSJ 8.31 * English 8.13 Spanish 8.35 18-22 years old 8.39 35 years old or more 8.18

Source: W-2 Participant Survey. *Indicates statistically significant at .05 level.

22 The Barrier Screening Tool portion of the telephone survey is conducted in English and Spanish.


SCREENING AND ASSESSMENT According to W-2 policy,23 if the BST results indicate potential employment barriers, the FEP is required to refer the participant to a formal assessment. That formal assessment must contain specified elements and it must be conducted by a qualified practitioner.

Potential Barriers The most frequently identified potential barriers among participants were medical/physical problems (54 percent) and trauma (52 percent). In 15 percent of the cases, medical/physical barriers were the only potential barrier identified. In most instances, a potential barrier was identified in combination with one or more other potential barriers. For example, while 52 percent (47 had trauma combo and 5 had trauma only) of participants had trauma identified as a potential barrier, only 5 percent identified trauma as their only potential barrier (Table 9).

Table 9: Percent of Participants with Potential Barriers, January - July 2004 (n=209)24

Potential Barrier Percent AODA 17 AODA only 2 AODA with something else 15 Brain Injury 13 Brain injury only 0 Brain injury with something else 13 Domestic Abuse 19 Domestic abuse only 2 Domestic abuse with something else 17 Learning Disability 21 Learning disability only 0.5 Learning disability with something else 21 Mental Health 31 Mental health only 1 Mental health with something else 30 Medical/Physical 54 Medical/physical only 15 Medical/physical with something else 39 Trauma 52 Trauma only 5 Trauma with something else 47

Source: Intensive case review

Quality of Assessments All participants in the intensive case reviews had at least one potential barrier identified where a formal assessment was required. These cases were divided into three categories in Table 10. The category “all assessments good” included participants with either a valid formal assessment for all potential barriers or a valid reason for not completing a formal assessment.25 The category “mixed good and inadequate assessments” included participants with at least one valid assessment and at least one

23 Wisconsin Works Screening and Assessment Policy, Department of Workforce Development, January 2003. 24 There were 210 cases in the sample. This table is based on the 209 cases with valid information. It excludes one case which had unusable results from the BST. 25 Valid reasons for not completing an assessment include those instances where the case closed before the assessment could be completed or where the participant refused the assessment.


assessment that was missing or failed to meet the policy criteria of a valid assessment.26 The category “all assessments inadequate” includes participants who lacked valid formal assessments for all potential barriers. Statewide, 56 percent of participants had all good assessments. While there appears to be a difference among the three areas of the state, the difference was not statistically significant. In 23 percent of the cases, all assessments for the case were inadequate. For 21 percent of the participants, there was a mix of good and inadequate assessments (Table 10). Table 10: Quality of Assessments for Cases with Potential Barriers Identified by Area of State,

January - July 2004 (n=181)27

Percent

Area of State

Number

All Assessments

Good

All Assessments

Inadequate

Mixed Good & Inadequate

Assessments Small Agencies 26 58 19 23Large Agencies 45 69 9 22Milwaukee County 110 50 30 20Statewide 181 56 23 21

Source: Intensive case review. A valid formal assessment was completed in 38 percent of the cases where AODA was identified as a potential barrier. In contrast, there was a valid assessment completed in only 5 percent of the cases where domestic abuse was identified as a potential barrier (Table 11).28

Table 11: Percent of Participants with Potential Barriers Identified and with a Formal Assessment Completed, January - July 2004 (n=193)29

Barrier

Number with Barriers

Identified

Percent with Good

Assessments AODA 38 38 Brain Injury 25 34 Domestic Abuse 55 5 Learning Disability 41 38 Medical/Physical 113 60 Mental Health 69 52 Trauma 106 39

Source: Intensive case review.

26 Removed from this analysis are cases that closed before an assessment could be completed, where the participant did not follow through with the assessment and where assessment information was not provided by the agency. 27 There were 210 cases in the sample. This table is based on the 181 cases with data on quality of assessments. It excludes cases where the participant did not follow through with an assessment (10 cases), the case closed before an assessment could be completed (8 cases) and where no assessment information was returned for the case (12 cases). The apparent one case discrepancy is due to weighting and rounding. For more information on the weighting mechanism, see Appendix Note 6. 28 It was sometimes noted by the reviewer that the event that caused the BST to identify domestic abuse as a potential barrier occurred in the distant past and was not a current concern. In interviews, FEPs sometimes reported that, when domestic abuse was identified as a potential barrier, they provided the participant with referral information but did not schedule a formal assessment. 29 There were 210 cases in the sample. This table is based on the 193 cases with valid information. It excludes those cases where no assessment information was returned for the case (12 cases) or cases where some combination of barrier or assessment information was missing (5 cases).


Assessment Providers Formal assessments must be completed by a qualified assessing agency or individual, such as a medical or mental health professional, or a Division of Vocational Rehabilitation counselor. The assessing agency or individual must demonstrate competency or completion of training in the appropriate field, be certified by an appropriate accreditation organization and understand the objectives of W-2 screening and assessment procedures.30 Case reviewers were asked whether the provider of the assessment met the criteria in the guidelines in the BST follow-up instructions. With the exception of domestic abuse barriers, providers met the guidelines in a high proportion of the cases (Table 12).

Table 12: Assessment Providers that Met the Guidelines in BST Instructions, January - July 2004 (n=193)31

Barrier

Number of Cases with

Provider Identified

Percent Who Met Guidelines

AODA 11 95 Brain Injury 7 100 Domestic Abuse 4 55 Learning Disability 15 88 Medical/Physical 42 89 Mental Health 24 68 Trauma 42 91


30 Wisconsin Works Screening and Assessment Policy, Department of Workforce Development, January 2003, pp. 3-4. 31 There were 210 cases in the sample. This table is based on the 193 cases with valid information. The numbers reported in the table are much smaller because they reflect only those cases with an assessment for a potential barrier and the provider of the assessment identified. For example, of the 193 cases analyzed, only 11 identified the provider that conducted an AODA formal assessment. In 95 percent of these 11 cases, the provider met the policy criteria. Participants with more than one assessment may appear more than once in the table.


CASE OUTCOMES

Formal assessments establish the extent and severity of a participant’s employment barriers and must include recommendations for appropriate activities, services or accommodations. These recommendations must be identified in the participant’s EP, unless there are valid reasons for not implementing them.

Recommendations in Assessments and EPs For cases where at least one valid formal assessment had been completed, reviewers identified the recommendations in the assessment and whether they were reflected in the activities, accommodations and services in the participant’s EP. Where the recommendations were not included in the EP, reviewers determined if there was a valid reason for not implementing the recommendations. The most frequently cited recommendations that appeared in the assessments were for counseling (38 percent) and medication (26 percent). The largest category of recommendations, “Other” (46 percent), included a variety of types of counseling, services or medical treatment, including AODA treatment, anger management, back surgery and SSI advocacy (Table 13).

Table 13: Percent of Participants with Recommendations in their Assessment among All Participants with at Least One Valid Assessment, January - July 2004 (n=90)32

Recommendation

Percent with Recommendations

in Assessment Physical therapy 18 Counseling 38 Reduced hours 22 Rest periods 4 Varied positions 5 Limited lifting 17 Medication 26 Training 9 Limit stress 14 Limit contact33 1 Other 46

Source: Intensive case review. For the 67 participants whose assessments had recommendations for specific activities, all the recommendations were reflected in the EP in 64 percent of the cases, some recommended activities were reflected in the EP in 17 percent of the cases and none of the recommended activities were in the EP in 19 percent of the cases. For the 34 participants whose assessments included a recommendation for an accommodation, all were reflected in the EP. Similarly, for the 13 cases where a service was recommended in the assessment, all were reflected in all EPs (Table 14, next page).

32 Only cases with at least one completed assessment are included in this analysis. Participants with more than one assessment may appear more than once in the table. 33 The term on the review instrument was “Limit contact with environmental elements.” This means that the participant’s activities may be limited in contact with other participants or the public.


Table 14: Percent of Cases with Specific Recommendations Reflected in Activities, Accommodations and Services, January- July, 200434

Percent Reflected in EP

Recommendations Activities (n=67)

Accommodations (n=34)

Services (n=13)

None 19% 0% 0%Some 17% 0% 0%All 64% 100% 100%


Assessments and Follow-Through Participant outcomes were categorized into three groups (Table 15). The category “good follow-through” included those cases that had good assessments for all potential barriers (or a valid reason for not completing a formal assessment) and at least one recommendation in the EP. The category “inadequate follow-through” included cases that were missing an assessment (or lacked a valid reason for failing to complete an assessment) or were missing recommendations in the EP. The “no agency action required” category included cases where there was a valid reason for not completing an assessment or when a participant failed to follow-through on a referral. Among the cases that did not require agency action, many closed before the assessment could be scheduled or completed. In other instances, the potential barrier was identified as late-term pregnancy, the participant refused the assessment or the participant did not appear for a scheduled assessment. Among the 154 cases where agency action was required, 53 percent (81 of 154) had inadequate follow-through. About half of the cases with inadequate follow-through (41 of 81) had no formal assessments in the file and lacked a valid reason for this omission. The other half had at least one assessment with recommendations in the EP (Tables 15 and 16).

Table 15: Distribution of Participant Outcomes (n=197) 35

All Cases

Cases Requiring Action

Outcome Number Percent Number PercentGood Follow-Through 73 37 73 47Inadequate Follow-Through 81 41 81 53No Agency Action Required 43 22 Total 197 100 154 100


34 Only cases where the assessment included at least one recommendation for an activity, accommodation or service are included in this analysis. Participants with more than one recommendation type may appear more than once in the table. 35 There were 210 cases in the sample. No information was returned from local agencies for 19 cases. This includes 11 cases for which the W-2 agency did not respond to the information request because of concerns about sharing confidential medical information with the DWS. In the eight remaining cases, local agencies did not respond to requests for the information. While the analysis is based on 191 actual cases, the data reported were weighted in order to reconstruct their true representation in the population of W-2 cases. The weighted number of cases reported in the table is 197. For more information on the sampling and weighting mechanism, see Appendix Note 6.


Detailed Follow-Through Among cases examined, 29 percent had good assessments for all potential barriers and recommendations in the EP. An additional 8 percent of the cases had either good assessments or valid reasons for not completing an assessment, and recommendations in the EP. Policy requires that participants referred through the BST receive a formal assessment for all identified barriers and that the assessments would lead to recommendations in the EP. Forty-one percent of all cases examined had inadequate follow-through. Breaking down this group into further detail, 21 percent had no formal assessments and no valid reason for their absence; 16 percent had a combination of good and inadequate (or missing) assessments and at least one recommendation in the EP; and 3 percent had both good and inadequate (or missing) assessments, and no recommendations in the EP.

Table 16: Detailed Outcomes for Intensive Case Review Sample (n=197)36

Outcome Number PercentGood Follow-Through All good assessments, recommendations in EP

58 29

All good assessments or valid reasons, recommendations in EP

15 8

Inadequate Follow-Through Good & inadequate assessments, recommendations in EP

32 16

All good assessments, no recommendations in EP

1 < 1

All good assessments or reasons, no recommendations in EP

2 1

Good and inadequate assessments, no recommendations in EP

5 3

No assessment, no valid reason 41 21No agency action required Participant refused or did not follow through 10 5 No assessment but valid reason 33 17Total 197 100

Source: Intensive case review

Impact on Case Planning The case plan activities of participants identified for possible inclusion in the W-2 Participant Survey were examined to determine if completion of the BST had an impact on assignment to activities.

36 There were 210 cases in the sample. No information was returned from local agencies for 19 cases. This includes 11 cases for which the W-2 agency did not respond to the information request because of concerns about sharing confidential medical information with the DWS. In the eight remaining cases, local agencies did not respond to requests for the information. While the analysis is based on 191 actual cases, the data reported were weighted in order to reconstruct their true representation in the population of W-2 cases. The weighted number of cases reported in the table is 197. For more information on the sampling and weighting mechanism, see Appendix Note 6.


In Milwaukee County, those who completed a screening were more likely than those who declined a screening to be participating in the following activities (Table 17):

Assessment: AODA assessment Domestic abuse assessment Mental health assessment Physical assessment

Treatment: AODA counseling Mental health counseling Physical rehabilitation

Education: Adult basic education Literacy

Work related: Employment counseling

Other: Personal care Personal development

In Milwaukee County, those who completed a screening were less likely than those who declined a screening to be participating in the following components: Work related: Employment search

Job readiness

Other: Parenting life skills In the balance of the state, those who completed a screening were more likely than those who declined a screening to be participating in the following components: Assessment: Domestic abuse assessment

Mental health assessment

Treatment: Employment counseling Mental health counseling

Work related: Employment search Job readiness

Other: Personal development Parenting life skills

Participants who completed the screening tended to be more involved in activities associated with assessment and counseling, both in Milwaukee and in the balance of the state. Participants who completed the BST in Milwaukee were more likely to be in AODA assessment, physical assessment and AODA counseling than those who completed a screening in the balance of the state. Participants who completed the BST in the balance of the state were more likely to be in job readiness and employment search than those who completed a screening in the Milwaukee County.


Table 17: Percent of Screenings Completed by Activities in Milwaukee County and Balance of

State, February 2004 Through April 2005 (n=15,673)

Milwaukee Balance of State Current Activity Percent

Completed Percent Declined

Percent Completed

Percent Declined

Assessment AODA assessment 1.2 ** 0.7 ** 0.8 0.3 Domestic abuse assessment 1.1 ** 0.7 ** 2.5 ** 0.6 ** Learning disability assessment 1.3 1.2 3.0 4.0 Mental health assessment 5.7 ** 3.7 ** 5.3 ** 2.4 ** Physical assessment 7.4 ** 6.1 ** 9.8 10.2 Treatment AODA counseling 3.9 ** 2.7 ** 3.8 2.7 Employment counseling 22.8 ** 2.5 ** 9.9 ** 4.0 ** Mental health counseling 14.0 ** 9.3 ** 26.1 ** 15.8 ** Physical rehabilitation 5.3 * 4.5 * 6.0 8.3 Education Adult basic education 33.6 ** 30.5 ** 19.6 19.1 Drivers education 0 0 0 0 ESL 1.8 1.7 1.4 1.9 GED 4.7 4.4 5.4 5.3 High school equivalency 0.5 0.7 1.8 1.4 Literacy 1.3 ** 0.5 ** 2.5 1.9 Regular school 1.7 * 3.1 * 1.1 1.4 Technical college 1.7 ** 2.6 ** 0.5 0.5 Work related Career counseling 0.1 0.1 1.0 0.6 Employer intervention 0 0 0 0 Employment search 45.9 ** 48.7 ** 35.0 ** 25.8 ** FSET work experience 0 0 0 0 Job skills 5.0 5.0 3.5 2.7 Job readiness 8.9 ** 11.4 ** 22.4 ** 14.2 ** Job retention 0 0 1.6 * 0.3 * Occupational assessment 3.7 3.1 3.8 4.1 OJT 0 0 0 0 Trial job-private 0.4 0.3 0.1 0 Trial job-public 0 0 0 0 Work experience 47.8 47.3 23.0 20.3 Other Housing 2.4 2.1 3.8 2.7 Legal issues 0.7 ** 0.3 ** 1.3 1.0 Mentor coach 0 0 0.5 0.5 Parenting life skills 10.2 ** 13.6 ** 21.5 ** 16.6 ** Personal care 18.5 ** 16.1 ** 40.9 43.3 Personal development 7.7 ** 5.3 ** 21.1 ** 10.7 ** SSI/SSDI advocacy 0.5 0.5 1.8 2.2

Source: CARES and Work Programs Warehouse, WP038 extract. *Indicates statistically significant at .05 level. ** Indicates statistically significance at .01 level.


SUMMARY OF KEY FINDINGS

Completion Rates

About half (49 percent) of CSJ and W-2 T participants completed the Barrier Screening Tool. Those in W-2 T placements were more likely than those in CSJ placements to complete the BST (53% vs. 44% statewide).

There was considerable variation in the BST completion rate by agency. Among the 28

agencies with 100 or more participants eligible for a BST screening, completion rates varied from 19 to 99 percent.

There was considerable variation in the BST completion rates of FEPs within agencies. Among

the eight agencies that had one or more FEPs with 40 or more cases, completion rates varied from 0 to 100 percent.

The longer a participant had been in the W-2 program, the more likely they were to have

completed the BST.

In the balance of state, racial differences in the proportion of participants with completed screenings occurred for both CSJ and W-2 T placements. Among CSJ participants, 66 percent of whites had been screened, compared to 43 percent of blacks. Among W-2 T participants, 59 percent of whites had completed a screening while 46 percent of blacks had been screened. The differences between Hispanics and whites in the balance of state were less pronounced. There were no statistically significant racial differences in Milwaukee County.

The most frequently cited reason in the participant survey for participants declining a screening

was that they “did not have time.” This reason was cited by 11 percent of those who declined to be screened.

The second most frequently cited reason in the participant survey for declining a screening was

that the participant did not feel that the BST would be useful. For example, 8 percent of those who declined said they did not need the services and 6 percent said they were not interested in receiving a screening.

In the participant survey, a smaller group of participants stated that they declined to be

screened because of reasons related to the administration of the BST or the tool itself. For example, 4 percent of participants said that the questions were too personal and 2 percent said they did not understand the tool.

In the participant survey, participants who completed the BST indicated that they felt

“comfortable” with it, giving the tool an average score of 8.1 on a scale of 1-10 (10 = most comfortable). Black (vs. white) participants and participants in W-2 T (vs. CSJ) placements were less likely to say they were “comfortable” with the tool (8.0 blacks vs. 8.4 whites and 8.0 W-2 T vs. 8.3 CSJ participants).


Screening and Assessment

The most common types of potential employment barriers identified by the BST were “medical or physical barriers” and “trauma.” Potential medical/physical barriers were cited in 54 percent of the cases reviewed; trauma was cited in 52 percent of the cases. In most cases, a potential barrier was identified in combination with other potential barriers.

In those instances where the BST indicated that the participant needed a formal assessment, 56

percent of participants either had a formal assessment that met policy criteria or a valid reason for not completing an assessment.

Twenty-three percent of participants had inadequate formal assessments and 21 percent had a

combination of good and inadequate formal assessments.

Participants with potential medical/physical or mental health barriers were most likely to have had a “good assessment.” In 60 percent of the cases where medical/physical potential barriers were identified, the case had a “good assessment.” Fifty-two percent of the participants with potential mental health barriers had “good assessments.” In contrast, a “good assessment” occurred in only 5 percent of the cases where the potential barrier was domestic abuse.

Most of the providers that conducted formal assessments met the “qualified assessing agency

or individual” criteria set by W-2 policy criteria. For example, 95 percent of the providers who conducted AODA assessments were qualified.

Case Outcomes

In those instances where action was required on the part of the agency: 47 percent of the participants had a required formal assessment with

recommendations that addressed all potential barriers in their employability plan, or the agency had a valid reason for not completing the assessment.

53 percent of the participants either lacked a required formal assessment or a

recommendation from the assessment was not found in the participant’s employability plan.

Participants who completed the BST were more likely than those who declined to be assigned

to activities related to assessments, counseling and rehabilitation. For example, in the balance of the state, 26 percent of those who completed the BST were in the “mental health counseling” while only 16 percent of those who declined were engaged in that activity.


Appendix: Research Notes

Note 1: Measuring the Accuracy of the Barrier Screening Tool Ensuring the accuracy of the BST improves both the effectiveness and efficiency of the W-2 program. If the tool identifies potential barriers that are not confirmed by subsequent formal assessments (i.e., false positive results), assessment resources are wasted. If the tool fails to detect actual employment barriers (i.e., false negative results), case plans may not effectively ensure that participants receive those services necessary to move to self-sufficiency. The initial research design included a mechanism to identify false positive results. The plan was to use the intensive case review sample (210 cases) for which the BST identified a potential barrier to determine if the formal assessments confirmed the BST finding. This approach did not prove feasible because the number of cases with formal assessments was too small to support a reliable analysis. For example, of the 210 cases in the sample, only nine had a good assessment for a potential brain injury barrier and only three had a good assessment for a potential domestic abuse barrier. To conduct an adequate analysis, it would have been necessary to have a minimum of 30 cases with completed and good formal assessments for each of the seven barrier types identified by the tool. While the study did not directly measure the accuracy of the BST regarding false positive results, there are some indications of this concept in the findings. Table 16, for example, describes the outcomes of cases where the BST identified at least one potential barrier. A total of 53 percent of the cases (first three specific outcomes listed in the table) where the BST identified a potential barrier had a least one recommendation from a formal assessment incorporated into the W-2 participant’s case plan. If only those cases where some agency action was required are examined (i.e., when cases where there was a valid reason for not conducting an assessment are removed), 68 percent of the cases had at least one confirmed barrier. The findings from Table 17 also provide an indication of the BST’s ability to accurately identify barriers. This table compares the assigned activities of W-2 participants who were administered the BST with those who declined to complete the tool. For example, while 14 percent of Milwaukee County participants who were administered the tool were subsequently assigned to a mental health counseling component, only 9 percent of those who did not use the tool were assigned to that component. These results suggest that use of the tool does result in the identification of barriers that would not have been identified without use of the tool. The initial research design did not include a mechanism to identify false negative results as this would have required resources well beyond those available for the study. Such an analysis would have first entailed independently verifying the existence of a barrier for a set of cases. The findings for these cases would then have had to be compared to BST findings to confirm if the BST also detected the barrier. This approach would have required that formal assessments be conducted for each of the seven barrier types until a minimum of 30 cases with a barrier were identified for each barrier type. The BST would then have had to be applied to these cases to determine if it accurately identified the actual barriers previously identified by the formal assessments. While not definitively addressing the accuracy issue, the fact that the BST findings were followed-up in the case plan recommendations and in assigned activities suggests that the tool can accurately identify actual barriers. However, the question of whether the BST misses actual barriers is not addressed.


Note 2: Telephone Survey Questions The W-2/FSET Customer Satisfaction Survey telephone survey instrument was designed to gauge satisfaction with aspects of the W-2 program. That survey was modified beginning in February 2004 to include a component to identify participants’ experience with the Barrier Screening Tool. That component is identified in this study as the W-2 Participant Survey. Since participants may receive services from different providers within agencies, the first part of the survey was designed to identify the specific provider the respondent was being asked to evaluate. The survey then asks a combination of seven questions to determine satisfaction with aspects of the program. Questions are phrased to elicit answers on a 1 to 10 scale, “where ‘10’ means ‘strongly agree’ and ‘1’ means ‘strongly disagree.’” The combination of questions that are asked of any individual respondent depends on answers to previous questions and the “skip logic” of the instrument. But all respondents are asked about their “overall” satisfaction with the service they received from the W-2 provider. After the W-2 satisfaction questions are completed, the survey gathers information on participants’ experience with the BST. First there is a description of the BST, to remind respondents of the topic of the subsequent questions. The script also assures the respondent that the surveyor has no information about the participant’s responses to the BST questions and that information collected is completely confidential. The first question in the series is:

“Our records show that you (declined or agreed) to answer the questions on the Barrier Screening Tool on (insert date). Is that correct?”

If the respondent says “yes” or “yes, but not on that date” or “don’t know,” the survey proceeds to the next question. If the respondent says “no,” the survey is terminated. For those who declined to complete the BST, the next question is:

“Why did you decline to answer the questions?” The answers are keyed and categorized. The initial categories were created on the basis of pre-test data. New categories have been added as multiple answers occurred in the “other” category. At the time of this analysis, there were 18 categories of responses, in addition to the “other” category. For those who completed the BST, the next question is:

“Using a scale of 1 to 10, where ‘10’ means ‘very comfortable’ and ‘1’ means ‘very uncomfortable,’ how comfortable did you feel answering the questions?”

Respondents who were not screened out by the first BST question were asked:

“Do you have any additional comments about the Barrier Screening Tool?” Their answers are keyed and categorized with a procedure similar to that of the other open-ended questions.


The survey script closes with an additional assurance that the respondent’s answers are confidential and will not affect services in any way.

Note 3: Telephone Survey Sampling and Weighting There is an unequal distribution of W-2 cases across the state, with nearly 87 percent of the cases in Milwaukee County and fewer than 10 open cases in several small agencies in the balance of the state. If the telephone survey used a simple statewide random sample, either there would be too few cases to generalize to smaller agencies or the statewide sample would be much larger and produce more cases in Milwaukee County than would be needed. Therefore, the survey uses a stratified sample, with proportionally fewer cases drawn from Milwaukee County and large balance of state agencies and proportionally more cases from “medium” and “small” agencies in the balance of the state. Because of the sample stratification, if one were to use these survey results in their original form, smaller agencies would disproportionately influence the results and cases in Milwaukee County and large balance of state agencies would be underrepresented. Therefore, the analysis weights cases from the four sampling tiers to reconstruct their proportionate representation in the statewide W-2 population. Weighting factors were calculated to ensure that the proportion of cases in the analysis from each agency reflected the proportion of actual cases from that agency in the statewide population. The weighting factor was determined by dividing the proportion of the case in the population for each tier by the proportion of the cases in the sample for that tier. Weighting factors are as follows: small agencies were multiplied by .433, medium sized agencies by .473, large agencies by .598 and Milwaukee County agencies by 1.209. This method not only corrects for the stratification of the sample, but it produces numbers in tables that approximates the actual number of cases in the population for each geographic area.

Note 4: Administrative Data There are two sources of administrative data used in this analysis. The first source was the CARES information system. That system is designed primarily to determine eligibility and to aid FEPs in case management. It also contains historical data for each participant. The second source was the Work Programs Warehouse. This system includes detailed information on participants’ employability plans, including the “components’ or activities in which the participant was enrolled. The WP038 extract was produced monthly from data in the Warehouse and includes whether or not the participant has completed the BST and the participant’s components as of the end of the reporting month. Each month, selected information from these two sources was merged in order to prepare the sample for the Participant Survey. Case, agency and demographic information were drawn from the CARES. This includes information on how long the participant has been in a CSJ or W-2 T placement and on the W-2 program as of the month of the sample. Activities and whether the participant had completed the BST were drawn from the WP038. These data were matched on the participants’ Personal Identifying Number (PIN). This merged data set was later merged with the results form the Participant Survey, including the responses to the questions about the BST.


The resulting data set has a wide range of uses in the current analysis and may be useful for additional related research. Note 5: Methods for Intensive Case Review The intensive case review was conducted on a sample of 210 cases that had at least one potential barrier identified by the BST and were identified as having been referred for at lest one formal assessment. A case review team, made up of seven DWS Regional Office and Central Office staff, was selected. In order to ensure consistency among reviewers, the project developed a case review instrument. To pre-test the review procedure and instrument, the review team analyzed the same sub-set of cases, entered results on the review instrument and then met to ensure consistency in interpretation and responses. The review instrument required responses for each of the potential barriers addressed by the BST. These include:

• Medical and physical limitations • Alcohol and other drug (AODA) barriers • Domestic abuse • Learning/Cognitive needs • Evidence of trauma • Mental health issues • Evidence of traumatic brain injury

Under each barrier the reviewer indicated:

• Was the barrier identified in the BST for the participant? • If yes, did the agency submit documentation for the barrier? • If so, did the documentation meet the W-2 definition of a formal assessment? • If no documentation was submitted, does CARES or the BST include a valid reason for not

completing an assessment? • The number of months between BST completion and assessment completion.

Policy requires that a formal assessment include all of the following six elements:

• Personal conditions/diagnosis that impact ability to function in activities of daily living and the ability to perform work

• General aptitude/cognitive level (for cases referred for Learning Disability) • Areas of strength • Areas of deficit • Range of recommended accommodations/assistive technology for the participant’s

employability plan • Summary of findings, including rationale for any disability determinations/diagnosis; prognosis;

and recommendations for additional services, as appropriate The next section of the case review instrument addressed the provider the agency used to conduct the formal assessment. As indicated, the BST recommends types and qualifications of providers for each barrier type. Reviewers were asked:

• Do the providers match the ones recommended in the BST follow-up steps? • What type of provider was used?


The next section asked for the types of recommendations that were made by the assessment. Note that policy states that a formal assessment should include specific recommendations for addressing confirmed barriers. Reviewers could indicate all of the following:

• Physical therapy • Counseling • Reduced hours • Rest periods • Very position • Limit lifting • Medication • Training • Limit stress • Limit contact with environmental elements • Other recommendations (specified)

Reviewers are then asked

• Are the recommendations in the assessment reflected in the activities assigned? • Are the recommendations in the assessment reflected in the additional services provided? • Is there a legitimate reason for not implementing any of the recommendations on the

assessment in the Employability Plan? The reviewer could also enter comments at various points throughout the review instrument.

Note 6: Intensive Case Review Sampling and Weighting There is an unequal distribution of W-2 cases across the state, with nearly 87 percent of the cases are in Milwaukee County and fewer than 10 open cases in several agencies. If the sample were a simple statewide random sample, either there would be too few cases to generalize to smaller agencies or the statewide sample would be much larger and produce more cases in Milwaukee County than would be needed. Therefore, the survey uses a stratified sample, with proportionally fewer cases drawn from Milwaukee County and large balance of state agencies and proportionally more cases from “small” agencies in the balance of the state. “Milwaukee” includes three agencies operating W-2 in Milwaukee County: MAXIMUS, OIC and UMOS. “Large agencies” include Brown, Kenosha, Outagamie, Racine and Rock Counties and Capitol Consortium, which includes Dane County. “Balance of state” includes 19 county agencies and seven consortia. If one were to use these survey results in their original form, smaller agencies would disproportionately influence the results and cases in Milwaukee County and large balance of state agencies would be underrepresented. Therefore, the analysis weights cases from the three sampling tiers to reconstruct their proportionate representation in the statewide W-2 population. Weighting factors were calculated to ensure that the proportion of cases in the analysis from each agency reflect the proportion of actual cases from that agency in the statewide population. The weighting factor was determined by dividing the proportion of the cases in the population for each tier by the proportion of the cases in the sample for that tier. Weighting factors are as follows: Small balance of state agencies are multiplied by 165, large agencies by .678 and Milwaukee County agencies by 1.824. This method not only corrects for the stratification of the sample, but it produces numbers in tables that approximates the actual number of cases in the population for each geographic area.

Barrier Screening and Assessment Evaluation

Documents

Transcript of Barrier Screening and Assessment Evaluation