Comparison of Alternative Employment Services for Persons with Serious Mental Illness: Effects on Patient-Reported Mental Health Functional Status, Substance Use, and Arrests.

Salkever D, Steinwachs DM, Abrams M, Gibbons B, Baier K, Stuart EA, Skinner EA, Wu AW, Salzberg C.

ABSTRACT

Objective:  To evaluate the effect of initiation of evidence-based supported employment services (Individual Placement and Support-IPS) for persons with serious mental illnesses (SMI) and qualifying for state or federal disability on patient-reported mental health functional status, substance use, and arrests.Data Sources: Maryland Medicaid data was matched to data from the Division of Rehabilitative Services and the Maryland Public Mental Health System to identify the study groups.   Data from Maryland Medicaid, Maryland Mental Hygiene Administration (MHA) and the Maryland Division of Rehabilitation Services (DORS) were used to select study subjects, to identify initiation dates of employment services that occurred between Sept 1, 2006 and Dec 31, 2007, and to assign each study subject to a treatment group. Data from MHA's Outcomes Measurement System (OMS) were used to measure baseline levels of the self-reported outcome measures and follow-up levels of these same measures.Study Design:  An intent-to-treat design is used to compare three groups of Medicaid persons with SMI: those initiating evidence-based IPS (placing people in competitive employment with support services) meeting fidelity criteria (IPS-F), those initiating supported employment services without evidence of fidelity to IPS (SE-NF), and those initiating traditional vocational services, emphasizing training that is expected to lead to competitive employment (TVS).  Propensity score weights are used in comparing the three groups on a number of baseline variables from the Medicaid claims data measured over the 12 months prior to the assigned initiation date of employment services for each study subject. Outcome variables were measured from OMS data for follow-up periods of 12-months, 24-months, and 36-months following each subject’s date of employment services initiation.

Data Extraction Methods: Data sets were provided by the Maryland Medicaid Program, the Maryland DORS and the Maryland MHA's Public Mental Health System.

Principal Findings: Results based on a variety of regression methods with inverse-probability propensity weights, showed inconsistent patterns of differences among the three treatment groups. Failure to reject null hypotheses of no difference, however, must be viewed as tentative due mainly to the relatively small number of study subjects with available OMS dependent variable data in each of the three study groups.

 Conclusions: The tentative conclusion is that the 3 study groups did not differ on these patient-reported outcomes. Further assessment with additional data is, however, indicated because of sample size and OMS data availability issues. 

Keywords: Supported employment; individual Placement and Support (IPS); Patient-reported mental health outcomes.

Acknowledgements: This research was supported by Contract No. HHSA290201000009I from the Agency for Healthcare Research and Quality, US Department of Health and Human Services, as part of the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Program.

1

INTRODUCTION

Nationally, it is estimated that persons with serious mental illnesses (SMI) and serious functional impairment comprise 4.1% of U.S. adults (SAMHSA, 2013). Recent data also suggest that over 80% of persons with schizophrenia being treated in the community are not employed (Salkever, et al., 2007). Moreover, employment is cited as a goal by almost 2/3 of all persons with SMI in the U.S. public mental health system (Bond and Drake, 2014; Bedell et al. 1998; Frounfelker et al. 2011; McQuilkenet al. 2003; Mueser, Salyers and Mueser, 2001; Ramsay et al. 2011; Rogers et al. 1991; Watkins et al. 2011; Woltmann 2009). In contrast studies show that only about 15% report any current paid employment (Bond and Drake, 2014; Lindamer et al. 2003; Pandiani and Leno 2012; Perkins and Rinaldi 2002; Rosenheck et al. 2006; Salkever et al. 2007).

Given this large gap between those wanting to work and those actually working, increasing paid employment has been viewed as a principal recovery goal for rehabilitation programs for those with SMI. Research shows that Individual Placement and Support (IPS), an evidence-based supported employment intervention, can achieve competitive employment positions for half or more of persons with SMI desiring work (Bond, Drake, and Becker, 2008). In randomized trials of IPS vs. traditional rehabilitation programs that are not vocationally oriented, IPS has shown strong positive effects in helping patients to gain paid (mainly competitive) employment (Marshall et al., 2014). As Salkever (2010, 2013) has noted, comparisons with more traditional alternative vocationally-oriented services, such as an enhanced vocational rehabilitation (EVR) (Drake et al., 1999) or a diversified placement approach (DPA) (Bond et al. 2007) show significantly higher competitive employment rates for IPS but similar overall paid employment rates (including agency jobs and other non-competitive paid employment) for EVR and DPA.

In contrast to clear results about IPS effectiveness in achieving competitive employment, IPS effectiveness in achieving nonvocational outcomes has been the subject of debate in the recent literature. A recent article by two of the leading authorities in the field (Bond and Drake, 2014) forcefully argues that IPS does have positive nonvocational effects stemming from the combination of (1) positive IPS effects on employment and (2) the positive effects of employment on nonvocational outcomes. Their argument is as follows:

IPS is a highly effective approach to vocational rehabilitation…(Becker et al. 2011)….systematic reviews conclude that IPS enhances vocational outcomes (Bond 2004; Bond et al. 2008; Burns et al. 2007; Crowther et al. 2001; Dixon et al. 2010; Twamley et al. 2003). About two-thirds of IPS participants succeed in competitive employment…(Bond et al. 2012a) and sustaining employment for years (Becker et al. 2007b; Salyers et al. 2004)….

People who obtain competitive employment through IPS enhance their income, self-esteem, quality of life, social inclusion, and control of symptoms (Bond et al. 2001; Burns et al. 2009; Kukla et al.2012; Mueser et al. 1997;Turner et al. 2012). These enhancements to well-being persist at 10-year follow-ups (Becker et al. 2007b; Salyers et al. 2004). People with SMI often report that IPS is good treatment and central to their recovery (Bailey 1998;Becker et al. 2007b; Strickler et al. 2009)….employment leads to decreased mental health costs (Bond et al. 1995; Burns et al. 2009;Clark 1998; Henry et al. 2004; Latimer 2001; Perkins et al. 2005; Rogers et al. 1995; Schneider et al. 2009). Long-term cost reductions appear to be even greater (Bush et al. 2009).

2

An alternative view is expressed in a recent article by Kukla and Bond (2013), who report on a two-year randomized trial of an IPS program vs. a DPA program. The nonvocational outcomes studied pertained to symptoms, psychiatric hospitalizations, quality of life, and social networks. They summarize their results and conclusion as follows:

Although the total sample showed improvement in several nonvocational domains over time, there were largely no differences between groups in nonvocational outcomes at follow-up or in their rates of improvement over time…Participation in supported employment alone is not sufficient to positively impact most nonvocational outcomes in people with severe mental illness.

To further reinforce Kukla and Bond’s (2013) alternative view that questions IPS effects on these nonvocational outcomes, we have previously noted (Salkever 2010 and 2013) that evidence of a positive contemporaneous association between employment status and these nonvocational measures should not be interpreted as a causal influence of employment because of the strong likelihood of unmeasured selection factors that are positively correlated with both employment and the nonvocational measures. This same concern about inferring causation is in fact explicitly noted in Kukla et al. (2012), Bond et al. (2001), Burns et al. (2009), and Mueser et al. (1997), all being papers cited by Bond and Drake in support of their argument for the nonvocational effects of IPS.

Given the divergence of views (e.g., between Kukla and Bond (2013) and Bond and Drake (2014)) about the effects of IPS on nonvocational outcomes, we undertook our own analyses that expanded the range of the research in several ways. First, it presented comparisons between certified fidelity-compliant IPS vs. the alternative “treatments” of (1) SE that was not certified as fidelity compliant and (2) a range of other “traditional” vocational rehabilitation services provide by a state vocational rehabilitation agency. Second, these analyses involved new outcome measures. In the present analysis, we study patient-reported outcome measures from a new statewide Outcomes Measurement System (OMS) used in the state of Maryland for all patients served under Maryland’s Public Mental Health System-managed specialty mental health care program. In a companion study, state Medicaid claims data are used to measure IPS vs. alternative treatment effects on measures of continuity and coordination of treatment services, testing the hypothesis that continuity and coordination of these services for persons with severe mental illness may be influenced by IPS efforts to coordinate vocational services with other mental health treatment services (Steinwachs, et al., 2014).

Traditional employment services in Maryland differ from the IPS supported employment approach in that they emphasize training and non-competitive job experiences (e.g., enclave employment) prior to or instead of placement in a competitive employment environment.

In 2002, the Maryland Mental Hygiene Administration implemented a statewide program to provide evidence-based employment services for all persons with SMI wanting to work, largely replacing traditional employment training and placement programs. For persons with SMI, competitive employment has been recognized as key outcome of the recovery process (President’s New Freedom Commission on Mental Health (2003); US Surgeon General (1999); National Institute of Mental Health (1999)). We hypothesized that the IPS employment intervention would have desirable effects on non-employment outcomes. Specifically, we expect that the IPS employment intervention would have positive effects on individuals’ self-reported functional status, and would reduce the frequency of negative outcomes such as substance use and arrests.

In this study, comparisons are made among consumers meeting SMI and continuous Medicaid enrollment criteria who entered either an IPS program meeting established fidelity standards (IPS-F), a supported employment program without fidelity (SE-NF) or a traditional

3

employment services program (TVS). We hypothesized that IPS-F would be associated with improved coordination of services as compared to TVS and possibly similar to SE-NF employment program outcomes. IPS-F is expected to:

Increase the positivity of each of 5 patient-reported ratings of their functional status with respect to 1) participating in meaningful activities, 2) taking care of personal needs, 3) coping with problems, 4) doing things the patient wants to do, 5) reducing the perceived burden of mental illness symptoms.

Reduce the frequency of self-reported alcohol use, drug use, and substance abuse. Reduce the frequency of self-reported arrests.

BACKGROUND

Individual Placement and Support (IPS) is a version of supported employment that uses a “place and train” approach (Wehman and Moon, 1988) in contrast to traditional practices in which extensive pre-vocational training is provided prior to job placement (“train and place” model). The supported employment model was adapted to meet the needs of persons with SMI and multiple randomized controlled trials have established its efficacy in improvement employment outcomes (Bond, Drake and Becker, 2008). In nine RCTs conducted in eight states (including Maryland), IPS employment rates ranged from 27% to >75%, compared to controls that achieved 7% - 40% IPS. A recent Cochrane Review of 14 RCTs in the U.S. and elsewhere compared IPS with other programs (largely emphasizing training prior to work placement) and focused on the employment outcomes of time to obtain competitive paid job sand duration of employment. IPS programs consistently outperformed traditional employment services (Kinoshita, 2013; Marshall et al., 2014)).

According to the Dartmouth Psychiatric Research Center (2011), IPS is based on 8 principles: (1) involvement in competitive employment from the outset instead of placement in a sheltered or non-competitive work setting, (2) all consumers (the person with the SMI) desiring to work are eligible for IPS, (3) rapid job search (placement frequently within a month), (4) integration of mental health and employment services, (5) attention to consumer preference in the job search, (6) time-unlimited individualized job supports, and (7) personalized benefits counseling, and (8) systematic job development.

DATA AND METHODS

Study Population - The sample selection criteria identified persons with serious mental illness (SMI) diagnoses and state or federal certified disability as identified in the Medicaid enrollment files with at least 10 months per year of Medicaid enrollment over a 4-5 year period, 2 years prior and 3 years post initiation of an employment services intervention. Persons were eligible for inclusion if their first employment services contact occurred between September 1, 2006, and December 31, 2007, which marked the beginning of statewide implementation of an Outcomes Measurement System (OMS) that is used by Maryland’s Public Mental Health System (PMHS) to track the progress of the patients that they serve. Persons were included in the study if they met all the following selection criteria. The figures in parentheses show the numbers remaining after completion of the step.

1. Had a serious mental illness diagnosis in calendar years 2006 or 2007 (n=131,820)2. Was 20-63 years of age between 9/1/2006 and 12/31/2007) (n=79,759) 3. Eligible for disability benefits according to state or federal criteria during the period 7/1/2006

to 12/31/2007 (n=35,253)4. Was not a Medicare enrollee at initiation of employment services (n=22,556), given that

Medicare data were not available for this investigation4

5. Had at least 10 months of Medicaid enrollment in each 12 month period two years before and two years after the employment services initiation period, 9/1/2006 and 12/31/2007 (n=16,321)

6. Had a matching DORS or PMHS record indicating initiation of employment services between 9/1/2006 and 12/31/2007 (n=618)

7. Did not receive employment services in the year before the initiation period and lived in Maryland during the same period (n=433)

Serious mental illness was defined as the presence of at least one of the following International Classification of Diseases Version 9 (ICD-9) diagnostic codes, as recorded in the Medicaid data:

295.xx Schizophrenic disorders296.xx Episodic mood disorders (mania, bipolar, major depression)297.xx Delusional Disorders298.xx Other Nonorganic Psychoses299.8x, 299.9xOther pervasive developmental disorders 300.xx Anxiety states301.xx Personality disorders302.xx Sexual and gender identity disorders (excluding 302.7x,

psychosexual dysfunction)310.xx Specific nonpsychotic mental disorders due to brain damage311.xx Depressive disorders, not elsewhere classified

Classification of subjects in three treatment groups - Individuals in our sample of 433 were classified into one of three mutually exclusive groups based on the type of service they received at the first employment service received between 9/1/2006 and 12/31/2007:

Individual Placement and Support (IPS-F) programs certified as meeting the criteria for fidelity to the IPS model at time of service initiation (N=136);

Supported employment services that place clients in competitive employment (SE-NF); these programs may or may not follow the IPS model and were not certified as meeting the IPS fidelity criteria at time of service initiation (N=171); and

Traditional vocational services (TVS) which offer a “train and place” approach (N=126).

Outcome Measurement System Data - The Outcome Measurement System (OMS) collects self-reported data from persons in Maryland’s PMHS on mental health status and related subject areas such as substance use, employment status, and arrests. The reported interview data were obtained by clinical provider staff who administered the interview to the PMHS patients. (Note that dual-eligible patients were not included in the OMS interview process.)

The study start date is September 1, 2006, which is also the start date for OMS implementation in the PMHS system. OMS data were obtained for the study initiation period, September 2006 through December 2008, as well as for three years thereafter, 2009 through 2011. Persons receiving outpatient mental health services covered by the PMHS who are not dually eligible should have one or more OMS records as long as they are not receiving services from an individual practitioner. Completed OMS questionnaires are submitted by Outpatient Mental Health Centers (OMHC’s), Federally Qualified Health Centers (FQHC’s) and hospital based clinics (known as HSCRC’s). The OMS questionnaire is reported approximately every 6 months for persons in continuous treatment and helps track patient progress for providers as well as for the overall PMHS.

5

The number of persons in our study sample who have OMS records depends on whether the person received mental health services covered by the PMHS in the study period and on whether persons received services at an OMHC, FQHC, or HSCRC. Of our 433 study subjects, 52 persons did not have any OMS records. Across treatment groups, the numbers without any OMS records are 10 (6.3%) for the IPS-F group, 13 (8.8%) for the SE-NF group, and 29 (23.0%) from the TVS group. For those that have at least 1 OMS record, persistence in treatment helps determine the number of OMS records each person has. Table 1 below displays descriptive statistics on the number of records per person in the study period (excluding the 52 cases with 0 records). The TVS group has a lower average number of records (7.1) than both the IPS-F group (10.1 records) and the SE-NF group (10.0).

Table 1: Total OMS records per person in Study Period

N mean sd min maxN with >0 OMS

RecordsMean # OMS Records if >0

IPS-F 159 9.6 4.9 0 20 149 10.2SE-NF 148 9.3 5.3 0 17 135 10.2TVS 126 5.4 5.1 0 17 97 7.1

While many persons in the PMHS who are persistent in treatment have OMS records at regular intervals of approximately 6 months, this is not the case for a substantial portion of this population who are in and out of treatment. There is also an issue with matching the OMS records with the initiation date. Table 2 below gives the mean and standard deviation for the number of days between the initiation date and the closest OMS record, before or after the initiation date. The average for the IPS-F and the SE-NF groups are 101 days and 136 days while the TVS group is even higher at 308 days. These are highly skewed distributions with large outliers for persons who had one or more records much later or in some cases much earlier than their initiation date.

Table 2: Descriptive Statistics for OMS-Initiation Date Time Intervals, by treatment groupIPS-F SE-NF TVS

nMeanDate

(# of days)

sd nMeanDate

(# of days)

sd nMeanDate

(# of days)

sd

initiation date 1594-May

07 132 148 25-Apr-07 139 12618-Feb-

07 110

OMS Baseline record date 102

17-May 07 130 88

18-May 07 144 52

2-Mar-07 116

# of days between initiation date and closest OMS record 149 101 163 135 136 234 97 308 488

1 yr. post-inititation date (initiation date+365) 159

3-May-

08 132 148 24-Apr-08 139 12618-Feb-

08 110OMS 1 yr. Outcome record date 106

21-Apr-08 144 103 21-Apr-08 158 56

12-Feb-08 122

2 yr. post-initiation date (imitation date+730) 159

3-May-

09 132 148 24-Apr-09 139 12617-Feb-

09 110OMS 2 yr. Outcome record date 111

27-Apr-09 146 91 15-Apr-09 152 50

11-Feb-09 117

3 yr. post-initiation date 159 3- 132 148 24-Apr-10 139 126 17-Feb- 1106

(imitation date+1,095)May-

10 10

OMS 3 yr. Outcome record date 102

3-May-

10 144 8611-May-

10 147 4023-Feb-

10 106# of days between baseline OMS record date and OMS 1-yr. follow-up record date 75 371 61 74 367 65 35 372 49# of days between baseline OMS record date and OMS 2-yr. follow-up record date 77 730 71 59 735 65 30 752 70# of days between baseline OMS record date and OMS 3-yr. follow-up record date 73 1093 68 58 1102 66 23 1104 61

Since the correspondence between the OMS dates and initiation dates was variable and rarely exact, we adopted a set of rules for matching the various OMS records with the initiation date and three annual follow-up dates. An OMS record for each person is determined to be a baseline OMS record if the service date is 1) within 90 days before or after the initiation date and 2) is the closest OMS record date to the initiation date for that person. The band of 90 days before or after initiation date was chosen as a reasonable timeframe that balances the desire to have OMS records be as close to the initiation date as possible with the need to have as many persons as possible in the analysis. Descriptive statistics are presented in Table 2 by treatment groups for a) the initiation date, b) the number of days between the initiation date and the closest OMS record and c) the baseline OMS record chosen.

The OMS record assigned to the 1-year follow-up was chosen as the closest OMS record to the 1-year anniversary of the initiation date, with the proviso that this OMS record must be within 90 days of that anniversary date. Similar rules were applied to assign the OMS records for the 2-year and 3-year anniversaries of the initiation date for each person. The number of study subjects for whom we found OMS records assigned to each of the anniversary dates are also reported in Table 2.

Since the initiation dates could be as early as 9/1/2006, and the OMS surveys only began on 9/1/2006, almost all of the persons with no baseline OMS record had an initiation date in the latter part of 2006 or early 2007. The share of persons missing a baseline record ranges from 35.8% for the IPS-F group up to 58.7% for the TVS group. A total of 265 persons were found to have had an OMS record within the time-window for the 1-year follow-up, including 66.7% of the IPS-F group, 69.6% of the SE-NF group and only 44.4% of the TVS group. The number of persons with an OMS record within the 2-year follow-up time window was 252; the corresponding figure for those within the 3-year follow-up time window only dropped to 228.

The first 2 rows of the table compare the mean initiation dates and mean baseline OMS dates across the 3 groups. The correspondence of the two dates is close for each of the groups. Row 3 in the table shows that the difference between the initiation date and the closest OMS record (which may not be in the time window to be designated as a baseline OMS record) is larger for the SE-NF and TVS groups. The next 6 rows show that for each of the 3 groups the mean follow-up dates based on the initiation date and the mean OMS follow-up record dates are similar. The final 3 rows show that the mean time durations between the baseline OMS dates and the follow-up OMS dates are also very similar across the groups. The latter result is of particular interest since some of our analyses will examine OMS outcome variables controlling for baseline values of these same variables.

7

Dependent Variables in the Analysis - The OMS dependent variables for this study are in three subject areas: functional status, alcohol and substance use, and criminal justice issues. All dependent variables come from the self-reported answers to the OMS questionnaire that is facilitated by the mental health provider. A list of dependent variables names, definitions, sources, and coded values are shown in Table 3. Descriptive data on these dependent variables are reported in Table 4. Note that for each of these variables, in each of the three follow-up years, the number of reported values is not quite as large as the numbers of persons with follow-up OMS records shown in Table 2.

As part of the OMS interview, there are 24 questions that together make up a psychiatric instrument called the BASIS-24 (Eisen et al., 2004). The BASIS-24 includes an overall score and seven sub-scores. BASIS-24 scores are continuous variables that are constructed from categorical questions whose responses have an ordinal nature. More detail is provided on each dependent variable below. Other questions from the OMS, besides those in the BASIS-24, either have ordered categorical or binary responses.

As shown in Table 3, the dependent variables include: 5 variables describing responses to each of 5 questions about functioning (included in the OMS but not in the BASIS-24), a re-coded BASIS-24 substance abuse sub-score (which covers both problematic alcohol and other substance use in the past week), one binary variable pertaining to a question about alcohol use in the past month, one binary variable pertaining to a question about drug use in the past month, and one binary variable pertaining to a question about arrests in the past 12 months.

Table 3. OMS Dependent Variable definitions

VARIABLE NAME* DEFINITION VALUE RANGEq1func_y1 I do things that are meaningful to me 1=strongly agree, 2=agree, 3=neutral,

4=disagree, 5=strongly disagree

q2func_y1 I am able to take care of my needs 1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

q3func_y1 I am able to handle things when they go wrong

1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

q4func_y1 I am able to do things that I want to do 1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

q5func_y1 My symptoms bother me 1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

alc_pastmo_y1** Alcohol use in the past month 1=any use; 0=No use

du_pastmo_y1** Drug use in the past month 1=any use; 0=No use

b24sa_y1 BASIS-24 Substance Use 1=BASIS-24 Substance Abuse score>0 ; 0=BASIS-24 Substance Abuse score of 0

arrest12mo_y1 Arrested in the past 12 months 1=arrest in past 12 months; 0=no arrests

* “y1” refers to the 1-year follow-up. Analogous dependent variables are defined for the 2-year and 3-year follow-ups, designated by the suffixes “y2” and “y3”.**No 3-year follow-up is included in our study because the OMS dropped this question from its survey and numbers of responses in the 3-year follow-up were very small in all treatment groups.

8

Descriptive statistics on these dependent variables are shown in Table 4.

Table 4: Descriptive Statistics (Unweighted) for Dependent Variables by Treatment Group

EBP-SE non-EBP SE TVS

N mean Sd N mean sd Nmea

n SdYr.1 Follow-Upq1func_y1 96 2.06 0.71 90 2.08 0.82 52 2.15 1.02q2func_y1 97 2.10 0.74 90 1.98 0.83 52 2.04 0.88q3func_y1 97 2.52 0.95 89 2.72 0.98 52 2.73 1.12q4func_y1 96 2.39 0.90 89 2.31 0.97 52 2.77 0.96q5func_y1 97 3.38 1.19 89 3.39 1.09 52 3.38 1.21alc_pastmo_y1 95 0.16 0.37 89 0.15 0.36 52 0.15 0.36du_pastmo_y1 94 0.04 0.20 88 0.05 0.21 52 0.04 0.19b24sa_y1 90 0.29 0.46 82 0.30 0.46 50 0.16 0.37arrest12mo_y1 102 0.06 0.24 88 0.08 0.27 52 0.13 0.34

Yr.2 Follow-Upq1func_y2 100 2.09 0.87 84 2.14 0.97 48 2.08 1.07q2func_y2 100 2.05 0.76 84 2.15 0.78 48 2.25 0.96q3func_y2 100 2.44 0.94 84 2.67 1.03 48 2.90 1.06q4func_y2 100 2.28 0.87 84 2.58 1.01 48 2.48 1.13q5func_y2 100 2.92 1.19 84 3.18 1.10 48 3.88 1.12alc_pastmo_y2 81 0.16 0.37 67 0.16 0.37 47 0.15 0.36du_pastmo_y2 81 0.02 0.16 66 0.08 0.27 47 0.02 0.15b24sa_y2 96 0.33 0.47 79 0.39 0.49 46 0.26 0.44arrest12mo_y2 111 0.01 0.09 91 0.08 0.27 50 0.04 0.20

Yr.3 Follow-Upq1func_y3 91 1.99 0.75 73 2.00 0.76 39 2.23 0.90q2func_y3 91 2.04 0.76 73 2.15 0.79 39 2.13 0.83q3func_y3 91 2.48 0.81 73 2.68 0.93 39 2.54 1.05q4func_y3 91 2.40 0.89 73 2.51 0.96 39 2.51 0.88q5func_y3 89 2.80 1.17 73 2.58 1.12 39 2.13 1.15b24sa_y3 88 0.34 0.48 72 0.46 0.50 36 0.33 0.48arrest12mo_y3 102 0.01 0.10 86 0.02 0.15 40 0.03 0.16

A number of salient features should be noted. First, the numbers of study subjects with reported values are generally slightly below the numbers with OMS records for the corresponding follow-up years (see Table 2). It thus appears that some OMS interviews did not have a complete set of responses. Second, average rates of non-zero responses were very low for several of the outcome variables, including the indicators for alcohol use and separately drug use in the prior month, and the indicator for arrests in the prior 12 months. The small amount of variance in these indicators, combined with the small sample size, implies that lack of power is a potential problem for statistical tests across treatment groups for these outcomes. Third, there are no cases where

9

the differences between treatment groups exceeds the sizes of the standard errors. This suggests that significant differences across these groups are unlikely to be observed in the absence of controlling on differences between the groups in covariates either by propensity weighting or by regression controls.

Covariates – The covariates used as regression controls in our analysis are defined in Table 5. They include baseline OMS values for our dependent variables as well as diagnostic flags from Medicaid claims data and baseline OMS data on employment status and on the person’s overall BASIS-24 mental health status score.

Table 5: Sources and Definitions for CovariatesVARIABLE

NAMEDATA

SOURCE DEFINITION VALUE RANGEq1func_bline OMS I do things that are

meaningful to me1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

q2func_bline OMS I am able to take care of my needs

1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

q3func_bline OMS I am able to handle things when they go wrong

1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

q4func_bline OMS I am able to do things that I want to do

1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

q5func_bline OMS My symptoms bother me 1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=strongly disagree

alc_pastmo_bline

OMS Alcohol use in the past month in binary form

1=any use; 0=No use

du_pastmo_bline OMS Drug use in the past month in binary form

1=any use; 0=No use

b24sa_bline OMS (B24) BASIS-24 Substance Use in binary form

1=BASIS-24 Substance Use score>0 ; 0=BASIS-24 Substance Abuse score of 0

arrest12mo_bline OMS Arrested in the past 12 months binary

1=arrest in past 12 months; 0=no arrests

emp_past6 OMS currently employed or in past 6 mos.

1=employed in past 6 mos.; 0=not

b24ovrall_bline OMS (B24) BASIS-24 overall score continuous score from 1 (fewest symptoms) to 5 (most symptoms)

schiz_diag Medicaid claims

Schizophrenia diagnosis 1=schizophrenia diagnosis; 0=other

dep_bip_diag Medicaid claims

Bipolar or major depression diag.

1=bipolar or depression diag.; 0=other

other_diag Medicaid claims

Other SMI diagnosis 1=other SMI diagnosis; 0=not

Table 6 presents descriptive statistics on these variables across the 3 treatment groups. As in the case of the dependent variables, we see relatively small differences across the groups in all variables. The most substantial difference is in the primary diagnosis variables, where the fractions of cases with a schizophrenia diagnosis is considerably smaller in the TVS group, while the fractions of cases with bipolar or major depression primary diagnoses and with other primary diagnoses is larger in the TVS group.

Table 6: Descriptive Statistics for Covariates

10

IPS-F SE-NF TVS

Variables Nmea

n sd NMea

n sd NMea

n sdq1func_bline 98 2.06 0.82 82 2.11 0.98 52 2.15 1.07q2func_bline 98 2.08 0.81 82 2.00 0.86 52 2.25 1.12q3func_bline 98 2.66 0.91 82 2.45 1.01 52 2.62 1.12q4func_bline 98 2.46 0.99 82 2.41 0.97 52 2.50 1.32q5func_bline 98 3.45 1.15 81 3.25 1.21 52 3.79 1.07alc_pastmo_bline 94 0.13 0.34 81 0.17 0.38 51 0.22 0.42du_pastmo_bline 93 0.04 0.20 81 0.05 0.22 51 0.08 0.27b24sa_bline 85 0.26 0.44 76 0.36 0.48 49 0.37 0.49emp_past6 102 0.06 0.24 88 0.08 0.27 52 0.13 0.34b24ovrall_bline 102 0.26 0.44 88 0.40 0.49 52 0.33 0.47schiz_diag 84 1.21 0.60 76 1.09 0.67 49 1.31 0.68dep_bip_diag 159 0.54 0.50 148 0.51 0.50 126 0.20 0.40other_diag 159 0.37 0.48 148 0.41 0.49 126 0.48 0.50

Demographic Comparisons – Table 7 provides comparative data, across the 3 treatment groups, on age, race, gender, and location. The source for these data items is the Medicaid enrollment records. Data are reported for those respondents with at least one OMS record.

Table 7: Descriptive Statistics for 0-1 Demographic Variables

IPS-F SE-NF TVSN mean N mean N mean

female 159 0.47 148 0.4512

6 0.59

white_race 159 0.40 148 0.3612

6 0.32

black_race 159 0.55 148 0.5712

6 0.67

other_race 159 0.06 148 0.0712

6 0.02

age 159 39.7 148 38.512

6 34.9

balt_suburbs 159 0.26 148 0.2612

6 0.16

wash_suburbs 159 0.40 148 0.1112

6 0.22

western_md 159 0.08 148 0.0712

6 0.05

southern_md 159 0.01 148 0.0512

6 0.10

easternshore_md 159 0.00 148 0.0212

6 0.05

baltcity 159 0.21 148 0.3812

6 0.41

The demographic differences between the groups are somewhat more pronounced. The IPS-F group has lower percentages of females and blacks. It also has a much lower percentage of persons residing in Baltimore City and a much higher percentage residing in the Washington

11

suburban areas. The TVS group has higher percentages of females, blacks, and persons residing in Baltimore City and in Southern Maryland and the Eastern Shore.

Regression Analysis Methods - Separate regressions were estimated separately for the 1-year follow up, the 2-year follow-up and the 3-year follow-up. In each case, 2 different model specifications were used:

a simple model with only the treatment group dummies and the lagged (baseline) value of the dependent variable as explanatory variables, and

a second model that included as additional covariates two baseline diagnosis dummies, baseline employment status, and baseline overall BASIS-24 mental health status score

The first model was estimated with un-weighted data while the second model used weighted data with the weights derived from a propensity score analysis (described in the on-line supplement). Thus, for each dependent variable we report 2 regression models for each of the 3 follow-up years.

The types of regression models that we applied varied with the type of dependent variables. The 5 functional status variables are ordered qualitative responses so we applied the maximum-likelihood ordered probit regression estimation procedure. The four remaining dependent variables were coded as binary outcomes and thus we applied the maximum-likelihood probit regression model with binary outcomes. (In all regressions, TVS treatment is the reference category and the reported results for the IPS-F and SE-NF groups indicated outcome differences relative to the TVS group.

RESULTS

Self-Reported Functional Status Items from the OMS -We examined possible treatment effects on responses to 5 different specific items in the OMS that were used to track functional status. These items are to following statements:

1. I do things that are meaningful to me2. I am able to take care of my needs3. I am able to handle things when they go wrong4. I am able to do things I want to do5. My symptoms bother me

For each statement the respondent was asked to select one of five responses indicating their level of agreement: strongly agree, agree, neutral, disagree, and strongly disagree. Since these responses are ordered but do not represent interval data, we have estimated ordered probit regression to obtain treatment effect estimates. It should also be noted that for the first 4 items, higher responses imply poorer outcomes while for the last item, a higher response implies a better outcome.

Item 1: I do things that are meaningful to me - Table 8 gives the estimated effects on the responses to item 1 (I do things that are meaningful for me). Results indicate a consistent absence of treatment effects regardless of the weights used or the set of covariates included in the regression. The only consistently significant covariates (with positive coefficients) were the baseline value of the outcome question and the baseline value of the BASIS-24 index (in 2 of 3 regressions).

12

Table 8. Ordered Probit Coefficients (and p-values) for Functional Status 1 (I do things that are meaningful to me). (Negative sign indicates improving effect.)

1-year Follow-up 2-year Follow-up 3-year Follow-upunweighted weighted unweighted weighted unweighted weighted

IPS-F 0.0793 0.153 -0.031 0.279 -0.031 0.079(0.738) (0.510) (0.900) (0.254) (0.912) (0.770)

SE-NF -0.0864 -0.046 -0.081 0.019 -0.084 0.080(0.716) (0.844) (0.748) (0.939) (0.770) (0.779)

q1func_bline 0.283 0.223 0.274 0.272 0.394 0.493(0.002) (0.023) (0.007) (0.020) (0.001) (0.000)

schiz_diag 0.001 -0.074 0.272(0.998) (0.850) (0.482)

dep_bip_diag 0.502 0.030 0.363(0.184) (0.937) (0.348)

emp_past6 0.056 0.367 0.060(0.774) (0.069) (0.789)

b24ovrall_bline 0.284 0.435 0.040(0.066) (0.009) (0.830)

N 163 150 149 135 129 121pseudo R-sq 0.03 0.071 0.02 0.082 0.04 0.083AIC 360.2 335.5 378 344.9 284.1 255.1BIC 378.8 365.6 399.1 376.9 301.3 283.1

Item 2: I am able to take care of my needs - Table 9 gives the estimated coefficients on the responses to item 2. Here again there are no significant treatment effects. The baseline value of the outcome is again consistently significant (with positive coefficients), as is the baseline overall BASIS-24 covariate (in 2 of 3 regressions). The only other significant covariate (in only 1 regression) is the baseline indicator that the respondent was employed in the 6 months prior to baseline. The positive coefficient for this covariate implies that it is predictive of a poorer follow-up outcome on this functional status indicator.

Table 9. Ordered Probit Coefficients (and p-values) for Functional Status 2 (I am able to take care of my needs). (Negative sign indicates improving effect.)

1-year Follow-up 2-year Follow-up 3-year Follow-upunweighted weighted unweighted weighted Unweighted weighted

IPS-F 0.354 0.292 -0.099 -0.111 0.146 -0.060(0.135) (0.207) (0.691) (0.663) (0.603) (0.832)

SE-NF 0.181 0.141 0.149 0.305 0.253 0.144(0.452) (0.551) (0.565) (0.236) (0.387) (0.618)

q2func_bline 0.435 0.495 0.353 0.242 0.216 0.197(0.000) (0.000) (0.001) (0.059) (0.063) (0.158)

schiz_diag 0.008 0.157 0.265(0.983) (0.711) (0.506)

dep_bip_diag 0.105 0.152 -0.137(0.783) (0.713) (0.728)

emp_past6 0.470 0.064 -0.123(0.015) (0.760) (0.588)

13

b24ovrall_bline -0.018 0.242 -0.109(0.902) (0.129) (0.528)

N 164 151 149 135 129 121pseudo R-sq 0.057 0.072 0.04 0.046 0.014 0.032AIC 365.6 336 314.2 277.6 282.7 240.4BIC 387.3 369.2 332.2 306.7 299.9 268.4

Item 3: I am able to handle things when they go wrong - Table 10 presents ordered probit results for the third of the five functioning questions. The only significant treatment effects are positive (i.e., detrimental) effects of non-fidelity-certified SE vs. TVS, but only in year 3. Results for the baseline outcome and baseline BASIS-24 covariates conform to our previous findings. There is also some evidence, albeit inconsistent across years, of significant differences by diagnosis in this particular outcome.

Table 10. Ordered Probit Coefficients (and p-values) for Functional Status 3 (I am able to handle things when they go wrong). (Negative sign indicates improving effect.)

1-year Follow-up 2-year Follow-up 3-year Follow-upUnweighted weighted unweighted weighted unweighted weighted

IPS-F -0.0544 0.136 -0.362 -0.322 0.201 0.340(0.810) (0.537) (0.129) (0.181) (0.460) (0.205)

SE-NF 0.133 0.378 -0.076 -0.025 0.511 0.576(0.564) (0.097) (0.757) (0.918) (0.073) (0.041)

q3func_bline 0.265 0.226 0.449 0.340 0.548 0.702(0.003) (0.026) (0.000) (0.003) (0.000) (0.000)

schiz_diag -0.632 0.199 -0.010(0.076) (0.605) (0.979)

dep_bip_diag -0.457 0.656 0.250(0.202) (0.087) (0.497)

emp_past6 0.177 0.024 0.324(0.341) (0.902) (0.136)

b24ovrall_bline 0.460 0.253 0.232(0.002) (0.097) (0.196)

N 163 150 149 135 129 121pseudo R-sq 0.022 0.08 0.061 0.098 0.087 0.171AIC 435.5 387.1 389.7 342.7 319.2 285.2BIC 457.1 420.2 410.8 374.6 339.2 316

I tem 4: I am able to do things I want to do - Results in Table 11 indicate only two significant treatment effects, both for the SE-NF group, but the signs differ between year 1 (negative) and year 2 (positive). Other results conform to our previous findings, with only the baseline dependent variable and the baseline BASIS-24 value being significant.

Table 11. Ordered Probit Coefficients (and p-values) for Functional Status 4 (I am able to do things I want to do). (Negative sign indicates improving effect.)

14

1-year Follow-up 2-year Follow-up 3-year Follow-upunweighted weighted unweighted Weighted unweighted Weighted

IPS-F -0.302 0.004 0.023 0.136 -0.074 0.039(0.183) (0.985) (0.924) (0.573) (0.782) (0.881)

SE-NF -0.515 -0.238 0.350 0.609 0.111 0.035(0.025) (0.293) (0.160) (0.013) (0.690) (0.899)

q4func_bline 0.269 0.156 0.366 0.284 0.242 0.199(0.001) (0.107) (0.000) (0.005) (0.016) (0.071)

schiz_diag 0.240 -0.074 0.249(0.511) (0.850) (0.492)

dep_bip_diag 0.532 0.051 0.021(0.155) (0.896) (0.955)

emp_past6 -0.137 -0.154 -0.226(0.464) (0.437) (0.290)

b24ovrall_bline 0.511 0.362 0.369(0.000) (0.017) (0.028)

N 162 149 149 135 129 121pseudo R-sq 0.037 0.079 0.05 0.08 0.021 0.044AIC 422.5 384.9 392.3 350.9 327.6 306.5BIC 444.1 417.9 413.3 382.8 347.7 337.2

Item 5: My symptoms bother me – Results in Table 12 for this dependent variable show significant treatment effects for both the IPS-F and SE-NF groups for years 1 and 2, but again the signs are not consistent between the years. We also now observed inconsistent, and not always significant results for the baseline dependent variable and BASIS-24 covariates. IN year 1, the prior employment covariate is significant with a sign again suggesting a deleterious effect on the outcome item, but results for years 2 and 3 are clearly not significant.

Table 12. Ordered Probit Coefficients (and p-values) for Functional Status 5 (My symptoms bother me). (Positive sign indicates improving effect.)

1-year Follow-up 2-year Follow-up 3-year Follow-upunweighted weighted unweighted Weighted unweighted weighted

IPS-F 0.0304 0.496 -0.822 -0.632 0.347 0.205(0.892) (0.025) (0.001) (0.008) (0.203) (0.440)

SE-NF 0.214 0.529* -0.619 -0.456 0.215 0.243(0.352) (0.020) (0.015) (0.062) (0.444) (0.367)

q5func_bline 0.314 0.121 0.040 0.081 -0.178* -0.098(0.000) (0.207) (0.597) (0.411) (0.032) (0.324)

schiz_diag 0.257 0.265 0.445(0.464) (0.488) (0.216)

dep_bip_diag 0.615 0.431 0.164(0.084) (0.251) (0.649)

emp_past6 -0.405 -0.003 0.060(0.027) (0.986) (0.773)

b24ovrall_bline 0.579*** 0.155 -0.545(0.000) (0.323) (0.002)

15

N 163 150 148 134 126 118pseudo R-sq 0.035 0.089 0.027 0.036 0.021 0.071AIC 484.7 440.9 453.7 426.2 378.7 345.9BIC 506.3 474.1 474.6 458.1 398.5 376.4

Self-reported prior alcohol use and drug use – Regressions were estimated for 3 different binary dependent variables pertaining to prior drug and alcohol use: 1) self-reported use of any alcohol in the 30 days prior to the OMS interview, 2) 1) self-reported use of any illegal drugs in the 30 days prior to the OMS interview, and 3) a 0-1 recoding of the BASIS-24 substance abuse sub-score.. This sub-score is derived from four of the questions from the BASIS-24 instrument and is in the form of a cumulative score in which a lower score is better. Each of the four questions from which the score is calculated ask whether a particular aspect of substance abuse occurred in the past week. The four questions are the following: 1) Did you have an urge to drink alcohol or take street drugs? 2) Did anyone talk to you about your drinking or drug use? 3) Did you try to hide your drinking or drug use? 4) Did you have problems from your drinking or drug use? Answers to these questions are in terms of the frequency they occurred in the past week: a) Never b) Rarely c) Sometimes d) Often e) Always. Due to the sub-score having a high percentage of zeros indicating no substance abuse problems, the decision was made to re-code the score into a binary variable with 0 representing no substance abuse problems and 1 representing any substance abuse problems.

Results for the first of these dependent variables are shown in Table 13. The only significant treatment effect is a positive coefficient in year 1 for the IPS-F implying an increased probability of alcohol use. The usual pattern of positive coefficients for the baseline dependent and baseline BASIS-24 covariates and insignificant results for all other covariates was again observed.

Table 13. Marginal Effects (and p-values) for Probability of Alcohol use past 30 days.

1-year Follow-up 2-year Follow-upunweighted weighted unweighted weighted

IPS-F 0.0841 0.119 0.017 -0.006(0.303) (0.014) (0.831) (0.905)

SE-NF 0.0754 0.056 0.051 -0.020(0.359) (0.267) (0.532) (0.706)

alc_pastmo_bline 0.0214 0.017 0.021 0.021(0.004) (0.001) (0.031) (0.002)

schiz_diag -0.100 -0.049(0.130) (0.549)

dep_bip_diag -0.045 -0.032(0.507) (0.695)

emp_past6 0.064 -0.016(0.091) (0.705)

b24ovrall_bline -0.068 -0.066(0.019) (0.044)

N 161 150 124 116pseudo R-sq 0.066 0.095 0.051 0.064AIC 140.9 283 102 197.6BIC 153.2 307.1 113.3 219.6

16

The pattern of results for the binary indicator of self-reported drug use in the past 30 days, as shown in Table 14, is similar. There is a finding of a positive treatment effect of IPS-F on probability of drug use in year 2 when other (almost all insignificant) covariates are also included but no other evidence of such an effect in other regressions. Positive effects are also observed for the baseline value of the dependent variable.

.Table 14. Marginal Effects (and p-values) for Probability

of Drug use past 30 days1-year Follow-up 2-year Follow-up

unweighted Weighted unweighted WeightedIPS-F 0.0109 0.023 0.023 0.081

(0.741) (0.262) (0.701) (0.081)SE-NF -0.00459 0.016 0.065 0.090

(0.897) (0.470) (0.270) (0.054)sa_pastmo_bline 0.0364 0.024 0.009 0.006

(0.045) (0.001) (0.020) (0.016)schiz_diag 0.094 0.352

(0.997) (0.990)dep_bip_diag 0.158 0.391

(0.996) (0.989)emp_past6 -0.144 0.001

(0.962) (0.974)b24ovrall_bline -0.018 -0.026

(0.157) (0.270)N 160 149 124 116pseudo R-sq 0.33 0.479 0.137 0.125AIC 42.27 63.18 54.49 108.7BIC 54.57 87.21 65.77 130.7

Regression results for the BASIS-24 substance abuse problem indicator are shown in Table 15. Treatment effect results for both year 1 and year 3 show significant increases in the probability of any substance abuse problem for the IPS-F and SE-NF treatment groups; however in year 2 both groups show significantly negative coefficients (implying reduced probabilities of substance use problems. Among the covariates, in year 1 we observe significantly negative coefficients for prior employment and baseline overall BASIS-24, and we also observe a significantly positive coefficient for the schizophrenia dummy in year 2. As in most other cases, the coefficient of the baseline dependent variable is consistently and significantly positive.

Table 15. Marginal Effects (and p-values) for Probability of Any substance use problem in the past week (BASIS-24).

1-year Follow-up 2-year Follow-up 3-year Follow-upunweighted weighted unweighted Weighted unweighted weighted

IPS-F 0.217 0.235 0.049 -0.115 0.108 0.154(0.026) (0.000) (0.633) (0.081) (0.379) (0.035)

SE-NF 0.197 0.186 0.040 -0.139 0.163 0.15417

(0.045) (0.001) (0.703) (0.043) (0.189) (0.047)b24sa_bline 0.291 0.315 0.299 0.259 0.309 0.327

(0.000) (0.000) (0.000) (0.000) (0.000) (0.000)schiz_diag -0.089 0.197 0.053

(0.257) (0.092) (0.596)dep_bip_diag -0.016 0.044 -0.061

(0.844) (0.711) (0.555)emp_past6 -0.124 -0.076 -0.059

(0.005) (0.172) (0.323)b24ovrall_bline -0.099 -0.014 -0.006

(0.004) (0.727) (0.884)N 143 143 132 132 117 117pseudo R-sq 0.202 0.263 0.13 0.103 0.123 0.128AIC 143.2 291.4 158.5 362 145.6 328.2BIC 155.1 315.2 170 385 156.6 350.3

Arrest in the Past 12 months – The last binary regression model is for the probability of an arrest in the prior 12 months for each of the 3 follow-up years. In this case, no significant coefficients are found for any treatment indicators or other covariates except for the baseline overall BASIS-24 score. This is not surprising since the observed rate of arrests in our data was very low. As shown in Table 4, the fraction of respondents reporting any arrests in the prior 12 months did not exceed o.1 for any of the three treatment groups for any of the three follow-up years with the sole exception of the TVS group in the year 1 follow-up.

Table 16. Marginal Effects for Prob. of an Arrest in the past 12 months

1-year Follow-up* 2-year Follow-up* 3-year Follow-up*unweighted weighted unweighted weighted unweighted weighted

IPS-F -0.0689 -0.037 -0.049 -0.054 -0.016 0.00135(0.123) (0.212) (0.225) (0.115) (0.520) (0.930)

SE-NF -0.0454 -0.024 0.026 0.013 -0.00131 0.00889(0.302) (0.435) (0.394) (0.649) (0.952) (0.563)

schiz_diag -0.065 0.403(0.149) (0.988)

dep_bip_diag 0.039 0.432(0.342) (0.987)

emp_past6 -0.037 -0.045(0.182) (0.115)

b24ovrall_bline 0.109 0.043(0.000) (0.012)

N 242 209 156 143 144 228pseudo R-sq 0.018 0.208 0.074 0.136 0.057 0.007AIC 141.6 260.8 64.42 127 33.51 85.05BIC 152.1 284.2 73.57 147.7 42.42 95.34

* The baseline arrest variable perfectly predicts the outcome variable and is therefore dropped for all regressions.

18

Conclusion

The analyses of patient-reported outcomes in the OMS provide little substantial evidence in support of the alternative hypotheses that, compared to persons receiving traditional vocational services, persons receiving IPS-F supported employment or SE-NF non-fidelity-certified supported employment would have positive (improving) non-employment outcomes. There are, however, a number of considerations which suggest that the power of our analysis to reject the corresponding null hypotheses may have been low.

A principal concern is the relatively small sample sizes for the OMS regressions. Across almost all regressions, the range of values observed for N was between 100 and 185; the only exception was the regressions for year 1 and year 3 for arrest in the prior 12 months. Given the variability inherent in individual self-report data, the multiple comparisons groups, and the need to also include covariate controls, it is perhaps surprising that any regression coefficients turned out to be significant.

Another aspect of problematic power in our analysis is that the rates of positive responses for the binary outcome variables tended to be very low, indeed, well below our expectations.For example, the Substance Abuse and Mental Health Services Administration (Epstein et al., 2004) found that in 2002 28.7% of persons with a severe mental illness reported substance abuse in the past year. Since the populations in our study are very different (e.g., all participants sought and were receiving vocational services), it may not be surprising that percentages for reported substance use were much lower, ranging from 2% to 6%, varying by treatment gr. These lower self-reported percentages clearly imply a low level of power in the analysis that makes rejection of the null hypothesis very difficult.

We also note the real possibility of selection on unobservable factors in our study samples that bias the estimated effects of the treatment groups on the non-employment outcomes. The observational nature of this study, with subjects self-selected into each of the three treatment groups, introduces this possibility. While propensity-score weights help to balance the treatment groups on observables, they do not necessarily correct for bias due to unobservable factors. There is also some concern that the propensity-weights may not adequately balance the treatment groups because the TVS group was small and may not have enough overlap.

In conclusion, we do not provide not evidence for the IPS-F treatment group having superior non-employment outcomes relative to the TVS. There are a few particular results from the different non-employment outcomes in certain follow-up years that are supportive. Overall though, there are enough instances of contradictory results and a lack of consistency that undercut this finding.

19

References

Bailey, J. 1998. “I’m Just An Ordinary Person.” Psychiatric Rehabilitation Journal, 22(1), 8–10.

Becker, D. R., Drake, R. E., and Bond, G. R. 2011. “Benchmark Outcomes In Supported Employment.” American Journal of Psychiatric Rehabilitation, 14, 230–236.

Becker, D. R., Whitley, R., Bailey, E. L., and Drake, R. E. 2007. “Long-Term Employment Outcomes Of Supported Employment For People With Severe Mental Illness.” Psychiatric Services, 58, 922–928.

Bedell, J. R., Draving, D., Parrish, A., Gervey, R., and Guastadisegni, P. 1998. “A Description And Comparison Of Experiences Of People With Mental Disorders In Supported Employment And Paid Prevocational Training”. Psychiatric Rehabilitation Journal, 21(3), 279–283.

Bond, G. R. 2004. “Supported Employment: Evidence For An Evidence-Based Practice.” Psychiatric Rehabilitation Journal, 27, 345–359.

Bond, G. R., Campbell, K., and Drake, R. E. 2012. “Standardizing Measures In Four Domains Of Employment Outcome For Individual Placement And Support.” Psychiatric Services, 63, 751–757.

Bond, G. R., Dietzen, L. L., Vogler, K. M., Katuin, C. H., McGrew, J. H., and Miller, L. D. 1995. “Toward A Framework For Evaluating Costs And Benefits Of Psychiatric Rehabilitation: Three Case Examples.” Journal of Vocational Rehabilitation, 5, 75–88.

Bond, G.R. and Drake, R.E. 2014. “Making The Case For IPS Supported Employment.” Administration and Policy in Mental Health and Mental Health Services Research, 41:69–73.

Bond, G.R., Drake, R.E., and Becker, D.R. 2008. “An Update on Randomized Controlled Trials of Evidence-Based Supported Employment”. Psychiatric Rehabilitation Journal 31(4):280-290.

Bond, G. R., Resnick, S. G., Drake, R. E., Xie, H., McHugo, G. J., and Bebout, R. R. 2001. “Does Competitive Employment Improve Nonvocational Outcomes For People With Severe Mental Illness?” Journal of Consulting and Clinical Psychology, 69, 489–501.

Bond, G.R., Salyers, M.P, Dincin, J., Drake, R., Becker, D.R., Fraser, V.V., and Haines, M. 2007. “A Randomized Controlled Trial Comparing Two Vocational Models For Persons With Severe Mental Illness.” Journal of Consulting and Clinical Psychology 75(6): 968-982.

Burns, T., Catty, J., Becker, T., Drake, R. E., Fioritti, A., Knapp, M., Lauber. C., Rössler, W., Tomov, T., van Busschbach, J., White, S., Wiersma, D.; EQOLISE Group. 2007. “The Effectiveness Of Supported Employment For People With Severe Mental Illness: A Randomised Controlled Trial.”Lancet, 370, 1146–1152.

Burns, T., Catty, J., White, S., Becker, T., Koletsi, M., Fioritti, A., Rossler, W., Tomov T., van Busschbach, J., Wiersma, D., Lauber, D., and for the EQOLISE Group. 2009. “The Impact Of Supported Employment And Working On Clinical And Social Functioning: Results Of An International Study Of Individual Placement And Support.” Schizophrenia Bulletin, 35, 949–958.

Bush, P. W., Drake, R. E., Xie, H., McHugo, G. J., & Haslett, W. R. 2009. “The Long-Term Impact Of Employment On Mental Health Service Use And Costs.” Psychiatric Services, 60, 1024–1031.

20

Clark, R. E. 1998. “Supported Employment And Managed Care: Can They Coexist?” Psychiatric Rehabilitation Journal, 22(1), 62–68.

Crowther, R. E., Marshall, M., Bond, G. R., and Huxley, P. 2001. “Helping People With Severe Mental Illness To Obtain Work: Systematic Review.” British Medical Journal, 322, 204–208.

Dartmouth Psychiatric Research Center. “Practice Principles of IPS Supported Employment.” 10/11/2011. http://www.dartmouth.edu/~ips/page48/page79/files/ips-practice-principles-002880029.pdf. Accessed August 20, 2014.

Dixon, L. B., Dickerson, F. B., Bellack, A. S., Bennett, M., Dickinson, D., Goldberg, R. W., Lehman, A., Tenhula, W.N., Calmes, C., Pasillas, R.M., Peer, J., and Kreyenbuhl, J. 2010. “The 2009 Schizophrenia PORT Psychosocial Treatment Recommendations And Summary Statements.” Schizophrenia Bulletin, 36, 48–70.

Drake RE, McHugo GJ, Bebout RR, Becker DR, Harris M, Bond GR and Quimby E. 1999. “A Randomized Clinical Trial Of Supported Employment For Inner-City Patients With Severe Mental Disorders.” Archives of General Psychiatry 56: 627-633

Eisen, S. V., Normand, S.-L., Belanger, A. J., Spiro, A., and Esch, D. 2004. “The Revised Behavior and Symptom Identification Scale (BASIS-R): Reliability And Validity.” Medical Care, 42(12), 1230–1241.

Epstein J., Barker, P., Vorburger, M., and Murtha, C. 2004) “Serious Mental Illness And ItsCo-Occurrence With Substance Use Disorders, 2002”. DHHS Publication No. SMA 04-3905, Analytic Series A-24. Rockville, MD: Substance Abuse and Mental Health Services Administration, Office of Applied Studies.

Frounfelker, R. L., Wilkniss, S. M., Bond, G. R., Devitt, T. S., and Drake, R. E. 2011. “Enrollment In Supported Employment Services For Clients With A Co-Occurring Disorder.” Psychiatric Services, 62, 545–547.

Henry, A. D., Lucca, A. M., Banks, S., Simon, L., and Page, S. 2004. “Inpatient Hospitalizations And Emergency Service Visits Among Participants In An Individual Placement And Support (IPS) Model Program.” Mental Health Services Research, 6, 227–237.

Kinoshita, Y., Furukawa, T.A., Kinoshita, K., Honyashiki, M., Omori, I.M., Marshall, M., Bond, G.R., Huxley, P., Amano, N. and Kingdon, D. 2013. “Supported Employment for Adults with Severe Mental Illness.” Cochrane Database of Systematic Reviews, Issue 9. Art. No,: CD008297. DOI: 10.1002/14651858.CD008297.pub2

Kukla, M., Bond, G. R., and Xie, H. 2012. “A Prospective Investigation Of Work And Nonvocational Outcomes In Adults With Severe Mental Illness.” Journal of Nervous and Mental Disease, 200,214–222.

Kukla M. and Bond GR. 2013. “A Randomized Controlled Trial Of Evidence-Based Supported Employment: Nonvocational Outcomes.” Journal of Vocational Rehabilitation 38: 91–98.

Latimer, E. 2001. “Economic Impacts Of Supported Employment For The Severely Mentally Ill.” Canadian Journal of Psychiatry, 46, 496–505.

21

Lindamer, L. A., Bailey, A., Hawthorne, W., Folsom, D. P., Gilmer, T. P., Garcia, P., Hough, R.L., Jeste, D.V. 2003). “Gender Differences In Characteristics And Service Use Of Public Mental Health Patients With Schizophrenia.” Psychiatric Services, 54, 1407–1409.

Marshall, T., Goldberg, R. W., Braude, L., Dougherty, R. H., Daniels, A. S., Ghose, S. S., George, P., Delphin-Rittmon, M.E. 2014. “Supported Employment: Assessing The Evidence.” Psychiatric Services, 65, 16-23.

McQuilken, M., Zahniser, J. H., Novak, J., Starks, R. D., Olmos, A., and Bond, G. R. 2003. “The Work Project Survey: Consumer Perspectives On Work.” Journal of Vocational Rehabilitation, 18,59–68.

Mueser, K. T., Becker, D. R., Torrey, W. C., Xie, H., Bond, G. R., Drake, R. E., and Dain, B.J. 1997. “Work And Nonvocational Domains Of Functioning In Persons With Severe Mental Illness: A Longitudinal Analysis.” Journal of Nervous and Mental Disease, 185, 419–426.

Mueser, K. T., Salyers, M. P., and Mueser, P. R. 2001. “A Prospective Analysis Of Work In Schizophrenia.” Schizophrenia Bulletin, 27, 281–296.

National Institute of Mental Health (1999). Bridging Science And Service: A report by the National Advisory Mental Health Council’s Clinical Treatment and Services Research Workgroup. Rockville, MD: National Institute of Mental Health.

Pandiani, J. and Leno, S. 2012. “Employment Of CRT Clients: Third Quarters (Jan–Mar) Of FY2000–FY2012.” Waterbury: Vermont Agency of Human Services.

Perkins, R., and Rinaldi, M. 2002. “Unemployment Rates Among Patients With Long-Term Mental Health Problems: A Decade Of Rising Unemployment.” Psychiatric Bulletin, 26, 295–298.

Perkins, D. V., Born, D. L., Raines, J. A., and Galka, S. W. 2005. “Program Evaluation From An Ecological Perspective: Supported Employment Services For Persons With Serious PsychiatricDisabilities.” Psychiatric Rehabilitation Journal, 28, 217–224.

President’s New Freedom Commission on Mental Health (2003). Report to the President from the President’s New Freedom Commission on Mental Health. Washington DC: U.S. Government Printing Office

Ramsay, C. E., Broussard, B., Goulding, S. M., Cristofaro, S., Hall, D., Kaslow, N. J., Killackey, E., Penn, D., and Compton, M.T. 2011. “Life And Treatment Goals Of Individuals Hospitalized For First-Episode Nonaffective Psychosis.” Psychiatry Research, 189, 344–348.

Rogers, E. S., Walsh, D., Masotta, L., and Danley, K. 1991. “Massachusetts Survey Of Client Preferences For Community Support Services: Final Report.” Boston: Center for PsychiatricRehabilitation.

Rogers, E. S., Sciarappa, K., MacDonald-Wilson, K., and Danley, K. 1995. “A Benefit-Cost Analysis Of A Supported Employment Model For Persons With Psychiatric Disabilities.” Evaluation and Program Planning, 18, 105–115.

Rosenheck, R. A., Leslie, D., Keefe, R., McEvoy, J., Swartz, M., Perkins, D., Stroup, D., Hsiao, J.K., Lieberman, J., and the CATIE Study Investigators Group. 2006. “Barriers To Employment For People With Schizophrenia.” American Journal of Psychiatry, 163, 411–417.

22

Salkever, D., Karakas, M.C., Slade, E.P., Harding, C.M., Hough, R.L., Rosenheck, R.A., Swartz, M.S., Barrio, C., Yamada, A.M. 2007. “Measures and Predictors of Community-Based Employment and Earnings f Persons wth Schizophrenia in a Multisite Study.” Psychiatric Services 58(3):315-324.

Salkever, D. 2013. “Social Costs Of Expanding Access To Evidence-Based Supported Employment: Concepts And Interpretive Review Of The Evidence.” Psychiatric Services 64:111– 119,

Salkever D. 2010. “Toward a Social Cost-Effectiveness Analysis of Programs to Expand Supported Employment Services: An Interpretive Review of the Literature.” U.S. Dept. of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation, Westat, Inc.December 31, 2010. http://aspe.hhs.gov/daltcp/reports/2010/supempLR.htm (accessed August 20, 2014).

Salyers, M. P., Becker, D. R., Drake, R. E., Torrey, W. C., and Wyzik, P. F. 2004. “Ten-Year Follow-Up Of Clients In A Supported Employment Program.” Psychiatric Services, 55, 302–308.

Schneider, J., Boyce, M., Johnson, R., Secker, J., Slade, J., Grove, B., and Floyd, M. 2009. “Impact Of Supported Employment On Service Costs And Income Of People With Mental Health Needs.” Journal of Mental Health, 18, 533–542.

Strickler, D. C., Whitley, R., Becker, D. R., adn Drake, R. E. 2009. “First Person Accounts Of Long-Term Employment Activity Among People With Dual Diagnosis.” Psychiatric Rehabilitation Journal, 32, 261–268.

Substance Abuse and Mental Health Services Administration (SAMHSA), 2013. “Results from the 2012 National Survey on Drug Use and Health: Mental Health Findings.” NSDUH Series H-47, HHS Publication No. (SMA) 13-4805. Rockville, MD: Substance Abuse and Mental Health Services Administration. http://www.nimh.nih.gov/statistics/SMI_AASR.shtml

Turner, N., O’Mahony, P., Hill, M., Clarke, M., Larkin, C., Waddington, J., et al. 2012. “Employment Outcome Over 12 Years In A First Episode Psychosis Cohort.” Paper presented atthe International Early Psychosis Association, San Francisco.

Twamley, E. W., Jeste, D. V., and Lehman, A. F. 2003. “Vocational Rehabilitation In Schizophrenia And Other Psychotic Disorders: A Literature Review And Meta-Analysis Of Randomized Controlled Trials.” Journal of Nervous and Mental Disease, 191, 515–523.

U.S. Public Health Service Office of the Surgeon General. 1999. Mental Health: A Report of the Surgeon General. Rockville, MD: Department of Health and Human Services, U.S. Public Health Service.

Watkins, K. E., Pincus, H. A., Paddock, S., Smith, B., Woodroffe, A., Farmer, C., Sorbero, M.E., Horvitz-Lennon, M,. Mannle, T. Jr., Hepner, K.A., Solomon, J., and Call, C. (2011). “Care For Veterans With Mental And Substance Use Disorders: Good Performance, But Room To Improve On Many Measures.” Health Affairs, 30, 2194–2203.

Wehman, P., and Moon, M.S. 1988. Vocational Rehabilitation And Supported Employment. Baltimore: Brookes Publishing.

Woltmann, E. 2009. “Development And Evaluation Of A Consumer Directed Decision Support

23

System To Support Shared Decision Making In Community Mental Health.” Dartmouth: Unpublished dissertation.

24