The National Evaluation of the PATH Program

Pamela J. Fischer, Ph.D.SAMHSA’s Center for Mental Health Services

Homeless Programs Branch

Presented at the 2010 PATH Grantee Meeting, Washington, DC

• Findings from the 2005 National Evaluation of the PATH Program—Pamela Fischer, Ph.D.

• Design and upcoming activities of the current evaluation—Paul Brounstein, Ph.D.

• Questions and discussion

Today’s Presentation

• Section 528 of the Public Health Service (PHS) Act requires the SAMHSA Administrator to evaluate the expenditures of PATH grants at least once every 3 years to ensure they are consistent with legislative requirements and to recommend changes to the program design or operations

• Evaluation design is determined by the Federal program staff and conducted through a competitively awarded contract

Congressional Mandate for the National Evaluation of the PATH Program

3

• Outreach is emphasized by PATH providers and is carried out conscientiously and with quality

• PATH program uses funds appropriately and effectively• Flexibility is widely praised • High levels of job satisfaction • PATH not necessarily recognized as a program as PATH

funds represent only one of many small, but important funding streams received by providers

Findings from Previous Evaluations

National Evaluation of the 2005 PATH Program

Conducted during 2006-9 by the MANILA Consulting Group, Inc.:

Walter Leginski, Ph.D., and Caroline Fernandez, Project Directors

5

• Are services appropriate?

• Are services well-administered?

• Are outcome and process goals achieved?

• increases in the number of persons contacted, enrolled, and receiving mental health services

• clients placed in permanent housing

Evaluation Questions

Focused Analyses

• PATH and rural homelessness

• Effect of local context on PATH performance – Housing– Poverty– Other Federal funding

Methods

• 5 VPGs informed the evaluation approach– emphasis on literal homelessness – active State management – specific guidance on reporting and definitions – focus on exemplary practices – active support of transition to mainstream services

• 2005 selected for analysis due to completeness of reports

• Data sources included administrative data and contextual information

• Design and interpretation guided by Technical Panel

Results

• Consistent with the findings of prior evaluations, PATH demonstrated that it continues to meet the three objectives of the evaluation

Appropriateness of Services

• Enrolled PATH clients match the eligibility profile and intent of the PATH legislation exceedingly well

• PATH providers offer a wide range of services from the PATH menu

• High proportions of enrolled clients participate in these services

• Engagement of clients into services and their transition to mainstream services are emphasized

Quality of Service Administration

• Program structure and approach• Responsiveness of States and providers to

information requested in applications• Program monitoring• States’ efforts toward adoption of Voluntary

Performance Goals (VPGs)• Variety of administrative approaches adopted by

providers featuring collaborations that would expedite clients’ access to mainstream services and housing

Achievement of Identified Outcome and Process Goals

• GPRA goal achievement was similar to past years

• Housing collaborations developed to assure access to affordable housing for PATH clients

PATH Providers in Rural Locations

• Received smaller awards

• Challenges to collaboration with providers in resource-poor environments

• Rural providers perform equally and in some cases better than providers in metropolitan and mixed (urban/rural) areas

Correlating PATH Performance with Poverty Rates, Affordable Housing, and Other Targeted Homelessness Funding Levels

• PATH provider performance was strong, even in locations where high poverty and lack of affordable housing were evident

• Additional sources of Federal homelessness funding in the same locale as PATH providers did not detract from PATH performance and supported the concept that multiple providers function as a system of homeless services, enhancing PATH performance

Summary

• PATH is a critical funding stream• PATH leverages resources and systemic approaches

directed to the most vulnerable subpopulations of homeless persons

• As populations and policies change in State and national environments, the triennial evaluations are ideal opportunities to assess whether adjustments are needed to sustain PATH’s important contributions to the welfare of homeless persons with serious mental illnesses

For additional information contact:

Pamela J. Fischer, Ph.D.Homeless Programs BranchCenter for Mental Health ServicesSubstance Abuse and Mental Health Services

Administration(240) 276-1901Pamela.Fischer@samhsa.hhs.gov

The National Evaluation of the PATH Program

Paul J. Brounstein, Ph.D.MANILA Consulting Group

Project Director

Presented at the 2010 PATH Grantee Meeting, Washington, DC

PATH

• Formula Grant to 50 states, D.C., and 5 Territories

• To enable States and providers to better allocate resources and improve service availability, accessibility and fill gaps in existing service system to decrease homelessness for individuals and families affected by serious mental illness

Key Services

• Community-based outreach• Screening and diagnostic services• Habilitation and rehabilitation services• Community mental health services• Alcohol or drug treatment services (for people with

mental illnesses and co-occurring substance abuse disorders)

• Case management services• Supervisory services in residential settings• Limited housing services and services to help clients

use housing resources

Most Common Services

• 88% of grantees offer outreach services• 82% of grantees offer case management

services

Evaluation

• Congress requires triennial evaluation answering 3 questions:– Are services using PATH monies appropriate

(Stewart B. McKinney Homeless Assistance Amendments Act of 1990)?

– Are services well-administered?– Have process and outcome goals been achieved?

Current Evaluation Plan goes further and assesses the role of:

• Grant Structure• Organizational Structure• Level of Collaboration• Mix of Services• Populations Served/Characteristics• Role of Context on Outcomes

– Urbanicity, Poverty, – Labor and Housing Markets

Evaluation Process

• Developed initial evaluation specifications in response to solicitation for proposals

• Assembled expert panel– Multiple telephone conferences to review plan

• Plan finalized and approved by SAMHSA

Data Requirements

• Answers to the multiple questions and purposes of the evaluation require data from several different sources:– Archival/secondary data (e.g., Annual reports, State

Applications, IUPs, Census data)– Primary data collection (SPC and provider surveys,

site visit assessments and focus groups with PATH clients)

• Data selected to support analysis of Program Model

PATH Logic Model

Other Measures

• Identification of PATH clients• Sustained engagement of clients• Linkage of PATH and other services• Accessibility to and use of housing and MH care

opportunities• Acceptability of services• Gaps (training, outreach, assessment, services,

linkages, measurement and reporting• Trends in homelessness (populations, mix of services)• Identification of innovation and use of best practices

Data Sources

• Archival/Secondary Data– 2009 State Applications and Annual Reports

(context, grant/organizational structure, population and service characteristics, GPRA)

– Bureau of Census (American Community Survey), HUD (HMIS homelessness count), National Survey of Homeless Assistance Providers (context, community and population characteristics)

Primary Data Collection

• 14 Site Visits– 4 site visits between 12/6/10 and 1/30/10

• Pre-test in-person interview and online survey data collection instruments and client focus group questions for comprehensiveness and usefulness

– Interviews w/SPC, at least 2 provider staff and 1-2 clients

– 10 assessment visits after OMB approval• Detailed assessment of grant structure; process;

perceived outcomes and service gaps. • Recommendations

Data Requirements

• Site Assessments– SPC and key state staff– 1-8 provider Directors and key staff offering direct

service to clients • Depends on size and distribution of provider network

– Focus group with 8-12 clients re: acceptability, quality, gaps.

Online Surveys

• Developed with recommendations from SAMHSA staffers, Expert Panel and pre-test participants

• Will be approved by OMB– Focused on assessing grant implementation

process, throughput, outcomes and gaps– 56 SPCs– 483 Local providers (1/provider agency—staff who

provide direct face-to-face services)

Online Surveys cont.

• Survey will require about 30 minutes to complete

• Unique password assigned to respondent will allow completion over multiple sessions

• Your cooperation in completing these surveys is ESSENTIAL!!! Please help.

Analyses

• Describe PATH implementations– Look for commonalities in structure, populations served

and service mix• Create a typology of program implementations• Relate PATH program models to outcomes to see if

there is differential effectiveness• Identify best practices• Inform SAMHSA about service, training gaps and

inform decisions on technical assistance still needed

Evaluation Project Status

• Developed reliable coding and data extraction procedures for State applications and IUPs– Currently coding both, expect completion by mid-February

• Preparing Annual Report Data to merge with coded information extracted from Annual Reports and State Applications

• Pilot testing of protocols and instruments begun on December 6. Will complete around mid January

• OMB draft with revised instruments to SAMHSA by the end of January

• Field site assessments and surveys should begin by July 2011.

Questions and Comments

• Lisa Kleppel or Paul Brounstein • MANILA Consulting Group• 6707 Old Dominion Drive• McLean, VA 22101• 571-633-9797