Using HMIS for Program Evaluation September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S....

Using HMIS for Program Evaluation

September 13-14, 2005St. Louis, Missouri

Sponsored by the U.S. Department of Housing and Urban Development

Daria Zvetina, Zvetina & Associates, Inc.

Christine Mirasy-Glasco, Los Angeles Homeless Services Authority

Charlene Flaherty, Arizona Dept. of Econ. Security




Daria Zvetina Zvetina & Associates, Inc.

September 13-14, 2005 St. Louis, MissouriSponsored by the U.S. Department of Housing and Urban Development

3

What Is Program Evaluation?

Program evaluation, also called performance measurement, is a systematic effort to assess whether your program is making an impact in the areas that you intend.


4

Why Is It Important?

• Allows you to determine whether you are achieving your intended results

• Allows you to modify your approach or activities if you are not achieving your intended results

• Allows you to use outcome data to communicate the effectiveness of your services to funders and your community


5

• And, program evaluation is a necessity in the increasingly performance/outcomes driven funding environment.



6

You can measure program performance at several levels

• Process evaluation – are we on track to achieve our outcome targets?

• Outcome evaluation – did we achieve our outcomes to the extent and within the time frame anticipated?


7

How Do We Get There?

• Dare I say, “logic model”?


8

Outcome Logic Models

Logic models are a simple, straightforward way to summarize your resources, efforts, and end.

Inputs Activities Outputs OUTCOMES


9

Why Use a Logic Model?

Logic models provide a framework to help you:

• Organize your thoughts and plans• Be intentional about your efforts and allocation of resources• Ensure a common understanding exists among all partners,

staff, and consumers about what you’re attempting to achieve and how you intend to do it

• Clearly communicate to others (e.g., funders, the public and media, other stakeholders) the purpose and value of your efforts

• Regularly evaluate whether your resources, activities, and program approach are aligned and sufficient to achieve your intended results


10

InputsActivitiesOutputs OUTCOMES

• Inputs include resources dedicated to or consumed by the program, such as money, staff and staff time, volunteers and volunteer time, facilities, equipment and supplies.

• Activities are what the program does with the inputs to fulfill its mission, such as sheltering and feeding homeless families or training and counseling adults to help them prepare for jobs.

• Outputs are the direct products of program activities. They usually are measured in terms of the volume of work accomplished, such as the numbers of participants served and the units of service.

• Outcomes are benefits or changes for individuals or populations during or after participating in program activities. Outcomes may relate to knowledge, attitudes, values, skills, behaviors, conditions, or other attributes.


11

Outcomes

• An outcome should be:• Focused on what the individual will gain from the

program.• Measurable with clear targets, specified timeframes,

and methods for measuring change.• Specific to the program and directly attributable to it.

Attainable.• Understandable to someone outside of the program.


12

Sample Outcome for HUD Permanent Housing Outcome Domain

HUD Program Goal: Obtain and remain in permanent housing

Sample outcome for an SHP rolling stock transitional housing program:

• 100% (30) of program families will be placed in subsidized housing at program entrance; at least 90% (27) will remain in this housing for 6 months; 80% (24) will remain in this housing at program exit or move to comparable or better housing; and 75% (23) will still be stably housed 6 months later.

• Measurement strategy: HMIS data gathered at program entrance, exit, and 6-month follow-up


13

Sample Outcomes for HUD Skills/Income Outcome Domain

HUD Program Goal: Increase skills and income

• 90% (24) of program participants will increase their income from program entrance to exit via access to additional benefits and/or employment; 70% (21) will have achieved another increase in income within six months of program exit.

• 70% (21) of participants will have obtained livable wage employment at program exit; 70% (15) of those employed will be employed at equal or greater pay six months after program exit.

Measurement strategy: HMIS data gathered at program entrance, exit, and 6-month follow-up


14

Potential Benefits of Using HMIS to Collect and Analyze

Program Outcome Data

• Information is systematically gathered• Easy to extract (theoretically)• Is potentially available in real time• Can be used to complete reports for funders (e.g.,

HUD APR) and other constituents• Permits richer, more nuanced analysis (e.g.,

allows you to examine whether your program is working better for some participants than others)

• Permits examination of data trends and cross year or multi-year analysis


15

Challenges to Using HMIS for Program Evaluation

• Programs must enter information routinely, accurately, and reliably (e.g., data must be coded consistently within and across programs)

• Many providers will have to conquer their technology anxiety (akin to math anxiety?)

• To ensure use, providers must have access to IT help desk

• To garner ANY of the benefits listed previously, participating agencies/programs MUST have DIRECT ACCESS to their data


16

HMIS Data Elements

Universal Data Elements: data elements that must be collected by ALL agencies serving persons who are homeless

Program Specific Data Elements: elements that must be collected by all HUD funded agencies required to complete the APR


17

HMIS Universal Data Elements

• 2.1 Name• 2.2 SSN• 2.3 DOB• 2.4 Ethnicity & Race• 2.5 Gender• 2.6 Veteran Status• 2.7 Disabling Condition

• 2.8 Residence Prior to Program entrance• 2.9 Zip Code Last Permanent Address• 2.10 Program Entry Date• 2.11 Program Exit Date• 2.12 Unique Person ID #• 2.13 Program ID #• 2.14 Household ID #


18

Program Specific Data Elements

• 3.1 Income and Sources• 3.2 Non-Cash Benefits• 3.3 Physical Disability• 3.4 Developmental Disability• 3.5 HIV/AIDS• 3.6 Mental Health• 3.7 Substance Abuse• 3.8 Domestic Violence

• 3.9 Services Received• 3.10 Destination• 3.11 Reasons for Leaving• 3.12 Employment• 3.13 Education• 3.14 General Health Status• 3.15 Pregnancy Status• 3.16 Veteran’s Information• 3.17 Children’s Education


19

Using HMIS Data Elements to Gather Housing Outcome Data

Permanent Housing Outcome: 100% (30) of program families will be placed in subsidized housing at program entrance; at least 90% (27) will remain in this housing for 6 months; 80% (24) will remain in this housing at program exit or move to comparable or better housing; and 75% (23) will still be stably housed 6 months later.

Relevant HMIS data elements: 2.10 Program Entry Date 3.10 Destination 2.11 Program Exit Date 3.11 Reasons for Leaving

To assess housing status, 6-months post-exit an additional data collection timeframe would need to be added.


20

Using HMIS Data Elements to Gather Income Outcome Data

Increased Income/Skills Outcome: 90% (24) of program participants will increase their income from program entrance to exit via access to additional benefits and/or employment; 70% (21) will have achieved another increase in income within six months of program exit.

• 70% (21) of participants will have obtained livable wage employment at program exit; 70% (15) of those employed will be employed at equal or greater pay six months after program exit.

Relevant HMIS data elements: • 2.10 Program Entry Date 3.12 Employment• 2.11 Program Exit Date 3.2 Non-Cash Benefits• 3.1 Income and Sources


21

Value Added HMIS Program Evaluation Capabilities

Beyond determining whether housing and income/skill outcome targets have been achieved, the universal and program-specific data elements afford programs the potential to ask whether housing or income/skill outcomes vary systematically according to:

• Participant characteristics or circumstances such as:• Disability status and type• Health status• Education• Employment status at intake or exit

• As a result of specific program features such as:• Length of time in the program• Services received




Christine Mirasy-Glasco, Los Angeles Homeless Services Authority


23

Background

• Los Angeles Homeless Services Authority’s (LAHSA) primary role is to coordinate and manage funding for homeless programs in the Los Angeles Continuum of Care.

• Challenge: Fund programs that demonstrate success in ending homelessness.

• Tool: HMIS• Pilot Program Experience


24

Summary

• Mindset shift – from activities to outcomes (results)

• Necessary prerequisites• Critical data elements• Measures of success – verification and

evaluation• Benefits of new approach• Lessons learned


25

Mindset Shift

• Traditional approach – provision of services measured by frequency or intensity of program activities. Data collected to report to other entities i.e. grantor or funder

• Outcome approach – provision of services measured by verifiable results. Meaningful data is the cornerstone of implementing a successful program. Hence, data collected should be primarily used to inform the program implementers as well as grantors.

• Bottom line: Both funder and program implementer need and want to know how to ask and answer the result question.


26

Necessary Prerequisites

• Understanding the funder’s intent, i.e. ending homelessness, increasing affordable housing stock etc.

• Identifying the goals of the program, i.e. obtain and maintain permanent housing; increase skills and/or income; and achieve greater self-determination.

• Establishing performance measures that express specific changes in participant condition or behavior as it relates to identified goals.


27

Critical Data Elements

• What question(s) are you answering?• What data elements do you need to answer

the question(s)?

Example• Question: How successful is my program in

assisting participants in obtaining and maintaining permanent housing?


28

Critical Data Elements

Performance measure:• 45% (54) of participants will be placed in PH when

they exit the program.• 30% (16) of those participants placed in PH will

remain in housing for 4 months after they exit the program.

Data elements required: • # of participants placed in PH• Types of PH i.e. market rate apartment, shared

housing, Section 8 housing. • Program exit date.• Follow-up mechanism with dates at 1 month, 2

months, 3 months and 4 months.


29

Measures of Success

• Verification – establishing the achievement of a result by documentation.

• This is obtained by generating reports from HMIS that query specific data elements established in advance.

• It provides a quick yes or no response to the question “did we achieve our targets?”


30

Measures Of Success

• Evaluation – at the end of the day, both funder and program implementer want to know whether the program is effective. Therefore, the question being asked here is “what have we learned?”

• The information obtained through this exercise should be used to change or modify program aspects that are not producing results.

• he information is also useful for funders in making funding and policy decisions.


31

Benefits of Outcome Approach

• Ability to detect the need for correction in real time throughout program implementation.

• Immediate feedback on program success to key stakeholders.

• Early evidence of program effectiveness that can be used to help build a case for continued funding.


32

Lessons Learned

What worked• Dispelling the misconception that the more clients

served the better. Emphasis should be on what the clients achieved and whether it is consistent with the program’s goals.

• HMIS is a potent tool for program implementers to manage programs in a proactive manner. It is not a tool for funders to penalize program implementers who are not meeting their targets.

• Realization that program implementers and funders are real partners.


33

Lessons Learned

What did not work• Use of HMIS as a daily activity. As a result, very

limited information could be retrieved to measure program progress.

• Overall agency buy-in. The benefits of HMIS were not well communicated to executive staff, line staff and clients. As a result, making changes to the program using information from HMIS was difficult.


34

Next Implementation Phase

• Deeply involve agency staff from all levels in the implementation process from the beginning.

• Invest time in explaining the benefits of performance targets and milestones/benchmarks to program implementers.

• Agree on protocol of checking utilization of the system by users.

Investing in Outcomes:The Arizona Homeless Program

Evaluation Project



Charlene Moran FlahertyArizona Department of Economic Security


36

Objectives

• Background

• Partners

• Tools/Methodology

• Timeline

• Next Steps


37

What Is It?

Tools and methodology for…

• Evaluating the effectiveness of homeless programs

• Improving outcomes in order to make a real difference in the lives of homeless people

• Making informed decisions regarding investments

• Identifying best practices


38


• Conscious shift from focus on outputs to investing in outcomes

HUD McKinney-Vento grant

State Performance Measures

Private Investors


39

Goal

• Reduce the gap between knowledge and practice

Which programs are effective?

What interventions produce results?

Where do we target investment?


40

Partners

• Arizona Dept. of Economic Security

• Arizona Dept. of Housing

• Continua of Care

• Virginia G. Piper Charitable Trust

• HUD


41

Provider Engagement

• Data to self-evaluate programs

• System for obtaining feedback

• Celebrate best practices

• Potential for technical assistance


42

Developing the System

• Statewide Steering Committee

• Provider Meetings

• Surveys of Providers

• Testing of Pilot

• Outcomes Training


43

Project Components

• Self-Sufficiency Matrix

• Annual Progress Report data

• Housing placement success

• Provider-developed outcomes


44

Self-Sufficiency Matrix

Domain Score 1 2 3 4 5

Income no income Inadequate income and/or spontaneous or inappropriate spending

Can meet basic needs with subsidy; appropriate spending

Can meet basic needs and manage debt without assistance

Income is sufficient, well managed; has discretionary income and benefits

Employment No job Temporary, part-time or seasonal; inadequate pay, no benefits.

Employed full time; inadequate pay; few or not benefits

Employee full time with adequate subsidized housing.

Maintains permanent employment with adequate income and benefits.

Housing Homeless or threatened with eviction

In transitional, temporary or substandard housing; and/or current rent/mortgage payment is unaffordable (over 30%of income)

In stable housing that is safe but only marginally adequate.

Household is in safe, adequate, subsidized housing.

Household is safe, adequate, unsubsidized housing.

Food No food or means to prepare it. Relies to a significant degree on other sources of free or low-cost food.

Household is on food stamps

Can meet basic food needs, but requires occasional assistance.

Can meet basic food needs without assistance.

Can choose to purchase any food household desires.

Childcare Needs childcare, but none is available/accessible and/or child is not eligible.

Child case is unreliable or unaffordable, inadequate supervision is a problem for child care that is available

Affordable subsidized childcare is available, but limited.

Reliable, affordable childcare is available, no need for subsidies

Able to select quality childcare of choice

Safety Home or residence is not safe; immediate level of lethality is extremely high; possible CPS involvement

Safety is threatened/temporary protection is available; level of lethality is high




Parenting Skills

There are safety concerns regarding parenting skills






45

Self-Sufficiency Matrix, cont.

Children's Education

One or more eligible children not enrolled in school.

One or more eligible children enrolled in school, but not attending classes.

Enrolled in school, but one or more children only occasionally attending classes.

Enrolled in school and attending classes most of the time.

All eligible children enrolled and attending on a regular basis

Adult Education Literacy problems and/or no high school diploma/GED are serious barriers to employment.

Enrolled In literacy and/or GED program and/or has sufficient command of English to where language is not a barrier to employment.

Has high school diploma/GED

Needs additional education/training to improve employment situation and/or resolve literacy problems to where they are able to function effectively in society.

Has completed educational/training needed to become employable. No literacy problems

Legal Current outstanding tickets or warrants.

Current charges/trial pending, noncompliance with probation/parole.

Fully compliant wit probation/parole terms.

Has successfully completed probation/parole within past 12 months, no new charges filed.

No active criminal justice involvement in more than 12 months and/or no felony criminal history

Health Care No medical coverage with immediate need.

No medical coverage and great difficulty accessing medical care when needed. Some household members may be in poor health.

Some members (eg Children on AHCCCS

All members can get medical care when needed, but may strain budget.

All members are covered by affordable, adequate health insurance.

Life Skills Unable to meet basic needs such as hygiene, food, activities of daily living.

Can meet a few but not all needs of daily living without assistance.

Can meet most but not all daily living needs without assistance.

Able to meet all basic needs of daily living without assistance

Able to provide beyond basic needs of daily living for self and family.

Mental Health Danger to self or others; recurring suicidal ideation; experiencing severe difficulty in day-to-day life due to psychological problems.

Recurrent mental health symptoms that may affect behavior, but not a danger o self/others; persistent problems with functioning due to mental health symptoms.

Mild symptoms may be present but are transient; only moderate difficulty in functioning due to mental health problems.

Minimal symptoms that are expectable response to life stressors; only slight impairment in functioning

Symptoms are absent or rare; good or superior functioning in wide range of activities; no more than every day problems or concerns.


46

Pilot Data Collection

• 21 programs committed to participate

• 13 programs provided pre- data

25% emergency 50% transitional 25% permanent

supportive


47

Pilot Data Collection, cont.

• 9 programs provided both pre- and post- data

• Attrition pre- and post-

12% emergency 42% transitional 99% permanent

supportive


48

Scoring Systems

• System #1

Independence Dysfunction Child Functioning

• System #2

Self-Sufficiency Community Connectedness


49

Level of Functioning

•Permanent Supportive Housing

•Transitional Shelter

•Emergency Shelter


50

Results

• More challenging clients show greatest increase in

self-sufficiency

Lower expectations; greatest gains

• Less challenging clients show smallest increase in

self-sufficiency

Higher expectations; smaller gains


51

Timeline

• Outcomes Training – March 2005

• Integrate Matrix w/ HMIS – June 2005

• Provider Training on Matrix – July 2005

• Statewide Rollout – August/September 2005

DES projects HUD projects


52

Vertical Integration

Service Integration

Homelessness

State PlanImplementation


53

Next Steps

• Data collection

• Analysis/Mathematical modeling

• Develop data warehouse

• Identify and replicate best practices

• State reports

• Provider reports

Using HMIS for Program Evaluation September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S....

Documents

Transcript of Using HMIS for Program Evaluation September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S....