Approaches to Evaluating and Measuring Outcomes in Integrated Care

Key Issues for Consideration

Dr Nick GoodwinCEO, International Foundation for Integrated Care

Senior Associate, The King’s Fund

Paper to: Health Quality and Safety Commission New Zealand Workshop: Towards Integrated Care in New Zealand

Wellington, New Zealand, November 14th 2013

What is a ‘programme evaluation’?

• A systematic method of collecting, analysing and using information to answer questions about projects, policies or programmes

• In health care systems, they are particularly concerned with quality of care, patient safety, system efficiency and/or cost effectiveness

• They also seek to examine whether programme goals are, of have been, appropriate and/or useful so can be used to change and adapt strategic directions

• They tend to utilise both qualitative as well as quantitative methods

• They can be ‘formative’ or ‘summative’

Typical components of a ‘programme evaluation’

• Assessment of the need for the programme– Needs assessment/gap analysis/population health planning

• Assessment of the design and/or theory and logic of the programme in supporting its desired influence– Are the assumptions of the programme justified?

• Assessment of how the programme is being implemented– Process evaluation - is it going to plan?

• Assessment of the programmes outcome or impact; cost and efficiency– Outcome level, outcome change, programme’s effect

Rossi, Lipsey and Freeman (2004) Evaluation: a systems approach, Sage

Understanding what to evaluatein an integrated care programme

What are you evaluating – some key questions

• Who and what is the programme seeking to influence?

Need to clarify aim and design of the integrated care intervention by looking at the needs of patients/users

• What is the timeframe over which outcomes are expected to be achieved?

Given this timeframe, which categories of outcomes have the potential to be improved?

• Is there sufficient opportunity in a given population to achieve this targeted improvement in outcomes?

• How can you measure the impact? How can you ensure attribution?

What are you evaluating – some key questions

Before developing questions and/or survey instruments to examine the experience and impact of integrated care from a person’s perspective, there is a need to understand four things:

the programme theory of change – what are the assumptions that lie behind the programme (why?)

the (set of) problems to be addressed (where and who?) the (set of) interventions best suited to address the problem

(what and who?) the strategy best suited to develop, implement, and

evaluate the (set of) interventions (how, when and who?)

What are you evaluating – some key questions

For integrated care to be successful, it needs to execute the following three functions:

accurate identification of individuals within target population (e.g. reliable predictive modelling, health risk assessment, medication list and/or laboratory values from EMRs);

individuals must be enrolled and actively participate in the program for a meaningful period of time (e.g. readiness to change, motivational interviewing, incentives);

the program must include a set of interventions that modify or close deficits in participant and provider behaviour (e.g. tailoring to needs).

Key Points to Consider

Baseline data Define a comparison group Define nature and structure of integrated care being

implemented Include measures of the professionals’ perspective where care

is delivered through multidisciplinary teams Identify what good looks like from a patients’ perspective and

evaluate this through user feedback Include analysis of utilisation and costs of care

Experiences, care outcomes, utilisation & costs

Understanding what to measurein an integrated care programme

Measurement Types - 1Care Outcome Measures Patient outcomes

– e.g. mortality, morbidity, functional status, quality of life Cost and utilisation outcomes

– e.g. hospital admissions, bed days, LOS, nursing home placementsCare Process Measures Occurrence of recommended care activities

– e.g. presence of a care plan; patient follow-up – often processes that are set out in best practice guidance

The measurement tools we have on outcomes and processes are mostly disease-specific - for people with multiple needs, the process of care is less well understood

Examples of outcome measures Hospital utilisation

– Emergency admissions; hospital readmissions; lengths of stay; number of bed days etc …– Disease-specific hospital admissions etc …

‘Social care’ utilisation– Levels of home care support packages– Rates of long-term nursing home/residential home stays

Mortality and disease-specific mortality Short-term clinical outcomes

– e.g., glycated haemoglobin levels for diabetic patients Functional status

– e.g., for CHF patients Quality of life

– e.g., functional dependence Other patient outcomes

– e.g., missed school days for children due to illness; experiences with system Treatment and service adherence

– e.g., remaining in contact with services for mentally ill patients

Measurement Types - 2

Measures of care co-ordination Information exchanges and transfers Relational co-ordination between organisations and professionals

– levels of awareness/interaction among participants– is there a common understanding of care activities and goals– shared culture and mission?

Co-ordination Mechanism Measures - Approaches

Direct observation Interviews and staff surveys (self-report) Medical record audits on information transfer Measures of inter-professional collaboration within teams and

organisations

Examples:ITMA - Integrated Team Monitoring and Assessment Tool – see http://www.readiness-tools.com/tool-full.aspx?toolguid=0d6382ad-f017-4623-8d10-93f2f314e346POET – Partnership Outcomes Evaluation Toolkit – see www.dhcarenetworks.org.uk/asset.cfm?aid=1479

Care Co-ordination Measures Atlas

McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J, and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdf and http://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools

Domains for measuring care co-ordinationCo-ordination activity: Establish accountability/negotiate

responsibility Communication – informational and

inter-personal Facilitate transitions – e.g. across

settings or as coordination needs change

Assess multiple needs and goals Pro-active care planning Monitor, follow-up, review Support self-management Link or refer to community resources Align resources to meet individual or

community needs

Service delivery approaches: Care management Medicines management Healthcare at Home Multi-disciplinary teams ICT-enabled integrated care (e.g.

telehealth)

Perspectives:Family/patient

ProfessionalSystem/organisation

Measurement Types - 3Patient, carer, family-reported perceptions PROMs

• do you feel better? • have your symptoms improved?

PREMs• recommend to a friend? • how satisfied are you?

* PACIC patient assessment of chronic illness care http://www.improvingchroniccare.org/downloads/2004pacic.doc.pdf

* Care Transition Measures (CTM) – patient assessment of discharge from hospital http://www.caretransitions.org/documents/CTM-15.pdf

The more ‘robust’ measures are those that ask about specifics of care co-ordination rather than generalities, and which link data sets together to understand association between responses. Two key things: measures that are specific to a particular group of people; measures that translate to actionable service improvements

Patient Reported Outcome Measures in England

Since 2009, PROMs have been collected for four elective procedures: hip surgery, knee surgery, hernia repair and varicose veins – these costs the English NHS £800m/year

Cataract surgery was to be included, but concerns on methods of data collection

250,000 patients a year invited to complete survey – all NHS providers (100%) collect data

PROMS being extended to cover: anxiety and depression, cancer care, asthma, COPD, diabetes, epilepsy, heart failure, stroke

It’s a careful process – identify the right PROMs instrument; pilot before roll-out; implement data collection; evaluate programme

Uses EQ-5D – part 1 about mobility, self-care, usual activities, pain/discomfort, anxiety/depression; part 2 gives overall assessment of health on 0-100 scale

NOTE: Not sensitive to looking at people with multiple conditions/needs

Some Conclusions on Measuring Integration

Many different tools available: Need to define the client group Need to understand the goal in terms of outcomes to patients and service users Need to create ‘measurable’ outcomes and experiences Measures need to mean something – i.e. that actions can follow

Patients and users tend to understand the term ‘care co-ordination’ or ‘continuity of care’ – e.g. to what extent they feel that care is co-ordinated around their needs

Baseline on measures required on which to base progress over time Link measures to other data – e.g. on clinical outcomes, utilisation, costs Where possible, benchmark performance with others or investigate with a

matched ‘control’ Use data in ‘real time’ to monitor progress and drive performance

Case Example

Integrated Care Metrics NHS London (2012)

Activity metrics currently used in IC systems in London, across all settings of careMetric Definition Data source ICS Emergency admissions (No.)

Number of unplanned admissions for ACS conditions that should not usually require hospital admissionNumber of emergency admissions for the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)Case file audit results for reviews of avoidable ED admissions

SUS / HA data NWL, ONEL, NCL, ONEL,ELC, KHP

Emergency bed days (No.) Number / total number of emergency hospital bed days for IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)Emergency bed days associated with multiple acute hospital admissions

SUS/ HA data ONEL, NWL, ELC

A and E attendances (No.) Number / Total number of A and E attendances for the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)

SUS data NWL, ONEL, NCL, KHP, ELC

Acute re-admissions (No/ %)

All emergency re-admissions that occur within 30 days of discharge for IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)

SUS / HA data NWL, ONEL, KHP

Length of stay (No.) Difference between the discharge time and the admittance time for both elective and non elective episodes divided by the total number of spells for IC cohort Average Length of Stay for patients in the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)

Dr Foster/ SUS / HA data

NWL, ONEL, NCL, ELC

Permanent admissions to residential and nursing care homes (No./ %)

Number of patients in long term care homes as a proportion of the total number of patients in the pilot (%) Number of new permanent admissions to residential and nursing care homes, 65+/ -65Case file audit results for reviews of recent care home admissions

Referral Assessment Provision (RAP) stats

KHP, NWL, Greenwich, NCL, ONEL

Number of people completing re-ablement (No.)

Proportion of service users independent following re-ablement Readmission within 1 year for patients who have completed re-ablement

Referral Assessment Provision (RAP) stats

Greenwich

Community nursing hours per person (No.)

Total number of community nursing visits or units delivered divided by number of patients in pilot

RiO NWL

People supported at home with low/med/high care packages (No.)

Number of people 65+supported at home with low/med/high care packages Referral Assessment Provision (RAP) stats

KHP, Greenwich

Urgent GP response DH survey (Q14-17 seen on same day within primary care 'how convenient' score)

QOF KHP

Process metrics currently used by IC systems in London

Metric Definitions Source ICS

Number of health assessments completed (No.)

Number of assessments complete versus target Outcomes of health assessments Emergency admissions of patients who had health assessments

Local ICS collection KHP

Number of people case managed (No.)

Numbers of case managed patients Caseload, analysed by length of stay, source of referral and state of care plan % currently case managed patients with no ED contact ‘Referral bounce’ for case managed patients

Local ICS collection KHP

Number of care plans completed (No.)

Numbers of patients with personalised care plans Number of people in cohort on a care plan as a proportion of total number of people that should be on a care plan

Local ICS collection ONEL , NWL

Adherence to care plan (No.) Number of patients with one or more delay or incomplete actions as a proportion of the total number of patients in the pilot (%)

Local ICS collection NWL

MDT operation (No.) Numbers of case managed patients discussed at MDT for whom follow up actions took place

Local ICS collection KHP

Attendance at MDT meetings (No.)

Number of meetings scheduled / held / fully attended Number of clinicians attending monthly MDT case conferences and quarterly review meetings as proportion of total number expected at these meetings (%)

Local audit KHP

Frequency of MDT meetings (No.)

Virtual Ward: number of monthly MDT meetings held Local ICS collection ELC

Degree of joint working Attendance rates at joint meetings Exception reporting – local ICS collection

ONEL

Metric Definition Source ICS Control measures Patients with T1/T2 DM with HbA1c less than or equal to 10

% of patients with blood pressure targets achieved as per NICE guidance Percentage of patients with cholesterol less than or equal to 5BMI less than or equal to 30proportion of patients receiving medication review who should

QOF NWL

Hard outcomes Speed of referral for recognised foot complications, Amputation rate below the ankle, Falls rate among the frail elderly, Number of fractures

QOF NWL

Outcome metrics currently used by IC systems in London

Patient and staff experience measures used by IC systems in London

Metric Definition Source ICS PROMS Measure of quality from the patient perspective covering

four clinical procedures: hip replacements, knee replacements, hernia, varicose veins

NHS Information Centre NWL, ONEL

PREMS Measure of patients experience based on range of different possible metrics: patient recommendation, overall satisfaction, complaints, patient confidence

Patient surveys and questionnaires, focus groups, complaints data, one on one patient interviews

NWL , ONEL

Patient satisfaction surveys

Results of ICS service user survey Percentage of people with LTCs who feel supported to manage their condition The proportion of people who use services who feel safe

DH GP survey 032 KHP, ONEL

LTC6 survey 1. Did you discuss what was most important for you in managing your own health?

2. Were you involved as much as you wanted to be in decisions about your care or treatment?

3. How would you describe the amount of information you received to help you to manage your health?

4. Have you had enough support from your health and social care team to help you to manage your health?

5. Do you think the support and care you receive is joined up and working for you?

6. How confident are you that you can manage your own health?

Local health observatory

Staff satisfaction Staff survey to determine satisfaction of IC pilot, to determine number of staff responding very satisfied and satisfied to survey as proportion of total number of staff surveyed

Local ICS collection NWL

Contact

Dr Nick GoodwinCEO, International Foundation for Integrated Care

nickgoodwin@integratedcarefoundation.orgwww.integratedcarefoundation.org

@goodwin_nick @IFICinfo

Appendix 1: From a measure to an indicator that can be used for quality improvement

In developing ‘indicators’ to judge comparative performance, the following criteria are important:

Statistical validity Accurate – measures what is says it measures! Reliable – can be tracked over time Consistent – data collection robust and reproducable Avoids bias

Data considerations Data source – sample or full population; existing or new data source? Unit of assessment – country, region, locality, practice etc Client groups covered – what is the targeted population? account for bias/case mix? Significance – smaller groups, reduced statistical power Patient reported data is indirect – how cross-compare to other data (multi-methods) Coverage – health, health & social care, health, social care and housing ….

Face validity Meaningful to public; clinically credible; Potential to support quality improvement; Cost and value for money

Appendix 2: Key Resources on Care Co-ordination MeasuresMcDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J,

and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdf ttp://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools

King et al - 1995/2004 - Measures in the process of care – MPOC http://www.canchild.ca/en/measures/mpoc56_mpoc20.asp

Flocke SA. Measuring attributes of primary care: development of a new instrument. J Fam Pract. 1997 Jul;45(1):64-74.

Safran DG, Kosinski M, Tarlov AR, et al. The Primary Care Assessment Survey: tests of data quality and measurement performance. Med Care. 1998 May;36(5):728-39.

Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA. Measuring consumer experiences with primary care. Pediatrics. 2000 Apr;105(4 Pt 2):998-1003.

Starfield B, Cassady C, Nanda J, Forrest CB, Berk R. Consumer experiences and provider perceptions of the quality of primary care: implications for managed care. J Fam Pract. 1998;46(3):216-26.

Grimmer K, Moss J. The development, validity and application of a new instrument to assess the quality of discharge planning activities from the community perspective. Int J Qual Health Care. 2001 Apr;13(2):109-16

McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of health care. Int J Qual Health Care. 2003 Aug;15(4):309-18.

Radwin L, Alster K, Rubin KM. Development and testing of the Oncology Patients' Perceptions of the Quality of Nursing Care Scale. Oncol Nurs Forum. 2003 Mar- Apr;30(2):283-90.

Billings J, Coxon K, Alaszewski A. Empirical research methodology for ‘Procare’ version 3. University of Kent at Canterbury: Centre for Health Services Studies; 2003. Available from: http://www.kent.ac.uk/chss/docs/procare_version3.pdf.

Morita T, Hirai K, Sakaguchi Y, Maeyama E, Tsuneto S, Shima Y. Measuring the quality of structure and process in end-of-life care from the bereaved family perspective. J Pain Symptom Manage. 2004 Jun;27(6):492-501.

Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient's perspective: the care transitions measure. Med Care. 2005 Mar;43(3):246-55. (also CTM website)

Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the Patient Assessment of Chronic Illness Care (PACIC).Med Care. 2005 May;43(5):436-44

Drewes et al (2012) PCIC+ at http://www.ijic.org/index.php/ijic/article/view/862/1896 Haggerty J, Roberge D, Freeman G, Beaulieu C, Breton M. Review. When patients encounter

several providers: validation of a generic measure of continuity of care. Annals of Family Medicine (In press) and Breton et al at http://www.ijic.org/index.php/ijic/article/view/682/1534

Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): Conceptualizing and Measuring Activation in patients and consumers. Health Services Research 2004: 39:1005-1026.

Powell, R. Powell, H. Baker, L. & Greco, M. (2009) Patient Partnership in Care: A new instrument for measuring patient–professional partnership in the treatment of long-term conditions. Journal of Management & Marketing in Healthcare. Vol. 2 No. 4. PP 325–342.

Long Term Conditions 6 (LTC 6) (part of the QIPP programme) Department of Health and also Living with your Long Term Condition (LWYLTC)

Safran DG, Karp M, Coltin K, Chang H, Li A, Ogren J, et al. Measuring Patients' Experiences with Individual Primary Care Physicians Results of a Statewide Demonstration Project. Journal of General Internal Medicine. 2006;21(1):13-21.

Adair CE, McDougall GM, Mitton CR, Joyce AS, Wild TC, Gordon A, et al. Continuity of Care and Health Outcomes Among Persons With Severe Mental Illness. Psychiatric Services. 2005;56(9):1061-9.

King M, Jones L, Nazareth I. Concern and continuity in the care of cancer patients and their carers: A multi-method approach to enlightened management. London: National Coordinating Centre for NHS Service Delivery and Organisation 2006.

Dolovich LR, Nair KM, Ciliska DK, Lee HN, Birch S, Gafni A, et al. The Diabetes Continuity of Care Scale: the development and initial evaluation of a questionnaire that measures continuity of care from the patient perspective. Health Soc.Care Community. 2004;12(6):475-87

Kowalyk KM, Hadjistavropoulos HD, Biem HJ. Measuring Continuity of Care for Cardiac Patients: Development of a Patient Self-report Questionnaire. Canadian Journal of Cardiology. 2004;20(2):205-12.

Hadjistavropoulos H, Biem H, Sharpe D, Bourgault-Fagnou M, Janzen J. Patient Perceptions of Hospital Discharge: Reliability and Validity of a Patient Continuity of Care Questionnaire. International Journal for Quality in Health Care : Journal of the International Society for Quality in Health Care / ISQua. 2008;20(5):314-23.

Borowsky SJ, Nelson DB, Fortney JC, Hedeen AN, Bradley JL, Chapko MK. VA community-based outpatient clinics: performance measures based on patient perceptions of care. Med Care. 2002;40(7):578-86.