The Tripadvisor approach for health? Using patients' online descriptions of their care to understand healthcare quality

Felix Greaves

Imperial College London

@felixgreaves

England is proud of the National Health Service

But it is not without problems Quality can be variable, and we often fail to hear the patient perspective

At the same time, we love rating things on the internet

We now do it for our experiences of healthcare too

The number of patients describing their care online is increasing

USA UK

Gao et al., JMIR, 2012 Greaves et al., JMIR, 2012

Aug-0

8

Nov-0

8

Feb-0

9

May

-09

Aug-0

9

Nov-0

9

Feb-1

0

May

-10

Aug-1

0

Nov-1

0

Feb-1

1

May

-11

Aug-1

1

Nov-1

10

5000

10000

15000

20000

25000

Nu

mb

er

of h

osp

ital r

atin

gs

There is resistance from many clinicians to the idea of being rated online:

Arguments for and against online rating

Why it’s bad

1. Strain doctor-patient relationships

2. Reviews may be malicious or fake

3. Selection bias by those leaving reviews

4. Lack of meaningful data on technical quality of health care

Why it’s good

1.Doctors can often be poor judges of their patients’ experience

2.Feedback changes doctors’ performance

3.People are using the Internet to voice opinions, so why not capture this information in a useful form

Research Question

Is there a relationship between ratings online and ‘hard’ measures of quality?

The opportunity for a natural experiment

NHS Choices website allows patients to review all NHS services in England

Method:

Obtained national review data from the Department of Health

We thought we could compare ratings with some traditional measures of quality

Vs.

Average scores for 10,000 ratings

Score Mean rating (out of 5) Range

The environment where I was treated was…

3.6 2.6-5.0

I was treated with dignity and respect by the hospital staff…

4.0 2.7-5.0

I was involved with decisions about my care…

3.8 2.4-5.0

The hospital staff worked well together…

4.1 2.9-5.0

Greaves et al. BMJ Quality and Safety, 2012

Average scores for 10,000 ratings

Score Mean rating (out of 5) Range

The environment where I was treated was…

3.6 2.6-5.0

I was treated with dignity and respect by the hospital staff…

4.0 2.7-5.0

I was involved with decisions about my care…

3.8 2.4-5.0

The hospital staff worked well together…

4.1 2.9-5.0

Greaves et al. BMJ Quality and Safety, 2012

• The majority of ratings are positive• 67% would recommend to a friend

Ratings compared to patient experience surveys

NHS Choices Measure Survey question: Spearman

Rho p value

Proportion of patients recommending

“Overall, how would you rate the quality of care you received”

0.40 <0.001

Rating of being treated with dignity and respect

“Overall, did you feel you were treated with dignity and respect while in hospital?”

0.33 <0.001

Rating of staff working together

“How well would rate how well the doctors and nurse worked together?”

0.32 <0.001

Rating of cleanliness “How clean was the hospital ward or room you were in?”

0.48 <0.001

Greaves et al. BMJ Quality and Safety, 2012

Ratings compared to patient experience surveys

NHS Choices Measure Survey question: Spearman

Rho p value

Proportion of patients recommending

“Overall, how would you rate the quality of care you received”

0.40 <0.001

Rating of being treated with dignity and respect

“Overall, did you feel you were treated with dignity and respect while in hospital?”

0.33 <0.001

Rating of staff working together

“How well would rate how well the doctors and nurse worked together?”

0.32 <0.001

Rating of cleanliness “How clean was the hospital ward or room you were in?”

0.48 <0.001

There is a moderate, highly significant association between ratings online and large surveys of patient experience

Greaves et al. BMJ Quality and Safety, 2012

Ratings compared to outcomes

NHS Choices Measure Other variable Spearman Rho p value

Proportion of patients recommending

Hospital Standardised Mortality Ratio

-0.20 0.01

Proportion of patients recommending

Standardised morality rate for high risk conditions

-0.22 0.01

Proportion of patients recommending

Standardised morality rate among surgical inpatients with serious treatable complications

-0.00 0.99

Proportion of patients recommending

Emergency readmission rate within 28 days

-0.31 <0.001

Patient perception of cleanliness

Rate of MRSA bacteraemia (per 1,000 bed days)

-0.30 <0.001

Greaves et al. Arch Int Med, 2012

Ratings compared to outcomes

NHS Choices Measure Other variable Spearman Rho p value

Proportion of patients recommending

Hospital Standardised Mortality Ratio

-0.20 0.01

Proportion of patients recommending

Standardised morality rate for high risk conditions

-0.22 0.01

Proportion of patients recommending

Standardised morality rate among surgical inpatients with serious treatable complications

-0.00 0.99

Proportion of patients recommending

Emergency readmission rate within 28 days

-0.31 <0.001

Patient perception of cleanliness

Rate of MRSA bacteraemia (per 1,000 bed days)

-0.30 <0.001

There is a weak, significant association between ratings online and mortality rates

Greaves et al. Arch Int Med, 2012

A study in the US found the same results comparing Yelp reviews with the HCAHPS survey

A wisdom in the crowd of patients?

Sir Francis Galton won ‘guess-the-weight-of-the-bull’ competitions by asking lots of local farmers for their best guess, and then taking the average – he gave us the idea of a ‘wisdom of crowds’

Comparison of the NHS Inpatient Survey and ratings on the NHS Choices website

NHS Inpatient Survey NHS Choices ratings

Mechanism Paper-based survey Ratings left on a website

Number of responses

69,000 per year 5,000 per year

Selection Random; patients receive a survey requesting completion after leaving hospital

Self-selecting; patients are not solicited

Proportion positive

79% rated their overall care as excellent or very good

67% would recommend to a friend

Cost Likely more expensive Likely less expensive

Comparison of the NHS Inpatient Survey and ratings on the NHS Choices website

NHS Inpatient Survey NHS Choices ratings

Mechanism Paper-based survey Ratings left on a website

Number of responses

69,000 per year 5,000 per year

Selection Random; patients receive a survey requesting completion after leaving hospital

Self-selecting; patients are not solicited

Proportion positive

79% rated their overall care as excellent or very good

67% would recommend to a friend

Cost Likely more expensive Likely less expensive

There were 10,000 hospital ratings in the UK over 2 years

Over the same time period there were 29,118,009 hospital admissions

0.04% of hospital admissions are rated

The stereotypical reviewer?

Where are people rating their care?

Associations between whether a practice is rated with population and practice characteristics

Independent variable

Z statistic p value

Practice population size 

15.38 <0.001

IMD score of patients 

-7.82 <0.001

Population density 

6.72 <0.001

Singlehander  -4.50 <0.001Proportion of population aged over 65 years 

-3.88 <0.001

Proportion of population who are white

-1.58 0.11

Type of contract -0.71 0.48Training practice  0.35 0.73

Associations between whether a practice is rated with population and practice characteristics

Independent variable

Z statistic p value

Practice population size 

15.38 <0.001

IMD score of patients 

-7.82 <0.001

Population density 

6.72 <0.001

Singlehander  -4.50 <0.001Proportion of population aged over 65 years 

-3.88 <0.001

Proportion of population who are white

-1.58 0.11

Type of contract -0.71 0.48Training practice  0.35 0.73

• Practices serving younger people are more likely to be rated

• Practices serving less deprived people are more likely to be rated

• Practices in urban areas are more likely to be rated

Cloud of patient experience ?

A problem

Free Text

A new trend: ‘Big Data’ and social media analytics

Sentiment analysis

Another natural experiment

We can compare patients’ free text descriptions of care with their own quantitative ratings

What we did: Machine learning

• You need to teach an algorithm how to recognise particular words and phrases

• We used all comments and ratings from 3 years as a training set (13,802 comments)

We tried to predict patients ratings of their care from their comments in 2010

• Whether the patient would recommend the hospital or not• Whether the hospital was clean or not• Whether the patient was treated with dignity or not

Used open source Weka software

Sentiment analysis can be tricky

Simple punctuation contains little information

(

But when paired with other punctuation, it’s very important

:(

And small changes can have big effects on sentiment

:)

Some words the algorithm thought were important

Overall Cleanliness Dignity• rude • dirty • rude• excellent • floor • told• hours • filthy • thank• pain • bed • friendly• communication • blood • attitude

How good is it at predicting ratings?

Question we are predicting the answer to

Total number of comments

Prediction accuracy

Kappa P value

Would you recommend the hospital?

 6412 88.7 0.75 <0.0001

How clean was the hospital room or ward that you were in?

 6139 81.2 0.40 <0.0001

Overall, did you feel you were treated with respect and dignity while you were in the hospital?

 6239 83.6 0.56 <0.0001

How does this compare to patient surveys?

Patient Survey Question Machine learning prediction

Spearman rho Probability

Overall, how would you rate the care you received?  

Whether the patient would recommend the hospital 0.46 P<0.001

Overall, did you feel you were treated with respect and dignity while you were in the hospital?  

Whether the patient was treated with dignity 0.50 P<0.001

In your opinion, how clean was the hospital room or ward that you were in? 

Standard of cleanliness 0.37 P<0.001

Limitations

• Selection bias• Sarcasm / Irony• Culturally specific

Phrases like ‘cup of tea’ important,

but effect depends on context

The UK quality regulator nowincludes online reviews and comments in its performance measurement framework

The NHS has started publically reporting social media sentiment

ConclusionsOnline rating and reviewing are on the rise

Evidence of some wisdom in the crowd of patients

Substantial professional resistance to the idea

Selection bias in those commenting online

The ‘cloud of patient experience’ is a potentially valuable source of information to understand patient views

• Christopher Millett• Ara Darzi• Dominic King• Henry Lee• Utz Pape• Liam Donaldson• Azeem Majeed• Robert Wachter• Daniel Ramirez-Cano

Thanks to my co-authors, and funders