Post on 15-Jan-2016
Consumers’ Appraisal of AnecdotalAccounts of Patient Experience
AHRQ Annual Conference
Steven C. Martino, PhD
September 10, 2012
Martino-2 Sep-12
Customer Reviews of Products and Servicesare Omnipresent Online
• Amazon.com first began allowing customers to post reviews of products in 1995
• Now almost all retail websites include them
• 78% of internet users (58% of adults) have researched a product or service online, and 32% have posted a comment or review (Pew Internet & American Life Survey, 2011)
• Reviews are a trusted source of information on product or service quality
• Current retail industry focus is on increasing the relevance of online customer reviews
Martino-3 Sep-12
Martino-4 Sep-12
Martino-5 Sep-12
Martino-6 Sep-12
Narrative Accounts of Patient Experience are Increasingly Available Online
• Dozens of commercial websites provide narrative accounts of patients’ experiences with their doctors (sometimes accompanied by ratings)
– e.g., Vitals.com, HealthGrades, Yelp, Angie’s List, and Consumer Reports
• RateMDs currently hosts narrative reviews of at least 1 in 6 practicing doctors in the U.S. (Gao et al., 2012)
• Currently, the only equivalent in the public sector is NHS Choices (British National Health Service)
Martino-7 Sep-12
Currently, the Audience for these Sitesis Small
• 80% of internet users (59% of adults) have looked online for health information
• 16% of internet users have consulted online rankings or reviews of doctors or other providers (4% have posted such reviews)
• 15% have consulted rankings or reviews of hospitals or other medical facilities (3% have posted such reviews)
Data source: Pew Internet & American Life Survey conducted Aug-Sep 2010
Martino-8 Sep-12
Martino-9 Sep-12
Martino-10 Sep-12
Promises and Limitations of Patient Narratives
• Patient narratives often cover the same domains as standardized surveys of patient experience but in ways that can be more vivid, concrete, and engaging
• Yet, the sample of respondents is often not representative (biased) and small (poor signal, low reliability)
– On RateMDs, the average number of reviews per physician is three (Lagu et al., 2010)
• If presented with standardized data, patient narratives could make unfamiliar, difficult-to-understand quality measures more salient and evaluable
• Yet, anecdotal data may overwhelm more formal statistical information
Martino-11 Sep-12
Extracting Meaning from Narratives is aComplex Cognitive Task
Enduring Characteristics of the Doctor
• Technical ability• Interpersonal skill
Enduring Characteristics of the Environment
• Health system guidelines• Case load and mix
Unique Aspects of the Situation
• Individual patient• Circumstantial context
Doctor’s Behavior
Martino-12 Sep-12
Extracting Meaning from Narratives is aComplex Cognitive Task
Enduring Characteristics of the Doctor
• Technical ability• Interpersonal skill
Enduring Characteristics of the Environment
• Health system guidelines• Case load and mix
Unique Aspects of the Situation
• Individual patient• Circumstantial context
Doctor’s Behavior
Consumers try to infer this from this
Martino-13 Sep-12
Cognitive Strategies that People Use to Discern Among Multiple Possible Causes of Behavior . . .
• Typically, people look for– Consistency in behavior across time and
situations: Has the doctor behaved a certain way repeatedly across time and situations?
– Consensus in people’s experiences with the person: Do other patients report getting similar treatment from the doctor?
– Distinctiveness of behavior: Is this doctor the only doctor to have behaved this way toward the patient?
Martino-14 Sep-12
. . . Are Not Necessarily Applicable in the Case ofPatient Narratives
• Consumers often do not have the information needed to answer questions about consistency, consensus, and distinctiveness.
– Usually limited to a small number of narratives– Do not get to “observe” the doctor’s behavior
over multiple occasions with the same patient or get the patient’s view of other doctors
– Narratives rarely mention situational factors that may have facilitated or constrained behavior; instead, they strongly imply dispositional causes
Martino-15 Sep-12
Consumers’ Values, Disposition, and Past Experiences May Affect Their Appraisal
• Consumers differ in how much they value data on patient experience and the importance they ascribe to various facets of patient experience and doctor quality
• Consumers with different decision-making styles may use narrative accounts differently
• Consumers have different levels of experience with healthcare quality data
Martino-16 Sep-12
Overview of Research Questions
• What cues do people use to judge the value and authenticity of patient narratives?
• How salient (memorable) are patient narratives relative to standardized measures of healthcare quality?
• How trustworthy and useful do consumers perceive narratives to be? What are bases for skepticism?
• Do people’s preferred decision-making styles, health status, or past exposure to healthcare quality data affect their attention to and appraisal of patient narratives?
Martino-17 Sep-12
• A fictitious public reporting website custom built for this and related studies
• Designed to be consistent with current public reports in terms of content, format, and functionality
• Presents three types of data– Standardized patient experience measures (CAHPS)– Clinical process and outcome measures (HEDIS)– Patient narratives (reviews)
• Hidden tracking system to monitor:– Click patterns– Time spent on each page
The SelectMD Website: Basic Design
17
Martino-18 Sep-12
• Modeled on actual narratives collected from RateMDs
• Matched to actual narratives based on length, cognitive complexity, and clarity of writing
• Pilot testing to compare actual (RateMD) narratives to a large set of fictitious ones constructed for this study
• Matched to reported CAHPS scores:– Focused on areas of experience covered by CAHPS– Higher scores associated with more positive
comments (but imperfectly)– Comments varied in emotional tone for all doctors
The SelectMD Website: Creating Realistic Patient Narratives
18
Martino-19 Sep-12
Martino-20 Sep-12
Martino-21 Sep-12
Martino-22 Sep-12
Martino-23 Sep-12
23
Martino-24 Sep-12
Pretesting of Patient Narratives
• Three rounds of testing, n = 8-9 per round
• Convenience sample of internet users with recent healthcare experience, mix of demographic characteristics
Online Experiment
• Random sample drawn from Knowledge Networks online research panel– Representative national sample (>60,000 households)– Limited to households with computer-based internet
access (89% of total panel), aged 25-64 years
• Of 1,757 invitations, over 48% accepted (n = 849)
Study Participants
Martino-25 Sep-12
Experimental Design:Six Study Arms
25
ARM MEASURES DOCTORS
1 CAHPS 12
2 CAHPS + HEDIS 12
3 CAHPS + COMMENTS 12
4 CAHPS + HEDIS + COMMENTS 12
5 CAHPS + COMMENTS 24
6 CAHPS + HEDIS + COMMENTS 24
Martino-26 Sep-12
Experimental Design:Six Study Arms
26
ARM MEASURES DOCTORS
1 CAHPS 12
2 CAHPS + HEDIS 12
3 CAHPS + COMMENTS 12
4 CAHPS + HEDIS + COMMENTS 12
5 CAHPS + COMMENTS 24
6 CAHPS + HEDIS + COMMENTS 24
Martino-27 Sep-12
• Questions asked of pretest participants– Emotional valence of narratives– Perceived informativeness & authenticity of narratives
• Questions asked prior to MD Choice– Prior exposure to public reports on plans, hospitals,
and doctors– Health status
• Questions asked after MD Choice– Recall of measures– Perceived usefulness and trustworthiness of measures– Decision making styles
Study Measures
27
Martino-28 Sep-12
Qualities of Narratives Associated with Their Perceived Usefulness and Authenticity
• Strongly worded narratives were perceived as less authentic (p = .001) but more informative (p < .001) than mildly worded narratives
• Narratives about negative experiences were perceived as less authentic (p < .001) than narratives about positive experiences
• Length of narratives was positively related to perceived authenticity, r (129) = 0.25, p = .005, and positively related to perceived informativeness, r (129) = 0.30, p = .001
Martino-29 Sep-12
Recall of Patient Narratives
• Of Ss who were shown patient narratives on the SelectMD site, 86% recalled seeing them
• Of Ss shown narratives plus CAHPS measures– 58% recalled both– 32% recalled narratives only– 4% recalled CAHPS measures only– 7% recalled neither
• Of Ss shown narratives plus HEDIS measures– 29% recalled both– 53% recalled narratives only– 3% recalled HEDIS measures only– 15% recalled neither
Martino-30 Sep-12
Recall of Patient Narratives (cont.)
• Ss low in avoidant decision-making were more likely to recall patient narratives than Ss high in avoidant decision-making (89% vs. 84%; p = .07)
• Ss high in intuitive decision-making were more likely to recall patient narratives than Ss low in intuitive decision-making (88% vs. 83%; p = .10)
• Ss high in intuitive decision-making and low on rational decision-making were especially likely (94%) to recall patient narratives
Martino-31 Sep-12
Recall of Patient Narratives (cont.)
• 81% of Ss who recalled the narrative data provided open-ended data on what they remembered specifically
Content recalled % of subjects who recalled this
content
% of narratives that contained this
content
Favorable comment 27 48
Unfavorable comment 28 52
Doctor communication/time spent 28 47
Access to care 14 33
Office staff 16 36
Doctor showed concern 10 36
Martino-32 Sep-12
Perceived Usefulness of Patient Narratives
• Among Ss who recalled seeing the narratives, 44% said that they were very useful for decision-making, 44% said somewhat useful, 9% said only a little useful, and 3% said not at all useful
• Unexpectedly, Ss high in intuitive decision-making judged the narratives as less useful than did Ss low in intuitive decision-making (p = .06)
• Ss with recent serious or chronic health conditions saw the narratives as less useful than Ss without those conditions (p = .01)
• Past exposure to healthcare quality data was not related to perceived usefulness of the narratives (p = .34)
Martino-33 Sep-12
Trustworthiness of Patient Narratives
• Ss perceptions of the usefulness and trustworthiness of narratives were strongly related, r = 0.59, N = 588, p <.001
• Among Ss who recalled seeing the narratives, 18% said that they were very trustworthy, 65% said somewhat trustworthy, 15% said only a little trustworthy, and 2% said not at all trustworthy
• Decision-making style and prior exposure to healthcare quality data were unrelated to how trustworthy narratives were perceived to be
• Ss with recent serious or chronic health conditions trusted narratives less than Ss without those conditions (p = .04)
Martino-34 Sep-12
Reasons Given for Mistrust of Patient Narratives
Reasons* Percentage of responses
Prefer to base my opinions on firsthand experience or information from people I know
35%
Everyone is looking for something different in a doctor; comments are not relevant to me or my situation
11%
Comments are subjective/biased 20%
Contextual information is missing 10%
Comments are too varied or contradict one another 7%
Too few comments to draw a reliable inference 5%
Comments could be faked or edited 12%
Distrust any information on the web 8%
* Given by 101 subjects who judged narratives to be “only a little” or “not at all” trustworthy
Martino-35 Sep-12
Attention Given to Patient Narratives
• Overall, Ss (N = 593) read reviews of 17.1% (SD = 24.4) doctors (includes those who did not drill down to reviews)
• Ss who drilled down to the reviews (N = 157) read the reviews of 30.2% (SD = 25.6) of doctors
• Ss high in avoidant decision-making read reviews of a smaller percentage (11.2%) of doctors than Ss low in avoidant decision-making (14.9%; p = 0.10)
Martino-36 Sep-12
Attention Given to Patient Narratives (cont.)
• Ss low in analytic decision-making and low in intuitive decision-making read reviews of a smaller percentage (11.7%) of doctors than Ss with other combinations of these two styles (who read the reviews of 17-20% of doctors; F = 3.2, p = .02)
• Ss with prior exposure to healthcare quality information read reviews of a larger percentage of (20.3%) of doctors than Ss without prior exposure (15.1%; p = .01)
• Ss with recent serious or chronic health conditions read reviews of a larger percentage of doctors (19%) than did Ss without such conditions (16%; p =.09)
Martino-37 Sep-12
Conclusions• People use cues in narratives to determine their value and relevance
• When presented together, patient narratives appear to be more memorable (salient) than standardized data on quality
• Most people trust patient narratives and find them useful for decision-making
• A minority recognizes the limitation of patient narratives as a source of data on data on doctor quality
• Positive and negative accounts of patient experience are recalled equally; no domain of patient experience is more salient than others
• Attention to and evaluation of patient narratives depends on people’s decision-making tendencies, health status, and prior experience with healthcare quality data
Martino-38 Sep-12
Implications
• It may be possible to increase healthcare consumers’ engagement in and use of reports of standardized quality data by incorporating patient narratives
• Need to elicit patient narratives in a way that makes them most useful to consumers (requires attention to issues of reliability and validity)
• Need to present narratives in a way that clarifies their value (as illustrative rather than representative), relevance (by building trust in the methods of elicitation and reporting) and limitations