Post on 02-Nov-2014
description
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Make or Break Customer Satisfaction
Improving Customer Satisfaction Measurement With New Methods
Keith Chrzan
Chief Research Officer, Maritz Research
1
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Current practice for customer satisfaction modeling
• “Driver analysis” using linear, compensatory customer satisfaction models (regression, correlation, PLS, SEM)– Each attribute has an importance weight
– Sum of attribute importances times their respective performance scores reflects overall satisfaction
– When done well, we account for the multicollinearity that’s pervasive in customer satisfaction research (e.g. Theil’sinformation-theoretic averaging-over-orderings regression model)
• Entered the marketing and economics fields in the 1950s and 1960s and never left
• Fits well with statistical models
• But how realistic is this model?
2
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
A visit to <NAME REMOVED> hotel
• Accurate reservation
• Quick check in process
• Nice room, upgraded bath
• Larger TV than most movie theater screens
• Easy internet access
• Excellent workout room
• Tasty room service
• Comfortable bed, neatly folded
3
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
A visit to <NAME REMOVED> hotel
• Accurate reservation
• Quick check in process
• Nice room, upgraded bath
• Larger TV than most movie theater screens
• Easy internet access
• Excellent workout room
• Tasty room service
• Comfortable bed, neatly folded with dead cockroaches between the sheets
4
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 20105
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 20106
Non-compensatory effects
• Sometimes performance on an attribute is so bad that, all by itself, it causes dissatisfaction– It doesn’t matter how well the brand performs on other attributes,
poor performance on just one ruins the entire experience
– This is a “non-compensatory” effect because adding the good effects still can’t overcome the effect of the poor performance on overall satisfaction
• Of course the opposite is also possible – great performance on one attribute outweighs shortcomings on others
• Standard regression models miss these kinds of effects
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 20107
A non-compensatory model
• Joffre Swait (1997) developed a questionnaire method that allowed analysts to incorporate non-compensatory effects to conjoint modeling– His method created better-fitting models, models that explain
brand choice better
– His models generated additional insights not otherwise available• Which are “must have” attribute levels and for which and how many
people?
• Which are dealbreakers for which and how many people?
– Within three years, however, the advent of HB analysis made thismodel obsolete, for conjoint studies
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 20108
A non-compensatory customer satisfaction model
• Adapting Swait’s approach to customer satisfaction modeling creates a non-compensatory model featuring rewards and penalties
• The result is Make or Break customer satisfaction
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Test case
• Online survey, May 2010
• 599 respondents rating a recent trip to a mobile phone retail store
• Overall satisfaction
• Nine attributes identified as drivers of retailer satisfaction in qualitative research
9
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201010
Sample design
• Two cells – Control cell (n=305): Standard customer sat survey
– Test cell (n=294): Penalty/Reward approach
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201011
Questionnaire Outline
• First respondents rate their overall satisfaction
• We ask of satisfied respondents – If any attributes were so wonderful that, all by themselves, they
made the experience great
– We use a checklist, to make this easy for respondents
• We ask dissatisfied respondents to check any attributes that were so terrible as to ruin, by themselves, the overall experience
• We ask respondents to rate only the attributes not checked above
• Thus there are no additional keystrokes required from respondents
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201012
Questionnaire
Q1. Please indicate how satisfied you were with your mobile phone shopping experience at <INSERT MOBILE PHONE RETAIL STORE> using the scale below. [ ] Completely satisfied [ ] Somewhat satisfied [ ] Neither satisfied nor dissatisfied [ ] Somewhat dissatisfied [ ] Completely dissatisfied
Q2c. ASK IF Q1< 3. Was the performance on any of these aspects so good as, all by itself, to make your overall mobile phone shopping experience satisfactory?
Q2d. ASK IF Q1> 3. Was the performance on any of these aspects so bad as, all by itself, to make your overall mobile phone shopping experience satisfactory?
Aspect Yes (for the good) Yes (for the bad) Store location [ ] [ ]Speed of service [ ] [ ]Friendliness of sales representative [ ] [ ]Professionalism sales representative [ ] [ ]Information the sales representative had for me [ ] [ ]Phone prices [ ] [ ]Network coverage [ ] [ ]Availability of the phone I wanted [ ] [ ]Price of the calling plans [ ] [ ]None of these CANNOT BE CHOSEN WITH ANY OTHER [ ] [ ]
Q2e.
Strongly Agree Agree
Neither agree nor disagree Disagree
Strongly Disagree
The store was conveniently located [ ] [ ] [ ] [ ] [ ]
I was waited on quickly [ ] [ ] [ ] [ ] [ ]
The sales representative was friendly [ ] [ ] [ ] [ ] [ ]
The sales representative was professional [ ] [ ] [ ] [ ] [ ]
The sales representative had the information I needed [ ] [ ] [ ] [ ] [ ]
The phones were reasonably priced [ ] [ ] [ ] [ ] [ ]
The network has adequate coverage [ ] [ ] [ ] [ ] [ ]
The phone I wanted was available [ ] [ ] [ ] [ ] [ ]
The calling plans were reasonably priced [ ] [ ] [ ] [ ] [ ]
Please indicate how much you agree or disagree with each of the following statements about your mobile phone shopping experience at <MOBILE PHONE RETAIL STORE>. SHOW ONLY ATTRIBUTES NOT CHECKED IN 2C OR 2D. RANDOMIZE ORDER.
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Results
• We get these results– Basic coefficients (weights) for each attribute
– An additional bonus weight for those people saying each attribute was wonderful and made their experience great
– An additional penalty, a negative weight, that detracts from theoverall rating for those people reporting attributes that ruined their experience
– Patterns of which attributes were particularly wonderful or terrible vary
• Not all respondents get the same attribute weights
• The model accommodates respondent heterogeneity
• This test case study uses regression analysis and shows the statistically significant attribute coefficients
13
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Standard regression model
14
Attribute CoefficientRep had info I needed -My phone was available -Price of phone .13Price of plan -Coverage .15Rep friendly .38Rep professionalism -Quick service -
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201015
Base of Make or Break model
15
Attribute CoefficientRep had info I needed .18My phone was available .07Price of phone .15Price of plan
Coverage
Rep friendly
Rep professionalism
Quick service
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201016
Adding in penalties
16
Attribute Coefficient
% reporting attribute
ruined the overall
experience PenaltyRep had info I needed .18 5 -.40My phone was available .07 5 -.68Price of phone .15Price of plan 4 -.48Coverage 1 -.64Rep friendly
Rep professionalism
Quick service 6 -.96
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201017
Topping it off with gains
17
Attribute Coefficient
% reporting attribute
ruined the overall
experience Penalty
% reporting attribute perfected the overall experience Reward
Rep had info I needed .18 5 -.40My phone was available .07 5 -.68 31 .26Price of phone .15Price of plan 4 -.48Coverage 1 -.64 30 .25Rep friendly 39 .22Rep professionalism 34 .17Quick service 6 -.96
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Correcting for multicollinearity
• Using Theil’s model (True Driver Analysis) shows the impact of all the penalties and gains, taking into account shared variance among the scale questions, the penalties and the rewards
18
Scale Questions 42%
Reward 32%
Penalty 26%
-10% -5% 0% 5% 10%
Represenative - hadinfo I needed
Representative -professional
Representative -friendly
Waited on quickly
Phone i wanted wasavailable
Phones werereasonably priced
Calling plans werereasonably priced
Network hasadequate coverage
Conveniently located
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Evaluation
• The non-compensatory Make or Break model yields additional insights
• The model incorporates respondent heterogeneity– Different respondents can have different patterns of penalties and
rewards
– In this case, 47 distinct patterns
• The model VASTLY improves prediction– Control cell with standard customer sat questions: R2 = 30%
– Above plus non-compensatory penalties/rewards for ruining/making my experience: R2 = 65%
19
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Maybe that was too easy?
• Experiences at mobile phone retailers vary quite a bit (long waits for service, phone availability, etc.)
• How does the model perform when most respondents are happy with their experiences?
• Will a shortage of penalties allow the model enough to work with?
• Let’s try retail banking
20
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Case studies 2-6
• Web surveys fielded October 2010
• Control groups doing standard ratings and test cells identifyingnon-compensatory penalty/rewards
• Five surveys of banking satisfaction– Branch satisfaction
– ATM satisfaction
– Call center satisfaction
– Customer service representative satisfaction
– Online banking satisfaction
• Overall satisfaction and 3-11 attributes, depending on study
21
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Sample sizes – case studies 2-6
22
Aspect Test SampleControl Sample
Branch 395 377
ATM 367 369
Call center 113 128
CSR 181 180
Online 363 363
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201023
Bank branch satisfaction model
23
AttributeStandard
regression Base
% reporting attribute
ruined the overall
experience Penalty
% reporting attribute perfected the overall experience Reward
Wait time in line .15 .10 2 -1.42
Staff courteous 41 .27Speed of completing request
.19 .17
Staff knowledgeable .28 .22
Staff provides accurate answers
.18 .17 1 -.40
R2 .62 .68 .70
• Significant effects for 4/10 attributes
• Model improves with addition of penalties and then again with addition of rewards
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201024
ATM satisfaction model
24
AttributeStandard
regression Base
% reporting attribute
ruined the overall
experience Penalty
% reporting attribute perfected the overall experience Reward
Safe and secure .14 .14 71 .11
Ease of transaction .37 .19 2 -.99 42 .36Wait time .13 .13 1 -1.61
R2 .51 .61 .64
• Significant effects for all three attributes
• Model improves with addition of penalties and rewards
• Service failures are uncommon and catastrophic – 1.61 points on a 5 point scale!
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201025
Automated call satisfaction model
25
AttributeStandard
regression Base
% reporting attribute
ruined the overall
experience Penalty
% reporting attribute perfected the overall experience Reward
Easy to get live rep 8 -.53
Easy to navigate .31 .13Useful response options
.53 .42 5 -.72
Reasonable hold time
4 -.85 37 .50
R2 .74 .81 .84
• Significant effects for all four attributes
• Model improves with addition of penalties and of rewards
• Service failures are more common
• Captures eight different varieties of customer experience
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201026
Phone rep satisfaction model
26
AttributeStandard
regression Base
% reporting attribute
ruined the overall
experience Penalty
% reporting attribute perfected the overall experience Reward
Authority to address your issue
.22 .18 1 -.49
Explains things clearly
2 -.80
Takes responsibility to resolve your issue
.28 .22 4 -.58
Handles call quickly 43 .31
Provides complete answers
.27 .19
R2 .71 .75 .77
• Significant effects for 5/11 attributes
• Three significant and injurious penalties
• Improved model fit
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201027
Online banking satisfaction model
27
AttributeStandard
regression Base
% reporting attribute
ruined the overall
experience Penalty
% reporting attribute perfected the overall experience Reward
Safe and secure 1 -1.45
Easy to navigate .21 .19Easy to complete tasks
.15 .14 50 .17
Able to conduct desired transactions
.19 .13 1 -1.22
Helps you manage your finances
.11 .11
R2 .57 .61 .62
• Significant effects for 5/10 attributes
• Rewards are less good than penalties are bad (but more common)
• Improved model fit
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Location of penalty/reward questions
• If before attribute ratings, as in Study 1, an adaptive survey flow can keep the amount of respondent effort (in terms of number of keystrokes) the same as standard customer satisfaction studies
• If after, we require additional work from respondents
• We split respondents, half with penalty/reward questions before and half with after
• Similar models, same R2 either way
• Adaptive set-up doesn’t seem to hurt the resulting data
28
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010
Summary
• In all six cases tested, the Make or Break model significantly outperforms the standard customer satisfaction ratings measurements– Higher R2 – explains more variance on overall satisfaction
– Additional insight about non-linear penalties and boosts for excellent/poor performance
– Additional insight about respondent heterogeneity
• These benefits remain even if we only use the penalties part of the model
29
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 201030