Conjoint Adaptive Ranking Database System ( CARDS )
description
Transcript of Conjoint Adaptive Ranking Database System ( CARDS )
![Page 1: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/1.jpg)
ConjointConjoint Adaptive Ranking Database SystemAdaptive Ranking Database System ( (CARDSCARDS))
Ely DahanEly Dahan
Michael Yee, John Hauser & Jim OrlinMichael Yee, John Hauser & Jim Orlin
EXPLOR Award Winning Presentation – September 22, 2004EXPLOR Award Winning Presentation – September 22, 2004
Good, Fast, Cheap Good, Fast, Cheap andand Easy? Easy?
![Page 2: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/2.jpg)
The Problem• Current methods require many
questions for few answers
• Respondents must rate products they don’t like
• Simplifying rules to narrowchoices not typically captured
• Respondents make mistakesdue to fatigue, causing inconsistency
• Is there a better way?
![Page 3: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/3.jpg)
Smart Phone ExampleSmart Phone Example
![Page 4: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/4.jpg)
SmallSmallServiceService
Phone BrandPhone Brand
Example: Smart Phone• Respondent: Alex Bell• How does Alex choose
a smart phone?
Mini KeyboardMini Keyboard
FlipFlip 1010
55
7766
99 Utility Scores:Utility Scores:Alex makes tradeoffsAlex makes tradeoffs
![Page 5: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/5.jpg)
SmallSmallServiceService
Phone BrandPhone Brand
• Respondent: Alex Bell• How does Alex choose
a smart phone?
Mini KeyboardMini Keyboard
FlipFlip 1616
11
4422
88 Process of Elimination:Process of Elimination:Focus on key featuresFocus on key features
Example: Smart Phone
![Page 6: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/6.jpg)
Consider this tough task:Rank 32 Smart Phones based on your preferences
There are a billion, billion, billion, billionbillion, ways for a respondent to rank 32 smartphones!
Are weAre wesurprised thatsurprised that
respondents becomerespondents becomefatigued and makefatigued and make
mistakes!?mistakes!?
![Page 7: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/7.jpg)
Prior Research on Adaptive Questioning• Johnson (1987, 1991) & Orme and King (‘02) Sawtooth ACA• Huber and Zwerina (1996), Aggregate utility balance• Arora and Huber (2001), Aggregate customization• Sandor and Wedel (2001), Aggregate + prior beliefs• Louviere, Hensher, and Swait (2000), Aggregate CBC
Prior Research on Fast & Frugal Rules• Tversky (1969, 1972), lexicographic semi-order, elimination by aspects• Dawes and Corrigan (1974), unit weights, linear models• Montgomery and Svenson (1976), 2-stage processing• Thorngate (1980), efficient decision heuristics• Shugan (1980), cost of thinking (pair wise comparisons)• Johnson, Meyer, et. al. (1984, 1989), protocol anal., choice models can fail• Roberts and Lattin (1991, 1997), two-stage w/greedy• Gigerenzer and Goldstein (1996), Take the Best & others• Bettman, Luce, Payne (1996, 1998), Accuracy vs. effort, lexicography• Martignon and Hoffrage (2002), fast and frugal is robust
![Page 8: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/8.jpg)
Two new ideas:
• IDEA 1: We can now measure Alex’s process of elimination
• IDEA 2: We can help Alex avoid inconsistent answers
![Page 9: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/9.jpg)
Customer Insight:Respondents may be using a simple
process of eliminationprocess of eliminationto narrow choices for consideration
IDEA 1:
Phone BrandPhone BrandMini KeyboardMini Keyboard
FlipFlip
“I will only considerflip phones, with mini-keyboards, from Blackberry”
![Page 10: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/10.jpg)
Customer Insight:Respondents may be using a simple
process of eliminationprocess of elimination
IDEA 1:
How hard is it to identify each respondent’s simplifying rule?
AGB
![Page 11: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/11.jpg)
How can we identify each respondent’s
process of eliminationprocess of elimination
IDEA 1:
?
•Tougher than it seems, because they may be using one of a huge number of possible rules
•We solved this problem with a new computer technique (speedy)
•We tested our theory and it works!
![Page 12: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/12.jpg)
The big benefit of identifying respondents’
process of eliminationprocess of elimination?
•GoodGood Accuracy, customer insight
•FastFast 1 minute for them, quick for us
•CheapCheap Pack more into the same study
•EasyEasy Reduce drudgery
![Page 13: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/13.jpg)
RankSome
RankAll
2 minutes 7 minutes
2 7
Process of elimination Benefit: CheapCheap
Could you use 5 extra minutes of survey time?Could you use 5 extra minutes of survey time?
FastFast
![Page 14: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/14.jpg)
Ranksome
Rankall
kind of funokay
Somewhatinteresting
about rightlong
Benefit: EasyEasy
![Page 15: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/15.jpg)
How do we know if theprocess of elimination
idea is
GoodGood?
![Page 16: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/16.jpg)
Holdout sample for rankings
Pretty GOODGOOD
![Page 17: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/17.jpg)
Holdout sample for first choice
Pretty GOODGOOD
![Page 18: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/18.jpg)
The consistency criterion,a new approach
Reduce response error by “guiding” respondents towards consistent answers
Each choice must be 100% consistent with at least one set of utility scores
IDEA 2: Avoiding inconsistent answers
![Page 19: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/19.jpg)
• Show product features
• Click on favorite cards
• Inconsistentcards just“disappear”
• GetUtilityscores
• Save lotsof clicks
Keeping people consistent:Conjoint Adaptive Ranking Database System (CARDSCARDS)
IDEA 2: Consistency
![Page 20: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/20.jpg)
How do we keep people consistent?
IDEA 2: Consistency
SmallSmall
ServiceServicePhone BrandPhone BrandMini KeyboardMini Keyboard
FlipFlip
1010 5577 6699
Imagine we knew Alex Bell’s utility scores…
We would know how he would rank all 32 phones
![Page 21: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/21.jpg)
How do we keep people consistent?
IDEA 2: Consistency
Imagine we knew every possible set of scores…
Each set of utility scores are consistent witha unique ranking of all 32 phones
Surprise: Consistent rankings are atiny percentagetiny percentage of the possible answers
Eliminate rankings not on the consistent list
![Page 22: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/22.jpg)
The big benefit of keeping respondents
consistentconsistent?
•GoodGood Accuracy, consistent
•FastFast minutes for them, quick for us
•CheapCheap Pack more into the same study
•EasyEasy 50% to 75% effort reduction
![Page 23: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/23.jpg)
Consistency reduced effort 73%!
Without consistency With consistency
1717cards
7cards
Consistency Benefit: EasyEasy
![Page 24: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/24.jpg)
•Scalable
•Utility Scores as you go
•Emphasizes likes
•Measures uncertainty
Extra benefits of consistency
![Page 25: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/25.jpg)
How do we know if consistency is
GoodGood?
![Page 26: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/26.jpg)
Holdout sample for rankings
Pretty GOODGOOD
![Page 27: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/27.jpg)
Holdout sample for first choice
Just OKAYOKAY
![Page 28: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/28.jpg)
Key Takeaways:
•GoodGood Predictive; Customer insight
•FastFast For them and for us
•CheapCheap Pack more into one study
•EasyEasy Reduce drudgery & mistakes
![Page 29: Conjoint Adaptive Ranking Database System ( CARDS )](https://reader036.fdocuments.us/reader036/viewer/2022081515/568146f2550346895db4261d/html5/thumbnails/29.jpg)
Thank you for this exciting award!
Good, Fast, Cheap, Easy Good, Fast, Cheap, Easy demonstrations:
http://orc-pumba.mit.edu/~myee/CARDS/conjoint.php http://orc-pumba.mit.edu/~myee/CARDS/conjoint.php Use email: nodebug7@nodebug7@
http://wow.mit.eduhttp://wow.mit.edu