Jeffrey henning main stage - 2011
-
Upload
ray-poynter -
Category
Marketing
-
view
35 -
download
2
Transcript of Jeffrey henning main stage - 2011
Main Stage, November 3, 2011
Crowd-Shaped Surveys: Adapting the Experience Based on Prior Respondents Jeffrey Henning – Affinnova
A Presenta*on from the Fes*val of NewMR Main Stage – November 3, 2011
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Crowd-Shaped Surveys: Adapting the Experience Based on
Prior Respondents Jeffrey Henning
Chief Marketing Officer
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
THE SPIRIT OF THE WEB
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 15 Apr 03
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 15 Apr 03
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 28 Apr 03
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 17 May 03
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 25 May 03
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 24 Jun 04
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 4 Dec 08
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Metal Umlaut – 12 Oct 11
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
NOW IMAGINE THIS AS AN ONLINE SURVEY
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Define “Metal Umlaut”
• The "heavy metal umlaut" is over the ö in the names of Motörhead, and the Blue Öyster Cult
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Define “Metal Umlaut”
• The "heavy metal umlaut" is over the ö and ü in the names of Motörhead, Mötley Crüe
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Define “Metal Umlaut”
• The heavy metal umlaut was ridiculed by the spoof band Spinal Tap by putting the umlaut over the letter 'n'
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Define “Metal Umlaut”
• The heavy metal umlaut can be seen in the term "nü-metal” to describe nu-metal with added umlaut-ness.
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Define “Metal Umlaut”
• The heavy metal umlaut is in the name of the band Assück.
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Define “Metal Umlaut”
• The heavy metal umlaut is in the name of the band Assück.
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Define “Metal Umlaut”
• The use of an umlaut over some of the letters in the names of heavy metal bands like Queensrÿche.
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
YOUR MISSION: GIVE ONLINE SURVEYS THE SPIRIT OF THE WEB
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
An Old Trick That Deserves a New Look
21 CONFIDENTIAL MATERIALS
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
• 99% of online surveys unfold the same way for a respondent whether they are the first or last respondent
• They provide little in the way of qualitative exercises and less in the way of co-creation
Where We Are Today
22
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Trends Rise of Online
Qual
Surveys with Huge Sample Sizes
Pressure for Shorter, More
Engaging Surveys
23
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Your Mission
How can we make our surveys adapt to early responses to better serve later respondents or to provide better data?
24 CONFIDENTIAL MATERIALS
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora>on
Idea Explora>on Concept Explora>on
Crowd-‐Sourced Ques>onnaires
25
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Question Buckets What Likes
• Prompt if fewer than 100 responses
What Dislikes
• Prompt if fewer than 100 responses
Reasons for Usage
• Prompt if fewer than 100 responses
Sugges>ons • Prompt if fewer than 100 responses
26 CONFIDENTIAL MATERIALS
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora>on
Idea Explora>on Concept Explora>on
Crowd-‐Sourced Ques>onnaires
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
28
In the past, why have you thought the Atkins Diet was not right for you? (Choose all that apply.)
q Love pasta too much to go without q Love bread too much to go without q Love sweets too much to go without q Try to avoid ea>ng red meat q Worried about health problems that Atkins might cause
q Other _________________________________
Develop Choice Lists for Closed-‐Ended Ques>ons
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Crowd-Sourced Choice Lists
• The “Other – please specify” entries are added to the list of options • Over time, only the 10 most frequently selected choices are
presented • To protect against choices entered early always being selected, the
10 choices can be selected probalistically • The “Other” text box uses autocomplete to suggest the unshown
choices • Analysis ramifications: track display
29 CONFIDENTIAL MATERIALS
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora>on
Idea Explora>on Concept Explora>on
Crowd-‐Sourced Ques>onnaires
30
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
BrainJuicer MindReader
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora9on
Idea Explora>on Concept Explora>on
Crowd-‐Sourced Ques>onnaires
34
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Comment Exploration: GMI Consensus Question Component
• Respondents read and evaluate other respondents’ comments
• Comments consistently scoring below a certain level can be screened out if required
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Respondents Select the Comments to Rate
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Management Portal for Comment Moderation
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora>on
Idea Explora9on Concept Explora>on
Crowd-‐Sourced Ques>onnaires
38
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Idea Exploration
• Idea voting communities do a poor job of quantifying the appeal of ideas
• Large idea spaces are problematic • Alternative is to show 7 ideas 10 times,
each time having respondent choose the one they’d must like to see done
39 CONFIDENTIAL MATERIALS
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora>on
Idea Explora>on Concept Explora9on
Crowd-‐Sourced Ques>onnaires
40
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Large Innovation Spaces
Number of Variants per AUribute
64
117,649
46,656
15,625 4,096 729
1 2 3 4 5 6 7
Content example: Concept with 6 attributes
Innovation space example:
Number of unique concepts increases exponentially.
Insight
Product Benefits
Clinical Endpoint
Positioning
Product Features
Product Image
1
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Optimization vs. Measurement
Simulated Best Concept #3 Simulated Best
Concept #2
Simulated Best Concept #1
42
• Traditional methods are measurement focused – Build a preference model to map a landscape – Search the map to find preferred concepts
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Optimization vs. Measurement (continued)
Optimized Top Concept #3
Optimized Top Concept #2
Optimized Top Concept #1
43
• Evolution is Optimization Focused – Directly searches the landscape – Does not need to estimate a model
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT) 44
Evolutionary Optimization Preferred Concepts New Concepts
Random Jumps
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Real-Time Optimization
45
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora>on
Idea Explora>on Concept Explora>on
Crowd-‐Sourced Ques9onnaires
46
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Crowd-Sourced Questionnaire
• Challenge online community to come up with questions to meet a specific research objective 1. Brainstorm on questions 2. Vote on questions 3. Have researcher vet, validate and create final
questionnaire - Source: Pat Molloy, ConfirmIt
47 CONFIDENTIAL MATERIALS
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Adaptive Questionnaires
• Imagine EFM questionnaires that improve their respondent engagement over time
• Length optimization – Shortening themselves if abandonment is high – Adding questions if abandonment is low
• Question optimization – Hiding open-ended questions once sufficient
verbatims collected – Hiding questions that cause abandonment
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
7 Examples of Crowd-Shaped Surveys
Ques>on Buckets
Crowd-‐Sourced Choice Lists Crowd-‐Sourced
Laddering Comment Explora>on
Idea Explora>on Concept Explora>on
Crowd-‐Sourced Ques>onnaires
49
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Your Mission • Currently most surveys don’t adapt or learn based on the
behavior of other respondents
• Crowd-shaped surveys solve interesting qualitative and quantitative issues that static surveys can’t
• While two of the examples premiered in 2001, crowd-shaped surveys are an important technique to explore further!
50
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Your Mission
Can you create online surveys that better embody the Spïrït of the Wëb?
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Thank you
Jeffrey Henning
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
53
Q & A
Jeffrey Henning Affinnova
Greg Coops Asian Strategies
Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)
Jeffrey Henning
Email Jeffrey @ [email protected] Read Jeffrey’s blog @ hUp://www.affinnova.com/blog/ Follow Jeffrey’s tweets at @ hUp://twiUer.com/#!/jhenning