SAS presentation at the Chief Analytics Officer Forum, Sydney
State Farm presentation at the Chief Analytics Officer, Fall 2016
-
Upload
corinium-global -
Category
Data & Analytics
-
view
1.085 -
download
2
Transcript of State Farm presentation at the Chief Analytics Officer, Fall 2016
Contains CONFIDENTIAL information which may not be disclosed without express written authorization
Do You Know or Do You Think You Know?
Incorporating Business Experiments Into
Strategy Development
Andy Pulkstenis
Director, Advanced Analytics
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 1
• Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance
• Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance
What are we doing today?
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 2
What is a “designed experiment?”
Designed testing is a way of treating the business environment like a controlled experimental setting, randomly assigning strategies to customers as in a clinical drug study or lab
Benefits
• Clear, accurate insights about causation • Ultra-clean data for model development = more accurate
prediction • Enables strategy optimization at the customer level
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 3
Innovators, on Testing
“By thorough experimentation and trial-and-error in the market, [successful companies] iterate towards a strategy and a business model that really is viable.”
- Harvard Business School Professor Clayton Christensen, a recognized authority on “disruptive innovation”
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 4
- Rich Fairbank, Chairman & CEO Capital One
“What we’ve done is to create an innovation machine. This is not just a credit card strategy, testing is a revolution that can be applied to many businesses.”
Innovators, on Testing
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 5
- Jeff Bezos, Amazon Founder
“If you double the number of experiments you do per year, you’re going to double your inventiveness.”
Innovators, on Testing
Fun fact: Amazon once fired a team for implementing strategic change without proper statistical testing first!
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 6
How do businesses use testing?
Strategy Optimization
Targeted Messaging
Pricing
Real-time Website Design/Optimization
Customer-level Decisioning
Retention & Save Strategies
Product & Strategy Development
Operations Improvement
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 7
Specific business examples
Sources: “How to Design Smart Business Experiments” by Tom Davenport, HBR 2/2009 “The Discipline of Business Experimentation” by Stefan Thomke and Jim Manzi, HBR 12/2014
Do visible lobster tanks increase lobster sales?
Are higher bids driven by CC payment option?
Will opening stores an hour later lead to a drop in sales?
Nearly all aspects of business.
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 8
Typical Testing Evolution
None A/B “one factor at a time”
(OFAT) Multivariate
(MVT) MVT w/
covariate analysis
Pick “one winner for all” Increased business insight while progressing left to right
Enables customer or segment-level strategy/message
optimization
Companies typically flow through an evolution when embarking on testing agendas
Continuous testing
A pervading culture of evolving, continuous, and cyclical testing provides
strategic foundation
Many stop at A/B or OFAT through lack of knowledge or lack of skillset, missing out on benefits that increase with each successive milestone.
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 9
• Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance
What are we doing today?
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 10
Which banner led to the most visits to the
“Agent Search” page?
Version B:
Version A: (Click leads to “Drive
Safe & Save” page)
(Click leads to “Agent
Search” page)
Contains CONFIDENTIAL information which may not be disclosed without express written authorization
Answer: Version B led more people to the Agent Search page
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 12
Which statefarm.com banner led to the most visits to
the “Drive Safe & Save” page?
Version B:
Version A: (Click leads to “Drive
Safe & Save” page)
(Click leads to “Agent
Search” page)
Contains CONFIDENTIAL information which may not be disclosed without express written authorization
“Find an Agent” drove more people to the “Drive Safe &
Save” page than the “Drive Safe & Save” experience did!
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 14
Version A: Version B:
Which mailer generated the highest lift in
Life insurance sales?
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 15
Claims Contact Center
Claims Contact Center – “Out of Scope” Calls
Can proactive sharing of contact info reduce misdirected or “out of scope” Claims
calls?
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 16
Multivariate Discount Double Check Test
• Explored 7 different page features as tested ‘factors’ (text, graphics, layout,
images, colors, etc.)
• Results in 384 distinct web page variations (with a subset of 200+ valid
combinations)
• Only needed to physically test 58 combinations
• Improved quote starts by over 50% and quote completes by 10%
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 17
• Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance
What are we doing today?
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 18
My roadmap for launching testing
Be Persistent It will take legwork and determination to find the partner in step two
Be Ambitious Aim for moderate or greater business impact
Be Careful Carefully execute the approach with business partners
Be Ready Build the “success story” deck before you need it
Be A Teacher There is a significant initial education component
Be Opportunistic Follow the path of least resistance
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 19
Best Practices: Set Up for Success
Assemble the
right structure Identify key roles at the outset (business, stats/DOE, execution, structural or creative design)
Assemble the
right players
Don’t just fill seats with “good people.” If you need an SME in a certain area, then staff that role with an SME
Assemble the
right questions
A surprising amount of time in early DOE engagements is spent helping the business partner define the problem as a quantifiable hypothesis
Assemble the
right scope
Too ambitious and you fail spectacularly Not ambitious enough and you really don’t move the needle
Assemble the
right controls
Make sure to monitor execution and data collection from Day 1 and throughout the process
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 20
• Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance
What are we doing today?
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 21
Common Arguments
Regulation
Initial resistance to statistical testing typically falls into
three categories:
Complication
Organizational
Tradition
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 22
Regulation
Our industry is too heavily regulated for this
Didn’t stop Finance, Pharma, and others from
successful transition to DOE
Even if allowed, the regulators would never
understand it
It’s not “fair” to randomly assign strategies to
customers in a test
Successful testing industries proactively educate
regulators on key concepts
• Current strategy isn’t “fair” either
• Jonas Salk & polio
• “Beautiful experiment over which the
epidemiologist could become quite ecstatic but
would make the humanitarian shudder”
Perceived Barrier Response
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 23
Complication
Too complicated Nonsense – no more complicated than
predictive modeling, just new to your firm
Ignoring complex relationships doesn’t mean
they aren’t there
Start small, manageable
• Design according to cost constraints
• Much cheaper than hoping we guessed
correctly (think of it as “buying data”)
This can be handled by applying exclusions
to the design
Complexity of analysis
Too difficult to execute
Some combinations don’t make
business sense
Too expensive
Perceived Barrier Response
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 24
“We tried it once and we saw no
improvement” (i.e. learned nothing)
“We intuitively know our business”
Perceived Barrier
“Our external consulting firms tell us
all we need to know about course
correcting”
Organizational Tradition
• You don’t hit a home run every time
• You learned that those strategies were
equivalent
• Often gain fresh, real insights that defy
historical logic or perceived wisdom
• “The greatest obstacle to discovery is not
ignorance - it is the illusion of knowledge.”
- Daniel Boorstin
Response
Designed testing consistently outperforms
consultant-provided solutions
“I have not failed 10,000 times. I have succeeded in proving that those 10,000 ways will not work. When I have eliminated the ways that will not work, I will find the way that will work.” - Thomas Edison
Today WD-40 can be found in 4/5 US homes
WD-1 WD-39 “didn’t work!”
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 25
Questions/Comments
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 26
Appendix
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 27
Engagement Model
Business Partner
Statistical Design
Implementation/Execution
Structural and Creative Design Test
ing
Team
Ensure alignment with broader
strategic goals
Strategic Approval
“Testing Council” of SME’s ensures alignment with
statistical best practices and production constraints
Structural Approval
• Statistical DOE • Production Environment Document key hypotheses,
findings, participants, and actions taken
Test Results Repository
Execute/Monitor/Analyze/Take Action
Ensure alignment with broader strategic goals
Strategic Approval
Execute/Monitor/Analyze/Take Action “Testing Council” of SME’s
ensures alignment with statistical best practices and
production constraints
Structural Approval
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 28
What do I mean by “test-and-learn culture?”
• Designed experiments inform (and determine) strategy development
• Understanding of when to settle for simple “A/B” testing and when to use more sophisticated multifactor tests
• Internal standards/best practices for test design • Formal internal governance
A true testing culture demonstrates the following elements:
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 29
Four “Test Factors”:
• Engagement (3)
• Primary Emphasis (2)
• Tabs (2)
• George (2)
“Quote” “Yes”
“Phone”
A Multivariate Test Example
“Simple”
“Active”
“Yes”
“No”
“Passive”
“No”
This results in 24 combinations or “test cells”
Metric of interest: quote starts
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 30
So we tested a statistically chosen subset of 12 that allows us:
• To draw conclusions about all 24 combinations
• Make inferences about the impact of individual factors
and associated interactions
Physically testing all 24 combinations is undesirable:
• Implementation complexity
I applied a “reduced” design
Contains CONFIDENTIAL information which may not be disclosed without express written authorization 31
• We learned “Why?” not just “Which one?” • Discovered interactions that had been downplayed as unlikely • Better business result:
- 12% > than current page - 10% > than vendor-recommended combo
• Set the stage for a 7-factor 200+ combination comprehensive test
• “George”
Benefits of using multifactor testing