DV 2016: Building a Culture of Testing and Optimization
-
Upload
tealium -
Category
Technology
-
view
283 -
download
0
Transcript of DV 2016: Building a Culture of Testing and Optimization
![Page 1: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/1.jpg)
© 2016 Tealium Inc. All rights reserved. | 1
Building a Culture of Testing and Optimization
Joel Ring, HUB International
![Page 2: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/2.jpg)
2000
2006
2015
2009
Joel RingDirector of Analytics, HUB International
15+ Years in digital analytics
Created and managed the Web Analytics department at bestbuy.com
Sr. Digital Marketing Consultant
working with some of the
biggest brands on the web
Created the Digital
Strategy & Solutions team helping clients
with their analytics and
digital marketing
Manage the Analytics team
focusing on digital
analytics, data sciences, and
testing & optimization
![Page 3: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/3.jpg)
A lot has changed
2000
2006
2015
2009
![Page 4: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/4.jpg)
The great thing about the
internet is you can measure
anything!
We’re ‘gonna fail-fast and
break things!Like Facebook!
The only way to know what will work best is to
test it!
![Page 5: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/5.jpg)
With all the data generated through testing we will be able to understand our customers
like never before and learn
new ways to meet their
needs.
![Page 6: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/6.jpg)
Every minute…• Facebook users share nearly 2.5 million pieces of
content.• Twitter users tweet nearly 300,000 times.• Instagram users post nearly 220,000 new photos.• YouTube users upload 72 hours of new video content.• Email users send over 200 million messages.• Amazon generates over $80,000 in online sales.
![Page 7: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/7.jpg)
“Two-thirds of business are unable to extract value from their information.”
Source: 2015 Information Value Index from PwC and Iron Mountain
![Page 8: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/8.jpg)
If we have access to all
this data, how do we create
an environment
that enables us to derive value
from that data?
![Page 9: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/9.jpg)
Definition of an
effective test and
learn organization look like?
![Page 10: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/10.jpg)
Implemented roles and procedures to create,
execute, and evaluate testsAND
are actively communicating results to drive change.
“The key to great analysisis not math,
it’s communication.”
![Page 11: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/11.jpg)
Keys to creating a test and
learn organizatio
n
Sponsors Ideators
Executors Analyzers
The sponsor helps evangelize the testing organization and helps drive changes based on the results.
The ideators help develop testing ideas. These can be part of the testing team or stakeholders.
This is the group that executes the tests. This will generally encompass multiple departments and can include creative and technical resources.
The analyzers tabulate and communicate the results of the tests. They will be the team members who declare the test “winners”.
• Individuals can, and will, fall into multiple roles
• Include people from multiple departments
• Make sure to include stakeholders
• How many people in these roles?• Forrester estimates 5-15
![Page 12: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/12.jpg)
How do I get the initial buy in?
![Page 13: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/13.jpg)
Demonstrate value using existing
channels that already allow for testing
Email – Display Advertising – Paid Search
How do I get the initial buy in?
![Page 14: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/14.jpg)
Conduct a kick off meeting
![Page 15: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/15.jpg)
Conduct a kick off meeting
• Bring together people who will be either involved in or interested in the execution and effectiveness of your testing program
• Goal of meeting is to educate generate excitement through collaboration
• Kick off agendao Testing 101
• Types of test (A/B, multivariate, etc.)• What makes a good test• What is a testing plan
o Intro to our testing platformo Intro to our testing processo Ideation workshop
• Used the ideation ideas to start our testing prioritization
![Page 16: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/16.jpg)
Develop a testing plan
![Page 17: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/17.jpg)
Develop a testing plan
• Guides your testing prioritization, status and results
• What is in a testing plan?o List each test ideao Conversiono Status (Live, scheduled, complete, removed, on-hold)o Date launchedo Date completedo Results (winner)o Test type (A/B, A/B/C, multivariate)o Test location (site, email, SEM, display advertising)
• Keep in a central location accessible by the team
![Page 18: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/18.jpg)
Test NameTest Cell Test Description Conversion Test Type Test Location Status
Launch Date
End Date Winner
Link to test variations Results
Control: Current quote first position Control Group
Test: The IRS is expecting “many filings” to be in error at some level. No one can just blow off the law, but there will be something of a pass on reporting penalties as long as it can be demonstrated that effort went into compliance. Walter Fagan
Test Group
Control: What’s dominating the Employee Benefits agenda in 2016?Get the answers to the big questions →
Control Group
Test A: What’s dominating the Employee Benefits agenda in 2016?Get expert insights and tools you can use in the year ahead →
Test Group A
Control: current template designTest: Test: plain text format that will appear personalized including 1. Dear [name] 2. Sincerely [regional EB president]
Control: no nurture emailTest: CM nurture series to those who completed ACA nurture Ems
Control: always onTest A: second panel of ACA siteTest B: new contact form to be designed for campaignControl: no button/SM buttons Control
Test: remove SM buttons and add contact us button that goes to contact a rep form TestControl: https://www.hubinternational.com/employee-benefits/contact-us/
Test: modal pop-upControl: Current copy Control
Test:Headline: Optimize your Employee Benefits in 2016
Box 1: ACA Reporting & Verification AuditsBox 2: Cost Management Strategies
Test
Q1 Campaign Site
Live 9-Feb TBD
Complete 4-Feb 1-Apr Control ResultsEB Always On HP Interstitial Link clicks A/B Q1 ACA Campaign Site
ACA Panel Jack Quote 1 Share button clicks A/B
TBD TBD
CM nurture to ACA list Increase lead pipeline
A/B email
Live 9-Feb TBD Results
CM nurture email #2 template Email CTR A/B On-Hold
Results
Contact Us button location Button clicks A/B Q1 ACA Campaign Site
ACA Ad words Landing page test Form Completion Rate
A/B/C Ad words
Live 7-Mar
Complete 22-Feb 11-Apr Test
TBD
Results
Contact Rep Form Design - sub pages
Form Completion Rate
A/B Always On Site Live 9-Mar
L1 page CTA Link clicks A/B Q1 L1 page Complete 9-Mar 6-Apr Test Results
![Page 19: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/19.jpg)
Conduct regular testing/analytics meetings
![Page 20: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/20.jpg)
Conduct regular testing/analytics meetings
• Include core team and stakeholders
• Use the testing plan as a guideline for the agendao Review tests in progress
• Determine any status changeso Review tests scheduled to launcho Discuss any new testso Reprioritize based on new tests (if necessary)o Present deep dives from previous meeting questions
• Use the meeting to generate new test and analysis ideas
![Page 21: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/21.jpg)
Standardize the test process and embed it in business as usual
![Page 22: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/22.jpg)
DK Cool Crayon
Test approved Build testAsset(s)
needed?Add to test
plan N
Create asset(s)
Y
Review test Launch test
Wait 7 days
![Page 23: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/23.jpg)
DK Cool Crayon
Test approved Build testAsset(s)
needed? Review test
Create asset(s)
Launch testAdd to test plan
Y
N
Wait 7 days
Why wait 7 days?
• Early results show greater variance.
• Behavior often varies by day of week, want to ensure all days are represented.
![Page 24: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/24.jpg)
DK Cool Crayon
Test approved Build testAsset(s)
needed? Review test
Create asset(s)
Launch testAdd to test plan
Y
N
Wait 7 days
Declare winner?
Y
Review meeting
N
Determine next steps
Update test plan End
Suspend test
Declare Winner
Continue testing
Update test plan
Update test plan
End
Revise test?
N
Y
Review results
![Page 25: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/25.jpg)
Final thoughts
![Page 26: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/26.jpg)
Start Slow
![Page 27: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/27.jpg)
Start Slow
Use basic tests to establish your
process, then grow into complexity
![Page 28: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/28.jpg)
Don’t panic if your tests don’t have significant results
![Page 29: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/29.jpg)
You still learn from tests with no differences
Don’t panic if your tests don’t have significant results
![Page 30: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/30.jpg)
Implemented roles and procedures to create,
execute, and evaluate testsAND
are actively communicating results to drive change.
![Page 31: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/31.jpg)
Questions?
![Page 32: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/32.jpg)
THANKYOU
![Page 33: DV 2016: Building a Culture of Testing and Optimization](https://reader035.fdocuments.us/reader035/viewer/2022070515/5878f5681a28ab49608b54cb/html5/thumbnails/33.jpg)