Widemile and Microsoft Multivariate Testing Case Study
-
Upload
widemile -
Category
Technology
-
view
5.013 -
download
3
description
Transcript of Widemile and Microsoft Multivariate Testing Case Study
Microsoft Case Study:
40% Conversion Rate Increase of SEM Traffic using
Multivariate Testing
Michael Schultz Director Product Management & Marketing, Microsoft Office Unmanaged Services
Frans KeylardDirector of Optimization, Widemile
Agenda
• A/B & Multivariate Testing Overview
• Microsoft Office Live Small Business Case Study
– Business Overview
– Case Study Details
• Top 10 Page Optimization Tips
Cus
tom
er
Sal
es F
unne
l
• Email marketing• Loyalty Programs• Newsletters• Personalization• Coupons /Surveys
Widemile Focus
• Key web pages• Sales funnels• Landing pages• Registration pages
• Search marketing• Display advertising• Affiliate marketing• Contextual ads• Ad targeting
Conversion Optimization
Conversion
Acquisition
Retention
*Source: eMarketer – April 2009
Investment recommendation:5% to 15% of acquisition budget
Testing and segmentation can improve retention performance
Automated, data-driven action that increases revenue by optimizing a user’s online experience.
$37 Billion annual US spend by 2013. $23 Billion spent in 2008.*
Testing Options – Approach Overview
Basic Analytics
Ideal use: > Validates traffic and conversion rates
Limitation: > Simple measurement; no optimization
A/B Split Testing
Ideal use: > Best for testing two or more radically different pages <i.e. template testing>
Limitation: > No insight into why one page is better, just which won among traffic tested
Multivariate Testing (MVT)
Ideal use: > Test different content, simultaneously determining most effective combination> Segmentation can be applied for more granular optimization in follow-on tests
Limitation: > Requires sufficient traffic & conversions
MVT Testing Options - Full & Fractional Factorial
Fractional FactorialFull Factorial
5
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170
Full Factorial
Fractional Factorial
Number of Days to Complete Test
Expertise
Requirements for Success
6
User Interface Proven Architecture Reporting Interface
Analyst Creative Web Developer
Technology
2009 – Year for Conversion Optimization
Microsoft Office Live Small Business
Audience:
• Small businesses, 1-10 employees(real estate agent, local bakery, video store, restaurant)
• Non-profits/Organizations(community groups, soccer clubs, PTSAs)
Benefits:
• Free website and email
• Free services to view, store and share documents online
Goals Summary
Goals:
• Improve sign up conversions
• Better understand audience
• Identify highest impact page factors
Testing Strategy:
• Leverage proven landing page template
• Execute multivariate test
• Fractional factorial approach to minimize time
Microsoft Case Study
Getting started…1. Define page to be tested
2. Analyze outside factors
1. In-bound traffic
2. Traffic volume
3. UX - Funnel
3. Review page & conversion goals
Original web page
Testing Considerations
What is the best hero shot?
Does the headline grab the customer’s attention?
Do these resource links belong here?
Does the button maximize conversions?
Does this copy help or hurt?
Indentifying Factors to Test
1
2
3
4
5
Defining a Test Matrix
No. Factor (Variable) Levels (versions)
1 Hero Shot 4
2 Title 4
3 Description 4
4 Call-To-Action Button 4
5 Resource Links 2
Audience Participation – Best CTA Button
Level 2
Level 3
Level 4
Level 1
Efficiency Through Technology
1 of 16 2 of 16
3 of 16 4 of 16
0 100 200 300 400 500
Full Factorial
Fractional Factorial
Comparison - # of Experiments Required for Test by Approach
512
16
And the Winner is…
40%Improvement
Which Factors Made the Biggest Difference
High Influence
Medium Influence
Low Influence
Hero Shot
Title
Description
Call-To-Action
Resource Link Location
High Influence Factor - Key Learning
Factor/level insights
• “Get Started Free” won
• The free aspect of the service is very important to get people to try it out
• More task focused terminology like “Set up my business!” or “Build My Online Business” lost
• “Sign Up Free!” which is the typical text found on so many CTA buttons also lost out.
• Future testing around the positioning of the FREE service
Case Study Summary
Microsoft Office Live Small Business’ testing and optimizing investment:
1. Delivered a 40% lift in conversion
2. Reduced customer acquisition cost
3. Maximized SEM performance
Testing and optimizing content is a fast and effective way to maximize the performance of your SEM spend.
10 Optimization Tips
1. Offer Clarityo Make sure the offer is clear, relevant, and compelling
o Keep your pages simple, clear, and appealing
2. Path to Conversiono Make sure the path to conversion is simple and straightforward
o Keep conversion paths efficient, focused, uncluttered
3. Performanceo Prospects will not tolerate slow pages or links
o Google penalizes slow page loads
4. Test Design and User Experienceo Avoid confusing landing page designs and difficult to read text
o Make transitions seamless and keep the consistency high (both messaging and look & feel)
5. Seasonalityo Seasonally optimize your pages and PPC campaign
o Leverage historical SEM in planning current and future campaigns
20
10 Optimization Tips
6. Target Audienceo Optimize your content based on audience insight
o Capture contact information early to better inform your marketing
7. Segment Audienceo Optimize for distinct audiences to convert
o Segment your traffic and test to see if unique landing pages would perform better
8. Offers and Goalso Create and test offers that have a high perceived value but a low cost to fulfill
o Force your page to have only one clear goal and link ads to landing pages
9. Consistencyo Build consistency and relevancy between PPC campaigns and where you send your traffic
o Reinforce the call to action/offer on order and lead forms
10. Competitiono Look at what your competition is doing, but also what they are not doing
o Differentiation can often help, so test new approaches, offers, and incentives
4 Additional Tips for Marketers
1. Drink the Kool-Aid and start now!
2. Think of multivariate testing as a “fixed” marketing expense and allocate budget appropriately
3. Ensure collaboration between Creative, Optimization Testing and SEM
4. Make sure Optimization vendor learns the business and understands the audience
22
Questions?
PresentersFrans Keylard – [email protected]
Michael Schultz – [email protected]
Content Informationwww.widemile.com
www.officelive.com