MEASURE RIGHT THE FIRST TIME

Infusionsoft Partnercon 2012

vuurr.com

WHAT WE’LL COVER

• What you need• How to track• How to design experiments• What to experiment on• Putting it all together

TOOLS

TOOLS YOU MUST HAVE

• Google Analytics (or SiteCatalyst, or similar)• Optimizely (or similar, but you need killer

statistics)• Call tracking => analytics platform of choice• Link builder appropriate for your analytics

platform of choice• AdWords/AdCenter/Facebook properly

configured (if advertising)

DETERMINE YOUR MODEL

CPA ROI MARGIN

Minimize the cost of

each customer

acquisition

Maximize revenue for every dollar

spent

Minimize ad spend as a percent of revenue

DETERMINE YOUR MODEL

CPA ROI

Ad Spend ($)Conversion (qty)

Revenue - SpendSpend

all in dollars ($), answer is %

Revenue - SpendRevenue

all in dollars ($), answer is %answer is in dollars ($) per

conversion

MARGIN

DETERMINE YOUR MODEL - FINDING ROI & MARGIN

MARGIN

ROI

DETERMINE YOUR MODEL - FINDING CPA (MANUALLY)

Completed Conversions

Total Cost

Advertising => AdWords => Campaigns => Site Usage

Advertising => AdWords => Campaigns => Clicks

DETERMINE YOUR MODEL - FINDING CPA (AUTOMATICALLY)

• Use the Google Analytics API to query for total number of conversion by source & medium, and then properly divide them into the total marketing expense for each source & medium.

• Use a third party tool

TRACKING

INSTALLING ANALYTICS

• Follow your vendor’s instructions

• Don’t just dump the tracking code in and “go”

• Setup proper funnels, events, goals, and filters

• Know the difference between accounts, properties, and profiles

• Attach GWT and AdWords (if applicable)

• Track ALL OF THE THINGS

GOALS & ECOMMERCE

• Assign relevant goal values. A video view != purchase

• Track eCommerce as accurately as possible

• Explicit funnels are only useful if the supporting steps are relevant and provide information - use sparingly

TRACKING CAMPAIGNS

Campaign

Name/Content

Source Medium

• Every click is valuable

• Every traffic source matters

• Proper campaign tracking is crucial to adequately gauging traffic value

PROPER URL TAGGING

TRACKING RICH MEDIA: VIDEOS

• Instantiate video with YouTube/Vendor JavaScript API

• Track state changes, namely Plays and Completes

• On these changes, push GA events

TRACKING RICH MEDIA: SOCIAL BUTTONS

_gaq.push(['_trackSocial', network, socialAction, opt_target, opt_pagePath]);

Instantiate by subscribing to edge.create and message.send

Google Analytics handles for you

Instantiate by subscribing to the ‘tweet’ intent event

TRACKING RICH MEDIA: FORM ABANDONMENT

• Track full and empty exits as events

• Determine fields causing abandonment

TRACKING OFFLINE: PHONE CALLS

• Build GA gif request and HTTP GET on phone call

• Push calls as pageviews ( /call/+14805551234/abcdefg )

• Use Twilio & Twimlbin or another 3rd party service

• Consider filtering out those pageviews from main profile

• Setup goals in Google Analytics to fire on calls

• Give each ad, landing page, social source, different number

TRACKING PAID CLICKS

• Properly configure AdWords to push to Analytics

• Verify all cost data is applied to Analytics account

• Put MSN AdCenter {AdId} & {Keyword} in destination URL, then use AdCenter API to pull the cost and associate with relevant clicks from Analytics API

EXPERIMENT DESIGN

INTERNET MARKETING IS ABOUT EXPERIMENTATION

y = f( x1, x2, x3, . . . . xn )

y CTRCTR

CRCR

GoalsGoals

X1= X2= X3= Xn=

Page or Ad CopyPage or Ad Copy

Page LayoutPage Layout

Button Color & SizeButton Color & Size

Other FactorsOther Factors

DESIGNING EXPERIMENTS - ONE FACTOR AT A TIME

Let’s run an experiment where we test two different button colors.

Run Button Color ( x1 ) CTR ( y )

1 Green 5.2%

2 Orange 7.7%

Factors are tested in series, one at a time. This is extremely time intensive, requiring many tests for very gradual improvement.

DESIGNING EXPERIMENTS - FULL FACTORIAL DESIGN

Let’s run an experiment where we test two different button colors, button positions, and button labels simultaneously.

This tests multiple changes in parallel to find optimal combination of factors (x variables) to optimize results (y variable).

3 FACTORS WITH 2 VARIATIONS EACH

Number of Unique Tests = 23 = 8

DESIGNING EXPERIMENTS:FULL FACTORIAL DESIGN

Run Color (x1) Position (x2) Text (x3) CTR (y)

1 Green Left Try Now 5.2%

2 Orange Left Try Now 7.7%

3 Green Right Try Now 9.3%

4 Orange Right Try Now 12.7%

5 Green Left Buy Today 21.2%

6 Orange Left Buy Today 19.3%

7 Green Right Buy Today 14.9%

8 Orange Right Buy Today 17.7%

DESIGNING EXPERIMENTS: OFAT VS. FULL FACTORIAL

ONE FACTOR AT A TIME

IterativeSimple to Setup

Easy to Measure Results

SLOW & GRADUALIMPROVEMENT

VS.FULL FACTORIAL

More Complex to SetupRequires more visitor

data to test all combinations

QUICK & DRASTIC IMPROVEMENT

DESIGNING EXPERIMENTS:STATISTICAL SIGNIFICANCE

Preliminary results - is orange better?

Button Color ( x1 )

Visitors Clicks CTR ( y )

Green 38 2 5.2%

Orange 39 3 7.7%

Too small of a sample size leads to incorrect conclusions.

DESIGNING EXPERIMENTS: STATISTICAL SIGNIFICANCE

It turns out our original conclusion of CTR was incorrect.

Button Color ( x1 )

Visitors Clicks CTR ( y )

Green 8,238 486 5.9%

Orange 7,893 734 9.3%

Too large of a sample size wastes time, effort, & money.

SO WHAT QUANTITY OF VISITORS IS ENOUGH?

DETERMINING MINIMUM SAMPLE SIZE

x = Sample mean

μ = Population mean

Z = Number of standard deviations above mean

α = Confidence distance from 100%σ = Standard deviation

n = Minimum sample size

E = x - μ

E = Z α/2 σ

√n

DETERMINING MINIMUM SAMPLE SIZE

α/2 - α/2 Z

DETERMINING MINIMUM SAMPLE SIZE (CONTINUED)

E = Z α/2 σ√n

Standard Deviation

Minimum Sample Size

n = [Z α/2 σE

]

A/B TESTING SOFTWARE

TESTING

EVERY SINGLE CLICK HAS A COST ASSOCIATED WITH IT

KNOW YOUR LEVERS

• Product Pricing• Organic Rankings• Ad bids, placement,

content• Email subject lines &

content• Social content• Site Content

FUNNEL ANALYSIS & OPTIMIZATION

• Improve Quality Score• Split Test Ad Copy• Adjust Keyword Bids• Increase Organic Rankings• Increase Social Reach

• Split Test Landing Page Copy

• Improve Call to Action• Experiment with Deep

Linking• Look at Bounce Rates• Look at Cart Abandonment

IMPRESSIONSIMPRESSIONS

CONVERSIONCONVERSIONSS

(Sales/Leads)(Sales/Leads)

CLICKS(Site Visits)

Traffic Sources drive impressions & clicks

Optimizing those sources get more clicks

Optimized landing pages & site content increase conversionsCR

CTR

• SEM, SEO, Social, Email, & More

FUNNEL ANALYSIS & OPTIMIZATION

EMAILEMAIL

PPCPPC

ORGANIC ORGANIC SEARCHSEARCH

SOCIALSOCIAL

OFFLINE OFFLINE MARKETINGMARKETING

PHONE PHONE CALLSCALLS

AFFILIATESAFFILIATES

ALL OF THESE SOURCES HAVE ALL OF THESE SOURCES HAVE LEVERS THAT MANIPULATE THEIR LEVERS THAT MANIPULATE THEIR

QUALITY & LIKELIHOOD QUALITY & LIKELIHOOD TO BECOME TO BECOME

BUYERSBUYERS

EMAIL OPTIMIZATION LEVERS

EMAILS SENTEMAILS SENT

OPENED OPENED EMAILSEMAILS

LINKS LINKS CLICKEDCLICKED

LEVER 1: List Size

• Completely dependent on conversion rate of subscription form

LEVER 2: Open Rate

LEVER 3: CTR

Increase List Size

Test Subject Line

Test Email Copy

• Optimize CR on subscription page

• Increase sales to lure repeat buyers

EMAIL OPTIMIZATION LEVERS

EMAILS SENTEMAILS SENT

OPENED OPENED EMAILSEMAILS

LINKS LINKS CLICKEDCLICKED

LEVER 1: List Size

LEVER 2: Open Rate

Quantity Opened

LEVER 3: CTR

Quantity Sent

OPEN RATE =

Increase List Size

Test Subject Line

Test Email Copy

• Use an email application with A/B testing

• Call to action is important

EMAIL OPTIMIZATION LEVERS

EMAILS EMAILS SENTSENT

LINKS LINKS CLICKEDCLICKED

LEVER 1: List Size

LEVER 2: Open Rate

LEVER 3: CTR

Clicks from Email

Quantity Opened

CTR =

Increase List Size

Test Subject Line

Test Email Copy

• Treat as a landing page

• Test layout, buttons, content, etc.

EMAILS SENTEMAILS SENT

OPENED OPENED EMAILSEMAILS

PPC OPTIMIZATION LEVERS

AD IMPRESSIONSAD IMPRESSIONS

AD CLICKSAD CLICKS

LEVER 1: Budget

LEVER 2: Bids

LEVER 3: Quality Score

Increase Budget

Optimize for KPI

Maximize QS

LEVER 4: CTR Test Ad Copy

• Directly drive conversions factoring in conversion rate & CPC

• Optimize for maximum conversions at current CPC

PPC OPTIMIZATION LEVERS

AD IMPRESSIONSAD IMPRESSIONS

AD CLICKSAD CLICKS

LEVER 1: Budget

LEVER 2: Bids

LEVER 3: Quality Score

Increase Budget

Optimize for KPI

Maximize QS

LEVER 4: CTR Test Ad Copy

Ranking Score of Position Below

Quality Score of Your Ad

+Actual Cost $0.01= +

PPC OPTIMIZATION LEVERS

AD IMPRESSIONSAD IMPRESSIONS

AD CLICKSAD CLICKS

LEVER 1: Budget

LEVER 2: Bids

LEVER 3: Quality Score

Increase Budget

Optimize for KPI

Maximize QS

LEVER 4: CTR Test Ad Copy

• Function of ad content, landing page content, historical CTR, & more

• Relevant ad• Relevant landing page• Display URL

PPC OPTIMIZATION LEVERS

AD IMPRESSIONSAD IMPRESSIONS

AD CLICKSAD CLICKS

LEVER 1: Budget

LEVER 2: Bids

LEVER 3: Quality Score

Increase Budget

Optimize for KPI

Maximize QS

LEVER 4: CTR Test Ad Copy

• Continuously A/B test ad

• Keep highest CTR ad• Delete lower CTR ads• Repeat until returns

diminish

Clicks from AdAd Impressions

CTR =

SOCIAL CONTENTOPTIMIZATION LEVERS

SOCIAL SOCIAL IMPRESSIONSIMPRESSIONS

SOCIAL CONTENT SOCIAL CONTENT CLICKSCLICKS

LEVER 1: Network

LEVER 2: Content

LEVER 3: Interactions

Add Networks

Quantity/Quality

Followers/Frequency

• Pinterest• Reddit• Others

• Be there in the first place

SOCIAL CONTENTOPTIMIZATION LEVERS

SOCIAL SOCIAL IMPRESSIONSIMPRESSIONS

SOCIAL CONTENT SOCIAL CONTENT CLICKSCLICKS

LEVER 1: Network

LEVER 2: Content

LEVER 3: Interactions

Add Networks

Quantity/Quality

Followers/Frequency

• Frequency• Better content drives

more shares/retweets

• Links to deep content• Polite, relevant posts

SOCIAL CONTENTOPTIMIZATION LEVERS

SOCIAL SOCIAL IMPRESSIONSIMPRESSIONS

SOCIAL CONTENT SOCIAL CONTENT CLICKSCLICKS

LEVER 1: Network

LEVER 2: Content

LEVER 3: Interactions

Add Networks

Quantity/Quality

Followers/Frequency

• Increase quantity and value of followers

• Don’t “buy” followers

• More interactions drives more clicks

WEBSITE LAYOUT & CONTENT

VISITSVISITS

CONVERSIONSCONVERSIONS

LEVER 1: Layout

LEVER 2: Content

LEVER 3: Click Areas

Forms, Phone #s

Quantity, Quality

Size, Color, Position

LEVER X: etc.....

EXECUTION

THE WEB IS A SYSTEM

• Your marketing strategy is a system of equations• Some levers are independent of others• Some levers manipulate others• Some levers have a larger impact on KPI than

others

Ad Bid SEO Spend

Email Subject

Social Effort

THE WEB IS A SYSTEM

• Think about moving several levers at once• Always consider what a single and multiple-lever move

will do to the system, and accordingly, your end result• Design a good experiment, continue to measure, and

repeat

y = f( x1, x2, x3, . . . . xn )

EXAMPLE SCENARIO

• Current Budget: $7,500 / month ( $250 / day )• Conversions: 48 / month ( 1.6 / day )• Average CPA = $158.17• Average CPC = $2.29• Landing Page CR = 1.45%

New Target: 10 Conversions / Day AND Lower CPA

SCENARIO ONE: ADJUSTING BUDGET

• Average CPA at $158.17 means 10 conversions per day is $1581.70

• $47,000 per month• Easy, but too expensive

SCENARIO TWO: ADJUSTING CPC

• Budget stays at $250 / day, conversion rate steady at 1.45%

• Need 689 clicks to get 10 conversions• Average CPC must be $0.36• Improbable to bring $2.29 down to $0.36 in

this space while maintaining volume

SCENARIO THREE: ADJUST CONVERSION RATE

• Budget stays at $250 / day, CPC at $2.29• Can purchase 110 clicks for $250• To get 10 conversions, landing page needs to

convert at 9.1%• Improbable to quickly jump from 1.45% to

9.1%

FINAL SCENARIO: MOVE ALL THREE LEVERS

• Double the budget to $15,000• Reduce average CPC by 50% ( $2.29 to $1.15 ) via

QS & Keywords• Improve landing page CR from 1.45% to 2.3% via

A/B testing• ( $15k / $1.15 per click ) at 2.3% drives 300

conversions monthly at a CPA of $50• Used 3 levers simultaneously to improve

conversions 6x and reduce average CPA by 68%

ITERATING

CREATE A CULTURE OF CONTINUOUS IMPROVEMENT

• Define success• Measure current performance and compare to

targets• Determine biggest levers and how they contribute

to KPIs• Design an experiment• Test repeatedly• Finalize improvements based on adequate data• Repeat

Questions?

Scott Yacko - scott@vuurr.com - @scottmyacko

Jonathan Kressaty - jonathan@vuurr.com - @kressaty