Post on 22-Feb-2017
Holiday Catalog Testing
• Why testing this year matters next year
• Best practices for test set up and results
assessment that empower you with dependable
data for predicting future performance.
• Test variables that provide actionable learnings
most likely to most significantly improve
performance
Holiday Catalog Testing
• Why testing this year matters next year
• Best practices for test set up and results
assessment that empower you with dependable
data for predicting future performance.
• Test variables that provide actionable learnings
most likely to most significantly improve
performance
Holiday Catalog Testing
• Why testing this year matters next year
• Best practices for test set up and results
assessment that empower you with dependable
data for predicting future performance.
• Test variables that provide actionable learnings
most likely to most significantly improve
performance
Holiday Catalog Testing
• Why testing this year matters next year
• Best practices for test set up and results
assessment that empower you with dependable
data for predicting future performance.
• Test variables that provide actionable learnings
most likely to most significantly improve
performance
$1.0M $1.2M$1.5M
$1.8M $1.8M 2.0M 2.0M
2.5M3.0M 3.0M
$4.0M
$6.0M $6.3M
$7.5M $7.2M
Spend
Circulation
Revenue
Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual
Test Winner More Costly Device Offer Lists Lists
Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ
Spend $1M $1.2M $1.5M $1.8M $1.8M
CPP $0.50 $0.60 $0.60 $0.60 $0.60
Circ 2M 2M 2.5M 3M 3M
Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40
Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M
Incr ROAS 4.0 5.0 4.2 4.2 4.0
$ Chg YOY 50% 4% 20% 15%
20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY
Ch
ange
vs
Yea
r 1
80
%
80
%
50
%
This
is w
he
re w
e w
ant
to b
e…
Assumes a margin and ROI target that is acceptable for this fictitious business.
Conservative long-term vision…
what any manager would like to see
$1.0M $1.2M$1.5M
$1.8M $1.8M 2.0M 2.0M
2.5M3.0M 3.0M
$4.0M
$6.0M $6.3M
$7.5M $7.2M
Spend
Circulation
Revenue
Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual
Test Winner More Costly Device Offer Lists Lists
Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ
Spend $1M $1.2M $1.5M $1.8M $1.8M
CPP $0.50 $0.60 $0.60 $0.60 $0.60
Circ 2M 2M 2.5M 3M 3M
Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40
Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M
Incr ROAS 4.0 5.0 4.2 4.2 4.0
$ Chg YOY 50% 4% 20% 15%
20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY
Ch
ange
vs
Yea
r 1
80
%
80
%
50
%
This
is w
he
re w
e w
ant
to b
e…
Assumes a margin and ROI target that is acceptable for this fictitious business.
Conservative long-term vision…
what any manager would like to see
$1.0M $1.2M$1.5M
$1.8M $1.8M 2.0M 2.0M
2.5M3.0M 3.0M
$4.0M
$6.0M $6.3M
$7.5M $7.2M
Spend
Circulation
Revenue
Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual
Test Winner More Costly Device Offer Lists Lists
Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ
Spend $1M $1.2M $1.5M $1.8M $1.8M
CPP $0.50 $0.60 $0.60 $0.60 $0.60
Circ 2M 2M 2.5M 3M 3M
Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40
Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M
Incr ROAS 4.0 5.0 4.2 4.2 4.0
$ Chg YOY 50% 4% 20% 15%
20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY
Ch
ange
vs
Yea
r 1
80
%
80
%
50
%
This
is w
he
re w
e w
ant
to b
e…
Assumes a margin and ROI target that is acceptable for this fictitious business.
Conservative long-term vision…
what any manager would like to see
$1.0M $1.2M$1.5M
$1.8M $1.8M 2.0M 2.0M
2.5M3.0M 3.0M
$4.0M
$6.0M $6.3M
$7.5M $7.2M
Spend
Circulation
Revenue
Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual
Test Winner More Costly Device Offer Lists Lists Lists
Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ Expanded Circ
Spend $1M $1.2M $1.5M $1.8M $1.8M
CPP $0.50 $0.60 $0.60 $0.60 $0.60
Circ 2M 2M 2.5M 3M 3M
Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40
Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M
Incr ROAS 4.0 5.0 4.2 4.2 4.0
$ Chg YOY 50% 4% 20% 15%
20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY
Ch
ange
vs
Yea
r 1
80
%
80
%
50
%
This
is w
he
re w
e w
ant
to b
e…
Assumes a margin and ROI target that is acceptable for this fictitious business.
Conservative long-term vision…
what any manager would like to see
Conservative long-term vision…
what any manager would like to see
$1.0M $1.2M$1.5M
$1.8M $1.8M 2.0M 2.0M
2.5M3.0M 3.0M
$4.0M
$6.0M $6.3M
$7.5M $7.2M
Spend
Circulation
Revenue
Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual
Test Winner More Costly Device Offer Lists Lists
Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ
Spend $1M $1.2M $1.5M $1.8M $1.8M
CPP $0.50 $0.60 $0.60 $0.60 $0.60
Circ 2M 2M 2.5M 3M 3M
Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40
Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M
Incr ROAS 4.0 5.0 4.2 4.2 4.0
$ Chg YOY 50% 4% 20% 15%
20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY
Ch
ange
vs
Yea
r 1
80
%
80
%
50
%
This
is w
he
re w
e w
ant
to b
e…
Assumes a margin and ROI target that is acceptable for this fictitious business.
Execute roll-outs and tests each year with a
view across at least 3 holiday seasons…
• THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS.
– That’s just hard work – execution. Has to be done perfectly.
• THIS year’s TESTS are selected informed by LASTyear’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year.
– Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign.
– Don’t let testing get lost! Make testing a priority!
– Plus – it’s FUN! It’s THE fun!
This Year Next YearLast Year
Execute roll-outs and tests each year with a
view across at least 3 holiday seasons…
• THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS.
– That’s just hard work – execution. Has to be done perfectly.
• THIS year’s TESTS are selected informed by LASTyear’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year.
– Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign.
– Don’t let testing get lost! Make testing a priority!
– Plus – it’s FUN! It’s THE fun!
This Year Next YearLast Year
Execute roll-outs and tests each year with a
view across at least 3 holiday seasons…
• THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS.
– That’s just hard work – execution. Has to be done perfectly.
• THIS year’s TESTS are selected informed by LASTyear’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year.
– Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign.
– Don’t let testing get lost! Make testing a priority!
– Plus – it’s FUN! It’s THE fun!
This Year Next YearLast Year
Execute roll-outs and tests each year with a
view across at least 3 holiday seasons…
• THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS.
– That’s just hard work – execution. Has to be done perfectly.
• THIS year’s TESTS are selected informed by LASTyear’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year.
– Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign.
– Don’t let testing get lost! Make testing a priority!
– Plus – it’s FUN! It’s THE fun!
This Year Next YearLast Year
Execute roll-outs and tests each year with a
view across at least 3 holiday seasons…
• THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS.
– That’s just hard work – execution. Has to be done perfectly.
• THIS year’s TESTS are selected informed by LASTyear’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year.
– Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign.
– Don’t let testing get lost! Make testing a priority!
– Plus – it’s FUN! It’s THE fun!
This Year Next YearLast Year
Execute roll-outs and tests each year with a
view across at least 3 holiday seasons…
• THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS.
– That’s just hard work – execution. Has to be done perfectly.
• THIS year’s TESTS are selected informed by LASTyear’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year.
– Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign.
– Don’t let testing get lost! Make testing a priority!
– Plus – it’s FUN! It’s THE fun!
This Year Next YearLast Year
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Simple Campaign Structure
Mailed Circulation Audience Device Offer IHDExpiration
Date
Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec
Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec
Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec
Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec
Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec
Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec
Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec
Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec
Total Circulation Mailed 2,000,000 - - - - - - - - - - - -
No Mail Control Audience - - - - - - Start End
No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec
No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec
Total No Mail Control 50,000 - - - - - - - - - - - -
Audience Audience - - - - - - - - - - - -
Total Required 2,050,000 - - - - - - - - - - - -
Building Testpools
• For our example:– Take the 2.05M Households
– Split into 10 cohorts, n-thing off for equal balance, so that if all cohorts were treated equally in the mail they should all behave the same
• We need 9 cohorts of 25k and 1 rollout cohort of 1.825M
– For house cohorts, check that they’re balanced by checking each cohort’s transaction history – they should each have the same 0-6 month revenue per household
– For prospect cohorts, test that they’re balanced by checking if zips are equally represented
Building Testpools
• For our example:– Take the 2.05M Households
– Split into 10 cohorts, n-thing off for equal balance, so that if all cohorts were treated equally in the mail they should all behave the same
• We need 9 cohorts of 25k and 1 rollout cohort of 1.825M
– For house cohorts, check that they’re balanced by checking each cohort’s transaction history – they should each have the same 0-6 month revenue per household
– For prospect cohorts, test that they’re balanced by checking if zips are equally represented
Building Testpools
• For our example:– Take the 2.05M Households
– Split into 10 cohorts, n-thing off for equal balance, so that if all cohorts were treated equally in the mail they should all behave the same
• We need 9 cohorts of 25k and 1 rollout cohort of 1.825M
– For house cohorts, check that they’re balanced by checking each cohort’s transaction history – they should each have the same 0-6 month revenue per household
– For prospect cohorts, test that they’re balanced by checking if zips are equally represented
Building Testpools
• For our example:– Take the 2.05M Households
– Split into 10 cohorts, n-thing off for equal balance, so that if all cohorts were treated equally in the mail they should all behave the same
• We need 9 cohorts of 25k and 1 rollout cohort of 1.825M
– For house cohorts, check that they’re balanced by checking each cohort’s transaction history – they should each have the same 0-6 month revenue per household
– For prospect cohorts, test that they’re balanced by checking if zips are equally represented
Example of n-thing
We have an audience of 100k
We want 5 testpools of 10k each• No Mail Control• Mail Control• Test 1 of 3• Test 2 of 3• Test 3 of 3
We want the remainder, 50k, for the rollout
50% 10.0% 10.0% 10.0% 10.0% 10.0%
HouseholdRollout
NoMail
Control
Control
Test1
of3
Test2
of3
Test3
of3
1 1
2 1
3 1
4 1
5 1
6 1
7 1
8 1
9 1
10 1
11 1
12 1
13 1
14 1
15 1
16 1
17 1
18 1
19 1
20 1
Etc….
Test Pool Size: What’s Optimal?
• Best Practice– Work with an analyst who is fully informed on your campaign
structure and your testing objectives to determine optimal test pool size
• Second-Best Practice– 100 orders minimum “rule of thumb”
• Have your test pool size large enough that you are confident that you will definitely get 100 orders or more
• Assume prospect response will be just 20-30% of house response
• If possible, shoot for test pools sizes that will deliver much more than 100 orders – particularly if the results of your tests will inform decisions on a grand scale
– Example: we don’t want make a change to a 20M rollout campaign that would significantly impact costs or revenue based on a test conducted with a testpool size of just 5,000
Test Pool Size: What’s Optimal?
• Best Practice– Work with an analyst who is fully informed on your campaign
structure and your testing objectives to determine optimal test pool size
• Second-Best Practice– 100 orders minimum “rule of thumb”
• Have your test pool size large enough that you are confident that you will definitely get 100 orders or more
• Assume prospect response will be just 20-30% of house response
• If possible, shoot for test pools sizes that will deliver much more than 100 orders – particularly if the results of your tests will inform decisions on a grand scale
– Example: we don’t want make a change to a 20M rollout campaign that would significantly impact costs or revenue based on a test conducted with a testpool size of just 5,000
Test Pool Size: What’s Optimal?
• Best Practice– Work with an analyst who is fully informed on your campaign
structure and your testing objectives to determine optimal test pool size
• Second-Best Practice– 100 orders minimum “rule of thumb”
• Have your test pool size large enough that you are confident that you will definitely get 100 orders or more
• Assume prospect response will be just 20-30% of house response
• If possible, shoot for test pools sizes that will deliver much more than 100 orders – particularly if the results of your tests will inform decisions on a grand scale
– Example: we don’t want make a change to a 20M rollout campaign that would significantly impact costs or revenue based on a test conducted with a testpool size of just 5,000
No Mail Control vs. Mail Control Results
Audience
FullyLoaded
Matchback
FullyLoaded
Matchback/Household
Incremental
$/Household
Incremental
Revenue
NoMailControl1 25,000 298,750$ 11.95$
NoMailControl2 25,000 296,250$ 11.85$
TotalNoMailControl 50,000 595,000$ 11.90$
MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$
MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$ 155,000$
Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$
TotalMailControls/Rollout 1,875,000 28,125,000$ 15.00$ 3.10$ 5,812,500$
$15.00-$11.90=$3.10
– Matchback Incremental Lift Assessment Model (above)
• Mail Control Matchback $ or Orders minus No Mail Control
Matchback $ or Orders (you can also add in Pass-Along Promo Code
Redemption $ or Orders) = Incremental Result
No Mail Control vs. Mail Control Results
Audience
FullyLoaded
Matchback
FullyLoaded
Matchback/Household
Incremental
$/Household
Incremental
Revenue
NoMailControl1 25,000 298,750$ 11.95$
NoMailControl2 25,000 296,250$ 11.85$
TotalNoMailControl 50,000 595,000$ 11.90$
MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$
MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$ 155,000$
Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$
TotalMailControls/Rollout 1,875,000 28,125,000$ 15.00$ 3.10$ 5,812,500$
$15.00-$11.90=$3.10
– Matchback Incremental Lift Assessment Model (above)
• Mail Control Matchback $ or Orders minus No Mail Control
Matchback $ or Orders (you can also add in Pass-Along Promo Code
Redemption $ or Orders) = Incremental Result
No Mail Control vs. Mail Control Results
Audience
FullyLoaded
Matchback
FullyLoaded
Matchback/Household
Incremental
$/Household
Incremental
Revenue
NoMailControl1 25,000 298,750$ 11.95$
NoMailControl2 25,000 296,250$ 11.85$
TotalNoMailControl 50,000 595,000$ 11.90$
MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$
MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$ 155,000$
Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$
TotalMailControls/Rollout 1,875,000 28,125,000$ 15.00$ 3.10$ 5,812,500$
$15.00-$11.90=$3.10
– Matchback Incremental Lift Assessment Model (above)
• Mail Control Matchback $ or Orders minus No Mail Control
Matchback $ or Orders (you can also add in Pass-Along Promo Code
Redemption $ or Orders) = Incremental Result
No Mail Control vs. Mail Control Results
Audience
FullyLoaded
Matchback
FullyLoaded
Matchback/Household
Incremental
$/Household
Incremental
Revenue
NoMailControl1 25,000 298,750$ 11.95$
NoMailControl2 25,000 296,250$ 11.85$
TotalNoMailControl 50,000 595,000$ 11.90$
MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$
MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$ 155,000$
Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$
TotalMailControls/Rollout 1,875,000 28,125,000$ 15.00$ 3.10$ 5,812,500$
$15.00-$11.90=$3.10
– Matchback Incremental Lift Assessment Model (above)
• Mail Control Matchback $ or Orders minus No Mail Control
Matchback $ or Orders (you can also add in Pass-Along Promo Code
Redemption $ or Orders) = Incremental Result
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 5-15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 5-15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 5-15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 5-15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Tests vs. Mail Control
(Rollout) Results
• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M
more in incremental revenue vs. this year (but cut that back 5-15%).
• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT
AudienceFullyLoadedMatchback
FullyLoaded
Matchback/Household
Incremental$/Household
Incremental
vs.IncrMailControl
TotalNoMailControl 50,000 595,000$ 11.90$
TotalMailControl 50,000 750,000$ 15.00$ 3.10$
TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$
TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$
TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$
$5.60 - $3.10 = $2.50
Results Assessment Models
• 2 primary ways to measure results
– Matchback Incremental Lift Model• Match back your Mail and No Mail Controls
• Mail minus No Mail Control + Pass-Alongs = Incremental Revenue
– Promotion Code Redemptions• Match Back Redemptions
• Measure Pass-Alongs– promo code usage by customers not mailed – somehow they
received the coupon
• Total Redemptions
• Redemption Entries (entered into account but not used) vs. Usage
Results Assessment Models
• 2 primary ways to measure results
– Matchback Incremental Lift Model• Match back your Mail and No Mail Controls
• Mail minus No Mail Control + Pass-Alongs = Incremental Revenue
– Promotion Code Redemptions• Match Back Redemptions
• Measure Pass-Alongs– promo code usage by customers not mailed – somehow they
received the coupon
• Total Redemptions
• Redemption Entries (entered into account but not used) vs. Usage
Results Assessment Models
• 2 primary ways to measure results
– Matchback Incremental Lift Model (Best)• Match back your Mail and No Mail Controls
• Mail minus No Mail Control + Pass-Alongs = Incremental Revenue
– Promotion Code Redemptions• Match Back Redemptions
• Measure Pass-Alongs– promo code usage by customers not mailed – somehow they
received the coupon
• Total Redemptions
• Redemption Entries (entered into account but not used) vs. Usage
What results can show and not show
• Matchbacks Incremental Lift Model only tells you
incremental number of responders and revenue –
we don’t know WHO was the incremental
responder
– And your catalog drives not just incremental responders,
but incremental revenue from those responders who were
going to order anyway whether they received the catalog
or not – they spent more than they might have if they
hadn’t received it
– You can look at incremental revenue by product category
• Promo Code Redemptions tell you who engaged
– But NOT who would have engaged anyway if you hadn’t
mailed at all
What results can show and not show
• Matchbacks Incremental Lift Model only tells you
incremental number of responders and revenue –
we don’t know WHO was the incremental
responder
– And your catalog drives not just incremental responders,
but incremental revenue from those responders who were
going to order anyway whether they received the catalog
or not – they spent more than they might have if they
hadn’t received it
– You can look at incremental revenue by product category
• Promo Code Redemptions tell you who engaged
– But NOT who would have engaged anyway if you hadn’t
mailed at all
What results can show and not show
• Matchbacks Incremental Lift Model only tells you
incremental number of responders and revenue –
we don’t know WHO was the incremental
responder
– And your catalog drives not just incremental responders,
but incremental revenue from those responders who were
going to order anyway whether they received the catalog
or not – they spent more than they might have if they
hadn’t received it
– You can look at incremental revenue by product category
• Promo Code Redemptions tell you who engaged
– But NOT who would have engaged anyway if you hadn’t
mailed at all
What results can show and not show
• Matchbacks Incremental Lift Model only tells you
incremental number of responders and revenue –
we don’t know WHO was the incremental
responder
– And your catalog drives not just incremental responders,
but incremental revenue from those responders who were
going to order anyway whether they received the catalog
or not – they spent more than they might have if they
hadn’t received it
– You can look at incremental revenue by product category
• Promo Code Redemptions tell you who engaged
– But NOT who would have engaged anyway if you hadn’t
mailed at all
How many tests in holiday season?
• Conduct as many tests as practical within bounds of circ size, budget, keeping it simple, and prioritizing tests likely to improve performance the most
• Some tests will be winners, most will be losers vs your roll-out control– You may lose some short-term revenue due to testing but
that’s the cost of increasing revenue next year.
• One chance per year – don’t miss it – so certainly don’t under-test.
• More tests increase chance you find something that will move the meter and have a meaningful impact on next year’s results
How many tests in holiday season?
• Conduct as many tests as practical within bounds of circ size, budget, keeping it simple, and prioritizing tests likely to improve performance the most
• Some tests will be winners, most will be losers vs your roll-out control– You may lose some short-term revenue due to testing but
that’s the cost of increasing revenue next year.
• One chance per year – don’t miss it – so certainly don’t under-test.
• More tests increase chance you find something that will move the meter and have a meaningful impact on next year’s results
How many tests in holiday season?
• Conduct as many tests as practical within bounds of circ size, budget, keeping it simple, and prioritizing tests likely to improve performance the most
• Some tests will be winners, most will be losers vs your roll-out control– You may lose some short-term revenue due to testing but
that’s the cost of increasing revenue next year.
• One chance per year – don’t miss it – so certainly don’t under-test.
• More tests increase chance you find something that will move the meter and have a meaningful impact on next year’s results
How many tests in holiday season?
• Conduct as many tests as practical within bounds of circ size, budget, keeping it simple, and prioritizing tests likely to improve performance the most
• Some tests will be winners, most will be losers vs your roll-out control– You may lose some short-term revenue due to testing but
that’s the cost of increasing revenue next year.
• One chance per year – don’t miss it – so certainly don’t under-test.
• More tests increase chance you find something that will move the meter and have a meaningful impact on next year’s results
What to test?
Here are come
categories:
– Lists
– Devices
– Promotional
Offers
– Merchandise
– Timing
– Expiration
Dates
– Covers
– Messaging
– Design
– Color
– Frequency
– Digital Integration (Catalog + Email,
Catalog + Display)
What to test?
Here are come
categories:
– Lists
– Devices
– Promotional
Offers
– Merchandise
– Timing
– Expiration
Dates
– Covers
– Messaging
– Design
– Color
– Frequency
– Digital Integration (Catalog + Email,
Catalog + Display)
What to test?
Here are come
categories:
– Lists
– Devices
– Promotional
Offers
– Merchandise
– Timing
– Expiration
Dates
– Covers
– Messaging
– Design
– Color
– Frequency
– Digital Integration (Catalog + Email,
Catalog + Display)
What matters most?Factors most likely to increase performance
• In my experience, what will move the meter the most is the interaction between:(Assuming timing & merch mix are adequate.)
– Lists• Relevance
• Breadth
• Depth
– Device “in-the-mail”• You may call it a “Form Factor”?
• Catalog / Pages / Merch / Bells & Whistles / Wraps / Wallet Cards / Blow-Ins / Inserts / Envelopes / Things to Peel / Greeting Card / Postcard
– Promotional Offer (Incentive)• Save! Free! Survey! Exclusive Access!
• Be uncomfortably generous
What matters most?Factors most likely to increase performance
• In my experience, what will move the meter the most is the interaction between:(Assuming timing & merch mix are adequate.)
– Lists• Relevance
• Breadth
• Depth
– Device “in-the-mail”• You may call it a “Form Factor”?
• Catalog / Pages / Merch / Bells & Whistles / Wraps / Wallet Cards / Blow-Ins / Inserts / Envelopes / Things to Peel / Greeting Card / Postcard
– Promotional Offer (Incentive)• Save! Free! Survey! Exclusive Access!
• Be uncomfortably generous
What matters most?Factors most likely to increase performance
• In my experience, what will move the meter the most is the interaction between:(Assuming timing & merch mix are adequate.)
– Lists• Relevance
• Breadth
• Depth
– Device “in-the-mail”• You may call it a “Form Factor”?
• Catalog / Pages / Merch / Bells & Whistles / Wraps / Wallet Cards / Blow-Ins / Inserts / Envelopes / Things to Peel / Greeting Card / Postcard
– Promotional Offer (Incentive)• Save! Free! Survey! Exclusive Access!
• Be uncomfortably generous
What matters most?Factors most likely to increase performance
• In my experience, what will move the meter the most is the interaction between:(Assuming timing & merch mix are adequate.)
– Lists• Relevance
• Breadth
• Depth
– Device “in-the-mail”• You may call it a “Form Factor”?
• Catalog / Pages / Merch / Bells & Whistles / Wraps / Wallet Cards / Blow-Ins / Inserts / Envelopes / Things to Peel / Greeting Card / Postcard
– Promotional Offer (Incentive)• Save! Free! Survey! Exclusive Access!
• Be uncomfortably generous
Test: House Lists
• Segmentation RFMP– Recency
– Frequency
– Monetary
– Product
– Seasonal
– Inquiries / Non-Buyers
• Reactivation Scoring (lapsed “look-alike” modeling)
• Match your e-mail only customers to postal addresses and mail them
• Match your anonymous site visitors to postal addresses and mail them
Test: House Lists
• Segmentation RFMP– Recency
– Frequency
– Monetary
– Product
– Seasonal
– Inquiries / Non-Buyers
• Reactivation Scoring (lapsed “look-alike” modeling)
• Match your e-mail only customers to postal addresses and mail them
• Match your anonymous site visitors to postal addresses and mail them
Test: House Lists
• Segmentation RFMP– Recency
– Frequency
– Monetary
– Product
– Seasonal
– Inquiries / Non-Buyers
• Reactivation Scoring (lapsed “look-alike” modeling)
• Match your e-mail only customers to postal addresses and mail them
• Match your anonymous site visitors to postal addresses and mail them
Test: House Lists
• Segmentation RFMP– Recency
– Frequency
– Monetary
– Product
– Seasonal
– Inquiries / Non-Buyers
• Reactivation Scoring (lapsed “look-alike” modeling)
• Match your e-mail only customers to postal addresses and mail them
• Match your anonymous site visitors to postal addresses and mail them
Test: Prospect Lists
• Test from many list categories:– Vertical Lists (ex: Pottery Barn Buyers)
– Compiled Lists (ex: Kindergarten Teachers)
– Look-Alike Modeled Lists (co-ops)
– Subscriber Lists (magazines)
– Association Lists (conferences)
– Business Partner Lists• Do you sell to same target, but not competitive products?
• Use each other’s lists
– Competitor List Swaps• Trust, good relations, and a common belief that both parties
benefit (that both sell more) by mailing each other’s lists are required. This is a rare arrangement, but it works for those who partner to make it possible.
Test: Prospect Lists
• Test from many list categories:– Vertical Lists (ex: Pottery Barn Buyers)
– Compiled Lists (ex: Kindergarten Teachers)
– Look-Alike Modeled Lists (co-ops)
– Subscriber Lists (magazines)
– Association Lists (conferences)
– Business Partner Lists• Do you sell to same target, but not competitive products?
• Use each other’s lists
– Competitor List Swaps• Trust, good relations, and a common belief that both parties
benefit (that both sell more) by mailing each other’s lists are required. This is a rare arrangement, but it works for those who partner to make it possible.
Test: Prospect Lists
• Test from many list categories:– Vertical Lists (ex: Pottery Barn Buyers)
– Compiled Lists (ex: Kindergarten Teachers)
– Look-Alike Modeled Lists (co-ops)
– Subscriber Lists (magazines)
– Association Lists (conferences)
– Business Partner Lists• Do you sell to same target, but not competitive products?
• Use each other’s lists
– Competitor List Swaps• Trust, good relations, and a common belief that both parties
benefit (that both sell more) by mailing each other’s lists are required. This is a rare arrangement, but it works for those who partner to make it possible.
Use Promo Codes AND Matchbacks for
list results analysis
• Promo Codes are very helpful for LIST analysis – NMCs (No Mail Controls) may be too small at list segment
level to meaningfully measure the incremental revenue
– You may not want to pay for the prospect NMC names for every list – particularly at large cohort sizes that provide statistically significant data
– You will at least get a read on the promo code performance of list segments relative to each other
– If your match back reports end up unreliable for some reason, promo code redemption usage is a back-up path to results analysis (though not preferrable vs. match back and incremental lift model).
– In my experience, the incremental performance of prospect lists lands at 90-110% of the promo code redemptions – not too far off (house lists not so)
• Therefore, assign a separate promo code group for each list test, each segment depth
Test: Device
• Test devices that make you feel uncomfortably radical – don’t just be a “catalog”– They stand out in the mailbox, violate, and engage,
triggering a pause for further consideration, an interception on the way to the trash can
– What sparkles? Catches the eye? Feeling of touch?
• Thick plastic postcards
– What conveys value?
• Wallet Cards (they look like credit or gift cards)
– What conveys a trigger to take action?
• Sticky Notes (anything that peels)
• It’s only a test! • Remind everyone at your company that only a small portion of
the mailing will receive this radial device – it won’t “destroy the brand” if mailed once to a small cohort
OBJECTIVE: Intercept the trash can and get recipient from the direct mailer / catalog to your web site. Everything about your device should be serving this objective!
Costlier Devices can PAY out BIG time!
• Costlier devices can have significant impact in the mailbox.– You’re not just competing with your compeitors in your
sector, you’re competing with whatever else arrives in the recipient’s mailbox that day – are you standing out?
• If $0.10 extra spend per piece in the mail for some violating bells & whistles engagement brings in $2.50 more revenue, you have a ROAS of 25 on that incremental $0.10 spend – an insanely high ROAS.
• A blow-in card, with an extra ink-jetted promo code promoting a featured product, costs about $0.01. If that lifts your result $0.05 you have a ROAS of 5.
Costlier Devices can PAY out BIG time!
• Costlier devices can have significant impact in the mailbox.– You’re not just competing with your compeitors in your
sector, you’re competing with whatever else arrives in the recipient’s mailbox that day – are you standing out?
• If $0.10 extra spend per piece in the mail for some violating bells & whistles engagement brings in $2.50 more revenue, you have a ROAS of 25 on that incremental $0.10 spend – an insanely high ROAS.
• A blow-in card, with an extra ink-jetted promo code promoting a featured product, costs about $0.01. If that lifts your result $0.05 you have a ROAS of 5.
Costlier Devices can PAY out BIG time!
• Costlier devices can have significant impact in the mailbox.– You’re not just competing with your compeitors in your
sector, you’re competing with whatever else arrives in the recipient’s mailbox that day – are you standing out?
• If $0.10 extra spend per piece in the mail for some violating bells & whistles engagement brings in $2.50 more revenue, you have a ROAS of 25 on that incremental $0.10 spend – an insanely high ROAS.
• A blow-in card, with an extra ink-jetted promo code promoting a featured product, costs about $0.01. If that lifts your result $0.05 you have a ROAS of 5.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Promotional Offers
• Here’s what I am seeing from my experience:
– Save 10-30% isn’t meaningfully increasing response
– Save $ offers seem to perform better than Save % offers
– BOGO where margins make it possible tends to be powerful
– Free Shipping isn’t meaningfully increasing response
– Free Gifts aren’t meaningfully increasing response
– Free Products – if you can give products away, do!
– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response
– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.
– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.
Test: Merchandise
• More or less merch?
• Breadth? Depth? – Pack more merch into your existing catalog size
– Cut pages – can you spend less?
– Add in a centerfold stitched-in insert of extra pages, a “gift guide” booklet
– Try a postcard instead
• Plastic postcards perform 2-3x better than card-stock postcards
– Try a greeting card instead – very little merch
• Invitation
• Wallet Card
• Works great with existing customers
Test: Promotional Offer Expiration Dates
• I’ve seen both short-term expiration dates and
extended expiration dates perform well
• Test to see what works best for your business
• Don’t test small differences – test big differences:
– Nov 30
• Careful! Don’t run the risk that your offer expires before it lands
in mailboxes!
– Dec15
– Dec20
– Dec31
– Jan15
Test: Timing – when you’re landing in
mailboxes
• Holiday catalogs are hitting homes in October
these days…what works best for your catalog?
• Don’t guess! Test!
• Test a variety of dates
– Pre-Halloween
– Post-Halloween
– Pre-Thanksgiving
– Land in mailbox Thanksgiving weekend
– Post-Thanksgiving
Test: Covers
• Covers are difficult to test because a control won’t help you; you can’t repeat the same cover twice (or the same cover next year)
• Cover wraps, on the other hand, can be repeated and serve as a roll-out control to be beat each year
• If you’re going to test cover imagery, try:– “product” vs. “aspirational”
• Bottles of Wine VS.
• Beautiful people laughing and drinking wine at a Vineyard
– If “aspirational” works best, then always use “aspirational” covers of people partying at a Vineyard until you find some other approach that works better
– Variations on the same theme: every cover is of friends on a dramatic outdoor excursion
Test: Covers
• Covers are difficult to test because a control won’t help you; you can’t repeat the same cover twice (or the same cover next year)
• Cover wraps, on the other hand, can be repeated and serve as a roll-out control to be beat each year
• If you’re going to test cover imagery, try:– “product” vs. “aspirational”
• Bottles of Wine VS.
• Beautiful people laughing and drinking wine at a Vineyard
– If “aspirational” works best, then always use “aspirational” covers of people partying at a Vineyard until you find some other approach that works better
– Variations on the same theme: every cover is of friends on a dramatic outdoor excursion
Test: Covers
• Covers are difficult to test because a control won’t help you; you can’t repeat the same cover twice (or the same cover next year)
• Cover wraps, on the other hand, can be repeated and serve as a roll-out control to be beat each year
• If you’re going to test cover imagery, try:– “product” vs. “aspirational”
• Bottles of Wine VS.
• Beautiful people laughing and drinking wine at a Vineyard
– If “aspirational” works best, then always use “aspirational” covers of people partying at a Vineyard until you find some other approach that works better
– Variations on the same theme: every cover is of friends on a dramatic outdoor excursion
Test: Covers
• Covers are difficult to test because a control won’t help you; you can’t repeat the same cover twice (or the same cover next year)
• Cover wraps, on the other hand, can be repeated and serve as a roll-out control to be beat each year
• If you’re going to test cover imagery, try:– “product” vs. “aspirational”
• Bottles of Wine VS.
• Beautiful people laughing and drinking wine at a Vineyard
– If “aspirational” works best, then always use “aspirational” covers of people partying at a Vineyard until you find some other approach that works better
– Variations on the same theme: every cover is of friends on a dramatic outdoor excursion
Test: Covers
• Covers are difficult to test because a control won’t help you; you can’t repeat the same cover twice (or the same cover next year)
• Cover wraps, on the other hand, can be repeated and serve as a roll-out control to be beat each year
• If you’re going to test cover imagery, try:– “product” vs. “aspirational”
• Bottles of Wine VS.
• Beautiful people laughing and drinking wine at a Vineyard
– If “aspirational” works best, then always use “aspirational” covers of people partying at a Vineyard until you find some other approach that works better
– Variations on the same theme: every cover is of friends on a dramatic outdoor excursion
Test: Messaging
• In my experience, messaging relevant to a
targeted audience can increase results a bit –
5%.
– Never ordered?
– Ordered just once?
– Previous products purchased indicates they qualify
for relevant upsell?
Test: Colors
• Color in a vacuum has no impact except to the
degree that it:
– Helps or impairs the conveyance to the recipient of
your intended message
– Helps recipients recognize your company, such as
Shutterfly, Netflix
Test: Colors
• Color in a vacuum has no impact except to the
degree that it:
– Helps or impairs the conveyance to the recipient of
your intended message
– Helps recipients recognize your company, such as
Shutterfly, Netflix
Test: Colors
• Color in a vacuum has no impact except to the
degree that it:
– Helps or impairs the conveyance to the recipient of
your intended message
– Helps recipients recognize your company, such as
Shutterfly, Netflix
Test: Colors
• Color in a vacuum has no impact except to the
degree that it:
– Helps or impairs the conveyance to the recipient of
your intended message
– Helps recipients recognize your company, such as
Shutterfly, Netflix
Test: Digital Integration
• Email Test– Cohort A: Send Catalog Only
– Cohort B: Send Email Only
– Cohort C: Send Catalog + Email
• Display– Cohort A: Send Catalog Only
– Cohort B: Serve Display Only
– Cohort C: Send Catalog + Serve Display
• In my experience this integration isn’t a major lever, is a lot of work and cost with little payout– Difficult to read results, so much noise in those channels already
– Most catalog recipient households are probably getting hit with emails and display ads anyway.
– Not clear this integration has any impact, but it does cost!
Test: Digital Integration
• Email Test– Cohort A: Send Catalog Only
– Cohort B: Send Email Only
– Cohort C: Send Catalog + Email
• Display– Cohort A: Send Catalog Only
– Cohort B: Serve Display Only
– Cohort C: Send Catalog + Serve Display
• In my experience this integration isn’t a major lever, is a lot of work and cost with little payout– Difficult to read results, so much noise in those channels already
– Most catalog recipient households are probably getting hit with emails and display ads anyway.
– Not clear this integration has any impact, but it does cost!
Test: Digital Integration
• Email Test– Cohort A: Send Catalog Only
– Cohort B: Send Email Only
– Cohort C: Send Catalog + Email
• Display– Cohort A: Send Catalog Only
– Cohort B: Serve Display Only
– Cohort C: Send Catalog + Serve Display
• In my experience this integration isn’t a major lever, is a lot of work and cost with little payout– Difficult to read results, so much noise in those channels already
– Most catalog recipient households are probably getting hit with emails and display ads anyway.
– Not clear this integration has any impact, but it does cost!
• Don’t conclude from my experience – try it yourself…
Test: Digital Integration
• Email Test– Cohort A: Send Catalog Only
– Cohort B: Send Email Only
– Cohort C: Send Catalog + Email
• Display– Cohort A: Send Catalog Only
– Cohort B: Serve Display Only
– Cohort C: Send Catalog + Serve Display
• In my experience this integration isn’t a major lever, is a lot of work and cost with little payout– Difficult to read results, so much noise in those channels already
– Most catalog recipient households are probably getting hit with emails and display ads anyway.
– Not clear this integration has any impact, but it does cost!
• Don’t conclude from my experience – try it yourself…
Test: Frequency
– Test A • Single-Touch Mail Control Touch 1 of 1 (NOV15)
vs.
– Test B – The total combined revenue of…• 2-Touch Mail Touch 1 of 2 (NOV15)
• 2-Touch Mail Touch 2 of 2 (DEC1)
• Trigger Direct Mail – 24-hour follow-up to account activity (log-in, placed order, new customer)
Test: Frequency
– Test A • Single-Touch Mail Control Touch 1 of 1 (NOV15)
vs.
– Test B – The total combined revenue of…• 2-Touch Mail Touch 1 of 2 (NOV15)
• 2-Touch Mail Touch 2 of 2 (DEC1)
• Trigger Direct Mail – 24-hour follow-up to account activity (log-in, placed order, new customer)
Take-aways
• If you’re not testing, take some steps now to test something meaningful this year – it can be simple
• If you can test:– be sure to set up tests using best practices so that the
results are actionable
– Prioritize• Lists
• Promotional Offers
• Devices
• Don’t believe what anyone else says about what is working – test it yourself before taking action. – Believe in the best practice methods for testing.
– Believe in what you have learned yourself by testing using best practices!
Take-aways
• If you’re not testing, take some steps now to test something meaningful this year – it can be simple
• If you can test:– be sure to set up tests using best practices so that the
results are actionable
– Prioritize• Lists
• Promotional Offers
• Devices
• Don’t believe what anyone else says about what is working – test it yourself before taking action. – Believe in the best practice methods for testing.
– Believe in what you have learned yourself by testing using best practices!
Take-aways
• If you’re not testing, take some steps now to test something meaningful this year – it can be simple
• If you can test:– be sure to set up tests using best practices so that the
results are actionable
– Prioritize• Lists
• Promotional Offers
• Devices
• Don’t believe what anyone else says about what is working – test it yourself before taking action. – Believe in the best practice methods for testing.
– Believe in what you have learned yourself by testing using best practices!
Thanks!
Contact Info:
Christopher Werler
christopher@amherstdirect.com
781-962-7902