1

Accelerated Testingfor Retention and CRM Programs

Brian FichtelSenior Director, Customer RetentionHearst Magazines

Gordon BellPresident, LucidViewManaging Partner, Artestry

gbell @ lucidview.com

2

Overview of Retention / CRM programs

• Multiple effort subscription renewal series utilizing a mix of direct mail, e-mail, telephone and on-magazine efforts

• Generally segment by original acquisition source (e.g. Direct Mail, Internet) and subscriber tenure

• Series runs for as long as 9 months• Retention efforts are highly responsive and highly

profitable but have small quantities compared to Acquisition Direct Mail

• All of the above makes testing a long, challenging process with limited testing opportunities

3

Control Renewals / Retention Program

4 monthsbefore expiration

- 3 months

- 2 months

- 1 month

Subscriptionexpiration

1 month afterexpiration

+ 2 months

+ 3 months

+ 4 months

4

CRM Testing: Real-World Challenges

4 monthsbefore expiration

- 3 months

- 2 months

- 1 month

SubscriptionExpiration

+ 2 months

+ 3 months

+ 4 months

5

“Business Case” for Multivariate Testing

• Many creative, offer and segmentation test ideas but small retention quantities that limit testing opportunities

• Started with 41 ideas to test…

Envelope size Stronger copy pushing paymentPostage Laser magazine title on all lettersReturn slip position More aggressive in promotion of RAB discountBranding Increase urgency around expireFormat Attached mailOffer and payment info Same form with all effortsUrgency Postcard with drive-to-webMagazine benefits copy Offer expression - $0.97 vs. rounded up (9.97 vs. 10)Guarantee No discount for late effort High segmentsPayment chit Incentive - extra issues vs. pricingHighlight discount Credit card renewal optionChange price 2- and 3-year renewal optionIncentive for early renewal Hearst Rewards (earn points)Late-series incentive Renew now for digital copy/appMulti-year offer Special offer upon cancel requestCompress contact stream Add a chit - gift/cross-sell subscriptionMore e-mail Premium with 2- or 3-year termStart renewal cycle earlier Shorter term offer (6 issues) offerAdd telemarketing efforts Sweepstakes entry for early renewalE-mail beforehand Magnitude of price change (small/medium)

Decrease time between efforts

6

“Business Case” for Multivariate Testing

• Multivariate testing provided opportunity to test many ideas in a short period of time – Enabling results in 9 months instead of 3 years

• Multivariate testing also provided an opportunity to gain insights about test element interactions that’s not possible with traditional A/B testing

Test Design: Universe of Possibilities2-level

Test elements

Optimal designs• Complete freedom• Cost = complexity and error if assumptions are wrong

Taguchi designs• Use many levels• Bold assumption that interactions don’t exist

Orthogonal test designs• Full factorial• Fractional factorial• Plackett-Burman

• Centerpoints• CCD (etc.)

7

A/B split-runtests

Selected designs

8

“Mosaic” Test Design

• Two test designs1. 11 creative changes2. 6 elements of the offer and contact strategy

• Why?...a) Number of test elementsb) Magnitude of impact (creative changes tend to have a smaller effect)c) Value of interactions – mix of contacts and offer elements tend to have

larger, more valuable interactions

• Choices1. 12-recipe test for 11 creative elements ( = fewer package versions)2. 16-recipe test for 6 offer and creative elements ( = more interactions)

9

CRM Testing: Real-World Challenges

4 monthsbefore expiration

- 3 months

- 2 months

- 1 month

SubscriptionExpiration

+ 2 months

+ 3 months

+ 4 months

10

Test Elements

Test Elements - Creatives (-) Control (+) New idea

A Envelope size Large SmallB Postage Metered IndiciaC Return slip position Top of form Bottom of formD Branding Magazine branding Hearst corporation brandingE Format Letter (with signature) Statement (no signature)F Offer and payment info On right side Move to left sideG Urgency Current urgency copy Different urgency copyH Magazine benefits copy Control Stronger benefitsJ Guarantee None Add guarantee copyK Payment chit No Add chitL Highlight discount No Yes

Test Elements - Offer and Strategy (-) Control (+) New idea

A Change price Control +/- 20%B Incentive for early renewal No Yes, offer incentive in 2nd effortC Late-series incentive Discount PremiumD Multi-year offer Only once Add multi-year option to additional effortsE Compress series timing Control (every 4 weeks) Compress (every 2-3 weeks) and add efforts at endF More e-mail efforts Control More e-mail following DM efforts

Brainstormed 41 new ideas selected 17 test elements (split into 2 test designs)

11

“Mosaic” Test Designs E

nvel

ope

size

Pos

tage

Ret

urn

slip

pos

ition

Bra

ndin

g F

orm

at O

ffer a

nd p

aym

ent i

nfo

Urg

ency

cop

y M

agaz

ine

bene

fits

copy

Gua

rant

ee c

opy

Pay

men

t chi

t H

ighl

ight

dis

coun

t (la

ter

Recipe A B C D E F G H J K L Sample

1 – – – – – – – – – – – 205632 + + + – + + – + – – – 205883 + – + + – + – – – + + 204804 – – – + + + – + + – + 204715 + + – + + – + – – – + 204666 + – – – + + + – + + – 204577 + – + – – – + + + – + 205818 – + + – + – – – + + + 206339 – – + + + – + + – + – 20569

10 – + – – – + + + – + + 2045511 – + + + – + + – + – – 2051612 + + – + – – – + + + – 20528

Creative Test

Cha

nge

pric

e

Ince

ntiv

e fo

r ear

ly re

new

al

Lat

e-se

ries

ince

ntiv

e

Mul

ti-ye

ar o

ffer

Com

pres

s se

ries

timin

g

Mor

e e-

mai

l effo

rts

Recipe A B C D E F Sample

1 – – – – – – 20,4372 + + – – – – 19,8803 – – – + – + 20,5854 + + + – + – 19,9455 + + + + + + 20,0816 – + – + + – 20,4677 – – + – + – 20,3908 + – + – – + 19,9799 + – + + – – 20,083

10 + – – + + – 20,09011 – + – – + + 20,19012 + + – + – + 19,98813 – + + + – – 20,51614 – – + + + + 20,55715 + – – – + + 19,95116 – + + – – + 20,170

Offer & Strategy Test

12

Creative Test (control)

13

Creative Test (2 versions)

E+

F+

D+C+

G+

H+

14

Mosaic Creative Test – Results

-0.10%

-0.16%

-0.23%

+0.39%

+0.58%

-0.73%

-0.81%

-1.22%

-1.22%

-1.48%

-3.59%

0.0% 1.0% 2.0% 3.0%

B-: Postage

H-: Magazine benef its copy

C-: Return slip position

F+: Of fer and Payment Info

K+: Payment Chit

E-: Format

G-: Urgency

L-: Highlight discount (no)

A-: Envelope size (large)

J-: Guarantee (no)

D-: Branding (magazine)

Effect (avg RR of test as % change vs. control)

Hearst Magazines CreativeTest: Renewal Rate|Effects| as % change versus the control

Elements: sign of effect (optimal setting)

Four significant effects:

D-: Brand letters with the magazine title (not Hearst Magazines)

J- and L-: Skip the guarantee language and highlighting the discount

A-: Keep the larger envelope

15

Mosaic Offer & Strategy Test – Design

3-levels of price means that each A+Recipe is divided into two groups:Half with a 20% drop in priceHalf with a 20% price increase

Cha

nge

pric

e

Ince

ntiv

e fo

r ear

ly re

new

al

Lat

e-se

ries

ince

ntiv

e

Mul

ti-ye

ar o

ffer

Com

pres

s se

ries

timin

g

Mor

e e-

mai

l effo

rts

Recipe A B C D E F Sample

1 – – – – – – 20,4372 + + – – – – 19,8803 – – – + – + 20,5854 + + + – + – 19,9455 + + + + + + 20,0816 – + – + + – 20,4677 – – + – + – 20,3908 + – + – – + 19,9799 + – + + – – 20,083

10 + – – + + – 20,09011 – + – – + + 20,19012 + + – + – + 19,98813 – + + + – – 20,51614 – – + + + + 20,55715 + – – – + + 19,95116 – + + – – + 20,170

16

Recipe A B C D E F Model 1 Model 2 Model 3 Model 4 Model 1 Model 2 Model 3 Model 4 Model 1 Model 2 Model 3 Model 4

1 – – – – – – 20,4372 + + – – – – 19,8803 – – – + – + 20,5854 + + + – + – 19,9455 + + + + + + 20,0816 – + – + + – 20,4677 – – + – + – 20,3908 + – + – – + 19,9799 + – + + – – 20,083

10 + – – + + – 20,09011 – + – – + + 20,19012 + + – + – + 19,98813 – + + + – – 20,51614 – – + + + + 20,55715 + – – – + + 19,95116 – + + – – + 20,170

Cha

nge

pric

e

Ince

ntiv

e fo

r ear

ly re

new

al

Lat

e-se

ries

ince

ntiv

e

Mul

ti-ye

ar o

ffer

Com

pres

s se

ries

timin

g

Mor

e e-

mai

l effo

rts

Segment 1 Segment 2 Segment 3

Total Sample Size

Mosaic Offer & Strategy Test – analytical plan

17

Offer & Strategy Test – Execution: recipes 1 vs. 5

- 4 months

- 3 months

- 2 months

- 1 month

Expiration

+ 1 month

+ 2 months

+ 3 months

+ 4 months

+ 5 months

$ $

$ $

$ $

$ $

$ $

$ $

$ $

$ $

$ $

$ $

18

Mosaic Offer Test – Results

0.07%

0.11%

0.12%

0.17%

0.25%

0.35%

0.36%

0.65%

0.67%

+0.86%

+1.22%

+1.56%

-2.43%

+3.04%

-5.86%

0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0%

AEACF

ACACD

CDABADAFCF

E+: Compress contact stream (yes)F+: More e-mail (yes)

A+: Change price (yes)D-: Multi-year of fer (no)

B+: Incentive for early renewal (yes)C-: Late-series incentive (discount)

Effect (avg RR of test as % change vs. control)

Hearst Magazines Billing & Renewals Test: Renewal Rate|Effects| as % change versus the control (optimal = 6.7% lift)

Elements: sign of effect (optimal setting)

All 6 elements are significant(but no interactions)

C-, B+: Keep the later discount, but add and early-renewal incentive E+, F+: Add more contacts – more e-mails

and compressed timing D-: Skip the multi-year offer A+: Change price (but how?)

19

-50%

0%

50%

100%

150%

200%

250%

-20% Control +20%

% c

hang

e in

rev

enue

vs.

con

trol

A: Price

Effect (A): Price change(change in Revenue versus control)

Magazine #1

Magazine #2

Magazine #3

Mosaic Offer Test – Effect of Price…

To maximize revenue

• For magazine #1, increase price 20% (Revenue increases 19% but response is 9% lower)

• Increase price 20% for magazine #2(Revenue increases 23%

but response is 22% lower)

• For magazine #3, remainat the control price

20

-50%

0%

50%

100%

150%

200%

250%

-20% Control +20%

% c

hang

e in

rev

enue

vs.

con

trol

A: Price

Effect (A): Price change(change in Revenue by Segment)

#1 - Cancel risk

#1 - Loyal

#2 - Cancel risk

#2 - Loyal

#3 - Cancel risk

#3 - Loyal

… Depends upon segmentationDifferences bySegmentation Model

• “Loyal” customers(defined by a combinationof statistical models) Respond well

to higher prices

• Subscribers who are a“Cancel risk” are better served at the control or lower price…

Because… Need to consider response(not shown) along with revenue Renewal rate drops up to 26%

for this high-risk segment And we need to deliver promised rate base Plus renewals are lower cost

than new subscribers

21

Summary of Mosaic Tests

• Tested 17 marketing-mix elements at once• Quantified 10 significant elements

– Analyzing main effects, interactions, and “curvature” in the price effect

• Pinpointed differences by segment, title, and statistical model

• Tested in 9 months what would require more than 3½ years with A/B splits

22

Conclusions & Insights

• Benefits of multivariate testing for Hearst retention marketing­ Identified creative, offer, segmentation elements with most leverage

with some rolled out immediately, others fine-tuned via more tests

­ Low-leverage test elements removed from future test consideration

­ Additional tests to “fine-tune” high-leverage elements led to roll outs New control series that outperformed existing control series of the

last 7 years (response lifts as high as 15%) New model-based offer and segmentation strategy based on a

subscriber’s likelihood to renew or cancel

23

Risks & Recommendations• Risks of multivariate testing for Retention efforts

­ Testing more elements increases risk of negative impact­ Normally low risk test elements could be high risk when mixed with

other test elements to form counter-intuitive recipes needed for test­ Control creative may not accommodate variety of recipes required

for testing, which may force additional creative changes and response risk

• Recommendations on when to use multivariate testing ­ Statistically significant A/B test volumes can’t accommodate

all test elements­ There’s a lot to test and results are needed quickly­ The organization supports bold, high risk/reward testing

24

Accelerated Testingfor Retention and CRM Programs

Brian FichtelSenior Director, Customer RetentionHearst Magazines

Gordon BellPresident, LucidViewManaging Partner, Artestry

gbell @ lucidview.com