MARK2038 Data Base Marketing Strategies II Week 11 Instructor: Santo Ligotti Email:...

MARK2038 Data Base Marketing Strategies II

Week 11

Instructor: Santo Ligotti Email: [email protected]

Testing, Metrics, and Post Analysis

This week

Testing, metrics, and post analysisIn-class assignment #5Structure/content of final test (July 18th, 2006)

Learning Objectives:

You just learned: why testing of DBM programs is important;4 steps you can take to test DBM programs; how to analyze the effectiveness of direct response campaigns including response rate, ROI and cost per response.

Campaign Management Process

1. Planning List Budget Offer/call to action Fulfillment Creative format Messages and copy Response device Testing process Response tracking Financial success

measures

CampaignPlanning

List Compilation

Implementation

Measurement


2. List compilation Purchase response

lists/compiled lists Ensure any last-minute

field edits are complete Select list members Forward records to

agency/suppliers Flag records for inclusion

in CRM system

CampaignPlanning

List Compilation

Implementation

Measurement


CampaignPlanning

List Compilation

Implementation

Measurement

3. Implementation Campaign is activated Customer inquiries and orders

are acted upon Information is received from

selected media channels

4. Measurement Monitor the results of the

campaign for effectiveness Input recommendations to

direct marketing planning

Time to MarketMarketing campaigns require an average of 2.5 months to implement.

Reducing Time to Market

The longer the campaign lead time,

The less likely the message will be relevant to its audience…

… and the less likely it will be “highly effective.”

A partnership between Marketing and Analytics will maximize campaign results

Involve the data analytics team at the beginning of the campaign to establish key business objectives, pre-analysis, targeting and key metrics/tracking

Continually integrate the data analytics team’s tracking and key insights into future campaigns to maximize ROI of all marketing initiatives

Getting the right mix, requires internal partnerships

With increasing pressure from shareholders/analysts to continually improve financial results, marketers need to able to illustrate that their campaigns are delivering strong results

In order to ensure marketing dollars are maximized, data analytics needs to become a key partner in the ongoing measurement & tracking of campaigns

A number of marketer’s are still struggling to demonstrate that their campaigns deliver quantifiable results

The Business Challenge

So how do we as marketers achieve this?

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE

CREATEAPPROPRIATE MESSAGE

IDENTIFYPOTENTIALCUSTOMERACTIONS

TEST ANDLEARN

Data Analytics is key to CRM Process

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE



TEST ANDLEARN

Knowing Your Customer starts with Data Analytics

Analyze customer behaviour to determine key drivers of valueKnow recent key events and interaction with your company

Utilizing Data Analytics allows you to Identify Potential Customer Actions

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE



TEST ANDLEARN

Continuously target and tailor offerings based on testing and learning

Marketing and Data Analytics allows you to Create Appropriate Message

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE



TEST ANDLEARN

Explicitly manage the flow and sequence of marketing communications to each customer

Marketing Delivers the Message to the Customer

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE



TEST ANDLEARN

Create a dynamic and consistent messaging and response capability at all customer touch (communication) points

Data Analytics allows you to Listen to the customers response

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE



TEST ANDLEARN

Capture and remember relevant customer conversations

Data Analytics allows you to Track the Customer Responses and gain Insights

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE



TEST ANDLEARN

Customer responds to the message Key Learning’s are integrated into future programs by marketing

Establishing a Test & Learn Partnership between marketing & data analytics will maximize results

ACTION

KNOW THECUSTOMER

LISTEN

DELIVERMESSAGE



TEST ANDLEARN

Conduct sophisticated tests, share learning widely, and implement fast read and re-launch capability

The Concept of Testing

Why Test?Good economics: Use a sample to learn what works and what

doesn’t work before rolling to entire databaseContinuous Improvement Learn how to improve marketing programs to

ensure they’re the most effective

Testing Multiple Variables

Test all or some variablesWhy? Learning Loop: Generates constant feedback

on how to improve effectiveness of communications

Commonly tested variables: Lists Offers Creative execution Channel Content

Testing an Idea

Four Steps1 Plan Test

Define objectives Set up test and control groups

2 Execute Test3 Track Results4 Analyze Results

Response rate ROI Cost per response LTV

Example - Department Store

AssumptionsStore has a house credit card tied to customer database containing 400,000 men and women Store credit card allows capture of information about purchasesStore has new line of designer clothes for women, being promoted through print adsWould like to increase sales of new clothing lineDecide to test a direct mail program with a small group of women customers, before roll out to entire databaseOffer: If buy new suit by May 30, will receive a free piece of costume jewelry worth $20 by presenting this offer

Step 1: Plan Test

i) Define Marketing ObjectivesWhat are you trying to accomplish?Objectives should be measurable and time-bound.

Department Store Example: To increase sales to existing customers by 4% within 1

year. To achieve sales of new clothing line of $4.2 million. To increase LTV per customer from $80 to $125 over

next 12 months.

ii) Set up test and control groups

Test Group Control Group

Total Customers

Gets Offer Does NOT get offer

Step 1: Plan Test

Why use a Control Group?

• Allows you to measure the effect of the promotion versus not running it

• No offer or promotional piece sent to the control group

• Can be larger/smaller than test group

Set up test and control groupsQuery the database to determine how many women have credit cards in their name

Example - Department store200,000 women with department store credit card in their nameMust select 2 groups from this 200,000: Women who get the direct mail offer (Test) Women who do not get the DM offer (Control)

Step 1: Plan Test

Test & Control Groups: How large?

Cost considerations: make as small as possibleStatistical validity: make as large as possible

Rule of Thumb:Each group must be big enough so that you receive at least 500 responses from the promoted group

ExampleIf anticipate response rate of 2%Test group needs to be (500/2%) = 25,000

Example: Department StoreAnticipate response rate of 2.5%200K women in databaseTest group size = 500/.025 = 20,000

Test & Control Groups: How large?

Set up test and control groupsConstruct Test Group using ‘Nth’ method (per RFM)YOUR CONTROL WOULD BE THE SAME FOR THE ENTIRE MAILING UNIVERSE, REGARDLESS OF HOW MANY CELLS

Nth = Total customers in databaseTest Group Quantity

Example: Department store• Test group of 20K• Add another 20K for control group … total = 40K• Nth = 200,000/40,000 = 5• Select every 5th customer from master database

• That is, select customer record #5, #10, #15 ...

Step 1: Plan Test

Why use ‘Nth’ select?

Test and Control groups must be exact statistical replicas of the master databaseMust mirror the master database - will have the same percentage of people with similar characteristics: Same postal codeSame incomeSame # of childrenSame lifestyleSame purchase behaviour etc.

Step 2: Execute Test Execute Program among test group,

interacting normally with control group


200k Women customers

No Mailed Offer

Step 3: Track Results

Assign a source codeA “source code” is assigned to each test variable to facilitate measurement and analysisA source code is a series of letters or numbers used to identify a particular offer Rule: different source code for each new variable

Example: Women who got offer: OFFERMAY03 Women who did not get offer:

NOOFFERMAY03

Step 4: Analyze Results

What is the key learning?

15%

10%

0%

2%

4%

6%

8%

10%

12%

14%

16%

Response Rate


3,000responses

2,000responses

What is a response?

A response can be ...

Phoning a 1-800 number

Providing information (e.g. survey answers)

Entering a contest

Purchasing a product

Signing up for a service

Our example

Step 4: Analyze ResultsEvaluate success using a number of factors:

How did the program perform relative to objectives?Did the promotion come in on budget?

Metrics used to analyze performance:Response Rates Analysis (RR%)Cost per Response (CPR)Return on Investment (ROI)LTV

Response Rate Analysis

Calculate response rate for Test Group

Calculate response rate for control group

Step 2

Step 1

Calculate incremental lift between test and control groups

Step 3

First, calculate response rate for Test group

Department Store ExampleDirect mail offer: Get free piece of costume jewelry if buy suit by May 3020,000 mailed, 3,000 responded

Test RR% = Responder Quantity x 100=15% Test Quantity


Then calculate response rate for the Control group

Department Store Example20,000 in Control Group do not receive direct mail offer Still, 2,000 people respond to print advertising and buy a suit by May 30

Control RR% = Responder Quantity x 100 Control Quantity


Third, calculate % Lift between groups

% Lift = Test RR% – Control RR% x 100 Control RR%

EvaluationThe higher the lift, the betterPositive % Lift = Test performed better than ControlNegative % Lift = Control performed better than Test

Based on the Department Store example, what is the incremental lift percentage?


Cost per Response Analysis

Campaign Costs / Budget Include:Planning & Campaign Development Agency Costs (e.g. Fees, Creative

Development) List Development (e.g. data work)

Campaign Execution Printing, Laser/Lettershop, Postage

Response Costs The marketing cost associated with response

to a database marketing campaign BRC postage, data entry, offer fulfillment, call

centre

Cost per Response

Cost per response = Total cost of program

# responses

Department Store ExampleTotal program costs = $210,000 (includes campaign development, execution, response costs)Cost/response = $210,000/3,000 = $70

Evaluation: the lower the cost, the better

Return on Investment (ROI) Analysis

ROI = what you earn on a campaign relative to what you spent on a campaign

Evaluation: the higher, the better

Objective: To determine if you made money from your database marketing investment

Return on Investment Analysis

ROI = Revenue – Program Costs x 100 Program Costs

Department Store ExampleTotal program costs = $210,000 Sales revenue = $450/suit=(450*3000)3,000 responses to program

What is the ROI ?

Next step:Determine promotion effect on lifetime value Increased lifetime value, rather than immediate short-term payout, should be the real goal of database marketing Test effectiveness of alternative ways of

increasing LTV

Lifetime Value

Testing an Idea

Four Steps1 Plan Test

Define objectives Set up test and control groups

2 Execute Test3 Track Results4 Analyze Results

Response rate ROI Cost per response LTV

Metrics Example:CIBC: Direct Mail Creative

Execution Test

Example: CIBC Creative Test

• 3 different Direct Mail pieces created for launch of CIBC Adventura Gold Visa card

• Packages all the same except the outer envelope:» Cell A: High-end envelope & CIBC logo» Cell B: High-end envelope & Adventura logo» Cell C: High-end envelope & CIBC logo &

Aventura logo

Calculate the % lift, cost per response and ROI for each cellWhich envelope creative would you roll out to the entire database of customers?



Cell A Cell B Cell CCell Quantities Mail 50,000 50,000 50,000

Control 10,000 10,000 10,000

Responders Mail 5,000 2,500 7,500Control 500 200 1,000

Program Costs $100,000 $100,000 $100,000

Net Profit $500,000 $300,000 $500,000

Response Rate MailControlLift

Cost/Response

ROI

10,000

3,500


CELL A CELL B CELL CCell Quantities Mail 50,000 50,000 50,000

Control 10,000 10,000 10,000

Responders 5,000 2,500 7,500 Control 500

Program Costs 100,000.00$ 100,000.00$ 100,000.00$

Response Rate Mail 10% 5% 15%ControlLift 100% 0% 200%

Cost/Response 20.00$ 40.00$ 13.33$

5%7%

40%

10,000

3,500

28.57


ROI% = Revenue – Program Costs x 100Program Costs

Cell A Cell B Cell C% Lift vs. Control 100% 150% 50%

Net Profit $500,000 $300,000 $500,000

Program Costs $100,000 $100,000 $100,000

ROI

100% 40% 200%

Revenue


ROI = Revenue – Program Costs x 100Program Costs

Cell A Cell B Cell C% Lift vs. Control 100% 150% 50%

Net Profit $500,000 $300,000 $500,000

Program Costs $100,000 $100,000 $100,000

ROI 400% 200% 400%

100% 40% 200%

Revenue

Based on the results, which envelope creative would you roll out to all customers?


» Cell A: High-end Envelope + CIBC logo» Cell B: High-end envelope & Adventura logo» Cell C: High-end envelope & CIBC logo &

Adventura logo

In-class Exercise (Worth 10%)-Part 1

Read Luring ‘em back to school, Strategy Magazine, November 2003

Write 2 measurable, timebound objectives for the integrated marketing programs executed by CMC. What was the CMC strategy?What direct marketing tactics were used?How would you measure campaign success?

In-class Exercise-Part II:

Luring ‘em back to school Complete the following table comparing the differences between the direct mail and e-mail catalogue mailings. Which program appears to be more successful? Why?

DM EMcatalogues sent 60,000 78,000response rate 2.5% 1.5%total responses

mktg cost/catalogue $3.00 $0.25total catalogue mktg costsother mktg costs (agency fees etc.) $300,000 $300,000total mktg costs

avg profit /course $700 $700total profit

ROIcost/response

Statistical Significance

Statistical certainty is impossibleWe normally talk of level of confidence in statistical predictions In DM this is often 95% (19 out of 20 times) or 90% (18 out of 20 times) confidence - results will be repeated within an acceptable margin of errorThe confidence level set normally depends on financial risk

Where Are the Other 95% - the Direct Marketer’s Non-respondents

Research evidence suggest that it is all due to poor timing!!Not ready or unable to transact because: lack of funds don’t know how the product or service will perform domestic upheaval (e.g. moving house)

Is this the reason why repeat mailings and follow-ups are often successful?Also, is this the reason behind the possible discrepancy between test results and roll-up?

Selecting Response Channels

How do you want them to respond?The 3 main channels are:MailPhone InternetAdditional Channels include:

Mobile Devices

Response Channel Specific Metrics

Direct MailResponse Rate versus no mail groupCreative Tests-different letter versionsOffer Tests-different offer typesResponse Mechanisms (call/in-person)

TelemarketingResponse Rate versus no calll groupPercentage Right Party ConnectWrap code analysisCross and Up sellsCreative Testing-Scripts

InternetResponse Rate versus no contactView RateAbandon RateAccept RateClick Through RateRe-visit RateCreative Tests-different content pagesPush versus Pull tactics

Channel CombinationsResponse Rate versus single channel

Measurement

It’s not enough to count responses. Response does not indicate the level

of customer commitment. Measuring response doesn’t tell us

WHY consumers behave the way they do.

Response builds only limited knowledge of customer behaviour.

Beyond Response

What kind of people are responding?What other market segments are there?What offers trigger different groups to respond?How many ways can we present a message?Where are the overlaps in media used?What messages are appropriate for various media?

Performance Measurement

Historical data can be useful in evaluating the performance of similar marketing campaigns.

Performance MeasurementMEASURE OPERATIONALIZATION

Response rate Percentage of prospects contacted who replied

Number of inquiries Number of fulfillments

Number of qualified leads Number of leads who expressed interest that were converted into

sales or opportunities

New customers acquired Number of purchasers who had not purchased before

Customer lifetime value Net present value of customer over a specified period of time

Customer acquisition cost Total marketing costs divided by number of new customers

TEST

TEST

TEST

1. Products/Services

2. Media e.g.. Lists, print, Internet

3. The Offer

4. Formats/Layouts

5. Timing Schedules

Testing Variables

Common Experimental Designs

Split-run experimentCompare responses of campaign A to

campaign B using the same list (split in two)

Before-and-after experimentCompare the outcomes of campaign A

recipients to a control group that did not receive it.

Good Pre-test Design

A good experiment will measure the effect of ONE variable on another (response rate).

Compare, on a limited audience:1. (Offer A) vs. (Offer B) vs. (Offer C)2. (Creative A) vs. (Creative B)3. (Segment A) vs. (Segment B)

Bad Pre-test Design

Marketer attempts to:alter more than 1 variable per test

cell in the same experiment compare results in one medium to

anothertest different response channelssplit the list into test cells that are too

small (n<30 responses)

Next Week: Test Structure (25%)

Class Test: July 18th, 20062 hours

Final ExamResponsible for everything covered in class, including handoutsCovers Materials from Week 1-Week 10

StructureMultiple choice Short AnswerMetrics ProblemCase Study

MARK2038 Data Base Marketing Strategies II Week 11 Instructor: Santo Ligotti Email:...

Documents

Transcript of MARK2038 Data Base Marketing Strategies II Week 11 Instructor: Santo Ligotti Email:...