Perry D. Drake, Drake DirectPage 1. Perry D. Drake, Drake DirectPage 2 Chapter 14 Planning and...

Perry D. Drake, Drake Direct

Perry D. Drake, Drake Direct

Chapter 14

Planning and DesigningMarketing Tests

Objectives

The 5 rules of marketing test design Outside list test design considerations Determining appropriate test sample sizes Alternative testing approaches for smaller direct marketers Establishing strategic tests Testing best practices checklist

Planning Marketing Tests/The Five Rules of Test Design

When preparing to test new formats, copy alternatives, prices or offers to your house or outside list names, several rules must be followed to ensure your results will be readable, reliable and projectable.

Rule 1: For Mailers, Include the Control Package in the Test plan

In most cases, the promotional pieces associated with the test mailing will be delivered to the post office in one batch and the promotional pieces associated with the “bulk” mailing in a separate batch. Because quantities will be different, the post office will handle each differently. Therefore, you cannot compare response rates of the test to the bulk plan response rate.

Planning Marketing Tests/The Five Rules of Test Design (Continued)

Rule 2: Reverse Test Package Changes

As previously mentioned, when changing to a new promotional format one should always reverse test (re-test) the old promotional format to validate the lift in response you had forecasted. Without reverse tests you will not be able to determine if the cause for a campaign that is under forecast is due to the list selection or the promotion or both.

For example, ACME Direct tested the addition of a new “mini radio” premium to the control package last year to a sample of 10,000 names and received a +25% in response. Based on an assessment of a 95% confidence interval the product manager determined to roll-out with the new premium in the next major campaign.

Within the next major marketing campaign (of size 1,450,000 names), the product manager will conduct a reverse test of this change to the control package:

10,000 names to receive the old control package 1,440,000 names to receive the new package (old control + “mini radio” premium offer)

Should the response rate of the 1,440,000 names come in under forecast, the product manager will be able to determine if the cause is list related or if the cause is due to the mini radio not yielding the plus seen in the test. Without the reverse test, the product manager would not be able to separate out the two to determine what was the cause of the mailing coming in under forecast.


Rule 2: Reverse Test Package Changes (Cont.)


Rule 3: Test One Change at a Time

When applicable, test various changes to the control promotional package one at a time. You may be mislead otherwise by the testing results and be left in a situation where no action can be taken.

For example, suppose you are planning to test a new premium and new copy. Do not simply test the two changes combined into one test. Test each change separately. You will gain much more information from two separate test results than if they were combined into one test panel.


Rule 3: Test One Change at a Time (Cont.)

In this example, it may be that the copy change increases response while the new format decreases response. This will not be apparent if tested simultaneously.

Consider the following results to this test series:

What would our decision have been had we not tested each element separately in the above test series?

Test PanelResponse Rate Index to Control

Control Package 3.50% 100As Control with New Format and Copy Change 3.47% 99As Control with New Format 3.15% 90As Control with Copy Change 3.79% 108


Rule 4: Test for Only Meaningful Package Element InteractionsGenerally it is unnecessary to test every possible package element combination in your test plan. For example, the marketing manager may be interested in testing the following changes to the control package:

Price increase Addition of a premium Color change to outer envelop New format

Testing every possible combination of price, premium, outer envelop color and format yields a total of 16 test panels. Testing all 16 is called a “full factorial test design.”

Should a direct marketer always consider a full factorial test design, why or why not?

The only reason a marketer would test a full factorial test design is if it was truly believed interactions will occur between all four elements with respect to response.

In this example, the only possible interaction to be concerned with would be one between price and premium. In other words, if testing a higher price, perhaps the minus in response (due solely to pricing) would be less for the package with a premium versus the package without the premium.

NOTE: Other interactions may also be possible but one would need to know more details about the other panels in order to make a proper judgment call. For example, if the action device is a “scratch-off card” where everyone gets a $1 discount, it too may interact with price.


Rule 4: Test for Only Meaningful Package Element Interactions (Cont.)


Rule 4: Test for Only Meaningful Package Element Interactions (Cont.)

Assuming ACME Direct is only interested in assessing a possible interaction between price and premium, the test series would appear as shown below:

Based on this test series, how will ACME determine if the addition of a premium to the control package offset any or all of the negative effect that a price increase might have on response?

Test Panel Description1 Control test package

2 Price test package - as control package with $2 price increase

3 Premium test package - as control but with premium for order added

4 Format test package - as control but with a new format

5 Action device test package - as control but with an action device added

6 Price and premium test package - as test panel #3 with $2 price increase


Rule 5: Define the Universe for Testing Carefully

Careful consideration must be given to the names selected for testing.

If you want to take results from one test series and apply to another, you will need consistency in your test universe definitions.

You cannot take the results of a test conduct to one group of customers and assume you will receive the same response rate if used on a different group of customers.

Planning Marketing Tests/Outside List Test Design Considerations

Regarding outside list testing, you may be well advised to test for interactions between lists and the package elements (format, price, offer, etc.). In other words, certain lists, depending on how they were sourced, may react differently to different package elements.

For example, a list sourced via a sweepstakes promotion will in all likelihood respond better to a sweepstakes promotion than a non-sweepstakes promotion. A list sourced via a soft free-trial offer will in all likelihood respond better to a similar offer than to a hard “cash with order” offer.

If you do not see any reason to test for list by package element interactions, don’t. Use your experience to guide you in setting up the test.

Planning Marketing Tests/Outside List Test Design Considerations (Cont.)

For example, if you are testing 5 lists, 2 formats and 2 offers, a full factorial test design will require that you test 20 different combinations of list, format and offer.

If you have a fixed testing budget, you may not be able to test enough names within each cell to get meaningful results. In other words, the confidence intervals constructed around the response rate associated with each cell may be so wide that you cannot make a decision (as learned yesterday).

If this is the case, you are advised to reduce the number of test cells keeping only those interactions that make the most sense. A knowledgeable list broker should be able to help you. Based on his experience, your list broker should know what types of offers work best for which lists. Take advantage of his expertise.


For example, ACME Direct is going to test 5 lists, 2 formats and 2 offers all within a budget of 25,000 names total. A full factorial design will yield 20 test cells with 1,250 names per cell as shown below:

Format 1 Format 2List Offer 1 Offer 2 Offer 1 Offer 2List A 1,250 1,250 1,250 1,250List B 1,250 1,250 1,250 1,250List C 1,250 1,250 1,250 1,250List D 1,250 1,250 1,250 1,250List E 1,250 1,250 1,250 1,250


With 1,250 names per cell, the test results will have so much error variance associated with them that they will be difficult to interpret. The 95% confidence bounds around various response rates based on sample sizes of 1,250 names as shown below. As you will notice, the widths of each interval are so wide that you cannot make a sound decision.

Response Ratesof Test on 1,250 Names

95% Confidence BoundsAround the Test Response Rate

1.5% Response Rate 0.83% to 2.17%2.0% Response Rate 1.22% to 2.78%3.0% Response Rate 2.05% to 3.95%4.0% Response Rate 2.91% to 5.09%5.0% Response Rate 3.79% to 6.21%6.0% Response Rate 4.68% to 7.32%


If increasing the sample sizes of the test cells is not an option, ACME Direct will need to eliminate any cells that are not necessary. For example, if there is no reason to believe “list by package element” interactions will exist, ACME Direct will not test a full factorial design. Instead, they will test the format on one list only and the offer on one list only. This will allow ACME Direct to test more names per cell and increase the precision of each cell’s response rate while staying within their testing budget (see below).


If ACME Direct sees no reason to believe there will be an interaction between format and offer, they can further reduce the number of test cells by one more as shown below.


Based on this final test design, how will ACME Direct determine the winning format?

How will ACME Direct determine the winning offer?

How will ACME Direct forecast the response rate for List C if promoted with

format 1 & offer 2?

Planning Marketing Tests/Sample Size Determination

Prior to conducting your marketing tests, the most important part of the planning process is to determine the appropriate sample sizes for your test panels.

Without adequate sample sizes, the time and money spent creating and developing your marketing tests will be wasted.

This is true for any type of test discussed here today.

Without proper sample sizes your test results will have variance so great that the results will be unreliable.

As such, odds will be the test results will not resemble what you can expect should you decide to roll-out with the new test “promotion” to a larger audience.

Planning Marketing Tests/Sample Size Determination (Continued)

When planning your testing program, you will:

First determine the appropriate sample sizes Once results of the tests are final, you will analyze them by means of

confidence intervals or hypothesis testing as previously discussed.

And, if you properly planned the sample sizes, you will be able to feel confident in the results of your marketing tests.

Sample Size Estimation (Continued)

Two formulas will be reviewed:

Sample size determination when concerned with the accuracy of a single test response rate. Sample size determination when concerned with accurately measuring the difference between two test panels.

Sample Size Estimation/A Single Test Response Rate (continued)

To calculate the sample size required when concerned with the reliability of a sample response rate, the following information is required:

The maximum allowable error variance E An estimate of the expected response rate p The desired confidence level c


To determine the value of E, you need to answer the following question:

After your test results are final and you build a confidence interval around the sample response rate, what is the maximum width you are willing

to accept for the confidence interval?

The answer to this question is the value of E. For example, you need the test result to be within plus or minus 0.25% of the true response rate you can expect in roll-out. In this case E equals .0025.


You will also need an estimate of the expected test response rate p. Base it on past experience. It does not have to be exact but make sure you error on the side closest to 50%. That is:

If you expect the response rate to be 5.00% but no greater than 6.00%, use 6.00% since it is closest to 50%. If you expect the response rate to be 75% but no less than 70%, use 70% since it is closest to 50%.

A confidence interval will be at it’s widest for a response rate of 50% (all else constant).


The formula for deriving the sample size n is:

n = ((z2)( p)(1- p))/E2

Where p is the estimated response rate

E is the maximum error you are willing to accept

z equals 1.645, 1.96 and 2.575 for a 90%, 95% and 99% confidence level

NOTE: The sample size formula is derived from the formula for a confidence interval for a single test response rate previously discussed.


Consider the following example:

The marketing manger at ACME Direct is prepared to test a new source of names from a list broker to determine the list’s viability as a source of new names for their home products catalogue. The marketing manager believes the response rate for this list will be no greater than 4.75% (the maximum response rate conceivable by the marketing manager for this list). If she requires the test response rate to be within 0.25% (or .0025) of the actual response rate to expect in roll-out with 90% confidence, how many names should she test?


The marketing manager will calculate the required sample size as follows:

n = ((z2)( p)(1- p))/E2

= ((1.6452)(.0475)(1- .0475))/(.00252)

= ((2.706025)(.0475)(.9525))/(.0000062)

= (.1224306)/(.0000062)

= 19,747


Therefore, if the marketing manager tests 19,747 names she will be guaranteed, with 90% confidence, the response rate obtained from the test will be within 0.25% of the true response rate she can expect in roll-out.

Basically what the marketing manager has done is, with sample size planning, she has controlled what the resulting confidence interval will look like once the test results for this new list are final.


Question: If the marketing manager decides to increase the confidence level of her sample response rate from 90% to 95%, do you think the number of names she must test will go up or down?

For a 95% confidence level, the value of z = 1.96:

n = ((z2)( p)(1- p))/E2

= ((1.962)(.0475)(1- .0475))/(.00252)

= ((3.8416)(.0475)(.9525))/(.0000062)

= (.1738083)/(.0000062)

= 28,034


Question: If the marketing manager decides to keep the confidence level at 90% but needs more precision in her test estimate and therefore reduces the allowable error from .25% to .20%, do you think the number of names she must test will go up or down?

n = ((z2)( p)(1- p))/E2

= ((1.6452)(.0475)(1- .0475))/(.00202)

= ((2.706025)(.0475)(.9525))/(.000004)

= (.1224306)/(.000004)

= 30,608

Determining the Tolerable Error Variance

To determine the tolerable error variance (E), the Marketing Manager must take into consideration how close the expected response rate for the new list is to the “rent/no-rent” response rate level. The closer the expected response rate of the new list is to this cutoff level, the more important it will be to have a test estimate with little error variance.

Assume the Marketing Manager in our example will roll-out with a new prospect list as long as it exceeds 4.50% (which relates to a break even situation). If the marketing manager truly expects the list test to yield at least a 4.75% response rate, then for decision making purposes she only needs to test enough names to ensure that her resulting confidence interval is no wider than ± 0.25% (derived by taking 4.75% minus 4.50%).


Determining the Tolerable Error Variance (Cont.)All that is needed for decision making purposes (rent the names or do not rent the names) is to be certain that the response rate of the new list in roll-out will not be below 4.50%.

Testing more names to yield less variance would be a waste of testing dollars. It is that simple!


Setting the Confidence Level

To determine the appropriate confidence level, the marketing manager will go through the same steps outline yesterday for determining the appropriate confidence level for confidence intervals and hypothesis tests.

For example, if we assume the new prospect list costs $10 less per 1,000 names rented compared to other prospect lists, a 90% confidence level seems appropriate. A 99% or 95% confidence level are not warranted as the impact of misreading the test here is less risky than doing so for a test that costs more or even the same.



Luckily, you do not need to use the complicated formula previously given to calculate the sample size for a single test.

With the help of The Plan-alyzer, a software package created by Drake Direct, you can easily determine the appropriate sample size for your test.

Sample Size Estimation/The Difference Between Two Test Response Rates

When interest revolves around measuring the difference in response rates between two test panels in order to be able to accurately determine if one test has beaten another, you will employ the formula provided in this section to determine the required sample sizes.

This is one of the most commonly used formulas when designing format, creative concept, pricing, and offer tests while needing to compare the results against your control offer.

Sample Size Estimation/The Difference Between Two Test Response Rates (continued)

To calculate the sample sizes required when concerned with the reliability of the difference in two test response rates, the following information is required:

An estimate of the expected response rate p1 for one of the two samples

The minimum difference d to detect as significant The desired confidence level c

You will base the estimate for one of the test panels p1 on past experience. In all likelihood, one of the two test panels for comparison will be the control package for which you should have historical response rate figures. You do not need to be exact but you should try to be as close as possible.

When estimating the response rate for one of the two test panels, as was the case when determining sample sizes for a single test response rate, error on the side closest to 50%.


Also required is the minimum difference d to detect as significant. This value represents the difference between the two test response rates (p1 and p2 ) you will want to detect as significant between the two test response rates. It should represent a critical difference for decision making purposes.


For example, a control package is expected to yield a 4.00% response rate. The addition of a four color pamphlet to the control package is being tested and at least 5 additional responses per 1,000 names promoted are needed to break-even.

Therefore the minimum difference of interest is 0.50% or .0050.

In other words, you are doing this “planning” because it is important for you not to be in a situation (after the test is complete) where you end up obtaining, for example, a 3.98% response rate for the control and a 4.48% response rate for the test and your resulting confidence interval contains zero simply because you have too much sampling error due to small sample sizes. If you observe a 0.50% difference in response you want to be able to say “yes the test won.”


The formula for deriving the sample sizes n1 and n2 is:

n1= n2= [( z2)] [( p1)(1 - p1) + ( p2)(1 - p2)] / [( d2)]

Where p1 is the estimated response rate for the control test panel

p2 is the estimated response rate for the second test panel

d is the minimum difference to be detected and must equal the difference between p1 and p2

z equals 1.645, 1.96 and 2.575 for a 90%, 95% and 99% confidence level

NOTE: The sample size formula is derived from the formula for a confidence interval for a difference between two test response rates previously discussed.


Consider the following example:

The marketing director at ACME Direct is testing the addition of a new “what’s included with your subscription benefits pamphlet” to the current control format. Assume the control format is known to yield a response rate of 3.40%. Also assume the marketing director determined that 4 additional responses per 1,000 names mailed are required to cover the additional costs of the pamphlet. Therefore, the marketing director will require the test to yield at least a 3.80% (.034 + .004) response rate in order to cover the incremental costs associated with this test.

What sample sizes should the control and test panels be to ensure the marketing director will be in a good position to make a decision should the observed difference in test response rates yields the required 0.40% difference?


What do we know so far:

p1 = .034

the response rate of the known control.

d = .004the minimum difference of concern for the marketing director. Should the test and control panels reveal this difference, the marketing director wishes to be in a position to ensure he can read the difference as significant.

p2 = .038

the response rate of the control with the required increase of 4 additional responses per thousand or an increase of .004.

c = ?

what should the marketing director use as the confidence level?


To set the confidence level, the marketing director will go through the same process as was done when calculating confidence intervals:

The new package is only moderately more expensive than the control Therefore, the risk to the business should the test tell us we can roll-out with the new action device when in fact we should not is only moderate As such, a 95% confidence level seem appropriate.

Note: Had the new package been the same cost or even less than the control, the risk of making an incorrect decision would be less and as such a confidence level as low as 90% could be used. If the new package had been significantly more expensive than the control, the risk of making an incorrect decision would be much greater and as such might warrant a confidence level as high as 99%.



The marketing manager will calculate the required sample sizes for both the control and test panels as follows:

n1= n2= [( z2)] [( p1)(1 - p1) + ( p2)(1 - p2)] / [( d2)]

= [( 1.962)] [( .034)(1 - .034) + ( .038)(1 - .038)] / [( .0042)]

= [( 3.8416)] [( .034)(.966) + ( .038)(.962)] / [( .000016)] = [( 3.8416)] [( .032844) + ( .036556)] / [( .000016)] = [( 3.8416)] [( .0694)] / [( .000016)] = [( .266607)] / [( .000016)] = 16,663


In order to guarantee, with 95% confidence, the marketing director will be able to conclude the two response rates 3.40% and 3.80% as significantly different from one another (should the results of the test and control panel yield these response rates), he will be required to test 16,663 names per panel.


Question: What is the marketing manager decided (due to his ultra conservative nature) it was important to detect an even smaller difference than was really required, say .002 instead of .004. How would this decision impact the required sample sizes for the test and control panels?

n1= n2= [( z2)] [( p1)(1 - p1) + ( p2)(1 - p2)] / [( d2)]

= [( 1.962)] [( .034)(1 - .034) + ( .038)(1 - .038)] / [( .0022)]

= [( 3.8416)] [( .034)(.966) + ( .038)(.962)] / [( .000004)] = [( 3.8416)] [( .032844) + ( .036556)] / [( .000004)] = [( 3.8416)] [( .0694)] / [( .000004)] = [( .266607)] / [( .000004)] = 66,652


There is clearly a trade-off. The smaller the difference to detect as significant or the higher the level of confidence, the larger the required sample sizes will be. It depends on the importance of the tests and how accurate the rest results need to be.

The rule of thumb is to only test the quantity of names required to make your decision. If all you need to detect (from a marketing point of view) as a significant difference is .004 then it is a waste of testing dollars to establish your tests so you can detect even a smaller difference. It is not necessary.

Luckily, you do not need to use the complicated formula previously given tocalculate the sample sizes when interested in detecting a significant difference between two tests.

With the help of The Plan-alyzer, a software package created by Drake Direct, you can easily determine the appropriate

sample sizes required for both test panels.


Regardless of the amount of time spent planning a test, the results of what you might achieve in roll-out are only valid if nothing outside of your control occurs from the time of the test to roll-out which may impact your test universe in some way.

Various scenarios which could negatively impact the likelihood that the true response rate obtained in roll-out will not fall with the bound of your confidence interval include:

the timing of the test vs. roll-out (e.g. test was conducted in May with a roll-out the following January) a major competitor emerged on the market after the test

Sample Size Estimation/Facts

Scenarios continued include:

increased competition in the mail box from the time of the test bad press regarding direct marketing practices since the time of the test and before roll-out a major natural disaster in a specific region of the country since the time of the test and before roll-out changes in the promotional package or offer prior roll-out economic recession or boom

Sample Size Estimation/Facts

Unfortunately we cannot always afford to test as many names as is required statistically. That is reality.

Many smaller to medium sized direct marketer test the majority of their new format, creative and pricing test panels to 5,000-10,000 names.

In most all instances 5,000 names are simply not enough to base a final decision on prior roll-out. And, in many instances neither is 10,000.

Assessing Test Results with Small Quantities/Format and Creative Tests

Larger quantities should be considered for:

Tests that have back end implications such as pricing tests. For such tests, your risk to the business, should you misread the test results, could be quite devastating.

Tests that were expensive to create (not development costs, but rather creative costs) relative to others. Your risk is much greater here if you misread such a test since you have invested more money into it’s creation.

Quite expensive formats (on a per piece basis) where you need to cover the incremental costs via additional responses in order to break-even versus the control. Determine the minimum difference to detect and then

calculate the required sample size. For such tests, let this be your guide.

Assessing Test Results with Small Quantities/Format and Creative Tests

For outside list tests, many direct marketers employ 5,000 name samples.

Can one still gauge the results with confidence based on such small sample sizes?

Certainly, but more careful analysis is needed for the assessment of such tests. And, in most all cases, test panels that do look good will need to be confirmed tested based on a larger sample so as not to put the business at risk.

Assessing Test Results with Small Quantities/Outside List Tests

In general, do not base winners solely on statistical significance when dealing with small sample sizes such as 5,000.

When you cannot test enough names to yield reasonable sampling error in the list test or to detect a meaningful difference between your control and new format test, do not base your decision on significance alone. Doing so will cause you to overlook potential winners.

In such cases, assessment of the lower and upper bounds of the confidence interval, incorporating marketing cost and revenue figures, is a must.


Steps to assessing the results in such cases are: If the P&L of the test does not look good (or come close to looking good) based on the upper bound in response at the 90% level, you are advised not to roll-out nor re-test. In other words, you have used the most liberal confidence level (90%) along with the upper bound on response and the P&L still does not comfortably meet your requirements. Toss it! Use the names to test a more promising format.

If the P&L of the test looks good at both the 95% and 99% levels, taking into account the worse case scenario, then by all means roll-out and do not re-test. In other words, you have used the most conservative confidence level (99%) along with the lower bound on response and the P&L is meeting (or coming very close to meeting) your requirements. Go for it! You have a definite winner on your hands.

If neither of these conditions are true, then you are advised to re-test to a larger sample of size 10,000 or greater. In other words, you were not able to definitively confirm the test to be a winner or a loser. Re-test to a larger sample to ensure a decision can be made next time around.


Strategic Testing

Besides product, list, promotion and offer tests, there are many other types of tests you may want to consider. They include but are not limited to:

Streaming Tests Intensity Tests Breakpoint Tests Seasonality Tests

Strategic Testing (continued)

Streaming Tests

These tests aim to determine the correct sequence of product/offers to be sent a specific universe within a given period of time.

For example, a streaming test may look as follows:

Test Time 1 Time 2 Time 3Test #1 Prod A Prod B Prod CTest #2 Prod B Prod A Prod CTest #3 Prod C Prod A Prod B


Intensity TestsIf a direct marketer begins to experience a significant reduction in response rates across the board, he or she may want to monitor promotional intensity. These tests are specifically designed to find out how many products/offers should be sent to a specific market, during a given period of time. They are designed using “mail” and “mark but do not mail” customer groups.

For example, a intensity test may look as follows:

Test Time 1 Time 2 Time 3Test #1 Prod A None Prod CTest #2 Prod A Prod B Prod C


Breakpoint Tests

You conduct these tests to find out the impact to a customer if they receive two different product/offers on the same day, rather than sequenced over a specific interval.

For example, a breakpoint test may look as follows:

Test Time 1 Time 2Test #1 Prod A + B NoneTest #2 Prod A Prod B


Seasonality Tests

Designed to learn whether one time of the year is inherently stronger for a given product offer than some other time.

For example, a seasonality test may look as follows:

Test Spring Fall Test #1 Prod A NoneTest #2 None Prod A

Things to Consider in the Advent of Unusual Test Results

From time to time we are faced with a test series for which some or all of the test results are called into question.

A check list of things to investigate include: Were the test samples representative of the universe of concern? Yes No Was there another mailing on top of your mailing? Yes No Was material properly created and personalized? Yes No Did the lettershop drop the mail according to your schedule? Yes No Did each test receive the appropriate handling by fulfillment? Yes No

Check list when establishing and setting up marketing tests: Universe clearly defined and understood by programming Universe for testing was appropriately chosen Appropriate sample sizes Special instructions sent to fulfillment and customer service Control included in the test plan Reverse tests established if changing to a new control Element testing incorporates only one change and not many Tests include any likely interaction panels

Testing Best Practices Checklist

Check list when reading marketing test results: Ensured that all test results look as expected and nothing unusual to be

investigated Established appropriate confidence levels for testing by panel Ran profit calculations using the worse and best case response rate scenarios

based on the confidence intervals If comparing tests across different test series, applied proper bridging

methodology

Testing Best Practices Checklist (Continued)

The three most common testing mistakes made by direct marketers today: Not enough names tested to yield reliable results.

– Lost revenue presents itself when you ignore significance and go with the leaning plus only to find that it did not hold up in roll-out

– Lost opportunity may presents itself when you assume a test lost against the control based on the leaning minus only.

You do not up the quantities on your most expensively created packages.

– There is more risk in misreading such test results

– Test more names to safeguard against your investment

You did not reverse test changes to your current bulk package

– if the bulk came in under forecast you will not know which element change may have been the cause or if it was list selection.

Three Most Common Testing Mistakes

Perry D. Drake, Drake DirectPage 1. Perry D. Drake, Drake DirectPage 2 Chapter 14 Planning and...

Documents

Transcript of Perry D. Drake, Drake DirectPage 1. Perry D. Drake, Drake DirectPage 2 Chapter 14 Planning and...