Experimental Design: Part II

MAR 6648: Marketing ResearchFebruary 8, 2010

Overview

• Let’s review experiments!• What can experiments do that other

techniques can’t?• How do experiments get implemented in

marketing?

An experiment

An experiment?

• The owner of two McDonalds franchises here in Gainesville wants to see if transactions run more quickly if he uses both drive-thru windows or only one. He picks one restaurant to use both windows at all times for a month, and the other he has closed at all times for a month. He finds that the drive-thru that uses both windows has notably faster service times.

An experiment?

An experiment?

A Case Study in Causal Investigation

• Broken Windows Theory (Kelling & Wilson, 1982)– “If a window in a building is broken and is left

unrepaired, all the rest of the windows will soon be broken”• Implication: People see the building as the sort of place

where it is OK to break windows• Implication #2: If you fix the window, people see this as

a place where people behave nicely• Broad sweeping implication: If you eliminate petty

crimes, people will stop perpetrating major crimes

A Case Study in Causal Investigation

• Primary concern: The theory is only interesting if it is causal. Consider the alternatives:– 1. Reverse causation: When you reduce major crimes,

people are also less likely to commit minor crimes (e.g., if I stop people from stealing cars, I also stop them from breaking windows)

– 2. Third variable: If I remove all the criminals from circulation, there will be fewer broken windows AND fewer stolen cars

• Now, given the methodologies we have thus far, how would you try to evaluate the theory?

A Case Study in Causal Investigation

• Possibilities:– Qualitative Data– Survey Design– Observational Research– Archival Research/Data Mining

• Archival Analysis: • Crime in New York City

Corman and Mocan, 2000

Primary Conclusion:

“… an increase in the size of the police force generates a decrease in robberies and burglaries.”

A Case Study in Causal Investigation

• Possibilities:– Qualitative Data– Survey Design– Observational Research– Archival Research/Data Mining– Experimental Design

1. Experimentation is the conscious manipulation of one or more variables by the experimenter in such a way that its effect on one or more variables can be measured.

2. The variable being manipulated is called the independent variable (a.k.a. cause).3. The variable being measured is called the dependent variable (a.k.a. effect).4. Elimination of other possible causal factors: i.e., the research design should rule out the

other factors (exogenous variables) as potentially causal ones. 5. This is typically done through random assignment to condition

Experimental Design: Example 1

• Independent Variable– “Policing Disorder” vs. Control

• Dependent Variable– Service calls for five serious crimes

Control

Policing Disorder

Braga and Bond, 2008

Results

But what limitations do we see in this?

Control

Disorderly Setting

Experimenters attached a paper flyer to each bicycle and recorded whether or not people dropped the flyer or took it with them.

Experimental Design

Keizer, Lindenberg, & Steg, 2008

Results

Percentage LitteringControl 33%

Disorderly Setting 69%

Independent Variable Dependent VariableExpt. 1 Graffiti Littering

Further Experiments

Independent Variable Dependent VariableExpt. 1 Graffiti LitteringExpt. 2 Illegally Parked Bicycles TrespassingExpt. 3 Unreturned Shopping Carts LitteringExpt. 4 Setting off Fireworks LitteringExpt. 5 Graffiti StealingExpt. 6 Litter Stealing

But what are the shortcomings of this design?

Key Points

• Experimentation is necessary to infer causality • A poorly designed experiment will not allow you to

infer causality • A good experiment should achieve:– High internal validity through appropriate choice of

experimental design.– High external validity by keeping the experimental setting

as close to the real marketing environment as possible.– There is a trade-off here…

• A good control group is often a key requirement of a good experimental design.

How should experimentation be used in marketing?

?New Customer Service Program Customer Satisfaction

Focus is on detecting causal relationships between variables

Causal Research in Marketing

• Many examples of the need for causal effects in Marketing

• 4P’s alone...• How do we actually identify causal effects?

Key tool: Experiments

How to Run Experiments?

• How do we actually run experiments?• In the sciences there is a long history of lab

experiments • In a lab it is relatively easy to control external

conditions that might affect the validity of the experiment

• What is the situation in Marketing?

Experiments in Marketing

Usually take the form of a comparison between a test and (at least one) control group

Experiments are frequently run as field experiments

Example: Price Experiment

• A catalog retailer selling women’s apparel• Conducted price experiment to estimate

demand curves• One control group and 4 test groups • Each group consisted of 15,000 randomly

sampled customers

Version 1 Version 2 Version 3 Version 4 Version 5

The Concept of Validity• Internal Validity:– Refers to the ability of the experiment to unambiguously

show a cause and effect relationship, i.e., to what extent can we attribute the effect that was observed to the experimental variable and not other factors?

• External Validity:– Refers to the extent to which the results of the experiment

can be generalized from the experimental environment to the environment of the decision maker; i.e., the real world

• There is a trade-off between internal and external validity, from a managerial perspective.

Test Markets• Many uses:– Controlled introduction of a new product– Change of pricing strategy– Change of product design

• Choose representative markets• Often as long as one year • Expensive – but highly informative• Three Types:– Simulated Test Markets– Controlled Test Marketing– Sell-In test Marketing

Simulated Test Marketing (STM)• (a.k.a Laboratory Test Markets)• Simulates an actual test market to estimate

– initial purchase rates– ultimate repeat purchase rates

• Advantages: Compared to true test markets…– Fast (3 months)– Relatively cheap ($250,000)– Flexible– Impressive accuracy rates

• Limitations– assumes preference data and purchase/repurchase decisions are

valid predictors of what would actually happen in the market place– convenience sample– attrition

• Despite this, laboratory test markets are one of the biggest success stories in market research

Example: Typical STM ProcedureStep 1: Recruit

Qualified Shoppers

Step 2: Background Questions

Familiarity, Preferences, Usage

Step 3: Screening of Ads

Ad for Target Product + Others

Step 4: Simulated Shopping

• Respondents given money

• Invited into mock/real store

• Where they may buy any item

Step 5: Debriefing

• Choices recorded (TRIAL RATE)

• Reasons for (non) purchase

• Non buyers given free sample

Step 6: Reinterview

• Contacted after few weeks

• Product attitude, usage, satisfaction

• Repurchase (REPURCHASE RATE)

• Intended & Actual

Controlled Test Marketing• Cities for distribution is prearranged• Purchases of a panel of consumers are monitored

through scanner data• Example– IRI BehaviorScan– 3,000 households in 7 cities– ID card presented to supermarket– In-store conditions - price, promotion, displays -

controlled and monitored– Device on TV allows channel selection to be

monitored and ads to be substituted

Sell-In Test Marketing

• Cities where product is sold just as it would be in a national launch

• Must gain distribution space• Issues– selecting the test cities– implementing and controlling the test– timing– evaluative measures– costs

Six U.S. Common Test Markets

Downsides to Test Markets?

Other Downsides?

Key Points

• Experimentation is very useful in marketing to determine true causal effects of marketing decisions

• Test markets are the primary example, although smaller scale experiments are common– Test markets represent a trade off between

internal and external validity– They have other downsides, too

Summary

• Experiments are awesome, right?• They can demonstrate causality, which makes

them useful tools for marketers• When you design an experiment, keep in mind

the balance you are looking for with regard to internal and external validity– Test markets give you a lot of external validity,

which can be good and bad