*Lecture 2: Experimental DesignDr. Ghazala AzmatMonday 19th January 2015

ECN374: Behavioural Economics

*

*Todays LectureFirst Principles for Experimental Design

Experimental Design

Phases of an Experiment

Important Nuisances

*First Principles

*Creating an Economy in the LabA microeconomics system is a complete, self-contained economyIt consists of a set of agents and the institutions through which they interactE.g., buyers and sellers operating in a particular marketThis general description applies equally well to:Theoretical modelsNaturally occurring economiesArtificial economies in the lab

*The AgentThe agents are the individual participants in the economyEach agent has his or her own characteristicsType (e.g., buyer)Endowment of resources (e.g., time, goods, cash)Information (e.g., regarding others endowment)Technology (e.g., production function)Preferences over outcome

*The InstitutionThe institution specifies which interactions are allowed among the agentsThe institution consists of:a message space (or choice set) for each agent typeE.g., a range of allowable bid prices in an auction or a simple matrix gameAn outcome function, given the agents choiceE.g., the winner and price at auction, or payoff matrix for a simple game

*PredictionsTypically we assume optimization and equilibriumOften we use a theoretical structure to give a unique prediction of what each agent will do and what the overall outcome will beWe advance economic knowledge by testing the predictions against observations in the field We refine the model description (of agent characteristics or the institution) when significant discrepancies are found

*ReplicabilityWith proper implementation we achieve replicabilityThis is the hallmark of controlled laboratory work (in any science)Replicability means that any competent investigator can produce functionally similar data

*PracticalitySo how do we do it?To implement the experiment you must first recruit human subjects to fill the role of agentsIf appropriate, you construct computerized agents (robots)To control the institution, you give the agents the desired message space and enforce the outcome function.We will discuss this in more detail later.

*Induced Value TheoryIVD is the key methodological innovation for experimental economicsIt is based on the idea that proper use of a reward medium allows the experimenter to induce pre-specified characteristics in the subjectsSuch that their innate characteristics become irrelevantThree conditions are sufficient:

MonotonicitySalienceDominance

*1. MonotonicityIn a suitable reward medium, more is always better (or, alternatively, less is always better)E.g., we can safely assume that every human subject prefersMore cash earning to lessMore grade points to lessPrefers less tedious work to moreLater we will discuss the practical advantages of using cash payments to other reward mediums

*2. SalienceFor each agent, the reward component corresponds to a clear outcome function, e.g., utility or profitAnd the subject understands thisSalience connects the outcome in the microeconomic system to a reward medium that the subject cares aboutThe connection cannot work properly unless the subjects are fully aware of it

*3. DominanceThe reward increments are much more important than other components of subjects utility that are affected by the experimentSubjects may have rivalrous or altruistic motives towards other subjects or towards the experimenterBut these motives cannot upset dominance if the subjects do not know how their own actions affect others payoffs or the experimenters goal

*Experimental Design

*Experimental DesignYour purpose determines the appropriate design for the experimentIt defines focus variables: those whose effects you want to understandBut nuisance variables may also have an effect and you need to account for themThe whole point of experimental design is to deal appropriately with both kinds of variablesThere are two basic devices to separate the effects of the focus from nuisance variables: Control and RandomizationThese help achieve independence among the variables affecting the outcomes

*Control IYou, the experimenter, can freely choose the values of many variables, e.g., two different auction formats and the sorts of cost to induce on the sellerThe deliberate choice, or control, of key variables is what distinguishes experimental data from happenstance dataYou have two options for controlling a variable:Hold it constant keeping it at the same level throughout the experimentVary it between two or more levels this is called a treatment variable

*Control IIAs you hold more variables constant, the experiment becomes simpler but you can also learn about direct effects and interactionsIt sometimes takes serious thinking about and careful work through the theory before you can decide on the right control variables

*Independence Treatment variables are independent if knowing the value of one variable does not give any information about the level of the other variablesWe want to vary the treatment variable independently because:If two variables are dependent then their effects are harder to separateAn experiment could helpHow do we make control variables independent?This is easy for variables you can OBSERVE or are aware about, as you can control for them as constants

*ButWe are not out of the woods yet!

Other factors may affect independence

May have an effect on the subjects behaviour

We need randomization

*Randomization ISome potential nuisance variables are not controllable, so independence seems problematicThe lack of control is especially serious when the nuisance variable is not even observable and may interact with the focus variableE.g., some subjects are more altruistic than others and you cant observe itWhat happens if you assign the more altruistic subjects to the first set of instruction conditions?Your conclusions will be wrong!!

*Randomization IIThere is a simple solutionIf you assign the conditions in a random order, your treatments will become independent of all unobserved variables (observable or not)E.g., do not assign the first half of the subjects to arrive to the encouraging instructions; they may be more (less) altruistic Instead use a random devise to choose which instructions to use for each subjectRandomization ensures independence as the number of subjects increases

*Phases of an Experiment

*Phases of an Economic ExperimentResearch QuestionExperimental DesignRecruitment of SubjectsExplanation of the RulesPlayPaymentsAnalysisWriting up

*1. Research QuestionYou should always start with a research question.Think clearly what you want to study.Research question always before observing data. New questions? New designs?Replications are valid and needed.Ask yourself which methodology is best to study your question at hand: Theory? Empirical Data? Experiments?If experiment, why?

*Why experiments?Control/Precise measurement.True exogeneity such that causes may be isolated. This can be difficult with happenstance data because explanatory variables may be highly correlated. Relatively cheap Legal issuesE.g., Welfare-to-work experiments would probably not be feasible in some countries because of equal-treatment acts.

**

*2. Experimental DesignThink of the best experimental design for your research question.Inspiration from others is valid, but think twice before setting on a design.Typically, you want to conduct an experiment to find out about the effect of just a few variables. Focus variables? Nuisance variables?What is focus and what is nuisance may depend on the purpose of your study.

**

*Good DesignA good experimental design helps you:

to sharpen the effects of your focus variable.

to disentangle the effects of different focus variables.

to minimize the effects of nuisance variables.

**

*Direct control: Constants and TreatmentsIn a lab you can directly control for many variables.

Controlling these variables makes the difference between experimental and happenstance data.

You can keep these variables constant or vary them.

The ones you vary are your treatment variables.

The more treatment variables you have, the more you might learn about these variables but at the same time your experiment will be more expensive.

**

*Validity of laboratory dataTwo issues: Internal validity: Do the data permit correct causal inferences?

Thats a matter of good design.

External validity: Can we generalize from lab to field?

The experimentalists traditional answer to this is: We should.Thats the idea of parallelism.

**

*Avoid confounded treatmentsSuppose you are interested in the revenue of two auction formats with different numbers of bidders.Two treatments:

1) Auction type I with 4 bidders. 2) Auction type II with 10 bidders.Whats wrong here?Effects will be confounded. You only learn about the interaction; nothing about the two variables on their own.Never change two variables at the same time!

Vary all treatment variables independently.

**

*Within-Subject and Between-Subject DesignIn a within-subject design, a single subject is observed in different treatmentsThe subject serves as her own control groupIn a between-subject design the norm in experimental economics different subjects are tested in treatments A and BStatistical variation across the subjects then muddies the water of what is observed by comparing A and BWithin-subject designs are statistically powerful than between-subject designs because they automatically control for individual differencesThese are often a large source of variation

*3. Recruitment of subjectsRecruit without false promises

Subject database is extremely precious resource.

Record subjects history!!! (In particular, if you do more than one session or related studies at a later point in time).

The problem of no-shows

Be NICE to your subjects. Pay show-up fee if you have over recruited. Think airlines.

**

*Who Should Your Subjects be?Students vs. Non-studentsAdvantages of students: accessibilityconvenient recruiting low opportunity costs steep learning curves lack of exposure to confounding material.Disadvantages of students: something to provecontaminationlow income.

**

*ProfessionalsThe use of business professionals may solve some problems but create othersBehave often worse.Harder to motivate, confuse abstract situations with real life.Not much evidence in favour.But also not much evidence at all.

**

*Alternative subject poolsExperiments in developing countries

Experiments with representative samples

Experiments with survey participants

**

*Where to Conduct Experiments?Classroom experiments are extremely convenient for one-shotBut students may feel more empathy/sympathy towards class mates than to others they would normally interact withAnonymity extremely importantGrades as reward mediumIn any case, dont teach them the topic before (except if you plan this as a treatment).

**

*How many subjects? IThere are two issues:How many per game? How many altogether?How many observations? But what is an observation?

The first depends on the theory/model you look at. Simple in game experimentsMore difficult if you test competitive markets/models with continuum of traders

**

*4. Explanation of the rulesInstructions are KEY to your experiment.

Clarity is of utmost importance. Test them!!

Examples? Can be dangerous

Statement of purpose?

General rules: No talking etc.

Read them aloud?

**

*InstructionsInstructions tell subjects what they need to knowUseful to have a clear instructional script that enables precise replicationReading instructions out load is common practice to establish public knowledgeAs close as we can get to common knowledge assumption in game theoryCommon practice to explain how actions lead to payoffs

*5. PlayPen and Paper vs. Computers?Pen and paper experiments allow a great deal of freedom in changing your design, treatment, parameters, and procedures with little effort and delayComputer experiments often require the software to be written and rewrittenAnd computer facilities are expensive in equipment, maintenance, and support staff timeBut there are advantages to computer experiments.

**

*ComputerizationYou can exercise tighter control on the flow of information in the computer network e.g., eye contact, body language, and voice inflectionFast and accurate dataMay play role in the experimentComputers permit you to have less interaction with the subjects

*Matching ProtocolsExperiments are usually designed with several periods of play so subjects can learn from experienceBut if a pair of subjects play together several times, the possibility of reputation building can affect the prediction that game theory makesi.e., may be game-theoretical equilibria for the repeated game that differ from the one-shot, stage-game equilibriumUnless that is what we care about, we should avoid having subjects play with each other more than once in experiment session

*Lab logIt is good practice to keep a lab log, i.e., record all experiments in a bound log book by date, purpose, experiments, software, parameter valuesAlso to note any unusual eventsoften we forget what we did, when and why.Experiments should be replicable!

**

*Pilot experimentIf you are running a new experiment, then it is sensible to conduct a pilot This will help you to see ambiguities in instructions, missing information, too much (little) time, problems with equipment etc

*Lab setup & RegistrationDont underestimate time for preparations.In particular, when you have to book space.

Registration should be done to keep track of subjects history in your data base.

Who should conduct your experiment? (There is no final advice. But do NOT try to influence your subjects; do NOT stare over their shoulders, etc.)

Subject monitors (to make processes believable)

**

*How to handle queriesDifferent approaches

Often subjects are not allowed to say/ask anything allowed because this might spoil others. E.g., think of questions like Doesnt that mean that I should always play X?

If there is anything of general interest, repeat for all.

**

*6. PaymentsMonotonicity: Subjects must prefer more reward medium over less and not become satiated.

Salience: The reward must depend on choices of participants. Reward (payoff) function implements institutions (rule of the game).

Dominance: Subjects utility depends predominantly on the reward medium and other influences are negligible.

*RewardsVermon Smith (1962) reported the earliest experiments comparing behaviour of subjects who were rewarded in points with those rewarded with dollarsHe found that those paid only in points tended to approach competitive equilibrium more erratically and seemed to grow bored with the experiment faster than those who were paid moneyIn other words, by inducing value using money payment, the experimenter need rely only on the assumption that everybody likes having more money and nobody gets tired of having more of it

*7. AnalysisExperimentrics are econometric techniques customized to experimental applicationsGood knowledge of Statistics and Econometrics will give you a comparative advantageButdont use fancy techniques for the sake of being fancy!Use the technique (graphs, tests, descriptive statistics, econometrics) which best answers to your research question.We will talk more specifically about techniques soon

**

*8. Writing upIt should be the easiest part if:

Your research question was clear.Your design is most appropriate for such questionYour analysis is correct and insightful.

**

*

Structure of every experimental paper:

Introduction:Motivation (research Question)Related Literature (Highlight your novelties!)Experimental design and Procedures.TreatmentsRecruitmentIncentives

**

*

Structure of every experimental paper:

3. Results:Descriptive StatisticsGraphsTests and other Econometrics

4. Conclusion:How your goal was achieved.Suggest further research (and experiments?).

**

*Important Nuisances

*Important Nuisances IIn choosing your design it is worth thinking through what nuisance variables are likely to be important And how to deal with them

Here are some:LearningSubjects behaviour usually changes over time as their understanding of game deepens in the sessionYou can control for it by keeping it constantUse only the last few periods or runs

*Important Nuisances II2. ExperienceSimilar to 1 but occurs across sessionsKeep a database to track which subjects already came and played in a particular experimentThe easiest solution is to use inexperienced subjects

3. Boredom and FatigueKeep sessions to no more than 2 hours and the shorter the betterYou may save money and time by having them longer but you will compromise on salience and dominance when data comes from tired/bored subjects

*Important Nuisances III4. Extracurricular contactPrevent any uncontrolled communication among subjects during sessionIf they talk they may collude!If you cant monitor them, change the parameters if there is a break

5. Self-SelectionHave a long list from which you can choose your subjectsYou should actively choose balanced subject pools

*Important Nuisances IV6. Idiosyncrasies of individual subjectsSubjects (or groups) with a particular background may lead to unrepresentative behaviourReplicate with different subject pools

*

**

**

**

**

**

**

**

**

**

**

**

**

**

**

**

**

**

**

**

**

**