Sampling, sample size estimation, and randomisation

PS302

OverviewSampling

representative sampling (e.g. for surveys)homogenous sampling (e.g. for experiments)

Sample size estimationBased on power

Gathering the information you needPower calculations (G*Power software)

- ANOVA- regression

Rules of thumb for multivariate testsPresentation of power analysis in your report

Practical randomisingRandom selection (e.g. for surveys)Random allocation (e.g. for experiments)

Getting a representative sample

Survey of UK Households

want a sample from each SES group

each age group

each sex

Proportions should match the population

Matching the population

Percent of population percent of sample

Assume, sample size = 1200

Population = Women 60%, Men 40%

Sample: Women 720, Men 480

Problem for you to try

Population figures:Men 65 years+ = 1 millionWomen 65 years+ = 1.5 millionMen 25-65 years = 8 millionWomen 25-65 years = 8.5 millionMen < 25 years = 5 millionWomen < 25 years = 5.2 million

Given a sample size of 200, how many women <25 years should be included?

Total population size = 29.2 million

Percent W25-65 = (5.2 / 29.2) * 100 = 17.8%

quota sampling

Recruiters are given a quota of each stratum

Problem – biased selection by recruiter/interviewer

Advantage – random selection very difficult to achieve, quota sampling a good compromise

Homogenous sampling

Restrict sampling to a narrow group

Sample only Warwick studentsSample only one SexSample only one Age group

Advantagesreduces error variance by reducing

individual differences

Homogenous sampling ctd

Disadvantage – may reduce generalisabilitygeneralisability will need to be considered and assessed separately

Suitability– experimental work– studies where individual differences are

not directly relevant and power is more important concern

power

Probability that any particular (random) sample will produce a statistically significant effect

Eg. power = 0.9

90% chance of detecting an effect if there really is an effect

Researchers usually aim to have power at 80-90%

Power and sample size

All else being equal, to get more power you need more participants

Where “all else” means:reliability of measuresother sources of error variancep-valuethe true size of the effect

These concepts are inter-related

Desired power ↑ N ↑

Acceptable p-value ↓ N ↑

Effect size to detect ↓ N ↑

Reliability of measures ↓ N ↑

Other error variance ↑ N ↑

if you know these…

effect size

variance of measures

you can often work out what the sample size should be

So where can you find them?

Previous research studies

Calculating using G-power

• First step, assemble the figures needed

For between subjects ANOVA:1. Effect size (Cohen’s f, or partial eta squared)

2. Significance level [.05, usually]

3. Power [.8, usually]

4. Numerator degrees of freedom (df)

5. Number of cells in design (groups)

1. Effect size

… from previous studies

Easy – they reported effect sizes“There was not a significant main effect of Sex on response time, F(1, 42) = 2.03, p = .16, η2 = 0.046”

Harder – they reported only the F and df, so you have to make a calculation

partial η2 = (dfeffect * F) / [(dfeffect * F) + dferror]

= (1 * 2.03) / [(1 * 2.03) + 42]= 0.046

measures of effect size for ANOVA

eta squaredThe proportion of variance in the outcome variable (DV)

that is explained by the IV SSeffect / SS[corrected] total

partial eta squared (SPSS prints this out)The proportion of the effect + error variance explained by

the effectSSeffect / (SSeffect + SSerror)

Roughly, the correlation between an effectand the outcome (DV)

4. Numerator df

“There was a non-significant main effect of Gender on response time, F(1, 42) = 2.03, p < .05, η2 = 0.09”

5. Number of cells (groups)

Two way ANOVA

2 x 3 ANOVA 6 cells

4 x 2 ANOVA 8 cells

Etc.

Calculating using G-power

• First step, assemble the figures needed

For this 2 X 3 between subjects ANOVA:1. Effect size (η2 = 0.046)2. Significance level [.05, as usual]3. Power [.8, normal]4. Numerator degrees of freedom (df = 1, 2 for

the respective main effects, or 2 for the interaction)

5. Number of cells in design (groups = 6)

tip: power & ANOVA

Each effect in the ANOVA has its own power

Eg. 2 x 3 ANOVA

Main effect A

Main effect B

Interaction effect A * B

Tip: power is lower for interactionsthan for main effects

Sample size – ethical issues

Too small a sample

-- can’t detect significant effects

waste all participants’ time

Too large a sample

-- waste resources

-- waste the extra participants’ time

Sample size – practical issues

ResourcesTimeCost of running each participant

AvailabilityClinical populations are often smallAccess can take time & require permission

Choosing an appropriate sample size for established laboratory paradigms

Shortcut

Base sample size on sample size used in previous research

This is often perfectly appropriate

(but make sure the previous research is of high quality!)

Rules of thumb for multivariate tests

multiple regression

cases (N) / predictors (p)

N at least 50 + 8p for R2

N at least 104 + p for testing a predictor

Need more cases if outcome is skewed, anticipated effect size is small, measures less reliable…

Rules of thumb for multivariate tests

PCA (exploratory FA)

50 no good

100 poor

300 good, but ideally need more

Random allocation

For example

3 between subjects conditions (e.g. control, happy, sad)

Who does which condition?first come? Interviewer choice?

Must avoid confounds. But can’t check all possible. Solution is random allocation.

Random allocation needs truly random numbers

Different ways to do that

SPSS

random.org

Research randomiser

scripting language like python

Python

to randomly assign 9 participants to 3 conditions:

from random import shuffle

numbers = [1,1,1,2,2,2,3,3,3]

shuffle(numbers)

numbers[3, 2, 1, 2, 3, 1, 3, 1, 2]

Research randomiserhttp://www.randomizer.org/form.htm

3 conditions, 48 planned participants

randomly: allocate each participant (identified by order of recruitment) to one of 3 conditions

How many sets of numbers to generate? [1]numbers per set? [48]Number range? [From 1 To 3]Do you wish each number in a set to remain unique? [No][Don’t “sort”!]

Result

Set #1:3, 3, 1, 3, 3, 1, 2, 2, 2, 1, 2, 2, 1, 1, 2, 1, 3, 3, 1, 3, 2, 1, 3, 3, 1, 2, 1, 3, 1, 2, 2, 2, 3, 3, 1, 3, 3, 1, 1, 3, 1, 3, 3, 2, 3, 3, 1, 2

Research randomiserhttp://www.randomizer.org/form.htm

3 sentence types, 48 sentences

16 in each group, create a random sequence, but limit runs of the same type

How many sets of numbers to generate? [16]numbers per set? [3]Number range? [From 1 To 3]Do you wish each number in a set to remain

unique? [Yes]

Research randomiserhttp://www.randomizer.org/form.htm

3 types, 48 sentences, 16 of each type limit run of a given type, while still

randomising order of presentation

16 Sets of 3 Unique Numbers Per SetRange: From 1 to 3 -- UnsortedJob Status:      Set #1:2, 3, 1           Set #2:3, 1, 2           Set #3: ….

Web links

http://www.randomizer.org/

http://www.random.org/

measures of effect size for ANOVA

eta squaredThe proportion of variance in the outcome variable (DV)

that is explained by the IV SSeffect / SS[corrected] total

partial eta squared (SPSS prints this out)The proportion of the effect + error variance explained by

the effectSSeffect / (SSeffect + SSerror)

Roughly, the correlation between an effectand the outcome (DV)