Choosing the correct sample size

Roger Watson

Sample size

Question:

Why is it important?

Answer:

• We usually cannot access the whole population

• We usually do not need to access the whole population

• We need samples to be representative of the population in surveys

• We need samples to be large enough to show real effects where inferential statistics are being used to analyse data

Sample size - surveys

How many people should the questionnaire be sent to?

Answer:

• Enough to make the results representative of the population

• No more than is necessary to achieve the above

• Sample size will be related to the size of the population

Minimum sample sizes for selected small populations

95% confidencePopulation size with 5% margin of error

<200 Sample whole population500 2181,000 2782,000 3235,000 35710,000 370100,000 383

(Jackson & Furnham 1999)

Minimum sample sizes for selected small populations

95% confidencePopulation size with 5% margin of error

<200 Sample whole population500 2181,000 2782,000 3235,000 35710,000 370100,000 383

The confidence level is the amount of uncertainty you can tolerate. Suppose that you have 20 yes-no questions in your survey. With a confidence level of 95%, you would expect that for one of the questions (1 in 20), the percentage of people who answer yes would be more than the margin of error away from the true answer. The true answer is the percentage you would get if you exhaustively interviewed everyone. 

Higher confidence level requires a larger sample size.

Minimum sample sizes for selected small populations

95% confidencePopulation size with 5% margin of error

<200 Sample whole population500 2181,000 2782,000 3235,000 35710,000 370100,000 383

The margin of error is the amount of error that you can tolerate. If 90% of respondents answer yes, while 10% answer no, you may be able to tolerate a larger amount of error than if the respondents are split 50-50 or 45-55. 

Lower margin of error requires a larger sample size.

http://www.raosoft.com/samplesize.html

Populations and samples

Population: all the members of a particular group

Sample: a subset of a population

Representative sample: sample is like population

Biased sample: sample is unlike population

Sampling error: difference between the above

Biased samples

• Result from poor sampling

• May result from poor response rates

• For example, only middle class, educated and motivated people may respond to a survey

Sampling methods

Probability sampling:

simple randomstratified random

Non-probability sampling:

opportunisticsystematicpurposive

Sample size

Also depends on the statistical tests you are going to use

Power analysis*

Related to effect size

* Cohen J (1992) A power primer Psychological Bulletin 112 155-159

Effect size

In statistics, an effect size is a quantitative measure of the strength of a phenomenon. Examples of effect sizes are the correlation between two variables, the regression coefficient, the mean difference, or even the risk with which something happens, such as how many people survive after a heart attack for every one person that does not survive. For each type of effect-size, a larger absolute value always indicates a stronger effect. Effect sizes complement statistical hypothesis testing, and play an important role in statistical power analyses, sample size planning, and in meta-analyses. (Wikipedia)

Cohen suggested that d=0.2 be considered a 'small' effect size, 0.5 represents a 'medium' effect size and 0.8 a 'large' effect size. This means that if two groups' means don't differ by 0.2 standard deviations or more, the difference is trivial, even if it is statistically significant. (Walker)

Total sample(100)

Men Women(50) (50)

Young Old Young Old

(25) (25)(25) (25)

Well Sick Well Sick Well SickWell Sick

(12) (13) (10) (15) (13) (12) (11) (14)

email: r.watson@hull.ac.uk

@rwatson1955