SI0124 – Introduction to Social Science Research

Week 5Luke Sloan

Asking the Unanswerable:Good Questions & Bad Questions

Introduction

• Research Questions• Hypotheses• Data Collection Questions

• The Problem of Falsification• The Problem of Access• The Problem of Perception• The Problem of Ambiguity

• Commissioning the Census 2021

Research Questions

• The research question (RQ) is the over-arching guiding principle of any project

• It is the ‘big question’ that you set out to answer e.g…

• What is the relationship between ethnicity and income?

• How does social class affect individual health?

• Why do differences in GCSE results exist between the sexes?All of these questions are very broad – some of them too much so! They need to

be broken down before they can be answered.

Hypotheses• Hypotheses are not questions – they are statements that can be proved wrong

• Note that they cannot be proved right!

• Natural science model of falsification

• We answer the research question using hypotheses e.g. for varying education ability at GCSE between the sexes…

– H1) Males do worse than females at GCSE English– H2) Females do worse than males at GCSE Maths– H3) Males spend more time playing computer games than females– H4) Females revise for longer than males– H5) Males do better than females at exams– H6) Females do better than males at coursework

Data Collection Questions• These are the questions that allow us to test our hypotheses

• They form part of the data collection instrument (interview schedule, survey etc…)

• For example, to test H1 (that males do worse than females at GCSE English)…– Are you male or female?– What was your final result for GCSE English?

• If you’re distinguishing between exams and coursework you might also ask…– What was your final coursework mark for GCSE English?– What was your final exam mark for GCSE English?

You must make sure that your data collection questions answer your hypotheses!

The first part of the lecture will be about designing good research questions.

The second part of the lecture will be about designing good data collection questions.

The Problem of Falsification

• Does God exist?

• A fascinating question?• Interest does not equate to usefulness!• How can you possibly investigate this?• Remember the scientific method• Popper’s Falsification Principle?

You must be able to use empirical evidence to answer your question

The Problem of Access

• How much time do British Prime Ministers spend on defence matters?

• Beware classified information!• How many PMs are available for comment?• How many would know?• Is this information even recorded?

You should only ask questions to which the data is available or can be collected

The Problem of Perception

• Do downtrodden women earn less than downtrodden men in the Celtic Fringe?

• What do you think ‘downtrodden’ means?• What an Earth is the ‘Celtic Fringe’?• What is being earned? Money? Respect?

Avoid using slang or subjective/contested concepts (e.g. community)

The Problem of Ambiguity

• Do people from Pakistani backgrounds earn less?

• Less than who?• What is a ‘Pakistani background’?• Who are the ‘people’? Age? Sex?• Geographic location?

Be specific – ask yourself who, what, where and how to check

Just Plain Nuts!• Do 60% of people like Anchorman 100%

of the time whilst playing the harmonica?

• Important question for Social Science?

• Too specific – likely to not be true

• How could you investigate this?

• What is the sample?

• Where is the theory?

5 Minute Break

Commissioning the Census 2021

• I’m going to show you a series of questions with response sets (data collection questions)

• Working in pairs or small groups, have a look at the questions and discuss whether they are ‘fit for purpose’

• Do they meet the ‘gold standard’ – would you commission these questions for the Census 2021?

• Beware the flying killer whale…

Candidate 1

What is your religion?

This question is voluntary

☐ No Religion☐ Christian (including Church of England, Catholic…)☐ Buddhist☐ Hindu☐ Jewish☐ Muslim☐ Sikh☐ Any Other Religion, write in…

• Talk about eliciting a response!• Assumes that you ‘should’ have a religion• ‘No Religion’ option makes no sense

• No room for Atheism/Humanism• Is the order of responses randomised?• What do you think the result will show?

Candidate 2

How is your health in general?

☐ Very Good☐ Good☐ Fair☐ Bad☐ Very Bad

• What does this even mean?• ‘Good health’ is subjective• What about asthma?

• Trying to simplify a complex concept• What if answered by proxy?• What if I have the flu at the time?

Candidate 3

How well can you speak English?

☐ Very well☐ Well☐ Not well☐ Not at all

Seriously?

Candidate 4

Which of these qualifications do you have?

☐ 1-4 O levels/CSEs/GCSEs (any grades), Entry Level, Foundation Diploma☐ NVQ Level 1, Foundation GNVQ, Basic Skills☐ 5+ O levels (passes)/CSEs (grade 1)/GCSEs (grades A*-C), School Certificate, 1 A

level/2-3 AS levels/VCEs, Higher Diploma☐ NVQ Level 2, Intermediate GNVQ, City and Guilds Craft, BTEC First/General Diploma,

RSA Diploma☐ Apprenticeship☐ 2+ A levels/VCEs, 4+ AS levels, Higher School Certificate, Progression/Advanced

Diploma☐ NVQ Level 3, Advanced GNVQ, City and Guilds Advanced Craft, ONC, OND, BTEC

National, RSA Advanced Diploma☐ Degree (for example BA, BSc), Higher Degree (for example MA, PhD, PGCE)☐ NVQ Level 4-5, HNC, HND, RSA Higher Diploma, BTEC Higher Level☐ Professional qualifications (for example teaching, nursing, accountancy)☐ Other vocational/work-related qualifications☐ Foreign qualifications☐ No qualifications

Candidate 4 – A Commentary• Firstly – does anyone know how to answer?

• Very strange ordering and distinctions

• Overlapping categories (e.g. ‘PGCE’ and ‘professional qualification – teaching’ in different groups?

• Scottish Highers? Students who moved to England? Surely not a ‘foreign qualification!

• What about the International Baccalaureate? Certainly not ‘foreign’…

• Distinguishes between 1 and 2 A levels (or 2-3 and 4+ AS levels) but not a BSc/Ba and postgraduate qualification

• Last point is odd considering the growing number of university graduates and the increased demand for postgraduate training

Candidate 5

Think about your weekly grocery shop. What proportion of your expenditure is spent on the following items in a typical week? (e.g. 15% spent on fruit)

☐ Fruit☐ Vegetables☐ Meat☐ Confectionary☐ Alcohol☐ Entertainment☐ Clothes

• Such ‘ratio’ questions are common in market research• Very difficult to answer• Proportion of expenditure requires complex thought processes

Your Turn…

• In pairs or small groups, design the worst possible question that you can think of!

• Break as many rules as possible

• Don’t just go for unintelligible language, think about response options too

The Worst Question in the World…

Any Questions?