AN EXAMPLE OF COOPERATION & SOME WIDER ISSUES

Ian Plewis (Bedford Group, Institute of Education) &

Stephen Morris (Social Research Division, Department for Work &

Pensions)

OVERVIEW• Government research identifies important

methodological issues• These issues often/sometimes ignored or

assumed not to be a problem – exception ONS work on methods & quality

• Where methods research takes place no government forum for exchange exists

• Child outcomes work (IoE/DWP) – a methods project - to illustrate the problem

• We suggest why methods often not addressed & thoughts on potential solution

THE RESEARCH

• A feasibility study

• Are adult outcomes for individuals (parents) linked to (or caused) by changes in childhood circumstances?

• Effects of change in household income (household deprivation & parental employment) in childhood on individual’s adult outcomes

• Pathways through which changes take effect?

POLICY CONTEXT• Government child poverty targets – end child

poverty by 2020.• This research informs development of policies to

meet targets• Marginal £ - what be the most effective way of

addressing child poverty– Income transfers?– Services?

• Our focus on changes in income thus whether transfers are likely to be effective

WHY NEW RESEARCH?• Not much UK research looking at change (Our

interest is in establishing causality)• Tends to focus on short-range educational

outcomes • Focus on single data sources• Most useful sources – birth cohorts – have

limitations• Can data sources be combined to better address

questions of relevance – What assumptions do we have to make?– What statistical models could be used?

THE PROBLEM

• Most quantitative social research has to deal with methodological issues.

• Some of these issues are ‘cutting edge’ and need to be addressed within a research council project.

• Some are ‘standard’.• But many fall between these two extremes and

tend to get ignored in Government funded projects

MEASUREMENT ERROR• Suppose we want to estimate a regression

model with a measure of income as one of possibly several explanatory variables.

• We know that measures of income are unreliable and we also know that this unreliability, if ignored, can lead to biased estimates of coefficients of interest.

• We should correct for measurement error because not to do so could lead to misleading inferences for policy.

• But such corrections are not standard and the problem is often swept under the carpet. .

NON-RESPONSE

• Unit non-response and attrition are features of all longitudinal datasets.

• Certain kinds of cases tend to be lost – at the outset and over time.

• Ideally, we should carry out sensitivity analyses of some kind to establish whether losses from the sample are likely materially to affect policy conclusions.

• But these analyses take time and resources

INCOME MISSING

• Data are often missing for income even when other variables are measured.

• And some surveys do not collect income data.• Imputation is a way round some of these

difficulties.• But there are a number of imputation methods –

how does their application affect our conclusions?

IMPLICATIONS FOR POLICY• Measurement error/non-response = biased

estimates• Biased estimates can lead to bad policy

decisions• Example (hypothetical):

– Under-estimate of effect of change in income on child outcomes

– Assume income transfers less effective– Direct policy toward services– Sub-optimal policy response

WHY MIGHT THIS HAPPEN?

• No cross-government forum for exchange of information on methods research

• Within spending departments not always easy to make the case for methods research– Benefits of methods research diffuse – Not priority for policy customers

• Researchers don’t always appreciate implications

• Time – complex problems take time to solve

WHAT COULD BE DONE?• Establish a GCSRO/NCRM methods group

(representatives: depts., ONS, ESRC & NCRM)• Forum for raising methods related issues with policy

focus – disseminate existing work from ONS, depts. & academics/researchers

• Identify gaps and commission – requires funding– Policy relevance– Wider application

• Dissemination – publications, seminars and training courses

• CASE studentships

CONTACT DETAILSIan Plewis – Institute for Education(T) 020 7612 6238(E) i.plewis@ioe.ac.uk

Stephen Morris – Department for Work & Pensions(T) 020 7962 8193(E) stephen.morris4@dwp.gsi.gov.uk