Imputation in the 2011 Census NILS Brownbag Talk – 6 May 2014 Richard Elliott.

Imputation in the 2011 Census

NILS Brownbag Talk – 6 May 2014

Richard Elliott

Overview

• Background• What is imputation• How did we impute the 2011 Census

• Strategy• Process• Implementation

• Considerations• Information• Next steps

Background

• Legal obligation on the public to complete a Census Questionnaire accurately

• A minority didn’t provide such information• Item non-response

– Leave questions unanswered– Make mistakes (i.e. neglect to follow questionnaire

instructions)– Provide values that are out of range (e.g. Born in 1791)

• Item inconsistency– Captured values not consistent with other values on the

questionnaire (e.g. 6 year old mother)• Non-response

– Don’t fill in the questionnaire at all

Background

• It is NISRA’s policy to report estimates for the entire population. Therefore Imputation was utilised to:• Correct for non-response

– Estimate the missing persons and households• Correct for Item non-response

– Fill the gaps left by unanswered questions• Correct for Item inconsistency

– Ensure that the information provided is consistent

• These types of data quality issues apply equally to any data collection exercise• Not specific to Census• Census Office recognises that users need to be aware

Background

• While imputation was used to “fill the gaps”, its strength comes from the information that was recorded

• Therefore, it is important to recognise the following:• Responses to the Census represent a self-assessment of a

respondents circumstances– Proxy responses given by main householder

• Respondents didn’t always complete the questionnaire correctly• 85% of questionnaires were completed on paper forms

– handwriting that had to be captured using an electronic character recognition system

– While Service Levels in place for capture, errors still possible

Two Types of Imputation

• Item Edit and Imputation• Correcting a dataset for inconsistencies and item non-response• Making each record “complete and consistent”

• Record imputation• The addition of whole records to a dataset• Estimate and adjust for persons missed, duplicated and counted

in the wrong place• Increases the accuracy of the overall estimates

Item Edit and Imputation Strategy

• Primary Objective:• to produce a complete and consistent database where

unobserved distributions were estimated accurately by the imputation process

• There were three key principles1. All changes made maintain the quality of the data

2. The number of changes to inconsistent data are minimised

3. As far as possible, missing data should be imputed for all variables to provide a complete and consistent database


• In adhering to these principles, the following key aims were defined• Editing must not introduce bias or distortion in the data• Editing facilitates the production of output data that is fit for

purpose• Editing methods help to ensure that pre-determined levels of

data quality are met– Highest priority given to variables which define population

bases (e.g. Age and Sex)• Editing supports the production of the population estimates by

ensuring that the basic population estimates are accurate


• Used a similar but enhanced version of the framework adopted in 2001• One Number Census Process• Tried and tested in 2001

• Was undertaken as part of the Downstream Processing (DSP) project at ONS• Included both Item and Record Imputation• Supplemented by detailed QA at every stage by NISRA Census

Office• NISRA benefitted from enhancements to the system found

through ONS data processing• Ultimately NISRA responsible for processing of NI data and any

parameter tweaking / re-runs

1. Cleansing the Data

2. Item Imputation

Imputation Process – 4 Key Stages

Capture and Coding RMR FRDVP

Edits Donor Imputation

Manual Imputation

3. Coverage Assessment and

Adjustment

4. Post-Coverage Item

Imputation


2. Item Imputation


Capture and Coding RMR

FRDVP EditsDonor

ImputationManual

Imputation


Adjustment

4. Post-Coverage Item Imputation

Implementation – Capture and Coding

• Capture and coding rules• Turning tick and text responses into data that could be edited and

imputed– Complex coding used to assign numerical values to written text

and ticked boxes (e.g. occupation and industry coding)– Invalid responses flagged for imputation (V, W, Y and Z)

• Determinations made to responses to resolve combinations of tick and text

– Ticks that could not be determined set to W (failed multi-tick)– Text that was uncodeable set to V (uncodeable text response)

• Data subject to checks to ensure each question response was within a predefined range (e.g. No year of birth before 1896 or after 2011)

– Invalid responses set to Z (out of range)• Missing data flagged as Y (missing requires imputation)


• Determining combinations of ticks• Single tick

RESPONSE

CODING RULES

OUTPUT CODE Text Tick

N/A Single tick Accept single tick 1 – 4 1 – 4

N/A Two ticks

Two ticks 1 + 2, code as 2 2



Any other combination of two ticks code as W

W

N/A Three or more ticks

Code as W W

N/A None Code as Y Y


• Determining combinations of ticks• Resolvable multi-tick

RESPONSE

CODING RULES



N/A Two ticks





W


Code as W W



• Determining combinations of ticks• Irresolvable multi- tick

RESPONSE

CODING RULES



N/A Two ticks





W


Code as W W


This will be assumed missing and imputed.


• Missing data

RESPONSE

CODING RULES



N/A Two ticks





W


Code as W W


This will be imputed.


• Resolving write-ins• Numbers

RESPONSECODING RULES OUTPUT

CODE Text Tick

If the answer is written as a word within the 2 CB constrained field then input the equivalent numeric value. If only one digit is entered with a space or a non-numeric character then right justify and precede with a zero e.g. – 1 code as 01

Text N/A Code text, accept 00 to 99.Any number above 99 code as ZZ, invalid text code as ZZ

00 – 99 ZZ

None N/A Code as YY YY

1

01


• Resolving write-ins• Numbers


CODE Text Tick



00 – 99 ZZ


two

02


• Resolving write-ins• Range Check


CODE Text Tick



00 – 99 ZZ


199



• Resolving write-ins• Codeable response

F R A NC E

“FRANCE” gets coded to 250


• Resolving write-ins• Uncodeable response

S U G A R

“SUGAR” is clearly not a country so set to set to VVV.



2. Item Imputation



FRDVP EditsDonor

ImputationManual

Imputation


Adjustment


Implementation – RMR

• Reconcile Multiple Responses (RMR)• Removal of false persons

– Removal of persons generated by capture anomalies– For example: strike throughs, inadequately completed

questionnaires• Removal of duplicates (multiple persons / households)

– Individuals who included themselves more than once– Separated parents who included their children at both

addresses• Creating households / communals from multiple questionnaires

– Consolidating H4 / HC4 / I4 etc• Validation

– Renumbering person records within households / communals


2. Item Imputation



FRDVP EditsDonor

ImputationManual

Imputation


Adjustment


Implementation – FRDVP

• Filter Rules and Derived Variables for Processing (FRDVP)• Correct data by applying edits to correct for questionnaire

routing errors• Apply hard edits to keep individual records consistent

– Minimal at this stage (mostly applied within imputation system)

• Information not required set to X– No imputation done on any variable set to X

• Create high level variables to be used within the Item Imputation system

– Blocking variables for donor searching– Makes it easier to find donors


• Surplus information – questionnaire routing

In this scenario the respondent should have skipped question 6 and gone straight to question 7.

Therefore, since the respondent should not have answered question 6, it is set to:

X (not required)


• Surplus information – questionnaire consistency

9 L O RD WAR DE N S

In this scenario, since the respondent is aged under 1 on Census day, and therefore did not have a usual address one year ago, the captured address information is set to X.

C RE S C E N T

B T 1 9 1 Y J

0 1 0 1 2 0 1 1


2. Item Imputation



FRDVP EditsDonor

ImputationManual

Imputation


Adjustment


Implementation – Item Imputation

• Achieved using CANCEIS• Canadian Census Edit and Imputation System• Developed specifically for Census type data

– ie a mix of categorical and numerical variables• Donor-based edit and imputation system that can

simultaneously:– Apply nearest-neighbour donor imputation– Apply deterministic edits and maintain consistency

• Evaluated and endorsed as the 2011 Census imputation tool– Faster– Less resource intensive– Allowed for more joint-imputation

Implementation – CANCEIS

• How did CANCEIS work in practice• The database was divided up into processing units for the

purposes of resource management and maximising donor pools

Three Geographic

units

Household questions

Person questions

Individual imputationDonor unit = household

Household Persons 1 to 6


Communal Persons

Joint Household imputationBetween person edits and relationships

Donor unit = household of same size

Individual imputationRelationship to Person 1

Donor unit = individual person

Individual imputationDonor unit = individual person

Delivery Groups (Processing Units)


• How did CANCEIS work in practice• The database was divided up into processing units for the

purposes of resource management and maximising donor pools

Three Geographic

units

Household questions

Person questions

Individual imputationDonor unit = household



Communal Persons

Joint Household imputationBetween person edits and relationships

Donor unit = household of same size

Individual imputationRelationship to Person 1

Donor unit = individual person

Individual imputationDonor unit = individual person


• How did CANCEIS work in practice• The household questions were imputed within a single module• Person data was divided up into 4 modules

– Aim was to group variables that help predict each other– Attempt to maximise the number of donors for a given group

Demographics

e.g. Age, Sex, Marital status,

Student, Activity last week

Culture

e.g. Ethnicity, Country of birth,

Language, Passports

Health

e.g. General health, Disability,

Long-term condition

Labour Market

e.g. Economic activity, Hours

worked, Qualifications


• How were the donors selected?• Within each module a number of matching variables were used to

select donors• Matching variables were weighted according to several factors

– How well they would predict other values and how highly they should be prioritised when resolving inconsistencies

– For example, age is often a good predictor of other demographic variables

– Age was given a high weight, therefore observed ages were prioritised over other values if there was an inconsistency and changes were required

• Northings and Eastings were used to control for geographical differences and find donors from similar areas

– These were given a small weight compared to demographic characteristics like age, sex and marital status etc


• Matching variables (example)• Suppose someone omitted to fill in their occupation details• The record would be flagged for imputation under the Labour

Market module• Donor pool identified by matching on (for example):

– Economic Activity– Industry– Hours worked– Qualifications

• These variables deemed to influence Occupation• Occupation information imputed from a donor with similar

Labour Market characteristics


• Editing and Imputing was done simultaneously• Each record was checked for consistency before imputation• Any items that failed the checks were marked for imputation along with the missing

items

• A single donor was selected to resolve inconsistencies and non-response• Only values which satisfied the edit constraints were imputed into the recipient

record• CANCEIS sought to minimise the number of changes required to repair a record

when edit constraints were in place

• There were 31 edit rules which were broadly based on 2001• e.g. If aged between 5 and 15 then must be in full-time education

• Some rules had to be updated to account for changes since 2001• e.g. Removal of rule that did not allow same-sex couples• Replaced with rules that said married couples had to be opposite-sex and civil

partners had to be same-sex


• Say we have the following (oversimplified) example:

• Student is missing• Requires imputation under the demographic module

• This record is subject to two edit constraintsA. Must be aged 16+ to be married

B. Aged 5 to 15 must be a student

• Fails Rule A since aged 10 and married• Therefore, both Age and Marital Status are also flagged for

imputation

Record ID Age Marital Status

Student

Obs1 10 Married -9 (Missing)


• Say we have the following (oversimplified) example:

• Student is missing• Requires imputation under the demographic module

• This record is subject to two edit constraintsA. Must be aged 16+ to be marriedB. Aged 5 to 15 must be a student

• Fails Rule A since aged 10 and married• Therefore, both Age and Marital Status are also flagged for

imputation


Student



• The system searches for potential donors• Matching on demographic variables• Uses Northings and Eastings to find a donor in the area

• The following records are returned:


Student

Donor1 4 Single No

Donor2 12 Single Yes


A. Must be aged 16+ to be marriedB. Aged 5 to 15 must be a student


Student


Donor1 4 Single No


New1



• Donor1• Using Donor1 would mean that “Single” is taken as well as “No”


Student


Donor1 4 Single No


New1 10 Single No



• Donor1• Using Donor1 would mean that “Single” is taken as well as “No”• The new record fails Rule B• Therefore Age is taken from the donor as well


Student


Donor1 4 Single No


New1 10 Single No



• Donor1• Using Donor1 would mean that “Single” is taken as well as “No”• The new record fails Rule B• Therefore Age is taken from the donor as well


Student


Donor1 4 Single No


New1 4 Single No

Two observed value changes



• Donor2• Using Donor2 would mean that “Single” is taken as well as “Yes”


Student


Donor1 4 Single No


New1 10 Single Yes



• Donor2• Using Donor2 would mean that “Single” is taken as well as “Yes”• The new record passes both Rule A and Rule B


Student


Donor1 4 Single No


New1 10 Single Yes

Only one observed value change



• Donor2• Using Donor2 would mean that “Single” is taken as well as “Yes”• The new record passes both Rule A and Rule B

• Donor2 given a higher probability of selection


Student


Donor1 4 Single No


New1 10 Single Yes

Only one observed value change


• Points to note• Variables were imputed in blocks of similar variables (modules)

– there was no individual model for any one question• There is independency between the modules

– for example, cultural characteristics might come from a different donor to employment characteristics

• Imputed person data was combined in a way that maintained relationship consistency within a household

• Given the processing approach, quality was maintained at the geographic unit level


2. Item Imputation



FRDVP EditsDonor

ImputationManual

Imputation


Adjustment


Implementation – Manual Imputation

• Manual Imputation kept to a minimum but was necessary• Manual Imputation – QA checks

• Quality Assurance at every stage of processing• Distributional checks and checks against comparator data sources• Edits made through Data File Amendments (DFAs)• DFAs not taken lightly

– Involved detailed questionnaire image analysis– Mostly correcting for capture errors– e.g. Centenarians

• Manual Imputation to increase donor pool• Temporary changes sometimes required when donor pool too small• E.g. Postcode matching

(would have been done later in processing but brought forward)


2. Item Imputation



FRDVP EditsDonor

ImputationManual

Imputation


Adjustment


Implementation – Coverage

• Coverage Assessment and Adjustment (Record Imputation)• Estimating wholly missed households and/or missing persons

within households– Enumerated persons (92%)– Census Under-enumeration project (CUE) (4%)– Census Coverage Survey (CCS) (5%)

• Further information can be found at http://www.nisra.gov.uk/Census/pop_QA_2011.pdf

http://www.nisra.gov.uk/Census/pop_QA_2011.pdf


2. Item Imputation



FRDVP EditsDonor

ImputationManual

Imputation


Adjustment


Implementation – Post-Coverage

• Post-Coverage Item Imputation• Making wholly imputed records complete and consistent• Using same methods as initial Item Imputation• Required only basic demographic information to be available for

each record• Final check for consistency

Considerations

• Self completion• Incorrect information provided (mother putting down wrong age of baby)• Bad understanding of question or layout (marital status / relationships)

• Some capture errors exist• eg dob captured as 1961 instead of 1981 – valid in family• Strike throughs

• Item Imputation assumes missingness is at random (MAR)• It has to – cant make any other assumption• Attempt to control for dependency by using modules• Negligible change to marginal distributions

• Record Imputation doesn’t assume MAR• Designed to correct for under-coverage which is not uniform• This imputation will change variable distributions• Extent of change driven by CCS and CUE

Considerations

• While based on similar approach in 2001, some differences exist that can affect Imputation rates• Changes to definitions (eg Marital Status)• Some questions are quite similar but subtly different

– eg Religion / Qualifications• Change in processing ability

– Workplace postcode matching much easier in 2011

• Census QA undertaken at every stage• Census assessed against various comparator datasets• However, unable to compare Census to Census unit records

– 2001 to 2011 link not available when processing

Information

• Information already available• ONS paper on Item Edit and Imputation process

– Item Edit and Imputation Process paper• ONS Evaluation report on Item Edit and Imputation

– Item Edit and Imputation Process Evaluation paper• 2011 NI Census Methodology Overview

– http://www.nisra.gov.uk/Census/pop_meth_2011.pdf• Details on the NI Census Under-enumeration project

– 2011 Census Under Enumeration Project: Methodology paper• 2011 NI Census Quality papers

– http://www.nisra.gov.uk/Census/pop_QA_2011.pdf– http://www.nisra.gov.uk/Census/pop_QA_2_2011.pdf– http://www.nisra.gov.uk/Census/key_QA_2011.pdf

• Census Quality Survey– http://www.nisra.gov.uk/archive/census/2011/census_quality_survey.pdf

http://www.ons.gov.uk/ons/guide-method/census/2011/census-data/2011-census-user-guide/quality-and-methods/quality/quality-measures/response-and-imputation-rates/item-edit-and-imputation-process.pdf

http://www.ons.gov.uk/ons/guide-method/census/2011/how-our-census-works/how-did-we-do-in-2011-/evaluation---item-edit-and-imputation.pdf

http://www.nisra.gov.uk/Census/pop_meth_2011.pdf

http://www.nisra.gov.uk/archive/census/2011/methodology/under-enumeration.pdf

http://www.nisra.gov.uk/Census/pop_QA_2011.pdf

http://www.nisra.gov.uk/Census/pop_QA_2_2011.pdf

http://www.nisra.gov.uk/Census/key_QA_2011.pdf

http://www.nisra.gov.uk/archive/census/2011/census_quality_survey.pdf

Next Steps

• Census Imputation rates will be published in due course

• Change Rates available on NILS website• Note that rates are change rates and not Imputation Rates

– Imputation rates are expressed as a percentage of expected response rather than total response

• Most people filled in most of the questionnaire• A small proportion didn’t• Robust procedures applied to “fill the gaps”

Questions

Imputation in the 2011 Census NILS Brownbag Talk – 6 May 2014 Richard Elliott.

Documents

Transcript of Imputation in the 2011 Census NILS Brownbag Talk – 6 May 2014 Richard Elliott.