2011 Census2007 Census Test – emerging findings

Garnett Compton, ONS

Updated 4 September 2007

BSPS – 12 September 2007

Session Aim

Aim:

• To share emerging findings from the 2007

Census Test• Outline plan for publishing 2007 Census Test

evaluation• Present a brief update on other 2011 Census

developments

Contents

• Test background• Results:

• Address register development• Post-out/hand delivery• Income question

• Key findings of 2007 Test• Further evaluation and publication• Did you know ….

2007 Census Test - Objectives

Test objectives:• Assess the effect on response of:

– Inclusion of an income question; and– the use of post-out to deliver questionnaires.

• Assess the feasibility of major innovations in proposed 2011 Census operational procedures:

– Outsourcing of field staff recruitment, pay and training;– development of an operational intelligence system to

enable individual questionnaires to be tracked; and,– development of an address list and address checking

procedures.

2007 Census Test - Sample sizes by LA and ETC

0

5000

10000

15000

20000

25000

30000

35000

40000

Cam

den

L'pool

Stoke

Bath &

NE

S

Car'shire

5

4

3

2

1

ETC

2007 Census Test – High-level Design

• Address checking– Conducted in all Test areas during Sept and October– Split discretionary and full contact methods– Used controlled errors

• Delivery – 50% Post-out, 50% hand delivery– 50% questionnaires income, 50% no income – For hand delivery 3 attempts at contact over 2 week period

• Collection/Follow-up– Central post-back– 23 May – 22 June– 3 attempts everywhere– reminder letter to all outstanding addresses as at 31 May

2007 Census Test – Key constraints

Key constraints affecting evaluation of the Test:

• Voluntary– Relied on public’s good will to complete a return

• Publicity• Sample – skewed to harder to enumerate areas• Follow-up

– Fixed number of follow-up attempts everywhere

Results

Household* return rates by LA

0%

10%

20%

30%

40%

50%

60%

70%

Cam

den

L'pool

Stoke

Bath &

NE

S

Car'shire

All areas

* As at 15 July

Household* return rates by ETC

0%

10%

20%

30%

40%

50%

60%

70%

ETC* As at 15 July

Address register development

Results

Key results

• Address checking (AC) found about 12% new addresses– Time consuming to validate and update AR

• 4.4% (4,400) of questionnaires undelivered– Of which 17% were added by address checkers– Reasons include vacant properties, non-existing

addresses, duplicate addresses.

• 1,200 new addresses found during enumeration– Of which, over 50% found at follow-up.– Nearly 70% of new addresses were sub-premise

addresses – suggest existed at time of AC.

Early conclusions

• Suggests we need to plan to do a 100% address check in 2011 no matter what the delivery design.

• Plan is for a rolling address check over a longer period of time (4-6 months).

– enables better quality address checking; and,– enables more time to update the final address register

with Address check findings.

• Consider a re-address check shortly before the Census in a small %’age of areas

Early conclusions cont …

• Main enumeration and controlled errors didn't identify all missing addresses – need to improve methods and guidance

• Issues around whether information with address suppliers can be shared;

• Criteria for deciding which address products to use as a base under consideration.

Delivery Method

Results

Household return rates* by delivery method by ETC

0%

10%

20%

30%

40%

50%

60%

70%

80%

1 2 3 4 5 All areasETC

Post-out Hand delivery * As at 15 July

Success rates* at follow-up by delivery method by ETC

* As at 15 July

ETC Post-out Hand delivery

1 38.7% 39.9%

2 29.1% 29.4%

3 24.2% 26.1%

4 19.1% 18.4%

5 14.9% 16.3%

Overall 23.9% 24.3%

Delivery method – estimated cost savings

Estimated cost savings between 100% post-out and 100% hand delivery

Initial return* rate difference

(%’age points)

Estimated savings

5 £28m - £35m

6 £25m – £33m

10 £6m - £21m

15 -£18m - £1m

* At the start of follow-up – 23 May

Address register coverage

New addresses found during the 2007 Census Test by delivery method

During delivery During follow-up Total

Delivery method

No. %’age No. %’age No. %’age

Post-out 9 0.02% 478 0.94% 487 0.95%

Hand delivery 540 1.06% 181 0.36% 721 1.42%

Total 549 0.54% 659 0.65% 1,208 1.19%

Delivery method - conclusions

• Post-out has an impact on return rates, minimal impact which can be addressed through additional follow-up.

• A post-out methodology will provide significant savings (£25-£33m) to invest in targeted follow-up and community liaison.

• Improvements identified for the address register and follow-up procedures suggest that the levels of AR undercoverage will be small and manageable.

Decision: In E&W, post-out will be the primary means (at least 85%) of delivering questionnaires in 2011.

Income – Setting the scene

Including income depends on:• results from the Test; and,• consultation on other topics and relative priority of

income in relation to other demands.

Further analysis required as follows:• Quality and accuracy of responses to income

question• Item imputation rates• Public perception• Other data sources

Household return rates* by income/no income question by ETC

0%

10%

20%

30%

40%

50%

60%

70%

1 2 3 4 5 All areas

Income No Income * As at 15 July

2007 Census Test - Key findings

Delivery method• In E&W, post-out will be the primary method (at

least 85%) of delivering questionnaires in 2011.

Address register development• Address checking will be required for 2011,

currently planning 100% for E&W.

Outsourcing recruitment, training and pay:• Worked well in the Test and are considering

outsourcing for 2011;

Other key findings:

Recruitment:• More difficult than expected in some areas, mostly delivery

enumerators.• Good MI to identify and manage problems early.Training:• Some development issues with e-learning but overall well

received/effective.Pay:• Much simpler system defined thereby avoiding some of the

problems experienced in 2001.• Hourly pay worked well, provided good control and

flexibility; travel expenses still considered cumbersome. Refinement for 2011 required.

Other key findings:

LA Liaison:• Principles and benefits of LA Liaison proven.• Variable engagement across the LAs – some more engaged than

others • Going forward we need to consider:

– methods for achieving more consistency across LAs; – Making it simpler for ONS and LAs– Achieving Chief Exec buy-in.

Follow-up:• Transfer of information held centrally to field staff worked well within the

constraints of the Test.• A good start to follow-up is imperative – need to review start dates.• Organisation and management of field staff worked well, but more

development required on doorstep interaction to convince respondent to respond.

2007 Census Test Evaluation– planned publications

Publication

Statistical Evaluation of the 2007 Census Test

2007 Census Test - Evaluation of the delivery method

2007 Census Test - Evaluation of the Income question

Evaluation of 2007 Census Test – Summary Report

Did you know?

Some other key Census developments:

• Rehearsal - Spring 2009• Route A – contract to be let in January 2008• Route C – start procurement in January 2008• Finalising questionnaire – Spring 2008• White Paper – Autumn 2008

Questions ????

Household return rates* by delivery method by LA

0%

10%

20%

30%

40%

50%

60%

70%

Post-out Hand delivery * As at 15 July

Household return rates* by income/no income question by LA

0%

10%

20%

30%

40%

50%

60%

70%

Cam

den

L'pool

Stoke

Bath

Carm

arthen

All areas

Income No Income * As at 15 July