Whom are we talking about?
Exclusion, noncoverage and nonresponse in
social statistics (and social surveys)
Ineke Stoop
Vienna, Q2014, June 2014
Whom are we talking about?
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
No landline phone
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Health problems
Lang
uage
prob
lems
No landline phone
No inte
rnet
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
No landline phone
No inte
rnet
Not in population register
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
No landline phone
No inte
rnet
Not in population register
No-go a
rea
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
No landline phone
No inte
rnet
Not in population register
No-go a
rea Noncontact
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
No landline phone
No inte
rnet
Not in population register
No-go a
rea Noncontact
Mentally or physically unable
Vienna, Q2014, June 2014
Whom are we talking about?
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
No landline phone
No inte
rnet
Not in population register
No-go a
rea Noncontact
Mentally or physically unable
Refu
sal
Vienna, Q2014, June 2014
People living in The Netherlands
Non-citizens
Age limits
Non-re
siden
tial h
ouseh
olds
Abroad longer period
Lang
uage
prob
lems
No landline phone
No inte
rnet
Not in population register
Noncontact
Mentally or physically unable
Refu
sal
Whom are we talking about?
Conduct an interview
Vienna, Q2014, June 2014
ContentWhat do we want?• Allow all members of the target population to participate in
our surveyDo we know what is happening?• Exclusion mechanisms• Availability of information on survey data collection process
What can we do?• Better documention• Become more inclusive
www.europeansocialsurvey.orgVienna, Q2014, June 2014
Aims ESS
• To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions
• To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond
www.europeansocialsurvey.orgVienna, Q2014, June 2014
ESS top data user countries (>100) (March 2014) (N=68,204)
Luxembourg
Korea, Republic of
Croatia
Cyprus
Australia
Philippines
Slovakia
Canada
Romania
Israel
Greece
Russian
Ireland
Finland
Austria
Switzerland
Italy
Spain
Norway
Netherlands
Belgium
0 1000 2000 3000 4000 5000 6000 7000
www.europeansocialsurvey.orgVienna, Q2014, June 2014
Strategy European Social Survey
• Face-to-face interviewing• CAPI/PAPI
• Random sampling• Different sampling frames
• High coverage• Translation +5% language minorities• But
– Exclusion non-residential population– Exclusion far away areas (Caribbean islands)
• Nonresponse• High target response rates: 70%• Specification of efforts, guidelines
CAPI
PAPI
www.europeansocialsurvey.orgVienna, Q2014, June 2014
Response rates ESS (selection all rounds)
Switzerl
and
Spain UK
German
y
Belgium
Norway
Denmark
Netherl
ands
Portug
al
Sweden
Sloven
ia
Finlan
d
Poland
0
10
20
30
40
50
60
70
80
2002 2004 2006 2008 2010 2012
Vienna, Q2014, June 2014
Exclusion, noncoverage, nonresponse, cultural bias Different sampling frames
• Homeless• Illegal aliens• Opt-out register
Small language minority groups• No on-the-fly translation
Non-residential population• No upper age limit, but exclusion senior housing facilities• Cross-national differences
Not able due to mental or physical incapability• Survey mode
Cultural background• How important is it that immigrants have a Christian background?
Vienna, Q2014, June 2014
Response outcomes European Social Survey, R5
BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IL LT NL NO PL PT RU SE SI SK UA0
10
20
30
40
50
60
70
80
90
100interview noncontact refusal not able/other
Report on quality assessment of contact data files in Round 5: Final report 27 countriesHideko Matsuo and Geert LoosveldtWorking paper Centre for Sociological Research (CeSO) Survey Methodology CeSO/SM/2013-3
www.europeansocialsurvey.orgVienna, Q2014, June 2014
40.00 45.00 50.00 55.00 60.00 65.00 70.00 75.00 80.00 85.000.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
BE
BG
HR
CYCZ
DK EEFI
FR
DE
GR
HU
IE
IL
NLNO
PLPTRU
SK
SI
ES
SECH
UA
UK
Response R4 %
R5%
www.europeansocialsurvey.orgVienna, Q2014, June 2014
0.00 1.00 2.00 3.00 4.00 5.00 6.000.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
BE
BGHR
CY
CZ
DK
EE
FI
FR
DEGR
HU
IEIL
NLNO
PL PTRUSK
SI
ES
SE
CH
UA
UK
Language R4 %
R5%
www.europeansocialsurvey.orgVienna, Q2014, June 2014
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.000.00
1.00
2.00
3.00
4.00
5.00
6.00
BE
BG
HR
CY CZ
DK
EE
FI
FR
DE
GR
HU
IE
IL
NL
NO
PL
PT
RUSK
SIES
SE
CH
UA
UK
Not able R4 %
R5%
www.europeansocialsurvey.orgVienna, Q2014, June 2014
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.000.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
10.00
BE
BG
HR
CY
CZ DK
EE
FI
FR
DE
GR
HU
IE
IL
NL
NO
PL
PTRU SK
SI
ES
SE
CH
UA
UK
Unavailable R4 %
R5%
Vienna, Q2014, June 2014
Comparison official statisticsHealth surveys, LFS, EU-SILCMethodological information hard to findComparable response rates, effective sample size?• (EU-SILC 2010): “The effective sample size is not included
in the following table due to a lack of comparability among the methodologies used by countries for computing the design effect”.
Vienna, Q2014, June 2014
Response outcomes EU SILC 2010
AT BE BGCH CY CZ DE DK EE EL ES FI FR HRHU IE IS IT LT LU LV MT NL NO PL PT RO SE SI SK UK0
10
20
30
40
50
60
70
80
90
100interview Interview rejected Refusal to cooperate AwayUnable to respond Other reasons Noncontacted
2010 COMPARATIVE EU INTERMEDIATE QUALITY REPORT Version 3 – October 2012
Vienna, Q2014, June 2014
Undercoverage LFS
Eurostat, Quality report of the LFS 2012
People living in new dwellings Recent immigrants People without a fixed phone (LU, CH) Students in student halls People in non-private households Homeless people and other people without registered residence (e.g. people living in
huts, caravans) (DE). Hard-to-access groups are characterized either by extremely bad traffic conditions to
get to their place or by collective reluctance – usually within a small community – towards being interviewed (HU).
Population living in collective households or in dwellings outside the borders of built areas is not covered (EL).
New dwellings underrepresented in the sample, also population living in collective (institutional) households staying/or planning to stay in this places for over a year, homeless people and other people without registered residence (e.g. people living in huts, caravans) are out of the frame, emigrants staying abroad for more than one year (PL).
Substitution (IT) In many cases no or little information provided
Vienna, Q2014, June 2014
Exclusion mechanismsOutdated or incomplete sampling framesExclusion particular groups• Non-private households• Far away, no-go areas• Difficult
Nonparticipation• Not able for mental or physical reasons• Language problems
Survey mode• Functional illiteracy
Vienna, Q2014, June 2014
Do we know what is happeningNot reallyNon-standardised informationUnavailable informationMany groups drop out for different reasonEffect on statistics• The very elderly in the LFS• The illiterate in the LFS• The very elderly in health surveys
Vienna, Q2014, June 2014
General population Subgroups Minority group
Precision and bias
Comparing groups Specific survey
Vienna, Q2014, June 2014
1. General populationComplete population• Unemployment• Health• Political efficacy• Social participation
Survey problem• No identification necessary, as long as they take part• Precision: dependent on group size• Bias: Survey participation related to output variable
(through subgroup membership)
General population
Vienna, Q2014, June 2014
2. General population: subgroupsDistinctions• Socio-demographic groups, deviant groups, analysis
Survey problems• In surveys, registrations• Have to be identified• Comparing groups
Standardised design
Subgroups
Vienna, Q2014, June 2014
3. Specific groupTarget groupsSpecific topicsAdapted questionnaire
Survey problems• Sampling frame• Cooperation• Language• Culture• Measurement issues
Minority group
Vienna, Q2014, June 2014
Paying more efforts to make it possible that everybody can
participate
Vienna, Q2014, June 2014
Cons Pros• Efforts to include groups that
are not going to respond• Questionnaire not relevant• Costs• Small effect on final results• Responsibility respondent
• Completeness• Data quality• Cross-national comparability• Reporting on small subgroups• Fairness• Responsibility statistician,
researcher
Vienna, Q2014, June 2014
Thank you for your [email protected]
Top Related