Quality strategies in cross- national surveys The case of the European Social Survey Ineke Stoop.

Quality strategies in cross-national surveys

The case of the European Social Survey

Ineke Stoop

www.europeansocialsurvey.org

European Conference on Quality in Official Statistics 8-11 July 2008

Quality and comparability

Harmonisation strategies and cross-national surveys

Optimal comparability

European Social Survey

Nonresponse



Harmonisation strategies(Körner and Meyer, 2005)

Harmonised concept

Harmonised concept

Harmonised concept

Measurementprocedure



National survey/Specific concept



Input harmonisationEx-ante outputharmonisation

Ex-post outputharmonisation



Type of harmonisation

Ex post output harmonisation• Use existing sources

Ex ante output harmonisation• Set up new survey (or develop new instrument)

Output harmonisation• Best national quality• Or, national survey tradition• But, what about optimal comparability

Input harmonisation• Design new survey• Optimal comparability• Or, are identical methods and instruments really equivalent in different countries?• But, what about optimal quality



Strategies in comparative survey

Quality

Optimal national approach• Comparability?• Poor performance

accepted

Some restrictions and standards

Consistency

Identical approach• Lowest standards

• Possible?• Pull down level

• Highest standards• Possible?• Pull up level

High standards and optimal national approach



Mixed approach

Focus on those key aspects that may hamper comparability• Data collection mode• Sampling approach• Response rates• Question formulation



European Social Survey

Attitudes, values and beliefs

Bi-annual: 2002, 2004, 2006, 2008

Fixed core, rotating modules

Effective size 1500 or more

Central co-ordination

National implementation• 30 National Coordinators responsible for fieldwork

Face-to-face



Aims European Social Survey

To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions

• Provide data on beliefs, attitudes and values for scientific and policy making purposes

• Measure attitude change in a changing Europe

To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond

• Collect data according to highest standards

• Generate methodological research

• Develop and disseminate new best practices

• Develop and improve social survey research infrastructure in Europe



EVERYTHING DOCUMENTED

EVERYTHING AVAILABLE



Nonresponse: consistency

Strict probability sampling

No substitution

Fieldwork monitoring

Guidelines response enhancement• Incentives• Brochure

Guidelines interviewer training

Target noncontact rate: 3%• Minimum number of calls• Timing of calls (evening,

weekend)

Target response rate: 70%• Refusal conversion

suggested

Contact form and call records

Standard response rate calculation



Nonresponse: divergence

Population• Residential population

• Minority languages

Sampling frame• Addresses, households

• Individuals

At home pattern• Female employment

• When does the evening start?

Fieldwork organisation• Field director’s meeting

Interviewers• Experience

• F2F

• Random sampling

• Training and briefing

• Remuneration• Response, hours, staff

Mode of contacting

Use of advance letters

Incentives

Contact forms



Response rates ESS (deviation 70%)



Why do response rates differ?

Survey climate• Used to surveys?

Popularity topic• Europe?

Country size, urbanicity and contactability

Survey modes• Predominantly telephone?

Fieldwork efforts



UP TILL NOW

NO UNIVOCAL PREDICTOR

OF

RESPONSE RATES

ACROSS COUNTRIES



% noncontact and refusal (R1, R2, R3)

% noncontact

% r

efu

sal



Age, gender, education (Vehovar & Zupanic)

Age: underrepresented• 35+ (Austria)• 55+ (Belgium and Luxemburg)• 15-34 (Spain, Ireland, Netherlands, UK)

Education: underrepresented• Middle (Austria, France, Iceland, Luxembourg, the Netherlands, Norway, Sweden

and the United Kingdom)• Less (Switzerland, Czech Republic, Germany, Estonia, Hungary, Norway,

Slovenia, Slovakia and Ukraine)



Results multivariate analysis: reluctance(Beullens, Billiet and Loosveldt)

Switzerland• Members of larger families

Germany• Female, aged, city dweller• Internet, unemployment history• Less political participation

Estonia• Female, not in village• Employed

Slovakia• Older, average education, church

life• Never job, less safe, comfortable

family income

Netherlands• Female, average education, more

TV, internet• Paid job, healthy• Immigrants threat, trust political

institutions• Civil obedience, political

participation• Socially isolated, dissatisfied



Nonresponse bias in cross-national studies

How to measure bias?• Different type of auxiliary information available

• Sample frame/register information?

• Reluctant respondents?• Neighbourhood information?

Different size of bias?

Different type of bias?

Different bias over time?

ESS-experiments in several countries• Mixed results



Is output harmonisation better?

Maybe, but in that case do you know about• Response rates (standard calculation)• Intensity and effect of field efforts• Nonresponse composition• Underrepresented groups• Nonresponse bias

Black box:• Differences between countries due to methodology or

substance?



Are comparative surveys possible?

If not, national surveys are not possible either• Subgroups

• Socio-demographic• Survey interest• Topic relevance

• Regions• National languages• Interviewers

There is so much we don’t know, and now we know so much more



Thank you for your attention

Quality strategies in cross- national surveys The case of the European Social Survey Ineke Stoop.

Documents

Transcript of Quality strategies in cross- national surveys The case of the European Social Survey Ineke Stoop.