Census Microdata: findings and futures

Manchester 1 – 3 September 2008

The future of microdata

Professor Denise Lievesley UN African Centre for Statistics

Addis Ababa, Ethiopia and President, International Statistical Institute

Graduating from being data producers to generators of information and knowledge

attention to data collection at expense of generation of information and knowledge– collection costly and difficult– importance of quality of data

mountains of data – insufficiently processed and analysed

most people not adept at understanding data– even more important for statisticians to get

involved in interpretation and use of information

Partnerships for data use

with– subject experts– data analysts– researchers in government, universities,

private sector and civil society

who can contribute to development of data production systems in countries

this requires making census and survey data accessible to these stakeholders

The importance of international guidelines on ensuring data access

The UN Principles and Recommendations for the 2010 Population and Housing Census urge countries to create census databases as part of the process of census data management.– “in order to expand the life and usability of

data, and as a complement to the standard production of tables, NSOs are encouraged to store census data in various computerized database forms” including macro and micro- databases.

These recommendations give one of the main advantages of micro databases as “permitting the retrieval of data at least in principle, at any level of detail”.

Additional advantages:– to broaden data use and reuse;– to foster diversity and deepen the quality of

data analysis thereby extracting more information from the data;

– to add value to data by bringing subject-matter knowledge to data analysis;

– to improve data quality (Data analysts can and often do detect errors in data and when they provide feedback to statistical agencies, this can lead to improvements in future data collection.)

According to the International Household Survey Network established as one of the six action points of the Marrakech Action Plan for Statistics, national and international micro databases should be established to:– promote the acquisition, documentation,

dissemination and preservation of microdata essential for the production of national statistics, for research and for instruction in the social sciences,

– promote the effective use of existing survey and census data,

– ensure the continued viability and usability of microdata now and in the future, and,

– promote equitable access to these data within the framework of the national statistical legislation.

Obstacles

Despite these international guidelines and exhortations for data access, obstacles do exist and we must be sensitive to them. They include:– legal obstacles– technical and financial obstacles including in-

house capacity to handle the complex aspects of micro-data dissemination such as data anonymization

– political obstacles– psychological obstacles: the tendency to

control access perhaps because of concerns over its mis-interpretation or because ‘data is power’

Legal constraints

in many countries the statistical legislation in use is out dated and does not recognise the dissemination of electronic data particularly micro-data

some legislation, or the interpretation of it, actually prevents such dissemination on account of confidentiality

new legislation needed– the African Centre for Statistics is working with

countries to help them to prepare both legislation and professional frameworks relevant to today’s era of electronic information.

Data are collected using valuable resources – both financial but also the time of survey and census respondents – and are increasingly seen as being a vital part of democratic systems, since people are empowered though information.

Official data are a “public good”: part of the enabling environment for national and international development, which all stakeholders in society should have access to and benefit from.

Conclusion - I

It is the responsibility of statisticians to ensure that the widest possible use is made of data; consistent of course with the legal constraints and ethical undertakings.

Preservation is essential Having collected data at some cost to the

taxpayer, it behoves official statisticians to manage them well.

Alongside dissemination, this entails data preservation.

Due to poor data management, human error as well as technical change and inadequate use of technology, many data sets including critical census data are no longer readable.

Thus all that remains of this important legacy are the, often quite superficial, reports that were produced at the time.

To this extent an important part of our heritage is lost and we will be severely limited in our analysis of change.

Long term preservation of electronic material is not a straightforward task, especially in resource-poor and technology-weak developing country statistical offices.

It can be hard to persuade financial authorities to spend money on the preservation of data for historians and statisticians of the future, when there are so many pressing problems today.

Partnerships for data preservation

To this end, partnership - for both technical work and advocacy – across the data archiving, data librarian, statistical and research communities is to be encouraged.

Welcome the formation of new organisations such as the African Association of Statistical Data Archivists

Value of international networks and support systems

Metadata It is necessary not only to preserve

data but also to create and preserve metadata and contextual information. This is essential to ensure that the interpretation of the data will be informed.

The documentation should include– data collection instruments and forms– instruction manuals– definitions and concepts– descriptions of scope and coverage and

other aspects of quality– codebooks– basic tables– records of validation and post-enumeration

checks

Documentation

The documentation should be clear and easy to understand and should help users to:– identify and find the data they are

interested in– understand what the data are measuring

and how they have been created– assess the quality of data and fitness for

their purpose

International standards for metadata creation such as the DDI should be implemented

Conclusion - II

The leadership of the national statistical agencies should ensure that census and survey macro- and micro-data are well documented and archived.

Case made for data dissemination and preservation.

Data alone have no “intrinsic value”.– Their value is “extrinsic”: derived from the fact that

they can be disseminated to those who need them, are understood and are used for a variety of purposes in a timely fashion.

– So one way to justify the high cost of data production is to disseminate and mine them for information

– especially for evidence-based policy and comparative analysis

Data dissemination in a variety of ways (through reports, compilations of data, electronic extracts etc) is, therefore, important not only to complete the data cycle but also to enhance data relevance and usability.

Data grow in value the more they are used, unlike most commodities which are diminished with use.

Evidence-based policy

What is evidence-based policy? Need for use evidence at all stages in the

policy cycle Why is evidence-based policy important? Evidence but one input into policy process What is evidence? Evidence initiatives Challenges for statisticians Examples from the developing world

What is evidence-based policy ?

Helping people to make well-informed decisions about policies, programmes and projects, by putting the best available evidence from research at the heart of policy development and implementation

Enlightening through making explicit what is known through scientific evidence and importantly what is not known

Better statistics, better decisions, better outcomes Vision of the African Centre for Statistics

In contrast to opinion based policy

which relies heavily on – either the selective use of information– or the untested views of individuals or

groups often inspired by ideological standpoints, prejudices or speculative conjecture.

Need an evidence base at all stages in the policy cycle

in shaping agendas, in defining issues, in identifying options, in making choices of action, in delivering them and in monitoring their impact and outcomes.

Context in the UK

Modernising Government White Paper commits to policy making that is – strategic – outcome-focussed – joined up (works across organisational boundaries) – inclusive (is fair and takes account of the interests of

all – gender sensitive, promotes equity) – flexible – innovative (means taking risks – and so risks must be

identified, monitored and managed)– robust (works and continues to work)

Tackles causes not symptoms. Is acceptable to the public.

Why is evidence-based policy important?

Sherlock Holmes “it is a capital mistake to theorise before you have all the evidence. It biases the judgement”

Policy makers may be well-intentioned but misguided

Results of a meta-analysis

Collation of the results of many studies contradict this advice

Extract from publicity prepared for the UK ‘Reduce the Risk’ Campaign (early 1990s)

“The risk of cot death is reduced if babies are not put on the tummy to sleep. Place your baby on the back to sleep. ….Healthy babies placed on their backs are not more likely to choke.”

Iain Chalmers

“No doubt like millions of his other readers, I passed on and acted on this apparently rational and authoritative advice.”

“We now know that the advice promulgated so successfully in Spock's book led to thousands, if not tens of thousands, of avoidable cot deaths.” (Letter to BMJ)

Judging what works

Outcomes often defined too narrowly eg performance management whereby targets are often set ‘top down’ and may have unintended consequences.– Hitting the target but missing the point– The good, the bad and the ugly report of the

Royal Statistical Society– Difficulty is that sometimes different things work

in different circumstances and this is interpreted as nothing works.

(Ref Robert Martinson’s 1974 review of American offender rehabilitation programmes- He concluded that there was no single approach which worked consistently – different things worked in different circumstances but this was misread as ‘nothing works’.)

The policy making processPolicy making is the process by which governments translate their political vision into programmes and actions to deliver desired changes in the real world.

Evidence but one input into policy process

Ideology

Lobbies

Values/beliefs

Tradition

Self interest

Judgement

Campaign promises

Expert viewsExperience Resource

constraints

Acceptabilityto public

“ There is nothing a government hates more than to be well-informed: for it makes the process of arriving at decisions much more complicated and difficult. ”

John Maynard Keynes

What is evidence?

Expert knowledge; published research; existing statistics; stakeholder consultations; previous policy evaluations; the Internet; outcomes from consultations; costings of policy output from economic and statistical modelling. UK Cabinet Office 1999

Scientific, rigorous, critically appraised, well documented

Fit for purpose Beware policy-based evidence? Unbiased (note problems of publication

bias) But data are never value-free

All scientific evidence is imperfect.

“The absence of excellent evidence does not make evidence-based decision making impossible: what is required is the best evidence available not the best evidence possible”

Muir Gray 1997

“ Evidence rarely provides neat and tidy prescriptions to decision makers as to what they should do. Often it generates more questions to be resolved ”

Petrosino et al 2001

Challenge : is it essential that evidence is based on experiments?

Adrian Smith ‘Mad cows and ecstasy : choice or chance in evidence-based society’

Experiments used too little in social and economic research

Rich data from different sources Science is often misrepresented as the body

of knowledge acquired by performing replicated controlled experiments – it is much broader – the acquisition of reliable knowledge about the world

Jared Diamond Collapse

Challenges for analysts

increase understanding of the policy process, where and how evidence can feed into it (eg SPATS)

improve interpretation and communication of data particularly in relation to uncertainty

speak the language of policy makers forge strong relationships of official

statisticians with policy analysts, increase number of policy analysts

combine humility and confidence improve training for analysts

– (International Research Forum on statistical reasoning, thinking and

literacy http://www.srtl.stat.auckland.ac.nz)

Particular challenges with respect to timing

demand for quick fixes means that policy makers often do not have time for in-depth research – “Demands by Ministers are short term, quick fix

solutions rather than major carefully considered strategic initiatives” (Bullock et al)

collection of quality data takes time – hence the existence of public use samples can accelerate research

to be effective the research strategy needs to look beyond the timescales of one government

many issues are chronic – beware “unwarranted impatience” (Boruch)

Evidence compilations – example initiatives

Micro-data based on samples from Censuses

Data archives Cochrane collaboration Campbell collaboration National Library of Health Communities of practice

Communities of practice

European social survey CROP - the Comparative Research

Programme on Poverty whose major aim is to produce sound and reliable knowledge, which can serve as a basis for poverty reduction

RENCORE - encourage and enhance comparative empirical research of individual, national and institutional level data from the states of western, central and eastern Europe

Cleveland conference on education research

African Programme on Rethinking Development Economics

Evidence base urgently needed in developing countries

Problems are severe and urgent Need ownership by countries… and to empower them Develop policies which are relevant to

their needs… and ensure effective implementation Need to counteract corruption Paucity of data is a major obstacle Lack of recognition /acceptance of

much data

Joint Marrakech Memorandum

Affirm a commitment to fostering a global partnership on managing for development results. Awareness is growing that getting better development results requires management systems and capacities that put results at the center of planning, implementation and evaluation. We need to align cooperation programs with desired country results, define the expected contribution of our support to country outcomes and rely on and strengthen countries’ monitoring and evaluation systems to track progress and assess outcomes … better distil the lessons of countries’ experiences and disseminate knowledge about what gets results in different country contexts.

C.Scott for Paris 21Use of good statistics having a positive effect on policy

Uganda– Poor public service delivery caused by

government’s failure to ensure that budgeted funds reached front line agencies

Brazil and Mexico– Tackled child poverty and education by a

programme to give child benefits to mothers according to the attendance of their children in school

C.Scott for Paris 21Absence of data or failure to use available information has negative effect on policy making

Malawi– Data from Save the Children on child malnutrition

disregarded by government because of conflict with crop data

Botswana– Deficiencies in data on HIV/AIDS (from sentinel

surveillance systems) legitimised the rejection of the message on the scale of the problem

Current challenge in policy development in poor countries Development literature of the 1990s dominated by

the view that growth is central to any strategy aimed at poverty reduction.

Since modified: not growth per se but structure of growth that matters.

Further recognised that income inequality matters when making progress on poverty reduction.

Example Nigeria– Aigbokhan asks why the rate of poverty is so high in

Nigeria despite the strong growth performance and concludes it is because of a lack of explicit concern with inequality in public policy. He analyses data on Nigeria and on other countries for comparative purposes to argue that if growth occurs in the sectors that require skills that the poor of Nigeria do not possess although the economy grows it has little impact on poverty reduction.

– Essential that access is provided to researchers which allows them to examine the distribution of resources within countries

The gross national product does not allow for the health of our children, the quality of their education, or the joy of their play; it does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials. It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short except that which makes life worthwhile.

Robert Kennedy

Comparative research – building of cross-national resources

Knowledge of the self is gained through knowledge of others

Tension between the value and importance of cross-national data and its fragility

To what extent are differences artefacts of the measurement ?

Benefits of cross-national research

Contacts, enabling researchers to capitalise on their experience and knowledge of different intellectual traditions and to compare and evaluate a variety of conceptual approaches.

Comparisons can lead to fresh, exciting insights and a deeper understanding of issues that are of central concern in different countries - the identification of gaps in knowledge and sharpen the focus of analysis of the subject under study by suggesting new perspectives.

Cross-national projects give researchers a means of confronting findings in an attempt to identify and illuminate similarities and differences, not only in the observed characteristics of particular institutions, systems or practices, but also in the search for possible explanations in terms of national likeness and unlikeness.

Cross-national comparativists are forced to attempt to adopt a different cultural perspective, to learn to understand the thought processes of another culture and to see it from the native's viewpoint, while also reconsidering their own country from the perspective of a skilled, external observer.

Cross-national data is a special case of comparative research

Spatially comparative data – within countries– between countries

Collecting data over time (the past is a foreign country ) Many parallel issues in spatially and time

related comparative data such as achieving a balance between

specificity versus generalisability, and coping with differences/changes.

Purposes of cross-national data

To learn from one another (contrast and similarity) For purposes of national accountability (the

indicator movement) To build a greater global understanding through

comparison and through multiple instances of the same phenomena (Jared Diamond)

To aggregate across national boundaries for a regional or global picture– for advocacy– for resource mobilisation

To accelerate progress through sharing resources To make research more credible/ defensible

recognising that research which displeases is attacked rather than accepted

To distance the research from the political process (tension – policy relevance v. autonomy)

Challenges to comparability

Language Culture Social systems and structure Administrative systems Ideology and politics Economics and resources History Context – events Different methodologies and types of

methodological expertise (often deeply ingrained)

Different research climate

Over-interpretation

Jowell “Good science should not turn a blind eye to known imperfections – nor should these be concealed from users”– Concerns about lower standards being applied

for cross-national studies than for national studies

Goldstein (“Surprising naivety”) cross-national educational studies– Concerns about validity of cross-national

measures, inappropriate conclusions about causation, over-simplification of complex relationships

– Overall Goldstein argues for celebrating diversity and encouraging transparency

The UN model

Attempting to achieve comparable data across the world

Assumptions about universality of aims The survey ‘hammer’ is owned by richer

countries Under-resourced at UN and at country level Countries with resources play a bigger role (city

groups) Fragmentation within the UN decentralised

system (regionally and by substantive area) Concentrates on ‘facts’ and often assumes that

comparison is unproblematic Exhorted to “Think globally, act locally”

Problems with the UN model

One size fits all– Difficulty of accommodating national

differences– Cutting edge and trailing tail countries’

needs are very different– Context is critical but is often ignored

The nation is too often the unit of analysis– variation within countries ignored

Need to encourage country ownership and recognition of their data

Accountability of countries v. their autonomy

Combining administrative with survey and census data

Adds value to dataProvides contextEnables better understandingImproves communication – tells a better storyExcites and interests usersEngages the media

Summary

Case made for data preservation and dissemination

Data resources are critical for evidence based policy making

and for cross-national, comparative research

Welcome this meeting where we will discuss the value of this work, the challenges which face us and ways in which we can redouble our efforts to build these resources