Kees Mandemakers

Les Grandes Bases de Données et l’Histoire Sociale des Populations

Bordeaux, Université Michel de Montagne, 7-9th of February 2008

Development, database and results of the Historical Sample of the Netherlands

HSN

- What’s the HSN ?

- GENLIAS-index of civil certificates

- Best practices of large databases

- International data model

- Historical Sample of Europe

HSN

- What’s the HSN ?

- Sources and development database

- Projects

- Results

HSN

Five periods of Investment

1987-1990 Conceptual stage1991-1993 Pilot project province of Utrecht1993-1995 Extension pilot,

first collaboratory projects1996-2002 Extension to the whole country

2003-2007 Life courses 1850/1863 -1922

Data entry birth certificates and deaths, 1994-2006

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Births Total deaths Cards

0

5000

10000

15000

20000

25000

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Data entry marriage certificates, 1994-2006

Parental family

On the road lodger maid servant soldier

Own family

Typical Sequence of Entries in Population Registers

Status 44.000 Life Courses, 1st of January 2008

0

5000

10000

15000

20000

25000

MAIL CODED ENTERED FINISHED

HSN

IN CONNECTION WITH OUTPUT REGISTER 1850-1940

Adding Personal Card/List RP

Adding Marriage Certificates RP

HSN

Collaboratory projects

Extension datasets - Other kind of data - Other (related) persons

Upkeeping HSN-infrastructure

Widening circle HSN-users

Migration Dutch East Indies 1820 – 1940

Ulbe Bosma

Retrieving HSN-research persons

- Indian archives (Directories)- Registers Military, Civil Servants- Shipping Lists etc

Migration Dutch East Indies 1820 – 1940

Ulbe Bosma

Comparison Indian and Dutch Careers

(HSN Control Group)

HSN Publications and Presentations, 1991-2007

0

5

10

15

20

25

30

35

40

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

Publications

Presentations

Development of staff, HSN, 1991-2007

0

10

20

30

40

50

60

70

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

End of year

Total per year

HSN

Unique position HSN

. Sampling birth certificates . A whole nation (not regional)

. Following persons all over the country

HSN

Spanning Past and Present (SPAN)

. Combination NKPS (sample 1923-1985) . Completing Families RP’s . Sampling and Following Siblings . Sampling from Period 1923-1938 . Reinforcement technical infrastructure

(e.g. integrated SPAN-database)

HSN Future

. Life courses 1812-1862 . Back into 18th century . Simulating census 1850-1920 . New sources: land registers, tax registers,

poor law files, personal records army . Enrichment geo-referencing all addresses

HSN

Standard HSN-dataPARENTAL

family

OWN

family

Life career RP

HSN

Collaboratory projectsPARENTAL

family

1st

generation

Life career RP

2nd

generation

Life career 2nd RP

HSN

PARENTAL

family

OWN

family

Life career 2nd RP

OWN

familyLife career 1st RP

Collaboratory

projects

Index names from

2,5 million marriage certificates 1812-1922 70 %

4,0 million death certificates 1812-1950 40 %

2,0 million birth certificates 1812-1902 20 %

0,6 million from sources before 1812

Coverage by index ofmarriage certificates

NO T

COVERED

With occupational titles

Bride and groom

Lastname, First name, Age at marriage

Parents of bride and groom

Lastname, First name

Marriage certificates

Linking parents of bride and groom with their own certificate

between 15 and 50 years earlier

matching on combination of two persons

(last name and first name)

Marriage certificates

MarriageCertificate

BRIDE

GROOM

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

MarriageCertificate

BRIDE

GROOM

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

Linking three or more generations ??

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

Linking three or more generations ??They have already been linked !!!

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

BUT HOW ?????

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

EASILY

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

ANN

JOHN

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

A B

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

ANN

JOHN

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

A B

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

ANN

JOHN

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

A B

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

ANN

JOHN

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

ANN

JOHN

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

A B [= A] C [=B]

MarriageCertificate

ANN

JOHN

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

MarriageCertificate

BRIDE

GROOM

FATHER

MOTHER

FATHER

MOTHER

Chains of three generations

• Research on social mobility

• Marriages of cousins

• Marriages of uncles/aunts with nieces/newphews

. Passing down of the age of marriage

Marriage certificates

HSN

Standard HSN-dataPARENTAL

family

OWN

family

Life career RP

HSN

PARENTAL

family

OWN

family

Life career RP

HSN

PARENTAL

family

OWN

family

Life career RP

HSN

PARENTAL

family

OWN

family

Life career RP

HSN

PARENTAL

family

1st

generation

Life career RP

2nd

generation

3rd

generation

Life career 2nd RP

Life career 3rd RP

 Best practices with

large databases on historical populations

Historical Methods 37 (2004), nr. 1, 34-38.

History and Computing 13 (2001, published 2004), 2, 207-216

Kees Mandemakers and Lisa Dillon

Best practices with large databases

Purpose of the rules

- Articulate standards

- Trustworthy results

- Benefit from previous experience

- Quality standard

- Read as list of recommendations

Best practices with large databases

Three stages database-building

A Definition object and content

B Data entry, integration, standardization

C Enrichment and release data

Best practices with large databases

A Definition object and content

A1 Description universe database

A2 Whole source, unless…

A3 Documentation each included source

A4 Specific rules in case of samples

Best practices with large databases

B Data entry, integration, standardization

Components Database

SOURCES

CENTRAL DATABASE

DATA RELEASE

Best practices with large databases

B Data entry, integration, standardization

B5 Distinction original and corrected data

B6 Clearly distinction of inferred data

B7 Archiving in simple format (ASCI)

B8 Good back-up system

Best practices with large databases

C Enrichment and release data Values

C5 Standardized and original values

(international standards, e.g. HISCO)

C6 Negative standard values missing values

C7 Geo-referenced data

Best practices with large databases

C Enrichment and release Documentation

C8 List of model studies C9 Easy user versions (for ‘beginners’)C10 DDI-based meta-data C11 Website databaseC12 Progress reports

HSN

PARENTAL

family

OWN

family

Life career RP

RELEASES Dissemination data

HSN

PARENTAL

family

OWN

family

Life career RP

EASY to handle????

HSN

PARENTAL

family

OWN

family

Life career RP

NO, dynamic output

HSN

HSN

HSN

 

 

  

HSN

 

 

  

HSN

 

 

  

HSN

Tough nuts Dynamic outputUncertain datesCohort/period Definition householdLinking research persons

 

 

  

Dynamic Output

HSN

UNCERTAIN

DATES

HSN

Over 20 CODES on day number

33 Arrival date based on interpretation

35 Estimated declaration occupational title, religion etc

42 Estimated change of address

44 Estimated departure (group departing)

46 Estimated departure (based on census date)

HSN

COHORT &

PERIOD

HSN

1850 1860 1870 1880 1890 1900Life career RP

Life career RP

Life career RP

Life career RPLife career RP

Life career RPLife career RP

Life career RP

Life career RP

Life career RP

Life career RP

Life career RP

Life career RP

Life career RP

HSN

DEFINITION HOUSEHOLD

or where is my RP?

HSN

PARENTAL

family

OWN

familyLife career 1st RP

HSN

PARENTAL

family

OWN

family

OWN

familyLife career 1st RPLife career 2nd RP

HSN

PARENTAL

family

OWN

family

OWN

familyLife career 1st RPLife career 2nd RP

OWN

family

Life career 3th RP

HSN

PARENTAL

family

OWN

family

OWN

familyLife career 1st RPLife career 2nd RP

OWN

family

OWNfamily

Life career 3th RP

HSN

Complicated but

TWO SOLUTIONSInternationalisingSimplifying

DATASETS for ANALYSIS

Social Mobility

Fertility

Mortality

Migration

Etcetera

Population registers

Census records

LandregistersCivil Certificates

Church records

SOURCES

Etcetera

DATABASES

England &* Wales

Netherlands

Sweden

Switzerland

Spain

Italy

Poland

Pop. registers 1850-1910

Family Cards1910-1940

Civil Certificates

Individual Cards1940-present

SOURCESDATASETS for

ANALYSIS

Social Mobility

Fertility

Mortality

Migration

Etcetera

Italy

Italy

Sweden

Belgium

Landregisters

HOW TO BUILD MY DATASET ??

Pop. registers 1850-1910

Family Cards1910-1940

Civil Certificates

Individual Cards1940-present

SOURCESDATASETS for

ANALYSIS

Social Mobility

Fertility

Mortality

Migration

Etcetera

HOW TO MANAGE DATA FLOW ??

Italy

Italy

Sweden

Belgium

Landregisters

Pop. registers 1850-1910

Family Cards1910-1940

Civil Certificates

Individual Cards1940-present

SOURCESDATASETS for

ANALYSIS

Social Mobility

Fertility

Mortality

Migration

Etcetera

HOW TO MANAGE DATA FLOW ??

Italy

Italy

Sweden

Belgium

Landregisters

HOW TO BUILD MY DATASET ??

Pop. registers 1850-1910

Family Cards1910-1940

Civil Certificates

Individual Cards1940-present

SOURCESDATASETS for

ANALYSIS

Social Mobility

Fertility

Mortality

Migration

Etcetera

HOW TO MANAGE DATA FLOW ??

Italy

Italy

Sweden

Belgium

Landregisters

HOW TO BUILD MY DATASET ??

Pop. registers 1850-1910

Family Cards1910-1940

Landregisters

Civil Certificates

Individual Cards1940-present

SOURCES

Etcetera

DATA-HUBS

Sweden Lund

Spain

Sweden Umea

Italy

HSN

Poland

Pop. registers 1850-1910

Family Cards1910-1940

Landregisters

Civil Certificates

Individual Cards1940-present

SOURCESDATASETS for

ANALYSIS

Etcetera

Social Mobility

Fertility

Mortality

Migration

Etcetera

DATA-HUBS

Sweden Lund

Spain

Sweden Umea

Italy

HSN

Poland

Pop. registers 1850-1910

Family Cards1910-1940

Landregisters

Civil Certificates

Individual Cards1940-present

SOURCESDATASETS for

ANALYSIS

Etcetera

Social Mobility

Fertility

Mortality

Migration

Etcetera

DATA-HUBS

Sweden Lund

Spain

Sweden Umea

Italy

HSN

Poland

IN TERMEDIATE

STRUCTURE

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCESDATASETS for

ANALYSIS

Etcetera

Social Mobility

Fertility

Mortality

Migration

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec

IN TERMEDIATE

STRUCTURE

DATA DATA

??

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCES

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec

TASK EACH DATABASE

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCES

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec

LINKING appearances of persons to individuals

TASK EACH DATABASE

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCES

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec

LINKING appearances of persons to individuals

DATING attributes

TASK EACH DATABASE

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCES

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec

LINKING appearances of persons to individuals

DATING attributes

STANDARDIZING attributes (HISCO, GIS, etc)

TASK EACH DATABASE

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCES

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec

LINKING appearances of persons to individuals

DATING attributes

STANDARDIZING attributes (HISCO, GIS, etc)

ALL META-DATA

TASK EACH DATABASE

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCES

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec

LINKING appearances of persons to individuals

DATING attributes

STANDARDIZING attributes (HISCO, GIS, etc)

ALL META-DATA

TASK EACH DATABASE

at least

HSN

Simplifying withHSN data-machine

flat structureonly main datano uncertaintydocumentation per variable

HSN

Simplifying

HSN data machine

with UmeåHistorical Sample of Europe[Clio_infra project]

Historical Sample of the Netherlands

InternationalInstitute forSocial History

Problems for researcher

Problem of time-dependent data

- more values same variable- changing environment (family

structure cause as such)- age / period /cohort

Problems for researcher

. Selection (movers / stayers/ disappearing)

. Fuzzy or no dating

. Household definition

. Multi-level data

. Need for rectangularization

. Analysing in a comparative way

Population. registers

Family Cards

Landregisters

Civil Certificates

Individual Cards

SOURCES

Etcetera

DATABASES

UTAH

Spain

Sweden Umea

Italy

HSN

Québec