Introduction to DDI with a Technical Perspective

7th Workshop of German Panel Surveys

01.02.2013

Thomas Bosch

M.Sc. (TUM)PhD student

boschthomas.blogspot.comGESIS - Leibniz Institute for the Social Sciences

Outline

2

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

Outline

• PhysicalDataProduct

• PhysicalInstance

• DDIProfile

• Colectica

Outline

4

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

General Information About Missy

• Missy (Microdata Information System)

• Provides detailed information about individual datasets

• Contains metadata of microdata relevant for the social sciences

• Facilitates data usage for research

• Currently consists of microcensus survey• Largest household survey in Europe

• statistics about• general population in Germany

• situation about employment market

(occupation, professional education, income, legal insurance, …)

5

General Information About Missy

• May be split in two parts• Missy Web (end-user front-end part)

• Missy Editor for documentation (back-end part)

• Consists of approx. 500 Variables & Questions per year

• Captures 25 years, since 1973

6

Missy 3

• That’s what it’s all about! The “next generation Missy”

• Integration of further surveys, e.g. EU-SILC, EU-LFS, …

• Implementation of Missy Editor as a Web-Application

7

Use-Cases in Missy

• General Information about survey 'microcensus'

• Variables by thematic classification and year

• List of variables by year

• List of generated variables by year

• Show details of a variable with statistics

• Variable-Time Matrix• List of variables by thematic classification and year (selectable)

• Questionnaire Catalogue

• In future: browse by survey and country!8

9

Use-Cases in Missy

• General Information about survey 'microcensus'

• Variables by thematic classification and year

• List of variables by year

• List of generated variables by year

• Show details of a variable with statistics

• Variable-Time Matrix• List of variables by thematic classification and year (selectable)

• Questionnaire Catalogue

• In future: browse by survey and country!10

11

Use-Cases in Missy

• General Information about survey 'microcensus'

• Variables by thematic classification and year

• List of variables by year

• List of generated variables by year

• Show details of a variable with statistics

• Variable-Time Matrix• List of variables by thematic classification and year (selectable)

• Questionnaire Catalogue

• In future: browse by survey and country!12

13

Use-Cases in Missy

• General Information about survey 'microcensus'

• Variables by thematic classification and year

• List of variables by year

• List of generated variables by year

• Show details of a variable with statistics

• Variable-Time Matrix• List of variables by thematic classification and year (selectable)

• Questionnaire Catalogue

• In future: browse by survey and country!14

Use-Cases in Missy

• General Information about survey 'microcensus'

• Variables by thematic classification and year

• List of variables by year

• List of generated variables by year

• Show details of a variable with statistics

• Variable-Time Matrix• List of variables by thematic classification and year (selectable)

• Questionnaire Catalogue

• In future: browse by survey and country!15

16

Use-Cases in Missy

• General Information about survey 'microcensus'

• Variables by thematic classification and year

• List of variables by year

• List of generated variables by year

• Show details of a variable with statistics

• Variable-Time Matrix• List of variables by thematic classification and year (selectable)

• Questionnaire Catalogue

• In future: browse by survey and country!17

18

Use-Cases in Missy

• General Information about survey 'microcensus'

• Variables by thematic classification and year

• List of variables by year

• List of generated variables by year

• Show details of a variable with statistics

• Variable-Time Matrix• List of variables by thematic classification and year (selectable)

• Questionnaire Catalogue

• In future: browse by survey and country!19

20

21

Flow Diagram

UML Class Diagrams

22

Outline

23

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

24

25

26

Modules

Outline

27

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

28

DDIInstance

DDIInstance

29

30

ResourcePackage

ResourcePackage

31

Allows packaging of any maintainable item as a resource item

Structure to publish non-study-specific materials for reuse

ResourcePackage

32

Any module (except: StudyUnit, Group, LocalHoldingPackage)

33

Any Scheme (Banks)

Internal or external references to schemes and elements within schemes

ResourcePackage

Group

34

Group

35Groups allow inheritance

Group

36

Organization

37

Organization

38

Outline

39

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

Comparison (DDI 3.1)

1. ResourcePackage

2. Group

3. Comparison

40

Comparison (MISSY)

41

42

Comparison (DDI 3.1)

1. ResourcePackage

2. Group

3. Comparison

43

References to same contents of ResourcePackages

44

Comparison (DDI 3.1)

1. ResourcePackage

2. Group

3. Comparison

45

46

label = 'variable age'

label = 'age'

Group (study and variable information)

--> variables in this group are comparable

--> inherit

--> overwrite variable information

+ easy to understand

- Implementation

- changes --> reorganize groups

Comparison (DDI 3.1)

1. ResourcePackage

2. Group

3. Comparison

47

Comparison

48

Comparison

Comparison

VariableMap

CategoryMap

CodeMap

ConceptMap

QuestionMap

UniverseMap

Why not more? good understanding of these elements

49

50

Variable-Age Variable-AgeGroup

codes/categories codes categories

20 1 20 – 29

… … …

29 1 20 – 29

30 2 30 – 39

… … …

67 6 60 – 67

Comparison (Example)

VariableMap

Variable-Age --> Variable-AgeGroup

commonality, difference

51

CodeMap

commonality, difference

52

Variable-Age Variable-AgeGroup

20 1

… …

29 1

30 2

… …

67 6

CategoryMap

commonality, difference

53

Variable-Age Variable-AgeGroup

20 20 – 29

… …

29 20 – 29

30 30 – 39

… …

67 60 – 69

Outline

54

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

55

StudyUnit

56

Coverage

Coverage

57

TemporalCoverage

58

TopicalCoverage

59

SpatialCoverage

60

SpatialCoverage

61

TopicalCoverage

62

TemporalCoverage

63

64

Citation

Outline

65

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

ConceptualComponent

66

ConceptualComponent

ConceptualComponent

67

ConceptScheme

68

ConceptScheme

69

Concept

70

Thematic Classification (MISSY)

71

Concept

72

Universe

73

Universe

74

Outline

75

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

76

LogicalProduct

77

LogicalProduct

78

CodeScheme

CodeScheme (MISSY)

79

CodeScheme

80

Code

81

Code (MISSY)

82

Code

83

84

CategoryScheme

CategoryScheme (MISSY)

85

CategoryScheme

86

Category

87

MISSY (Category)

88

Category

89

90

VariableScheme

VariableScheme (MISSY)

91

VariableScheme

92

Variable

93

Variable (MISSY)

94

Variable

95

Representation

96

Representation

97

Outline

98

• MISSY

• DDI-L Main Structures

• DDIInstance

• Comparison

• StudyUnit

• ConceptualComponent

• LogicalProduct

• DataCollection

99

DataCollection

DataCollection

100

101

QuestionScheme

QuestionScheme

102

103

QuestionItem

QuestionItem (MISSY)

104

QuestionItem

105

106

Response Domain

Response Domain

107

108

Instrument

Instrument (SOEP)

109http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf

Instrument

110

111

ControlConstruct

ControlConstructScheme

112

ControlConstruct

113

Sequence

114

Sequence (bahavioral)

115

Sequence (SOEP)

116http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf

Sequence (structural)

117

118

StatementItem

Explanatory background related to questions

StatementItem (behavioral)

119

StatementItem (SOEP)

120http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf

StatementItem (structural)

121

122

QuestionConstruct

QuestionConstruct (behavioral)

123

QuestionConstruct (SOEP)

124http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf

QuestionConstruct (structural)

125

126

IfThenElse

IfThenElse (behavioral)

127

IfThenElse (SOEP)

128http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf

IfThenElse (structural)

129

130

Loop

Loop (behavioral)

131

Loop (SOEP)

132http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf

Loop (structural)

133

134

RepeatWhile

RepeatWhile (behavioral)

135

RepeatWhile (structural)

136

137

RepeatUntil

RepeatUntil (behavioral)

138

RepeatUntil (structural)

139

Outline

• PhysicalDataProduct

• PhysicalInstance

• DDIProfile

• Colectica

141

PhysicalDataProduct

PhysicalDataProduct

142

• Physical layout of data that do not change when physical instance data files are merged or split

• Physical Data Structure

Outline

• PhysicalDataProduct

• PhysicalInstance

• DDIProfile

• Colectica

144

PhysicalInstance

145

PhysicalInstance

• Physical instances of physical

data products

• Represents data files

Outline

• PhysicalDataProduct

• PhysicalInstance

• DDIProfile

• Colectica

147

DDIProfile

148

DDIProfile

149

DDIProfile

Describes which DDI-L elements you use in your institution’s DDI flavor

Outline

• PhysicalDataProduct

• PhysicalInstance

• DDIProfile

• Colectica

Colectica

151

• Commercial software tool

• http://colectica.com/

• Multiple versions for individuals, departments, and institutions

• Multiple videos on the website for getting started

Colectica Reader For Free

152

Colectica (Navigation)

153

Colectica (Variables)

154

Colectica (Variables)

155

Colectica (Questions)

156

Colectica (Codes & Categories)

157

Feel free to ask any unsolved questions!

158

Thomas Bosch

M.Sc. (TUM)PhD student

boschthomas.blogspot.comGESIS - Leibniz Institute for the Social Sciences