Zen of metadata 09212010

The Zen of MetadataScott NorthrupData Architect

September 21, 2010

biography

I work for CCSF DPW

(City and County of San Francisco

Department of Public Works)

I’m a DA

(that’s Data Architect)

with no particular credentials

except 30+ years experience in database design

HomERwin

biography continued

• Prior roles

– Analyst – business & operational

– Programmer – at least 6 languages

– Network tech

– Project manager – SDLC

– Data modeler – logical & physical

• Current roles

– Data Architect for Data Governance and MDM

– Metadata manager – data dictionary

– Data standards admin

agenda

– Intro and environment

– DATA

– DATA and METADATA

– Business Requirements

– Metadata types and philosophies

– CIA’s compilation and usage of Metadata

– Some examples and caveats

© 2010 Wells Fargo Bank, N.A. All rights reserved.

ZEN…

…emphasizes experiential wisdom, discernment, or understanding, particularly as realized in the form of meditation for the attainment of enlightenment…

??? Inmon or Kimball ???

who cares…

! BEST PRACTICES !


application environment

• Teradata

• ERwin

• E/R Studio

• Ab Initio

• Microstrategy

• SQL Server


data environment

• Data Warehouse

– 70 databases from 62 source systems

– DSL – Data Source Layer

• 62 databases

– DIL – Data Integration Layer (Data Mart)

• 3 databases

– DAL – Data Application Layer

• 5 databases

– Aggregates & Composites


classic rock

A little bit of data is better than no data

Even the bad data is better than no data

And any kind of data is better than no data at all

Inspired by BJ Thomas, “No Love At All”


data is…

DATA


KING… of OUR world !

DATA


data “analysis”

What does a BYTE of data cost in your organization (and does anyone know or care)?

WHAT is the ROI of your data usage? (Alton Brown)

HOW can you leverage your data if you don’t really know what it is?

WHO cares most about any given piece of data?


data management

Data Governance

Metadata Quality

Analysis Design Load

Data Acquisition


key components

• MDM

– Master Data Management (-OR- Master Data Mess)

• Metadata management

– robust metadata with acceptable content & presentation

• Data quality and governance

– EVERY data element needs a steward


so… what is?

DATA about DATA


co-existence of data and metadata

– –



Business Card Request

Joe Camel

4321 Puff St.

El Nicotine, CA

415–555–1212

Mascot

[email protected]



Business Card Request

Name Joe Camel

Location 4321 Puff St.

El Nicotine, CA

Phone 415–555–1212

Title Mascot

Email [email protected]


validating business requirements


ObjectivesGeneral statements of what we

intend to accomplish

ProblemsIssues that may

arise that are under our

control

OpportunitiesFactors that may enhance our

chances of success

GoalsTimed and

measurable milestones

ActivitiesManual and

automated processes and

procedures

MetricsThe performance measures we

use to assess our progressOperational DataData used and produced

by manual and automated activities

Business RequirementsInformation to be included in data

models

Constraints

Issues that may arise that are beyond our influence

Purpose and ScopeWhat the project is intended to

accomplish; what’s “in” and “out”

Defining and Planning a Project

Process

Data

Business

Systems

Technology

Organization

Physical

RisksUncertainties

that might unfavorably impact our

effort

INTRODUCTION

As CIA is responsible for maintaining a Data Warehouse, we do not create new sources

of data, per se. We extract data from other source systems and load it into the warehouse.

We may possibly manipulate some of the source data which could result in “new”

warehouse data.

To that end, designing a database in the warehouse requires a somewhat different

approach than designing a database from scratch that would be used to support an

application.

To aid in our design, we should understand things from these perspectives:

What the data functionality and make-up of the source system is

o What is the functional data flow, content and meaning

o What is the layout/design

Why we want to have this data included in the warehouse

o What business need will this support (high-level)

o What business question(s) will this answer (detail)

How the data will be added to our warehouse or how it may replace or change

existing data in our warehouse

o Will this be a new source

o Can it be integrated into an existing database

o How does it differ from other similar data sources

o Are there any downstream affects

How we can best make this data available to the widest possible audience

o How does it relate to the overall architecture

o What standards will be applied

o What is the level of granularity - are composites or aggregates needed

How the data might be integrated with our datamart layer

o Is it “Partyable”

o Can it incorporate, or be incorporated into, any existing objects (reference

tables, etc.)

JOINT REQUIREMENTS DISCOVERY (JRD)

I. DEFINING THE PROJECT (and Scope)

A. Project Purpose

B. Project Scope

D. Risk Assessment

E. Roles and Responsibilities

II. DESCRIBING THE CURRENT SITUATION - Source System

A. Systems and Procedural Analysis

B. Data Analysis

C. Technical and Environmental Analysis

III. GAP and/or IMPACT ANALYSIS - Data Warehouse

A. Current Systems and Procedures

B. Data Impacts

C. Technical and Environmental Impacts

IV. SPECIFYING BUSINESS REQUIREMENTS

A. Business Process Analysis

1. Process Analysis

a) How will we leverage existing documentation or

diagrams

2. Process Analysis Deliverables

B. Business Data Analysis 1. Data Analysis

a) What user output is needed (Views, Reports, Forms)

2. Data Analysis Deliverables

V. PLANNING the PROJECT

A. Resource allocation

joint requirements discovery


metadata management (80/20 principle)

Business Policies Information Policies Business Rules

Information Usage Information Quality Access Controls

Entity Models Relational Tables Master Objects

Critical Elements Data Formats Alias & Synonyms

Conceptual & Value

DomainsReference Tables Mapping & Lineage

Business Terms &

ConceptsDefinitions Semantics

Reference Metadata

Business Definitions

Business Metadata

Data Governance

Information Architecture

Data Elements


metadata types

• Bretheron & Singley

– Structural metadata is used to describe the structure of computer systems such as tables, columns and indexes

– Guide metadata is used to help humans find specific items and is usually expressed as a set of keywords in a natural language

• Kimball

– Business metadata describes data content (data dictionary, maps, rules)

– Technical metadata describes objects and processes (sources, systems)

– Process metadata describes results of operations (audits, outputs)

• NISO

– Descriptive metadata is the information used to search and locate an object such as title, author, subjects, keywords, publisher

– Structural metadata gives a description of how the components of the object are organised

– Administrative metadata refers to the technical information including file type. Two sub-types of Administrative metadata are rights management metadata and preservation metadata


Kimball - DW architecture metadata worksheet

TABLE metadata

Metadata Item Short Description Possible Values

Description A short description of the table Free text

Is Audit Subsystem Is this table part of the audit subsystem? Y or N

Display Name The name displayed on the screen Free text

Table Type The type of table Free text

Used In Schemas The schemas that the table is used in Free text

View Name The name of the single table view Free text

COLUMN metadata

Metadata Item Short Description Possible Values

Description Short description of the column Free text

Display Name The name of the column as displayed on the screen Free Text

Example Values Example values stored in the column Free Text

Is Audit Subsystem Is this column part of the audit subsystem? Y or N

Is Key What type (if any) of key this column is

PK (Primary Key), SK

(Surrogate Key), or FK

(Foreign Key)


metadata benefits

– Information can be viewed in an understandable context

– Better usage of the organization’s data assets

– Preservation of corporate knowledge and business rules

– Improved impact and gap analysis (new or existing systems)

– Facilitation of standards and reusability

– Decrease in data redundancy and storage

– Easier transitions or interactions with new platforms

– Assists in answering security and regulatory questions

– Employees/contractors can get up to speed more quickly

– Potential reduction in costs for equipment and manpower

– Data becomes USEFUL INFORMATION

– Data becomes a TRUE ASSET


metadata matrix

PLANVIEW NUMBER: ___________________________________ 0

DELIVERABLE 1. C

olle

ct/u

pdat

e

busi

ness

defin

ition

s2.

Cre

ate

and

publ

ish

sour

ce-to-

targ

et m

app

ing

3. R

evie

w /

verif

y

busi

ness

defin

ition

s4.

Upd

ate

ER

Win

Mod

el5.

Upd

ate

and

publ

ish

data

bas

e m

ap6.

Upd

ate

MD

S u

sing

ER

Win

==>

MIM

B =

=>

MD

S p

roce

ss7.

Con

firm

Succ

essf

ul

Deliv

ery

of M

etad

ata

8. S

end

inte

rnal

notif

icat

ion

to d

evelo

pers

9. P

ublis

h da

ta m

odel

on

CIA

web

site

10. P

ubl

ish

data

bas

e

map

on

CIA

web

site

11. S

end

Use

r

Com

muni

catio

n re

gard

ing

Pro

duc

tion

Sta

tus

Entity names

Entity definitions

Attribute names

Attribute definitions

Relationships

Table names

Source-to-target file/table mapping

Column names

Column Transformations

Source-to-target element/column

mapping

Column data types

Column null options

Column valid values, decoded

(validation rules)

Column derivation rules (embedded

in definitions)

Primary Key and Primary Indexes

Relationships

Groupings of tables to define a

subject area

Cardinality between tables

Join strategies

Signature of person responsible

for project: Date:

Title:

Data Modeler: Date:

Physic

al

Model

Logic

al

Model

DB

Map


ERwin DM – data warehouse source


ERwin DM – UDP’s


source to target using MS Excel


ERwin DM – validation rules


domain of values


ERwin DM – data model


ERwin DM – import/export bridge


ERwin DM – column definitions via RTB

Table Name Column Name Column Datatype Attribute Definition

T_XYZ_ACCT XYZ_ACCT_ID DECIMAL(18) The system generated identifier for an existing user's Account

T_XYZ_ACCT XYZ_ACCT_NUM VARCHAR(17) The unique identifier used for an account within a given account type.

T_XYZ_ACCT XYZ_CO_ID VARCHAR(5) This identifier denotes the original (legacy) bank that booked or owned the associated account. type.

T_XYZ_ACCT XYZ_CRTE_DT_TM TIMESTAMP(0) The date and time a particular Account was added to the system.

T_XYZ_ACCT XYZ_STAT_CD INTEGER Status of the account

T_XYZ_ACCT XYZ_STAT_DT_TM TIMESTAMP(0) The STATUS is as-of this time, per the Server's local time.

T_XYZ_ACCT XYZ_IS_BUS_IND CHAR(1) This is a Business Account.

T_XYZ_ACCT XYZ_LST_UPDT_DT_TM TIMESTAMP(0) Server local time.

T_XYZ_ACCT XYZ_MASK_ACCT_NUM VARCHAR(64) This is a masked verison of or surrogate key for a real Account number,

T_XYZ_ACCT XYZ_PROD_CD VARCHAR(6) The BOS product code.

T_XYZ_ACCT XYZ_PROD_SUBCODE VARCHAR(6) a.k.a. Sub-Product Code.

T_XYZ_ACCT XYZ_SOR_ACCT_NUM VARCHAR(17) The Account Number, per the SOR. Generally a padded version of ACCOUNT_NO.

T_XYZ_ACCT XYZ_SOR_PROD_CD CHAR(3) The SOR doesn't use the BOS code as its Product Code, so this what it wants.

T_XYZ_ACCT XYZ_SOR_TYP INTEGER System of Record Type

T_XYZ_EMAIL_EV XYZ_EMAIL_ID DECIMAL(18) This identifier is the key randomly assigned to an email item when one is created for a given user.

T_XYZ_EMAIL_EV XYZ_CNTNT_TYP_CD VARCHAR(40) This is the MIME type of the email message (this could be "text/plain" or "text/html" ).

T_XYZ_EMAIL_EV XYZ_CRTD_DT_TM TIMESTAMP(0) Date and time this email item was created.

T_XYZ_EMAIL_EV XYZ_EDS_PROF_ID VARCHAR(30) The EDS profile used, in conjunction with a template Identifier, for sending the message.

T_XYZ_EMAIL_EV XYZ_EMAIL_TYP DECIMAL(18) Used for sub-type identification by Hibernate.

T_XYZ_EMAIL_EV XYZ_EVNT_ID DECIMAL(18) This identifier is the key randomly assigned to an inbox item when one is created for a given user.

T_XYZ_EMAIL_EV XYZ_INBOX_ID DECIMAL(18)

T_XYZ_EMAIL_EV XYZ_LST_UPDT_DT_TM TIMESTAMP(0) Date and time any attributes of this email item were last updated.

T_XYZ_EMAIL_EV XYZ_NNM VARCHAR(120) The nickname from the ContactPoint (at the time the Email was created), used for the "friendly name" of the recipient.

T_XYZ_EMAIL_EV XYZ_PRI_CD INTEGER Identifies the priority queue used to sequence handling. (loosely related to Urgent Indicator)

T_XYZ_EMAIL_EV XYZ_PROC_STAT_CD INTEGER Status of the email processing.

T_XYZ_EMAIL_EV XYZ_SNT_DT_TM TIMESTAMP(0) The date and time the email message was sent.

T_XYZ_EMAIL_EV XYZ_SMTP_FAILURE_CD VARCHAR(60) This is the failure code returned with failed email messages, per RFC821.

T_XYZ_EMAIL_EV XYZ_TMPLT_ID VARCHAR(100) The identifier of the template used to create this email item, if one was used.

T_XYZ_EMAIL_EV XYZ_TO_ADDR_LN VARCHAR(200) The destination email address.

T_XYZ_EMAIL_EV XYZ_URGENT_IND INTEGER This is the "importance" flag on a typical email item. It is carried along with the email message to indicate that the sender considers it important.

T_XYZ_EMAIL_EV XYZ_USR_ID DECIMAL(18) This identifier is the key randomly assigned to a user when he/she completes registration within the system.


SYS_MDS database using MS Access

Database Column Attribute Attribute Def

DB_DAD ABC_CN AbC_CN

AbC cleansed version of the CN: Customer Number assigned to the

customer.

DB_HIGH ABC_CN AbC customer number AbC customer number found in T_ABC_MBR_CN table.

DB_EVX ABC_CN ABC customer number A unique number assigned by Hogan for identifying a bank customer. This attribute stores the cleansed version of the raw input attribute "customer number".

DB_CGG ABC_CN ABC customer number AbC customer number found in T_ABC_MBR_CN table.

DB_RS ABC_CN ABC_CN See corresponding column in DB_AbC.

DB_OPS ABC_CN ABC_CN The Customer Number of the person associated with this account in AbC format.

DB_ABC ABC_CN AbC_CN @@

DB_BMT ABC_CN AbC customer number AbC datamart

DB_EDS ABC_CN AbC CN see AbC definition

DB_VSS ABC_CN ABC customer number Refer AbC

DB_RS ABC_CN AbC_CN See corresponding column in DB_AbC.

DB_OPS ABC_CN AbC_CN The Customer Number of the person associated with this account in AbC format.

DB_APK ABC_CN AbC customer number AbC customer number found in T_ABC_MBR_CN table.

DB_XYZ ABC_CN customer number AbC

DB_EDS ABC_CN AbC CN see AbC definition

DB_CM ABC_CN customer number undefined OR unable to load definition

DB_PE ABC_CN CN wells fargo CN of the visitor

DB_BMT ABC_CN ABC_CN AbC datamart

DB_EDS ABC_CN ABC_CN see AbC definition

DB_AA ABC_CN AbC customer number AbC customer number located in T_ABC_MBR_CN table.

DB_STS ABC_CN customer number See AbC Business Definitions

DB_APK ABC_CN AbC customer number AbC customer number found in T_ABC_MBR_CN table.

DB_OPS ABC_CN AbC_CN The Customer Number of the person associated with this account in AbC format.


SYS_MDS database continued

Database Column Attribute Attribute Def

DB_DAD ABC_CN AbC_CN

A unique number assigned for identifying a bank customer associated

with this account.

DB_HIGH ABC_CN AbC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_EVX ABC_CN ABC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_CGG ABC_CN ABC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_RS ABC_CN ABC_CN A unique number assigned for identifying a bank customer associated with this account.

DB_OPS ABC_CN ABC_CN A unique number assigned for identifying a bank customer associated with this account.

DB_ABC ABC_CN AbC_CN A unique number assigned for identifying a bank customer associated with this account.

DB_BMT ABC_CN AbC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_EDS ABC_CN AbC CN A unique number assigned for identifying a bank customer associated with this account.

DB_VSS ABC_CN ABC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_RS ABC_CN AbC_CN A unique number assigned for identifying a bank customer associated with this account.

DB_OPS ABC_CN AbC_CN A unique number assigned for identifying a bank customer associated with this account.

DB_APK ABC_CN AbC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_XYZ ABC_CN customer number A unique number assigned for identifying a bank customer associated with this account.

DB_EDS ABC_CN AbC CN A unique number assigned for identifying a bank customer associated with this account.

DB_CM ABC_CN customer number A unique number assigned for identifying a bank customer associated with this account.

DB_PE ABC_CN CN A unique number assigned for identifying a bank customer associated with this account.

DB_BMT ABC_CN ABC_CN A unique number assigned for identifying a bank customer associated with this account.

DB_EDS ABC_CN ABC_CN A unique number assigned for identifying a bank customer associated with this account.

DB_AA ABC_CN AbC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_STS ABC_CN customer number A unique number assigned for identifying a bank customer associated with this account.

DB_APK ABC_CN AbC customer number A unique number assigned for identifying a bank customer associated with this account.

DB_OPS ABC_CN AbC_CN A unique number assigned for identifying a bank customer associated with this account.


MetaSurf – Meta Data Services


MetaSurf search


Metadata Lite

T_XYZ_ACCT This represents an account.

T_XYZ_CNTCT A Contact is a channel for notifying a user. An Email address is the primary type

T_XYZ_EMAIL_MSG_TYP table level has NO definition

T_XYZ_IN_EV Table of In Items

T_XYZ_SUBSCR

555 AccountBasedSubscript 666 PasswordLockoutSubscript 777 CheckClearSubscript 888

CheckReorderReminderSubscript 999 DepositAccountUpdateSubscript

DATABASE TABLE COLUMN DESCRIPTION

DB_XYZ T_XYZ_CNTCT_PT XYZ_CNTCT_PT_ID

The system generated identitfier for each Contact Point entered

by a user.

DB_XYZ T_XYZ_CNTCT_PT XYZ_CNTCT_PT_TYP

This is the type of method or device used as a contact point by

the user.

DB_XYZ T_XYZ_CNTCT_PT XYZ_CRTD_DT_TM

The date and time a particular Contact Point was setup (created)

by the system.

DB_XYZ T_XYZ_CNTCT_PT XYZ_EMAIL_ADDR If the contact point type is an email then this is the address.

DB_XYZ T_XYZ_CNTCT_PT XYZ_LST_UPDT_DT_TM

Date and time any attributes of this contact point item were last

updated.

DB_XYZ T_XYZ_CNTCT_PT XYZ_PURP_CD

This code is set by the user to rank the priority or order in which

a contact point is used.

DB_XYZ T_XYZ_CNTCT_PT XYZ_USR_ID

This identifier is the key randomly assigned to a user when

he/she completes registration.


enterprise metadata repository – EMR


EMR lineage diagram


source systems’ (metadata) websites


Data Modeling Set of Standards – DMSOS


MODEL Submission Checklist (v5)

This Submission Checklist is used to ensure that all models meet CIA modeling standards. Check mark each entry to indicate that it has been successfully completed and reviewed.

If an item is not checked, explain in the comments section.

Model Name (in MM):

Date:

Data Modeler: other

Data Base Administrator: other

Check each requirement below if complete X

Logical model Entity

All lower case names

Singular nouns used (no plural or past tense)

English business name

Abbreviations not used

- AND/OR -

Standard acronyms used (when necessary)

Special characters not used

Naming standards followed for References & Composites

Definition entered

Attribute

All lower case names

Singular nouns used (no plural or past tense)

English business name (consisting of descriptors,

followed by a classword)

Classwords taken from template of logical domains

Abbreviations not used in descriptors

- AND/OR -

Standard acronyms used as needed

Special characters not used

Naming standards followed for:

Keys, Indicators, Codes, Identifiers, Dates or Timestamps

Definition entered

Logical Datatype

Assigned to every attribute

Datatypes assigned properly for:

Indicators, Codes, Dates or Timestamps

Keys & Key Groups

Primary key has been assigned for every entity

All attributes which participate in a key are either part

Physical model

Table

“T” plus acronym used in name prefix

All UPPER case names

Standard abbreviations applied

Column

All UPPER case names

Acronym used in name prefix

Standard abbreviations applied

Misc.

Template logical domains set to Physical Only, as needed

Indexes have been identified

Complete Compare done with previous Production model

Both (L/P)

Model

Model considers usage of third normal form in design

If colors used, there is a legend for color scheme

Subject Areas used for functional or project categorization

Text Boxes and comments use a standard format layout

Successful “export” and “import” of model via XML

Metadata Matrix form has been reviewed

Subject Area

Template Subject Areas have NOT been modified or deleted

Subject Areas added and/or used appropriately for

functional or project categorization

Website (to be completed by modeler and website Admin)

(See the document „Data Model Implementation.doc‟ for further detailed information)

PDF file of model diagram (w/o date) moved to staging

S2T (source to target) file for model moved to staging

New or Existing Source: Existing

Supplemental Documents: No

Source Description (required):

Source Summary:

PDF file of model diagram uploaded to website

S2T (source to target) file uploaded to website

model submission checklist


ERwin DM – database maps


corporate GLOSSARY of acronyms


intranet web site using Sharepoint


‘Data Sources’ page for a given Source System


integrating technical documents

• Business Requirements Document

• Data Modeling Checklist

• Database Design Spec

• ETL Design Spec

• Service Delivery Runbook


global objects

• Model Mart Reports

• Data Dictionary Search

• Data mining tools

Looking for common element names and/or datatypes and/or data values

Create reference or cross-reference tables


enticement !


user support and support of users

• Brown Bag presentations – business unit functionality

• Training Sessions - new projects & ongoing courses

– in person

– video / audio

– online

• CIA Communications

– What’s happening?

– How am I impacted?

– Questions – call…

• User meetings & group discussions


examples

• Expanding acronyms

– CIA =

– MOB =

• Abbreviations

– st_cd CHAR(2)

– sta_cd CHAR(2)

• Data element value/content

– 10 digit ID =


examples

• Expanding acronyms

– CIA = Certified Internal Auditor

– MOB = Mobile Browser

• Abbreviations

– st_cd status code CHAR(2)

– sta_cd state code CHAR(2)

• Data element value/content

– 10 digit ID = SSN + check digit (potential security violation)


caveats

• Metadata

– Data Element Definition – be detailed in your descriptors and be consistent in their use

– It’s better to be consistently wrong than inconsistently right

– A good example is worth a thousand words… of definition

• Models

– Normalize the physical data structure and de-normalize the view of that data or create separate (aggregate/composite) physical structures

– Create Reference and Cross Reference information that can be shared across models/databases


in conclusion

• Data and Metadata

• Standards and Roadmap

• Tools and Usage

• Communication


Vendor environment / products

• Computer Associates – ERwin DM

• Embarcadero Technologies – E/R Studio

• Axis Software Designs – Data Model Set of Standards

• Teradata – MetaSurf and Metadata Services

• ASG – Rochade / Enterprise Metadata Repository

• Microsoft – MS Excel, MS Word, MS Access, MOSS


thank you

Zen of metadata 09212010

Technology

Transcript of Zen of metadata 09212010