Data-Ed Webinar: Data Modeling Fundamentals

Peter Aiken, Ph.D.

Data Modeling Fundamentals10124 W. Broad Street, Suite C

Glen Allen, Virginia 23060 804.521.4056

Data Modeling Fundamentals

2Copyright 2016 by Data Blueprint Slide #

This presentation provides you with an understanding of the data modeling and data development components of data management. Participants will understand how the analysis, design, implementation, deployment, and maintenance of data solutions should be approached in order to maximize the full value of the enterprise data resources and activities. Architecting in quality is imperative at this level and complements a subset of project activities within the system development lifecycle (SDLC) focused on defining data requirements, designing data solution components, and implementing these components. Participants will understand the difficulties organizations experience when interacting with data development efforts and how to best incorporate these efforts into specific data projects.

Date: June 14, 2016

Time: 2:00 PM ET/11:00 AM PT Presenter: Peter Aiken, Ph.D. & John Sells

Executive Editor at DATAVERSITY.net


Shannon Kempe

Commonly Asked Questions


1) Will I get copies of the slides after the event?

2) Is this being recorded?

Get Social With Us!


Like Us on Facebook www.facebook.com/

datablueprint Post questions and

comments

Find industry news, insightful content

and event updates.

Join the Group Data Management &

Business Intelligence Ask questions, gain insights and collaborate with fellow

data management professionals

Live Twitter Feed Join the conversation!

Follow us: @datablueprint

@paiken Ask questions and

submit your comments: #dataed

• 30+ years in data management • Repeated international recognition • Founder, Data Blueprint (datablueprint.com) • Associate Professor of IS (vcu.edu)

• DAMA International (dama.org) • 9 books and dozens of articles • Experienced w/ 500+ data

management practices • Multi-year immersions:

– US DoD (DISA/Army/Marines/DLA) – Nokia – Deutsche Bank – Wells Fargo – Walmart – …

Peter Aiken, Ph.D.

• DAMA International President 2009-2013

• DAMA International Achievement Award 2001 (with Dr. E. F. "Ted" Codd

• DAMA International Community Award 2005

PETER AIKEN WITH JUANITA BILLINGSFOREWORD BY JOHN BOTTEGA

MONETIZINGDATA MANAGEMENT

Unlocking the Value in Your Organization’s

Most Important Asset.

The Case for theChief Data OfficerRecasting the C-Suite to LeverageYour Most Valuable Asset

Peter Aiken andMichael Gorman


John Sells• Data consultant with a background in Project

Management, Data Management, Verification and Validation, as well as Application Development

• Certified Data Management Professional • Experience working with large global clients

across many business functions • Skill-set includes in-depth analysis of clients’

business processes, analysis of data and data sources, and development and communication of data-centric tailored solutions that add business value

• Expertise focuses on eliciting business and technical requirements and facilitating communication between the business users and technical experts, including all levels of management

• Helped clients improve data flow logistics, develop data quality programs, implement data governance programs, and design and implement data warehouses and BI platforms for organizational divisions.



Data Modeling Fundamentals1. Data Management Overview 2. Why data modeling & what is it? 3. The power of the purpose statement 4. Understanding how to contribute to

organizational challenges beyond traditional data modeling

5. Guiding problem analyses using data analysis

6. Using data modeling in conjunction with architecture/engineering techniques

7. How to utilize data modeling in support of business strategy

8. Take Aways, References & Q&A

Tweeting now: #dataed

UsesUsesReuses

What is data management?


Sources

Data Engineering

Data Delivery

DataStorage

Specialized Team Skills

Data Governance

Understanding the current and future data needs of an enterprise and making that data effective and efficient in supporting business activitiesAiken, P, Allen, M. D., Parker, B., Mattia, A., "Measuring Data Management's Maturity: A Community's Self-Assessment" IEEE Computer (research feature April 2007)

Data management practices connect data sources and uses in an organized and efficient manner • Engineering • Storage • Delivery • Governance

When executed, engineering, storage, and delivery implement governance

Note: does not well-depict data reuse

What is data management?


Sources

Data Engineering

Data Delivery

DataStorage


Resources

(optimized for reuse)

Data Governance

Ana

lytic

Insi

ght


Data$Management$Strategy

Data Management GoalsCorporate CultureData Management FundingData Requirements Lifecycle

DataGovernance

Governance ManagementBusiness GlossaryMetadata Management

DataQuality

Data Quality FrameworkData Quality Assurance

DataOperations

Standards and ProceduresData Sourcing

Platform$&$Architecture

Architectural FrameworkPlatforms & Integration

Supporting$Processes

Measurement & AnalysisProcess ManagementProcess Quality AssuranceRisk ManagementConfiguration Management

Component Process$Areas

DMM℠ Structure of 5 Integrated DM Practice Areas

Data architecture implementation

Data Governance

Data Management

Strategy

Data Operations

PlatformArchitecture

SupportingProcesses

Maintain fit-for-purpose data, efficiently and effectively


Manage data coherently

Manage data assets professionally

Data life cycle management

Organizational support

Data Quality

You can accomplish Advanced Data Practices without becoming proficient in the Foundational Data Practices however this will: • Take longer • Cost more • Deliver less • Present

greaterrisk(with thanks to Tom DeMarco)

Data Management Practices Hierarchy

Advanced Data

Practices • MDM • Mining • Big Data • Analytics • Warehousing • SOA

Foundational Data Practices

Data Platform/Architecture

Data Governance Data Quality

Data Operations

Data Management Strategy

Technologies

Capabilities

Copyright 2016 by Data Blueprint Slide # 12

Data Management Body of Knowledge


Data Management

Functions

DAMA DM BoK: Data Development


from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Why Modeling


• Would you build a house without an architecture sketch?

• Model is the sketch of the system to be built in a project.

• Would you like to have an estimate how much your new house is going to cost?

• Your model gives you a very good idea of how demanding the implementation work is going to be!

• If you hired a set of constructors from all over the world to build your house, would you like them to have a common language?

• Model is the common language for the project team.

• Would you like to verify the proposals of the construction team before the work gets started?

• Models can be reviewed before thousands of hours of implementation work will be done.

• If it was a great house, would you like to build something rather similar again, in another place?

• It is possible to implement the system to various platforms using the same model.

• Would you drill into a wall of your house without a map of the plumbing and electric lines?

• Models document the system built in a project. This makes life easier for the support and maintenance!

Use Models to


• Store and formalize information

• Filter out extraneous detail

• Define an essential set of information

• Help understand complex system behavior

• Gain information from the process of developing and interacting with the model

• Evaluate various scenarios or other outcomes indicated by the model

• Monitor and predict system responses to changing environmental conditions

• Goal must be shared IT/business understanding – No disagreements = insufficient communication

• Data sharing/exchange is largely and highly automated and thus dependent on successful engineering – It is critical to engineer a sound foundation of data modeling basics

(the essence) on which to build advantageous data technologies

• Modeling characteristics change over the course of analysis – Different model instances may be useful to different analytical problems

• Incorporate motivation (purpose statements) in all modeling – Modeling is a problem defining as well as a problem solving activity - both are inherent to

architecture

• Use of modeling is much more important than selection of a specific modeling method • Models are often living documents

– It easily adapts to change

• Models must have modern access/interface/search technologies – Models need to be available in an easily searchable manner

• Utility is paramount – Adding color and diagramming objects customizes models and allows for a more engaging and

enjoyable user review process

Data Modeling for Business Value


Inspired by: Karen Lopez http://www.information-management.com/newsletters/enterprise_architecture_data_model_ERP_BI-10020246-1.html?pg=2

Data Modeling Ensures Interoperability• Who makes decisions about the range and scope of

common data usage?


Program F

Program E

Program DProgram G

Program H

Program I

Applicationdomain 2Application

domain 3

databasearchitectureengineering

effort

Data

DataData

Data

Data Data

Data

Focus of asoftware

architectureengineering

effort Program A

Program B

Program C

Program F

Program E

Program DProgram G

Program H

Program I

Applicationdomain 1

Applicationdomain 2Application

domain 3

Data

Focus of a

Data

Data

Data Architecture Focus has Greater Potential Business Value• Broader focus

than either software architecture or database architecture

• Analysis scope is on the system wide use of data

• Problems caused by data exchange or interface problems

• Architectural goals more strategic than operational


Primary Deliverables become Reference Material



Data Modeling Definition• Modeling = Analysis and design

method used to – Define and analyze data requirements – Design data structures that support these

requirements • Model = set of data specifications

and related diagrams that reflect requirements and designs

– Representation of something in our environment

– Employs standardized text/symbols to represent data attributes (grouped into data elements) and the relationships among them

– Integrated collection of specifications and related diagrams that represent data requirements and design



Data Modeling and Data Architecture• Data modeling is used to articulate data architecture

components • Data architectures are comprised of components – usually

expressed as models • Styles of data modeling exist – this is a challenge

– IE or information engineering

– IDEF1X used by DoD

– ORM or object role modeling

– UML or unified modeling language

• Data models are useful – In stand-alone mode

– As components of a larger information architecture


Standard definition reporting does not provide conceptual context


Bed

Something you sleep in

Entity: BED

Data Asset Type: Principal Data Entity

Purpose: This is a substructure within the room substructure of the facility location. It contains information about beds within rooms.

Source: Maintenance Manual for File and Table Data (Software Version 3.0, Release 3.1)

Attributes: Bed.Description Bed.Status Bed.Sex.To.Be.Assigned Bed.Reserve.Reason

Associations: >0-+ Room

Status: Validated

The Power of the Purpose Statement


• A purpose statement describing why the organization is maintaining information about this business concept

• Sources of information about it • A partial list of the attributes or

characteristics of the entity • Associations with other data

items; this one is read as "One room contains zero or many beds"

11

DISPOSITION Data Map


Data map of DISPOSITION• At least one but possibly more system USERS enter the DISPOSITION facts into the system.

• An ADMISSION is associated with one and only one DISCHARGE.

• An ADMISSION is associated with zero or more FACILITIES.

• An ADMISSION is associated with zero or more PROVIDERS.

• An ADMISSION is associated with one or more ENCOUNTERS.

• An ENCOUNTER may be recorded by a system USER.

• An ENCOUNTER may be associated with a PROVIDER.

• An ENCOUNTER may be associated with one or more DIAGNOSES.


ADMISSION Contains information about patient admission history related to one or more inpatient episodes

DIAGNOSIS Contains the International Disease Classification (IDC) of code representation and/or description of a patient's health related to an inpatient code

DISCHARGEA table of codes describing disposition types available for an inpatient at a FACILITY

ENCOUNTER Tracking information related to inpatient episodes

FACILITY File containing a list of all facilities in regional health care system

PROVIDER Full name of a member of the FACILITY team providing services to the patient

USER Any user with access to create, read, update, and delete DISPOSITION data

• Models

• Are usually for the purpose of understanding

• Can be

– Equations

– Simulations including video games

– Physical models

– Mental models

Models as an Aid to Understanding


What is a model?


drawcritique

testdialog

select decidefilter

summarizedesign

rankreview cluster

generate evaluate

list

visible to participants

Structure for organizing things

Framework for decision making

Requires tools for problem solving and decision making

Easy to review and validate

graphic

text

Prototype and mockupFramework for understanding and design

Source: Ellen Gottesdiener www.ebgconsulting.com

Don’t Tell Them You Are Modeling!


• Just write some stuff down

• Then arrange it • Then make

some appropriate connections between your objects

Keep them focused on the purpose


• The reason we are locked in this room is to: – Mission: Review proposal from

voice over IP providers • Outcome: Walk out the door with the

top two proposals selected and scheduled personal presentations from each.

– Mission: Discuss logo ideas for the Bore No More movement • Outcome: We will walk out the door

when we identify the top three traits that represent the Bore No More brand.

– Mission: Update all employees on the retirement plan options • Outcomes: Confirm that all team

members took part in the meeting and have access to review their plans privately with a financial consultant.

Entity Relationship View


C U S T O M E R

coins

soda

machine

(adapted from [Davis 1990])

Entity Relationship View


(adapted from [Davis 1990])

entity thing about which we maintain information

object entity encapsulated with attributes and functions

C U S T O M E R soda

machine

coinreturn

deposits

selects

given to

dispenses

coins

Modeling In Support of Requirements

Person Job Class

Employee Position

BR1) Zero, one, or more EMPLOYEES can be associated

with one PERSON

BR2) Zero, one, or more EMPLOYEES can be associated with one POSITION


Job Sharing

Moon Lighting

Data Modeling• Modeling = complex process involving interaction

between people and with technology that don’t compromise the integrity or security of the data

– Good data models accurately express and effectively communicate data requirements and quality solution design

• Modeling approach (guided by 2 formulas):

– Purpose + audience = deliverables

– Deliverables + resources + time = approach




Data Models Facilitate• Formalization

– Data model documents a single, precise definition of data requirements and data-related business rules

• Communication – Data model is a bridge to understanding data

between people with different levels and types of experience.

– Helps understand business area, existing application, or impact of modifying an existing structure

– May also facilitate training new business and/or technical staff

• Scope – Data model can help explain the data concept and scope of

purchased application packages


ANSI-SPARK 3-Layer Schema


For example, a changeover to a new DBMS technology. The database administrator should be able to change the conceptual or global structure of the database without affecting the users.

1. Conceptual - Allows independent customized user views:

– Each should be able to access the same data, but have a different customized view of the data.

2. Logical - This hides the physical storage details from users:

– Users should not have to deal with physical database storage details. They should be allowed to work with the data itself, without concern for how it is physically stored.

3. Physical - The database administrator should be able to change the database storage structures without affecting the users’ views:

– Changes to the structure of an organization's data will be required. The internal structure of the database should be unaffected by changes to the physical aspects of the storage.

Conceptual Models• Business

focused • Entity level • Provides focus,

scope, and guidance to modeling effort

• Sometimes thrown away - rarely maintained


Logical Models• Required to achieve the transition

from conceptual to physical • Developed to the attribute level via

3rd normal form - to a define level of understandability

• Logical models are developed to be refined to until it becomes a solution - sometimes purchased (as in EDW) always requires tailoring

• Used to guarantee the rigor of the data structures by formally describing the relationship between data items in a strong fashion - more often maintained


Physical Models• Becomes the blueprints for

physical construction of the solution

• Blueprints are used for future maintenance of the solution


Model Evolution (better explanation)


As-is To-be

TechnologyIndependent/Logical

TechnologyDependent/Physical

abstraction

Other logical as-is data architecture components

As Is InformationRequirements Assets

As Is Data Design Assets As Is Data Implementation Assets

Exi

stin

gN

ew

Modeling in Various Contexts

O2 Recreate Data Design

Reverse Engineering

Forward engineering

O5 Reconstitute Requirements

O9 Reimplement

Data

To Be Data Implementation Assets

O8 RedesignData

O4 Recon-stitute Data

Design

O3 Recreate Requirements

O6 Redesign Data

To Be Design Assets

O7 Re- developRequire-ments

To Be Requirements Assets

O1 Recreate Data Implementation

Metadata


Model Evolution Framework


Conceptual Logical Physical

Goal

Validated

Not Validated

Every change can be mapped to a transformation in this framework!

Preliminaryactivities Modeling

cyclesWrapupactivities

Evidencecollection &

analysis

Projectcoordinationrequirements

Targetsystemanalysis

Modelingcyclefocus

Activity

Refinement

Collection

Analysis

Validation

Declining coordination requirements

Increasing amounts of target system analysis




analysis



Modelingcyclefocus

Activity

Refinement

Collection

Analysis

Validation






analysis



Modelingcyclefocus

Activity

Refinement

Collection

Analysis

Validation






analysis



Modelingcyclefocus

Activity

Refinement

Collection

Analysis

Validation



Relative use of time allocated to tasks during ModelingPreliminary

activities Modelingcycles

Wrapupactivities


analysis



Modelingcyclefocus

Activity

Refinement

Collection

Analysis

Validation




How do Data Models Support Organizational Strategy?• Consider the opposite question:

– Were your systems explicitly designed to be integrated or otherwise work together?

– If not then what is the likelihood that they will work well together?

– In all likelihood your organization is spending between 20-40% of its IT budget compensating for poor data structure integration

– They cannot be helpful as long as their structure is unknown

• Two answers

– Achieving efficiency and effectiveness goals

– Providing organizational dexterity for rapid implementation


Design Styles – 3NF• A mathematical data design technique founded in the early 70s by E.F.

Codd. • Organizes data in simple

rows and columns - Entities • Creates connections

between the entities called relationships to show how the data is inter-related

• 3NF removes data redundancies – a piece of data is stored only once

• 3NF is based on mathematics, give the same facts to different modelers; the models they produce should be very similar.

• Creates a visual (Entity Relation Diagram - ERD) which may be understood by less technical personnel

• 3NF is the modeling style most popularly used for operationally focused data stores.


Design Styles – Dimensional• Created and refined by Ralph

Kimball in the 80s. • Organizes data in Facts

and Dimensions. Fact tables record the events (what) within the business domain and the Dimension tables describe who, when, how and where.

• The data design style was created to exploit the capabilities of the relational database to retrieve and report against large volumes of data.

• Dimensional modeling sacrifices storage efficiency for analytical processing speed

• There are 2 variations to Dimensional Modeling: Star Schema and Snowflake


Design Styles – Data Vault• One of the newer relational database modeling techniques • Data Vault modeling was conceived in the 1990s by Dan

Linstedt • Data Vault models are designed for central data

warehouses that store non-volatile, time-variant, atomic data

• Relationships are defined through Link structures which promote flexibility and extensibility


Data Models Used to Support Strategy• Flexible, adaptable data structures • Cleaner, less complex code • Ensure strategy effectiveness measurement • Build in future capabilities • Form/assess merger and acquisitions strategies


EmployeeType Employee

SalesPerson Manager Manager

Type

StaffManager

Line Manager

Adapted from Clive Finkelstein Information Engineering Strategic Systems Development 1992

Mission and Purpose• Develop, deliver and support products and services which

satisfy the needs of customers in markets where we can achieve a return on investment at least 20% annually within two years of market entry


Mission Model Analysis


Identify Potential Goals

G1. Market Analysis G2. Market Share G3. Innovation G4. Customer Satisfaction G5. Product Quality G6. Product Development G7. Staff Productivity G8. Asset Growth G9. Profitability


Mission Model Analysis


Next Step


Market

MarketCustomer

ProductNeed

Need

CustomerProduct

MarketNeed

ProductCustomer

CustomerNeed

MarketProduct

Subsequent Step for Business Value


Market

MarketPerformance

ProductPerformance

Need

CustomerPerformance

Need Performance

ProductCustomer

Performance

Questions?

It’s your turn! Use the chat feature or Twitter (#dataed) to submit

your questions to Peter & John now!

+ =


Upcoming EventsGoverning the Business Vocabulary – aligning the requirements of the business and IT to achieve a shared understanding of data across an organization June 27, 2016 @ 8:30 AM ETSan Diego, CAhttp://www.debtechint.com

Data Quality Success Stories July 12, 2016 @ 2:00 PM ET/11:00 AM PT

Sign up here: www.datablueprint.com/webinar-schedule or www.dataversity.net


Data-Ed Webinar: Data Modeling Fundamentals

Technology

Transcript of Data-Ed Webinar: Data Modeling Fundamentals