October 23, 2017 Sam Siewert

CS317File and Database Systems

Lecture 10 – Physical DBMS Designhttp://commons.wikimedia.org/wiki/Category:R-tree#mediaviewer/File:R-tree_with_Guttman%27s_quadratic_split.png

RemindersAssignment #4 – Grading in Progress

Assignment #5 (Available on Canvas)– Physical DB Design Practice– Propose your Final DBMS Project

Assignment #6, DBMS Project of Your Interest – FINAL ORAL PRESENTATION– Design Schema for DBMS project in a small team

Logical design focusNormalizationPhysical is MySQL on PRClab

– Combine Network Applications with DBMS in C/C++, JDBC, or Python - http://www.mysql.com/products/connector/

– Add Stored Programs and Triggers– Add Views– Create Transactions where needed

Sam Siewert 2

Goals for This WeekWrap-Up DBMS Conceptual Design

Wrap-Up DBMS Logical Design

Introduce DBMS Physical Design

DBMS Indexing [B-Tree & B+-Tree, Hash, R-Tree]– Indexing Fundamentals– ISAM – Indexed Sequential Access Method [Records in Files]

MyISAM – Index, Format, Data Files for Each Table– InnoDB – Cached Indexes and Data, Several Files or Raw Disk Partitions– Memory (Heap) – Unordered Set of Records, Hash Index, B-Tree Options– Introduction to B and B+ Trees (Bayer or perhaps Boeing or Balance Tree)– Hash - Fast Table Lookup, Collisions handled with List or Tree (Similar, but

different than data digest or cryptographic - e.g. MD5 Hash)– Exploration beyond B-Trees - R-Tree, X-Tree (Not covered)

Mapping to DBMS to Storage– Memory– Nand Flash and SSD– HDD– File Systems and Block Storage

Sam Siewert 3

DBMS Design in 10 StepsDBMS Design - Decomposition1. Conceptual Design – Analysis, Centralized or View Integration2. Build and Validate Logical Data Model – ER/EER,

Normalization, Schema Prototype (Workbench)3. Translate Logical Data Model for Target DBMS (MySQL)4. Design File Organization (or Block) and Indexing5. Triggers and SQL/PSM for Referential Integrity6. User View Design (Lossless Join Verification)7. Security8. Transactions for Complex Updates9. Redundancy (RTO/RPO and RAID)10. Connectors for App Interfaces

Covered, To be covered

Sam Siewert 4

Logical DBMS EngineeringIdeally use a CASE tool like Workbench – EER DB

Sam Siewert 5

ForwardReverse

SQL

EER

Workbench ManualModel Editor – Name your Model to match DB deployment Target name – e.g. siewertsDHv1_3, siewertsdb, etc.Test and Interaction Tab (Queries, Schemas,

Sam Siewert 6

Workbench ModelHigh Level View of Logical Model and Test SQL Scripts

Sam Siewert 7

Deployed DB from EER Logical Design

Logical to Physical MappingsBranch Table Logical Design Details– Primary Key

Designation, Not-NULL, Default, etc.

– Foreign Keys, Triggers, etc.

Branch Table Physical Design Details– Indexes – Engine

[InnoDB, MyISAM, MEMORY], Columns to Index, Storage Type [BTREE, HASH]

Sam Siewert 8

Engine = MEMORY

Index Type = BTREE

Indexing for Optimal Query

9

Physical Design - Objectives

Purpose of physical database design.

How to map the logical database design to a physical database design.

How to design base relations for target DBMS.

How to design general constraints for target DBMS.

10

Physical Design - Objectives

How to select appropriate file organizations based on analysis of transactions.

When to use secondary indexes to improve performance.

How to estimate the size of the database.

How to design user views.

How to design security mechanisms to satisfy user requirements.

11

Logical v. Physical Database Design

Sources of information for physical design process includes logical data model and documentation that describes model.

Logical database design is concerned with the what, physical database design is concerned with the how.

12

Physical Database Design

• Process of producing a description of the implementation of the database on secondary storage.

• It describes the base relations, file organizations, and indexes used to achieve efficient access to the data, and any associated integrity constraints and security measures.

13

Overview of Physical Database Design Methodology

Step 1 Conceptual Design (Chapters 10-12, Lifecycle, Analysis – Centralized, View Integration)

Step 2 Build and Validate Logical Data Model (Chapters 12 – 14 – ER/EER, Normalization)

Step 3 Translate logical data model for target DBMS– Step 3.1 Design base relations– Step 3.2 Design representation of derived data – Step 3.3 Design general constraints

14

Overview of Physical Database Design Methodology

Step 4 Design file organizations and indexes– Step 4.1 Analyze transactions– Step 4.2 Choose file organizations– Step 4.3 Choose indexes– Step 4.4 Estimate disk space requirements

15

Overview of Physical Database Design Methodology

Step 5 Design user views (Lossless Join)Step 6 Design security mechanismsStep 7 Consider the introduction of controlled redundancy Step 8 Monitor and tune operational system

Step 9 RAID and DR (Disaster Recovery), ScalingStep 10 Applications and Connectors

16

Step 3 Translate Logical Data Model for Target DBMS

To produce a relational database schema from the logical data model that can be implemented in the target DBMS.Need to know functionality of target DBMS such as how to create base relations and whether the system supports the definition of:– PKs, FKs, and AKs;– required data – i.e. whether system supports NOT

NULL;– domains;– relational integrity constraints;– general constraints.

17

Step 3.1 Design base relations

To decide how to represent base relations identified in logical model in target DBMS.

For each relation, need to define:– the name of the relation;– a list of simple attributes in brackets;– the PK and, where appropriate, AKs and FKs.– referential integrity constraints for any FKs identified.

18

Step 3.1 Design base relations

From data dictionary, we have for each attribute:– its domain, consisting of a data type, length,

and any constraints on the domain;– an optional default value for the attribute;– whether it can hold nulls;– whether it is derived, and if so, how it should

be computed.

19

DBDL for the PropertyForRent Relation

20

Step 3.2 Design representation of derived data

To decide how to represent any derived data present in logical data model in target DBMS.

Examine logical data model and data dictionary,and produce list of all derived attributes.

Derived attribute can be stored in database orcalculated every time it is needed.

21

Step 3.2 Design representation of derived data

Option selected is based on:– additional cost to store the derived data and keep it

consistent with operational data from which it isderived;

– cost to calculate it each time it is required.

Less expensive option is chosen subject to performance constraints.

22

PropertyforRent Relation and StaffRelation with Derived Attribute

noOfProperties

23

Step 3.3 Design general constraints

To design the general constraints for target DBMS.

Some DBMS provide more facilities than others for defining enterprise constraints. Example:

CONSTRAINT StaffNotHandlingTooMuchCHECK (NOT EXISTS (SELECT staffNo

FROM PropertyForRentGROUP BY staffNoHAVING COUNT(*) > 100))

24

Step 4 Design File Organizations and Indexes

To determine optimal file organizations to store the base relations and the indexes that are required to achieve acceptable performance; that is, the way in which relations and tuples will be held on secondary storage.

Must understand the typical workload that database must support.

25

Cross-referencing transactions and relations

26

Example Transaction Usage Map

27

Example Transaction Analysis Form

28

Step 4.2 Choose file organizations

To determine an efficient file organization for each base relation.

File organizations include Heap [Unordered Set of Records], Hash, Indexed Sequential Access Method (ISAM), B+-Tree, and Clusters.

Some DBMSs may not allow selection of file organizations.

29

Step 4.3 Choose indexes

To determine whether adding indexes will improve the performance of the system.

One approach is to keep tuples unordered andcreate as many secondary indexes as necessary.

30

Step 4.3 Choose indexes

Another approach is to order tuples in therelation by specifying a primary or clusteringindex.

In this case, choose the attribute for ordering orclustering the tuples as:– attribute that is used most often for join operations -

this makes join operation more efficient, or– attribute that is used most often to access the tuples in

a relation in order of that attribute.

31

Step 4.3 Choose indexes

If ordering attribute chosen is key of relation,index will be a primary index; otherwise, indexwill be a clustering index.

Each relation can only have either a primaryindex or a clustering index.

Secondary indexes provide a mechanism forspecifying an additional key for a base relationthat can be used to retrieve data more efficiently.

32

Step 4.3 Choose indexes

Have to balance overhead involved in maintenanceand use of secondary indexes against performanceimprovement gained when retrieving data.This includes:– adding an index record to every secondary index

whenever tuple is inserted;– updating secondary index when corresponding

tuple updated;– increase in disk space needed to store secondary

index;– possible performance degradation during query

optimization to consider all secondary indexes.

33

Step 4.3 Choose indexes –Guidelines for choosing ‘wish-list’

1. Do not index small relations. 2. Index PK of a relation if it is not a key of the file

organization. 3. Add secondary index to a FK if it is frequently

accessed. 4. Add secondary index to any attribute heavily used

as a secondary key.5. Add secondary index on attributes involved in:

selection or join criteria; ORDER BY; GROUP BY; and other operations involving sorting (such as UNION or DISTINCT).

34

Step 4.3 Choose indexes –Guidelines for choosing ‘wish-list’

6. Add secondary index on attributes involved in built-in functions.

7. Add secondary index on attributes that could result in an index-only plan.

8. Avoid indexing an attribute or relation that is frequently updated.

9. Avoid indexing an attribute if the query will retrieve a significant proportion of the relation.

10. Avoid indexing attributes that consist of long character strings.

Note on MySQL IndexesMySQL Create Index

MySQL 5.7 Provides Indexes to Speed Query for Query Engines as Follows

Spatial R-Tree Indexing Possible for MyISAM

Using Spatial Data Extensions

MEMORY Storage Engine –Hash Option Sam Siewert 35

Storage Engine Supported Index Types

MyISAM B-tree, R-Tree Spatial on POINT,GEOMETRY Columns

InnoDB B-tree

Memory (Heap) Hash, B-Tree

Defaults - Common

Options – Advanced Indexing

B-TreeRudolf Bayer, Edward McCreight – Boeing

Optimized for Read/Write Large Block Data – DBMS and File Systems

B+-Tree is Variant with Keys [Indexes] Only for Internal Nodes and all Keys and Data Found in Leaves– Keys [Indexes] Repeated in Tree– Keys [Indexes] + Data in Sequential Order in Leaves

Goal is to Increase the Size of Nodes to Match File System Block Size [E.g. 4K … 64K] with N Keys [Data] Stored in Each Tree Node [8x512 … 128x512 Sectors]

Sam Siewert 36

B-Tree ExamplestaffNo Key, staff tuples with multiple attributes in staffTableOrder N=4, 4 Pointers, 3 Keys per Node, 2 Pointers, 1 Key Minimum [Splits and Combines on Insert and Delete]Tuple Pointer for All Keys [Implicit]Insert Tuples for staffNo S005, S003, S021, S009, S001, S013

Sam Siewert 37

Ptr1 S003 Ptr2 S005 Ptr3 S0021 Ptr4

Ptr1 S001 Ptr2 S003 Ptr3 S005 Ptr4 Ptr1 S013 Ptr2 S021 Ptr3

Ptr1 S009 Ptr2

First 3 inserts, fill root node

S009 insert, splits root nodeS001 and S013 fill out children

B-Tree Inserts ContinuedSplits Continue on Staff Table InsertsInsert S002, …

Sam Siewert 38

Ptr1 S001 Ptr2 S003 Ptr3 S005 Ptr4 Ptr1 S013 Ptr2 S021 Ptr3

Ptr1 S009 Ptr2 S002 insert splits left child andpromotes S003 to root node

Ptr1 S001 Ptr2 S002 Ptr3 Ptr1 S005 Ptr2 Ptr1 S013 Ptr2 S021 Ptr3

Ptr1 S003 Ptr2 S009 Ptr3

B-TreeProvides Nice WHERE Clause Support

Index [Key] Appears ONCE in Each Node

Downside – SELECT * with Tuples in staffNo ASC or DSC Order is Time Consuming

Split and Combines Required on Insert/Delete

Key, Pointers, and Tuple Data in Nodes Limit “Bushiness”

B+-Tree – Stores Keys, Pointers ONLY on Internal Nodes, Increasing Bushiness [Key for File systems], Record/File Data ONLY on Leaf Nodes

Sam Siewert 39

B+-TreeReplication of Keys [Indexes]Tuple Data ONLY in LeavesUseful for Key [Index] Cache and Storage of Leaves

Sam Siewert 40

Ptr1 S001 Ptr2 S002 Ptr3

Ptr1 S003 Ptr2 S005 Ptr3

Ptr1 S009 Ptr2 S013 Ptr3 S021 Ptr4

Ptr1 S003 Ptr2 S009 Ptr3

DISK DRIVE

I/O CACHE

B-Tree DetailsCovered in Data Structures – B-tree Example in C

“The Ubiquitous B-Tree”, by Douglas Comer– Insertion Balancing– Deletion Balancing– Cost of Operations

Recall Disk Drive Performance– 100 to 200 Random I/Os per Second [Seek + Rotate is 5 to 10

msec]– Sequential Rates 100 to 200 Mbytes per Second

MySQL Memory (Heap) B-Tree for Small Tables – Fit in Memory Pages [Cache Lines] Sam Siewert 41

Hash TableIndex = Function (Candidate Key)E.g. Index = MD5(staffNo, staffName) % Table_Size

Perfect Hash Guarantees No 2 Staff will Map to Same Index

Hash Collision Requires Secondary Collision Structure –E.g. List

Great for Small Tables and Data Sets

Typically in Memory – Performs like LUT [Look Up Table or Array] Sam Siewert 42

Multi-Value QueryB-Tree Great for Selection and Theta-Join– Finding Tuples Satisfying WHERE clauses

What About Complex Query for Multiple Conditions?– E.g. WHERE clause and Distance from Current Location < 20 miles– Satisfy B-Tree WHERE clause– Check X,Y Distance Constraint as Post Filter?

Possible to Combine Multiple Dimensions for Query in the Tree Index?

R-Trees or Rectangular Trees, X-Tree, etc. [Not Covering, but Interesting]

Wrap-Up B-Trees and RAID Next Week …

Sam Siewert 43