Temple University – CIS Dept. CIS331– Principles of Database Systems V. Megalooikonomou Indexing...
-
Upload
reginald-russell -
Category
Documents
-
view
223 -
download
0
Transcript of Temple University – CIS Dept. CIS331– Principles of Database Systems V. Megalooikonomou Indexing...
Temple University – CIS Dept.CIS331– Principles of Database Systems
V. Megalooikonomou
Indexing and Hashing I
(based on notes by Silberchatz, Korth, and Sudarshan and notes by C. Faloutsos at CMU)
General Overview - rel. model Relational model - SQL
Formal & commercial query languages
Functional Dependencies Normalization Physical Design Indexing
Indexing- overview primary / secondary indices index-sequential (ISAM) B - trees, B+ - trees hashing
static hashing dynamic hashing
Basic Concepts
Indexing mechanisms speed up access to desired data
E.g., author catalog in library Search Key - attribute to set of attributes used to
look up records in a file An index file consists of records (called index
entries) of the form
Index files are typically much smaller than the original file
Two basic kinds of indices: Ordered indices: search keys are stored in sorted order Hash indices: search keys are distributed uniformly across
“buckets” using a “hash function”
search-key pointer
Indexing once the records are stored in a
file, how do you search efficiently? (e.g., ssn=123?)
STUDENTSsn Name Address
123 smith main str234 jones forbes ave125 tomson main str
Indexing once the records are stored in a
file, how do you search efficiently?
brute force: retrieve all records, report the qualifying ones
better: use indices (pointers) to locate the records directly
Indexing – main idea:
123125234
STUDENTSsn Name Address
123 smith main str234 jones forbes ave125 tomson main str
Measuring ‘goodness’ retrieval time?
insertion / deletion?
space overhead?
reorganization?
range queries?
Main concepts search keys are sorted in the index
file and point to the actual records
primary vs. secondary indices
Clustering (sparse) vs
non-clustering (dense) indices
Indexing
STUDENTSsn Name Address
123 smith main str234 jones forbes ave678 tomson main str456 stevens forbes ave345 smith forbes ave
123234345456567
Primary key index: on primary key (no duplicates)
Indexing
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
forbes avemain str
secondary key index: duplicates may exist
Address-index
Indexing
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
forbes avemain str
secondary key index: typically, with ‘postings lists’
Postings lists
Main concepts – cont’d Clustering (= sparse) index:
records are physically sorted on that key (and not all key values are needed in the index)
Non-clustering (=dense) index: the opposite
E.g.:
Indexing- Sparse index
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
123456
…
Clustering/sparse index on ssn
>=123
>=456
Sparse Index Files Sparse Index: contains index records for only some
search-key values Applicable when records are sequentially ordered on search-
key To locate a record with search-key value K we:
Find index record with largest search-key value < K Search file sequentially starting at the record to which the
index record points Less space and less maintenance overhead for
insertions and deletions Generally slower than dense index for locating records Good tradeoff: sparse index with an index entry for
every block in file, corresponding to least search-key value in the block
Indexing – Dense Index
Ssn Name Address345 tomson main str234 jones forbes ave567 smith forbes ave456 stevens forbes ave123 smith main str
123234345456567
Non-clustering / dense index
Summary
Dense Sparse
Primary usual
secondary
usual rare
• All combinations are possible…
• at most one sparse/clustering index
• as many as desired dense indices
• usually: one primary-key index (maybe clustering) and a few secondary-key indices (non-clustering)
Indexing- overview primary / secondary indices index-sequential (ISAM) B - trees, B+ - trees hashing
static hashing dynamic hashing
ISAM What if index is too large to search
sequentially?
use a multilevel index…
ISAM
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
123456
…
>=123
>=456
1233,423
…
block
ISAM - observations if index is too large, store it on disk
and keep index-on-the-index usually two levels of indices, one first-level entry per disk block
(why? )
ISAM - Multilevel Index
ISAM - observations What about insertions/deletions?
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
123456
…
>=123
>=456
1233,423
…
124; peterson; fifth ave.
ISAM - observations What about insertions/deletions?
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
123456
…
1233,423
…124; peterson; fifth ave.
overflows
Problems?
ISAM - observations What about insertions/deletions?
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
123456
…
1233,423
…124; peterson; fifth ave.
overflows
• overflow chains may become very long - what to do?
ISAM - observations What about insertions/deletions?
STUDENTSsn Name Address
123 smith main str234 jones forbes ave345 tomson main str456 stevens forbes ave567 smith forbes ave
123456
…
1233,423
…124; peterson; fifth ave.
overflows
• overflow chains may become very long - thus:
• shut-down & reorganize
• start with ~80% utilization
So far … indices (like ISAM) suffer in the
presence of frequent updates sequential scan using primary index is
efficient, but a sequential scan using a secondary index is expensive each record access may fetch a new block
from disk
alternative indexing structure: B - trees
Overview primary / secondary indices multilevel (ISAM) B - trees, B+ - trees hashing
static hashing dynamic hashing
B-trees the most successful family of
index schemes (B-trees, B+-trees, B*-trees)
can be used for primary/secondary, clustering/non-clustering index
they are balanced “n-way” search trees
B-trees Disadvantage of indexed-sequential files:
performance degrades as file grows, since many overflow blocks get created. Periodic reorganization of entire file is required
Advantage of B+-tree index files: automatic self-reorganization with small, local,
changes, in the face of insertions and deletions. Reorganization of entire file is not required
Disadvantage of B+-trees: extra insertion and deletion overhead, space overhead
Advantages of B+-trees outweigh disadvantages, and they are used extensively
B-treesE.g., B-tree of order 3 (i.e., at most 3 pointers from each
node):
1 3
6
7
9
13
<6
>6 <9 >9
B-tree properties: each node, in a B-tree of order n :
key order at most n pointers at least n/2 pointers (except root) all leaves at the same level if number of pointers is k, then node has
exactly k-1 keys
v1 v2 … vn-1
p1 pn
Properties “block aware” nodes: each node -> disk
page
O(log (N)) for everything! (ins/del/search)
typically, if N = 50 - 100, then 2 - 3 levels
utilization >= 50%, guaranteed; on average 69%
Queries Algorithm for exact match query? (e.g., ssn=8?)
1 3
6
7
9
13
<6
>6 <9 >9
Queries Algorithm for exact match query? (e.g., ssn=8?)
1 3
6
7
9
13
<6
>6 <9 >9
Queries Algorithm for exact match query? (e.g., ssn=8?)
1 3
6
7
9
13
<6
>6 <9 >9
Queries Algorithm for exact match query? (e.g., ssn=8?)
1 3
6
7
9
13
<6
>6 <9 >9
Queries Algorithm for exact match query? (e.g., ssn=8?)
1 3
6
7
9
13
<6
>6 <9 >9H steps (= disk accesses)
Queries what about range queries? (e.g.,
5<salary<8) Proximity/ nearest neighbor
searches? (e.g., salary ~ 8 )
Queries what about range queries? (e.g.,
5<salary<8) Proximity/ nearest neighbor searches?
(e.g., salary ~ 8 )
1 3
6
7
9
13
<6
>6 <9 >9
Queries what about range queries? (eg.,
5<salary<8) Proximity/ nearest neighbor searches?
(eg., salary ~ 8 )
1 3
6
7
9
13
<6
>6 <9 >9
B-trees: Insertion Insert in leaf;
on overflow, push middle up (recursively)
split: preserves B - tree properties
B-trees
Easy case: Tree T0; insert ‘8’
1 3
6
7
9
13
<6
>6 <9 >9
B-trees
Tree T0; insert ‘8’
1 3
6
7
9
13
<6
>6 <9 >9
8
B-trees
Hardest case: Tree T0; insert ‘2’
1 3
6
7
9
13
<6
>6 <9 >9
2
B-trees
Hardest case: Tree T0; insert ‘2’
1 2
6
7
9
133
push middle up
B-trees
Hardest case: Tree T0; insert ‘2’
6
7
9
131 3
22Ovf; push middle
B-trees
Hardest case: Tree T0; insert ‘2’
7
9
131 3
2
6
Final state
B-trees - insertion Q: What if there are two
middles? (e.g., order 4) A: either one is fine
B-trees: Insertion Insert in leaf; on overflow, push
middle up (recursively – ‘propagate split’)
split: preserves all B - tree properties (!!)
notice how it grows: height increases when root overflows & splits
Automatic, incremental re-organization (contrast with ISAM!)
INSERTION OF KEY ’K’
find the correct leaf node ’L’;
if ( ’L’ overflows ){
split ’L’, by pushing the middle key upstairs to parent node ’P’;
if (’P’ overflows){
repeat the split recursively;
}
else{
add the key ’K’ in node ’L’; /* maintaining the key order in ’L’ */
}
Pseudo-code
Overview primary / secondary indices multilevel (ISAM) B – trees
Dfn, Search, insertion, deletion
B+ - trees hashing
Deletion
Rough outline of algorithm: Delete key; on underflow, may need to merge
In practice, some implementors just allow underflows to happen…
B-trees – Deletion
Easiest case: Tree T0; delete ‘3’
1 3
6
7
9
13
<6
>6 <9 >9
B-trees – Deletion
Easiest case: Tree T0; delete ‘3’
1
6
7
9
13
<6
>6 <9 >9
B-trees – Deletion Case1: delete a key at a leaf – no underflow Case2: delete non-leaf key – no underflow Case3: delete leaf-key; underflow, and ‘rich
sibling’ Case4: delete leaf-key; underflow, and ‘poor
sibling’
B-trees – Deletion Case1: delete a key at a leaf – no underflow
(delete 3 from T0)
1 3
6
7
9
13
<6
>6 <9 >9
B-trees – Deletion Case2: delete a key at a non-leaf – no
underflow (e.g., delete 6 from T0)
1 3
6
7
9
13
<6
>6 <9 >9
Delete & promote, i.e:
B-trees – Deletion Case2: delete a key at a non-leaf – no
underflow (e.g., delete 6 from T0)
1 3 7
9
13
<6
>6 <9 >9
Delete & promote, i.e.:
B-trees – Deletion Case2: delete a key at a non-leaf – no
underflow (eg., delete 6 from T0)
1 7
9
13
<6
>6 <9 >9
Delete & promote, i.e.:3
B-trees – Deletion Case2: delete a key at a non-leaf – no
underflow (eg., delete 6 from T0)
1 7
9
13
<3
>3 <9 >9
3FINAL TREE
B-trees – Deletion Case2: delete a key at a non-leaf – no underflow (eg.,
delete 6 from T0) Q: How to promote? A: pick the largest key from the left sub-tree (or the
smallest from the right sub-tree)
Observation:
Every deletion eventually becomes a deletion of a leaf key
B-trees – Deletion Case1: delete a key at a leaf – no underflow Case2: delete non-leaf key – no underflow Case3: delete leaf-key; underflow, and
‘rich sibling’ Case4: delete leaf-key; underflow, and ‘poor
sibling’
B-trees – Deletion Case3: underflow & ‘rich sibling’ (eg.,
delete 7 from T0)
1 3
6
7
9
13
<6
>6 <9 >9
Delete & borrow, ie:
B-trees – Deletion Case3: underflow & ‘rich sibling’ (eg.,
delete 7 from T0)
1 3
6 9
13
<6
>6 <9 >9
Delete & borrow, ie:
Rich sibling
B-trees – Deletion Case3: underflow & ‘rich sibling’
‘rich’ = can give a key, without underflowing
‘borrowing’ a key: always THROUGH the PARENT!
B-trees – Deletion Case3: underflow & ‘rich sibling’ (eg.,
delete 7 from T0)
1 3
6 9
13
<6
>6 <9 >9
Delete & borrow, ie:
Rich sibling
NO!!
B-trees – Deletion Case3: underflow & ‘rich sibling’ (eg.,
delete 7 from T0)
1 3
6 9
13
<6
>6 <9 >9
Delete & borrow, ie:
B-trees – Deletion Case3: underflow & ‘rich sibling’ (eg.,
delete 7 from T0)
1
3 9
13
<6
>6 <9 >9
Delete & borrow, ie:
6
B-trees – Deletion Case3: underflow & ‘rich sibling’ (eg.,
delete 7 from T0)
1
3 9
13
<3
>3 <9 >9
Delete & borrow, through the parent
6
FINAL TREE
B-trees – Deletion Case1: delete a key at a leaf – no underflow Case2: delete non-leaf key – no underflow Case3: delete leaf-key; underflow, and ‘rich
sibling’ Case4: delete leaf-key; underflow, and
‘poor sibling’
B-trees – Deletion Case4: underflow & ‘poor sibling’ (eg.,
delete 13 from T0)
1 3
6
7
9
13
<6
>6 <9 >9
B-trees – Deletion Case4: underflow & ‘poor sibling’ (eg.,
delete 13 from T0)
1 3
6
7
9<6
>6 <9 >9
B-trees – Deletion Case4: underflow & ‘poor sibling’ (eg.,
delete 13 from T0)
1 3
6
7
9<6
>6 <9 >9
A: merge w/ ‘poor’ sibling
B-trees – Deletion Case4: underflow & ‘poor sibling’ (eg.,
delete 13 from T0)
Merge, by pulling a key from the parent exact reversal from insertion: ‘split and push
up’, vs. ‘merge and pull down’ Ie.:
B-trees – Deletion Case4: underflow & ‘poor sibling’ (eg.,
delete 13 from T0)
1 3
6
7
<6
>6
A: merge w/ ‘poor’ sibling
9
B-trees – Deletion Case4: underflow & ‘poor sibling’ (eg.,
delete 13 from T0)
1 3
6
7
<6
>69
FINAL TREE
B-trees – Deletion Case4: underflow & ‘poor sibling’ -> ‘pull key from parent, and merge’ Q: What if the parent underflows? A: repeat recursively
B-tree deletion - pseudocodeDELETION OF KEY ’K’
locate key ’K’, in node ’N’
if( ’N’ is a non-leaf node) {
delete ’K’ from ’N’;
find the immediately largest key ’K1’;
/* which is guaranteed to be on a leaf node ’L’ */
copy ’K1’ in the old position of ’K’;
invoke this DELETION routine on ’K1’ from the leaf node ’L’;
else {
/* ’N’ is a leaf node */
... (next slide..)
B-tree deletion - pseudocode/* ’N’ is a leaf node */ if( ’N’ underflows ){ let ’N1’ be the sibling of ’N’; if( ’N1’ is "rich"){ /* ie., N1 can lend us a key */ borrow a key from ’N1’ THROUGH the parent node; }else{ /* N1 is 1 key away from underflowing */ MERGE: pull the key from the parent ’P’, and merge it with the keys of ’N’ and ’N1’ into a new
node; if( ’P’ underflows){ repeat recursively } } }
B-trees in practiceIn practice: no empty leaves; pointers to records
1 3
6
7
9
13
<6
>6 <9 >9theory
B-trees in practiceIn practice: no empty leaves; pointers to records
1 3
6
7
9
13
<6
>6 <9 >9
practice
B-trees in practiceIn practice:
1 3
6
7
9
13
<6
>6 <9 >9
Ssn ……
3
7
6
9
1
B-trees in practice
In practice, the formats are:- leaf nodes: (v1, rp1, v2, rp2, … vn, rpn)- Non-leaf nodes: (p1, v1, rp1, p2, v2, rp2, …)
1 3
6
7
9
13
<6
>6 <9 >9
Overview primary / secondary indices multilevel (ISAM)
B – trees
B+ - trees
hashing
B+ trees - Motivation
B-tree – print keys in sorted order:
1 3
6
7
9
13
<6
>6 <9 >9
B+ trees - Motivation
B-tree needs back-tracking – how to avoid it?
1 3
6
7
9
13
<6
>6 <9 >9
Solution: B+ - trees Facilitate sequential ops
They string all leaf nodes together
AND
Replicate keys from non-leaf nodes, to make sure every key appears at the leaf level !!
B+ trees
1 3
6
6
9
9
<6
>=6 <9 >=9
7 13
B+-Trees (Cont.)
All paths from root to leaf are of the same length
Each node that is not a root or a leaf has between [n/2] and n children
A leaf node has between [(n–1)/2] and n–1 values
Special cases: If the root is not a leaf, it has at least 2 children If the root is a leaf (that is, there are no other nodes
in the tree), it can have between 0 and (n–1) values
A B+-tree is a rooted tree satisfying the following properties:
B+-Tree Node Structure Typical node
Ki are the search-key values Pi are pointers to children (for non-leaf
nodes) or pointers to records or buckets of records (for leaf nodes).
The search-keys in a node are ordered K1 < K2 < K3 < . . . < Kn–1
Leaf Nodes in B+-Trees - Properties
For i = 1, 2, . . ., n–1, pointer Pi either points to a file record with search-key value Ki, or to a bucket of pointers to file records, each record having search-key value Ki. Only need bucket structure if search-key does not form a primary key.
If Li, Lj are leaf nodes and i < j, Li’s search-key values are less than Lj’s search-key values
Pn points to next leaf node in search-key order
Non-Leaf Nodes in B+-Trees - Properties
Non leaf nodes form a multi-level sparse index on the leaf nodes. For a non-leaf node with m pointers: All the search-keys in the subtree to which P1 points
are less than K1
For 2 i n – 1, all the search-keys in the subtree to which Pi points have values greater than or equal to Ki–
1 and less than Km–1
B-Tree vs B+-Tree
B-tree (above) and B+-tree (below) on same data
B+ tree insertionINSERTION OF KEY ’K’ insert search-key value to ’L’ such that the keys are in order; if ( ’L’ overflows) { split ’L’ ; insert (ie., COPY) smallest search-key value of new node to parent node ’P’; if (’P’ overflows) { repeat the B-tree split procedure recursively; /* Notice: the B-TREE split; NOT the B+ -tree */ } }
B+-tree insertion – cont’d
/* ATTENTION:
a split at the LEAF level is handled by COPYING the middle key upstairs;
A split at a higher level is handled by PUSHING the middle key upstairs
*/
B+ trees - insertion
1 3
6
6
9
9
<6
>=6 <9 >=9
7 13
Eg., insert ‘8’
B+ trees - insertion
1 3
6
6
9
9
<6
>=6 <9 >=9
7 13
Eg., insert ‘8’
8
B+ trees - insertion
1 3
6
6
9
9
<6
>=6 <9 >=9
7 13
Eg., insert ‘8’
8
COPY middle upstairs
B+ trees - insertion
1 3
6
6
9<6
>=6 <9>=9
9 13
Eg., insert ‘8’
COPY middle upstairs
7 8
7
B+ trees - insertion
1 3
6
6
9<6
>=6 <9>=9
9 13
Eg., insert ‘8’
COPY middle upstairs
7 8
7
Non-leaf overflow – just PUSH the middle
B+ trees - insertion
1 3
6
6
<6
>=6>=9
9 13
Eg., insert ‘8’
7 8
7
9
<7 >=7
<9
FINAL TREE
B-Trees vs B+-Trees
Advantages of B-Tree indices: May use less tree nodes than a corresponding B+-Tree. Sometimes possible to find search-key value before reaching
leaf node. Disadvantages of B-Tree indices:
Only small fraction of all search-key values are found early Non-leaf nodes are larger, so fan-out is reduced. Thus B-Trees
typically have greater depth than corresponding B+-Tree Insertion and deletion more complicated than in B+-Trees Implementation is harder than B+-Trees.
Typically, advantages of B-Trees do not out weigh disadvantages
B*-tree In B-trees, worst case util. = 50%,
if we have just split all the pages how to increase the utilization of B
- trees?
… with B* - trees!
B-trees and B*-trees
E.g., Tree T0; insert ‘2’
1 3
6
7
9
13
<6
>6 <9 >9
2
B*-trees: deferred split! Instead of splitting, LEND keys to
sibling!(through PARENT, of course!)
1 3
6
7
9
13
<6
>6 <9 >9
2
B*-trees: deferred split! Instead of splitting, LEND keys to
sibling!(through PARENT, of course!)
1 2
3
6
9
13
<3
>3 <9 >9
2
7
FINAL TREE
B*-trees: deferred split!
Notice: shorter, more packed, faster tree
It’s a rare case, where space utilization and speed improve together
BUT: What if the sibling has no room for our ‘lending’?
B*-trees: deferred split!
BUT: What if the sibling has no room for our ‘lending’?
A: 2-to-3 split: get the keys from the sibling, pool them with ours (and a key from the parent), and split in 3.
Details: too messy (and even worse for deletion)
Conclusions all B – tree variants can be used for
any type of index: primary/secondary, sparse (clustering), or dense (non-clustering)
All have excellent, O(logN) worst-case performance for ins/del/search
It’s the prevailing indexing method
Overview ordered indices
primary / secondary indices index-sequential multilevel (ISAM)
B - trees, B+ - trees
hashing static hashing dynamic hashing