Post on 17-Jan-2018
description
CMPT 454File Storage and Indexing
Files and Indexing
File Organizations Indices
Types of index Tree based indexing Hash based indexing
Finding Records
Accessing a table Consider SELECT * FROM Customer
To make access to a table efficient Store the table on adjacent blocks On the same cylinder, or Adjacent cylinders
But many queries include where clauses File organizations should support the
efficient retrieval of records within a file
File Organizations
File Organizations Data in a database consists of collections of
records, or data files Each file consists of one or more blocks on a disk
A file organization is a method of arranging records in a file File organizations make some operations more
efficient In addition, files can be indexed, to provide
multiple ways to access records efficiently Index files contain search key values and
references to records
External Storage - Reminder Database data must be persistent, so must
be stored on secondary memory, such as a hard disk
Disk access is relatively inefficient, in the order of 10 to 15 milliseconds to access a single page Hundreds of thousands of times more than an
equivalent access to a main memory location The cost of disk I/O dominates the cost of database
operations The unit that data is read from or written to
disk is a block, typically 8 kilobytes Reading several pages in sequence from a disk
takes much less time than reading several random pages
File Organizations The minimum set of file operations is:
Create and destroy files Insert and delete records Scan an entire file▪ A scan brings all of the records in the file into main
memory A single DB table is usually stored as a single
file Every record in a file has a unique record ID, or rid ▪ A rid consists of a block address, and a slot number
The simplest file structure is an unordered file, or heap file
Heap Files Heap files support insertion and deletion of
records and file scans Because the entire file can be scanned, individual
records or collections of records can be found New records are inserted where there is room
Either in slots that contained previously deleted records, or
At the end of the file When a record is deleted, no other records
are affected That is, there is no need to reorganize remaining
records
Sorted Files The records in a sorted file, or sequential file,
are stored in order Based on the sort key of the file▪ The attribute of the record that the file is sorted on
The basic organization assumes that file pages are filled to conserve space
Pages should be maintained in sequence To allow for more efficient disk access
Insertions result in records being shuffled up Deletions result in records being shuffled
down
Sorted Files in Practice To avoid inefficiencies involved in inserting
and deleting records from sorted files Pages have only partial occupancy▪ i.e. space for future insertions is left in each page
Overflow pages can be attached (by pointers) to pages that become full
Records can be locally reorganized in adjacent pages
Sorted files may need to be periodically re-ordered
Indices (or Indexes)
Indexes An index is a data structure that organizes
data to optimize the retrieval of records on some criteria An index supports efficient retrieval of records
based on the search key of the index An index can be created for a file to speed up
searches that are not efficiently supported by the file's organization
A file can have more than one index An index is a collection of data entries which
must contain: A search key value, k, and Information to find data records with that search
key value
Primary Indexes
An index on a sequential file The search key of the index is the same
as the sort key of the file Primary indexes can be either dense
or sparseThere are different kinds of keysPrimary key
Sort keySearch key
Candidate keySuperkey
Dense Indexes A dense index is a sequence of blocks containing
search key : rid pairs The rid contains addresses to records with the search
key The blocks of the index are in the same order as the file
Searching an index is faster than searching the file The index is smaller The index is sorted so binary search can be used The index may be small enough to fit in main memory▪ If so, once the index has been read, records can be found with
one disk I/O
index
Dense Index
1020
3041
5360
7791
10203041
53607791
Two data records or four index records fit on one block
file
Sparse Indexes A sparse index usually contains one data
entry for each block of records in a data file It is only possible to use a sparse index if the
data file is sorted by the search key of the index Sparse indexes are smaller than dense indexes
Sparse indexes are searched in much the same way as dense indexes Except that the index is searched for the largest
key less than or equal to the target value The rid is then followed to a block of the data
file
index
Sparse Index
1020
3041
5360
7791
10305377
...
file
Two data records or four index records fit on one block
Multiple Level Indices
An index on a large data file can cover many blocks Even using binary search, multiple disk I/Os
may be needed to find a record An alternative is to build a multiple level index
The first level of the index may be dense or sparse Subsequent levels of the index are sparse
indexes on the preceding level of the index
multiple level index
Multiple Level Index
1020
3041
5360
7791
10203041
53607791
file1053...
Secondary Indexes
We define a primary index is an index whose search key is the same as the sort key of a sequential file There can only be one primary index for a file▪ Terminology – a primary index sometimes refers to an index where
the search key includes the primary key, we do not use this definition▪ More terminology – primary indices are also referred to as clustered
A secondary index is an index whose search key is not the sort key of the file Secondary indexes must be dense▪ Why?
Secondary indices are also referred to as unclustered
index
Secondary Index
10 bob20 dave
30 ann41 dave
53 kim60 ann
77 dave91 lee
annannbobdave
davedavekimlee
file
Using Secondary Indexes The pointers in a block of a
secondary index may point to many blocks of the data file Making secondary indexes less efficient
than primary indexes for retrieval of a range of records
Heap (data) files are not ordered so require secondary indexes
Clustered Files A clustered file contains the records of two
tables Consider two tables in a one to many relationship Where queries containing a join between the two
tables are made frequently Note the unfortunate re-use of the word clustered
...SELECT pet_name, speciesFROM Owner, PetWHERE Owner.sin = 111 AND Owner.sin = Pet.sin
Clustered File Organization
owner 1
owner 2
owner 3
owner 4
...Pets of
owner1
Pets of owner2
Pets of owner3
Pets of
owner4
Clustered Files and Indexes Consider a primary index on Owner
The file is sorted by owner Individual owners can be retrieve rapidly Pets of owners can also be retrieved
rapidly Efficiency is reduced when retrieving
a range of owner data with no pet data
Any index on attributes of Pets would be a secondary index
Indirection A secondary index may waste space if
search key values are repeated Since each record ID is paired with its own search
key Create a bucket for each set of rids
associated with a search key Follow a pointer to the bucket, then Follow the rids in the bucket to the records
Saves space if search key values are larger in bytes than record IDs And each key appears at least twice on average
index
Indirection
10 bob20 dave
30 ann41 dave
53 kim60 ann
77 dave91 lee
annbobdavekim
lee...
file
...
buckets
Multiple Secondary Indexes Multiple secondary indexes using
indirection can improve efficiency on queries with complex criteria Collect all the rids from the buckets that meet
each of the criteria Then intersect them And only retrieve records using the result
This avoids retrieving records that match some, but not all, of the criteria
Document Retrieval
There are issues related to the storage and efficient retrieval of documents Keywords are used to identify documents More and more documents are maintained on
the web A document can be considered as a record
in a table The record can be thought of as having
Boolean attributes for each possible word in the document
An attribute is true if the word is in the document and false otherwise
Inverted Indexes
Consider a secondary index on each attribute (word) of a document Only those records where an attribute is true is
contained in the index▪ So the index leads to documents with particular words
Indices for all attributes (words) are combined into a single index Known as an inverted index The index uses indirect buckets for space
efficiency
Matching Documents
Inverted indexes of words in multiple documents can be used to satisfy queries Using intersection, documents that
contain multiple target words can be found
Additional information can be maintained to match words in sections of documents Titles, headers, and other sections Number of occurrences of words ...
Composite Indices An index’s search key can contain several fields
Such search keys are referred to as composite search keys or concatenated keys▪ e.g. {fName, lName}
For searches on equality the values for each field in the search key must match the values in record e.g. 'Joe Smith' does not match 'Joe Jones' or 'Fred Smith'
For range queries, ranges may be specified for any fields in the search key. If no values are specified for a field it implies that any
value is acceptable for that field
Tree Based Indexes
Tree Index Introduction
Multiple level indexes can be very useful in speeding up queries There is a general data structure that is used in
commercial DBMSs Known as B trees▪ We will look at B+ trees, a commonly used variant
B trees have two desirable properties They keep as many levels as are required for the file
being indexed Space on tree blocks is managed so that each block is at
least ½ full
B Tree Structure
B trees are balanced structures All paths from the root to a leaf have the same
length Most B trees have three levels But any number of levels is possible
B trees are similar to binary search trees Except that B tree nodes contain more than two
children That is, they have greater fan-out B tree node size is chosen to be the same as a
disk block
B+ Tree Node Structure The number of data entries in a node is
determined by the size of the search key Up to n search key values and n + 1 pointers The value n is chosen to be as large as possible and
still allow n search keys and n + 1 pointers to fit on a block
Example If block size is 4,096, and the keys are 4 byte
integers and pointers are 8 byte addresses Find the largest value n such that 4n + 8(n + 1) ≤
4,096 n = 340
Leaf Nodes Search keys in leaf nodes are copies of the
keys in the data file The leaf nodes contain the keys in order The left most n pointers point to records in the data
file▪ A leaf node must use at least (n + 1)/2 of these pointers
The right most pointer points to the next leaf12
24
29
record with key
12
record with key
24
record with key
29
next leaf in the tree
Interior Nodes In interior nodes, pointers point to next level nodes
Label the search keys K1 to Kn, and pointers p0 to pn▪ Pointer p0 points to nodes whose search key values are less than K1
▪ Other pointers, pi, point to nodes with search keys greater than or equal to Ki and less than Ki+1
An interior node must use at least (n + 1)/2 pointers
12
24
29
K < 12
12 ≤ K < 24
K ≥ 29
24 ≤ K < 29
Example B+ Tree17
10
27
87
2 5 10
13
15 17
22
27
35
87
91
note that (n+1)/2
= 2
in this example n
= 3
B+ Tree Applications
B+ trees can be used to build many different indices A B+ tree could be a sparse index on a sorted
data file, or A dense index on a data file
We will assume for now that there are no duplicate values for search keys That is the search key is a candidate key for the
relation The meaning of interior nodes changes slightly
if there are duplicate search key values
Searching a B+ Tree The B+ tree search algorithm is similar to a BST
To search for a value K start at the root and end at a leaf
If the node is a leaf and the ith key has the value K then follow the ith pointer to the record
If the node is an interior node follow the appropriate pointer to the next (interior or leaf) node
Searching a B+ tree index requires a number of disk I/O operations equal to the height of the tree Plus one I/O to retrieve the record▪ If there are multiple searches on the same table the root of the
tree is probably in main memory
B+ Tree Searches17
10
27
87
2 5 10
13
15 17
22
27
35
87
91
which nodes are visited in a search for 22?which nodes are visited in a search for 16?
Range Queries B+ trees are useful for processing range
queries A range query typically has a WHERE clause that
specifies a range of values Assume a query specifies values from x to y
Search the tree for the leaf that should contain value x
Follow the leaf pointers until a key greater than y is found
The tree can also be used to satisfy queries that have no lower bound or no upper bound
B+ Tree Insertions Insert the record in the in the data file
Retaining the rid and the search key value, K Insert the entry in the appropriate place in a
leaf Use the search algorithm to find the leaf node Insert a data entry, if it fits, the process is complete
If the target leaf node is full then split it The first (n + 1) / 2 entries stay in the original node Create a new node with the remaining (n + 1) / 2
entries to the right of the original node Insert an entry with the first search key value from the
new leaf in its parent node that points to the new leaf
B +Tree Insertions Adding an entry to an interior node may cause it
to split After inserting a new entry there should be n + 1 keys
(and n + 2 pointers) The first (n + 2) / 2 pointers stay in the original node Create a new node with the remaining (n + 2) / 2
pointers to the right of the original node Leave the first n / 2 keys in the original node and move
the last n / 2 keys to the new node The remaining key 's value falls between the values in
the original and new node This left over key is inserted into the parent of the
node along with a pointer to the new interior node
B+ Tree Insertions continued Moving a value to a higher, interior level
of the tree, may again cause a split The same process is repeated until no further
splits are required, or until a new root node has been created
If a new root is created it will initially have just one key and two children So will be less than half full This is permitted for the root (only)
B+ Tree Insertion Example
2 11
21insert 2, 21 and
11
data file
the values are maintained in order in
the index pages
B+ Tree Insertion Example
11
insert 8
create a new node with the last ½ of the values
2 11
21
chain the new node to the original node
11
21
2 8
create new root with the first value of the new leaf
node
B+ Tree Insertion Example
11
insert 64, then 5
both leaf nodes are now full ...
11
21
2 82 5 8 11
21
64
insert 23 ...
2 5 8
B+ Tree Insertion Example
11
... inserting 23 ...
23
64
11
21
6411
21
11
23
11
23
11
21
2 5 8
B+ Tree Insertion Example
insert 97
23
64
23
64
97
and 6
11
23
11
21
6 8
B+ Tree Insertion Exampleinsert 6
23
64
6 11
23
2 5 82 5
6 11
23
11
21
6 8
insert 6
23
64
2 5 82 5
B+ Tree Insertion Example
6 8
6 11
23
6 8 9 11
21
B+ Tree Insertion Example
insert 19 and 9
23
64
2 5 82 5 11
19
21
6 11
23
6 8 9
B+ Tree Insertion Examplethe same
tree ...
2 5 11
19
21 23
64
6 8 9
B+ Tree Insertion Exampleinsert 7
and now insert 8 in the parent
6 72 5 11
19
21 23
64
8 9
6 11
23which will
require that it splits
23
6 7
6 11
23
11
B+ Tree Insertion Exampleinsert 7 –
inserting 8 and a pointer in the
root node
6 8
and make the middle value the
new root
2 5 11
19
21 23
64
8 9
move ½ the pointers and the last n/2
values
keep ½ the pointers and the first n/2
values
6 8 23
6 7
11
B+ Tree Insertion Exampletree after
inserting 7
2 5 11
19
21 23
64
8 9
6 8 23
45
60
6 7
11
B+ Tree Insertion Example
insert more values ...
2 5 11
19
21 23
31
398 9 45
51
60
64
93
6 8 23
45
60
6 7
11
B+ Tree Insertion Example
and insert 77
2 5 11
19
21 23
31
398 9
77
93
45
51
60
64
9360
64
60
64
6 8 23
45
60
6 7
11
B+ Tree Insertion Example
... and insert 77 ...
2 5 11
19
21 23
31
398 9
77
93
45
51
77
23
45
now insert 60 in the root
23
45
60
64
6 8
6 7
11
B+ Tree Insertion Example
... and insert 77 ...
2 5 11
19
21 23
31
398 9
77
93
45
51
77
11
60 now insert 60 in the
root
B+ Tree Deletions Find the entry in the leaf node and delete it
This may result in there being too few entries in the node
If so select an adjacent sibling of the node and Redistribute values between the two nodes
So that both nodes have enough entries If this is not possible
Coalesce the two nodes Delete the appropriate value and pointer in the
parent node
Redistributing Values A value and pointer are removed from an adjacent
sibling and inserted in the node with insufficient entries The sibling can be the left or the right sibling, although it
makes a slight difference to the process The chosen node must be a sibling to ensure that only a
single parent node is affected After redistribution, one of the two nodes will have a
different first search key value The corresponding value in the parent node must be
changed to this value If the node's sibling(s) have insufficient entries
redistribution may not be possible
Coalescing Nodes When redistribution is not possible, two
nodes can be combined (or coalesced if you prefer) Keep track of the value in the parent between the
pointers to the two nodes to be combined Insert all of the values (and pointers) from one node
into the other Re-connect links between leaves (if the nodes are
leaves) Make a recursive call to the deletion process,
deleting the identified value in the parent node This, in turn, may require non-leaf nodes to be
coalesced
B+ Tree Deletion Notes The deletion algorithm requires a choice
to be made between siblings Such a choice has to be implemented in the
algorithm Coalescing nodes requires more work
It may result in making changes up the tree, but
The tree height may be reduced Redistributing nodes requires less work,
but does not impact the height of the tree
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 1
delete 19
2 5 11
19
21 23
31
398 9
77
93
45
51
77
11
60
use search to find the value
11
21
delete the record in the data file first
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 2
delete 45
2 5 11
19
21 23
31
398 9
77
93
45
51
77
11
60
the node is less than half full: (n + 1)/2 pointers to records
51
51
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 2
delete 45
2 5 11
19
21 23
31
398 9
77
93
77
11
60
take a value from the left sibling
node
change the value in the parent
node
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 2
delete 45
2 5 11
19
21 23
31
398 9
77
93
51
77
11
60
take a value from the left sibling
node
39
51
23
31
23
39
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 3
delete 9
2 5 11
19
21 23
31
398 9
77
93
45
51
77
11
60
leaf nodes must use (n + 1)/2 =
2 pointers
not a sibling
so must coalesce the node with its
L sibling
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 3
delete 9
2 5 11
19
21 23
31
398 9
77
93
45
51
77
11
60
leaf nodes must use (n + 1)/2 =
2 pointers
not a sibling
so must coalesce the node with its
L sibling
6 7 8
now delete entry and pointer from
parent
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 3
delete 9
2 5 11
19
21 23
31
39
77
93
45
51
77
11
60
6 7 8
now delete entry and pointer from
parent
6
note that these nodes have enough
(2) pointers
23
45
60
64
6 8
6 7
11
B+ Tree Deletion Example 3
delete 6
2 5 11
19
21 23
31
39
77
93
45
51
77
11
60
6 7 8
6
7 8
the parent entry doesn't need to
change
23
45
60
64
6 8
11
B+ Tree Deletion Example 3
delete 8
2 5 11
19
21 23
31
39
77
93
45
51
77
11
60
2 5 7
now delete entry and pointer from
parent
6
7 8
6
2 5 7
23
45
60
64
11
B+ Tree Deletion Example 3
… delete 8 …
11
19
21 23
31
39
77
93
45
51
77
11
60
now delete entry and pointer from
parent
just 1 pointer
so take a pointer from
sibling
which value?
2 5 7
23
45
60
64
11
B+ Tree Deletion Example 3
… delete 8 …
11
19
21 23
31
39
77
93
45
51
77
11
60
45
which value?23
60
11
11
45
23
60
2 5 7 60
64
B+ Tree Deletion Example
… delete 8 finished
11
19
21 23
31
39 77
93
45
51
77
11
45
23
60
2 5 7 60
64
B+ Tree Deletion Example
delete 23 and 31?
11
19
21 23
31
39 77
93
45
51
77
B+ Tree Efficiency Splitting and merging of index blocks is rare
Typically the value of n will be much greater than 3!
Most splits or merges are limited to two leaves and one parent
The number of disk I/Os is based on the tree height It is a reasonable assumption that the majority of
B trees have a height of 3 And one level is the root (i.e. one block) which
can reside in main memory
B+ Tree Height
Assume a block size of 4,096 and rid and data entry size of 8 bytes Each tree node can contain 340 key
values and pointers If each node is 2/3 full that is 255
pointers How many records can be accessed
by such a tree with 3 levels? 2553 = 16,600,000 records 2554 = 4,228,250,625 records
Deletions Revisited
Some B+ tree implementations don't fix interior nodes for deletions If a leaf has too few keys and pointers it
is allowed to remain unchanged It is assumed that most DB files tend to
grow not shrink It also allows efficient access to
records replaced by tombstones in the data file
B+ Trees and Duplicates The search algorithm assumes that all search
key entries with a given key are in the same node If duplicate values are allowed this may not be the
case Three methods for dealing with duplicate
values are Maintain overflow pages for duplicates where
necessary Include the rid as part of the search key▪ Which ensures that there will not be duplicates
Modify the tree to change the meaning of interior nodes▪ Keys in interior nodes represent new keys, that is
keys where the same value does not appear to the left
▪ In some cases this requires null keys
6 - 31
57
6 11
19
B Trees Duplicates Example
find 11
2 5 23
23
23
31
4111
19
23 57
64
and 19and 23
i.e. no new keys in 2nd child
never follow this pointer!
Bulk Loading A B+ tree can be created by repeated
inserts of data using the insertion algorithm This process is likely to be inefficient as the same
disk pages may be accessed more than once An alternative is to initially sort the data and
fill the leaf nodes Non-leaf nodes can be created as necessary to
create the index Space should be left in all nodes to
accommodate future insertions
Key Compression The height of a tree is determined by the
fan-out The number of children of each node The fan-out is determined by the number of
search key values and pointers that can fit in one page
A smaller search key leads to a greater fan-out and a tree with fewer levels
It may be possible to compress search keys For example, a search key on last name can be
truncated to the extent that it is still sufficient to guide the search
Hash Indexes
Hash Tables in Main Memory In a hash table a hash function maps
search key values to array elements The array can either contain the data objects, or Linked lists containing data objects, call these
buckets Hash functions generate a value between 0
and B-1 Where B is the number of buckets A record with search key K is stored in bucket
h(K)
Hash Indexes on Disk Buckets should consist of single blocks
Full buckets can be chained to overflow blocks The bucket locations need to be recorded
An array of pointers to buckets, or The first block of each bucket is stored in
consecutive disk locations The number of buckets should be greater than
Number of data entries entries per page A hash function would compute the remainder of K/B▪ Where K is the key value and B is the number of buckets
Hash Functions
A good hash function should evenly distribute values over the buckets A hash function should be both uniform and
random▪ Buckets should be assigned the same number of
values from the set of all possible values, and▪ On average each bucket should contain the same
number of entries, i.e. evenly distribute the actual values
Note that buckets are expected to contain more than one search key value
A typical hash functions is a bit representation of the search key value modulo the number of buckets
Hash Index Operations
Compute h(K) when a new record is to be inserted Insert the record in the bucket (or its overflow
blocks)▪ If necessary add an overflow block
Deletion is similar to insertion Use the hash function to find bucket h(K) Delete any records with search key K Consider combining blocks if possible
Hash Index Efficiency
There should be enough buckets so that most fit on one block i.e. there are few overflow blocks Most lookups require only two disk I/Os
The number of buckets is fixed Such indexes are referred to as static
hash tables If the file grows most buckets will have
overflow chains, reducing efficiency Hash indexes do not support range
lookup
Dynamic Hashing
Two versions of dynamic hashing Extensible hashing, and Linear hashing
Both systems use the concept of a family of hash functions As the size of the index grows larger, more disk
pages are required to store the data entries Rather than creating overflow pages, additional
buckets are created, and The range of the hash function is increased
Hash Function Families
Consider a hash function that returns some bit value e.g. 0100 1101 0110 0101 The bucket can be derived by calculating the
bit value modulo n (the number of buckets) Note that if n is a power of 2 the bucket can be
determined by looking at the last k bits (where n = 2k)▪ e.g. if n = 8 the bit value shown above maps to bucket
5 The range of a hash function can be doubled
by increasing the number of relevant bits by one
Extensible Hashing Structure In extendible hashing there is a directory to
buckets The directory is an array of pointers The array's size is always a power of 2▪ If the directory needs to increase in size, it doubles
New buckets are only created as necessary The hash function computes a sequence of bits
The directory (and the associated buckets) uses a smaller number of i bits
The directory will have 2i entries▪ When the directory grows i+1 bits of the hash value are used
Extensible Hashing Directory The directory consists of an array of pointers
to buckets As the array only contains pointers it is relatively
small, so Can usually fit on one page
The array index is calculated with the hash function And is determined by the relevant bits of the hash
value The array size is determined by how many bits of
the hash function are being used Even though the directory may become large,
new buckets are only created when overflow occurs
Extensible Hashing Example
The array index is the last two bits of the hash value
Assume that three data entries fit in one bucket i.e. one disk page
The values shown are the decimal equivalents of the hash values Not search key values
Only four index pages are currently required
64
12
2
00011011
directory
1 17
5 2
6 2
31
15
2
j –the number of bits used to determine membership in bucket – appears in the
block header (local depth)
i = 2
i –number of bits of the hash value being used by the directory
(global depth)
Extensible Hashing Example
0100 0000
0100 0010
1
01
Only the last bit of the hash value is currently used to determine which blocks records are inserted into
i = 1
insert record where h(K) = 1100 1000
0000 1101
0100 0011
1
follow pointer in index 0 of the directory (as i = 1 only one bit of the
hash value is being used)
0100 0000
0100 0010
1100 1000
1
01
Extensible Hashing Example
00011011
i = 10000 1101
0100 0011
1
0100 0000
0100 0010
1100 1000
1
insert record where h(K) = 1101 1010
the block is fullcompare j to i
j = i so double the
directory size
distribute the new pointers based on the
previous index
split the block that was full
i = 2
00011011
Extensible Hashing Example
0100 0010
1101 1010
2
i = 20000 1101
0100 0011
1
0100 0000
0100 0010
1100 1000
1
insert record where h(K) = 1101 1010
split the block that was full 0100
00001100 1000
2
use the last two digits of
the hash value as i = 2
and increment j
adjust pointers to
the block that has split
00011011
0100 0000
1100 1000
2
Extensible Hashing Example
0100 0010
1101 1010
2
i = 20000 1101
0100 0011
10000 1101
0100 0011
1111 1101
1
insert record where h(K) = 1111 1101
00011011
0000 1101
0100 0011
1111 1101
1
0100 0000
1100 1000
2
Extensible Hashing Example
0100 0010
1101 1010
2
i = 20000 1101
1111 1101
1001 1001
2
insert record where h(K) = 1001 1001
so split the block but
don’t double directory size
increment j in the new blocks
and adjust pointers to
the new block
the block is full, but j < i
0100 0011
2
00011011
000
001
010
011
100
101
110
111
0000 1101
1111 1101
1001 1001
2
0100 0000
1100 1000
2
Extensible Hashing Example
0100 0010
1101 1010
2
i = 2
insert record where h(K) = 0110 1101
the block is full, and j = i
0100 0011
2
i = 3
000
001
010
011
100
101
110
111
i = 30000 1101
1111 1101
1001 1001
21001 1001
3
0100 0000
1100 1000
2
Extensible Hashing Example
0100 0010
1101 1010
2
insert record where h(K) = 0110 1101
split block and distribute
values
the block is full, and j = i
0100 0011
2
0000 1101
1111 1101
0110 1101
3
1001 1001
3000
001
010
011
100
101
110
111
i = 3 0100 0000
1100 1000
2
Extensible Hashing Example
0100 0010
1101 1010
2
insert record where h(K) = 1110 1101?
0100 0011
2
0000 1101
1111 1101
0110 1101
3
Extensible Hashing Deletion If a deletion empties a bucket it can
be merged with its counterpart The pointer entries in the directory are
reset and The existing bucket's local depth is
decremented In practice this is not usually
performed
Extensible Hashing Efficiency If the directory fits in main memory, performance is
identical to static hashing One disk read to use index and one to retrieve record If the directory does not fit in memory, another read is
required▪ In practice, the directory usually fits in memory
Many collisions in a bucket creates a large directory Reducing the chance that it will fit in memory More likely if the number of records per block is small
Increasing the directory size is relatively time consuming and interrupts access to the data file
Extendible Hashing Overflow If many entries have the same hash value
across the entire range of bits overflow pages are created The overflow pages are chained to the primary
pages This can occur if the hash function is poor, or If there are many insertions with the same
search key value (i.e. a skewed distribution) Otherwise repeated insertions of entries
with the same overall hash value would lead to The same bucket being repeatedly split, and The directory repeatedly doubling
Linear Hashing Linear hashing is another dynamic
hashing system The number of buckets (n) is selected to
maintain an average occupancy in buckets e.g. 80%
Buckets are not always split when full So overflow blocks are allowed The average number of overflow blocks per
bucket is less than 1 The number of bits used to number the
entries in the bucket array is log2n
Locating Buckets
Linear hashing does not use a directory The hash function determines which
bucket a record is mapped to The primary blocks of the buckets are stored
sequentially So that bucket m can be found by adding m to
the address of the first bucket Like extensible hashing only the right
most i bits of h(K) are used to determine a bucket
Hash Function
At any time i bits of the hash value are used to map records to n buckets Values for the i bits range from 0 to 2i-1 The value for n may be less than 2i-1
Computing h(K) on a search key value K results in a value m for the last i bits of h(K) If m < n place the record in bucket m If n ≤ m < 2i then place m in bucket m - 2i-1
▪ i.e. change the left most bit of m to 0
Hash Function Example
000
…001
…010
…011
…100
…101
…i = 3
insert record where h(K) = 1001 0100
n = 6
place record in bucket 100 as 4
(100) < n
insert record where h(K) = 0101 1110
using 3 bits record is mapped to bucket 110 which does not exist
so just use i-1 bits (or subtract 2i-1
from 110)
Splitting Buckets
Periodically buckets are added to the index When the ratio r / n (r records, n buckets)
exceeds a threshold value a new bucket is created
The bucket that is added to the index may not have any relationship to the bucket that was just inserted into
When a bucket is added values in its related bucket are distributed That is the bucket whose index is the index of
the new bucket - 2i-1
When n > 2i, i is incremented by one
Linear Hashing Example
00 0100 0000
0100 010001 0000
11010100 0011
1001 010110 0100
0010
insert record where h(K) = 1100 1011
1100 1011
n = 3
r = 6max occupancy
= 0.8 so max r / n = 2.4
i = 2
insert in bucket 11
r = 7
which does not exist so insert
in 01
add an overflow block to the
bucketr / n < 2.4 so no new bucket is
created
r = 7
Linear Hashing Example
00 0100 0000
0100 010001 0000
11010100 0011
1001 010110 0100
001011 0100 0011
1100 1011
insert record where h(K) = 1000 0110
n = 3
r = 8max occupancy
= 0.8 so max r / n = 2.4
i = 2
insert in bucket 10
r / n = 2.667, so make a new
bucket
the next bucket is 11, so distribute the
values in 01
1100 101110 0100
00101000 0110
01 0000 1101
1001 0101
n = 4
01 0000 1101
1001 0101
n = 4
r = 8
10 0100 0010
1000 0110
r = 9
Linear Hashing Example
00 0100 0000
0100 0100
10 0100 0010
1000 0110
1011 111011 0100
00111100 1011
insert record where h(K) = 1011 1110
max occupancy = 0.8 so max r /
n = 2.4
i = 2
insert in bucket 10
r / n = 2.25, so don't make a new bucket
01 0000 1101
1001 0101
n = 4
r = 9
10 0100 0010
1000 0110
1011 1110
r = 10
Linear Hashing Example
00 0100 0000
0100 0100
11 0100 0011
1100 101110
00100 0100
insert record where h(K) = 0101 0110
n = 5max occupancy
= 0.8 so max r / n = 2.4
i = 2
insert in bucket 10
r / n = 2.5, so make a new
bucket
the next bucket is 100, so distribute the
values in 00
0101 0110
n = 2i, so increase i to 3
00 0100 0000
i = 3
i = 3
n = 5
r = 10
010
0100 0010
1000 0110
1011 1110
Linear Hashing Example
000
0100 000000
10000 1101
1001 0101
011
0100 0011
1100 101110
00100 0100
insert record where h(K) = 1101 1101
max occupancy = 0.8 so max r /
n = 2.4
insert in bucket 101
r / n = 2.2, so don't make a new bucket
0101 0110
n – 1 (100) < 101 so insert in
001001
0000 1101
1001 0101
1101 1101
r = 11
r = 11
001
0000 1101
1001 0101
1101 1101
000
0100 0000
i = 3
n = 5
r = 12
010
0100 0010
1000 0110
1011 1110
Hash Function Example
011
0100 0011
1100 101110
00100 0100
insert record where h(K) = 1001 1100
max occupancy = 0.8 so max r /
n = 2.4
insert in bucket 100
r / n = 2.4, so don't make a new bucket
0101 0110
100
0100 0100
1001 1100
r = 12r = 13
001
0000 1101
1001 0101
1101 1101
000
0100 0000
i = 3
n = 5
010
0100 0010
1000 0110
1011 1110
Linear Hashing Example
011
0100 0011
1100 101110
00100 010010
10000 1101
1001 0101
1101 1101
insert record where h(K) = 1000 1000
max occupancy = 0.8 so max r /
n = 2.4
insert in bucket 000
r / n = 2.6, so make a new
bucket
0101 0110
100
0100 0100
1001 1100
000
0100 0000
1000 100000
1
n = 6
the next time a new bucket is created the values in 010 will be
distributed into it
Extensible and Linear Hashing Linear hashing does not require a
dictionary Linear hashing may result in less space
efficiency because buckets are split before they overflow
Multiple collisions in one bucket in extensible hashing will result in a large directory Such a directory may not fit on one disk page
Collisions in linear hashing lead to long overflow chains for the bucket with the collisions Requiring multiple disk reads for that bucket But no increase in the cost of accessing other
buckets
Specialized Index Techniques
Queries with Complex Conditions Queries may contain complex
conditions ... where name = 'bob' and age > 50
Indexes on the queried attributes can be used to satisfy the result By retrieving matching rids and taking
the intersection of the result Note that this general strategy works
poorly for disjunctions (or clauses)
Composite Search Keys If two attributes are frequently queried
together they can be combined in an index Called a composite search key A single index where the search key is a
concatenation of two (or more attributes) The search keys in both B+ trees and hash
indices can be composite A B+ tree composite index allows queries on just
the first attribute of the search key to be satisfied
Whereas a hash index does not
Specialized Indices
There are a number 0f motivations for specialized indexes That usually require some variant of complex
range queries Geographic information systems
Partial match queries Range queries Nearest neighbour queries
OLAP databases Queries on multidimensional daa
Multiple Key Indices
An index on indexes An index on one attribute is built
above an index on a second The first index refers to index pages on
the second attribute▪ Search key values in the lower index may be
repeated for different search key values from the first index
This can be generalized to more than two attributes
Multiple Key Index
name index
bob 50kate 33
ann 50dave
40
ann 40ann 30
bob 35dave
75
annbobdavekim ...
lee ...... ...
file
304050
3550
4075
age index
Multiple Key Index Perfomance Multiple key indices work well for
range queries If the individual indices support range
queries They do not support queries where
data for the first attribute is missing Similarly to composite search keys
kd Trees
A k-dimensional search tree that generalizes a binary search tree For multidimensional data
An in-memory data structure that can be adapted to block storage on disk
kd tree nodes contain an attribute name and an associated value Such as {salary, 40000}
kd Tree Structure A kd tree is structurally similar to a binary
search tree Values less than the node’s value are in its left
subtree Values greater than the node’s value are in its
right subtree The attributes at different levels of the tree
are different The levels rotate through the attributes of the tree▪ With two attributes, the levels alternate between the
attributes
kd Tree Example
name kate
age 60 age 47
name bob
age 30
name suejoe 65kat
60
joe 30hil 40
ann
25
art 22
ben
37
ada
40
sue
30
kim
40ned
49
ren 52
sue
47
zak
60
Adapting kd Trees to Disks Leaf nodes of kd trees should be
blocks The interior nodes can be adapted to
be more like B tree nodes With multiple key-pointer pairs Where each interior node is a block
Bitmap Indices
A bitmap index consists of multiple vectors of bits With one vector for each possible value
of the attribute▪ A bitmap to record if a patient was a smoker
would require two bit vectors▪ A bitmap on age might require 100 bit vectors
The ith bit of the index is set to 1 if the ith row of the table has the vector’s value for the attribute
Bitmap Indices A bitmap index can speed up queries on
sparse columns, that have few possible values One bit is allocated for each possible value
The indexes can be used to answer some queries How many male customers have a rating of 3? AND the M and 3 columns and count the 1s M F id nam
esex rating 1 2 3 4 5
1 0 112
Sam M 5 0 0 0 0 1
0 1 113
Sue F 3 0 0 1 0 0
0 1 121
Ann F 2 0 1 0 0 0
1 0 131
Bob M 3 0 0 1 0 0
rating index
gender
index
Using Bitmap Indices Bitmap indexes can satisfy conjunctions
By taking the logical AND of the appropriate vectors
They are also useful OR and NOT conditions By taking the appropriate Boolean combination
of the bit vectors Bitmap indices are often used in databases
for data mining and OLAP Which often have low cardinality attributes And change relatively infrequently
Join Indices
Joins are often expensive operations Join indexes can be built to speed up specific
join queries A join index contains record IDs of matching records
from different tables e.g. Sales, products and locations of all sales in B.C. The index would contain the sales rids and their
matching product and location rids▪ Only locations where province = "BC" are included
The number of such indices can be a problem where there are many similar queries
Alternative Join Indices To reduce the number of join indices
separate indexes can be created on selected columns Each index contains rids of dimension table
records that meet the condition, and rids of matching fact table records
The separate join indices have to be combined, using rid intersection, to compute a join query
The intersection can be performed more efficiently if the new indices are bitmap indices Particularly if the selection columns are sparse The result is a bitmapped join index