Query optimization in relational DBs

Leveraging the mathematical formal underpinnings of the relational model.

Pre-processing a query

Convert SQL to algebra

Create a binary parse tree

Assign execution fragments to each interior node

Optimizing a query: stage 1

Manipulate parse tree to create new, but equivalent trees that will be faster independently of the properties of the database

Select a handful that would most likely be the fastest

Stage 2: collect information about the DB’s physical state

Size of tables being manipulated (in both rows and space)

Sorting order of tables

Location of indices on tables

Distribution of PKs and FKs (to estimate hit ratio on joins)

Hit ratio of attributes referenced by 1-operator commands (select and project)

Stage 3: transform parse tree into a tree that is likely to be faster

Heuristics: unary operations Push unary operations down stream Bundle unary operations on common leaves Goal is to perform unary operations together and before

binary operations, to make binary operations faster

Heuristics: joins Most join algorithms tree the two operands differently Make use of sort orders and index locations

Stage 4: choose implementations of binary operations

If both tables sorted by desired attributes, do a merge join

If one table has an index, to a nested join where the inner table is accessed via the index and the outer one is accessed sequentially

Order unary in-line operations so that only necessary attributes are funneled upstream

Stage 4, Notes on joins

If no indices exist and the tables are not sorted the way we want, build an index on one, then do a sort merge or a nested join

If both tables have indices, use the one for the shortest table

If the hit ratio of a join will be very small, do it as early as you can

Stage 5: consider multiple execution plans

Estimate the costs of various execution plans and pick the best one

Balance time spent optimizing versus time spent executing

For one time only transactions, pick a plan fast

Other important issues

Use indices that are fast with respect to pulling in blocks from the disk Wide, but shallow trees, like b-trees

An index on a sorted attribute can be interpolated

Hash indices good for key-based selections, but are not as versatile as b-trees

A note on b-trees

They are dynamic and therefore do not suffer from the migrating artificial primary key problems

Since they are dynamic, the tradeoff is that they are slightly less space efficient and often lead to an extra layer in the tree

Important: manipulating parse trees

There is a sort of algebra of parse tree manipulation

For example, A join B and B join A are equivalent

(A join B) join C = A join (B join C)

If a and b are used as join attributes, we can strip away all but those and the ones that will be chosen by a later projection or selection

More notes on optimization

The smallest parse tree might not be the best Sometimes we add unary operations to make the

operands of binary operations smaller If a unary operation will have a very small hit ratio, do it

early

The cost of a complete execution plan is complex to calculate Statistics of the data size and value distribution Choice of parse tree Choice of execution plan for each tree !! It is not a linear process

In light of modern database applications

Often the cost of optimization is not worth it Many one of a kind transactions Database with very small transaction load, perhaps do to

multiple servers and hard drives Tables are small and queries are simple, and so almost

any execution plan is good

This is a key reason by traditional relational database servers are often far less efficient than no-SQL DBs – because they were built to efficiently run a high volume of similar transactions that involve multiple tables

Making use of parallelism

SQL, compared to various non-declarative languages, is very east to parallelize

But communication costs can be very high if the servers are not co-located

Often, disk arrays are the chosen form of parallelism

Optimization in the real world today

Languages and queries are often not set-based

The code in a query can be very imperative

Binary operations might be in the minority

Databases can be so huge that optimization has to be heavily biased toward certain kinds of queries