March 7, 2002 1

Using Pattern Recognition Using Pattern Recognition Techniques to Derive a Techniques to Derive a Formal Analysis of Why Formal Analysis of Why

Heuristic Functions WorkHeuristic Functions Work

B. John OommenA Joint Work with Luis G. Rueda

School of Computer ScienceCarleton University

March 7, 2002 2

Optimization Problems

Any arbitrary optimization problem:

• Instances, drawn from a finite set, X,

• An Objective function

• Some feasibility functions

The aim:

• Find an (hopefully the unique) instance of X,

• which leads to a maximum (or minimum)

• subject to the feasibility constraints.

March 7, 2002 3

An Example

The Traveling Salesman Problem (TSP)• Consider the cities numbered from 1 to n,• The salesman starts from city 1,• visits every city once, and • Returns to city 1.

An instance of X is a permutation of cities:For example, 1 4 3 2 5, if five cities considered

The objective function:• The sum of the inter-city distances:• 1 4, 4 3, 3 2, 2 5, 5 1

March 7, 2002 4

Heuristic FunctionsA Heuristic algorithm is an algorithm

which attempts to find a certain

instance X

that maximizes the objective function

It iteratively invokes a Heuristic function.

The heuristic function estimates (or

measures) the cost of the solution.

The heuristic itself is a method that

performs one or more changes to the

current instance.

March 7, 2002 5

An Open Problem

Consider a Heuristic algorithm that invokes any of

Two Heuristic Functions : H1 and H2

used in estimating the solution to an

Optimization problem

If Estimation accuracy of H1 >

Estimation accuracy of H2

Does it imply that

H1 has higher probability of leading to the

optimal QEP?

March 7, 2002 6

Pattern Recogniton Modeling

Two heuristic functions : H1 and H2

Probability of choosing a cost value of a Solution:two independent random variables: X1 and X2

Distribution -- doubly exponential:

c

where ,

and

cX ][E

2

2][Var

X

||

2

1)( cxexf

X

March 7, 2002 7

Pattern Recogniton Modeling

Our model:

Error function is doubly exponential.

Typical in reliability analysis and failure models.

How reliable is a Solution when only estimate known?

Assumptions:Mean cost of Optimal Solution: , then

shift the origin by E[X] = 0Variances:

Estimate X1 better than Estimate of X2

][Var22

][Var 222

21

1 XX

March 7, 2002 8

Main Result (Exponential) H1 and H2, two heuristic functions.

X1 and X2, two r.v. optimal solution obtained by H1 and

H2

X1’ and X2

’, other two r.v. for sub-optimal solution

cXEXXEX ]'[]'[E][][E0 2121

Let p1 and p2 the prob. that H1 and H2 respectively make the wrong decision.

Shown that:

]'[Var][Var22

]'[Var][Var if 2222

21

11 XXXX

then : 21 pp

March 7, 2002 9

Proof (Graphical Sketch)For a particular x, the prob. that x leads to wrong decision by H1 is given by:

I e dux

u c11

1

2 11

( )

X1(opt) X1

(subopt)

X2(subopt)X2

(opt)

March 7, 2002 10

Proof (Cont’d)

or

X1(opt) X1

(subopt)

X2(subopt)X2

(opt)

I e du e duc

u c

c

x

u c1 2

1

2

1

211

11

( ) ( )

if x < c

March 7, 2002 11

Proof (Cont’d)The total probability that H1 makes the wrong decision for all values of x is:

p I e I e I ex

c

x

c

x1 11

0

1 110

1 1 2 1

1

21 1 1

Similarly, the prob. that H2 makes the wrong decision for all values of x is:

p I e I e I exc

x

c

x2 11

0

2 110

2 1 2 2

1

22 2 2

March 7, 2002 12

Proof (Cont’d)

Solving integrals and making p1 p2, we have:

02

1ln2

1ln),( 11111

kkkF

which, using ln x x - 1, implies that p1 p2 QED

where 1 = 1 c and 2 = 2 c

Also 2 substituted for k 1

March 7, 2002 13

Second Theorem

F(1,k) can also be written in terms of 1 and k as:

02

1

2

1),( 1111

111 eeekekG kk

Suppose that 1 0 and 0 k 1,then G(1,k) 0, and

there are two solutions for G(1,k) = 0

}1,1{ 1 k },0{ 1 kand

Proof:Taking partial derivatives and solving:

and 0),( 1

k

kG 0),(

1

1

kG

March 7, 2002 14

Graphical Analysis (Histograms)

),( 1 kG

1k

R-ACM / Eq-width R-ACM / Eq-depthT-ACM / Eq-widthT-ACM / Eq-depth

G >>> 0, orp1 <<< p2

R-ACM / T-ACM Eq-width / Eq-depthG 0, or p1 p2

Minimum in 1 = 0 and 0 k 1

March 7, 2002 15

Analysis : Normal Distn’s)(

1suboptX

)(

2

suboptX

)(1

optX

)(

2

optX

No integration possible for the normal pdf

Shown numerically that p1 p2

March 7, 2002 16

Plot of the Function G

March 7, 2002 17

Estimation for Histograms

is estimated as where N is the # of samples

N

i ix

N

1||

March 7, 2002 18

Similarities of R-ACM and d-Exp

Estimated for RACM

True d-Exp

March 7, 2002 19

Simulations Details

Simulations performed in Query Optimization:

4 independent runs per simulation.

100 random Databases per run 400 per simulation.

6 Relations,

6 Attributes per relation,

100 tuples per relation.

Four independent runs on 100 databases:

R-ACM vs. Traditional using:

11 bins, 50 values

March 7, 2002 20

Empirical Results

Run R > W W > R R > D D > R

1 26 12 35 12

2 24 15 42 15

3 35 11 46 8

4 29 15 46 8

Total 114 53 169 41

# of times in which R-ACM

yields better QEP

# of times in which Eq-width

yields better QEP

# of times in which Eq-depth

yields better QEP

March 7, 2002 21

Conclusions

Applied PR Techniques to solve problem of relating Heuristic Function Accuracy and Solution Optimality

Used a reasonable model of accuracy (doubly exponential distribution).

Shown analytically how the high accuracy of heuristic function leads to a superior solutions.

Numerically shown the results for normal distributions

Shown that R-ACM yield better QEPs in a larger number of times than Equi-width and Equi-depth.

Empirical results on randomly generated databases also shown the superiority of R-ACM.

Graphically demonstrated the validity of our model.