Post on 20-Nov-2014
description
Summations, Probability, and Randomized Algorithms
Advanced Data Structures & Algorithm Design
Introduction
Often the result of analyzing an algorithm is a summation Loops directly translate to summations, and
recursion can often be reduced to a summation Section 3.1 discusses a number of useful
summation formulas and properties; refer to it as needed throughout the semester
Bounding Summations
Being able to determine the upper bound on a summation is important if we want to actually use the summation in our analysis
We will investigate four methods of bounding summations: Using induction Bounding the terms Splitting the summations Approximation by integrals
Bounding Summations Using Induction We’ve already discussed induction as a method for
bounding recurrences and for solving summations Induction can also be used to show the bound on a
summation, rather than the exact value of the summation As an example, we will show the bound for the following
equation:
nn
k
k O 330
Bounding Summations Using Induction
3/2cor 1,1/c)(1/3 as long as3
3131
33
333
:1for holdsit that provemust and ,for holds bound theassume We
.1 allfor 113 ,0For
1
1
1
0
11
0
0
0
n
n
nn
n
k
nkn
k
k
kk
c
cc
c
nn
ccn
Bounding Summations by Bounding the Terms Sometimes a series can be bound by
bounding the individual terms in the series We can quickly bound a series using the
largest term of the series, then derive a series bound from it:
2
11
n
nkn
k
n
k
Bounding Summations by Bounding the Terms
n
kk
kn
k k
naa
aaa
1max
max1: maximum thelet weif , series afor general,In
This technique is a weak method for bounding a summation if the series can instead be bound by a geometric series.
Bounding Summations by Bounding the Terms Suppose we have a series such that ak+1/ak
<= r for some constant r<1 and all k>=0 In other words, the ratio of consecutive elements
in the series is less than a constant value If this property holds for all k, then any
element in the series ak <= a0rk In this case, we can bound the series using
an infinite decreasing geometric series:
Bounding Summations by Bounding the Terms
ra
ra
raa
k
k
k
kn
kk
11
0
00
00
0
Bounding Summations by Bounding the Terms
13/21
131
32
31
3
:boundour create to of value thisuse nowcan We32
131
3/3/)1(
:is termseconsecutiv of ratio theand 1/3,just is first term The3
summation thebound want to wesuppose example,For
11
1
1
k
k
kk
k
k
kk
kr
kk
kkr
k
Bounding Summations by Splitting Summations Difficult summations can sometimes be “split
apart” into pieces that are easier to solve individually
In these situations, the summation range is split and the summation expressed as the sum of each partition
This technique can be used to ignore a small number of initial terms, when each term in the summation is independent of n
Bounding Summations Using Approximation by Integrals Approximating the summation through the
use of integration provides a convenient means of obtaining a bound
This technique can be used when the summation can be expressed as the sum of some f(k), where f(k) is monotonically increasing or decreasing In other words, any x > y implies f(x) > f(y)
(monotonically increasing) or f(x) < f(y) (monotonically decreasing)
Bounding Summations Using Approximation by Integrals If f(k) is monotonically increasing, it can be
approximated by the integrals:
n
m
n
mk
n
mdxxfkfdxxf
1
1)()()(
Similarly, if f(k) is monotonically decreasing, it can be approximated by the integrals:
1
1)()()(
n
m
n
mk
n
mdxxfkfdxxf
Bounding Summations Using Approximation by Integrals
nxdx
k
nxdx
k
nn
k
nn
k
n
k
2
12
2
1
11
1
log
1:similar is boundupper The
)1(log
1:obtain webound,lower aFor
:k1function harmonic heconsider t example,an As
Counting Theory
Attempts to answer the question of “how many” without enumerating all the possibilities E.g., how many permutations of the string
“discovery” are there? One way to find out is to write ‘em all down Counting theory lets us calculate the answer without
having to
Rules of sum and product
Given a set of items, we can sometimes count the items using one of these rules: Rule of sum: the number of ways to choose an
element from one of two disjoint sets is the sum of the size of the sets
Rule of product: the number of ways to choose an ordered pair is the number of ways to choose the first element times the number of ways to choose the second element
Strings
String = a sequence of elements from the same set If the string is of length k, we sometimes call it a k-
string Given a string s, a substring s’ is an ordered
sequence of consecutive elements of s A k-substring is therefore a substring of length k
Given a set S of size n, how many k-strings are in the set?
How many k-strings are in a set? If we have n elements, and can pick any
element for any position in the string, then there’s n choices for the first, n for the second, etc. The rule of product applies over the entire string
The answer is therefore nk
Permutations
A permutation is an ordered sequence of all the elements of set S, with each element appearing once For example, if S = { c, a, t }, there are 6
permutations of S: cat, cta, act, atc, tca, tac
For the entire set S consisting of n elements, there are n! permutations
K-Permutations
A k-permutation consists of k elements from S, with no element appearing more than once E.g., if S = { a, b, c, d }, then there are 12 2-
permutations: ab, ac, ad, ba, bc, bd, ca, cb, cd, da, db, dc
If S has n elements, then the number of k-permutations in S is (n!)/(n-k)!
Combinations
A k-combination of an n-set S is simply a k-subset of S Combinations must be distinct, but elements in
the combination are unordered (unlike permutations)
For every k-combination, there are k! permutations Each permutation is a distinct k-permutation of S The number of k-combinations is therefore just the
number of k-permutations divided by k!:n!/k!(n-k)!
Probability
Probability is defined in terms of a sample space S, a set whose elements are called elementary events Each elementary event is a possible outcome of
an experiment E.g., flipping two coins can result in one of 4 elementary
events, which makes up the sample space: S = { HH, HT, TH, TT }
Probability
An event is a subset of the sample space S E.g., the event of obtaining one head and one tail
is the subset { HT, TH } The event S is called the certain event The event {} is called the null event Two events A and B are mutually exclusive if
they cannot occur simultaneously I.e., AB={}
Probability
The probability of an event A is written Pr{A} A probability distribution is a way to map the
events of S to real numbers, such that these axioms are met: Pr{A} >= 0 for any event A Pr{S} = 1 Pr{AB} = Pr{A} + Pr{B} for any two mutually
exclusive events
Discrete Probability Distributions A distribution is discrete
if it is defined over a finite or countably infinite sample space
A uniform distribution is a distribution such that all events are equally likely I.e., picking an element
at random
As
sA }Pr{}Pr{
Ss /1}Pr{
Continuous Uniform Probability Distributions A probability distribution in which all subsets
of the sample space are not considered to be events
They are defined over a closed interval [a, b], with each point in the interval being equally likely Since the number of points are uncountable, we
cannot satisfy axioms 2 & 3 – the probability of each “point” is effectively 0
Continuous Uniform Probability Distributions Given a closed range [a, b], and any closed
interval on that range [c, d] such that a <= c <= d <= b, the continuous uniform probability distribution defines the probability of the event [c, d] to be
abcddc
]},Pr{[
Discrete Random Variables
A discrete random variable X is a function from a finite or countably infinite sample space S to the real numbers It associates a real number with each possible
outcome of an experiment This lets us work on the probability distribution
X is random in the sense that its value depends on the outcome of some experiment, and cannot be predicted with certainty before the experiment is run
Discrete Random Variables
To use discrete random variables, we must define a probability density function: This is the probability that X is some particular
value or event It is simply the sum of the probabilities of all the
individual events represented by the random variable
xsXSs
sxX)(:
}Pr{}Pr{
Discrete Random Variables
Let’s look at rolling 2 6-sided dice: X is a random variable defining the maximum of
the two values shown on the dice There are 36 possible elementary events (2 dice,
each has 6 faces) If we define X=3, meaning the highest value on
either die is three, then Pr{X=3} = 5/36 (five possible outcomes out of 36 total)
Expected Value of a Random Variable This is the “average” of the values it takes on
Expected value defines the center of the distribution of the variable, i.e., if we were to run the experiment an infinite number of times, the expected value is the mean value of X over all experiments
x
xXxXE }Pr{][
Variance and Standard Deviation Variance: Var[X] = E[X2] – E2[X]
This is a measure of how much the distribution varies
Standard Deviation: sqrt(Var[X])
QuickSortvoid QuickSort(ArrayType &A, int begin, int end){ if ( begin < end ) { int q = Partition(A, begin, end); QuickSort(A, begin, q-1); QuickSort(A, q+1, end); }} This is a great algorithm if all inputs are equally likely
That’s not always the case! We can overcome the problem of worst case input by
introducing some randomness into the algorithm
QuickSort
When does the worst case occur in QuickSort? Why? It occurs because partition algorithm does not
partition the array evenly One way to improve it might be to partition around
the middle element – but even so, there is still a worst case such that the array is anti-optimally partitioned
Randomized Algorithms
An algorithm is randomized if its behavior is determined not only by the input, but also by the output of a random number generator
Let’s assume a random number generator with a function Random(a, b), that returns a random number in the range a-b
We will use a random number generator to impose a distribution such that no particular input elicits its worst case behavior
Randomized Algorithms
A randomized strategy is useful when there are many ways an algorithm can proceed, but no good way to know which way is “good”; if many alternatives are good, you pick one randomly
The benefits of good choices must outweigh the cost of bad choices
Randomized QuickSort
How do we randomize QuickSort? Create a RandomizedPartition function, then use
that in our main QuickSort function:
void RandomizedQuickSort(ArrayType &A, int begin, int end){ if ( begin < end ) { int q = RandomizedPartition(A, begin, end); RandomizedQuickSort(ArrayType(A, begin, q-1); RandomizedQuickSort(ArrayType(A, q+1, end); }}
Randomized QuickSort
int RandomizedPartition(ArrayType &A, int begin, int end)
{ int i = Random(begin, end); ArrayType temp = A[end]; A[end] = A[i]; A[i] = temp; return Partition(A, begin, end);}
Analysis of Randomized QuickSort How does this change our previous analysis?
We have added a constant factor to the Partition running time, which can be ignored
However, we have made worst-case behavior nearly impossible – no particular input can create it, only an unlucky partitioning
So the analysis doesn’t actually change, however we have made the average or expected case much more likely, and the worst case much less likely
Random Numbers
Adding randomness to an algorithm implies an ability to generate random numbers
Computers are unfortunately not directly capable of generating truly random sequences
The approach that is generally taken is therefore to generate a sequence of “pseudo-random” numbers that exhibits good random behavior
Ref: Numerical Recipes in C, Chapter 7, available online at
http://www.ulib.org/webRoot/Books/Numerical_Recipes/
Random Numbers
Most languages have a set of library functions for generating pseudo-random numbers System supplied generators typically suffer from a
number of problems due to poor specification and implementation The sequence generally repeats with a period no greater
than 32767 The randomness of the sequence is highly dependent
on the implementor’s choice of constants used by the algorithm, and in many standard implementations the choice is poor
Random Numbers
One fast method of choosing random numbers is the linear congruential method Each number in the sequence is determined by a
mathematical operation performed on the previous choice
Ij+1 = aIj + c % m m is the modulus, and determines the periodicity
of the generated sequence a is called the multiplier c is called the increment
Random Numbers
The quality of the generator here is highly dependent on the choice of m, a, and c Poor choices will limit the period, and more
importantly, significantly impact the randomness of the sequence
We can eliminate the need for m by using 32-bit integers and choosing m = 232
The result will be 64 bits, but since we’re using 32 bit variables, the hi order bits will be truncated
Some good choices for a and c are: a = 1664525, c = 1013904223
A Random Number Generator Implementationclass Random{public: static const unsigned long RANDMAX, MULTIPLIER, INCREMENT;
explicit Random(unsigned long seed = 0) :m_seed(seed) {}
unsigned long operator()(void) { return (m_seed = MULTIPLIER*m_seed+INCREMENT); }
unsigned long seed(void) const { return m_seed; } void seed(unsigned long value) { m_seed = value; }
private: unsigned long m_seed;};
Using the Random Number Generator This generator will produce a number between 0 and
232-1 Note that prior to generating numbers, the generator
should be seeded with a non-zero value If repeatability isn’t required, time() is often used to
generate the seed To produce a number in an arbitrary range, use the
following: j = LO+(int) ((float)(HI)*rand() / (RAND_MAX+1.0)); This forces the use of the hi order bits, which are
much more random than the lo order bits