Algorithms and Architectures II Jens Myrup Pedersen (JMP...
Transcript of Algorithms and Architectures II Jens Myrup Pedersen (JMP...
Algorithms and Architectures II
Jens Myrup Pedersen (JMP), Rasmus Løvenstein Olsen (RLO)
Mm3: Algorithm examples: Genetic algorithms, advanced backtracking, min-max- February 25, 2008
Algorithms and Architectures II
1. Processor and Storage Architecture (JMP)2. Vehicular on-board networks and applications (RLO)3. Algorithm examples: Genetic algorithms, advanced backtracking (RLO)4. Complexity and complexity classes (RLO)5. Complexity and complexity classes 2 + algorithms for task scheduling (RLO)6. Advanced data structures 1 (JMP)7. Advanced data structures 2 (JMP)8. Graph algorithms 1 (JMP)9. Graph algorithms 2 (JMP)10. Guest Lecture by Rasmus H. Nielsen: Complex Problems in Network
Planning
NB: Slides originally prepared by Hans-Peter Schwefel, for D4-6: Lecture 2, Spring 2007
Program of today
• Introduction to algorithms• Binary trees • Backtracking • Spanning tree• Genetic algorithms• Assignments
Introduction to algorithms
• The word algorithms origins from a Persian mathmatician from 9th century: Abu Abdullah Muhammad ibn Musa al-Khwarizmi
• Algorithms has many uses, e.g. for light bulps
• Earliest known algorithm was doneby the Babylonians in 1600 BCfor factorisation and findingsquare roots
• Question: Today we have reallyfast computers, why do we needalgorithm and study of these?
• The Big O notation
Lamp does not work
Lamp pluggedIn?
Bulp burnedOut?
Buy new lamp
Plug in lamp
Replace bulp
g(x)=O(f(x))With a slight abuse of notation!
Program of today
• Introduction to algorithms• Binary trees • Backtracking • Spanning tree• Genetic algorithms• Assignments
Binary trees
Binary search tree property:Let x be a node in a binary search tree. If y is a node in the left subtree of x, then key[y] ≤ key[x]. If y is a node in the right subtree of x, then key[x] ≤ key[y]
5
3
2 5
7
8
23
7
58
4
Increasing key numbersDecreasing key numbers
Queries to binary trees
• Searching for a specific node (with key[y]) using a key x• If key x = key[y] then stop. Node has been identified• If key x < key[y] then search in the left branch• If key x > key[y] then search the right branch• If no more branches are left, then nothing has been found• Search time can be done in O(h) (h is height of tree)
• Maximum and minimum search• Minimum search: Follow the left branch until end • Maximum search: Follow the right branch until end • Search time can be done in O(h) (h is height of tree)
Operations on binary trees – Insert node
• Move down the tree comparing key x with key[y] until no more branches are available
• Inserted item is then a child of predecessor
12
5
2 9
18
15 19
1713
12
18
15
Operations on binary trees – Delete node
12
5
3 12
16
23
20
1810 13
6
7
Program of today
• Introduction to algorithms• Binary trees • Backtracking • Spanning tree• Genetic algorithms• Assignments
Backtracking – what is the problem?
• What if we do not have keys that allows us to do binary search?• A problem has been broken into a set of possible solution
and we need to find the best solution given some constraints
Examples: The Queens problem
• Consider a n by n chess board, and the problem of placing q queens on theboard without the queens threatening one another.
• The solution space is {1,2,3,…,n}n
• Trying all possible solutions implies nn cases, but realising that two queenscannot be on the same row/column, reduces this to n! cases!!
One queen threatensa lot of squares
Ø ØØØ Ø
Ø
Examples: Convex hull (Graham’s Scan)
• The problem asks to construct the shortest polygon which encloses a given set of points on a plan.
• Traverse the points in some order, adding them to the partial solution upon making a turn of less than 180 degrees, and backtracking whenmaking a larger turn.
• The algorithm requires O(n log n) time for sorting the points, and O(n) time to select an appropriate subset.
Intuitively
Start with extremepoint
Now, traverse throughPossible solutions
Backtracking
• A potential solution is represented by a set of vectors S={v1,v2,..,vn}
Is S a solution
For each v in S, do:
Is {v1,v2,..,vn, v}Acceptable? Try {v1,v2,..,vn, v}
Is S != Ø
Try(S)
Return S
Return S
Yes
Yes
Yes
No
No
No
Characteristics of the backtracking algorithm
• Backtracking can be used to traverse through a solution space• Identification of the problem is the firstmost important issue• Description and idenfitication of rules is necessary, but not necessarily
easy!• Can easily reduce the amount of iterations that needs to be done!!
Program of today
• Introduction to algorithms• Binary trees • Backtracking • Spanning tree• Genetic algorithms• Assignments
Graph definitions
• Graph G=(V,E)• V: set of vertices• E ⊆ VxV: Set of edges• Undirected graph: (v1,v2)∈E⇒(v2,v1)∈E
• Basic operations on graphs• On vertices: Add/Delete/Find• On edges: Add/Delete/Find
• Path := A sequence of edges• P(v1->vn)={(v1,v2), (v2,v3), (v3,v4),.., (vn-1,vn)}• Path length is n
• Application examples• Routing in computer networks• Representation and manipulation of (finite) automatas (parsers, compilers)• Representation of semantic relations• …
A word on greedy algorithm
• Greedy algorithms always picks the option that looks best at the given time
• They do not always lead to an optimal solution, but in many cases they do
The minimum spanning tree
• Problem: To connect a graph with minimum ”cost”
• Example applications: • Wiring PCB’s for electronic devices, • Network planning• Routing in ad hoc networks
∑∈
=Tvu
vuwTw),(
),()(
a
h
b
i
g
c
f
d
e
4
8
11
7
8
2
1
6
2
4
7
14
9
10
Growing a minimum spanning tree
• A is a subset of some minimum spanning tree, T• Generic algorithm for generating a minimum spanning tree
• Safe for A:• Problem is to find an edge that is safe for A
• Kruskal’s algorithm• Prim’s algorithm
1 A ← Ø2 while A does not form a spanning tree3 do find an edge (u,v) that is safe for A4 A ← A∪{(u,v)}5 return A
Kruskal’s algorithm
• ”Brute force” methodology to construct a minimum spanning tree• Idea is to build set of edges E’ in a greedy manner (’local’ decisions)• Algorithm
a
h
b
i
g
c
f
d
e
4
8
11
7
8
2
1
6
2
4
7
14
9
10
1 A ← Ø2 for each vertex v∈V[G]
do MAKE-SET(v)3 sort edges of E into increasing cost by w4 for each edge (u,v)∈E
do if FIND-SET(u)≠ FIND-SET(v)then A ← A∪{(u,v)}
UNION(u,v)5 return A
Prim’s algorithm
• ”Brute force” metode which grows the tree• All vertices not in the tree resides in a
min-priority queue, Q
• End goal, i.e. algorithm termination happenswhen
}}{:])[,{( QrVvvvA −−∈= π
a
h
b
i
g
c
f
d
e
4
8
11
7
2
1
6
2
4
7
14
9
10
8
1 for each u∈V[G]do key[u] ← ∞
π[u] ← NIL2 key[r]←03 Q←V[G]4 while Q≠Ø
do u←EXTRACT-MIN(Q)for each v∈Adj[u]
do if v∈Q and w(u,v)<key[v]then π[v]←u
key[v] ←w(u,v)}}{:])[,{( rVvvvA −∈= π
Properties of the two algorithms
• Running time for the algorithms• Kruskal’s algorithm O(E log V)• Prim’s algorithm (depending on how Q is implemented)
• Adjacency matrix: O(V2)• Binary heap: O(E log V)• Fibonacci heap: O(E+V log V)
• Prim’s algorithm contains at all time a tree, while Kruskal’s algorithm doesnot
Program of today
• Introduction to algorithms• Binary trees • Backtracking • Spanning tree• Genetic algorithms• Assignments
Genetic Algorithms
• Charles Darwin published in 1859The Origin of Species by Means of Natural Selection
• Survival of the fittest! • Stronger genes survives the battle of life• Weaker genes extinct
• It is the environment that dictates what is fit andwhat is not fit
• Probabilistic search algorithm
The travelling rock band (aka the travelling salesman) problem
The travelling rock band
• Constraints to be considered by the rock band• Minimize the time spend in the air/road• Minimize the cost of the tour in money• Well, the band is not very good so:
- They may not return to a city once visited!
• Example: three randomly picked tours• $17486.01 : Madrid, Vienna, Moscow, Berlin, Brussels, Munich, Milan,
Barcelona, London, Hamburg, Warsaw, Dublin, Kiev, Paris, Rome• $20198.92 : London, Rome, Brussels, Kiev, Hamburg, Warsaw, Barcelona,
Paris, Munich, Dublin, Vienna, Moscow, Madrid, Milan, Berlin • $17799.34 : Madrid, Milan, Kiev, Vienna, Warsaw, London, Barcelona,
Hamburg, Paris, Munich, Dublin, Berlin, Moscow, Rome, Brussels• The question: Are any of these the best trips (in relation to the requirements)?
Darwin put inside a computer
Surviving population ”=” Population ”+” EnvironmentNew population ”=” Surviving population ”+” new generation
Newpopulation
Selection/Environment press
Reproduction& Mutation
Start
Stop
A solution has been found
Describing the ”beast”
• Problem: How do we describe our creatures in our program?• Encoding of chromosomes
• Example of describing the tour of the rock band• Binary encoding for finding the right direction• The trip of the rock band could be
{London, Dublin, Paris, Brussels, Hamburg, Berlin, Warsaw, ….}
• is encoded as {1,0,2,3,8,4,5,...}
• or as binary{0001,0000,0010,0011,1000,0100,0101,...} ………
Brussels311
Paris210
London101
Dublin000
MeaningDecodedCode
Direction codes
Defining the fitness level
• The fitter a ”beast” is, the more likely it is to survive• Fitness level is individual to the problem needed solved• Example: The rock band tour
• Parameter is time of travelling• Definition of a matrix describing travel
time between cities• Parameter is price in e.g. Euros
• Definition of a matrix describing travelcost between cities
• A cost function could be defined as
• The fitness level could be defined as
00.5h2.5hWarsaw
0.5h01.5hBerlin
2.5h1.5h0London
WarsawBerlinLondon
0100750Warsaw
1000500Berlin
7505000London
WarsawBerlinLondon
i1i
i timeprice ⋅+⋅= ∑=
t
N
p wwC
CF 1
=
Setting up a population of ”beasts”
• Initiating the algorithm by defining N numbers of ”beasts” (genes)• Gene values can be based on
• Random selection among valid chromosomes• Qualified initial guesses• A combination of both random and qualified guesses
• This is our initial population
Newpopulation
Selection/Environment press
Reproduction
Start
Stop
Who will survive our environment
• First thing to do is to calculate the fitness of each ”beast”• If there exists solutions such that our goal has been reached (more or
less), then stop• Elitism is a concept that
• Ensures that the best ”beasts” will always pass on to the reproduction• Eliminates the worst of the worst ”beasts”• However, be careful with elitism:
• The population may easily becometoo homogene to make furtherprogress
Newpopulation
Selection/Environment press
Reproduction
Start
Stop
Reproduction is a stochastic process
• Who should pair with whom?• It is basically a game of roullette
Newpopulation
Selection/Environment press
Reproduction &mutation
Start
Stop
1
2
3
4
5
6
7
8
9
Drrrrr rrrr rrrTik tik tik tik tik
• Surface area is determined by thefitness of individual ”beasts”compared to the total sum• -> higher fitness value, the more likely it will be reproduced
Reproduction in a nutshell
• Once two pairs of ”beasts” have been found, they make childrens• Example of how a son/daughter ”beast” is created
• Other cross-over aproaches can also be applied, e.g.• Partially mapped cross-over• Alternating position cross-over• Maximal preservation cross-over• And many other types
Beast A{10,11,10,01,01,01}
Beast B{11,11,10,01,10,11}+
{01,10,11,10,11,10}Beast C (new)
Mutant children• Mutation is used to ensure diversity in the population• Randomly altering a gene, example:
• Similar, mutation can be applied in dedicated ways to produce valid ”beasts”
• Invalid mutations should be removed or as minimum handled in some way• Mutation rate, m, determines how often/how likely a mutation occurs
Beast A{10,11,10,01,01,01}
Beast B{11,11,10,01,10,11}+
{01,10,11,10,11,10}Beast C (new)
{01,11,11,10,11,00}Beast C (new and mutated)
Challenges, traps and disadvantages with Genetic Algorithms
• Genetic algortihms are useful if• A given problem can be specified in terms of a set of parameters and a
cost/fitness value• If there are too many parameters to fully explore the parameter space
within reasonable time
• However,• GA’s may not apply to all types of problems• It may be difficult to define the best way of describing the ”beasts”• It may be difficult to define the fitness functions• It is not guaranteed that the optimal solution for a given problem is
found
Program of today
• Introduction to algorithms• Binary trees • Backtracking • Spanning tree• Genetic algorithms• Assignments
Assignments
• Project Discussions (15min): Discuss the potential application of graphs in your project. • What type of graphs (directed/un-directed) and what type of edge-
marking may be useful? • What tasks/algorithms would you need for your project to work with
those graphs?
• How many queens can safely be put on a• 4x4 square chess board?• 8x8 square chess board?• 16x16 square chess board?• How is the results affected if you can put other types of pieces on the
chess board, e.g. one or two ponds?
Assignments
• Genetic algorithms: Design and implement an algorithm that is capable of doing simple arithmetics using n numbers• Use the operators +,-,/,*• Use the numbers 0,1,2,3..,8,9• Example: ((3*7+1+0+1+3)/2)+9=15• Experiment with e.g. and see how these affects the algorithm
• different genome types and length, • mutation rate, • selection process for reproduction, • amount of genomes in pool
Assignments
• Assume you are running a transport enterprise in Norway with n cities in your operational scope. The cities are connected by (one-way) streets, but due to the many tunnels there are different height limitations for vehicles. The height limitations are given by a positive cost function on the directed graph G=(V,E). We are interested to find for each pair (vi,vj) of cities the maximum height of the vehicle so that it can reach city vj from vi (possibly also on longer paths).
Assume that the graph including cost function is represented by a matrix A=(aij) as follows: aij=-1 if (vi,vj) ∉E, aij=c(vi,vj) otherwise
References
• GA: Online toturial at http://fog.neopages.org/helloworldgeneticalgorithms.php
• http://ai4r.rubyforge.org/geneticAlgorithms.html#chromosome-impl• Backtracking problems (online): http://www.cse.ohio-
state.edu/~gurari/course/cis680/cis680Ch19.html#QQ1-51-128• Introduction to Algorithms by T.H.Cormen, C.E.Leiserson, R.L.Rivest and
C.Stein• Chapter 12, p. 253-255• Chapter 23, p. 561-579