Advance analysis of algo

1

Dept of CS & ITUniversity of Sargodha

Advance Analysis of Algorithms

Fall 2011

Fahad MaqboolMS Computer Science

BS Software Engineering

2

Adv Analysis of Algo

Steps involved in writing a computer program:

• Problem formulation & specification

• Design of solution

• Implementation

• Testing

• Documentation

• Evaluation of the solution

3

Problem formulation & specification

• Half the battle is knowing what problem to solve.

• Most problems have no simple precise specification.

• Some problems are impossible to formulate.

• Example: Gourmet recipe, World peace, Etc.


4

How to formulate & specify a problem?

• Identify the problem parameters.

• Expressing the problem by a formal model.

• Looking for a solution for the model.

• In the absence of a solution, discover about the model.


5

How to model and solve a problem?

• Using any branch of Mathematics or Science.

• Identifying problems of numerical nature.

• With a suitable model, attempt to find a solution in terms of that model

• Simultaneous linear equations for electrical circuits.

• Differential equations for predicting chemical reaction rates.


6

Algorithm

• Finite sequence of instructions.

• Each instruction having a clear meaning.

• Each instruction requiring finite amount of effort.

• Each instruction requiring finite time to complete.


7

Algorithm

• Finite sequence of instructions. An input should not take the program in an infinite loop

• Each instruction having a clear meaning. Very subjective. What is clear to me, may not be clear to you.

• Each instruction requiring finite amount of effort. Very subjective. Finite on a super computer or a 486?

• Each instruction requiring finite time to complete. Very subjective. 1 min, 1 hr, 1 year or a lifetime?


8

Using a model to solve a complicated traffic light problem

GIVEN: A complex intersection.

OBJECTIVE: Traffic light with minimum phases.

SOLUTION:• Identify permitted turns, going straight is a “turn”.

• Make group of permitted turns.

• Make the smallest possible number of groups.

• Assign each phase of the traffic light to a group.


9


AB

C

D

E

An intersection

Rawalpindi


Roads C and E are one way, others are two way.

10


There are 13 permitted turns.

Some turns such as AB (from A to B) and EC can be carried out simultaneously.

Other like AD and EB cross each other and can not be carried out simultaneously.

The traffic light should permit AB and EC simultaneously, but should not allow AD and EB.


11


AB

C

D

E

An intersection

AB & AC

AD & EB


12


SOLUTION:• We model the problem using a structure called graph G(V,E).

• A graph consists of a set of points called vertices V, and lines connecting the points, called edges E.

•Drawing a graph such that the vertices represent turns.

• Edges between those turns that can NOT be performed simultaneously.


13

AD


AB

C

D

E

An intersection

BD

AB AC

BA BC

DA DB DC

EA EC ED

Partial graph of incompatible turns


EB

14

Partial table of incompatible turns



15


SOLUTION: The graph can aid in solving our problem.

A coloring of a graph is an assignment of a color to each vertex of the graph, so that no two vertices connected by an edge have the same color.

Our problem is of coloring the graph of incompatible turns using as few colors as possible.


16


More on SOLUTION (Graph coloring):

Graph coloring problem has been studied for decades.

The theory of algorithms tells us a lot about it.

Unfortunately this belongs to a class of problems called as NP-Complete problems.

For such problems, all known solutions are basically “try all possibilities”

In case of coloring, try all assignments of colors.


17


Approaches to attempting NP-Complete problems:

1. If the problem is small, might attempt to find an optimal solution exhaustively.

2. Look for additional information about the problem.

3. Change the problem a little, and look for a good, but not necessarily optimal solution.

An algorithm that quickly produces good but not necessarily optimal solutions is called a heuristic.


18

Using model to solve complicated traffic light problem

A reasonable heuristic for graph coloring is the greedy “algorithm”.

Try to color as many vertices as possible with the first color, and then as many uncolored vertices with the second color, and so on.

The approach would be:

1. Select some uncolored vertex, and color with new color.

2. Scan the list of uncolored vertices.

3. For each uncolored vertex, determine whether it has an edge to any vertex already colored with the new color.

4. If there is no such edge, color the present vertices with the new color.

19

This is called “greedy”, because it colors a vertex, whenever it can, without considering potential drawbacks.

1 5

3

4

21 2

3

4

5

1 5

3

4

21

3

4

5 2

Using model to solve complicated traffic light problem

Three colors used

Two colors used(optimum)

20


Work SMARTnot

HARD

Analysis of Algo

Questions that will be answered

21/31

• What is a “good” or "efficient" program?• How to measure the efficiency of a program?• How to analyse a simple program?• How to compare different programs?• What is the big-O notation?• What is the impact of input on program performance?• What are the standard program analysis techniques?• Do we need fast machines or fast algorithms?

Which is better?

22/31

The running time of a program.

Program easy to understand? Program easy to code and debug? Program making efficient use of resources? Program running as fast as possible?

Measuring Efficiency

23/31

Ways of measuring efficiency:

• Run the program and see how long it takes• Run the program and see how much memory it uses

•Lots of variables to control:

• What is the input data?• What is the hardware platform?• What is the programming language/compiler?• Just because one program is faster than another right now, means it will always be faster?

Measuring Efficiency

24/31

Want to achieve platform-independence

• Use an abstract machine that uses steps of time and units of memory, instead of seconds or bytes

• - each elementary operation takes 1 step

• - each elementary instance occupies 1 unit of memory

A Simple Example

25/31

// Input: int A[N], array of N integers// Output: Sum of all numbers in array A

int Sum(int A[], int N) { int s=0; for (int i=0; i< N; i++) s = s + A[i]; return s;}

How should we analyse this?

Analysis of Sum

26/31

1.) Describe the size of the input in terms of one ore more parameters: - Input to Sum is an array of N ints, so size is N.

2.) Then, count how many steps are used for an input of that size: - A step is an elementary operation such as +, <, =, A[i]

A Simple Example

Analysis of Sum (2)

27/31

// Input: int A[N], array of N integers// Output: Sum of all numbers in array A

int Sum(int A[], int N { int s=0;

for (int i=0; i< N; i++)

s = s + A[i];

return s;}

1

2 3 4

56 7

8

1,2,8: Once3,4,5,6,7: Once per each iteration of for loop, N iterationTotal: 5N + 3The complexity function of the algorithm is : f(N) = 5N +3

A Simple Example

How 5N+3 Grows

28/31

Estimated running time for different values of N:

N = 10 => 53 stepsN = 100 => 503 stepsN = 1,000 => 5003 stepsN = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear proportion toN for this Sum function.

Analysis: A Simple Example

What Dominates?

29/31

What about the 5 in 5N+3? What about the +3?• As N gets large, the +3 becomes insignificant•5 is inaccurate, as different operations require varying amounts of time

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large, concentrate on thehighest order term:• Drop lower order terms such as +3• Drop the constant coefficient of the highest order term i.e. N


Asymptotic Complexity

30/31

• The 5N+3 time bound is said to "grow asymptotically" like N

• This gives us an approximation of the complexity of the algorithm

• Ignores lots of (machine dependent) details, concentrate on the bigger picture


Comparing Functions

31/31

Definition: If f(N) and g(N) are two complexity functions, we say

f(N) = O(g(N))

(read "f(N) as order g(N)", or "f(N) is big-O of g(N)")if there are constants c and N0 such that for N N0, T(N) cN

f(N) £ c g(N)for all sufficiently large N.

Comparing Functions

0

50000

100000

150000

200000

250000

100n2 10 40 90 16 25 36 49 64 81 10 12 14 16 19 22 25 28 32 36 40 44 48 52 57 62 67 72 78 84 90 961E 1E1E

5n3 5 40 13 32 62 10 17 25 36 50 66 86 10 13 16 20 24 29 34 40 46 53 60 69 78 87 98 1E1E1E1E2E2E2E

1 2 3 4 5 6 7 8 910111213141516171819202122232425262728293031323334

32/31

100n2 Vs 5n3, which one is better?

Comparing Functions

Differenec of functions

-90000-80000-70000-60000-50000-40000-30000-20000-10000

01000020000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

33/31

100n2 Vs 5n3, which one is better?

Why is this useful?

34/31

As inputs get larger, any algorithm of a smaller order willbe more efficient than an algorithm of a larger order

Tim

e (s

teps

)

Input (size)

3N = O(N)

0.05 N2 = O(N2)

N = 60

Comparing Functions

35/31

f1(n)

f2(n)

f3(n)

f4(n)

F(n)

Input size (n)

Comparing FunctionsWhat is the relationship between the following?

What is the polynomial representation for the following?

Big-O Notation

36/31

• Think of f(N) = O(g(N)) as " f(N) grows at most like g(N)" or " f grows no faster than g" (ignoring constant factors, and for large N)

Important:• Big-O is not a function!• Never read = as "equals"• Examples: 5N + 3 = O(N) 37N5 + 7N2 - 2N + 1 = O(N5)

Big-O Notation

0

50000

100000

150000

200000

250000

300000

350000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

37/31

100n2

100n2 + 5n3

5n3

5n4

Common Orders of Growth

38/31

Let N be the input size, and b and k be constants

O(k) = O(1) Constant TimeO(logbN) = O(log N) Logarithmic Time

O(N) Linear TimeO(N log N)O(N2) Quadratic Time

O(N3) Cubic Time...O(kN) Exponential Time

Increasing Complexity

Size does matter

Size does matter

39/31

What happens if we double the input size N?

N log2N 5N N log2N N2 2N

8 3 40 24 64 256 16 4 80 64 256 65536 32 5 160 160 1024 ~109

64 6 320 384 4096 ~1019

128 7 640 896 16384 ~1038

256 8 1280 2048 65536 ~1076

If one execution takes 10-6 seconds, 2256 will take about 1061 centuries

Big Numbers

40/31

Suppose a program has run time O(n!) and the run time for n = 10 is 1 second

For n = 12, the run time is 2 minutesFor n = 14, the run time is 6 hoursFor n = 16, the run time is 2 monthsFor n = 18, the run time is 50 yearsFor n = 20, the run time is 200 centuries

Size does matter

Standard Analysis Techniques

41/31

Simplest case: O(1) time statements• Assignment statements of simple data types int x = y;• Arithmetic operations: x = 5 * y + 4 - z;• Array referencing: A[j] = 5;• Array assignment: j, A[j] = 5;• Most conditional tests: if (x < 12) ...

Constant time statements

Analyzing Loops

42/31

Any loop has two parts: 1. How many iterations are performed? 2. How many steps per iteration? int sum = 0,j; for (j=0; j < N; j++) sum = sum +j;

- Loop executes N times (0..N-1) - 4 = O(1) steps per iteration - Total time is N * O(1) = O(N*1) = O(N)


Analyzing Loops (2)

43/31

What about this for-loop?

int sum =0, j; for (j=0; j < 100; j++) sum = sum +j;

- Loop executes 100 times

- 4 = O(1) steps per iteration

- Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)


Analyzing Loops (3)

44/31

What about while-loops?Determine how many times the loop will be executed: bool done = false; int result = 1, n; scanf("%d", &n); while (!done){ result = result *n; n--; if (n <= 1) done = true; } Loop terminates when done == true, which happens after N iterations. Total time: O(N)


Nested Loops

45/31

Treat just like a single loop and evaluate each level of nesting as needed:

int j,k; for (j=0; j<N; j++) for (k=N; k>0; k--) sum += k+j;

Start with outer loop: - How many iterations? N - How much time per iteration? Need to evaluate inner loopInner loop uses O(N) timeTotal time is N * O(N) = O(N*N) = O(N2)


PRODUCT RULE

Nested Loops (2)

46/31

What if the number of iterations of one loop depends on thecounter of the other?

int j,k; for (j=0; j < N; j++) for (k=0; k < j; k++) sum += k+j;

Analyze inner and outer loop together:- Number of iterations of the inner loop is: 0 + 1 + 2 + ... + (N-1) = O(N2) How?


Digression

47/31

When doing Big-O analysis, we sometimes have to computea series like: 1 + 2 + 3 + ... + (N-1) + NWhat is the complexity of this?

Remember Gauss:


Sequence of Statements

48/31

For a sequence of statements, compute their complexity functionsindividually and add them up

for (j=0; j < N; j++) for (k =0; k < j; k++) sum = sum + j*k; for (l=0; l < N; l++) sum = sum -l; printf("sum is now %f", sum);

O(N2)

O(N)

O(1)

Total cost is O(N2) + O(N) +O(1) = O(N2)

SUM RULE


Conditional Statements

49/31

What about conditional statements such as

if (condition) statement1; else statement2;where statement1 runs in O(N) time and statement2 runs in O(N2) time?

We use "worst case" complexity: among all inputs of size N, what is the maximum running time?

The analysis for the example above is O(N2)


Fast machine Vs Fast Algorithm

50/31

Get a 10 times fast computer, that can do a job in 103 seconds for which the older machine took 104 seconds .

Comparing the performance of algorithms with time complexities T(n)s of n, n2 and 2n (technically not an algorithm) for different problems on both the machines.

Question: Is it worth buying a 10 times fast machine?

Advance analysis of algo

Documents

Transcript of Advance analysis of algo