Analysis of Insertion and Merge Sort

8/8/2019 Analysis of Insertion and Merge Sort

1/21

Sorting Algorithms

In computer scienceand mathematics, a sorting algorithm is analgorithm that putselements of a listin a certainorder.

The most-used orders are numerical order and lexicographical order.In-Place Sorting Algorithms

In computer science, an in-place algorithm is analgorithm which transforms a data

structure using a small, constant amount of extra storage space. The input is usually overwritten by the output as the algorithm executes. An algorithm which is not in-place is sometimes called not-in-place orout-of-place. A sorting algorithm is said to be in-place if it requires very little additional space besidesthe initial array holding the elements that are to be sorted.

Normally very little is taken to meanthat for sorting elements, O(log n) extra spaceis required.

There are a number ofsorting algorithms that can rearrange arraysinto sorted order in-place, including:

Bubble sort

Selection sort

Insertion sort

Heapsort

Quicksort and Mergesort are not in-place sorting algorithms.Stable Sorting Algorithms

Stable sorting algorithms maintain the relative order ofrecords with equal keys (i.e.,sort key values).

That is, a sorting algorithm is stable if whenever there are two recordsR and Swith thesame key and withR appearing before Sin the original list,R will appear before Sin the

sorted list.

Assume that the following pairs of numbers are to be sorted by their first component:

(4, 1) (3, 7) (3, 1) (5, 6) In this case, two different results are possible, one which maintains the relative order ofrecords with equal keys, and one which does not:

(3, 7) (3, 1) (4, 1) (5, 6) (order maintained)

(3, 1) (3, 7) (4, 1) (5, 6) (order changed)

Sorting Algorithms Stable

Bubble Sort Yes

Selection Sort No

Insertion Sort YesHeapsort No

Quicksort NoMergesort No

Internal Sort The data to be sorted is all stored in the computers main memory.

External Sort

Some of the data to be sorted might be stored in some external, slower, device.

In-Memory Sorting : When the size of memory is bigger than that of file to be sorted!
http://en.wikipedia.org/wiki/Computer_sciencehttp://en.wikipedia.org/wiki/Computer_sciencehttp://en.wikipedia.org/wiki/Mathematicshttp://en.wikipedia.org/wiki/Algorithmhttp://en.wikipedia.org/wiki/Algorithmhttp://en.wikipedia.org/wiki/List_(computing)http://en.wikipedia.org/wiki/List_(computing)http://en.wikipedia.org/wiki/Total_orderhttp://en.wikipedia.org/wiki/Total_orderhttp://en.wikipedia.org/wiki/Total_orderhttp://en.wikipedia.org/wiki/Lexicographical_orderhttp://en.wikipedia.org/wiki/Computer_sciencehttp://en.wikipedia.org/wiki/Algorithmhttp://en.wikipedia.org/wiki/Algorithmhttp://en.wikipedia.org/wiki/Data_structurehttp://en.wikipedia.org/wiki/Data_structurehttp://planetmath.org/encyclopedia/MatiyasevivcsTheorem.htmlhttp://planetmath.org/encyclopedia/Mean3.htmlhttp://planetmath.org/encyclopedia/Mean3.htmlhttp://en.wikipedia.org/wiki/Sorting_algorithmhttp://en.wikipedia.org/wiki/Arrayhttp://en.wikipedia.org/wiki/Arrayhttp://en.wikipedia.org/wiki/Bubble_sorthttp://en.wikipedia.org/wiki/Selection_sorthttp://en.wikipedia.org/wiki/Insertion_sorthttp://en.wikipedia.org/wiki/Heapsorthttp://en.wikipedia.org/wiki/Strict_weak_orderinghttp://en.wikipedia.org/wiki/Strict_weak_orderinghttp://en.wikipedia.org/wiki/Computer_sciencehttp://en.wikipedia.org/wiki/Mathematicshttp://en.wikipedia.org/wiki/Algorithmhttp://en.wikipedia.org/wiki/List_(computing)http://en.wikipedia.org/wiki/Total_orderhttp://en.wikipedia.org/wiki/Lexicographical_orderhttp://en.wikipedia.org/wiki/Computer_sciencehttp://en.wikipedia.org/wiki/Algorithmhttp://en.wikipedia.org/wiki/Data_structurehttp://en.wikipedia.org/wiki/Data_structurehttp://planetmath.org/encyclopedia/MatiyasevivcsTheorem.htmlhttp://planetmath.org/encyclopedia/Mean3.htmlhttp://en.wikipedia.org/wiki/Sorting_algorithmhttp://en.wikipedia.org/wiki/Arrayhttp://en.wikipedia.org/wiki/Bubble_sorthttp://en.wikipedia.org/wiki/Selection_sorthttp://en.wikipedia.org/wiki/Insertion_sorthttp://en.wikipedia.org/wiki/Heapsorthttp://en.wikipedia.org/wiki/Strict_weak_ordering


2/21

External Sorting: When the size of file to be sorted is bigger than that of availablememory!

Loop Invariants

Loop invariants provide us with a way to reason about loops, and can be used to verifythat the loops we design are correct.

Simply stated, a loop invariant is a relationship among variables in a program that istrue when control enters a loop, remains true each time the body of the loop is executed,and is still true when the loop is exited.

The General Form of a Loop

It is important to understand a bit about the anatomy of a loop, and to discuss some loopterminology. There are three essential components to any loop.

1. Initialization for the loop.

2. The reason for looping, or the termination condition for the

loop.

3. The loop body, made up of the statements to be executed each

time the program goes though the loop. The loop body must guarantee

that the loop terminates in some finite number of steps.Example

Consider the problem of computing the value of n! where n is a positive, non-zero integergreater than 1. We know from math that

(1) n! = 1 * 2 * 3 * ... * (n-1) * n

The strategy that we want to use to solve this problem is to come up with a loop thatcreates each new integer value in turn, and then multiples it by the product of the previously

computed integers.

To do this, define the program variables j and product. The variable j will hold each newinteger value as we compute it, and product will hold the product of j and the previously

computed integer values.

Clearly, when we are finished,j = n andproduct = j!

It should be easy to see that the loop body will computeproduct = product * j;

and we will want the loop to run as long as

j < n.

The loop can therefore be constructed as follows: Every time that we loop back and start to execute the loop body again, we know that jwill be less than n.

This is the reason for looping. Inside the body of the loop we need to generate one newinteger value with the statement

j = j + 1;

and use that new value to calculate a new product.

The loop body therefore has two statements:j = j + 1;

product = product * j;


3/21

Since we know that every loop needs some initialization, we need to provide that here.Our equation (1) for factorial starts with the integer 1. Furthermore, we know that 1! = 1. So,let's write

j = 1;

product = 1;

The finished loop now looks like

j = 1;product = 1;

//At this point we know that product = j! and j < n

while ( j < n )

{

j = j + 1;product = product * j;

//At this point we also know that product = j! and j < n

}// At this point, we know that product = j! and j = n

We see from the above that there is one condition that is true before we start the loop, it is

true each time we complete the body of the loop, and it is true when we exit the loop. Thiscondition, product = j!, is called the loop invariant.

Using Loop Invariants to Construct Loops

The steps that we just went through can be used in any similar situation, to construct aloop.

In summary, these steps are:1. Come up with a loop strategy that solves the problem.

2. Determine the set of variables required in the loop.3. Express the result desired when the loop exits, in terms of the loop variables.

4. Write down the reason for leaving the loop and the loop invariant. These can

usually be written in terms of the desired result when the loop exits.

5. Construct the initialization and the loop body.Loop Invariants

If a condition is true when you enter a loop it will be true when you leave each iteration

of the loop

Used to determine if you have written your loop correctly.

int fact(int n) {

int prod = 1;

int k = 0;

while(k < n) {

k = k + 1;

prod = prod * k;

}return prod;

}

Loop invariant : prod = k!

When loop terminates, k = n, hence prod = n!

Insertion Sort

Design approach: Incremental

Sorts in place: Yes

Best case: (n)


4/21

Worst case: (n2) Idea: like sorting a hand of playing cards

Start with an empty left hand and the cards facing down on the table.

Remove one card at a time from the table, and insert it into the correct position in the left

hand

compare it with each of the cards already in the hand, from right to left

The cards held in the left hand are sorted

these cards were originally the top cards of the pile on the table.

61

0

2

4

12

36

6 10 2436

12

6 1024

36

To insert 12, we need to make

room for it by moving first 36and then 24.


5/21

61

0

2

4

12

36

6 10 2436

12

To insert 12, we need to makeroom for it by moving first 36and then 24.


6/21

6 1024

36

12

5 2 4 6 13

input array

left sub-array right sub-array

at each iteration, the array is divided in two sub-arrays:

sorted unsorted


7/21

Loop Invariants and the Correctness of Insertion Sort

Proving loop invariants works like induction

Initialization (base case): It is true prior to the first iteration of the loop

Maintenance (inductive step): If it is true before an iteration of the loop, it remains true before the next iteration(

[i-1] true => [i] true ) Termination:

When the loop terminates, the invariant gives us a useful property that helps show

that the algorithm is correct

Stop the induction when the loop terminates

a8

a7

a6

a5

a4

a3

a2

a1

1 2 3 4 5 6 7 8

key


8/21

When the first two properties hold, the loop invariant is true prior to everyiteration of the loop.

Loop invariant:OriginalA[1 ..j-1] is permuted toA[1 ..j-1] but insorted order

At the start of each iteration offor loop, the subarrayA[1 ..j-1] consists ofthe elements originally inA[1 ..j-1] but in sorted order.

It can help us to understand why an algorithm is correct. Initialization:j=2,A[1 ..j-1]=A[1] consists of just the singleA[1] , which is the

original element inA[1], and is sorted. Obviously, the loop invariant holds prior to the firstiteration of the loop.

Maintenance: the while inner loop moves A[j -1], A[j -2], A[j -3], and so on, byone position to the right until the proper position for key (which has the value that started out

in A[j]) is found. At that point, the value of key is placed into this position. The second

property holds for the outer loop.

Termination:

The outer for loop ends when j = n + 1 j-1 = n Replace nwith j-1 in the loop invariant:

the subarray A[1 . . n] consists of the elements originally in A[1 . . n],

but in sorted order.

The entire array is sorted!

Analysis of Insertion Sort

Alg.:INSERTION-SORT(A) Cost Times


9/21

forj 2to n c1 ndo key A[ j ] c2 n-1

Insert A[ j ] into the sorted sequence A[1 . . j -1] c3 n-1

i j 1 c4 n-1while i > 0 and A[i] > key c5do A[i + 1] A[i] c6

i i 1 c7A[i + 1] key c8 n-1

tj: # of times the while statement is executed at iteration j

Best Case Analysis

The array is already sorted

A[i] key upon the first time the while loop test is run (when i =j -1)

tj= 1

T(n) = c1n + c2(n -1) + c4(n -1) + c5(n -1) + c8(n-1) = (c1 + c2 + c4 + c5 + c8)n + (c2 + c4 + c5+ c8)

= an + b = (n)Worst Case Analysis

The array is in reverse sorted order

Always A[i] > key in while loop test

Have to compare keywith all elements to the left of the j-th position comparewith j-1 elements tj = j, using

, we have

a quadratic function of n

T(n) = (n2) order of growth in n2

=n

j jt

2

= n

j jt

2)1(

= n

j jt

2)1(

( ) ( ) )1(11)1()1()( 827

2

6

2

5421 ++++++= === nctctctcncncncnTn

j

j

n

j

j

n

j

j

1 2 2

( 1) ( 1) ( 1)1 ( 1)

2 2 2

n n n

j j j

n n n n n nj j j

= = =

+ + = => = => =

)1(

2

)1(

2

)1(1

2

)1()1()1()( 8765421 +

+

+

+

+++= ncnn

cnn

cnn

cncncncnT

cbnan ++=2


10/21

Average-case Analysis

Want to determine the average number of comparisons taken over all possible inputs.

Determine the average no. of comparisons for a keyA[j].

A[j] can belong to any of thej locations, 1..j, with equal probability.

The number ofkey comparisons for A[j] isjk+1, ifA[j] belongs to location k, 1 < k jand isj1 if it belongs to location 1.

Average no. of comparisons for inserting keyA[j] is:

Summing over the no. of comparisons for all keys,

Therefore, Tavg(n) = (n2)Mergesort

Design approach: divide and conquer

Sorts in place: No

Running time: (nlogn) Recursive in structure

Divide the problem into sub-problems that are similar to the original but smaller

in size

Conquerthe sub-problems by solving them recursively. If they are small enough,just solve them in a straightforward manner.

Combine the solutions to create a solution to the original problemSorting Problem: Sort a sequence ofn elements into non-decreasing order.A[p . . r]:

Divide: Divide the n-element sequence to be sorted into two subsequences ofn/2

elements each

Conquer: Sort the two subsequences recursively using merge sort.

Combine: Merge the two sorted subsequences to produce the sorted answer.

( )

j

j

j

j

jk

j

jj

kj

j

k

j

k

1

2

111

2

1

11

1

)1(11

1

1

1

1

+

=+

=

+=

+

=

=

)(

)(ln)(

11

4

3

4

1

2

1

)(

2

2

2

2

2

n

nOn

i

nn

i

i

nC

n

i

n

iavg

=

=

+=

+

=

=

=


11/21


12/21

Merge Sort

Alg.: MERGE-SORT(A, p, r)

ifp < r Check for base casethen q (p + r)/2 Divide

MERGE-SORT(A, p, q) Conquer

MERGE-SORT(A, q + 1, r) Conquer

MERGE(A, p, q, r) Combine

Initial call:MERGE-SORT(A, 1, n)

1 2 3 4 5 6 7 8

62317425

p rq

Example n Power of 21 2 3 4 5 6 7 8

q = 462317425

1 2 3 4

7425

5 6 7 8

6231

1 2

25

3 4

74

5 6

31

7 8

62

1

5

2

2

3

4

4

7 1

6

3

7

2

8

6

5

Divide


13/21

Example n Power of 2

1

5

2

2

3

4

4

7 1

6

3

7

2

8

6

5

1 2 3 4 5 6 7 8

76543221

1 2 3 4

7542

5 6 7 8

6321

1 2

52

3 4

74

5 6

31

7 8

62

ConquerandMerge

Example n Not a Power of 2

62537416274

1 2 3 4 5 6 7 8 9 10 11

q = 6

416274

1 2 3 4 5 6

62537

7 8 9 10 11

q = 9q = 3

274

1 2 3

416

4 5 6

537

7 8 9

62

10 11

74

1 2

2

3

16

4 5

4

6

37

7 8

5

9

2

10

6

11

4

1

7

2

6

4

1

5

7

7

3

8

Divide


14/21

Merging

Input: Array Aand indices p, q, rsuch that p q < r

Subarrays A[p . . q] and A[q + 1 . . r] are sorted

Output: One single sorted subarray A[p . . r]

Idea for merging:

Two piles of sorted cards

Choose the smaller of the two top cards Remove it and place it in the output pile

Repeat the process until one pile is empty

Take the remaining input pile and place it face-down onto the output pile

Example n Not a Power of 2

77665443221

1 2 3 4 5 6 7 8 9 10 11

764421

1 2 3 4 5 6

76532

7 8 9 10 11

742

1 2 3

641

4 5 6

753

7 8 9

62

10 11

2

3

4

6

5

9

2

10

6

11

4

1

7

2

6

4

1

5

7

7

3

8

74

1 2

61

4 5

73

7 8

ConquerandMerge

1 2 3 4 5 6 7 8

63217542

p rq


15/21

Merge - Pseudocode

Alg.:MERGE(A, p, q, r)Compute n

1and n

2

Copy the first n1elements into L[1 . .

n1+ 1] and the next n

2elements into R[1 . . n

2+ 1]

L[n1+ 1] ; R[n

2+ 1]

i 1; j 1

fork pto r do ifL[ i ] R[ j ]

then A[k] L[ i ]

i i + 1

else A[k] R[ j ]

j j + 1

p q

7542

6321

rq + 1

L

R

1 2 3 4 5 6 7 8

63217542

p rq

n1

n2

Sentinels, to avoid having to

check if either subarray is fully

copied at each step.

n1qp + 1

n2rq

fori 1 ton1

doL[i] A[p + i 1]

forj 1 ton2

doR[j] A[q +j]


16/21

Example: MERGE(A, 9, 12, 16)

p rq

Example: MERGE(A, 9, 12, 16)


17/21

Example (cont.)

Example (cont.)


18/21

Done!

Correctness of Merge Sort

Loop invariant(at the start of the forloop)A[pk-1] contains the k-p smallest elements of

L[1 . . n1+ 1] and R[1 . . n

2+ 1] in

sorted order

L[i] and R[j] are the smallest elements not yetcopied back to A

p r

Example (cont.)


19/21

Proof of the Loop Invariant

InitializationPrior to first iteration: k = p

subarray A[p..k-1] is empty

A[p..k-1] contains the k p = 0 smallest elements

ofL and R

L and R are sorted arrays (i = j = 1)

L[1]

andR[1]

are the smallest elements inL

and

R


Maintenance

Assume L[i] R[j] L[i] is the smallest element not

yet copied to A

After copying L[i] into A[k], A[p..k] contains the k p

+ 1 smallest elements ofL and R

Incrementing k (forloop) and i reestablishes the loopinvariant


20/21


TerminationAt termination k = r + 1

By the loop invariant: A[p..k-1] = A[pr] contains the k

p = r p + 1 smallest elements ofL and R in sortedorder

Exactly the number of elements to be sortedMERGE(A, p, q, r) is correct

k = r + 1

Running Time of Merge

Initialization (copying into temporary arrays): (n

1+ n

2) = (n)

Adding the elements to the final array (the last forloop):n iterations, each taking constant time (n)

Total time for Merge:

(n)


21/21

Analyzing Divide-and ConquerAlgorithms

The recurrence is based on the three steps of theparadigm:T(n) running time on a problem of size n

Divide the problem into a subproblems, each of size

n/b: takes D(n)

Conquer(solve) the subproblems aT(n/b)

Combine the solutions C(n)

(1) ifn c

T(n) = aT(n/b) + D(n) + C(n) otherwise

Analysis of Insertion and Merge Sort

Documents

Transcript of Analysis of Insertion and Merge Sort