Sorting
description
Transcript of Sorting
![Page 1: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/1.jpg)
Sorting
15-211 Fundamental Data Structures and Algorithms
Peter Lee
February 20, 2003
![Page 2: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/2.jpg)
Announcements
Homework #4 is out
Get started today!
Reading:
Chapter 8
Quiz #2 available on Tuesday
![Page 3: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/3.jpg)
Objects in calendar are closer than they appear.
![Page 4: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/4.jpg)
Introduction to Sorting
![Page 5: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/5.jpg)
Comparison-based sorting
We assume
• Items are stored in an array.
• Can be moved around in the array.
• Can compare any two array elements.
Comparison has 3 possible outcomes:
< = >
![Page 6: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/6.jpg)
Flips and inversions
An unsorted array.
24 47 13 99 105 222
inversion flip
![Page 7: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/7.jpg)
Naïve sorting algorithms
Bubble sort.
24 47 13 99 105 22213 4713 24
Keep scanning for flips, until all are fixed
What is the running time?
![Page 8: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/8.jpg)
Insertion sort
105 47 13 99 30 222
47 105 13 99 30 222
13 47 105 99 30 222
13 47 99 105 30 222
13 30 47 99 105 222
105 47 13 99 30 222
Sorted sublist
![Page 9: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/9.jpg)
Insertion sort
for i = 2 to n do
insert a[i] in the proper place
in a[1:i-1]
![Page 10: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/10.jpg)
How fast is insertion sort?
Takes O(#inversions) steps, which is very fast if array is nearly sorted to begin with.
3 2 1 6 5 4 9 8 7 …
![Page 11: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/11.jpg)
How many inversions?
Consider the worst case:
n n-1 … 3 2 1 0
In this case, there are
n + (n-1) + (n-2) + … + 1
or
![Page 12: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/12.jpg)
How many inversions?
What about the average case? Consider:
p = x1 x2 x3 … xn-1 xn
For any such p, let rev(p) be its reversal.
Then (xi,xj) is an inversion either in p or in rev(p).
There are n(n-1)/2 pairs (xi,xj), hence the average number of inversions in a permutation is n(n-1)/4, or O(n2).
![Page 13: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/13.jpg)
How long does it take to sort?
Can we do better than O(n2)?
In the worst case?
In the average case
![Page 14: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/14.jpg)
Heapsort
Remember heaps:buildHeap has O(n) worst-case running
time.deleteMin has O(log n) worst-case
running time.
Heapsort:Build heap. O(n)
DeleteMin until empty. O(n log n)
Total worst case: O(n log n)
![Page 15: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/15.jpg)
N2 vs Nlog N
N^2Nlog N
![Page 16: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/16.jpg)
Sorting in O(n log n)
Heapsort establishes the fact that sorting can be accomplished in O(n log n) worst-case running time.
![Page 17: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/17.jpg)
Heapsort in practice
The average-case analysis for heapsort is somewhat complex.
In practice, heapsort consistently tends to use nearly n log n comparisons.
So, while the worst case is better than n2, other algorithms sometimes work better.
![Page 18: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/18.jpg)
Shellsort
Shellsort, like insertion sort, is based on swapping inverted pairs.
It achieves O(n4/3) running time.
[See your book for details.]
![Page 19: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/19.jpg)
Shellsort
Example with sequence 3, 1.
105 47 13 99 30 222
99 47 13 105 30 222
99 30 13 105 47 222
99 30 13 105 47 222
30 99 13 105 47 222
30 13 99 105 47 222
...
Several inverted pairs fixed in one exchange.
![Page 20: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/20.jpg)
Recursive Sorting
![Page 21: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/21.jpg)
Recursive sorting
Intuitively, divide the problem into pieces and then recombine the results.
If array is length 1, then done.
If array is length N>1, then split in half and sort each half.
Then combine the results.
An example of a divide-and-conquer algorithm.
![Page 22: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/22.jpg)
Divide-and-conquer
![Page 23: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/23.jpg)
Divide-and-conquer
![Page 24: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/24.jpg)
Why divide-and-conquer works
Suppose the amount of work required to divide and recombine is linear, that is, O(n).
Suppose also that the amount of work to complete each step is greater than O(n).
Then each dividing step reduces the amount of work by greater than a linear amount, while requiring only linear work to do so.
![Page 25: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/25.jpg)
Divide-and-conquer is big
We will see several examples of divide-and-conquer in this course.
![Page 26: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/26.jpg)
Recursive sorting
If array is length 1, then done.
If array is length N>1, then split in half and sort each half.
Then combine the results.
![Page 27: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/27.jpg)
Analysis of recursive sorting
Suppose it takes time T(N) to sort N elements.
Suppose also it takes time N to combine the two sorted arrays.
Then:T(1) = 1T(N) = 2T(N/2) + N, for N>1
Solving for T gives the running time for the recursive sorting algorithm.
![Page 28: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/28.jpg)
Remember recurrence relations?
Systems of equations such as
T(1) = 1
T(N) = 2T(N/2) + N, for N>1
are called recurrence relations (or sometimes recurrence equations).
![Page 29: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/29.jpg)
A solution
A solution for
T(1) = 1T(N) = 2T(N/2) + N
is given by
T(N) = Nlog N + Nwhich is O(Nlog N).
How to solve such equations?
![Page 30: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/30.jpg)
Recurrence relations
There are several methods for solving recurrence relations.
It is also useful sometimes to check that a solution is valid.This is done by induction.
![Page 31: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/31.jpg)
Checking a solution
Base case:
T(1) = 1log 1 + 1 = 1
Inductive case:
Assume T(M) = Mlog M + M, all M<N.T(N) = 2T(N/2) + N
![Page 32: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/32.jpg)
Checking a solution
Base case:
T(1) = 1log 1 + 1 = 1
Inductive case:
Assume T(M) = Mlog M + M, all M<N.T(N) = 2T(N/2) + N
![Page 33: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/33.jpg)
Checking a solution
Base case:
T(1) = 1log 1 + 1 = 1
Inductive case:
Assume T(M) = Mlog M + M, all M<N.T(N) = 2T(N/2) + N = 2((N/2)(log(N/2))+N/2)+N = N(log N - log 2)+2N = Nlog N - N + 2N = Nlog N + N
![Page 34: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/34.jpg)
Logarithms
Some useful equalities.
xA = B iff logxB = A
log 1 = 0
log2 2 = 1
log(AB) = log A + log B, if A, B > 0
log(A/B) = log A - log B, if A, B > 0
log(AB) = Blog A
![Page 35: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/35.jpg)
Upper bounds for rec. relations
Divide-and-conquer algorithms are very useful in practice.
Furthermore, they all tend to generate similar recurrence relations.
As a result, approximate upper-bound solutions are well-known for recurrence relations derived from divide-and-conquer algorithms.
![Page 36: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/36.jpg)
Divide-and-Conquer Theorem
Theorem: Let a, b, c 0.
The recurrence relation
T(1) = bT(N) = aT(N/c) + bN for any N which is a power of c
has upper-bound solutions
T(N) = O(N) if a<c
T(N) = O(Nlog N) if a=c
T(N) = O(Nlogca) if a>c
a=2, b=1,c=2 for recursivesorting
![Page 37: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/37.jpg)
Upper-bounds
Corollary:
Dividing a problem into p pieces, each of size N/p, using only a linear amount of work, results in an O(Nlog N) algorithm.
![Page 38: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/38.jpg)
Upper-bounds
Proof of this theorem later in the semester.
![Page 39: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/39.jpg)
Exact solutions
Recall from earlier in the semester that it is sometimes possible to derive closed-form solutions to recurrence relations.
Several methods exist for doing this.
As an example, consider again our current equations:T(1) = 1
T(N) = 2T(N/2) + N, for N>1
![Page 40: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/40.jpg)
Repeated substitution method
One technique is to use repeated substitution.
T(N) = 2T(N/2) + N2T(N/2) = 2(2T(N/4) + N/2) = 4T(N/4) + NT(N) = 4T(N/4) + 2N4T(N/4) = 4(2T(N/8) + N/4) = 8T(N/8) + NT(N) = 8T(N/8) + 3NT(N) = 2kT(N/2k) + kN
![Page 41: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/41.jpg)
Repeated substitution, cont’d
We end up with
T(N) = 2kT(N/2k) + kN, for all k>1
Let’s use k=log N.
Note that 2log N = N.
So:
T(N) = NT(1) + Nlog N = Nlog N + N
![Page 42: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/42.jpg)
Other methods
There are also other methods for solving recurrence relations.
For example, the “telescoping method”…
![Page 43: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/43.jpg)
Telescoping method
We start with
T(N) = 2T(N/2) + N
Divide both sides by N to get
T(N)/N = (2T(N/2) + N)/N
= T(N/2)/(N/2) + 1
This is valid for any N that is a power of 2, so we can write the following:
![Page 44: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/44.jpg)
Telescoping method, cont’d
Additional equations:
T(N)/N = T(N/2)/(N/2) + 1
T(N/2)/(N/2) = T(N/4)/(N/4) + 1
T(N/4)/(N/4) = T(N/8)/(N/8) + 1
…
T(2)/2 = T(1)/1 + 1
What happens when we sum all the left-hand and right-hand sides?
![Page 45: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/45.jpg)
Telescoping method, cont’d
Additional equations:
T(N)/N = T(N/2)/(N/2) + 1
T(N/2)/(N/2) = T(N/4)/(N/4) + 1
T(N/4)/(N/4) = T(N/8)/(N/8) + 1
…
T(2)/2 = T(1)/1 + 1
We are left with:T(N)/N = T(1)/1 + log(N)
![Page 46: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/46.jpg)
Telescoping method, cont’d
We are left with
T(N)/N = T(1)/1 + log(N)
Multiplying both sides by N gives
T(N) = N log(N) + N
![Page 47: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/47.jpg)
Mergesort
![Page 48: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/48.jpg)
Mergesort
Mergesort is the most basic recursive sorting algorithm.
Divide array in halves A and B.
Recursively mergesort each half.
Combine A and B by successively looking at the first elements of A and B and moving the smaller one to the result array.
Note: Should be a careful to avoid creating of lots of result arrays.
![Page 49: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/49.jpg)
Mergesort
![Page 50: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/50.jpg)
Mergesort
But don’t actually want to create all of these arrays!
![Page 51: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/51.jpg)
Mergesort
L LR L
Use simple indexes to perform the split.
Use a single extra array to hold each intermediate result.
![Page 52: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/52.jpg)
Analysis of mergesort
Mergesort generates almost exactly the same recurrence relations shown before.
T(1) = 1
T(N) = 2T(N/2) + N - 1, for N>1
Thus, mergesort is O(Nlog N).
![Page 53: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/53.jpg)
Quicksort
![Page 54: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/54.jpg)
Quicksort
Quicksort was invented in 1960 by Tony Hoare.
Although it has O(N2) worst-case performance, on average it is O(Nlog N).
More importantly, it is the fastest known comparison-based sorting algorithm in practice.
![Page 55: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/55.jpg)
Quicksort idea
Choose a pivot.
![Page 56: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/56.jpg)
Quicksort idea
Choose a pivot.
Rearrange so that pivot is in the “right” spot.
![Page 57: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/57.jpg)
Quicksort idea
Choose a pivot.
Rearrange so that pivot is in the “right” spot.
Recurse on each half and conquer!
![Page 58: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/58.jpg)
Quicksort algorithm
If array A has 1 (or 0) elements, then done.
Choose a pivot element x from A.
Divide A-{x} into two arrays:
B = {yA | yx}
C = {yA | yx}
Quicksort arrays B and C.
Result is B+{x}+C.
![Page 59: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/59.jpg)
Quicksort algorithm
105 47 13 17 30 222 5 19
5 17 13 47 30 222 10519
5 17 30 222 105
13 47
105 222
![Page 60: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/60.jpg)
Quicksort algorithm
105 47 13 17 30 222 5 19
5 17 13 47 30 222 10519
5 17 30 222 105
13 47
In practice, insertion sort is used once the arrays get “small enough”.
105 222
![Page 61: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/61.jpg)
Doing quicksort in place
85 24 63 50 17 31 96 45
85 24 63 45 17 31 96 50
L R
85 24 63 45 17 31 96 50
L R
31 24 63 45 17 85 96 50
L R
![Page 62: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/62.jpg)
Doing quicksort in place
31 24 63 45 17 85 96 50
L R
31 24 17 45 63 85 96 50
R L
31 24 17 45 50 85 96 63
31 24 17 45 63 85 96 50
L R
![Page 63: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/63.jpg)
Quicksort is fast but hard to do
Quicksort, in the early 1960’s, was famous for being incorrectly implemented many times.
More about invariants next time.
Quicksort is very fast in practice.
Faster than mergesort because Quicksort can be done “in place”.
![Page 64: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/64.jpg)
Informal analysis
If there are duplicate elements, then algorithm does not specify which subarray B or C should get them.Ideally, split down the middle.
Also, not specified how to choose the pivot.Ideally, the median value of the array,
but this would be expensive to compute.
As a result, it is possible that Quicksort will show O(N2) behavior.
![Page 65: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/65.jpg)
Worst-case behavior
105 47 13 17 30 222 5 195
47 13 17 30 222 19 105
47 105 17 30 222 19
13
17
47 105 19 30 22219
![Page 66: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/66.jpg)
Analysis of quicksort
Assume random pivot.
T(0) = 1
T(1) = 1
T(N) = T(i) + T(N-i-1) + cN, for N>1
where I is the size of the left subarray.
![Page 67: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/67.jpg)
Worst-case analysis
If the pivot is always the smallest element, then:
T(0) = 1T(1) = 1T(N) = T(0) + T(N-1) + cN, for N>1 T(N-1) + cN = O(N2)
See the book for details on this solution.
![Page 68: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/68.jpg)
Best-case analysis
In the best case, the pivot is always the median element.
In that case, the splits are always “down the middle”.
Hence, same behavior as mergesort.
That is, O(Nlog N).
![Page 69: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/69.jpg)
Average-case analysis
Consider the quicksort tree:
105 47 13 17 30 222 5 19
5 17 13 47 30 222 10519
5 17 30 222 105
13 47
105 222
![Page 70: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/70.jpg)
Average-case analysis
The time spent at each level of the tree is O(N).
So, on average, how many levels?That is, what is the expected height of
the tree?
If on average there are O(log N) levels, then quicksort is O(Nlog N) on average.
![Page 71: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/71.jpg)
Average-case analysis
We’ll answer this question next time…
![Page 72: Sorting](https://reader036.fdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c522/html5/thumbnails/72.jpg)
Summary of quicksort
A fast sorting algorithm in practice.
Can be implemented in-place.
But is O(N2) in the worst case.
Average-case performance?