Chapter 7Sorting

Introduction

2

sortingfundamental task in data managementwell-studied problem in computer science

basic problemgiven an of items where each item contains akey, rearrange the items so that the keys appear inascending orderthe key may be only of the item begin sorted

e.g., the item could be an entire block of informationabout a student, while the search key might be only thestudent's name

Introduction

3

we will assumethe array to sort contains only integersdefined < and > operators (comparison-based sorting)all array positions contain data to be sortedthe entire sort can be performed in _________________

number of elements is relatively small: < a few milliongiven a list of n items, we will use thenotation to indicate that the search for adoes not follow that ofsorts that cannot be performed in main memory

may be performed on disk or tapeknown as sorting

Sorting Algorithms

4

sortsinsertion sortselection sortbubble sortwhy ?

Shellsort: ___________________sorts

heapsortmergesortquicksorta lower bound on sorting by _ comparison

sorting algorithms: count sortstring sorts

Sorting

5

given keys to sort, there are possible ofthe keys!

e.g, given and the keys , there arepossible permutations:

brute force enumeration of permutations is notcomputationally once

Sorting

6

cost model for sortingthe basic operations we will count are and___________if there are array accesses that are not associated withcomparisons or swaps, we need to count them, too

programming notesif the objects being sorted are large, we should swap______________ to the objects, rather than the objectsthemselvespolymorphism: general-purpose sorting algorithms vstemplated algorithms

Insertion Sort

7

insertion sortsimple

passesin pass , , we move to its correctlocation amongfor passes to , insertion sort ensures that theelements in positions to are in sorted orderat the end of pass , the elements in positions to are insorted order

Insertion Sort

8

algorithm

for (p = 1; p < N; p++) {tmp =for (j = p; (j > 0) && (tmp < ); j--) {

swap and}

= tmp}

Insertion Sort

9

to avoid the full swap, we can use the following code:

for (p = 1; p < N; p++) {tmp =for (j = p; (j > 0) && (tmp < ); j--) {

=}

= tmp}

Insertion Sort

10

example

Insertion Sort

11

exampleposition 0 1 2 3 4 5initial sequence 42 6 1 54 0 7p = 1 6 42 1 54 0 7p = 2 6 1 42 54 0 7

1 6 42 54 0 7p = 3 1 6 42 54 0 7p = 4 1 6 42 0 54 7

1 6 0 42 54 71 0 6 42 54 70 1 6 42 54 7

p = 5 0 1 6 42 7 540 1 6 7 42 54

Insertion Sort

12

analysisbest case: the keys are alreadycomparisons, no swapsworst case: the keys are in sorted orderexpected (average) case: ?

Insertion Sort

13

analysisgiven that we wish to sort in ascending order, an_______________ is any pair that is out of order relative toone another: for which butthe list

contains the following inversions:

swapping an adjacent pair of elements that are out of order______________ exactly one inversion

thus, any sort that operates by swapping adjacent termsrequires as many swaps as there are inversions

Insertion Sort

14

number of inversionsa list can have between and inversions, thelatter of which occurs whenthus, counting inversions also says the worst-case behaviorof insertion is ________________

what do inversions tell us about the behavior?let be the probability space of all permutations ofdistinct elements with equal _____________________

Theorem: The expected number of inversions in a listtaken from isthus, the expected complexity of insertion sort isquadratic

Insertion Sort

15

proofobserve that any pair in a list that is an inversion is in thecorrect order in the of that list

this means that if we look at a list and its reverse and countthe total number of inversions, then the combined numberof inversions is

list inversions reverse lists inversions1, 2, 3, 4 0 4, 3, 2, 1 62, 1, 3, 4 1 4, 3, 1, 2 53, 2, 1, 4 3 4, 1, 2, 3 3

Insertion Sort

16

proof (cont.)since there are distinct pairs of lists and theirreverses, there is a total of

inversions among the possible lists of distinct objects

this means that the expected number of inversions in anygiven list is

Insertion Sort

17

in summaryfor randomly ordered arrays of length with distinct keys,insertion sort uses

comparisons and swaps on ___________comparisons and swaps in the _________

casecomparisons and swaps in the case

Selection Sort

18

selection sortfind the smallest item and exchange it with the first entryfind the next smallest item and exchange it with the secondentryfind the next smallest item and exchange it with the thirdentry

Selection Sort

19

algorithm

for (p = 0; p < N; p++) {m = pfor (j = p+1; j < 0; j++) {

if ( < ) {m = j

}}swap and

}

Selection Sort

20

example

position 0 1 2 3 4initial sequence 42 6 9 54 0after p = 0 0 42 9 54 6after p = 1 0 6 9 54 42after p = 2 0 6 9 54 42after p = 3 0 6 9 42 54

Selection Sort

21

complexitycomparisons and swaps to sort an array of

lengththe amount of work is of the inputselection sort is no faster on input than onrandom inputselection sort involves a smaller number of thanany of the other sorting algorithms we will consider

Selection Sort

22

proofthere is one swap per iteration of the loop,which is executed timesthere is one comparison made in each iteration of the_______________ loop, which is executed

Bubble Sort

23

bubble sortread the items from left to rightif two items are out of order, swap themrepeat until sorteda sweep with swaps means we are done

Bubble Sort

24

algorithm

not_done = truewhile (not_done) {

not_done = falsefor (i = 0 to N - 2) {

if ( > ) {swap andnot_done = true

}}

}

Bubble Sort

25

example

initial sequence 42 6 9 54 0while-loop 6 42 9 54 0

6 9 42 54 06 9 42 0 54

while-loop 6 9 0 42 54while-loop 6 0 9 42 54while-loop 0 6 9 42 54

Bubble Sort

26

complexityif the data are already sorted, we make only sweepthrough the listotherwise, the complexity depends on the number of timeswe execute the while-loopsince bubble sort swaps items, it will have_______________ worst-case and expected-casecomplexity

Shellsort

27

ShellsortDonald L. Shell (1959), A high-speed sorting procedure,Communications of the ACM 2 (7): 3032.swapping only adjacent items dooms us to quadratic worst-case behavior, so swap items!Shellsort starts with an sequence

it uses insertion sort to sortevery -th term starting at , then thenevery -th term starting at , then thenetc.every term starting at , after which the array issorted

Shellsort

28

Shellsortsuppose we use the increment sequence 15, 7, 5, 3, 1, andhave finished the 15-sort and 7-sortthen we know that

we also know that

putting these together, we see that

Shellsort

29

Shellsortafter we have performed the sort using increment , thearray is -sorted: all elements that are terms apart arein the order:

the key to Shellsort's is the following fact:an -sorted array remains -sorted after sorting withincrement

Shellsort

30

example

Shellsort

31

complexitya good increment sequence has theproperty that for any element , when it is time for the -sort, there are only a few elements to the left of that are___________ thanShell's original increment sequence has ____________behavior:

,

Shellsort

32

complexity (cont.)the sequences

(T. H. Hibbard, 1963)

(V. R. Pratt, 1971)

yield worst-case complexityother sequences yield worst-case complexity

Heapsort

33

priority queues can be used to sort instrategy

build binary of elements timeperform deleteMin operationselements (smallest first) stored in second array, then____________ back into original array time

requires extra , which doubles memoryrequirement

Heapsort

34

can avoid extra array by storing element in original arrayheap leaves an as each smallest elementis deletedstore element in newly opened space, which is no longerused by the heapresults in list of order in arrayto achieve increasing order, change ordering of heap to________ heapwhen complete, array contains elements in ascendingorder

Heapsort

35

max heap after buildHeap phase

Heapsort

36

max heap after first deleteMax

Heapsort

37

analysisbuilding binary heap of elements < comparisonstotal deleteMax operations ifheapsort in worst caseaverage case extremely complex to compute

improved to or simplyheapsort useful if we want to sort the or____________ elements and

Mergesort

38

mergesort is a algorithmin Vol. III of The Art of Computer Programming, Knuthattributes the algorithm to John von Neumann (1945)the idea of mergesort is simple:

divide the array in twosort each half

the two subarrays using mergesortmerging simple since sorted

mergesort can be implemented recursively and non-recursivelyruns in , worst case

Mergesort

39

merging algorithm takes two arrays, and , and _________array

also uses three counters: , ,initialized to beginning of respective arrays

smaller of and copied towhen either input array exhausted, the remainder of theother list is to

Mergesort

40

example

Mergesort

41

example (cont.)

Mergesort

42

example (cont.)another way to visualize

Mergesort

43

time to merge two lists is at mostcomparisons

every comparison adds an element tomergesort easy to characterize

if , only one element to sortotherwise, mergesort first half andsecond halfmerge these two halvesproblem is divided into smaller problems and solvedrecursively, and conquered by patching the solutionstogether

Mergesort

44

analysisrunning time represented by relationassume is a power of 2 so that list is always divided____________for , time to mergesort is _________________otherwise, time to mergesort numbers is time to performtwo recursive mergesort of size , plus the time tomerge, which is linear

Mergesort

45

analysis (cont.)standard recurrence relationcan be solved in at least two ways

telescoping divide the recurrence through bysubstitution

Mergesort

46

solving mergesort recurrence relation using ______________

Mergesort

47

solving mergesort recurrence relation using telescoping(cont.)

Mergesort

48

solving mergesort recurrence relation using ______________(cont.)

Mergesort

49

proof by (cont.)

Prove: is equivalent to

letBase: (or )

rr: by definitioncf:

I.H.: Assume

for some

Mergesort

50

proof by induction (cont.)

I.S.: Show:

by I.H.

By induction, we have therefore shown the original statementto be true.

Quicksort

51

historically, quicksort has been known genericsorting algorithmaverage running timeworst case running time , but can be made highly________________can be combined with to achieveaverage and worst case time

Quicksort

52

quicksort is a divide-and-conquer algorithmbasic idea

arbitrarily select a single itemform three groups:

those than the itemthose to the itemthose than the item

recursively sort the first and third groupsconcatenate the three groups

Quicksort

53

Quicksort

54

implementation performance good on most inputsif list contains many , performance is verygoodsome issues

making extra lists recursively consumes ______________not much better than mergesort

loop bodies too heavycan avoid category in loop

Quicksort

55

classic quicksort algorithm to sort an arrayif there are either 0 or 1 elements in , then ____________choose an element in to serve as the ____________partition into two disjoint subsets and with theproperties that

if andif

apply quicksort recursively to andnote the for elements equal to the pivot

ideally, half of the duplicates would go into each sublist

Quicksort

56

examplepivot chosen randomly

Quicksort

57

many methods for selecting pivot and partitioning elementsvery sensitive to even slight variances

in these choicescomparison with mergesort

like mergesort, recursively solves two subproblems andrequires additional workunlike mergesort, may not be of equalsize (bad)

Quicksort

58

quicksort vs. mergesortmergesort: is trivial; the work is in the mergequicksort: the work is in the partitioning; the is trivalmergesort: requires an auxiliary array to be efficient (in-place variants exist that are less efficient, or which sacrificean important property called stability)quicksort: faster since partitioning step can be performedefficiently in place (with a modest amount ( ) spaceneeded to handle the recursion)in both sorts, more efficient to switch to insertion sort oncethe arrays are sufficiently small to avoid the cost of theoverhead of on small arrays

Quicksort

59

examplepivots in red

Quicksort

60

choosing the pivotpopular, but bad, method: choose the element inthe list

OK if input is _______________not OK if input is presorted or in reverse order

happens consistently in recursive callsresults in time for presorted datafor doing nothing!occurs oftenalternative: choose larger of first two elements

could pick the pivot randomlysafe, but random number generation expensive

Quicksort

61

choosing the pivot (cont.)median-of-three partitioning

best choice would be median of sublist, but takes toolong to calculategood estimate by picking three elements andusing middle element as pivot

randomness not really helpfulselect first, middle, and last elements

eliminates bad case for inputreduces number of by about 15%

example: 8, 1, 4, 9, 6, 3, 5, 2, 7, 0from 8, 0, and , or 6, select 6

Quicksort

62

partitioning strategyfirst, get pivot out of the way by swapping with __________elementtwo counters, and

starts at first elementstarts at nest-to-last element

move all the smaller elements to and all largerelements to __________

Quicksort

63

partitioning strategy (cont.)while is to the left of

move right, skipping over elements smaller than pivotmove left, skipping over elements larger than pivot

when and stop, is at a element and isat a element themexample

Quicksort

64

partitioning strategy (cont.)example (cont.)

after and cross, swap location with pivot

Quicksort

65

partitioning strategy (cont.)at this point, all positions contain smaller elementsthan , and all positions contain larger elementshow to handle equal elements

should stop when element to pivot? what about ?and should behave similarly to avoid all elements

equal to pivot collecting in one sublistbest to have and stop and perform an unnecessaryswap to avoid sublists (and quadratic run time!)

for small arrays, and as sublists get small (< 20 elements),use insertion sort

fast and avoids degenerate median-of-three cases

Quicksort

66

implementationdriver

pass array and range (left and right) to be sorted

Quicksort

67

implementation (cont.)median-of-three pivot selection

sort a[left], a[right], and a[center] in ________smallest of three ends up in locationlargest in last locationpivot in a[right 1]

can be initialized to left + 1can be initialized to right 2

since a[left] smaller than pivot, it will act as a____________ and stop from going past the beginningof the arraystoring pivot at a[right 1] will act as a sentinel for

Quicksort

68

implementation (cont.)median-of-three

Quicksort

69

implementation (cont.)main quicksort

Quicksort

70

implementation (cont.)main quicksort (cont.)

Quicksort

71

implementation (cont.)main quicksort (cont.)

16: i and j start at one off22: swap can be written inline19-20: small inner loop very fast

Quicksort

72

analysisquicksort is interesting because its worst-case behaviorand its expected behavior are very _________________let be the run-time needed to sort items

pivot is constant timecost of the partition isif has elements, then has elements,and

Quicksort

73

worst-case analysisthe worst-case occurs when or i.e., whenthe is the smallest or largest element everytime quicksort() is calledin this case, without loss of generality we may assume that

, so

thus

Quicksort

74

worst-case analysis (cont.)combining these yields

or

quadratic!is it likely that at every recursive call to quicksort() wewill choose the smallest element as the pivot?

yes, if the data are already sorted

Quicksort

75

best-case analysisin the best case, the pivot is always the of thedata being operated on

we know from the analysis of mergesort that the solution is

Quicksort

76

average-case analysisassumption: any size is equally likelyfor instance, suppose ; since we remove the ,the possible sizes of the partitions are

in this case the value of is

Quicksort

77

Quicksort

78

Quicksort

79

Lower Bound for Pairwise Sorting

80

no algorithm based on comparisons can guaranteesorting items with fewer than comparisons

to show this, we first abstract the behavior of suchalgorithms using a treea decision tree is a binary tree in which each noderepresents a set of possible orderingsthe root consists of the possible orderings of the items tobe sortedthe edges represent the results of comparisons, and anode comprises the orderings consistent with thecomparisons made on the path from the root to the nodeeach consists of a single sorted ordering

Lower Bound for Pairwise Sorting

81

Lower Bound for Pairwise Sorting

82

a decision tree to sort items must have leavesthis requires a tree of depth byapproximationthus, the best case for sorting with pairwise comparisons is

Quickselect

83

thus far, the best performance to select the smallestelement is using a priority queue (heap)quicksort can be modified to solve the selection problem

quickselect

Quickselect

84

quickselect algorithmgiven a set , let be its cardinalityquickselect

if there is 1 element in , returnchoose an element in to serve as the pivotpartition into two disjoint subsets and with theproperties that

if andif

now the search proceeds on andif , then the smallest element must be in , soquickselect( , )if , then the pivot is the smallest, so returnotherwise, the smallest element must be in , and it isthe -th element of , so return quickselect( ,

)

Quickselect

85

example: find the median of 7 items ( )red denotes pivots, while grey denotes the partition that is_______________call quickselect ( ; partition, then call quickselect ( , );once again, partition; at this point, so the pivot isthe 4th element, and thus the answer

Quickselect

86

example: find the median of 7 items ( )call quickselect ( ; partition; since , we want the

smallest element of , socall quickselect ( , ); partition; since we are inside the callquickselect ( , ), we want the 1st smallest element, so wecall quickselect ( , ), which immediately exits, returning10

Quickselect

87

quickselect complexityat each recursive step quickselect ignores one partition will thismake it faster than quicksort?in the worst case, quickselect behaves like quicksort, and hascomplexity

this occurs if the one partition is at each partitioning,and we have to look at all the terms in the other partition.

best case behavior is ________________occurs if each partition is ____________since quickselect ignores one partition at each step, itsruntime satisfies the recurrence

this leads to being linear

Quickselect

88

quickselect complexity (cont.)expected behavior

suppose we choose our pivot from theterms we are searchingsuppose lies between the 25th and 75th percentiles of theterms (i.e., is larger than 1/4 and smaller than 1/4 of theterms)this means that neither partition can contain more than 3/4 ofthe terms, so the partitions can't be too ;call such aon average, how many do we need to choose before we geta good one?

a randomly chosen is good with probability ½ - a goodpivot lies in the middle 50% of the termschoosing a good pivot is like tossing a coin and seeingheads

Quickselect

89

quickselect complexity (cont.)expected behavior (cont.)

the expected number of tosses to see a heads is twoto see this, let be the expected number of tossesbefore seeing a headstoss the coin; if it's heads, we're done; if it's tails (whichoccurs with probability 1/2) we have to toss it again, so

, whence

thus, on average, quickselect will take two ___________to reduce the array to at most 3/4 of the original size

Quickselect

90

quickselect complexity (cont.)expected behavior (cont.)

in terms of ,expected value of expected time toreduce the array

since each partitioning step requires work, and weexpect to need 2 of them to reduce the array size to

, we have

Quickselect

91

quickselect complexity (cont.)expected behavior (cont.)

consider the more general recurrence, where

at the level of the recursion, starting with ,there is a single problem of size at mostthe amount of work done at each level is thus at most

the recursion continues until

so , or

Quickselect

92

quickselect complexity (cont.)expected behavior (cont.)

thus, the total amount of work is bounded above by

thus, the best case and expected case behavior ofquickselect with a randomly chosen pivot is

Introsort

93

from D. R. Musser, Introspective sorting and selectionalgorithms, Software: Practice and Experience 27 (8) 983-993, 1997introsort is a of quicksort and heapsortintrosort starts with quicksort, but switches to heapsortwhenever the number of levels of recursion exceed aspecified threshold (e.g., ).if it switches to heapsort, the subarrays that remain to besorted will likely be much than the originalarraythis approach gives an complexity,while making the most of the efficiency of quicksort

Stability

94

a sorting algorithm is stable if it preserves the relative orderof keysstable sorts:

insertion sortmergesort

unstable sorts:selection sortShell sortquicksortheapsort

Stability

95

C++ Standard Library Sorts

96

sort(): introsort, guaranteed to bestable_sort(): mergesort, guaranteed to be stable and

time, if a -sized auxiliary array can be allocatedtime for an in-place sort, otherwise

partial_sort(): sort the largest (or smallest) items

String / Radix Sort

97

we can use the special structure of character strings todevise sorting algorithmshere, a character should be understood in a general sense,as an element of an alphabet, which is a collection of items(presumably all requiring the same number of fortheir representation)the alphabet is also assumed to have an ,so we may sort charactersas a special case, we can think of treating base- numbersas a string of digits from the range to ( -digit base-numbers)since the base of a number system is sometimes called the___________, the sorting algorithms we will discuss arefrequently called radix sorts

String Sorting: LSD

98

in least-significant digit (LSD) first sorting, we sort using thedigits (characters) from least- to most-significant:

String Sorting: LSD

99

algorithm:for d from least- to most-significant digit {

sort using counting sort on the least-significant digit}

observations:since counting sort is stable, LSD sorting is stable

the stability of counting sort is essential to making LSDsorting work

algorithm assumes all the strings have the same lengthtime complexity: there are passes in the outmost loop; ineach loop iteration, we apply counting sort to digits in therange to , requiring ; total work:space complexity: same as counting sort

String Sorting: LSD

100

why not go left-to-right?the same idea, but starting with the most significant digit,doesn't work:

why does right-to-left digit sorting work but left-to-right doesnot?

String Sorting: LSD

101

in the old days, we used Hollerith cards:

early versions of and COBOL had limits of 72characters per linethis left columns 73-80 free for the card numberthat way, if you dropped your deck of cards, you could go to...

String Sorting: LSD

102

IBM Model 082 card sorter

String Sorting: LSD

103

these machines used LSD:pile the disordered cards in the input hopper and sort by the__________ digit in the sequence numbernow take the sorted decks and stack them in orderplace the combined deck back in the input hopper and sortby the next-to-last digit in the sequence numberrepeat steps 2 and 3 until sorted

Sorting Algorithms Summary

104

Shell sort is subquadratic with a suitable increment sequenceShell's original increment sequence is, in fact, quadratic inthe worst case