Algorithms and their complexityLecture 12

Waterford Institute of Technology

March 1, 2016

John Fitzgerald

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 1/33

Presentation Outline

Estimated duration presentationQuestions at end presentationTopics discussed:

• Methods to measure program complexity• Example algorithms

• Sorting• Searching

• Relative performance of algorithms

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 2/33

AlgorithmDescription

What is a software algorithm?

• Program that solves problem• Algorithm examples:

• Sorting data• Searching data• Graphs (social networks)• Cryptography• Image processing• From labs:

• Generate PIN• Validate PIN

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 3/33

AlgorithmRequirements

In developing an algorithm we areinterested in:

• Does it produce correct answer for allvalid inputs?

• How long to solve task?• How much computer memory used?• Is code easy to understand & thus

maintain?• Conceptual complexity• Computational complexity

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 4/33

Algorithm outputExpect correct result for all valid inputs

Easier said than doneExamples of incorrect output:

• Timsort• Extensively used on Android• Bug recently discovered

• Binary search• Small & conceptually simple• Yet took years to eliminate all bugs

for all inputs

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 5/33

Algorithm performanceHow long to run?

Goal not measurement run timebecause of variations in

• Computer speeds• Language implementation• Inputs

Instead, determine inherentcomplexity

• Classes of complexity exist• Provide comparison of

growth of numbercomputing operations

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 6/33

Linear search algorithmHow long to run?

Consider the linear search algorithm belowRunning times could vary significantly:

• If String search near start of String[] target• method returns immediately

• But if towards end of list• return might be much later• providing very different run times for same algorithm

boolean linearSearch(String search, String[] target)

for (int i = 0; i < target.length; i += 1)if (target[i].contains(search))

return true;return false;

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 7/33

Algorithm efficiencyChoices available

We have a choice to make:• Best case

• Considering all possible inputs• Linear search runs in constant time• Returns immediately

• Worst case• Search time proportional input size• Linear search is linear in size list

• AverageWe will choose worst case.

• We will seek an upper bound onnumber operations in algorithm.

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 8/33

Counting operationsAgree some rules

Basic assumptions• Concept of random access machine (RAM)• An abstract computer• Measure size of input• Sequential execution operations• Operations take constant time

• Assignment (a = 10)• Comparison (a==10)• Arithmetic (a/10)• Memory access (b[i])

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 9/33

Counting operationsSimple example

Approximate number operations or steps in countOperations:• 2001 + 2n + 2n2

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 10/33

Count operationsNumber steps or operations

2001 + 2n + 2n2 (ignoring for overhead)• If n small output dominated by 2000 term• For large n 2000 term insignificant

• If n is 10 output is 2221• If n is 10,000 output exceeds 200 million (2.0002E+08)

static int countOperations(int n)

int ans = 0;for (int i = 0; i < 1000; i += 1)

ans += 1;for (int i = 0; i < n; i += 1)

ans +=1;for (int i = 0; i < n; i += 1)

for (int j = 0; j < n; j += 1)ans += 1;

return ans;

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 11/33

Algorithm complexityComparing measurement formulae

In the case of 2001 + 2n + 2n2

we use n2

• We use the highest ordervariable

• And disregard any coefficient• Provides approximate upper

bound measurement• Importantly, it provides

representation of growth ofcomplexity as input becomesvery large.

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 12/33

Algorithm complexityUse of asymptotic notation

Last example: Some constant times n2

provides upper bound on numberoperationsAsymptotic notation used to describegrowth of algorithm operations as inputsize approaches infinity.We now introduce Big O notation

• Running time of 2001 + 2n + 2n2

• O(n2) or• Big-O(n2)

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 13/33

Algorithm complexityFurther asymptotic notation

Big Θ (Big Theta)• For large n :

• k · f (n) provides upperbound

• c · f (n) provide lowerbound

• k and c are some constants.

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 14/33

Algorithm complexityFurther asymptotic notation

Big Ω (Big Omega)• For large n, c · f (n) provides

lower bound run time• c is some constant.

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 15/33

Algorithmic complexityAlternative measurement approach

Tilde notation• Proposed by Robert

Sedegwick• Denoted by: ~f(n)• Differs from Big-O

• Coefficient of term withhighest exponent retained

• Consider: f(x) = 2000 + 2n+ 2n2

• ~2n2 is considered thecomplexity of thisalgorithm

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 16/33

Big-O complexityCategories

• Constant O(1) : independent of input• Linear O(n): find smallest item unsorted array• Logarithmic O(log n) : binary search• Linearithmic O(n log n) : mergesort• Polynomial O(nc) , c > 0: selection sort quadratic (c == 2)• Exponential O(2n) : Binary exponential backoff (networking)• Factorial O(n!) : Brute force search travelling salesman

problem

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 17/33

Big-O complexityCategories

• Constant O(1)• Logarithmic O(log n)• Linear O(n)• Linearithmic O(n log n)• Polynomial O(nc)• Exponential O(2n)• Factorial O(n!)

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 18/33

Algorithm complexityAcceptable categories

Very large performance differences between categoriesDoes this really matter?

• For small problems, not really• For very large problems, definitely yes.

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 19/33

Sort algorithmsSelection sort

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 20/33

Sort algorithmsMerge sort

This implementation comprises two methods:• public method sort• private method merge

public sortloop

subdivide array into pairs blocksloop all pairs blocks

invoke merge on each pairendloop

endloop

private mergeSuccessively compare elementsin left and right blocks.Incrementally populate target sorted

array.

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 21/33

Merge sortMethod sort

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 22/33

Merge sortMethod sort

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 23/33

Merge sortMethod sort

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 24/33

Merge sortMethod merge

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 25/33

Search algorithmsBinary search

Linear search complexity is O(n).A more efficient Binary search has complexity O(log n)

• Works with a sorted array• Array considered as upper and lower half• Check carried out: is item in upper or lower?• Check repeated in the half array containing item.• This search method repeated until item found if it exists.• Each iteration reduces the search space in two.• This explains the O(log n) order of complexity.

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 26/33

Search algorithmsBinary search

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 27/33

Search algorithmsBinary search

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 28/33

Big O NotationTime classification of algorithms

• One method of categorizing algorithmic processing time

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 29/33

Big O NotationSorting

Important to have regard to• Best• Average• Worst

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 30/33

Summary

• Methods to measure complexity• Elapsed time• Big O• Big Θ• Big Ω• Tilde notation

• Example algorithms• Selection sort• Merge sort• Linear search• Binary search

• Relative performance of algorithms

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 31/33

Referenced Material

1. Algorithms (Khan Academy)www.khanacademy.org/computing/computer-science/algorithms/asymptotic-notation/a/big-o-notation

[Accessed 2015-03-03]2. TimSort bug fixed with formal methods http://www.cwi.nl/news/2015/java-bug-fixed-formal-methods-cwi

[Accessed 2015-03-03]3. MOOC: edX | MITx: 6.00.1x Grimson. Introduction toComputer Science and Programming Using PythonMassachusetts Institute of Technologyhttps://www.edx.org

[Accessed 2015-03-05]

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 32/33

Referenced Material

4. MOOC: Algorithms I: Sedgewick & Wayne. PrincetonUniversity.https://www.coursera.org/course/algs4partI

[Accessed 2015-03-03]5. Robert Sedgewick : Algorithms for the masseshttp://osric.com/chris/accidental-developer/2012/04/robert-sedgewick-algorithms-for-the-masses/ [Accessed2015-03-05]

Waterford Institute of Technology, Algorithms and their complexity Lecture 12 33/33