Algorithmic Complexity:Algorithmic Complexity: Complexity Analysis of Time Complexity Analysis of Time

Complexity Complexity ComplexitiesComplexities

Nate the Great

The complexity of an algorithm is The complexity of an algorithm is determined by the amount of determined by the amount of resources it usesresources it uses

Usually the most important resource Usually the most important resource we are concerned with is time— we are concerned with is time— difficult problems take longer to difficult problems take longer to solve in generalsolve in general

But how can we measure the amount But how can we measure the amount of time required by an algorithm? of time required by an algorithm?

Algorithm ComplexityAlgorithm Complexity

Time ComplexityTime Complexity

We cannot simply use a stopwatchWe cannot simply use a stopwatch The time required by an algorithm The time required by an algorithm

depends on the particular machine depends on the particular machine and how efficient the source code isand how efficient the source code is

Example: It usually takes longer to Example: It usually takes longer to sort 100,000 numbers than to sort sort 100,000 numbers than to sort say 1,000 numbers but…say 1,000 numbers but…

So…what then?So…what then?

Time ComplexityTime Complexity Time is not measured in seconds but Time is not measured in seconds but

rather in basic operations or program rather in basic operations or program steps.steps.

The input problem is assigned a size.The input problem is assigned a size. The time required by an algorithm is then The time required by an algorithm is then

computed as a function of the size of its computed as a function of the size of its input.input.

But algorithms generally take different But algorithms generally take different amounts of time on different inputs, even amounts of time on different inputs, even of the same size.of the same size.

Crap. Now what?Crap. Now what?

Time ComplexityTime Complexity The time complexity function F of an algorithm A The time complexity function F of an algorithm A

is defined based on the worst case scenario: F is is defined based on the worst case scenario: F is the greatest possible amount of time required for the greatest possible amount of time required for A to solve a problem instance of size n.A to solve a problem instance of size n.

We do not worry about time used on small inputsWe do not worry about time used on small inputs—these can be handled as special cases. —these can be handled as special cases.

Instead, we are only concerned with time Instead, we are only concerned with time complexities based on large inputs.complexities based on large inputs.

Constant factors do not matter. For example, Constant factors do not matter. For example, special hardware can always be developed that is special hardware can always be developed that is faster by a constant factor.faster by a constant factor.

Running times that differ only by constant factors Running times that differ only by constant factors are always of the same Order.are always of the same Order.

Order NotationOrder Notation Order Notation is used to measure running time bounds on algorithmsOrder Notation is used to measure running time bounds on algorithms The 4 most common bounds are:The 4 most common bounds are:

• OO (big Oh): upper bound (for worst case scenario) (big Oh): upper bound (for worst case scenario)• Ω (omega): lower bound (for best case scenario)Ω (omega): lower bound (for best case scenario)• o (little oh): strict upper boundo (little oh): strict upper bound• Θ (theta): equality (exact bound)Θ (theta): equality (exact bound)

From this point forward, we will focus only on O, a worst case upper boundFrom this point forward, we will focus only on O, a worst case upper bound Precise Definition of O (pronounced “Big-Oh”)Precise Definition of O (pronounced “Big-Oh”)

• A function f(n) is A function f(n) is OO(g(n)) iff there exists a constant n(g(n)) iff there exists a constant n00 such that for all n ≥n such that for all n ≥n00, f(n) , f(n) ≤ c*g(n), where c is a positive constant.≤ c*g(n), where c is a positive constant.

• We say that f is order g, f has order g, or f is of order gWe say that f is order g, f has order g, or f is of order g Two Important NotesTwo Important Notes

• 1) It is NOT required that f(n) ≤ g(n). We can make the constant c as large as 1) It is NOT required that f(n) ≤ g(n). We can make the constant c as large as necessary.necessary.

• 2) The condition f(n) ≤ c*g(n) is not required to hold for ALL n, but only for n ≥n2) The condition f(n) ≤ c*g(n) is not required to hold for ALL n, but only for n ≥n00 Stated differently, time bound functions give us a measure of how fast or Stated differently, time bound functions give us a measure of how fast or

slow a function grows.slow a function grows.

Order Notation ExamplesOrder Notation Examples Two main Issues:Two main Issues:

• Determining the order of a given functionDetermining the order of a given function• Determining if a function grows faster or slower than another function, in terms Determining if a function grows faster or slower than another function, in terms

of order notationof order notation Suppose f(n) = 500n and g(n) = 2nSuppose f(n) = 500n and g(n) = 2n

• What is the order of f(n)?What is the order of f(n)?• Is f(n) O(g(n))?Is f(n) O(g(n))?• Is g(n) O(f(n))?Is g(n) O(f(n))?

Suppose h(n) = 2Suppose h(n) = 2nn and p(n) = n and p(n) = n2020

• What is the order of p(n)?What is the order of p(n)?• Is h(n) O(p(n))?Is h(n) O(p(n))?• Is p(n) O(h(n))?Is p(n) O(h(n))?

Other Examples:Other Examples:• f(x) = 9xf(x) = 9x• g(x) = 3xg(x) = 3x22

• Is f(x) Is f(x) OO(g(x)?(g(x)?• Is g(x) Is g(x) OO(f(x))?(f(x))?• Is either f(x) or g(x) Is either f(x) or g(x) OO(x(xxx)?)?• What about h(x) = 4log(x) + 3/x – 16x + 240000xWhat about h(x) = 4log(x) + 3/x – 16x + 240000x33 – 0.000001x – 0.000001x55

Moral: Consider only large values of n and ignore constant factors.Moral: Consider only large values of n and ignore constant factors.

Order Notation ReviewOrder Notation Review So f(n) being Θ(g(n)) means:So f(n) being Θ(g(n)) means:

• f(n) is f(n) is OO(g(n)) and f(n) is Ω(g(n))(g(n)) and f(n) is Ω(g(n)) The most common bound is O, which proves some upper bound on a The most common bound is O, which proves some upper bound on a

functionfunction We use O notation to specify an upper bound on the running time of We use O notation to specify an upper bound on the running time of

algorithmsalgorithms The running time is almost always based on the size of the input, and the The running time is almost always based on the size of the input, and the

variable n is normally used to describe the input sizevariable n is normally used to describe the input size To construct a bound, we assume that some operation takes a constant To construct a bound, we assume that some operation takes a constant

amount of time, and base the remaining operations and running time on amount of time, and base the remaining operations and running time on that metric. Exactly what kind of operation is assumed to take constant that metric. Exactly what kind of operation is assumed to take constant time depends on the context of the problem, but it is usually obvious.time depends on the context of the problem, but it is usually obvious.

ExampleExample• Consider a vector containing n elementsConsider a vector containing n elements• Let’s assume that we can print out a single element in constant time.Let’s assume that we can print out a single element in constant time.• What kind of upper bound can we place on a function that simply prints out each What kind of upper bound can we place on a function that simply prints out each

element in the vector?element in the vector?• Suppose it takes time proportional to n to print a single element though. What is Suppose it takes time proportional to n to print a single element though. What is

an upper bound on the overall running time of the function?an upper bound on the overall running time of the function?

Key ConceptsKey Concepts

Experience in Computer Science has Experience in Computer Science has shown that shown that • Algorithms tend to be applied to larger and Algorithms tend to be applied to larger and

larger problems, so the rate of increase in time larger problems, so the rate of increase in time as the problem gets larger is more important as the problem gets larger is more important than the time required for small problems.than the time required for small problems.

• The “value” of an algorithm tends to be The “value” of an algorithm tends to be determined by the worst-case behavior, so the determined by the worst-case behavior, so the worst-case behavior is always quoted.worst-case behavior is always quoted.

Running TimesRunning Times

Quick Sort: t(N) ≈ a + b* NQuick Sort: t(N) ≈ a + b* N22

Selection Sort: t(N) ≈ a + b* NSelection Sort: t(N) ≈ a + b* N22

Merge Sort: t(N) = a + b*NlogMerge Sort: t(N) = a + b*Nlog22(N)(N)

Final Time Complexity Final Time Complexity ComplexitiesComplexities

Helpful ConceptsHelpful Concepts• Logarithms with constant bases?Logarithms with constant bases?• Exponentials with constant bases?Exponentials with constant bases?• Bounding an uncommon function?Bounding an uncommon function?• Be careful not to over- or under-bound your Be careful not to over- or under-bound your

time bounds (when working with strange time bounds (when working with strange bounds).bounds).

It’s relatively easy to correctly bound above.It’s relatively easy to correctly bound above. It’s relatively difficult to know if you bounded below.It’s relatively difficult to know if you bounded below.

Relative order of time boundsRelative order of time bounds