461191 Discrete Mathematics Lecture 3: Algorithms, The Integers, and Matrices
description
Transcript of 461191 Discrete Mathematics Lecture 3: Algorithms, The Integers, and Matrices
461191 Discrete Mathematics
Lecture 3: Algorithms, The Integers, and Matrices
San RatanasanyaCS, KMUTNB
Adapted from several sources: Michael P. Frank, University of Florida, University of Nevada, and University of Maryland at Baltimore
Today’s Topics
Review Administrivia Algorithms Algorithmic and Problem Complexity Introduction to Number Theory Applications of Number Theory Matrices
…from last week… Sets and Set operations
Set is an unordered collection of objects Operations on Set: union, intersection, difference, complement,
symmetric difference Venn Diagram and equivalence
Functions Domain, Codomain, Range One-to-one, onto, bijection, inverse, and composite functions
Sequences Represents ordered list of elements
a1,a,2a3,…,an or {an} It is convenient to describe Sequence using formula
an = 2an-1
Summations
Example: Set
Prove that A A = U is true
A = {x | x A}A = {x | x A}A A = {x | x A x A}A A = {x | x U}A A = U
Example: Function Let f(x) = x2/3. Find f(S) if S = {x | xZ |x| 3}
f(3) = 32/3 = 3, f(2) = 22/3 = 1, f(1) = 12/3 = 0, f(0) = 0 f(S) = {0, 1, 3}
Suppose that g is a function from A to B and f is a function form B to C. Show that if both f and g are one-to-one functions, then fg is also one-to-one.
We need to show that x A y A (x y f(g(x)) f(g(y))).Or we need to show that x A y A (x y g(x) g(y)) x B y B (x y f(x) f(y)).Given that g is one-to-one, therefore g(x) g(y) in B.Given that f is one-to-one, therefore f(x) f(y) in C.We can now conclude that f(g(x)) f(g(y)).
Example: Sequence
Find a simple formula for an if the first 10 terms of the sequence {an} are 1, 7, 25, 79, 241, 727, 2185, 6559, 19681, 59047
a1 = 1
an = 3n - 2
Example: Summations
?3
1
3
0
i j
i
242
)13(*3*4
4
4
1
3
1
3
1
3
1
3
0
i
i
i j
i
i
i
0
1
2123
1201
aa
aa
aaaa
aaaa
n
nn
nn
2
2
111
2
)1)((*2
212
n
nnnnnn
nkkn
k
n
k
n
k
?)12(1
n
k
k
?1
1
n
iii aa
Administrivia Homeworks due today
Homework 1.1 – 1.4 Homework 2
Programs used in this class cannot be downloaded from class webpage
they are TOO BIG… There are only links to download those
programs on class webpage Prolog, ISETLW Python, Scheme
Information on what to download is also on the webpage
Algorithms The foundation of computer programming. Most generally, an algorithm just means a definite
procedure for performing some sort of task. A computer program is simply a description of an
algorithm, in a language precise enough for a computer to understand, requiring only operations that the computer already knows how to do.
We say that a program implements (or “is an implementation of”) its algorithm.
Algorithms You Already Know Grade-school arithmetic algorithms:
How to add any two natural numbers written in decimal on paper, using carries.
Similar: Subtraction using borrowing. Multiplication & long division.
Your favorite cooking recipe. How to register for classes at KMUTNB.
Programming Languages Some common programming languages:
Newer: Java, C, C++, C#, Visual Basic, JavaScript, Perl, Tcl, Pascal, many others…
Older: Fortran, Cobol, Lisp, Basic Assembly languages, for low-level coding.
In this class we will use an informal, Pascal-like “pseudo-code” language.
You should know at least 1 real language!
Algorithm Example (English) Task: Given a sequence {ai}=a1,…,an, aiN,
say what its largest element is. One algorithm for doing this, in English:
Set the value of a temporary variable v (largest element seen so far) to a1’s value.
Look at the next element ai in the sequence. If ai>v, then re-assign v to the number ai. Repeat then previous 2 steps until there are no
more elements in the sequence, & return v.
Executing an Algorithm When you start up a piece of software, we
say the program or its algorithm are being run or executed by the computer.
Given a description of an algorithm, you can also execute it by hand, by working through all of its steps with pencil & paper.
Before ~1940, “computer” meant a person whose job was to execute algorithms!
Executing the Max algorithm
Let {ai}=7,12,3,15,8. Find its maximum… Set v = a1 = 7. Look at next element: a2 = 12. Is a2>v? Yes, so change v to 12. Look at next element: a2 = 3. Is 3>12? No, leave v alone…. Is 15>12? Yes, v=15…
Algorithm CharacteristicsSome important general features of algorithms: Input. Information or data that comes in. Output. Information or data that goes out. Definiteness. Algorithm is precisely defined. Correctness. Outputs correctly relate to inputs. Finiteness. Won’t take forever to describe or run. Effectiveness. Individual steps are all do-able. Generality. Works for many possible inputs. Efficiency. Takes little time & memory to run.
Our Pseudocode Language
procedurename(argument: type)
variable := expressioninformal statementbegin statements end{comment}if condition then
statement [else statement]
for variable := initial value to final value statement
while condition statement
procname(arguments) Not defined in book:return expression
What is Pseudocode Pseudocode is an intermediate step between
language description, normally English, of the steps of a procedure and a specification of this procedure using an actual programming language.
เป็�นการอธิ�บายโป็รแกรมด้�วยภาษาที่��ใช้�ก�นที่��วไป็ ด้�วยลั�กษณะที่��คลั�ายภาษาที่��ใช้�ในการเขี�ยนโป็รแกรม
HumanLanguages
ProgrammingLanguages
Pseudocode
What is Program Program = Algorithm + Data Structure Each step of a program may make changes to variables Trace = a sequence of changes in variables as each step of
program is executed
Trace is a simple model of program executions State = variable
procedure procname(arg: type) Declares that the following text defines a
procedure named procname that takes inputs (arguments) named arg which are data objects of the type type. Example:
procedure maximum(L: list of integers)[statements defining maximum…]
variable := expression An assignment statement evaluates the
expression expression, then reassigns the variable variable to the value that results. Example assignment statement:
v := 3x+7 (If x is 2, changes v to 13.) In pseudocode (but not real code), the
expression might be informally stated: x := the largest integer in the list L
Informal statement Sometimes we may write a statement as an
informal English imperative, if the meaning is still clear and precise: e.g.,“swap x and y”
Keep in mind that real programming languages never allow this.
When we ask for an algorithm to do so-and-so, writing “Do so-and-so” isn’t enough! Break down algorithm into detailed steps.
begin statements end Groups a sequence of
statements together:begin statement 1 statement 2 … statement n end
Allows the sequence to be used just like a single statement.
Might be used: After a procedure
declaration. In an if statement after
then or else. In the body of a for or
while loop.Curly braces {} are used insteadin many languages.
{comment} Not executed (does nothing). Natural-language text explaining some
aspect of the procedure to human readers. Also called a remark in some real
programming languages, e.g. BASIC. Example, might appear in a max program:
{Note that v is the largest integer seen so far.}
Conditional Constructionsif condition then statement
if condition thenbegin Block of statementsend
if condition then statement 1else statement 2
if condition 1 then statement 1else if condition 2 then statement 2else if condition 2 then statement 2………………………………………………………else if condition n then statement nelse statement n+1
if condition then begin statement 1end
else begin statement 2end
if condition then statement Evaluate the propositional expression
condition. If the resulting truth value is True, then
execute the statement statement; otherwise, just skip on ahead to the next
statement after the if statement. Variant: if cond then stmt1 else stmt2
Like before, but iff truth value is False, executes stmt2.
Loop Constructionsfor variable := initial value to final
value statement
for variable := initial value to final value begin
Block of statementsend
for all element with certain property
while condition statement
while conditionbegin Block of statements 1end
sum := 0for i := 1 to n sum := sum + i
sum := 0while n > 0 begin sum := sum + n n := n - 1end
while condition statement Evaluate the propositional (Boolean)
expression condition. If the resulting value is True, then execute
statement. Continue repeating the above two actions
over and over until finally the condition evaluates to False; then proceed to the next statement.
while condition statement Also equivalent to infinite nested ifs, like so:
if condition begin statement if condition begin statement …(continue infinite nested if’s) end end
for var := initial to final stmt Initial is an integer expression. Final is another integer expression. Semantics: Repeatedly execute stmt, first with
variable var := initial, then with var := initial+1, then with var := initial+2, etc., then finally with var := final.
Question: What happens if stmt changes the value of var, or the value that initial or final evaluates to?
for var := initial to final stmt For can be exactly defined in terms of
while, like so: begin var := initial while var final begin stmt var := var + 1 endend
procedure(argument) A procedure call statement invokes the
named procedure, giving it as its input the value of the argument expression.
Various real programming languages refer to procedures as functions (since the procedure call notation works similarly to function application f(x)), or as subroutines, subprograms, or methods.
Nested Block of statements Conditional Loop Procedure – a call to procedure within procedure
Procedure statistic (L:list of integers)begin max(L) min(L) average(L) ………..end
Max procedure in pseudocode
procedure max(a1, a2, …, an: integers)
v := a1 {largest element so far}
for i := 2 to n {go thru rest of elems} if ai > v then v := ai {found bigger?}
{at this point v’s value is the same as the largest integer in the list}
return v
Inventing an Algorithm Requires a lot of creativity and intuition
Like writing proofs. Unfortunately, we can’t give you an
algorithm for inventing algorithms. Just look at lots of examples… And practice (preferably, on a computer) And look at more examples… And practice some more… etc., etc.
Algorithm-Inventing Example Suppose we ask you to write an algorithm
to compute the predicate:IsPrime:N→{T,F}
Computes whether a given natural number is a prime number.
First, start with a correct predicate-logic definition of the desired function:n: IsPrime(n) ¬1<d<n: d|n
Means d divides nevenly (without remainder)
IsPrime example, cont. Notice that the negated exponential can be
rewritten as a universal:¬1<d<n: d|n 1<d<n: d | n
2≤ d ≤ n−1: d | n This universal can then be translated directly into
a corresponding for loop:for d := 2 to n−1 { Try all potential divisors >1 & <n } if d|n then return F { n has divisor d; not prime }return T { no divisors were found; n must be prime}
Means d does not divide n evenly (the remainder is ≠0)
Optimizing IsPrime The IsPrime algorithm can be further optimized:
for d := 2 to n1/2 if d|n then return Freturn T
This works because of this theorem: If n has any (integer) divisors, it must have one less than n1/2.Proof: Suppose n’s smallest divisor >1 is a, and let b :≡ n/a. Then n = ab, but if a > n1/2 then b > n1/2 (since a is n’s smallest divisor) and so n = ab > (n1/2)2 = n, an absurdity. Further optimizations are possible:
- E.g., only try divisors that are primes less than n1/2.
Note smaller range of search.
Another example task Problem of searching an ordered list.
Given a list L of n elements that are sorted into a definite order (e.g., numeric, alphabetical),
And given a particular element x, Determine whether x appears in the list, and if so, return its index (position) in the list.
Problem occurs often in many contexts. Let’s find an efficient algorithm!
Search alg. #1: Linear Search
procedure linear search(x: integer, a1, a2, …, an: distinct integers)i := 1 {start at beginning of list}while (i n x ai) {not done, not found}
i := i + 1 {go to the next position}if i n then location := i {it was found}else location := 0 {it wasn’t found}return location {index or 0 if not found}
Search alg. #2: Binary Search Basic idea: On each step, look at the middle
element of the remaining list to eliminate half of it, and quickly zero in on the desired element.
<x >x<x <x
Search alg. #2: Binary Search
procedure binary search(x:integer, a1, a2, …, an: distinct integers) i := 1 {left endpoint of search interval}j := n {right endpoint of search interval}while i<j begin {while interval has >1 item}
m := (i+j)/2 {midpoint}if x>am then i := m+1 else j := m
endif x = ai then location := i else location := 0return location
binary search for the letter binary search for the letter ‘j’‘j’
a c d f g h j l m o p r s u v x za c d f g h j l m o p r s u v x z
center elementcenter element
search intervalsearch interval
a c d f g h j l m a c d f g h j l m o p r s u v x zo p r s u v x z
center elementcenter element
search intervalsearch interval
binary search for the letter binary search for the letter ‘j’‘j’
a c d f ga c d f g h j l m h j l m o p r s u v x zo p r s u v x z
center elementcenter element
search intervalsearch interval
a c d f ga c d f g h j h j l m o p r s u v x zl m o p r s u v x z
center elementcenter element
search intervalsearch interval
binary search for the letter binary search for the letter ‘j’‘j’
a c d f g ha c d f g h j j l m o p r s u v x zl m o p r s u v x z
center elementcenter element
search intervalsearch interval
found !found !
Practice exercises Devise an algorithm that finds the sum of
all the integers in a list. [2 min] procedure sum(a1, a2, …, an: integers)
s := 0 {sum of elems so far}for i := 1 to n {go thru all elems}
s := s + ai {add current item}{at this point s is the sum of all items}return s
Sorting Algorithms Sorting is a common operation in many
applications. E.g. spreadsheets and databases
It is also widely used as a subroutine in other data-processing algorithms.
Two sorting algorithms shown in textbook: Bubble sort Insertion sort
However, these are notvery efficient, and you shouldnot use them on large data sets!
We’ll see some more efficient algorithms later in the course.
Smallest elements “float” up to the top of the list, like bubbles in a container of liquid.
Bubble Sort
3031
13233
234
3
301
3132
233
334
13031
232
33334
130
231
3323334
12
303
31323334
123
3031323334
Insertion Sort Algorithm English description of algorithm:
For each item in the input list, “Insert” it into the correct place in the sorted
output list generated so far. Like so: Use linear or binary search to find the location where the
new item should be inserted. Then, shift the items from that position onwards down by
one position. Put the new item in the hole remaining.
Pseudocodes
Procedure insertionsort(a1,…,an: real numbers with n 2)
for j := 2 to nbegin
i := 1 while aj > ai
i := i +1 m : = aj
for k := 0 to j – i – 1aj-k := aj-k-1
ai := am
end {a1, a2,…,an are sorted}
Procedure bubblesort(a1,…,an: real numbers with n 2)
for i := 1 to n-1 for j := 1 to n-1 if aj > aj+1 then interchange aj and aj+1
{a1,…,an is in increasing order}
Greedy Algorithms
Any algorithms selecting the ‘best’ choice at each step are called Greedy
Can solve optimization problemProcedure change(c1, c2,…,cr: values of denomination of coins, where c1 > c2 >… > cr; n: a positive integer)for i := 1 to r while n ci begin add a coin with value ci to the change n := n - ci
end
Analysis of Algorithms What is the goal of analysis of algorithms?
To compare algorithms mainly in terms of running time but also in terms of other factors (e.g., memory requirements, programmer's effort etc.)
What do we mean by running time analysis? Determine how running time increases as the
size of the problem increases.
Is Binary Search more efficient? Number of iterations:
For a list of n elements, Binary Search can execute at most log2 n times!!
Linear Search, on the other hand, can execute up to n times !!
Average Number of Iterations Length Linear Search Binary Search
10 5.5 2.9100 50.5 5.8
1,000 500.5 9.010,000 5000.5 12.0
Is Binary Search more efficient? Number of computations per iteration:
Binary search does more computations than Linear Search per iteration.
Overall: If the number of components is small (say, less
than 20), then Linear Search is faster. If the number of components is large, then
Binary Search is faster.
How do we analyze algorithms?
We need to define a number of objective measures.
(1) Compare execution times? Not good: times are specific to a particular computer !!
(2) Count the number of statements executed? Not good: number of statements vary with the
programming language as well as the style of the individual programmer.
Example (# of statements)
Algorithm 1 Algorithm 2 arr[0] = 0; for(i=0; i<N; i++)arr[1] = 0; arr[i] = 0;arr[2] = 0; ...arr[N-1] = 0;
How do we analyze algorithms?
(3) Express running time as a function of the input size n (i.e., f(n)).
To compare two algorithms with running times f(n) and g(n), we need a rough measure of how fast a function grows.
Such an analysis is independent of machine time, programming style, etc.
Computing running time
Associate a "cost" with each statement and find the "total cost“ by finding the total number of times each statement is executed.
Express running time in terms of the size of the problem.
Computing running time (contd.)
Algorithm 1 Algorithm 2 Cost Cost arr[0] = 0; c1 for(i=0; i<N; i++) c2 arr[1] = 0; c1 arr[i] = 0; c1 arr[2] = 0; c1 ... arr[N-1] = 0; c1 ----------- ------------- c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 = (c2 + c1) x N + c2
Computing running time (cont.)
Cost sum = 0; c1
for(i=0; i<N; i++) c2 for(j=0; j<N; j++) c2 sum += arr[i][j]; c3 ------------c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N x N
Comparing Functions UsingRate of Growth Consider the example of buying elephants and
goldfish:Cost: cost_of_elephants + cost_of_goldfishCost ~ cost_of_elephants (approximation)
The low order terms in a function are relatively insignificant for large n
n4 + 100n2 + 10n + 50 ~ n4
i.e., n4 + 100n2 + 10n + 50 and n4 have the same rate of growth
Rate of Growth ≡Asymptotic Analysis
Using rate of growth as a measure to compare different functions implies comparing them asymptotically.
If f(x) is faster growing than g(x), then f(x) always eventually becomes larger than g(x) in the limit (for large enough values of x).
Example Suppose you are designing a web site to process
user data (e.g., financial records). Suppose program A takes fA(n)=30n+8
microseconds to process any n records, while program B takes fB(n)=n2+1 microseconds to process the n records.
Which program would you choose, knowing you’ll want to support millions of users?
Visualizing Orders of Growth On a graph, as
you go to theright, a fastergrowingfunctioneventuallybecomeslarger...
fA(n)=30n+8
Increasing n
fB(n)=n2+1
Valu
e of
func
tion
Big-O Notation We say fA(n)=30n+8 is order n, or O(n).
It is, at most, roughly proportional to n. fB(n)=n2+1 is order n2, or O(n2). It is, at most,
roughly proportional to n2. In general, an O(n2) algorithm will be slower
than O(n) algorithm. Warning: an O(n2) function will grow faster
than an O(n) function.
More Examples …
We say that n4 + 100n2 + 10n + 50 is of the order of n4 or O(n4)
We say that 10n3 + 2n2 is O(n3) We say that n3 - n2 is O(n3) We say that 10 is O(1), We say that 1273 is O(1)
Big-O Visualization
Computing running time
Algorithm 1 Algorithm 2 Cost Cost arr[0] = 0; c1 for(i=0; i<N; i++) c2 arr[1] = 0; c1 arr[i] = 0; c1 arr[2] = 0; c1 ... arr[N-1] = 0; c1 ----------- ------------- c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 = (c2 + c1) x N + c2
O(n)
Computing running time (cont.)
Cost
sum = 0; c1 for(i=0; i<N; i++) c2
for(j=0; j<N; j++) c2
sum += arr[i][j]; c3 ------------
c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N x N
O(n2)
Running time of various statements
while-loop for-loop
i = 0;while (i<N) {
X=X+Y; // O(1)result = mystery(X); // O(N), just an example...i++; // O(1)
} The body of the while loop: O(N) Loop is executed: N times
N x O(N) = O(N2)
Examples
if (i<j)for ( i=0; i<N; i++ )
X = X+i;else
X=0;
Max ( O(N), O(1) ) = O (N)
O(N)
O(1)
Examples (cont.’d)
The Growth of Functions
Example:Show that f(x) = x2 + 2x + 1 is O(x2).
For x > 1 we have:x2 + 2x + 1 x2 + 2x2 + x2
x2 + 2x + 1 4x2
Therefore, for C = 4 and k = 1:f(x) Cx2 whenever x > k.
f(x) is O(x2).
The Growth of Functions
Question: If f(x) is O(x2), is it also O(x3)?
Yes. x3 grows faster than x2, so x3 grows also faster than f(x).
Therefore, we always have to find the smallest simple function g(x) for which f(x) is O(g(x)).
The Growth of Functions
“Popular” functions g(n) aren log n, 1, 2n, n2, n!, n, n3, log n
Listed from slowest to fastest growth: 1 log n n n log n n2
n3
2n
n!
Complexity Examples
What does the following algorithm compute?
procedure who_knows(a1, a2, …, an: integers)m := 0for i := 1 to n-1
for j := i + 1 to nif |ai – aj| > m then m := |ai – aj|
{m is the maximum difference between any two numbers in the input sequence}Comparisons: n-1 + n-2 + n-3 + … + 1 = (n – 1)n/2 = 0.5n2 – 0.5n
Time complexity is O(n2).
Complexity Examples
Another algorithm solving the same problem:
procedure max_diff(a1, a2, …, an: integers)min := a1max := a1for i := 2 to n
if ai < min then min := ai
else if ai > max then max := ai
m := max - minComparisons: 2n - 2
Time complexity is O(n).
Asymptotic Notation
O notation: asymptotic “less than”:
f(n)=O(g(n)) implies: f(n) “≤” g(n)
notation: asymptotic “greater than”:
f(n)= (g(n)) implies: f(n) “≥” g(n)
notation: asymptotic “equality”:
f(n)= (g(n)) implies: f(n) “=” g(n)
Definition: O(g), at most order g
Let f,g are functions RR. We say that “f is at most order g”, if: c,k: f(x) cg(x), x>k
“Beyond some point k, function f is at most a constant c times g (i.e., proportional to g).”
“f is at most order g”, or “f is O(g)”, or “f=O(g)” all just mean that fO(g).
Sometimes the phrase “at most” is omitted.
Big-O Visualization
k
Points about the definition Note that f is O(g) as long as any values of c
and k exist that satisfy the definition. But: The particular c, k, values that make
the statement true are not unique: Any larger value of c and/or k will also work.
You are not required to find the smallest c and k values that work. (Indeed, in some cases, there may be no smallest values!)
However, you should prove that the values you choose do work.
“Big-O” Proof Examples Show that 30n+8 is O(n).
Show c,k: 30n+8 cn, n>k . Let c=31, k=8. Assume n>k=8. Then
cn = 31n = 30n + n > 30n+8, so 30n+8 < cn. Show that n2+1 is O(n2).
Show c,k: n2+1 cn2, n>k: . Let c=2, k=1. Assume n>1. Then cn2 = 2n2 = n2+n2 > n2+1, or n2+1< cn2.
Note 30n+8 isn’tless than nanywhere (n>0).
It isn’t evenless than 31neverywhere.
But it is less than31n everywhere tothe right of n=8.
n>k=8
Big-O example, graphically
Increasing n
Valu
e of
func
tion
n
30n+8cn =31n
30n+8O(n)
Common orders of magnitude
Order-of-Growth in Expressions “O(f)” can be used as a term in an arithmetic
expression .E.g.: we can write “x2+x+1” as “x2+O(x)” meaning
“x2 plus some function that is O(x)”. Formally, you can think of any such expression as
denoting a set of functions: “x2+O(x)” : {g | fO(x): g(x)= x2+f(x)}
Useful Facts about Big O Constants ... c>0, O(cf)=O(f+c)=O(fc)=O(f) Sums:
- If gO(f) and hO(f), then g+hO(f). - If gO(f1) and hO(f2), then
g+hO(f1+f2) =O(max(f1,f2))
(Very useful!)
More Big-O facts Products:
If gO(f1) and hO(f2), then ghO(f1f2)
Big O, as a relation, is transitive: fO(g) gO(h) fO(h)
More Big O facts f,g & constants a,bR, with b0,
af = O(f) (e.g. 3x2 = O(x2)) f+O(f) = O(f) (e.g. x2+x = O(x2)) |f|1-b = O(f) (e.g. x1 = O(x)) (logb |f|)a = O(f) (e.g. log x = O(x)) g=O(fg) (e.g. x = O(x log x)) fg O(g) (e.g. x log x O(x)) a=O(f) (e.g. 3 = O(x))
Definition: (g), at least order gLet f,g be any function RR. We say that “f is at least order g”, written (g), if
c,k: f(x) cg(x), x>k “Beyond some point k, function f is at least a constant c
times g (i.e., proportional to g).” Often, one deals only with positive functions and can
ignore absolute value symbols. “f is at least order g”, or “f is (g)”, or “f= (g)” all
just mean that f (g).
Big- Visualization
Definition: (g), exactly order g If fO(g) and gO(f) then we say “g and f are
of the same order” or “f is (exactly) order g” and write f(g).
Another equivalent definition: c1c2,k: c1g(x)f(x)c2g(x), x>k
“Everywhere beyond some point k, f(x) lies in between two multiples of g(x).”
(g) O(g) (g) (i.e., fO(g) and f(g) )
Big- Visualization
Rules for Mostly like rules for O( ), except: f,g>0 & constants a,bR, with b>0,
af (f) Same as with O. f (fg) unless g=(1) Unlike O.|f| 1-b (f), and Unlike with O. (logb |f|)c (f). Unlike with O.
The functions in the latter two cases we say are strictly of lower order than (f).
example Determine whether: Quick solution:
)( 2?
1
nin
i
1
2
( 1) / 2
( ) / 2
( )
( )
n
i
i n n
n n
n n
n
Other Order-of-Growth Relations o(g) = {f | c k: f(x) < cg(x), x>k}
“The functions that are strictly lower order than g.” o(g) O(g) (g).
(g) = {f | c k: cg(x) < f(x), x>k }“The functions that are strictly higher order than g.” (g) (g) (g).
Subset relations between order-of-growth sets.
Relations Between the Relations
RR( f )O( f )
( f ) ( f )o( f )• f
Strict Ordering of Functions Temporarily let’s write fg to mean fo(g),
f~g to mean f(g) Note that
Let k>1. Then the following are true:1 log log n log n ~ logk n logk n n1/k n n log n nk kn n! nn …
.0)(
)(lim
xg
xfgf
x
Common orders of magnitude
Review: Orders of Growth
Definitions of order-of-growth sets, g:RR
O(g) {f | c,k: f(x) cg(x), x>k } o(g) {f | c k: f(x) < cg(x),x>k } (g) {f| | c,k : f(x) cg(x),x>k } (g) {f | c k: f(x) >cg(x), x>k} (g) {f | c1c2,k: c1g(x)f(x)|c2g(x), x>k}
Examples
Algorithmic Complexity The algorithmic complexity of a
computation is some measure of how difficult it is to perform the computation.
Measures some aspect of cost of computation (in a general sense of cost).
Time complexity Best-case time = minimum time needed to
execute the algorithm for inputs of size n Worst-case time = maximum time needed t
o execute the algorithm for inputs of size n Average-case time = average time needed
focus on best and worst time
Problem Complexity The complexity of a computational problem
or task is the complexity of the algorithm with the lowest order of growth of complexity for solving that problem or performing that task.
E.g. the problem of searching an ordered list has at most logarithmic time complexity. (Complexity is O(log n).)
Tractable vs. Intractable Problems A problem or algorithm with at most polynomial
time complexity is considered tractable (or feasible). P is the set of all tractable problems.
A problem or algorithm that has more than polynomial complexity is considered intractable (or infeasible).
Note n1,000,000 is technically tractable, but really impossible.
nlog log log n is technically intractable, but easy. Such cases are rare though.
Dealing with Intractable Problems Many times, a problem is intractable for a small
number of input cases that do not arise in practice very often.
Average running time is a better measure of problem complexity in this case.
Find approximate solutions instead of exact solutions.
Unsolvable problems It can be shown that there exist problems that no
algorithm exists for solving them. Turing discovered in the 1930’s that there are
problems unsolvable by any algorithm. Example: the halting problem (see page 176)
Given an arbitrary algorithm and its input, will that algorithm eventually halt, or will it continue forever in an “infinite loop?”
NP and NP-complete NP is the set of problems for which there exists a
tractable algorithm for checking solutions to see if they are correct.
NP-complete is a class of problems with the property that if any one of them can be solved by a polynomial worst-case algorithm, then all of them can be solved by polynomial worst-case algorithms.
Satisfiability problem: find an assignment of truth values that makes a compound proposition true.
P vs. NP We know PNP, but the most famous
unproven conjecture in computer science is that this inclusion is proper (i.e., that PNP rather than P=NP).
It is generally accepted that no NP-complete problem can be solved in polynomial time.
Whoever first proves it will be famous!
Questions Find the best big-O notation to describe the
complexity of following algorithms: A binary search of n elements A linear search to find the smallest number in a
list of n numbers An algorithm that lists all ways to put the
numbers 1,2,3,…,n in a row.
Questions (cont’d) An algorithm that prints all bit strings of length n An iterative algorithm to compute n! An algorithm that finds the average of n
numbers by adding them and dividing by n An algorithm that prints all subsets of size three
of the set {1,2,3,…,n} The best case analysis of a linear search of a list
of size n.
Questions (cont’d) The wors-case analysis of a linear search of a list
of size n The number of print statements in the following
while n>1 {print “hello”n=n/2
}
Questions (cont’d) The number of print statements in the following for (i=1, i≤n; i++)
for (j=1, j ≤n; j++) print “hello” The number of print statements in the following for (i=1, i≤n; i++)
for (j=1, j ≤i; j++) print “hello”
Introduction to Number Theory
Introduction to Number Theory
Number theory is about integers and their properties.
We will start with the basic principles of divisibility, greatest common divisors, least common multiples, and modular arithmetic
and look at some relevant algorithms.
Division
If a and b are integers with a 0, we say that a divides b if there is an integer c so that b = ac.
When a divides b we say that a is a factor of b and that b is a multiple of a.
The notation a | b means that a divides b.
We write a χ b when a does not divide b(see book for correct symbol).
Divisibility Theorems
For integers a, b, and c it is true that
if a | b and a | c, then a | (b + c) Example: 3 | 6 and 3 | 9, so 3 | 15.
if a | b, then a | bc for all integers c Example: 5 | 10, so 5 | 20, 5 | 30, 5 | 40, …
if a | b and b | c, then a | c Example: 4 | 8 and 8 | 24, so 4 | 24.
Primes
A positive integer p greater than 1 is called prime if the only positive factors of p are 1 and p.Note: 1 is not a prime
A positive integer that is greater than 1 and is not prime is called composite.
The fundamental theorem of arithmetic:Every positive integer can be written uniquely as the product of primes, where the prime factors are written in order of increasing size.
Primes
Examples:
3·53·5
48 =48 =
17 =17 =
100 =100 =
512 =512 =
515 =515 =
28 =28 =
15 =15 =
2·2·2·2·3 = 22·2·2·2·3 = 244·3·3
1717
2·2·5·5 = 22·2·5·5 = 222·5·522
2·2·2·2·2·2·2·2·2 = 22·2·2·2·2·2·2·2·2 = 299
5·1035·103
2·2·72·2·7
Primes
If n is a composite integer, then n has a prime divisor less than or equal .
This is easy to see: if n is a composite integer, it must have at least two prime divisors. Let the largest two be p1 and p2. Then p1p2 <= n.
p1 and p2 cannot both be greater than , because then p1p2 > n.
n
n
The Division Algorithm
Let a be an integer and d a positive integer.Then there are unique integers q and r, with 0 r < d, such that a = dq + r.
In the above equation, d is called the divisor, a is called the dividend, q is called the quotient, and r is called the remainder.
The Division Algorithm
Example:
When we divide 17 by 5, we have
17 = 53 + 2.
17 is the dividend, 5 is the divisor, 3 is called the quotient, and 2 is called the remainder.
The Division Algorithm
Another example:
What happens when we divide -11 by 3 ?
Note that the remainder cannot be negative.
-11 = 3(-4) + 1.
-11 is the dividend, 3 is the divisor, -4 is called the quotient, and 1 is called the remainder.
Greatest Common Divisors
Let a and b be integers, not both zero.The largest integer d such that d | a and d | b is called the greatest common divisor of a and b.The greatest common divisor of a and b is denoted by gcd(a, b).Example 1: What is gcd(48, 72) ?The positive common divisors of 48 and 72 are 1, 2, 3, 4, 6, 8, 12, 16, and 24, so gcd(48, 72) = 24. Example 2: What is gcd(19, 72) ?The only positive common divisor of 19 and 72 is1, so gcd(19, 72) = 1.
Greatest Common Divisors
Using prime factorizations:
a = p1a1 p2
a2 … pnan , b = p1
b1 p2b2 … pn
bn ,where p1 < p2 < … < pn and ai, bi N for 1 i n
gcd(a, b) = p1min(a1, b1 ) p2
min(a2, b2 ) … pnmin(an, bn )
Example:
a = 60 = a = 60 = 2222 3 311 5 511
b = 54 = b = 54 = 2211 3 333 5 500
gcd(a, b) = gcd(a, b) = 2211 3 311 5 50 0 = 6 = 6
Relatively Prime Integers
Definition:Two integers a and b are relatively prime if gcd(a, b) = 1.
Examples:Are 15 and 28 relatively prime?Yes, gcd(15, 28) = 1.Are 55 and 28 relatively prime?Yes, gcd(55, 28) = 1.Are 35 and 28 relatively prime?No, gcd(35, 28) = 7.
Relatively Prime Integers
Definition:
The integers a1, a2, …, an are pairwise relatively prime if gcd(ai, aj) = 1 whenever 1 i < j n.
Examples:Are 15, 17, and 27 pairwise relatively prime?No, because gcd(15, 27) = 3.Are 15, 17, and 28 pairwise relatively prime?Yes, because gcd(15, 17) = 1, gcd(15, 28) = 1 and gcd(17, 28) = 1.
Least Common Multiples
Definition:The least common multiple of the positive integers a and b is the smallest positive integer that is divisible by both a and b.We denote the least common multiple of a and b by lcm(a, b).Examples:
lcm(3, 7) =lcm(3, 7) = 2121
lcm(4, 6) =lcm(4, 6) = 1212
lcm(5, 10) =lcm(5, 10) = 1010
Least Common Multiples
Using prime factorizations:
a = p1a1 p2
a2 … pnan , b = p1
b1 p2b2 … pn
bn ,where p1 < p2 < … < pn and ai, bi N for 1 i n
lcm(a, b) = p1max(a1, b1 ) p2
max(a2, b2 ) … pnmax(an, bn )
Example:a = 60 = a = 60 = 2222 3 311 5 511
b = 54 = b = 54 = 2211 3 333 5 500
lcm(a, b) = lcm(a, b) = 2222 3 333 5 51 1 = 4 = 427275 = 5405 = 540
GCD and LCM
a = 60 = a = 60 = 2222 3 311 5 511
b = 54 = b = 54 = 2211 3 333 5 500
lcm(a, b) = lcm(a, b) = 2222 3 333 5 51 1 = 540 = 540
gcd(a, b) = gcd(a, b) = 2211 3 311 5 50 0 = 6 = 6
Theorem: aTheorem: ab =b = gcd(a,b)gcd(a,b)lcm(a,b)lcm(a,b)
Modular Arithmetic
Let a be an integer and m be a positive integer.We denote by a mod m the remainder when a is divided by m.
Examples:
9 mod 4 =9 mod 4 = 11
9 mod 3 =9 mod 3 = 00
9 mod 10 =9 mod 10 = 99
-13 mod 4 =-13 mod 4 = 33
Congruences
Let a and b be integers and m be a positive integer. We say that a is congruent to b modulo m if m divides a – b.
We use the notation a b (mod m) to indicate that a is congruent to b modulo m.
In other words:a b (mod m) if and only if a mod m = b mod m.
Congruences
Examples:Is it true that 46 68 (mod 11) ?Yes, because 11 | (46 – 68).Is it true that 46 68 (mod 22)?Yes, because 22 | (46 – 68).For which integers z is it true that z 12 (mod 10)?It is true for any z{…,-28, -18, -8, 2, 12, 22, 32, …}
Theorem: Let m be a positive integer. The integers a and b are congruent modulo m if and only if there is an integer k such that a = b + km.
Congruences
Theorem: Let m be a positive integer. If a b (mod m) and c d (mod m), then a + c b + d (mod m) and ac bd (mod m).Proof: We know that a b (mod m) and c d (mod m) implies that there are integers s and t with b = a + sm and d = c + tm. Therefore,b + d = (a + sm) + (c + tm) = (a + c) + m(s + t) andbd = (a + sm)(c + tm) = ac + m(at + cs + stm).Hence, a + c b + d (mod m) and ac bd (mod m).
Congruences
Theorem: Let m be a positive integer. a b (mod m) iff a mod m = b mod m.Proof: Let a = mq1 + r1, and b = mq2 + r2.Only if part: a mod m = b mod m r1 = r2, therefore a – b = m(q1 – q2), and a b (mod m).If part: a b (mod m) implies a – b = mq mq1 + r1 – (mq2 + r2) = mq r1 – r2 = m(q – q1 + q2).Since 0 r1, r2 m, 0 |r1 - r2| m. The only multiple in that range is 0. Therefore r1 = r2, and a mod m = b mod m.
The Euclidean Algorithm
The Euclidean Algorithm finds the greatest common divisor of two integers a and b. For example, if we want to find gcd(287, 91), we divide 287 by 91:287 = 913 + 14We know that for integers a, b and c,if a | b and a | c, then a | (b + c).Therefore, any divisor (including their gcd) of 287 and 91 must also be a divisor of 287 - 913 = 14.Consequently, gcd(287, 91) = gcd(14, 91).
The Euclidean Algorithm
In the next step, we divide 91 by 14:
91 = 146 + 7
This means that gcd(14, 91) = gcd(14, 7).
So we divide 14 by 7:
14 = 72 + 0
We find that 7 | 14, and thus gcd(14, 7) = 7.
Therefore, gcd(287, 91) = 7.
The Euclidean Algorithm
In pseudocode, the algorithm can be implemented as follows:
procedure gcd(a, b: positive integers)x := ay := bwhile y 0begin r := x mod y x := y y := rend {x is gcd(a, b)}
Representations of Integers
Let b be a positive integer greater than 1.Then if n is a positive integer, it can be expressed uniquely in the form:
n = akbk + ak-1bk-1 + … + a1b + a0,
where k is a nonnegative integer,a0, a1, …, ak are nonnegative integers less than b,and ak 0.
Example for b=10:859 = 8102 + 5101 + 9100
Representations of Integers
Example for b=2 (binary expansion):(10110)2 = 124 + 122 + 121 = (22)10
Example for b=16 (hexadecimal expansion):(we use letters A to F to indicate numbers 10 to 15)(3A0F)16 = 3163 + 10162 + 15160 = (14863)10
Representations of Integers
How can we construct the base b expansion of an integer n?First, divide n by b to obtain a quotient q0 and remainder a0, that is,n = bq0 + a0, where 0 a0 < b.The remainder a0 is the rightmost digit in the base b expansion of n.Next, divide q0 by b to obtain:q0 = bq1 + a1, where 0 a1 < b.a1 is the second digit from the right in the base b expansion of n. Continue this process until you obtain a quotient equal to zero.
Representations of Integers
Example: What is the base 8 expansion of (12345)10 ?
First, divide 12345 by 8:12345 = 81543 + 1
1543 = 8192 + 7192 = 824 + 024 = 83 + 03 = 80 + 3
The result is: (12345)10 = (30071)8.
Representations of Integers
procedure base_b_expansion(n, b: positive integers)q := nk := 0while q 0begin ak := q mod b q := q/b k := k + 1end {the base b expansion of n is (ak-1 … a1a0)b }
Addition of Integers
How do we (humans) add two integers?
Example: 7583 + 4932
5511552211
111111 carrycarry
Binary expansions:Binary expansions: (1011)(1011)22
+ + (1010)(1010)22
1100
carrycarry11
1100
11
11(( ))22
Addition of Integers
Let a = (an-1an-2…a1a0)2, b = (bn-1bn-2…b1b0)2.
How can we algorithmically add these two binary numbers?First, add their rightmost bits:a0 + b0 = c02 + s0,where s0 is the rightmost bit in the binary expansion of a + b, and c0 is the carry.Then, add the next pair of bits and the carry:a1 + b1 + c0 = c12 + s1,where s1 is the next bit in the binary expansion of a + b, and c1 is the carry.
Addition of Integers
Continue this process until you obtain cn-1.
The leading bit of the sum is sn = cn-1.
The result is:a + b = (snsn-1…s1s0)2
Addition of Integers
Example:Add a = (1110)2 and b = (1011)2.
a0 + b0 = 0 + 1 = 02 + 1, so that c0 = 0 and s0 = 1.
a1 + b1 + c0 = 1 + 1 + 0 = 12 + 0, so c1 = 1 and s1 = 0.
a2 + b2 + c1 = 1 + 0 + 1 = 12 + 0, so c2 = 1 and s2 = 0.
a3 + b3 + c2 = 1 + 1 + 1 = 12 + 1, so c3 = 1 and s3 = 1.
s4 = c3 = 1.
Therefore, s = a + b = (11001)2.
Addition of Integers
procedure add(a, b: positive integers)c := 0for j := 0 to n-1begin d := (aj + bj + c)/2 sj := aj + bj + c – 2d c := dendsn := c{the binary expansion of the sum is (snsn-1…s1s0)2}
Applications of Number Theory
GCD as linear combination
If a and b are positive integers, then there exist integers s and t such that gcd(a,b) = sa + tb
ExampleExpress gcd(252, 198) = 18 as a linear combination of 252 and 198
252 = 1*198 + 54198 = 3*54 + 3654 = 1*36 + 1836 = 2*18 Start from next-to-last division
18 = 54 – 1*3636 = 198 – 3*5418 = 54 – 198 + 3*54 = 4*54 – 1*19854 = 252 – 1*198
18 = 4*252 – 5*198
Hashing Function How to assign memory location that the
data can be retrieved quickly h(k) = k mod m, where k is key and m is the
number of available memory locations
Example m = 111, h(064212848) = ? h(064212848) = 14 So this data will be placed at memory
location = 14
Caesar’s Encryption Invented by Julius Caesar to make
a secret messageAlgorithm1. Convert each alphabet in a message to its corresponding
number p: A = 0, B = 1,…, Z = 25, and p 25 only2. Select key k as a positive integer, Caesar himself often use k = 33. To encrypt, applies this formula:
f(p) = (p+k) mod 264. Then convert each number back to alphabet and pack the
ciphertext with its key 5. To decrypt, do step 1 and 2 then applies this formula:
f-1(p) = (p-k) mod 266. If p+k or p-k is greater than or lesser than 25, shift it with 25
Example: Caesar’s Encryption Encrypt “DISCRETE MATH”
D I S C R E T E M A T H
3 8 18 2 17 4 19 4 12 0 19 7
f(p+k) mod 26
6 11 21 5 20 7 22 7 15 3 22 10
G L V F U H W H P D W K
GLVFUHWH PDWK
Linear Congruences
Congruence of a form ax b(mod m), where m is positive integer, a and b are integers, x is a variable is called Linear Congruence
To solve this congruence is to find all possible x that satisfies the formula
Inverse modulo More details in Section 3.7
Chinese Reminder Theorem
from San-Tsu’s puzzle, ancient Chinese mathematician
can be solved using Modular arithmetic More details in Section 3.7
RSA Public key crypto system Based on modular exponential To encrypt
C = Me mod n To decrypt
Cd M(mod pq) More details see RSA section and
Example 11 and 12 on page 241 - 244
Matrices
Matrices A matrix is a rectangular array of numbers. An mn (“m by n”) matrix has exactly m
horizontal rows, and n vertical columns. An nn matrix is called a square matrix,
whose order is n. The plural of matrix is matrices.
07
15
32
a 32 matrix
Example
Row: 3 rows Column: 2 columns Size: 3 x 2 matrix Square: NO
3
2
1
1
0
1
Example
Row: 2 rows Column: 2 columns Size: 2 x 2 matrix Square: YES
43
21
Example
Rows and columns? Size? Square? Equal?
02
11A
2
1
1
1
2
0
B
0
1
1
2
0
2C
02
10D
Matrix Equality Two matrices A and B are equal iff they
have the same number of rows, the same number of columns, and all corresponding elements are equal.
061
023
61
23
Row and Column Order The rows in a matrix are usually indexed 1
to m from top to bottom. The columns are usually indexed 1 to n from left to right. Elements are indexed by row, then column.
nmmm
n
n
ji
aaa
aaa
aaa
a
,2,1,
,22,21,2
,12,11,1
, ][
A
Let The ith row of A is the 1 x n matrix
The jth column of A is the n x 1 matrix
nn
2n
1n
n2
22
12
n1
21
11
a
.
.
.
a
a
...
...
...
a
.
.
.
a
a
a
.
.
.
a
a
A
ini2i1 a...aa
nj
2j
1j
a
.
.
.
a
aith row
jth column
The (i,j)th element or entry of A is the element aij, that is, the
number in the ith row and jth column of A.
A convenient shorthand notation for expressing the matrix A is
to write A = [aij], which indicates that A is the matrix with its
(i,j)th element equal to aij.
nn
2n
1n
n2
22
12
n1
21
11
a
.
.
.
a
a
...
...
...
a
.
.
.
a
a
a
.
.
.
a
a
A
Matrix Arithmetic
Addition/Sum Multiplication/Product
Matrix Sums The sum A+B of two matrices A, B (which
must have the same number of rows, and the same number of columns) is the matrix (also with the same shape) given by adding corresponding elements.
A+B = [ai,j+bi,j]
511
911
311
39
80
62
Example: Addition
Find A + B, where:
43
21A
12
34B
Example: Addition
Find A + B, where:
65
43
21
A
54
32
10
B
Example: Addition
Find A + B, where:
65
43
21
A
543
210B
Matrix Products For an mk matrix A and a kn matrix B, the product AB
is the mn matrix:
i.e., element (i,j) of AB is given by the vector dot product of the ith row of A and the jth column of B (considered as vectors).
In other words, if AB = [cij], then
cij = ai1b1j + ai2b2j + … + aikbkj. Note: Matrix multiplication is not commutative!
k
jiji bac1
,,, ][
CAB
Matrix Product Example An example matrix multiplication to
practice in class:
31123
1501
1301
0202
0110
302
110
Example: Multiplication
Find AB, where:
Find BA
43
21A
12
34B
Example: Multiplication
Find AB, where:
65
43
21
A
543
210B
Example: Multiplication
Find AB, where:
12A
543
210B
Example: Multiplication
Suppose A is a 6 × 8 matrix, B is an 8 × 5 matrix, and C is a 5 × 9 matrix.
Find the number of rows, the number of columns, and the number of entries in A(BC).
Example: Multiplication
Find A2 where,
43
21A
Identity Matrices The identity matrix of order n, In, is the
order-n matrix with 1’s along the upper-left to lower-right diagonal and 0’s everywhere else.
100
010
001
if 0
if 1
ji
jinI
Identity matrix
Multiplying a matrix by an
appropriate sized identity matrix
does not change this matrix. In
other words, when A is an m x n
matrix, we have
AIn = ImA = A
Example: Identity matrix
Find identity matrix of each of the following matrices:
43
21A
12
34B
332
212
321
C
543
210D
Matrix Inverses For some (but not all) square matrices A,
there exists a unique multiplicative inverse A-1 of A, a matrix such that A-1A = In.
If the inverse exists, it is unique, and A-1A = AA-1.
We won’t go into the algorithms for matrix inversion...
Matrix Multiplication Algorithm
procedure matmul(matrices A: mk, B: kn)for i := 1 to m
for j := 1 to n begincij := 0
for q := 1 to kcij := cij + aiqbqj
end {C=[cij] is the product of A and B}
What’s the of itstime complexity?
(m)·
(n)·(
(1)+
(k) ·
(1))
Answer:(mnk)
Powers of Matrices
If A is an nn square matrix and p0, then: Ap AAA···A (A0 In)
Example:
p times
23
34
12
23
01
12
01
12
01
12
01
12
01
123
If A=[aij] is an mn matrix, the transpose of A (often written At or AT) is the nm matrix given by At = B = [bij] = [aji] (1in,1jm)
Matrix Transposition
Flipacross
diagonal
23
11
02
210
312t
Example: Transpose
Find transpose of the following matrices:
43
21A
543
210D
65
43
21
B
Symmetric Matrices A square matrix A is symmetric iff A=At. I.e.,
i,jn: aij = aji . Which is symmetric?
211
120
103
213
101
312
11
11
11
Definition 7: Symmetric
A square matrix A is called symmetric if A
= At. Thus A = [aij] is symmetric if aij = aji
for all i and j with 1 ≤ i ≤ n and 1 ≤ j ≤ n.
Example: Symmetric
14
42
024
231
412
12
13
31
Zero-One Matrices All elements of a zero-one matrix are 0 or 1
Representing False & True respectively. Useful for representing other structures.
E.g., relations, directed graphs The meet of A, B (both mn zero-one matrices):
AB : [aijbij] = [aij bij] The join of A, B:
AB : [aijbij]
Boolean Products Let A=[aij] be an mk zero-one matrix,
& let B=[bij] be a kn zero-one matrix, The boolean product of A and B is like
normal matrix , but using instead + in the row-column “vector dot product.”
ji
k
ij bac 1
][CA⊙B
Arithmetic/Boolean Products
ji
k
ij bac 1
][CA⊙B
k
jiji bac1
,,, ][
CAB
Boolean Powers For a square zero-one matrix A, and any
k0, the kth Boolean power of A is simply the Boolean product of k copies of A.
A[k] A⊙A⊙… A⊙
k times
Homework 3 Chapter 3
Section 3.1: 1, 5, 22, 39, 42
Section 3.2: 1, 7, 9, 19, 21
Section 3.3: 1, 3, 9, 11, 25, 27
Section 3.4: 1, 9, 17, 31, 32
Section 3.5: 1, 21, 25
Section 3.6: 1, 5, 19, 23
Section 3.7: 7, 21
Section 3.8: 1, 3, 5, 31, 33
Supplementary: none