CHAPTER 1. INTRODUCTION

Differentiation (I)

• Product rule(fg)′ = f ′g + g ′f .

so(fgh)′ = f ′(gh) + (gh)′f = f ′gh + g ′hf + h′gf .

Example

d

dx(x − x1)(x − x2)(x − x3) = (x − x2)(x − x3) + (x − x1)(x − x3)

+(x − x1)(x − x2).

Differentiation (II)

n∏i=1

(x − xi ) = (x − x1)(x − x2) . . . (x − xn).

Then

d

dx

n∏i=1

(x − xi ) = (x − x2)(x − x3) . . . (x − xn) +

(x − x1)(x − x3) . . . (x − xn) + . . .+

(x − x1)(x − x2) . . . (x − xi−1)(x − xi+1) . . . (x − xn) + . . .

+(x − x1)(x − x2) . . . (x − xn−1).

O notation (I)

Let h be a quantity. Then we say that a function p(h) is O(h) ifthere is a constant C independent of h such that:

|p| ≤ C |h|.

Examples: 2h = O(h).

Let n be a positive integer. Then

2

n= O(

1

n), and

n + 1

n2= O(

1

n)

because ∣∣∣n + 1

n2

∣∣∣ =1

n+

1

n2≤ 2

n.

O notation (II)

If |h| < 1 then |h2| < |h|, |h3| < |h2|, ....Then

|ah + bh2| ≤ |ah|+ |bh2| ≤ (|a|+ |b|)|h|,

so ah + bh2 is O(h).

Taylor expansion (I)

f (x) = f (x0)+f ′(x0)(x−x0)+f ′′(x0)

2!(x−x0)2+. . .+

f (n)(x0)

n!(x−x0)n+. . . .

The finite form

f (x) = f (x0) + f ′(x0)(x − x0) +f ′′(x0)

2!(x − x0)2 + . . .+

f (n−1)(x0)

(n − 1)!(x − x0)n−1 +

f (n)(ξ)

n!(x − x0)n,

where ξ is between x and x0: i.e. if x < x0 then x ≤ ξ ≤ x0; ifx0 < x then x0 ≤ ξ ≤ x .

Taylor expansion (II)

Taylor expansion converges if |x − x0| < a for a fixed a; a is theradius of convergence. Sometimes a can be infinity.Example:

ex = 1 + x +x2

2!+

x3

3!+ . . . .

This holds for all x ; here x0 = 0 and a =∞.

ln(1 + x) = x − x2

2+

x3

3− . . . ;

this holds for |x | < 1 i.e. a = 1.There are tests to determine the radius of convergence, but this isbeyond the scope of this course.

Taylor expansion (III)

When |x − x0| < 1, then

f (x) = f (x0) + O(x − x0)

= f (x0) + f ′(x0)(x − x0) + O((x − x0)2)

= f (x0) + f ′(x0)(x − x0) +f ′′(x0)

2(x − x0)2 + O((x − x0)3).

We therefore can take

f (x0), f (x0)+f ′(x0)(x−x0), f (x0)+f ′(x0)(x−x0)+(f ′′(x0)/2)(x−x0)2, . . .

as approximations for f (x).

Integration

Definition of integration:

Divide an interval [a, b] into n parts bya = x0 < x1 < . . . < xn−1 < xn = b.Pick up a point ξi in (xi−1, xi ).Then ∫ b

af (x)dx = lim

max |xi−xi−1|→0

n∑i=1

f (ξi )(xi − xi−1).

About numerical analysis (I)

• Example: ∫ 1

0exdx = ex

∣∣∣10

= e − 1 = 1.718281828.

But for ∫ 1

0ex

2dx ,

we cannot find it exactly. We will need to find approximations.

Numerical analysis uses mathematical analysis to approximatequantities which cannot be, or are difficult to be, computed exactly.

About numerical analysis (II)

Coming back to the example: from the definition of integrals, if wedivide [0, 1] into 1 interval (itself) i.e. x0 = 0, x1 = 1. Pick upξ1 = 1/2 then ∫ 1

0ex

2dx ≈ e1/4 = 1.284025.

If we divide [0, 1] into 2 intervals, i.e. x0 = 0, x1 = 1/2 andx2 = 1. Pick up ξ1 = 1/4 and ξ2 = 3/4, then∫ 1

0ex

2dx ≈ 1

2(e1/16 + e9/16) = 1.40977.

Dividing [0, 1] into three intervals [0, 1/3], [1/3, 2/3] and [2/3, 1],and pick up ξ1 = 1/6, ξ2 = 1/2 and ξ3 = 5/6, then∫ 1

0ex

2dx ≈ 1.43826

About numerical analysis (III)

The more points we take, the better. To illustrate this, consideranother simple example∫ 1

0x2dx = 1/3 = 0.333333.

Dividing [0, 1] into one interval, and take ξ1 = 1/2, then∫ 1

0x2dx ≈ (

1

2)2 = 0.25.

Dividing [0, 1] into two intervals as above, then∫ 1

0x2dx ≈ 1

2

((

1

4)2 + (

3

4)2)

= 0.3125.

Dividing [0, 1] into three intervals as above, then∫ 1

0x2dx ≈ 0.324074.

About numerical analysis (IV)

We do not need to choose xi so that the length of (xi−1, xi ), i.e.|xi − xi−1|, is exactly the same for all i .

Later on, we will see that it is better that these lengths are notequal, and we have special ways to choose ξi .

In this course, we will learn how to approximate roots of anequation, approximate the values of a function with limitedinformation, approximate derivatives and integrals, etc.

Error (I)

If p̄ is an approximation for p, then the:Absolute error is defined as |p − p̄|.

Relative error is defined as |p − p̄|/|p|.

Example: for the integral∫ 10 x2dx = 1/3. For the approximated

value 0.3125, the error is

|1/3− 0.3125| = 0.020833,

and the relative error is

|1/3− 0.3125|/(1/3) = 0.0625.

Error (II)

In practice, we do not know the exact quantity p (otherwise thereis no need to find an approximation). We can only estimate theerror, though this task may be difficult.

For example, from Taylor expansion:

f (x + h) = f (x) + f ′(x)h + O(h2),

if we only know f (x) and f ′(x) and nothing else, we can say thatf (x) + f ′(x)h approximates f (x + h) with the error

e = f (x + h)− [f (x) + f ′(x)h],

which is bounded by Ch2 when h→ 0, for a constant C .

Why relative error

How can we say which approximation is better?

If p = 100, if p̄1 and p̄2 are approximations with errors 1 and 2respectively, then obviously p̄1 is better.

If q = 1 and if q̄ is an approximation with error 0.1, then how canwe compare p̄1 and q̄?

We need to compare the error to the size of the exact quantity,that is the relative error.For p̄1, the relative error is 1/100 = 0.01, i.e. the error is 1%. Forq̄, the relative error is 0.1/1 = 0.1, i.e. the error is 10%. So theapproximation p̄1 for p is better than q̄ for q.

Computer rounding off errors

We can only work with a finite number of digits.There are two methods to take a finite number of digits: choppingand rounding off.

Consider π = 3.14159265.... We want to take 5 digits.

For chopping, we take 3.1415.For rounding off, we take 3.1416 as the next digit is 9, which islarge.

In any case, chopping and rounding off always introduce an error,which we call (for both cases) rounding off errors.

Floating point

• Floating point numbers of k digits:

±0.d1d2 . . . dk × 10n, 1 ≤ d1 ≤ 9, 0 ≤ di ≤ 9 for i > 1.

For example: 3.1415 = 0.31415× 10, 3.1416 = 0.31416× 10.

Given a number y = 0.d1d2 . . . dkdk+1 . . .× 10n.

Floating point number: for chopping we just take the first k digits:

0.d1d2 . . . dk × 10n.

For rounding off: if dk+1 < 5 then we do exactly as chopping, ifdk+1 ≥ 5, we add 1 into dk .

Loss of accuracy due to rounding off

• Solve ax2 + bx + c = 0, a = 1, b = 62.1, c = 1.The exact roots are

x1 =−b +

√b2 − 4ac

2a, x2 =

−b −√b2 − 4ac

2a,

i.e. x1 = −0.01610723, x2 = −62.08390.

• Taking 4 digits:√b2 − 4ac =

√3852 = 62.06.

so

x1 =−62.1 + 62.06

2= −0.0200.

We have lost some accuracy.If we take 6 digits, i.e.

√b2 − 4ac =

√3852.41 = 62.0678, then

x1 = (−62.1 + 62.0678)/2 = −0.016100.

Reformulate to avoid loss of accuracy due to rounding off

Sometimes, we can reformulate the problem to get better accuracy.

We can rewrite x1 as (by multiplying both the denominator andnumerator by b +

√b2 − 4ac)

x1 =−2c

b +√b2 − 4ac

.

Then if we only take 4 digits:√b2 − 4ac =

√3852 = 62.06, then

x1 =−2.000

62.10 + 62.06= −0.01611.

Algorithm

• An algorithm is a procedure describing steps to be performed.• Example: computing the sum x1 + x2 + . . .+ xN .

• Algorithm:

- Take the first number x1 and add x2- Note the sum- Add x3 to the sum- Note the new sum- Continue until xN .

Pseudocode

• We describe this algorithm by a pseudocode

Step 1: INPUT N, x1, x2,. . . , xN.Step 2: Set SUM = 0.Step 3: For i = 1, 2, . . . ,N

Set SUM = SUM + xi.Step 4: OUTPUT SUM.

We can implement this in a computer language such as Fortran,C++, Matlab, ...

Rate of convergence (I)

• Let {βn} be a sequence that converges to 0, and {αn} be asequence that converges to α. If there is a constant K such that

|αn − α| ≤ K |βn|,

then we say that αn converges to α with the rate of convergenceO(βn).

• Normally, βn is of the form

βn =1

np.

Rate of convergence (II)

Consider

αn =n2 + n + 1

n2.

Then αn → 1 as n→∞.

|αn − 1| =n + 1

n2=

1

n+

1

n2≤ 2

n,

so |αn − 1| = O(1/n) – the rate of convergence is O(1/n).

Consider

α̂n =n3 + n + 1

n3.

Then

|α̂n − 1| =n + 1

n3=

1

n2+

1

n3≤ 2

n2.

The rate of convergence is O(1/n2).

Rate of convergence (III)

For functions: Suppose that limh→0 G (h) = 0 andlimh→0 F (h) = L. If there is a positive constant K such that

|F (h)− L| ≤ K |G (h)|,

then we say that F (h) converges to L with the rate O(G (h)).

For example: From Taylor expansion:

f (h) = f (0) + f ′(0)h +f ′′(0)

2h2 +

f ′′′(ξ)

6h3.

Assume that |f ′′′(ξ)| ≤ K for all ξ. Then

|f (h)− [f (0) + f ′(0)h +f ′′(0)

2h2]| ≤ K

6|h3|.

so f (h)− f ′(0)h − (f ′′(0)/2)h2 converges to f (0) with the rateO(h3).