Lect14p4edegweewgwegweg

Topic and contents

School of Mathematics and Statistics

MATH1151 – Algebra

A/Prof Josef Dickbased on notes from A/Prof Rob Womersley✞✝ ☎✆Lecture 14 – Projections and Least Squares

1 Vector GeometryCross Product

Planes in R3

Distances

Matlab M-files

lssq1.m lssq2.m

MATH1151 (Algebra) L14 – Projections and Least Squares Session 1, 2015 1 / 12

Vector Geometry Projections

Projection – Revision

Definition (Projection onto a line)

O✏✏✏✏✏✏✏✏✏✏✏✏✏✏✏✶b

��✒

a

A

rPr

r❇❇❇❇❇◆

✏❇❇

λb

λb− a

The projection of the vector a onto the non-zero vector b is→OP where

→OP is parallel to b ⇐⇒

→OP= λb, λ ∈ R

→AP is perpendicular to b ⇐⇒

→AP ·b = 0 =⇒ (λb− a) · b = 0

Length of b 6= 0 does not change→OP

Projection of a onto bProjb a =

(a · b)(b · b) b



Projection onto Planes – 1

Plane Π = Span{b1,b2} = {x ∈ Rn : x = α1b1 + α2b2, α1, α2 ∈ R}

rO ✏✏✏✏✏✏✏✏✏✶

b1✁✁✁✁✁✁✕

b2

��✒a

❄

rArP

ProjΠ a =→OP= λ1b1 + λ2b2, λ1, λ2 ∈ R

→AP perpendicular to plane Π ⇐⇒

→AP perpendicular to b1, b2

b1 · (λ1b1 + λ2b2 − a) = 0

b2 · (λ1b1 + λ2b2 − a) = 0

Expand to get linear system

λ1(b1 · b1) + λ2(b1 · b2) = b1 · aλ1(b2 · b1) + λ2(b2 · b2) = b2 · a



Projection onto Planes – 2

Linear system Bx = d

x =

[λ1

λ2

], B =

[b1 · b1 b1 · b2b2 · b1 b2 · b2

], d =

[b1 · ab2 · a

]

B symmetric, positive diagonal elementsSolve for λ1, λ2

ProjΠ a = λ1b1 + λ2b2

Example

Find the projection of a = (6, −11, −4)T onto the plane Π spanned byb1 = (1, 2, 5)T and b2 = (−3, −1, 1)T .

Solution

B =

[30 00 11

], d =

[−36−11

]=⇒ x =

[−6

5−1

]

ProjΠ a = (95 , −75 , −7)T B diagonal – why?



Projection onto general span

Proposition

Projection of a ∈ Rm onto Π = span{bj , j = 1, . . . , n} ⊂ Rm is

ProjΠ a =n∑

j=1

λjbj

where x = (λ1, λ2, . . . , λn)T solves n× n linear system Bx = d,

B =

b1 · b1 b1 · b2 · · · b1 · bnb2 · b1 b2 · b2 · · · b2 · bn

...... · · · ...

bn · b1 bn · b2 · · · bn · bn

, d =

b1 · ab2 · a...

bn · a

{b1,b2, . . . ,bn } orthogonal ⇐⇒ bi ·bj = 0, i 6= j ⇐⇒ B diagonal

{b1,b2, . . . ,bn } orthonormal ⇐⇒ also bi · bi = 1 ⇐⇒ B = IProjΠ a =

∑nj=1(bj · a) bj


Vector Geometry Least squares problems

Least Squares Problems

Linear system Ax = b, A ∈ Mm,n(R), b ∈ Rm, x ∈ Rn

m > n =⇒ more equations than variables over-determinedGenerally don’t expect a solution

Residual r = Ax− b ∈ Rm

Ax = b ⇐⇒ r = 0 ⇐⇒ |r| = 0 ⇐⇒ |r|2 = 0 ⇐⇒ r · r = 0

Least Squares Problem Minimize |r|2 = ∑mj=1 r

2j

|r|2 = rT r = (Ax− b)T (Ax− b) = xTATAx− 2xTATb+ bTb

Normal equations (ATA

)x = ATb

ATA ∈ Mn,n(R), ATb ∈ Rn, square linear system

Projection of b onto span of columns of A

Projection: r = Ax− b orthogonal to columns of A ⇐⇒AT r = 0 ⇐⇒ AT (Ax− b) = 0 ⇐⇒ ATAx = ATb



Line of best fit – 1

Example (Line of best fit to data)

Find the line y(t) = α+ βt which best fits the data (ti, yi), i = 1, . . . ,min the least squares sense.

−2 0 2 4 6 8 10 121

2

3

4

5

6

7

8

9

t

y

Line 1.81 + 0.60 t of best fit to data, ||r||22 = 553.95

DataLine of best fit




Want line close to data points: α+ βti ≈ yi, i = 1, . . . ,m

n = 2 parameters α, β, m > 2 data values

Linear system Ax = y where

A =

1 t11 t2...

...1 tm

∈ Mm,2(R), x =

[αβ

], y =

y1y2...ym

∈ Rm

Normal Equations ATAx = AT y

ATA =

[1 1 · · · 1t1 t2 · · · tm

]

1 t11 t2...

...1 tm

=

mm∑

i=1

ti

m∑

i=1

ti

m∑

i=1

t2i




Normal Equations ATAx = AT y

AT y =

[1 1 · · · 1t1 t2 · · · tm

]

y1y2...ym

=

m∑

i=1

yi

m∑

i=1

tiyi

Solve 2 by 2 linear system

mm∑

i=1

ti

m∑

i=1

ti

m∑

i=1

t2i

[αβ

]=

m∑

i=1

yi

m∑

i=1

tiyi

Matlab x = A \ y, x = (A’*A) \ (A’*y), lssq1.m



Exponential fit – 1

Example (MATH1151 past exam)

Total cost T of claims in million $ is

Year t 0 1 2 3Cost T 2.0 2.2 2.5 2.8

Model by exponential growth, with parameters α, k ∈ R,

T = αekt

Use a log transformation to solve this as a linear least squares problem.

Solution

Log transformation

log(T ) = log(αekt) = log(α) + log(ekt) = log(α) + kt

Linear in variables x = (log(α), k)T




Solution

Linear system Ax = y where

A =

1 01 11 21 3

, y =

log(2.0)log(2.2)log(2.5)log(2.8)

Normal equations ATAx = AT y

ATA =

[4 66 14

], AT y =

[log(2.0)+log(2.2)+log(2.5)+log(2.8)

log(2.2)+2 log(2.5)+3 log(2.8)

]

Solution x = (ATA)−1(AT y)

x =1

20

[14 −6−6 4

](AT y) =

[0.68630.1137

]=⇒ α = ex1 = 1.9863

k = x2 = 0.1137




Matlab lssq2.m

0 0.5 1 1.5 2 2.5 31.8

1.9

2

2.1

2.2

2.3

2.4

2.5

2.6

2.7

2.8

Time t (years)

Cla

ims

T (

$mill

ion)

Cost of claims T(t) = 1.99 e0.11 t

DataExponential fit


Lect14p4edegweewgwegweg

Documents

Transcript of Lect14p4edegweewgwegweg