Review of Linear Algebra

Dr. Chang Shu

COMP 4900C

Winter 2008

Linear Equations

A system of linear equations, e.g.

1114

242

21

21

=+

=+

xx

xx

can be written in matrix form:

=

1

2

114

42

2

1

x

x

or in general:

bAx =

Vectors

=

nx

x

x

xM

2

1

=

5

3

2

xe.g.

The length or the norm of a vector is

22

2

2

1 nxxxx +++= L

38532 222 =++=xe.g.

Vector Arithmetic

+

+=

+

=+

22

11

2

1

2

1

vu

vu

v

v

u

uvu

−

−=

−

=−

22

11

2

1

2

1

vu

vu

v

v

u

uvu

Vector addition

Vector subtraction

Multiplication by scalar

=

=

2

1

2

1

u

u

u

uu

α

ααα

Dot Product (inner product)

=

5

3

2

a

−=

2

3

4

b

[ ] 925)3(342

2

3

4

532 =⋅+−⋅+⋅=

−==⋅ babaT

nn

Tbababababa +++==⋅ L2211

Linear Independence

• A set of vectors is linear dependant if one of

the vectors can be expressed as a linear

combination of the other vectors.

• A set of vectors is linearly independent if

none of the vectors can be expressed as a

linear combination of the other vectors.

nnkkkkk vvvvv αααα +++++= ++−− LL 111111

Vectors and Points

Two points in a Cartesian coordinate system define a vector

P(x1,y1)

Q(x2,y2)

x

yv

−

−=

12

12

yy

xxv

P(x1,y1)

Q(x2,y2)

x

yv

A point can also be represented as a vector, defined by the point

and the origin (0,0).

PQv −=

=

−

=

1

1

1

1

0

0

y

x

y

xP

=

−

=

2

2

2

2

0

0

y

x

y

xQ

or vPQ +=

Note: point and vector are different; vectors do not have positions

Matrix

A matrix is an m×n array of numbers.

[ ]ij

mnmm

n

n

a

aaa

aaa

aaa

A =

=

L

MMM

L

L

21

22221

11211

Example:

−=

111070

9314

4532

A

Matrix Arithmetic

Matrix addition

[ ]nmijijnmnm baBA

××× +=+

Matrix multiplication

pmpnnmCBA ××× = ∑

=

=n

k

kjikijbac

1

Matrix transpose

[ ]ji

TaA =

( ) TTTBABA +=+ ( ) TTT

ABAB =

BAAB ≠

Matrix multiplication is not commutative

Example:

=

1119

2917

51

26

13

52

=

1017

3218

13

52

51

26

Symmetric Matrix

We say matrix A is symmetric if

AAT =

Example:

=

54

42A

A symmetric matrix has to be a square matrix

Inverse of matrix

If A is a square matrix, the inverse of A, written A-1 satisfies:

IAA =−1 IAA =−1

Where I, the identity matrix, is a diagonal matrix with all 1’s

on the diagonal.

=

10

012I

=

100

010

001

3I

Trace of Matrix

The trace of a matrix:

∑=

=n

i

iiaATr

1

)(

Orthogonal Matrix

A matrix A is orthogonal if

IAAT = 1−= AA

Tor

Example:

−=

θθ

θθ

cossin

sincosA

Matrix Transformation

A matrix-vector multiplication transforms one vector to another

11 ××× =mnnm

bxA

=

26

16

41

7

3

24

13

52

Example:

Coordinate Rotation

−=

y

x

y

x

r

r

r

r

φφ

φφ

cossin

sincos

'

'

Eigenvalue and Eigenvector

We say that x is an eigenvector of a square matrix A if

xAx λ=

λ is called eigenvalue and is called eigenvector.

The transformation defined by A changes only the

magnitude of the vector x

Example:

x

=

=

1

15

5

5

1

1

41

23

−=

−=

−

1

22

2

4

1

2

41

23and

5 and 2 are eigenvalues, and

1

1

−1

2and are eigenvectors.

Properties of Eigen Vectors

• If λ1, λ2,…, λq are distinct eigenvalues of a

matrix, then the corresponding eigenvectors

e1,e2,…,eq are linearly independent.

• A real, symmetric matrix has real eigenvalues

with eigenvectors that can be chosen to be

orthonormal.

Least Square

When m>n for an m-by-n matrix A, bAx = has no solution.

In this case, we look for an approximate solution.

We look for vector such that x

2bAx −

is as small as possible.

This is the least square solution.

Least Square

Least square solution of linear system of equations

bAx =

bAAxATT =

AAT is square and symmetric

bAAAxTT 1)( −=

Normal equation:

2bxA −

The Least Square solution

makes minimal.

Least Square Fitting of a Line

x

y

mmydxc

ydxc

ydxc

=+

=+

=+

M

22

11

=

mm y

y

y

d

c

x

x

x

MMM

2

1

2

1

1

1

1

The best solution c, d is the one that minimizes:

.)()( 22

11

22

mm dxcydxcyAxyE −−++−−=−= L

Line equations:

yAx =

Least Square Fitting - Example

x

y

P1=(-1,1), P2=(1,1), P3=(2,3)

Problem: find the line that best fit these three points:

Solution:

32

1

1

=+

=+

=−

dc

dc

dc

=

−

3

1

1

21

11

11

d

c

bAAxATT =

=

6

5

62

23

d

c

or

is

The solution is and best line is74

79 , == dc yx =+ 7

479

SVD: Singular Value Decomposition

TUDVA =

An m×n matrix A can be decomposed into:

U is m×m, V is n×n, both of them have orthogonal columns:

IUU T =

D is an m×n diagonal matrix.

IVV T =

Example:

−=

−10

01

00

30

02

100

010

001

00

30

02