Chap. 14: General LinearLeast-Squares and

Nonlinear Regression

2

Polynomial Regression

• The least-squares procedure can be applied to higher-order polynomials.

• (a): first order polynomial fit.

• (b): second order polynomial fit.

3

Process of Polynomial Regression

• Least square error of second order polynomial:

• In general, for m-th order polynomial:

Sr = ei2

i=1

n

∑ = yi − a0 − a1xi − a2xi2( )2

i=1

n

∑

Sr = ei

2

i=1

n

∑ = yi − a0 − a1xi − a2xi2 −L− am xi

m( )2

i=1

n

∑

4

Process of Polynomial Regression (cont.)

• Minimizing the least square error

MSr

Ma0'&2j (yi&a0&a1xi&a2x

2i )'0

MSr

Ma1'&2j xi(yi&a0&a1xi&a2x

2i )'0

MSr

Ma2'&2j x 2

i (yi&a0&a1xi&a2x2i )'0

MSr

Ma0'&2j (yi&a0&a1xi&a2x

2i )'0

MSr

Ma1'&2j xi(yi&a0&a1xi&a2x

2i )'0

MSr

Ma2'&2j x 2

i (yi&a0&a1xi&a2x2i )'0

5

Process of Polynomial Regression (cont.)

A linear equation of a0, a1 and a2. Solvable.

Y

na0 % (j xi)a1 % (j x 2i )a2 ' j yi

(j xi)a0 % (j x 2i )a1 % (j x 3

i )a2 ' j xiyi

(j x2i )a0 % (j x 3

i )a1 % (j x 4i )a2 ' j x 2

i yi

Y

na0 % (j xi)a1 % (j x 2i )a2 ' j yi

(j xi)a0 % (j x 2i )a1 % (j x 3

i )a2 ' j xiyi

(j x2i )a0 % (j x 3

i )a1 % (j x 4i )a2 ' j x 2

i yi

6

Error and Measures of Fit

• The standard error for fitting an m-th order polynomial to n data points is:

because the m-th order polynomial has (m+1) coefficients.

• The coefficient of determination r2 is still found using:

sy / x =Sr

n − m +1( )

r2 =St − Sr

St

7

Example 14.1

y'2.4786%2.3593x%1.8607x 2

sy/x'1.1175r 2'0.99851

y'2.4786%2.3593x%1.8607x 2

sy/x'1.1175r 2'0.99851

y'&3.7238%11.663xsy/x'5.1576r 2'0.94708

y'&3.7238%11.663xsy/x'5.1576r 2'0.94708

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-10

0

10

20

30

40

50

60

70

x

y

8

Multiple Linear Regression

• Let y is a linear function of two or more independent variables:

• Minimizing the sum of the squares of the estimate residuals:

y = a0 + a1x1 + a2x2 +Lamxm

Sr = ei

2

i=1

n

∑ = yi − a0 − a1x1,i − a2x2,i −Lam xm,i( )2

i=1

n

∑

9

Multiple Linear Regression (cont.)

MSr

Ma0'&2j (yi&a0&a1x1,i&a2x2,i)'0

MSr

Ma1'&2j x1,i(yi&a0&a1x1,i&a2x2,i)'0

MSr

Ma2'&2j x2,i(yi&a0&a1x1,i&a2x2,i)'0

MSr

Ma0'&2j (yi&a0&a1x1,i&a2x2,i)'0

MSr

Ma1'&2j x1,i(yi&a0&a1x1,i&a2x2,i)'0

MSr

Ma2'&2j x2,i(yi&a0&a1x1,i&a2x2,i)'0

10

Multiple Linear Regression (cont.)

Y

na0 % (j x1,i)a1 % (j x2,i)a2 ' j yi

(j x1,i)a0 % (j x 21,i)a1 % (j x1,ix2,i)a2 ' j x1,iyi

(j x2,i)a0 % (j x1,ix2,i)a1 % (j x 22,i)a2 ' j x2,iyi

Y

na0 % (j x1,i)a1 % (j x2,i)a2 ' j yi

(j x1,i)a0 % (j x 21,i)a1 % (j x1,ix2,i)a2 ' j x1,iyi

(j x2,i)a0 % (j x1,ix2,i)a1 % (j x 22,i)a2 ' j x2,iyi

11

General Linear Least Squares

• General linear least-squares model:

where z0, z1, …, zm are a set of m+1 basis functions and e is the error of the fit.

y = a0z0 + a1z1 + a2z2 +Lamzm + e

12

Solving General Linear Least Squares Coefficients

• Evaluate the equation at data point i

where zij=zj(xi)• Recast into matrix form

y{ }= Z[ ] a{ }+ e{ }

yi'a0z10%a1z11%a2z12%@@@%amz1m%eiyi'a0z10%a1z11%a2z12%@@@%amz1m%ei

13

Solving General Linear Least Squares Coefficients (cont.)

Where

[Z]'

z10 z11 þ z1m

z20 z21 þ z2m

! ! " !

zn0 zn1 þ znm

[Z]'

z10 z11 þ z1m

z20 z21 þ z2m

! ! " !

zn0 zn1 þ znm

{y}'

y1

y2

!

yn

, {a}'

a0

a1

!

am

, {e}'

e1

e2

!

en

,{y}'

y1

y2

!

yn

, {a}'

a0

a1

!

am

, {e}'

e1

e2

!

en

,

14

Solving General Linear Least Squares Coefficients

• Generally, [Z] is not a square matrix (m!=n).• Simple inversion cannot be used to solve for

{a}.• Instead, minimize the sum of the squares of

the estimate residuals:

• The resulting equation is:

Sr = ei2

i=1

n

∑ = yi − ajz jij=0

m

∑⎛

⎝ ⎜ ⎜

⎞

⎠ ⎟ ⎟

2

i=1

n

∑

Z[ ]T Z[ ][ ] a{ }= Z[ ]T y{ }{ }

15

Example 14.3

• Given x and y data in columns, solve for the coefficients of the best fit line for y=a0+a1x+a2x2

Z = [ones(size(x)) x x.^2]a = (Z’*Z)\(Z’*y)

– Note also that MATLAB’s left-divide will automatically include the [Z]T terms if the matrix is not square, soa = Z\y

would work as well• To calculate measures of fit:

St = sum((y-mean(y)).^2)Sr = sum((y-Z*a).^2)r2 = 1-Sr/Stsyx = sqrt(Sr/(length(x)-length(a)))

16

Example 14.4

• Repeat example 14.3 but use polyfitfunction.

p = polyfit(x,y,n)% x, y: the data% n: the order of polynomial% p: the coefficients of the polynomial% with higher order first.

17

Nonlinear Regression

• Not all fits are linear equations of coefficients and basis functions.

• Two methods– variable transformation

• Not all equations can be transformed.• The best fit line represents the best fit for the

transformed variables, not the original variables.

– nonlinear regression

18

Nonlinear Regression in MATLAB

• Write a function that returns the sum of the squares of the estimate residuals.

• Use MATLAB fminsearch function to find the values of the coefficients where a minimum residuals occurs.x=fminsearch(fun,x0,options,p1,p2,...)% x0: the initial guess of the coef.

19

Example 14.4

• Re-work example 13.4 using nonlinear regression.y=αxβ

• Write a function called fSSR.m to be minimized as following:

function f = fSSR(a, x, y)yp = a(1)*x.^a(2);f = sum((y-yp).^2);

• Then, use fminsearch in the command window to obtain the values of a that minimize fSSR:

a = fminsearch(@fSSR, [1, 1], [], v, F)where [1, 1] is an initial guess for the [α, β] vector, [] is a placeholder for the options.

20

Nonlinear Regression Results

• Linear regressiony=0.2741x1.9843

sy/x=240.03r2=0.80883

• Nonlinear regressiony=2.5384x1.4359

sy/x=192.62r2=0.8769

21

Case Study: Fitting Sinusoids

• To fit a periodic data to sinusoidal funtions:f(t)=A0 +C1cos(ω0t+θ)where A0 , C1 and θ are unknowns.

• Nonlinear. Recast to linear form:f(t)=A0 +A1cos(ω0t) +B1sin(ω0t)

22

Case Study: Fitting Sinusoids (cont.)

• Use general linear regression model.[x]=([1 2 3 4 5 6 8 9 10 11 12]-0.5)*30;

[y]=[18.9 21.1 23.3 27.8 32.2 37.2 36.1 34.4 29.4 23.3 18.9];w0=2*pi/360;Z=[ones(size(x)) cos(w0*t) sin(w0*t)];a=(Z’*Z)\(Z’*x);

23

Case Study: Fitting Sinusoids (cont.)

sy/x=0.8972r2=0.9862Sr=6.4398