Final Versiongauss and the Method of Least Squares

8/8/2019 Final Versiongauss and the Method of Least Squares

1/25

1

Gauss and the Method of Least Squares

Teddy Petrou Hongxiao Zhu


2/25

2

Outline

Who was Gauss?

Why was there controversy in finding the method of leastsquares?

Gauss treatment of error

Gauss derivation of the method of least squares

Gauss derivation by modern matrix notation

Gauss-Markov theorem Limitations of the method of least squares

References


3/25

3

Johann Carl Friedrich Gauss

Born:1777 Brunswick, Germany

Died: February 23, 1855, Gttingen, Germany

By the age of eight during arithmetic class heastonished his teachers by being able to

instantly find the sum of the first hundredintegers.


4/25

4

Facts about Gauss Attended Brunswick College in 1792, where he

discovered many important theorems before even

reaching them in his studies Found a square root in two different ways to fifty

decimal places by ingenious expansions andinterpolations

Constructed a regular 17 sided polygon, the firstadvance in this matter in two millennia. He was only18 when he made the discovery


5/25

5

Ideas of Gauss

Gauss was a mathematical scientist with interests in so manyareas as a young man including theory of numbers, to algebra,analysis, geometry, probability, and the theory of errors.

His interests grew, including observational astronomy, celestialmechanics, surveying, geodesy, capillarity, geomagnetism,electromagnetism, mechanism optics, and actuarial science.


6/25

6

Intellectual Personality and Controversy

Those who knew Gauss best found him to be cold and

uncommunicative.

He only published half of his ideas and found no one to sharehis most valued thoughts.

In 1805 Adrien-Marie Legendre published a paper on themethod of least squares. His treatment, however, lacked aformal consideration of probability and its relationship to leastsquares, making it impossible to determine the accuracy of themethod when applied to real observations.

Gauss claimed that he had written colleagues concerning theuse of least squares dating back to 1795


7/25

7

Formal Arrival of Least Squares

Gauss

Published The theory of the Motion of Heavenly Bodies in 1809.He gave a probabilistic justification of the method,which wasbased on the assumption of a normal distribution of errors.Gauss himself later abandoned the use of normal error function.

Published Theory of the Combination of Observations LeastSubject to Errors in 1820s. He substituted the root mean squareerror for Laplaces mean absolute error.

Laplace Derived the method of least squares (between1802 and1820) from the principle that the best estimate should have thesmallestmean error -the mean of the absolute value of the error.


8/25

8

Treatment of Errors

Using probability theory to describe error

Error will be treated as a random variable

Two types of errors

Constant-associated with calibration

Random error


9/25

9

Error Assumptions

Gauss began his study by making two assumptions

Random errors of measurements of the same type lie withinfixed limits

All errors within these limits are possible, but not necessarily

with equal likelihood


10/25

10

Density Function

)()(likely,equallyaremaginitudesametheoferrorsnegativeandPositive

oneslargeoccur thanlikely tomoreareerrorsSmall

)(isdx)x(x,intervalewithin thlyingerrorsofyprobabilitThe

.propertiesfollowingthe

ithfunction wdensityaasmeaningsamewith the)(functionthedefineWe

xx

dxx

x

!

JJ

J

J


11/25

11

Mean and Variance

Define . In many cases

assume k=0 Define mean square error as

If k=0 then the variance will equal

dxxxk ! )(J

dxxxm )(22 J

g

g

!

2m


12/25

12

Reasons for is always positive and is simple.

The function is differentiable and integrable unlikethe absolute value function.

The function approximates the average value in

cases where large numbers of observations are beingconsidered,and is simple to use when consideringsmall numbers of observations.

2

m2m


13/25

13

More on VarianceIf then variance equals .

Suppose we have independent random variables

with standard deviation 1 and expected value 0. Thelinear function of total errors is given by

Now the variance of E is given as

This is assuming every error falls within standarddeviations from the mean

0{k22 km

,...}'',',{ eee

...! eeE PP

!!

!!k

i

i

k

i

iieM1

2

1

222PP

P


14/25

14

Gauss Derivation of the Method of Least Squares

Suppose a quantity, V=f(x), where V, x are unknown. Weestimate V by an observation L.

If x is calculated by L, L~f(x), error will occur.

But if several quantities V,V,Vdepend on the sameunknown x and they are determined by inexact observations,then we can recover x by some combinations of theobservations.

Similar situations occur when we observe several quantities thatdepend on several unknowns.


15/25

15


)variancesamethehavetheysoerrorsthescaledWe:(

ns'.observatiotheoferrorsmean'theofweightstheares'thewhere

,'

)''(',

)(:

:bensobservatioin theerrorsLet the

),,,("

),,,('

),,,(

.,,,unkownsoffunctionsare.,'',',where

.,'',',:nsobservatiotindependenby taking,'',',estimatewant toWe

:Problem

3

2

1

Note

p

p

LVv

p

LVv

zyxfV

zyxfV

zyxfV

zyxVVV

LLLVVV

.

.

.

.

.

..

..

!

!

!

!

!

VT


16/25

16


)"...,,space(nobservatio...),,(spaceparameter:or,RR:

:mappingadescribessystemThis2.

since,inedoverdetermissystem1.This:Note

kno n.are,,tscoe icienthehere

,...,,unko nsounctionslinearasrittenare,,,

vvvzyxF

cba

zyxvvv

pp

""

TV

VT

VT

.

.

!!

!

.

..

.

'''''''''''''''

:systemlinearfollowingheConsider t

lzcybxavlzcybxav

lczbyaxv


17/25

17

_ a

componet.otherthe

orconditionsimilaranteandpossible,assmallasbeto2

antWe

...."""...),,(

Then.ocomponentirsttheis"...),,(uppose

:beloasdescribedcondition,optimalityanstatis ies2.Ronidentiytheis1.

:such thatRtoRrom),",,G(mappinglinearaorlookingareWe

:asproblemthestatecanWe

.z,y,x,otindependenkconstantsomeor

.:s.t

,",,otscoe icienare,",,here

:problemonoptimizatianolve

222min

!!

!

!

O

OOO

V

VT

OOO

OOO

OOO

kvvvvvvgx

Gvvvgx

GFG

vvv

kxetcvvv

vvv

Q

.

.

..

.


18/25

18


Solutions:

Its still not obvious:

How do these results relate with the least squares estimation?

.)()()(.

222222

222

etce

tc !

EOEOEOEEE

OOO .

.,"",,enminimun hitsattains

sumthat theobviousisithichFromsystem.theo

neliminatiobyderivedetscoe icienthedenotetheallhere

222

etc

s

EOEOEO

OOO

E

!!!

.


19/25

19


It can be proved that

...'ofonminimizatitheasresultssameget thewillwe

,0,0..

vanish.,,,

partialstheallwhere,minimizethatvaluesparameterthepickssquaresLeast

'

)'),,,('()),,,((

"'

22

22222

!x

;x!

x

;x

x

;x

x

;x

x

;x

;

!!;

OO

.

.

.

..

.

yxei

zyx

p

LzyxV

p

LzyxVvvvLet


20/25

20

Gauss derivation by modern matrix notation:

lAxv

pv

xV

cb

RcbcxbxbV

xxx

VVV

iiii

ii

ii

iij

iijiiii

!

!

!

:becomessystemThe

/)(

:settingbysystemcoordinateneaitch to

onobservatioanisAssume

.theovaluesin er theattempt toaninthemeasureWe

.andtheallovaluestheknoe

,,...

such that,,,parametersounctions

linearare,,,quantitiesobservablethatAssume

11

21

21

VV

V

T

.

.


21/25

21

Gauss derivation by modern matrix notation:

:Proof

norm.minimunofrowshas)(matrixthematricessuchallamongand

,

:holdsfollowingthesuch thatmatrixaiseThen ther.rankofmatrix)(aisSuppose

1 TTAAAE

xKAxx

KA

!

!

v"v

V

TVVVTVT

Gauss results are equivalent to the following lemma:

possible.assmallasbeshould

ofentriesdiagonaltheofsumthat thedemandingtoequivalentisThis

possible.assmallasbeshould...

:thatisconditiononoptimizatiThe

condition.firstthesatisfies)(

222

1

T

iiii

TT

KK

KKK

AAAE

T!

!


22/25

22

optimal.isGofpartconsistent-nontheinverses,leftlinearallamongand

))(()(functiontheofinverselefttheis)(:)(

thatshowslemmaour,:equationoriginalourtoeturning

diagonal.

itsonentriespositivestrictlyhavewill))((entries,zero-nonany

has)(ifsinceone,optimalfact theinissolutionthat theshowsThis

))(())())(((inally,

0)(getwe,thatnotingandgmultiplyinightmatrix.zerotheis)(Thus.

0)(,:getweg,Subtractin

;havewelso,;and,:Thus,)(

)(,Thus.)(denote,invertibleis

:continuedProof

1

11

xxlAxxEllAxEvG

lAxv

EKEK

EKE

EKEKEEEKEEKEKK

EEKEADD

AE

KA

xE

Kx

xKAxxEAxDAEAAAE

AxDAAxAAAxxAADAA

T

TTTT

TTTT

TTT

TTTTT

!!!

!

!!!

!! !

!!!!

!!!

Q

.


23/25

23

Gauss-Markov theorem

ariance.smallest vwith theestimatorunbiasedbesttheisestimatorsquares-leasttheed,uncorrelatareandvariancesamethehave'whens,other wordIn

)~

()Var(Cand)E(havewe,Cof~

estimatorunbiasedanyforthen,)Var(and0)E(Ifor.error vect

theisandctor,unknown veanis,rankhmatrix witaniswhere

modellinearaIn

TT

2

s

Var

I

ppn

Ax

LSLS

I

FUUUUFF

WII

IU

IU

e!!

!!

v

!


24/25

24

Limitation of the Method of Least Squares

Nothing is perfect:

This method is very sensitive to the presence ofunusual data points. One or two outliers cansometimes seriously skew the results of a leastsquares analysis.


25/25

25

References

Gauss, Carl Friedrich, Translated by G. W. Stewart. 1995. Theory of theCombination ofObservations Least Subject to Errors: PartOne, PartTwo, Supplement. Philadelphia: Society for Industrial and Applied

Mathematics. Plackett, R. L. 1949. A Historical Note on the Method of Least Squares.

Biometrika. 36:458460.

Stephen M. Stiger, Gauss and the Invention of Least Squares. TheAnnals of Statistics, Vol.9, No.3(May,1981),465-474.

Plackett, Robin L. 1972. The Discovery of the Method of Least Squares.

Plackett, Robin L. 1972. The Discovery of the Method of Least Squares.

Belinda B.Brand, Guass Method of Least Squares: A historically-basedintroduction. August2003

http://www.infoplease.com/ce6/people/A0820346.html

http://www.stetson.edu/~efriedma/periodictable/html/Ga.html

Final Versiongauss and the Method of Least Squares

Documents

Transcript of Final Versiongauss and the Method of Least Squares