E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

1

MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

EduardoFontana

Dep.deEletrônicaeSistemasUniversidadeFederaldePernambuco

UFPE–CTG-BlocoA,Sala421,Recife–PE50740-550BrasilE-mail:fontana@ufpe.br

Published:January20,2014Abstract:Amatrixformulationispresentedtoallowlinearandfirst-ordernonlinearregressionanalysiswith multidimensional model functions. Examples of pseudocodes are presented illustrating theimplementationoftheformulationfornumericalcomputation.

1.Introduction

Regressionanalysisisanimportanttoolforthedeterminationoftherelationship

between variables from experimental datameasured in physical, biological, statistical

and other phenomena. If a goodmodel function is known to govern the experiment,

parameterscanbeinferredbyfittingthemodelfunctiontothedata.

The formulation broadly available in the literature employs a summation

procedurethatisgeneralizedtoamatrixmethodtoderivetheexpressionsthatleadto

thefinalsetofequationsfortheunknownparameters[1].Inthispaper,oneemploysa

matrix formulation [2]directly toease thedevelopmentof structuredalgorithms.The

formulation is derived for both linear and first-order nonlinear regression analysis.

Examples are shown on how to determine the intermediatematrices that lead to the

solution of the problem for both the linear and nonlinear cases. A pseudocode is

presentedforthedevelopmentofsimplenonlinearregressionalgorithms.

2.StatementoftheProblem Considerarealfunction modelingaphysicalquantityw,with

X = x1 x2 ... xM( )T , (1)

representinganindependentvariable,M-element,columnmatrix1and

1Inthepresentformulationmatricesarerepresentedascapitalletterswithanappendedtildesymbol.ThetransposeofamatrixisrepresentedbyaddingthesuperscriptTtothematrix.Them-throwisrepresentedbyappendingthesubscriptmtothematrix,andthen-thcolumn,byappendingthesuperscript<m>.

f X, A( )

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

2

A = a1 a2 ... aL( )T , (2)

acolumnmatrixwithLparametersthatdefinethemodelfunction.

Forinstance,forthemodelfunction

f = a + bx + cy + dxy ,

M=2andL=4andthematrices X and A are

X = x y( )T A = a b c d( )T .

Inthefollowing X willbereferredtoastheindependentvectorand A ,asthe

parametervector.Inpractice,thephysicalquantitywismeasuredonNdistinctpoints

representedasacolumnmatrix

W = w1 w2 ... wN( )T . (3)

The problem in regression analysis is to determine the parameter vector A

yieldingafunctionfthatbestdescribesthemeasuredquantityw.Onewaytosolvethis

problem is to determine a solution A that minimizes the square error [1]. This

parameter,inmatrixnotation,canbewrittenintheform

ξ ≡ W − F( )T W − F( ) , (4)

with

F = f Y 1 , A( ) f Y 2 , A( ) ... f Y N , A( )⎛⎝⎜

⎞⎠⎟T

(5)

representingacolumnmatrixinwhichthen-thelementisthefunctionfevaluatedatthe

point

X = Y n , (6)

with Y n representingthen-thcolumnoftheM × N matrix

Y = Y 1 Y 2 ... Y N( ) (7)

Given that ξ is a positive definite function, if a localminimumexists, it can be

obtainedbyimposingthecondition

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

3

∇ξ = 0 (8)

with

∇ ≡∂∂a1

∂∂a2

... ∂∂aL

⎛

⎝⎜

⎞

⎠⎟

T

(9)

representingthenablaoperator,intheL-dimensionalparameterspace,and

0 ≡ 0 0 ... 0( )T , (10)

theL-dimensionalnullvector.

Expression(8)representsasetofL-equationsontheLunknownparameters.Iff

isagoodmodelfunctionfortheproblem,ingeneralonlyminimawillbeobtained.The

bestsolutioncanthenbechosenasthatyieldingthesmallestvalueof ξ .

As detailed in the following sections, an exact solution can be obtained for the

caseofmodelfunctionsthatarelinearintheparameters.Fornonlinearmodelfunctions

anumericalsolutionhastobeobtained.

3.LinearRegression

Afunctionthatislinearintheelementsoftheparametervectorcanbewrittenin

theform

f X, A( ) = G X⎡⎣ ⎤⎦T A , (11)

with

G X = g1 X( ) g2 X( ) ... gL X( )( )T , (12)

being an L-element vector that defines the distinct functions associated to the L

elements of the parameter vectors. Superscript X is used to allow defining a more

general rectangular G matrix from which G Xwould represent a given column,

associatedwith the independentvector X . Thiscanbebetterunderstoodbynoticing

that(5)canbecastintotheform

F = GT A , (13)

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

4

with

G = G 1 G 2 ... G N( ) , (14)

representinganL × N matrixwiththen-thcolumnbeingdefinedaccordingto(12),i.e.,

G n = g1 Yn( ) g2 Y

n( ) ... gL Yn( )⎛

⎝⎜⎞⎠⎟T

(15)

Forexample,forthefunction

f = aexp(xy) + bcos x + y( )+ csin xyz( )+ d ,onehasM=3,L=4andonecoulddefine,accordingto(1),(2),(11)and(12),

X = x y z( )T , A = a b c d( )T . g1 X( ) = exp x1x2( ) , g2 X( ) = sin x1 + x2( ) , g3 X( ) = sin x1x2x3( ) , g4 X( ) =1 ,i.e.,

G X = exp x1x2( ) sin x1 + x2( ) sin x1x2x3( ) 1( )T ,with x1 = x, x2 = y, x3 = z .

To determine the solution to the parameter vector, the differential operation

givenby(8)isfirstcomputedusing(4)yielding

∇ξ = ∇ WT W − WT F − FT W + FT F( ) . (16)

Noticingthat W isindependentoftheparametervector,oneobtains

∇ WT F + FT W − FT F( ) = 0 (17)

Thel-thelementof(17)isoftheform

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

5

∂∂al

WT F + FT W − FT F( ) = 0 ,

orequivalently

∂ FT

∂alW − F( ) = 0 (18)

Forthelatterexpression,theproperty

AT B = BT A , (19)

wasused.

Before proceeding, it is instructive to introduce the generalized derivative

operator,foravectorofKelements,

′H ≡ ∇h1∇h2 ... ∇hK( )T . (20)

Expression(20)representsaK × L matrix.Accordingtothisdefinition,setting H = F ,

(18)canbegroupedassetofLequationsoftheform

′F T W − F( ) = 0 , (21)

with

′F T = ∇f Y 1 , A( ) ∇f Y 2 , A( ) ... ∇f Y N , A( )⎛⎝

⎞⎠ , (22)

representingan L× N matrixand 0 thenullvectoroflengthL.

Noticethat,byuseof(13),

∂ F∂al

= GT ∂ A∂al

,

orequivalently

∂ FT

∂al= ∂ AT

∂alG , (23)

whichcanbegeneralized,byuseofthedefinition(20)totherelation

′F T = ′A T G . (24)

Byapplyingthedefinition(22)tovector A ,itisstraightforwardtoshowthat

′A = ℑL , (25)

with ℑL representingthe L × L identitymatrix.From(24)and(25)oneobtains

′F T = G , (26)

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

6

Inserting(26)into(21)yieldsFrom(13),(18)canbecastintotheform

G W − F( ) = 0 (27)

By inserting (13) into (23) one obtains, after a few algebraicmanipulations, the final

solutionfortheunknownparametervector,

A = G GT( )−1 G W( ) (28)

Noticethat G GT isan L × L squarematrixand G W isavectoroflengthL.

4.First-OrderNonlinearRegression

Forafunctionthatisnonlinearontheelementsoftheparametervector,onecan

obtain an approximate solution by making a first-order linear approximation for the

modelvector.Smallcorrectionsareobtainedbyiteration.Atthek-thiterationstep,one

obtainsasetofLlinearequationsforthecorrectiontobemadeontheparametervector.

In order to develop the procedure, let A k( ) the parameter vector at thek-th iterationstep.Giventhisvector,oneexpandsthemodelfunction,tofirstorderintheform

f X, A( ) = f X, A k( )⎡⎣ ⎤⎦ +∇T f X, A k( )⎡⎣ ⎤⎦ A − A k( )⎡⎣ ⎤⎦ . (29)

Usingthedefinition(5),(29)canbegeneralizedtotheform

F A( ) = F k( )+ ′F k( ) A − A k( )⎡⎣ ⎤⎦ (30)

with

′F k( ) ≡ ∇f Y 1 , A k( )( ) ∇f Y 2 , A k( )( ) ... ∇f Y N , A k( )( )⎛⎝

⎞⎠

T

(31)

representing theN × L generalizedderivativematrixof thevector F , according to the

definition(20),asper(22).In(30)oneusesthenotation F k( ) = F A k( )⎡⎣ ⎤⎦ .

In order to determine the correction to be obtained at each iteration step, one

considerstheconditiongivenby(21),re-writtenas

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

7

′F A( )⎡⎣ ⎤⎦T W − F A( )⎡⎣ ⎤⎦ =

0 . (32)

Using(30)intheaboveequationandnoticingthattofirstorder

′F A( ) ≈ ′F k( ) ,

yields

′F k( )⎡⎣ ⎤⎦T W − F k( )− ′F k( ) A − A k( )⎡⎣ ⎤⎦{ } = 0 , (33)

whichafterafewalgebraicmanipulationsyields,

Δ A k( ) = ′F k( )⎡⎣ ⎤⎦T ′F k( ){ }−1

′F k( )⎡⎣ ⎤⎦T W − F k( )⎡⎣ ⎤⎦{ } , (34)

with

Δ A k( ) ≡ A− A k( ) , (35)

representingthedifferentialcorrectionintheparametervector.

Expression (34) is calculated iteratively until differential correction becomes

smallerthanacertainpreseterrorparameter.

In the following section, examples on how to define thematrices for both the

linearandnonlinear casesare shown.Apseudocode is shown for thedevelopmentof

algorithmsforthecaseofnonlinearregressionanalysisofdata.

5.Examples

5.1Linearregressionexample ConsideronceagaintheexampleofSection3with

f = aexp(xy) + bcos x + y( )+ csin xyz( )+ d ,

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

8

onehasM=3,L=4.AssumethatNvaluesareobtainedforthequantityw,i.e.

W = w1 w2 ... wN( )T . (36)

From(7)

Y =

x1 x2 ... xNy1 y2 ... yNz1 z2 ... zN

⎛

⎝

⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟

. (37)

From(12)

G X = exp xy( ) sin x + y( ) sin xyz( ) 1( )T (38)

andfrom(15),

G =

exp x1y1( ) exp x2y2( ) ... exp xN yN( )sin x1 + y1( ) sin x2 + y2( ) ... sin xN + yN( )sin x1y1z1( ) sin x2y2z2( ) ... sin xN yN zN( )

1 1 1 1

⎛

⎝

⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟

(39)

Oneobtainsa4× N Gmatrixandexpressions(36)and(39)aresufficienttodetermine

thesolutiongivenby(28).

5.2Nonlinearregressionexample

Forthesakeofsimplicity,considerthe3-parametergaussianmodelforasingle

variablefunction

f x, A( ) = aexp − x − x( ) / w⎡⎣ ⎤⎦2⎧

⎨⎩

⎫⎬⎭, (40)

wherea is theamplitude, x is thecentroidandw is thehalfwidthof thegaussian.The

parametervectoris

A = a x w( )T (41)

Assumeonceagain thatasetofNdatapoints isobtained, representedby(36).

Accordingto(34)allthatisnecessaryistodeterminethematrices F and ′F .Assuming

thatatthek-thiterationtheparametervectorisgivenby

A k( ) = a k( ) x k( ) w k( )( )T . (42)

From(5)

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

9

F k( ) =

a k( )exp − x1 − x k( )⎡⎣ ⎤⎦ / w k( ){ }2{ }a k( )exp − x2 − x k( )⎡⎣ ⎤⎦ / w k( ){ }2{ }

...

a k( )exp − xN − x k( )⎡⎣ ⎤⎦ / w k( ){ }2{ }

⎛

⎝

⎜⎜⎜⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟⎟⎟⎟

, (43)

andfrom(31),oneobtains,afterafewalgebraicmanipulations

′F k( ) =

e− x1−x k( )⎡⎣ ⎤⎦/w k( ){ }2 2a k( ) x1 − x k( )⎡⎣ ⎤⎦ / w k( )⎡⎣ ⎤⎦

2⎧⎨⎩

⎫⎬⎭e− x1−x k( )⎡⎣ ⎤⎦/w k( ){ }2 2a k( ) x1 − x k( )⎡⎣ ⎤⎦

2/ w k( )⎡⎣ ⎤⎦

3⎧⎨⎩

⎫⎬⎭e− x1−x k( )⎡⎣ ⎤⎦/w k( ){ }2

e− x2−x k( )⎡⎣ ⎤⎦/w k( ){ }2 2a k( ) x2 − x k( )⎡⎣ ⎤⎦ / w k( )⎡⎣ ⎤⎦

2⎧⎨⎩

⎫⎬⎭e− x2−x k( )⎡⎣ ⎤⎦/w k( ){ }2 2a k( ) x3 − x k( )⎡⎣ ⎤⎦

2/ w k( )⎡⎣ ⎤⎦

3⎧⎨⎩

⎫⎬⎭e− x3−x k( )⎡⎣ ⎤⎦/w k( ){ }2

... ... ...

e− xN −x k( )⎡⎣ ⎤⎦/w k( ){ }2 2a k( ) xN − x k( )⎡⎣ ⎤⎦ / w k( )⎡⎣ ⎤⎦

2⎧⎨⎩

⎫⎬⎭e− xN −x k( )⎡⎣ ⎤⎦/w k( ){ }2 2a k( ) xN − x k( )⎡⎣ ⎤⎦

2/ w k( )⎡⎣ ⎤⎦

3⎧⎨⎩

⎫⎬⎭e− xN −x k( )⎡⎣ ⎤⎦/w k( ){ }2

⎛

⎝

⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟⎟⎟⎟⎟

. (44)

Together with (36), (43) and (44) are sufficient to determine the differential

correctionfortheparametervector.

5.3Pseudocodefornonlinearregression Table I shows a pseudocode for the computational implementation of theprocedure, for the case of nonlinear regression analysis. As can be noticed from theprogram structure, by use of thematrix formulation presented in this paper one canorganizethealgorithmusingarathersimpleandmodularscheme.

References[1] Douglas M. Bates and Donald G. Watts, “Nonlinear regression analysis and itsapplications,”JohnWiley&Sons,Inc. (1988).

[2]G.B.ArfkenandH. J.Weber,“MathematicalMethodsforPhysicists,”Chapter3,5th.ed.,SanDiego,CA,AcademicPress(2001).

E.Fontana MatrixFormulationforLinearandFirst-OrderNonlinearRegressionAnalysiswithMultidimensionalFunctions

10

TableI–Pseudocodefornonlinearregressionanalysis.1.Initialization:

• Definemodelfunction • ReaddataandstoreintomatrixW• DefineparametervectorA• Setinitialguesstotheparametervector: • Setahighvalueforanerrorparameter: • Definefunction

• Definesystemmatrixfunction

• Defineinputvectorfunction

• Setamaximumvalueforthechangeinmagnitudeoftheparametervector 2.Calculation:

whileerr>ε

A Remarks:

• Mathcadprogrammingstyleisusedasamodel.• Thevectorizeoperation producesamatrixinwhicheachelementistheabsolutevalueofthedifferencebetweenvectors• Thefunctionmax()calculatesthemaximumelementofthevector.• ThelastvalueofAstorestheapproximatesolutiontotheparametervector.