Lecture 2: Covariance and correlation · Lecture 2: Outline 1. Covariance & correlation 2. Lagged...

Lecture2:

Covarianceandcorrelation

ShaneElipotTheRosenstielSchoolofMarineandAtmosphericScience,

UniversityofMiami

Createdwith{Remark.js}using{Markdown}+{MathJax}

Processingmath:99%

http://remarkjs.com/

https://daringfireball.net/projects/markdown/

https://www.mathjax.org/

References

[1]Bendat,J.S.,&Piersol,A.G.(2011).Randomdata:analysisandmeasurementprocedures(Vol.729).JohnWiley&Sons.

[2]Thomson,R.E.,&Emery,W.J.(2014).Dataanalysismethodsinphysicaloceanography.Newnes.dx.doi.org/10.1016/B978-0-12-387782-6.00003-X

[3]Taylor,J.(1997).Introductiontoerroranalysis,thestudyofuncertaintiesinphysicalmeasurements.

[4]Press,W.H.etal.(2007).Numericalrecipes3rdedition:Theartofscientificcomputing.Cambridgeuniversitypress.

[5]Kanji,G.K.(2006).100statisticaltests.Sage.

[6]vonStorch,H.andZwiers,F.W.(1999).StatisticalAnalysisinClimateResearch,CambridgeUniversityPress

Processingmath:100%

http://dx.doi.org/10.1016/B978-0-12-387782-6.00003-X

Lecture2:Outline1. Covariance&correlation2. Laggedcovariance&correlation3. Aquicklookat"Aleisurelylookatthebootstrap,thejackknife,

andcross-validation"4. Covarianceandcorrelationofbivariatevariables

1.Covariance&Correlation

Covariance(definitions)

Whereaswepreviouslydealtwithasingler.v.x,wenowdealwithtwor.vs.,xandy.Inparticular,weareinterestedinevaluatinghowmuchtheycovary,possiblytomakesomestatementaboutacausationfromonevariabletotheother,perhapsexplainedbyadynamicalrelationship.

Perhapsthefirstquantitytoconsideristhecovarianceofxandy:

Cov(x, y) = Cxy ≡ E[(x − μx)(y − μy)]

Thevarianceofxisaparticularcaseofcovariancewhenyandxarethesamer.v.:

Cov(x, x) = Cxx ≡ E[(x − μx)(x − μx)] = E[(x − μx)2] = Var(x) = σ2x

Covariance(definitions)

iscalledthejointprobabilitydensityfunctionorjointPDF.

If where and arethePDFsof and ,respectively,thenitissaidthat and areindependent.

Justlikewewereabletobuildhistogramsfromsamples ofasingler.v. ,wecanalsobuild2Dhistogramsfrompairsofsamples

inordertoestimatethejointPDF.

Cxy =

=

E[(x− )(y− )] = E[xy] −μx μy μxμy

(x− )(y− )p(x, y) dxdy∫ +∞

−∞μx μy

p(x, y)

p(x, y) = p(x)p(y) p(x) p(y) x yx y

Xnx

( , )Xn Yn

Correlation

Sincether.v. and canbeofdifferentnatureormagnitude,wecanconsiderthenormalizedcovariance,thatisthecorrelationbetween and

Sincewehavethepropertythat ,thecorrelationisanumberbetween and .

If itissaidthatthevariable and areuncorrelated.

Iftwor.vs.areindependent thentheyarealsouncorrelated,butiftwor.vs.areuncorrelated,theyarenotnecessarilyindependent(i.e.maybe )

x y

x y

= = =ρxyCxy

CxxCyy− −−−−−

√

Cxy

σxσy

E[(x− )(y− )]μx μy

{E[(x− ]E[(y− ]}μx)2 μy)21/2

| | ≤Cxy σxσy−1 1

= 0ρxy x y

[p(x, y) = p(x)p(y)]

p(x, y) ≠ p(x)p(y)

Covariance&Correlation:estimation

Justlike isanunbiasedestimateof ,anunbiasedestimateofthecovarianceis

andanestimateof is

iscalledthePearson'scorrelationcoefficient.Itmeasurestherelativestrengthofalinearrelationshipbetween and (seeLecture3).Anonlinearrelationshipornoisewillmake tendstozero.

s2x σ2x

= = ( − )( − )sxy Cxy1

N − 1∑n=1

N

Xn X¯ ¯¯

Yn Y¯ ¯¯

ρxy

rxy = = =ρxysxy

s2xs2y

− −−−√( − )( − )∑N

n=1 Xn X¯ ¯¯

Yn Y¯ ¯¯

[ ( − ( − ]∑Nn=1 Xn X

¯ ¯¯ )2 ∑Nn=1 Yn Y

¯ ¯¯ )21/2

rxyx y

rxy

Correlation:significance

Since and arer.vs., isalsoar.v.withagivendistribution.

Typically,onewantstotestif isdifferentfromzero.Kanji(2006),reference[5]recommendstocalculatethefollowingteststatistic(test12)

whichfollowstheStudent'st-distributionwith degreesoffreedom.

x y rxy

rxy

t = ∼ t(0,N − 2)rxy

1 − r2xy− −−−−−√

N − 2− −−−−√

N − 2

Correlation:confidenceinterval

Alternatively,onemaywanttoderiveCIsfor byusingtheFishertransformedvariable:

Since ,theCIderivedfor canbeusedtocalculateCIsfor .

Theteststatistic

canbeusedtotestthenullhypothesesthat ,seetest13ofKanji(2006),reference[5].

rxy

w = = ln( ) ∼ N [ ln( ), ]tanh−1 rxy12

1 + rxy1 − rxy

12

1 + rxy1 − rxy

1N − 3− −−−−√

= tanhwrxy wrxy

z = ∼ N (0, 1)w− w0

1/ N − 3− −−−−√

= tanhρxy w0

Correlation:effectivedegreesoffreedom

Acrucialaspectoftestingforthesignificanceof istousetherightvaluefor ,whichisthesizeofyoursampleifyourdatapointsareindependent.Otherwise,alowervalueof ,knownastheeffectivedegreeoffreedom,needstobeconsidered.Wewillexploretheseaspectslaterandduringthepracticalthisafternoon.

ρxyN

N

Serialcorrelation:example

Let'sconsiderarealization ofatimeseries ,andcalculatethecorrelations :

Since , and arecorrelated,hencetheyarenotindependentsamples.Itissaidthat isseriallycorrelated,thusthenumberofdegreesoffreedomislessthan .Thevarianceandbiasofcorrelationestimatesinvolving arelikelytobeaffected.

, ,… ,X1 X2 XN x(t)ρ( , )Xn Xn+1

≠ 0rxy Xn Xn+1x

Nx

Spearmancorrelation

ThePearson'scorrelationcoefficientisnottheonlycorrelationcoefficient.TherankcorrelationcoefficientorSpearmancorrelationcoefficientis

where istherankofthedatasample .

= 1 −rs6 ( −∑N

n=1 Rx;n Ry;n)2

N ( − 1)N 2

Rx;n Xn

Spearmancorrelation


where istherankofthedatasample .Asanexample,with

= 1 −rs6 ( −∑N

n=1 Rx;n Ry;n)2

N ( − 1)N 2

Rx;n XnN = 3

Xn

4−13

Rx;n

312

Yn

1123

Ry;n

132

−Rx;n Ry;n

2− 20

Spearmancorrelation


where istherankofthedatasample .Asanexample,with

measuresiftherelationshipbetween and ismonotonicallyincreasing( >0)ordecreasing( ).Forlarge ,

under .

= 1 −rs6 ( −∑N

n=1 Rx;n Ry;n)2

N ( − 1)N 2

Rx;n XnN = 3

Xn

4−13

Rx;n

312

Yn

1123

Ry;n

132

−Rx;n Ry;n

2− 20

rs x yrs < 0rs N

≈ N (0, 1/ )rs N − 1− −−−−

√ : = 0H0 rs

PearsonvsSpearman

Covariance:multivariatecase

Ifyouaredealingwithmorethantwor.vs.,let'ssay variablesforwhichyouhave samples,you

needtobuildthecovariancematrix

orthecorrelationmatrix

Methodstostudythesematriceswillbecoveredinlecture5(Eigentechniques).

P, ,… ,x1 x2 xP n = 1, 2,… ,N

≡Cxx

⎡

⎣⎢⎢⎢⎢⎢Cx1x1

Cx2x1

⋮CxP x1

Cx1x2

Cx2x2

⋮CxP x2

…

…

⋯…

Cx1xP

Cx2xP

⋮CxP xP

⎤

⎦⎥⎥⎥⎥⎥

≡ρxx

⎡

⎣⎢⎢⎢⎢⎢ρx1x1

ρx2x1

⋮ρxP x1

ρx1x2

ρx2x2

⋮ρxP x2

…

…

⋯…

ρx1xP

ρx2xP

⋮ρxP xP

⎤

⎦⎥⎥⎥⎥⎥

Covariance:multivariatecase;estimate

Ifyouhave r.vs. forwhichyouhavesamples,typicallyarrangedina datamatrix

Theestimateofthecovariancematrixis

P , ,… ,x1 x2 xPn = 1, 2,… ,N N × P

X = [ , , ⋯ , ] =X1 X2 XP

⎡

⎣⎢⎢⎢⎢⎢

(1)X1

(2)X1

⋮(N )X1

(1)X2

(2)X2

⋮(N )X2

…⋯

⋯⋯

(1)XP

(2)XP

⋮(N )XP

⎤

⎦⎥⎥⎥⎥⎥

= X =Cxx1

N−1 XT

⎡

⎣⎢⎢⎢⎢⎢⎢

Cx1x1

Cx2x1

⋮

CxP x1

Cx1x2

Cx2x2

⋮

CxP x2

…

…

⋯

…

Cx1xP

Cx2xP

⋮

CxP xP

⎤

⎦⎥⎥⎥⎥⎥⎥

Correlation:beware!

Itisnotbecauseyoufindazeroornonsignificantcorrelationbetweentwor.vs.thatthereisnoformalordynamicalrelationshipbetweenthetwo!

Correlation:beware!

ThisanimationshowsthecorrelationcoefficientbetweentwosinusoidsignalsFandGasafunctionofphaselag.

GifbyDivergentdata-Ownwork,CCBY-SA4.0,Link

http://creativecommons.org/licenses/by-sa/4.0

https://commons.wikimedia.org/w/index.php?curid=57768455

2.LaggedCovariance&Correlation

Laggedcovariance&correlationfunctions

Wenowgeneralizetheconceptofcovariancebyconsideringtwor.vs.forwhichthesamplesareordered,maybeasafunctionoftime(orofspace).Inthiscase,thesamplesarerealizationsoftime

series.t



series.

Thecovariancestatisticpresentedearlierisaspecialcaseofthe(cross-)covariancefunction,functionoflag

If , iscalledtheauto-covariancefunctionof .

t

τ

(τ) = E{[x(t) − ][y(t+ τ) − ]}Cxy μx μy

y ≡ x (τ)Cxx x



series.

Thecovariancestatisticpresentedearlierisaspecialcaseofthe(cross-)covariancefunction,functionoflag

If , iscalledtheauto-covariancefunctionof .

Notethatherewehaveassumedthatthepopulationmeans andareconstantwithtime,inwhichcaseitissaidthat and

arestationarytimeseries.

t

τ

(τ) = E{[x(t) − ][y(t+ τ) − ]}Cxy μx μy

y ≡ x (τ)Cxx x

μxμy x(t) y(t)


Instatisticsandengineeringsciencesthenamescanbedifferent.Inparticular,inMatlab,thefollowingfunctioniscalledthecross-correlationfunction

whichissimilartothecovariancefunctionbutwithoutsubtractingthemeans.Assuch

(τ) = E{x(t)y(t+ τ)}Rxy

(τ) = (τ) −Cxy Rxy μxμy


Instatisticsandengineeringsciencesthenamescanbedifferent.Inparticular,inMatlab,thefollowingfunctioniscalledthecross-correlationfunction

whichissimilartothecovariancefunctionbutwithoutsubtractingthemeans.Assuch

Withsuchnamingconvention, istheauto-correlationfunction.

(τ) = E{x(t)y(t+ τ)}Rxy

(τ) = (τ) −Cxy Rxy μxμy

(τ)Rxx


Theauto-covarianceandauto-correlationfunctionsareevenfunctionsof (i.e.symmetricaround ):

Thecross-covarianceandcross-correlationfunctionsareneitheroddnoreven,butsatisfies

τ 0

(−τ)Cxx

(−τ)Rxx

==

(τ)Cxx

(τ)Rxx

(−τ)Cxy

(−τ)Rxy

==

(τ)Cyx

(τ)Ryx


Theauto-covarianceandauto-correlationfunctionsareevenfunctionsof (i.e.symmetricaround ):

Thecross-covarianceandcross-correlationfunctionsareneitheroddnoreven,butsatisfies

Beware!Checktheconventionsofthesoftwaresyouuse!InMatlab,thecross-correlationfunctionisdefinedas

.

τ 0

(−τ)Cxx

(−τ)Rxx

==

(τ)Cxx

(τ)Rxx

(−τ)Cxy

(−τ)Rxy

==

(τ)Cyx

(τ)Ryx

E[x(t+ τ)y(t)] = (−τ) = (τ)Rxy Ryx

Laggedcorrelationcoefficient

Thelaggedcorrelationcoefficientis

(τ) = =ρxy(τ)Cxy

(0) (0)Cxx Cyy− −−−−−−−−−√

(τ)Cxy

σxσy



Itcanbechallengingbutlet'strynottoconfusecorrelationfunctionandlaggedcorrelation.

(τ) = =ρxy(τ)Cxy

(0) (0)Cxx Cyy− −−−−−−−−−√

(τ)Cxy

σxσy




Youmaywanttocall thelaggedauto-correlationcoefficient,butitisusuallycalledtheautocorrelationfunction(whichshouldbeusedfor ).

(τ) = =ρxy(τ)Cxy

(0) (0)Cxx Cyy− −−−−−−−−−√

(τ)Cxy

σxσy

(τ) =ρxx(τ)Cxx

(0)Cxx

(τ)Rxx




Youmaywanttocall thelaggedauto-correlationcoefficient,butitisusuallycalledtheautocorrelationfunction(whichshouldbeusedfor ).

Distributionsundernullhypothesesfor or aredifficulttocomebyandrequiretomakeseveralassumptions(seepracticalthisafternoon).

(τ) = =ρxy(τ)Cxy

(0) (0)Cxx Cyy− −−−−−−−−−√

(τ)Cxy

σxσy

(τ) =ρxx(τ)Cxx

(0)Cxx

(τ)Rxx

(τ)ρxy (τ)ρxx

Laggedcovariancefunctionestimates

Considertwor.vs. and forwhichwehave samplesseparatedbyconstantintervals .Oneestimateofthecross-covariancefunctionatlags is

where arethepossiblelags.Itcanbeshownthatthisestimatorisunbiased(ifthepopulationmeansareknownratherthanestimated).

x y NΔt

= kΔtτk

(kΔt) = ( − )( − )C(1)xy

1N − |k|

∑n=1

N−|k|

Xn X¯ ¯¯

Yn+k Y¯ ¯¯

k = 0,±1,±2,…,±(N − 1)

Laggedcovariancefunctionestimates

Considertwor.vs. and forwhichwehave samplesseparatedbyconstantintervals .Oneestimateofthecross-covariancefunctionatlags is

where arethepossiblelags.Itcanbeshownthatthisestimatorisunbiased(ifthepopulationmeansareknownratherthanestimated).Thereexistsanotherestimatorof

whichis

whichistypicallycalledthe"biased"estimatorbuthasasmaller

randomerrorcomparedto .Seepracticalthisafternoon.

x y NΔt

= kΔtτk

(kΔt) = ( − )( − )C(1)xy

1N − |k|

∑n=1

N−|k|

Xn X¯ ¯¯

Yn+k Y¯ ¯¯

k = 0,±1,±2,…,±(N − 1)

(τ)Cxy

(kΔt) = ( − )( − )C(2)xy

1N

∑n=1

N−|k|

Xn X¯ ¯¯

Yn+k Y¯ ¯¯

C(1)xy

Laggedcorrelationestimate

Anestimateofthelaggedcorrelationcoefficientis

wheretheuseof cancelsoutthefactor intheformulasforthevariances.ThisestimatorisaccessedinMatlabusingthescaleoption asin

[rhox,lags] = xcov(Y,X,'coeff');

or

[rhox,lags] = xcorr(Y-mean(Y),X-mean(X),'coeff');

(kΔt) = =ρxy(kΔt)C

(2)xy

[ ]C(2)xx C

(2)yy

1/2

( − )( − )∑N−|k|n=1 Xn X

¯ ¯¯Yn+k Y

¯ ¯¯

[ ( − ( − ]∑Nn=1 Xn X

¯ ¯¯ )2 ∑Nn=1 Yn Y

¯ ¯¯ )21/2

C(2)xy N

'coeff'

Auto-correlationasameasureofmemory

Lag scatterplotsandcor.coeff. forAgulhas

transport

m ρ( , )Xn Xn+m

Auto-correlationasameasureofmemory

LaggedautocorrelationcoefficientforAgulhastransporttimeseries.With ,possiblelagsareN = 808 τ = −807,…,807

Laggedcorrelationfortimedelay

Thelaggedcorrelationorcovariancefunctioncanbeusedtodetermineatimedelaybetweentwosignals.Let'sassumethatatransmittedsignal(atimeseries) isreceivedas

where isanattenuationfactor, isaconstanttimedelayequaltolet'ssayadistance dividedbyapropagationvelocity ,and isanaddednoiseuncorrelatedwith .

x(t)

y(t) = αx(t− ) + n(t)τ0

α = d/cτ0d c

n(t) x(t)



where isanattenuationfactor, isaconstanttimedelayequaltolet'ssayadistance dividedbyapropagationvelocity ,and isanaddednoiseuncorrelatedwith .Itiseasilyshownthat

x(t)

y(t) = αx(t− ) + n(t)τ0

α = d/cτ0d c

n(t) x(t)

(τ) = α (τ − )Rxy Rxx τ0



where isanattenuationfactor, isaconstanttimedelayequaltolet'ssayadistance dividedbyapropagationvelocity ,and isanaddednoiseuncorrelatedwith .Itiseasilyshownthat

Sincethemaximumof isfor ,thepeakvalueofoccurswhen whichallowsustodeterminethe

constanttimedelay.

x(t)

y(t) = αx(t− ) + n(t)τ0

α = d/cτ0d c

n(t) x(t)

(τ) = α (τ − )Rxy Rxx τ0

(τ)Rxx τ = 0(τ)Rxy τ = τ0

ExamplefromDiNezioetal.2009investigatingtherelationshipbetweenGulfStreamtransportthroughtheFloridaStraitandwindstresscurl.


http://dx.doi.org/10.1175/2008JPO4001.1

Auto-correlationandeffectivedegreesoffreedom

Theconceptofeffectivedegreesoffreedomand(de)correlationtimescaleareintimatelylinked.If isthenumberofevenlydistributedsamplesatinterval ,thenthenumberofeffectivedegreesoffreedomis

where isthelengthofyourtimeseriesand isthedecorrelationtimescale,alsoreferredtoasanintegraltimescale. givesyouthenumberofeffectivelyindependentsamplesinyourdata.

NΔt

= =NeffNΔtT0

T

T0

T T0Neff


Theconceptofeffectivedegreesoffreedomand(de)correlationtimescaleareintimatelylinked.If isthenumberofevenlydistributedsamplesatinterval ,thenthenumberofeffectivedegreesoffreedomis

where isthelengthofyourtimeseriesand isthedecorrelationtimescale,alsoreferredtoasanintegraltimescale. givesyouthenumberofeffectivelyindependentsamplesinyourdata.

Ifyoursamplesareindependent,thedecorrelationtimescaleisthetimestepofyourtimeseries and .Thereisnotonlyonewayofcomputing orestimating .Infact,itdependsonthestatisticsforwhichthesewillbeused.

NΔt

= =NeffNΔtT0

T

T0

T T0Neff

Δt = NNeffNeff T0

Decorrelationtimescale

Onecommondefinitionofthedecorrelationorintegraltimescaleforther.v. is(e.g.ThompsonandEmery(2014),section3.15.2)

Agraphicalinterpretationof isgiveninFigure3.13ofEmeryandThompson(2014):

x

= (τ) dτ = (τ) dτT01(0)Cxx

∫ +∞

−∞Cxx

2(0)Cxx

∫ +∞

0Cxx

T0× (0) = (τ) dτT0 Cxx ∫ +∞

−∞ Cxx


Anestimateof isobtainedbyapplyingthetrapezoidalintegrationformulafortheintegral,i.e.

where isthesamplevarianceestimateof and istheindexwherethesummationisstopped.

T0

= ΔtT02s2x

∑n=0

M−1 [(n+ 1)Δt] + [nΔt]Cxx Cxx

2

= (0)s2x Cxx x M


Anestimateof isobtainedbyapplyingthetrapezoidalintegrationformulafortheintegral,i.e.

where isthesamplevarianceestimateof and istheindexwherethesummationisstopped.

NotethatThompsonandEmery(2014),section5.3.5,giveanalternateformula

wheretheyomittedthefactor andessentiallyintegratedonlyonesideoftheauto-covariancefunction.ThisisalsotheformulatypicallyusedinLagrangianstudies.(AndthisiswhatIuse).

T0

= ΔtT02s2x

∑n=0


2

= (0)s2x Cxx x M

= ΔtT01s2x

∑n=0


2

2


Thereareseveralsourcesoferrorenteringformulasfor :estimateofthevariance,estimateoftheautocovariance,andtruncationofthetrapezoidalintegration.Thetruncationoftheintegrationmaybewheretheestimatedautocovariancefunctionreachesaconstantvaluebutthismaynothappeninpractice.Anotherpossibilityistointegrateuptothefirstzerocrossingoruptowheretheautocorrelationisinsignificant.Seepracticalsessionthisafternoon.

T0


Onceyouhaveobtainedanintegraltimescale andcalculatedtheeffectivedegreesoffreedom ,youmayusethisparametertocalculateCIsforsamplemeans.

T0Neff


Onceyouhaveobtainedanintegraltimescale andcalculatedtheeffectivedegreesoffreedom ,youmayusethisparametertocalculateCIsforsamplemeans.

Theformulasgivenpreviouslydonotseemtobeappropriatewhenisneededtoassessestimatorsofauto-covariance/correlation

orcross-covariance/correlation.Inthesecases,VonStorchandZwiers(1999),reference[6],givestherespectivetwoformulas

assumingyouhaveaconstanttimestep foryourtimeseries.

T0Neff

Neff

T0

Δt

T0

Δt

=

=

1 + 2 (kΔt)∑k=1

+∞

ρ2xx

1 + 2 (kΔt) (kΔt)∑k=1

+∞

ρxx ρyy

Δt

3.Aquicklookat"Aleisurelylookatthebootstrap,thejackknife,andcross-validation"byEfronandGong(1983)

Thebootstrapandthejackknife

Sofar,wehaveusedtheoreticalorassumeddistributionsofourdatainordertoderivestandarderrorsandCIsofourparameterestimates.However,inmanycases,likeforthecorrelationcoefficient,itisimpossibletoexpressthevarianceinclosedform.Inaddition,weareoftenstuckwithonesamplefromoneexperimentinordertoinvestigatethe population.Ontopofthis,oursamplesmaynotbeindependentandwerunintotheissueofestimatingtheeffectivedegreesoffreedom.

( , ,… , )X1 X2 Xnx

Thebootstrapandthejackknife

Sofar,wehaveusedtheoreticalorassumeddistributionsofourdatainordertoderivestandarderrorsandCIsofourparameterestimates.However,inmanycases,likeforthecorrelationcoefficient,itisimpossibletoexpressthevarianceinclosedform.Inaddition,weareoftenstuckwithonesamplefromoneexperimentinordertoinvestigatethe population.Ontopofthis,oursamplesmaynotbeindependentandwerunintotheissueofestimatingtheeffectivedegreesoffreedom.

Theideaofthebootstrapandjackknife,combinedwithMonteCarlomethods,istoresampleyouroriginalsamplewithreplacementsforthebootstrap,andwithdeletionsforthejackknife.Thesemethodsarerelativelyeasytoimplementwithourfastandmoderncomputers.

( , ,… , )X1 X2 Xnx

Thebootstrap:principle(1)

Let'sassumeyouareinvestigatingaunivariateormultivariater.v.forwhichyouwanttoestimateastatistic withestimator ,usingasample .

x

ϕ ϕ( , ,… , )X1 X2 Xn



First,youdrawrandomlyabootstrapsample ofthesamesizeasyouroriginalsample.Asanexample,ifyouroriginalsampleifofsize ,i.e. ,abootstrapsamplewithreplacementmaybe .

x

ϕ ϕ( , ,… , )X1 X2 Xn

( , ,… , )X∗1 X

∗2 X∗

N

N = 3 ( , , )X1 X2 X3( , , ) = ( , , )X∗

1 X∗2 X

∗3 X1 X2 X1




Fromthebootstrapsample,youcalculateabootstrapreplicationestimateofyourstatistic .

x

ϕ ϕ( , ,… , )X1 X2 Xn

( , ,… , )X∗1 X

∗2 X∗

N

N = 3 ( , , )X1 X2 X3( , , ) = ( , , )X∗

1 X∗2 X

∗3 X1 X2 X1

ϕ∗




Fromthebootstrapsample,youcalculateabootstrapreplicationestimateofyourstatistic .Yourepeatthisoperation timesto

obtain bootstrapreplications .

x

ϕ ϕ( , ,… , )X1 X2 Xn

( , ,… , )X∗1 X

∗2 X∗

N

N = 3 ( , , )X1 X2 X3( , , ) = ( , , )X∗

1 X∗2 X

∗3 X1 X2 X1

ϕ∗

B

B , ,… ,ϕ∗(1)

ϕ∗(2)

ϕ∗(B)




Fromthebootstrapsample,youcalculateabootstrapreplicationestimateofyourstatistic .Yourepeatthisoperation timesto

obtain bootstrapreplications .

Youfinallycalculate(estimate)thevarianceofyourreplicationsas

x

ϕ ϕ( , ,… , )X1 X2 Xn

( , ,… , )X∗1 X

∗2 X∗

N

N = 3 ( , , )X1 X2 X3( , , ) = ( , , )X∗

1 X∗2 X

∗3 X1 X2 X1

ϕ∗

B

B , ,… ,ϕ∗(1)

ϕ∗(2)

ϕ∗(B)

Var[ ] = ( − where =ϕ∗ 1

B− 1∑b=1

B

ϕ∗(b)

ϕ∗(.)

)2 ϕ∗(.) ∑b ϕ

∗(b)

B


Youcannowuse toderiveastandarderrorofyourestimate

as orusetheestimateddistributionofyourbootstrap

replications tocalculateCIs.

Example:BootstrapcorrelationsbetweenbetweenAgulhasjetandboundarytransportswith .

Var[ ]ϕ∗

Var[ ]ϕ∗− −−−−−√

, ,… ,ϕ∗(1)

ϕ∗(2)

ϕ∗(B)

B = 1000

Thejackknife:principle(1)

Let'sassumeagainthatyouareinvestigatingaunivariateormultivariater.v. forwhichyouwanttoestimateastatistic withestimator ,usingasample .

x ϕ

ϕ ( , ,… , )X1 X2 Xn


Let'sassumeagainthatyouareinvestigatingaunivariateormultivariater.v. forwhichyouwanttoestimateastatistic withestimator ,usingasample .Ajackknifesampleisobtainedbydeleting datapointsfromyouroriginalsampleof

points.Thenumberofsuchsamplethatcanbeobtainedisthenumberofpermutationsof objectstaken atatimewhichis

.Ifyouhave datapointsthereare permutationsforthe"delete-1"jackknife.Thenumberofofpossiblejackknifesamplescanbecomeverylargesothatasubsetneedtobechoosen.

x ϕ

ϕ ( , ,… , )X1 X2 XnJ

NN J

N !/(N − J)! N N


Let'sassumeagainthatyouareinvestigatingaunivariateormultivariater.v. forwhichyouwanttoestimateastatistic withestimator ,usingasample .Ajackknifesampleisobtainedbydeleting datapointsfromyouroriginalsampleof

points.Thenumberofsuchsamplethatcanbeobtainedisthenumberofpermutationsof objectstaken atatimewhichis

.Ifyouhave datapointsthereare permutationsforthe"delete-1"jackknife.Thenumberofofpossiblejackknifesamplescanbecomeverylargesothatasubsetneedtobechoosen.

Theformulaforthe"delete-1"jackknifevarianceofyourstatisticestimateis

x ϕ

ϕ ( , ,… , )X1 X2 XnJ

NN J

N !/(N − J)! N N

Var[ ] = ( − where =ϕJ N − 1

N∑j=1

N

ϕ∗(j)

ϕ∗(.)

)2 ϕ∗(.) ∑j ϕ

∗(j)

N


Thejackknifemethodcanbealsousedtoestimatethebiasofyourestimatoras

sothatthejackknifeestimateofyourstatisticis

[ ] = (N − 1)( − )bJ ϕ ϕ∗(.)

ϕ

= N − (N − 1)ϕJ ϕ ϕ∗(.)


Thejackknifemethodcanbealsousedtoestimatethebiasofyourestimatoras

sothatthejackknifeestimateofyourstatisticis

Therearesomeimportantdetailstothebootstrapandthejackknifemethods.PleaseseeThompsonandEmery,TichelaarandRuff(1989)andEfronandGong(1983).

[ ] = (N − 1)( − )bJ ϕ ϕ∗(.)

ϕ

= N − (N − 1)ϕJ ϕ ϕ∗(.)

http://dx.doi.org/10.1080/00031305.1983.10483087

4.Covarianceandcorrelationofbivariate

variables

Bivariaterandomvariable

Whatifyourprocessofinterest,oryourdata,isa"vectorvariable"?Itcouldbeanoceancurrent,anatmosphericwind,thepositionofadrifteretc.Suchvariablesarecalledbivariatevariablesinthestatisticslitterature,andarealsotreatedasr.vs.


Whatifyourprocessofinterest,oryourdata,isa"vectorvariable"?Itcouldbeanoceancurrent,anatmosphericwind,thepositionofadrifteretc.Suchvariablesarecalledbivariatevariablesinthestatisticslitterature,andarealsotreatedasr.vs.Herewewillusetheformalismoftimeseries,thatisweassumethatwehavesamplesindexedalonganaxis thatrepresentstime.t


Whatifyourprocessofinterest,oryourdata,isa"vectorvariable"?Itcouldbeanoceancurrent,anatmosphericwind,thepositionofadrifteretc.Suchvariablesarecalledbivariatevariablesinthestatisticslitterature,andarealsotreatedasr.vs.Herewewillusetheformalismoftimeseries,thatisweassumethatwehavesamplesindexedalonganaxis thatrepresentstime.

Let'sconsiderafirstbivariatevariable withCartesiancomponents and (east-westandnorth-south).

t

z(t)x(t) y(t)


Whatifyourprocessofinterest,oryourdata,isa"vectorvariable"?Itcouldbeanoceancurrent,anatmosphericwind,thepositionofadrifteretc.Suchvariablesarecalledbivariatevariablesinthestatisticslitterature,andarealsotreatedasr.vs.Herewewillusetheformalismoftimeseries,thatisweassumethatwehavesamplesindexedalonganaxis thatrepresentstime.

Let'sconsiderafirstbivariatevariable withCartesiancomponents and (east-westandnorth-south).

Wecanarrangethecomponentsintoa vectorfunctionoftime

oralternativelyuseacomplex-valuednotation

where and isthecomplexargument(orpolarangle)of intheinterval .

t

z(t)x(t) y(t)

1 × 2

z(t) = [ ] or = [ ]x(t) y(t) zn xn yn

z(t) = x(t) + iy(t) = |z(t)| ,ei arg (z)

i = −1−−−

√ arg (z)z [−π,+π]

ThemeanofbivariateData

Thesamplemeanofthevectortimeseries isalsoavector,

thatconsistsofthesamplemeansofthe and componentsof .

z(t)

μz =

=

≡ = [ ]z1N

∑n=1

N

zn 1N

∑Nn=1 xn

1N

∑Nn=1 yn

[ ] = [ ]x y μx μy

x y z

ThemeanofbivariateData

Thesamplemeanofthevectortimeseries isalsoavector,

thatconsistsofthesamplemeansofthe and componentsof .

Usingthecomplexnotation,thesamplemeanis

whichiscomplexnumber.

z(t)

μz =

=

≡ = [ ]z1N

∑n=1

N

zn 1N

∑Nn=1 xn

1N

∑Nn=1 yn

[ ] = [ ]x y μx μy

x y z

= + iz x y

Thevarianceofbivariatevariable

Thevarianceofthevector-valuedtimesseries isnotascalaroravector,itisa matrix

where“ ”isthematrixtranspose, , .

Carryingoutthematrixmultiplicationleadsto

Thediagonalelementsof arethesamplevariances,whiletheoff-diagonalgivesthesamplecovariancebetween and .Notethatthetwooff-diagonalelementsareidentical, .

zn2 × 2

≡ ( − )Σz1

N − 1∑n=1

N

( − )zn z T zn z

T = [ ]zn xn yn = [ ]zTnxn

yn

= [ ] = [ ]Σz

1N − 1

∑n=1

N ( − )xn x2

( − ) ( − )yn y xn x

( − ) ( − )xn x yn y

( − )yn y2

s2x

syx

sxy

s2y

Σz

xn yn=syx sxy


Thevarianceofacomplexr.v., ,needsadefinition:

where meansthecomplexconjuguate,i.e. .

z = x+ iy

Var[z] = E[(z− (z− )]μz)∗ μz

(. )∗ = x− iyz∗




Substitutingandexpandingthepreviousexpressiongives

Thevarianceofa"physicalvector"writtenasacomplexr.v.isasinglerealnumberwhichisthesumofthevarianceofitscomponents.Itisdifferentfromthe matrixofvariancesandcovariancesofitscomponentsseeninthepreviousslide.

z = x+ iy

Var[z] = E[(z− (z− )]μz)∗ μz

(. )∗ = x− iyz∗

Var[z] ==E[(x− ] + E[(y− ]μx)2 μy)2

Var[x] + Var[y]

2 × 2




Substitutingandexpandingthepreviousexpressiongives

Thevarianceofa"physicalvector"writtenasacomplexr.v.isasinglerealnumberwhichisthesumofthevarianceofitscomponents.Itisdifferentfromthe matrixofvariancesandcovariancesofitscomponentsseeninthepreviousslide.Ifrepresentsanoceancurrent, isproportionaltotheaverageeddykineticenergy(density)

z = x+ iy

Var[z] = E[(z− (z− )]μz)∗ μz

(. )∗ = x− iyz∗

Var[z] ==E[(x− ] + E[(y− ]μx)2 μy)2

Var[x] + Var[y]

2 × 2z

Var[z]KE = (< > + < >)/2x′2 y ′2

Thecovarianceofbivariatevariables

Inadditionto ,let'sconsiderasecondbivariatevariablewithCartesiancomponents and .

z(t) w(t)g(t) h(t)


Inadditionto ,let'sconsiderasecondbivariatevariablewithCartesiancomponents and .Thecrosscovariancefunctionatzerolag,orsimplythecovariancebetweenthecomponentsof and canbeformedas

z(t) w(t)g(t) h(t)

z w

E{ w}zT =

=

E{[ ]}(x− )(g− )μx μg

(y− )(g− )μy μg

(x− )(h− )μx μh

(y− )(h− )μy μh

[ ]Cxg

Cyg

Cxh

Cyh


Inadditionto ,let'sconsiderasecondbivariatevariablewithCartesiancomponents and .Thecrosscovariancefunctionatzerolag,orsimplythecovariancebetweenthecomponentsof and canbeformedas

Themeaningofeachindividualentryofthismatrixisobviousbuttheinterpretationofallofthematoncemaybelessso.

z(t) w(t)g(t) h(t)

z w

E{ w}zT =

=

E{[ ]}(x− )(g− )μx μg

(y− )(g− )μy μg

(x− )(h− )μx μh

(y− )(h− )μy μh

[ ]Cxg

Cyg

Cxh

Cyh


Alternatively,let'susethecomplexrepresentationsof and

butassumeforsimplicitythat .

z w

z

w

==x+ iyg+ ih

= = 0μz μw




Adefinitionofthecovariancebetweenthesetwocomplexr.v.is

z w

z

w

==x+ iyg+ ih

= = 0μz μw

Czw ===

E[ w]z∗

E[(x− iy)(g+ ih)]E[xg] + E[yh] + i(E[xh] + E[−yg]).




Adefinitionofthecovariancebetweenthesetwocomplexr.v.is

Bewarethatthedefinitionofthecovariancebetweencomplexr.v.maydiffer.Sometimesitisdefinedas .OurconventionhereisthesameasMatlab.

z w

z

w

==x+ iyg+ ih

= = 0μz μw

Czw ===

E[ w]z∗

E[(x− iy)(g+ ih)]E[xg] + E[yh] + i(E[xh] + E[−yg]).

E[z ]w∗


Simplegeometryrevealsthattherealpartofthecomplexcovarianceistheexpectationofthedotproduct[orinnerproduct,noted ]whiletheimaginarypartistheexpectationofthemagnitudeofthevectorcrossproduct[orouterproduct,noted

]:

where istheinstantaneousgeometricangleintheCartesianplanebetween and ,measuredpositivelycounterclockwiserelativetothegeometricangleof .

(⋅)

(×)

Czw ===

E[xg] + E[yh] + i(E[xh] + E[−yg])E[z ⋅ w] + iE[||z × w||]E[|z||w| cos θ] + iE[|z||w| sin θ]

θz w

z


= E[|z||w| cos θ] + iE[|z||w| sin θ]Czw


Thecovarianceof and isacomplexnumberwhichargument,orphase,is


z w

Arg[ ] = atan{ } ≠ E[θ].CzwE[|z||w| sin θ]E[|z||w| cos θ]



Theabsolutevalueof isclearlyameasureofthecovariancebetweenthetwobivariatevariables,butitsphaseisnot :itisnottheexpectation,ormean,ofthegeometricanglebetween and

.


z w


CzwE[θ(t)]

zw



Theabsolutevalueof isclearlyameasureofthecovariancebetweenthetwobivariatevariables,butitsphaseisnot :itisnottheexpectation,ormean,ofthegeometricanglebetween and

.

However, canstillbeseenasanindicationoftherelativeangleofcovariance.If thisindicatesthatthecovarianceoccurswith and alignedandpointinginthesamedirection,andif thisindicatesthatthecovarianceoccurswith atrightanglecounterclockwisefrom .


z w


CzwE[θ(t)]

zw

Arg[ ]CzwArg[ ] = 0Czwz w

Arg[ ] = π/2Czww z

Thecorrelationofbivariatevariables

Fromthecomplexcovariance,wecandefinethecomplexcorrelationbetween and as

Since and arerealnumbers,thephaseof isthesamephaseasthephaseof :

z w

≡ = | |ρzwCzw

CzzCww− −−−−−√

ρzw eiArg[ ]ρzw

Czz Cww ρzwCzw Arg[ ] = Arg[ ]ρzw Czw


Fromthecomplexcovariance,wecandefinethecomplexcorrelationbetween and as

Since and arerealnumbers,thephaseof isthesamephaseasthephaseof :

Notethatthe"vector"regressionmodelfor basedonthecovariancebetween and is(seeLecture3)

whichshowsthattheregressedvector isrotatedcounterclockwiseby fromthedirectionof .Thisisvalidafterremovingthemeans.

z w

≡ = | |ρzwCzw

CzzCww− −−−−−√

ρzw eiArg[ ]ρzw

Czz Cww ρzwCzw Arg[ ] = Arg[ ]ρzw Czw

wz w

(t) = z(t) = z(t)wmCzwCzz

| |Czw

CzzeiArg[ ]Czw

(t)wmArg[ ]Czw z(t)

Panelcisthecorrelationbetweenthe10-mwindat170E,0Nand10-mwindsatallotherlocationsfromECMWFreanalyses.Therealpartof isshowninpanelaandtheimaginarypartisshowninpanelbasapureimaginarynumbersothatc=a+b.


ρ

ρ

Thecomplementaryofavector

Thecomplementaryofavector ,noted ,is,usingcomplexnotation

z zc

≡ = x− iyzc z∗

Thecomplementaryofavector

Thecomplementaryofavector ,noted ,is,usingcomplexnotation

Geometrically,itisthevector flippedwithrespecttotherealaxis.Ifabivariatetimeseriesrepresentsavectorrotatinginonedirection,thenitscomplementaryisrotatingintheoppositedirection.

z zc

≡ = x− iyzc z∗

z

Thecomplementarycorrelationofbivariatevariables

Thecomplementarycovariancebetween and is

Thecomplementaryorreflectionalcorrelationis

where

z w

C wz c ===

E[( w]z∗)∗

E[(x+ iy)(g+ ih)]E[xg] − E[yh] + i(E[xh] + E[yg]

≡ρ wzcC wzc

Cz cz cCww− −−−−−−√

=Cz cz c Czz

Auto-covarianceandvarianceellipses

Thespecialcasesofstandardandcomplementaryautocovariancesofaphysicalvectorwithitselfcanberelatedtotheconceptofvarianceorstandarddeviationellipses.


Thespecialcasesofstandardandcomplementaryautocovariancesofaphysicalvectorwithitselfcanberelatedtotheconceptofvarianceorstandarddeviationellipses.

Forabivariater.v. ,thefixedangle betweentherealaxispositivedirectionandtheso-calledmajoraxisofthestandarddeviationellipseis

while definesthedirectionoftheso-calledminoraxis.

z θM

= Arg [ ] = arctan [ ] ,θM12

C zz c12

2E[xy]E[ ] − E[ ]x2 y2

+ π/2θM


Thevariances and ofthebivariatevariablealongthemajorandminoraxes,respectively,aregivenby

AnticipatingLecture5, and arethe2eigenvaluesofthecrosscovariancematrixoftheCartesiancomponent,whiletheassociatedeigenvectorsare and

.

a2v b2v

a2v =

=

[ + | |]12Czz C zz c

{E[ ] + E[ ] + } ,12

x2 y2 (E[ ] − E[ ] + 4(E[xy]x2 y2 )2 )2− −−−−−−−−−−−−−−−−−−−−−−√

b2v =

=

[ − | |]12Czz C zz c

{E[ ] + E[ ] − } .12

x2 y2 (E[ ] − E + 4(E[xy]x2 y2)2 )2− −−−−−−−−−−−−−−−−−−−−−√

a2v b2v 2 × 2

[cos , sin ]θM θM[− sin , cos ]θM θM


Theratiooftheabsolutevalueofthecomplementaryautocorrelationtotheabsolutevalueoftheautocorrelationisameasureoftheabsolutelinearityofthestandarddeviationellipse

If theellipseisflatandif theellipseisacircle. isarelativeofthemorecommonlyknowneccentricityparameteroftheellipsewhichis

SeeLillyandGascard(2006)andLillyJ,OlhedeS.(2010).

|λ| = = .C zz c

| |Czz

−a2v b2v

+a2v b2v

|λ| = 1 |λ| = 0 λ

e = .1 −b2v

a2v

− −−−−−√

http://www.nonlin-processes-geophys.net/13/467/2006/

http://dx.doi.org/10.1109/TSP.2009.2031729

Standarddeviationorvarianceellipses

Ifyourcomplexsignalis whichrepresentsanellipse,thenthemajoraxisandminorsemi-axisofthestandarddeviationellipseare and

.

z(t) = [a cos(t) + i sin(t)]eiθ

= < aav /2a2− −−−

√= < bbv /2b2

− −−−√


Example:Meancurrents(blue)andstandarddeviationellipses(lightblue)ofnearbottomcurrentsfromtheRAPIDWAVEexperiment.Orangeaxesshowtopographygradient.SeeHughesetal.2013

http://dx.doi.org/10.1175/JTECH-D-12-00149.1


Example:Complementaryautocorrelationof10-mwindsatalllocationsfromECMWFreanalyses.


Thestandardellipseisastatisticaldescriptionofthevarianceofabivariatevariableanddoesnotmeanthatanyunderlyingvariabilityisactuallyellipticalintime.StandarddeviationellipsescanbecomputedfromanumberofindividualpairsofCartesiancomponentsofabivariatequantitywithoutthesepointsactuallyformingconsecutivetimeseries.


Thestandardellipseisastatisticaldescriptionofthevarianceofabivariatevariableanddoesnotmeanthatanyunderlyingvariabilityisactuallyellipticalintime.StandarddeviationellipsescanbecomputedfromanumberofindividualpairsofCartesiancomponentsofabivariatequantitywithoutthesepointsactuallyformingconsecutivetimeseries.

Example:VarianceellipsesfromdriftersfromLumpkinandJohnson2013

http://dx.doi.org/10.1002/jgrc.20210

Practicalsession

Pleasedownloaddataatthefollowinglink:

PleasedownloadtheMatlabcodeatthefollowinglink:

MakesureyouhaveinstalledandtestedthefreejLabMatlabtoolboxfromJonathanLillyatwww.jmlilly.net/jmlsoft.html

https://www.jmlilly.net/jmlsoft.html

Lecture 2: Covariance and correlation · Lecture 2: Outline 1. Covariance & correlation 2. Lagged...

Documents

Transcript of Lecture 2: Covariance and correlation · Lecture 2: Outline 1. Covariance & correlation 2. Lagged...