Exploring and exploiting intrinsic synergies of...

Exploringandexploitingintrinsicsynergiesofmultivariatesystems

FernandoE.RosasResearchFellow

CentreofComplexityScience

DepartmentofMathematics

DepartmentofElectricalandElectronicEngineering

ImperialCollegeLondon

!!!!!Imperial)College)London)))Centre)for)Complexity)Science)

!

!

Complexity

Science

Acknowledges:

Thisworkistheresultofthefortunatecollaborationwith:

!2

PedroA.M.MedianoDepartmentofComputingImperialCollegeLondon HenrikJ.Jensen

DepartmentofMathematicsImperialCollegeLondon

MartinUgarteDepartmentofComputerScienceUniversitéLibreBruxelles

MichaelGastparSchoolofComputerandCommunicationSciencesEPFL

BorzooRassouliSchoolofComputerScienceandElectronicEngineeringUniversityofEssex

DenizGündüzDepartmentofE&EEngineeringImperialCollegeLondon

Acknowledges:MarieSlodowska-CurieActions

!3

Objectives:

(a) tounderstandandexploitintrinsicstatisticalsynergiesofmultivariatesystems

(b) applythistodataprivacyandneuraldataanalysis

!4

Today’smenu

1.Self-organisation

2.O-information

3.Dataprivacy

4.Summaryandcurrentwork

!5

1.Self-organisation:whatisapattern

Whatisapattern/structure?

Shannon—>compressibilityofanstatisticalsourceRegularities/interdepedencies(i.e.deviationsfromstatisticalindependence)

Organisationasstatisticalinterdependency

!6

1.Whatisself-organization?

Attractor:destination/resultoftheevolutionLinearrecurrence/evolutionissimplesimple/uninterestingattractorsNon-linearevolutioninteresting/strangeattractors

Challenge:Isitpossibletorelatetheattractorspropertiestoorganizationproperties?

!7

1.Whatisself-organization?

Attractor:destination/resultoftheevolutionLinearrecurrence/evolutionissimplesimple/uninterestingattractorsNon-linearevolutioninteresting/strangeattractors

Idea:high-ordercorrelationsallowtodistinguishqualitativelydifferentarrangements…

redundancy! synergy!

1.Self-organisation:ourapproach

Approachtostudyspontaneouscreationofcorrelations:

—Consideranflatinitialdistribution(independence)

—Computeit’stimeevolution()usingthecorresponding“masterequation”

—Iterateformanysteps,oruntilitreachastationarystate.

—Analysethe“structureofthecorrelations”of.

Whystartfromanuniformdistribution?Becauseanycorrelationfoundisduetotheevolutionandnottheinitialcondition.

!8

µ0

µt

µtst+1 = µtst

(X1tst , . . . , X

Ntst)

1.Self-organisation:dynamicsassculpture…

!9

“TheAtlas”MichelangeloCirca1530-1534

“TheAwakeningSlave”MichelangeloCirca1520-1523

�t

�0t

1.Self-organisation:entropyisnotenough

Iseasytoprovethat,becauseofdeterminism,thejointShannonentropyofthesystemisnon-increasing.

Thisnotsufficienttoguaranteeself-organization!!(example:fixpointattractors)

!10

H(X1t+1, . . . , X

Nt+1) H(X1

t , . . . , XNt )

InformationisbeingdissipatedIttakesmoreinformationtospecifyarandomstate

thanapointinanattractor

1.Self-organisation:entropyisnotenough

Iseasytoprovethat,becauseofdeterminism,thejointShannonentropyofthesystemisnon-increasing.

Thisnotsufficienttoguaranteeself-organization!!(example:fixpointattractors)

!11

H(X1t+1, . . . , X

Nt+1) H(X1

t , . . . , XNt )

InformationisbeingdissipatedIttakesmoreinformationtospecifyarandomstate

thanapointinanattractor

Keyidea:Thedestructionofinformation

cancreatecorrelations!!

H(X) H(Y )

H(X) H(Y )

I(X;Y )

1.Self-organisation:proposedframework

!12

Ourproposedtotalentropydecomposition:

F.Rosas,P.A.M.Mediano,M.Ugarte,H.J.Jensen,“Aninformation-theoreticapproachtoself-organization:emergenceofcomplexinterdependenciesincoupleddynamicalsystems”,Entropy20,no.10(2018):793.

"Residualentropy”

“Bindingentropy”


Decompositionbysharingmodes:

!13

bymeasuringiagentscanguessthestateofn…


Decompositionbysharingmodes:

where

!14

1.Self-organisation:proofofconcept

Rule232

!15


Rule232

!16


Rule30

!17


Rule30

!18


!19


!20

0 5 10 15

0.0

0.5

1.0

yL

Rule 232: Redundancy

0 5 10 15

0.0

0.5

1.0

Rule 30: Moderate synergy

0 5 10 15

0.0

0.5

1.0

Number of cells (L)

yL

Rule 90: Strong synergy

0 5 10 15

0.0

0.5

1.0

Number of cells (L)

Rule 106: Mixed profile

Rule106

!21

Today’smenu

1.Self-organisation

2.O-information

3.Dataprivacy


2.O-information:fundamentals

1.Interaction-information:

2.Totalcorrelation:

3.Dualtotalcorrelation:

Stateofaffairs:

(+)TCandDTCaremetricsofglobalcorrelationstrength.

(-)TC=0ifandonlyifDTC=0.Besidesthat,theirrelationshipisunclear.

(+)

(+)=TC-DTC.

(-)Themeaningoftheinteraction-informationforn>3isunclear.

!22

ispositiveforredundantsystems,andnegativeforsynergisticones.I(X1;X2;X3) = I(X1;X3) + I(X2;X3)� I(X1X2;X3)

I(X1;X2;X3)

TC(Xn) =nX

j=1

H(Xj)�H(Xn)

DTC(Xn) = H(Xn)�nX

j=1

H(Xj |Xn�j)

I(X1;X2; . . . ;Xn) = �X

�✓{1,...,n}

(�1)|�|H(X�)

2.O-information:fundamentals

1.Interaction-information:

2.Totalcorrelation:

3.Dualtotalcorrelation:

!23

HowcouldI“fix”theinteraction-infofor

anyn??

TCorDTC?thatisthequestion…

Theinformation-theoreticHamlet

TC(Xn) =nX

j=1

H(Xj)�H(Xn)

DTC(Xn) = H(Xn)�nX

j=1

H(Xj |Xn�j)

I(X1;X2; . . . ;Xn) = �X

�✓{1,...,n}

(�1)|�|H(X�)

2.O-information:thetwofacesofinterdependency

Considerasystemwhere,describedbyap.d.f..

!24

nX

j=1

log |Xj |

N (Xn) =X

j

log |Xj |�H(Xn)

H(Xn) = �X

xn

pxn log pxn

Negentropy

Entropy

Xn = (X1, . . . , Xn) Xj 2 Xj pXn(xn)

H(Xn)

N (Xn)


Considerasystemwhere,describedbyap.d.f..

!25

nX

j=1

log |Xj |

Negentropy

Entropy

X1, X2 2 {0, 1}

pX1X2(0, 0) = pX1X2(1, 1) = 0

pX1X2(0, 1) = pX1X2(1, 0) = 1/2

H(X1X2) = N (X1X2) = 1

(0, 0) (0, 1)

(1, 0) (1, 1)+

+

Example:

H(Xn) = �X

xn

pxn log pxn

N (Xn) =X

j

log |Xj |�H(Xn)

Xn = (X1, . . . , Xn) Xj 2 Xj pXn(xn)

H(Xn)

N (Xn)


!26

N (X5)N (X4)N (X3)N (X2)N (X1)

TC

N (X6)

nX

j=1

log |Xj |

H(Xn) = �X

xn

pxn log pxn

N (Xn) =nX

j=1

N (Xj) + TC

IndividualconstraintsCollectiveconstraints

Marginalnegentropy: N (Xj) = log |Xj |�H(Xj)

Collectiveconstraints: TC(Xn) = N (Xn)�nX

j=1

N (Xj) =nX

j=1

H(Xj)�H(Xn)


!27

N (X5)N (X4)N (X3)N (X2)N (X1)

TC

N (X6)

nX

j=1

log |Xj |

H(Xn) = �X

xn

pxn log pxn

N (Xn) =nX

j=1

N (Xj) + TC

(0, 0) (0, 1)

(1, 0) (1, 1)+

+Example: (0, 0) (0, 1)

(1, 0) (1, 1)+ +

N (X1X2) = 1

TC(X1X2) = 1

N (X1X2) = 1

TC(X1X2) = 0

IndividualconstraintsCollectiveconstraints


!28

R1R2 R3 R4 R5

N (X5)N (X4)N (X3)N (X2)N (X1)

TC

DTC

N (X6)

R6

N (Xn) =nX

j=1

N (Xj) + TC

H(Xn) =nX

j=1

Rj +DTC

nX

j=1

log |Xj |

Privaterandomness

Collectiveconstraints

Sharedrandomness

Individualconstraints

Residualentropies:

Bindingentropy: DTC(Xn) = H(Xn)�nX

j=1

H(Xj |Xn�j)

Rj = H(Xj |Xn�j)


!29

R1R2 R3 R4 R5

N (X5)N (X4)N (X3)N (X2)N (X1)

TC

DTC

N (X6)

R6

N (Xn) =nX

j=1

N (Xj) + TC

H(Xn) =nX

j=1

Rj +DTC

nX

j=1

log |Xj |

Privaterandomness


Sharedrandomness

Example: X1 = X2 = X3

TC = 2 > DTC = 1

X3 = X1 �X2

TC = 1 < DTC = 2




!30

R1R2 R3 R4 R5

N (X5)N (X4)N (X3)N (X2)N (X1)

TC

DTC

N (X6)

R6

N (Xn) =nX

j=1

N (Xj) + TC

H(Xn) =nX

j=1

Rj +DTC

nX

j=1

log |Xj |

Privaterandomness


Sharedrandomness

Example: X1 = X2 = X3

TC = 2 > DTC = 1

X3 = X1 �X2

TC = 1 < DTC = 2



2.O-information:definition

Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription

• Ifitisshortertodescribetheallowedstates.

• Ifitisshortershortertodescribetheconstraints.

!31

Definition

⌦(Xn) = TC(Xn)�DTC(Xn)O-information:

⌦(Xn) > 0

⌦(Xn) < 0

F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019





!32

Definition


⌦(Xn) > 0

⌦(Xn) < 0

Example:


X1 = X2 = · · · = Xn

TC = n� 1 > DTC = 1 TC = 1 < DTC = n� 1

Xn = X1 �X2 � · · ·�Xn�1

F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019





!33

Definition


⌦(Xn) > 0

⌦(Xn) < 0

Definition

1. Asystemisredundancy-dominatedif

2. Asystemissynergy-dominatedif

⌦(Xn) � 0

⌦(Xn) 0


Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription!



Note:

1.n=2:,becausei.e.sharedrandomnessisequaltopredictability

!34

Definition


⌦(Xn) > 0

⌦(Xn) < 0

⌦(X1X2) = 0 TC(X1X2) = DTC(X1X2) = I(X1;X2)


Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription!



Note:

1.n=2:,becausei.e.sharedrandomnessisequaltopredictability

2.n=3:—>interaction-information!

!35

Definition


⌦(Xn) > 0

⌦(Xn) < 0

⌦(X1X2) = 0 TC(X1X2) = DTC(X1X2) = I(X1;X2)

⌦(X1X2X3) = I(X1;X2)� I(X1;X2|X3) =: I(X1;X2;X3)

2.O-information:(1)itissumoftripleinteraction-informations

• DecompositionsfortheTC,DTCando-information:

!36

O-informationisanaggregationoftriplemulti-informations!!



!37

2.O-information:(2)characterizationofextremevalues


• Upperandlowerbounds:

• Characterizationofuniqueextremesoftheo-information:

!39

Maximum: X1 = X2 = · · · = Xn Minimum: Xn =n�1X

j=1

Xj (mod k)

2.O-information:(3)nullvalueofnon-overlappingpairwiseinteractions

• O-informationofindependentsubgroupsisadditive:

• Ifasystem’sisdecomposableindisjointpairwiseinteractionsthen

• Theconverseisnottrue!!(synergiesandredundanciescanceleachother)

• Localo-informationcangiveafine-graineddescriptionofthesystem…

!40

⌦(Xn,Y m) = ⌦(Xn) + ⌦(Y m)pXn,Y m(xn,ym) = pXn(xn)pY m(ym)

⌦(Xn) = 0

!i,j = I(Xi;Xj ;Xn�i�j)

2.O-information:(4)valueimpliesboundsoverdifferentscales

• ThevalueoftheO-informationprovideconstraintsovertheinterdependenciesofsubgroups!

•

!41

2.O-information:(5)redundancyoftrees

• O-informationisnon-negativeingraphicalmodelswiththreestructure!

• ForNaiveBayes,theo-informationisgivenby

• ForMarkovchains,theo-informationis

!42

⌦(Xn) = C(Xn2 ) � 0

1 2 n…3

2.O-information:(6)synergyispervasiveinlargesystems

• Themeanvalueoftheo-informationoverrandomdistributionsgrowsnegative!

!43

2 4 6 8

�4

�2

0

System size n

Gaussian systems

2 4 6 8

�0.8

�0.6

�0.4

�0.2

0

System size n

⌦

Binary systems

AnalyticalMonte CarloApproximation

(Williams & Beer, 2010)(Williams & Beer, 2010)(Williams & Beer, 2010)(Williams & Beer, 2010)

2.O-information:relationshipwithstatisticalmechanics

1.Connectedinformation(Schneidmanetal.2003):

Stateofaffairs:

(+)Interestingconnectionwithstatisticalmechanics.

(+)Intuitiveinterpretation.

(-)Veryhardtocompute…

!44

Con(Xn) =

Pnk=1 H(Xj)�H(Xn

maxent)

TC(Xn)

2.O-information:relationshipwithstatisticalmechanics

1.Connectedinformation(Schneidmanetal.2003):

• Hamiltonianwithhigh-ordertermstendtohavenegativeO-information

!45

Con(Xn) =

Pnk=1 H(Xj)�H(Xn

maxent)

TC(Xn)

2.O-information:analysispipeline

!46

Procedure

1.Computeandasmetricsofglobalcorrelationstrength.

2.Computetofinddominantglobalbehaviour(redundancyorsynergy).

3.Studythelocalo-informationterms,i.e.foralliandj,asameasureoflocalisedbehaviour.

⌦(Xn)

I(Xi;Xj ;Xn�i,�j)

B(Xn) C(Xn)

2.O-information:Casestudyonbaroquemusicscores

DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)

i)choralesforfourvoicesbyJ.S.Bach(1685–1750)

!47




Dataformat:foursynchronoustimeseries,alphabetof13values(12tones+silence)

Database:300~chorales,43kfour-notechords

!48




(43kfour-notechords)

ii)Op.1,3,4,5and6ofA.Corelli(1653–1713)(80kfour-notechords)

!49



!50



!51

Synergyallowsthecoexistenceoflocalindependencyandglobalcoordination

!52

Today’smenu

1.Self-organisation

2.O-information

3.Dataprivacy


3.Synergyanddataprivacy:keyideas

Privacyfunnel:maximizecorrelationwithfeatureofinterest,whilekeepingprivatestuffsecure

!53


Privacyfunnel:maximizecorrelationwithfeatureofinterest,whilekeepingprivatestuffsecure

However,itisoftenthecasewherethefeatureofinterestisunknown

ButtheDPinequalitymakeitseemunfeasible…

!54


Idea:makeYindependentofeachcoordinateofX,butcorrelatedwiththewhole!!

!55


Idea:makeYindependentofeachcoordinateofX,butcorrelatedwiththewhole!!

!56

3.Synergyanddataprivacy:mainresults

!57

3.Synergyanddataprivacy:mainresults

!58

Tolearnmoreaboutthis:1. B.Rassouli,F.E.Rosas,andD.Gunduz."DataDisclosureunderPerfectSamplePrivacy."Submitted

toIEEETransactionsinInformationForensicsandSecurity(TIFS),underreview.arXivpreprintarXiv:1904.01711(2019).

2. B.Rassouli*,F.E.Rosas*,andD.Gunduz,"LatentFeatureDisclosureunderPerfectSamplePrivacy."In2018IEEEInternationalWorkshoponInformationForensicsandSecurity(WIFS),pp.1-7.IEEE,2018.

!59

Today’smenu

1.Self-organisation

2.O-information

3.Dataprivacy


Furtherreading:

!60

1. B.Rassouli*,F.E.Rosas*,andD.Gunduz,"LatentFeatureDisclosureunderPerfectSamplePrivacy."In2018IEEEInternationalWorkshoponInformationForensicsandSecurity(WIFS),pp.1-7.IEEE,2018.

2. B.Rassouli,F.E.Rosas,andD.Gunduz."DataDisclosureunderPerfectSamplePrivacy."SubmittedtoIEEETransactionsinInformationForensicsandSecurity(TIFS),underreview.arXivpreprintarXiv:1904.01711(2019).

3. F.Rosas,P.A.M.Mediano,M.Ugarte,H.J.Jensen,“Aninformation-theoreticapproachtoself-organization:emergenceofcomplexinterdependenciesincoupleddynamicalsystems”,Entropy20,no.10(2018):793.

4. F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019

PD:currentwork

Drivingquestions:1.Isthebrainasynergisticmulti-agentsystem?2.Isconsciousnessrelatedwithhigh-orderstatistics?3.CanPIDprincipleshelpustounderstandtheeffectofpsychedelicdrugs?

!61

!!!!!Imperial)College)London)))Centre)for)Complexity)Science)

!

!

Complexity

Science

Acknowledges:

Thisworkistheresultofthefortunatecollaborationwith

- PedroMediano(ImperialCollegeLondon)- HenrikJ.Jensen(ImperialCollegeLondon)- MichaelGastpar(EPFL)- MartinUgarte(UniversitélibredeBruxelles)- BorzooRasouli(UniversityofEssex)- DenizGündüz(ImperialCollegeLondon)

SpecialthanksfortheMarieSlodowska-CurieprogramfromH20202andtheEuropeanCommission.

!62

!63

Thankyou!

Contactinformation:

FernandoRosas

[email protected]

Exploring and exploiting intrinsic synergies of...

Documents

Transcript of Exploring and exploiting intrinsic synergies of...