EME: Information Theoretic views of emergence and self-organisation

Continuing the search for

useful definitions of

emergence and self organisation

EME : 2

The plot so far …

• Researchers agree on just two things– We need a consistent definition of emergence

– We don’t have one

• Statistical complexity and mutual information help us to explore definitions in information theoretic terms– So long as we can extract a Shannon-compliant

representation of information in a system

• So, now we can give formal definitions of emergence and self-organisation– And to explore the relationships among them…

EME : 3

Entropy: recap and use

• Joint entropy • Measure of information in joint systems

• Conditional entropy (mutual information)• Measure of information in a system relative to that in another

– Evolution increases mutual information between a system and its environment

• Entropy can be compared – between systems, or, for one system:

• over space, or • over time

• Entropy measures to find how much emergent information is a direct consequence of low-level information

• If we can encode the information appropriately – But that does not help to define emergence

EME : 4

Phase transitions: recap and use

• System behaviour is most complex at phase transitions– Many emergent and self-organising phenomena are

associated with complexity

• First order transitions generate latent heat, entropy, and complex behaviour

• Turbulence and related behaviours

• Second order transitions are not abrupt changes in entropy

• But important in identifying complex behaviour (Ising model)

• Phase transitions are related to emergence– but do not help define it

EME : 5

Recognising emergence and self-organisation

• Emergence is essentially about the appearance of patterns at larger/higher scales than components

• Self-organisation is essentially about the “spontaneous” appearance of structure over time

• To define emergence and self-organisation, we need to identify pattern or structure, and compare it across space or time– Complexity measures can be used in their definition

• So long as the measures distinguish complexity from randomness

EME : 6

Emergence and complexity

• Emergence and self-organisation can be defined using statistical complexity– Used by Crutchfield (1994), and subsequently by Shalizi

(2001)

– Based on the ability of a measurable system (e.g., an ε-machine) to reproduce the statistical information characteristics of an actual system

• Essentially, looking for a robust way to identify and measure structure, or pattern

Crutchfield, The Calculi of Emergence, Physica D, 75, 1994Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf Prokopenko et al, An information-theoretic primer…, Advances in Complex Systems, 2006

EME : 7

Argument underlying definitions

• Exact models of the structure inherent in systems are uncomputable in general, and uninformative

• Needs a UTM, or a simulated universe, and precise measurement…

• Much low-level detail is unnecessary to approximate higher-level behaviour

• As in thermodynamics

• Approximations improve when statistical analysis of repeated measurement and recalculation is used

• Best representation of average properties

• “Given the ubiquity of noise in nature, this is a small price to pay” Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 8

Shalizi’s definition of emergence

A derived process is emergent if it has a greater predictive efficiency than the process it derives

from

• Predictive efficiency e relates excess entropy, E and statistical complexity, Cμ :

e = E / Cμ

– Cμ is amount of memory of past stored in a process or system

– E is mutual information between system’s past and future• amount of apparent information about past stored in observed

behaviour

– E ≤ Cμ , so 0 ≤ e < 1 • e = the fraction of historical memory stored in the process which does

“useful work” in telling us about the future– Cμ = 0 if no complexity, so no predictive interest, so set e = 0

• perfect predictive efficiency “unlikely”

Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 9

Intuitive interpretation of Shalizi’s definition

A derived process is emergent if it has a greater predictive efficiency than the process it derives from

• For process X that emerges from process Y , ex > ey

• e = E / Cμ

• So an emergent system has • either lower statistical complexity

• or greater excess entropy

• Compared to system from which it emerges, an emergent system:–either has fewer irreducible complex or random components

–or its past determines its future more completely

EME : 10

A informal check of Shalizi’s definition

• GoL glider vs CA with GoL rules– Glider’s past predicts its future absolutely

– CA rule prediction depends on attractor space

• Slime mould: Dicty slug vs Dicty amoebae– Slug moves to favourable site and fruits

– Amoeba may stay alive, become spore/cyst on its own, may become a slug pre-stalk or a slug pre-spore

• In these simple emergent behaviours predictive efficiency seems greater than the underlying system– … but what about all the others?

– Exhaustive studies are not yet being done

EME : 11

Excess entropy calculation

• How do we calculate predictive efficiency, e = E / Cμ ?

• Excess entropy (E) is mutual information between a system’s pasts and future– A system cannot have more mutual information than that in

either the past or future states

• System’s pasts are represented by causal states– Equivalence class of input states that all have same

conditional probability distribution of outputs

Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 12

Intuition for Causal States

• Causal states are produced by the modelling process– They are observations of the “state machine”, not the state

machine itself• Recreates a minimal model with equivalent statistical

behaviour

• Uses a series of spatial or temporal measurements

• Causal states are not states of the actual system– Recall lecture 12: logistic process

• 47 deduced causal states

Crutchfield, The Calculi of Emergence, Physica D, 75, 1994

EME : 13

ε-machines and causal states

• From Crutchfield, ε-machines used to extract causal states from discrete time series

• Discrete measurements are approximate indicators of a hidden environment with finite accuracy, ε

• An ε-machine detects causal states by identifying pasts that predict the future– Based on computation theory and prediction of bit-strings

• Various algorithms for reconstructing an ε-machine • Shalizi and Crutchfield give example calculations

Crutchfield, The Calculi of Emergence, Physica D, 75, 1994Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 14

ε-machine: optimal model of complexity

• We cannot measure complexity directly– An ε-machine approximates the system’s information

processing

– An ε-machine is the smallest possible explanatory model• Ockham’s razor – include only what is needed

– Maximal accounting for structure • Basic tenet of science – obtain prediction of nature

• Achieve appropriate balance between order and randomness

• Compromise between:– Smallest model with huge error ε and little prediction

– Model with minimal error that differs minimally from system

Crutchfield, The Calculi of Emergence, Physica D, 75, 1994

EME : 15

Can we “calculate” emergence?

• It is hard to calculate causal states in general

• In Markovian behaviour (e.g. thermodynamics) any pre-state is a causal state– Standard entropy formulae in thermodynamics and

statistical mechanics allow calculation of predictive efficiency

– Not surprisingly, the predictive efficiency of macro-scale is considerably better than that of the micro-scale

• 1020 independent gas particles with almost no predictive power give rise to predictable emergent behaviour expressible in a small number of macro-variables

• So, by this measure, thermodynamics is emergentShalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 16

A note on calculations for thermodynamics

• Shalizi’s predictive efficiency calculation uses recognised facts and formulae of thermodynamics and statistical mechanics– Many assumptions about quantities, accuracy, etc.

• e.g. assumes macro-measurement error factor <10-15

– Finds sub-nano-second predictive efficiency of micro-scale is high, but rapidly reduces over time

• Most information in statistical mechanics is irrelevant to thermodynamic macro-state

– But the cumulated assumptions and errors make the figures at best debatable

EME : 17

Emergent structures and ε-machines

• An ε-machine summarises the dynamics of a process

• The ε-machine could be divided in to sub-machines and transitions among them– At each time step, causal and previous state may or may

not be in same sub-machine

• If successive states are in one sub-machine, this is an emergent process of the process approximated by the ε-machine – Because knowing sub-machine reduces statistical

complexityShalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 18

Shalizi’s definition of self-organisation

• An increase in statistical complexity is a necessary condition for self-organisation

• A more realistic definition, relying only on statistical complexity, not excess entropy, causal states, etc.

– Successive states in different sub-machines of an ε-machine – Optimal prediction requires more information

• There are more irreducible complex or random components

• For non-stationary processes, distribution of causal states changes over time– Statistical complexity is measured as a function of time, Cμ(t)

– A system that spontaneously moves from uniform to periodic behaviour exhibits an increase in Cμ(t)

Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 19

Thermodynamics: a box of gas

• In the micro-state, all particles are independent, and all behaviours equally likely– What happens in one time or spatial unit has no later effect

• Statistical complexity remains constant, and low

• Thermodynamic micro-process is not self-organising

• Note that, at least in this case, the definitions allow emergence without self-organisation– But thermodynamics is perhaps an extreme case

Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 20

Summarising the definitions

• Emergence – Informally: behaviour observed at one scale is not apparent

at other scales

– Formally: processes have better predictive efficiency than those from which they emerge

• Lower statistical complexity or greater explanatory power of the past

• Self-organisation– Informally: structures that emerge without systematic

external stimuli

– Formally: processes with an increase in statistical complexity over time

EME : 21

Intuitions on flocking

• Individual birds are probably not independent, but follow simple local rules– For a collection of birds, statistical complexity > 0 but no

where near 1

• If a bird could see the whole flock it would see complex dynamics– For a flock of birds, statistical complexity is closer to 1

• The flock is a self-organised collection of birds

• When we recognise self-organisation we label it as an emergent behaviour at a higher scale– “Flock” denotes an emergent pattern of behaviour

• Predictive efficiency of flock is better than that of a group of birds

EME : 22

Reconciling the definitions

• Natural systems can self-organise– Independent of observation

• Systems that self-organise are studied by “cognitively-limited observers”– Seeking descriptions that have good predictive ability

– Patterns are at a more abstract level than self-organising system elements

• An abstract description with enhanced prediction becomes an emergent process of the original behaviour

EME : 23

So does self-organisation imply emergence?

• Shalizi states that he knows of no reason against self-organising non-emergent systems, but …

• Emergence may be a precondition of detectable self-organisation – In practice, when humans recognise self-organisation, they

identify the abstract result at an emergent process

• At least some of what humans call “noise” may be unrecognisable complex self-organisation– We know that chaos has complex structure …

Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

EME : 24

Self-organisation and emergence

Not self-organising

Self-organising

Not emergent Emergent

Not interesting Thermodynamics

Possibly verycomplex

self-organisation

EMER interest:Biology

Interesting physics …

Improved modeling and understanding

EME : 25

So where are we now?

• Statistical complexity gives us a nice definition of self-organisation– And an intuition for how self-organisation and emergence

are related

• Statistical complexity gives us a possible definition of emergence– In reality, predictive efficiency is hard to estimate with any

confidence• The definition does clarify features of emergence

• The definitions make assumptions about measurement and discrete spatial or temporal series

EME : 26

A note on discrete measurement

• All work on information theoretic definitions of emergence and self-organisation assumes discrete temporal or spatial observation

• Shalizi discusses using causal states with continuous trajectories:– No current mathematics of continuous conditional

probability

– Continuous entropy exists but depends on co-ordinates• Entropy changes if e.g., distance is measured in inches or

metres

– Reconstruction from data is hard, and seriously affected by measurement problems

• Not unlike problems of using PDEs for explanatory models of biological systems

EME : 27

Open questions

Observers and intrinsic emergence:• If emergence is a precondition of detectable self-

organisation, then it must be possible to observe the system

• In intrinsic emergence, the observer is a sub-process of the system

• monitors environment through sensors to construct an imperfect behavioural model

– Observer makes predictions of future • which, because internal, can interfere with that future

• But, are intrinsic observation and self-organisation compatible?– does observer have to be outside a self-organising system?

EME : 28

Open questions

How to determine agency in self-organisation:

• Organisation assumes that structure emerges over time

• Self-organisation assumes that there is no consistent external agency in the appearance of structure

• But, how can we distinguish self-organisation from organisation by external agency?– A complex system in a complex environment, where

external inputs may be stochastic, but might also be very complex

– Biological systems are usually in this category

EME : 29

And one last open question

• Informally, research has identified level, scope and resolution as relevant to emergence– Statistical complexity and mutual information assume that

processes or systems are well-defined– Statistical analysis (e.g., for causal states or PDEs) also

assumes that scope is known

• What happens if the chosen bounds exclude a key component of the emergence or self-organisation?

• What happens if slightly widening the bounds would reveal that emergence or self-organisation was an artifice of the scope?

• We have raised as many questions as we have solved