Post on 29-Dec-2015
EME: Information Theoretic views of emergence and self-organisation
Continuing the search for
useful definitions of
emergence and self organisation
EME : 2
The plot so far …
• Researchers agree on just two things– We need a consistent definition of emergence
– We don’t have one
• Statistical complexity and mutual information help us to explore definitions in information theoretic terms– So long as we can extract a Shannon-compliant
representation of information in a system
• So, now we can give formal definitions of emergence and self-organisation– And to explore the relationships among them…
EME : 3
Entropy: recap and use
• Joint entropy • Measure of information in joint systems
• Conditional entropy (mutual information)• Measure of information in a system relative to that in another
– Evolution increases mutual information between a system and its environment
• Entropy can be compared – between systems, or, for one system:
• over space, or • over time
• Entropy measures to find how much emergent information is a direct consequence of low-level information
• If we can encode the information appropriately – But that does not help to define emergence
EME : 4
Phase transitions: recap and use
• System behaviour is most complex at phase transitions– Many emergent and self-organising phenomena are
associated with complexity
• First order transitions generate latent heat, entropy, and complex behaviour
• Turbulence and related behaviours
• Second order transitions are not abrupt changes in entropy
• But important in identifying complex behaviour (Ising model)
• Phase transitions are related to emergence– but do not help define it
EME : 5
Recognising emergence and self-organisation
• Emergence is essentially about the appearance of patterns at larger/higher scales than components
• Self-organisation is essentially about the “spontaneous” appearance of structure over time
• To define emergence and self-organisation, we need to identify pattern or structure, and compare it across space or time– Complexity measures can be used in their definition
• So long as the measures distinguish complexity from randomness
EME : 6
Emergence and complexity
• Emergence and self-organisation can be defined using statistical complexity– Used by Crutchfield (1994), and subsequently by Shalizi
(2001)
– Based on the ability of a measurable system (e.g., an ε-machine) to reproduce the statistical information characteristics of an actual system
• Essentially, looking for a robust way to identify and measure structure, or pattern
Crutchfield, The Calculi of Emergence, Physica D, 75, 1994Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf Prokopenko et al, An information-theoretic primer…, Advances in Complex Systems, 2006
EME : 7
Argument underlying definitions
• Exact models of the structure inherent in systems are uncomputable in general, and uninformative
• Needs a UTM, or a simulated universe, and precise measurement…
• Much low-level detail is unnecessary to approximate higher-level behaviour
• As in thermodynamics
• Approximations improve when statistical analysis of repeated measurement and recalculation is used
• Best representation of average properties
• “Given the ubiquity of noise in nature, this is a small price to pay” Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 8
Shalizi’s definition of emergence
A derived process is emergent if it has a greater predictive efficiency than the process it derives
from
• Predictive efficiency e relates excess entropy, E and statistical complexity, Cμ :
e = E / Cμ
– Cμ is amount of memory of past stored in a process or system
– E is mutual information between system’s past and future• amount of apparent information about past stored in observed
behaviour
– E ≤ Cμ , so 0 ≤ e < 1 • e = the fraction of historical memory stored in the process which does
“useful work” in telling us about the future– Cμ = 0 if no complexity, so no predictive interest, so set e = 0
• perfect predictive efficiency “unlikely”
Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 9
Intuitive interpretation of Shalizi’s definition
A derived process is emergent if it has a greater predictive efficiency than the process it derives from
• For process X that emerges from process Y , ex > ey
• e = E / Cμ
• So an emergent system has • either lower statistical complexity
• or greater excess entropy
• Compared to system from which it emerges, an emergent system:–either has fewer irreducible complex or random components
–or its past determines its future more completely
EME : 10
A informal check of Shalizi’s definition
• GoL glider vs CA with GoL rules– Glider’s past predicts its future absolutely
– CA rule prediction depends on attractor space
• Slime mould: Dicty slug vs Dicty amoebae– Slug moves to favourable site and fruits
– Amoeba may stay alive, become spore/cyst on its own, may become a slug pre-stalk or a slug pre-spore
• In these simple emergent behaviours predictive efficiency seems greater than the underlying system– … but what about all the others?
– Exhaustive studies are not yet being done
EME : 11
Excess entropy calculation
• How do we calculate predictive efficiency, e = E / Cμ ?
• Excess entropy (E) is mutual information between a system’s pasts and future– A system cannot have more mutual information than that in
either the past or future states
• System’s pasts are represented by causal states– Equivalence class of input states that all have same
conditional probability distribution of outputs
Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 12
Intuition for Causal States
• Causal states are produced by the modelling process– They are observations of the “state machine”, not the state
machine itself• Recreates a minimal model with equivalent statistical
behaviour
• Uses a series of spatial or temporal measurements
• Causal states are not states of the actual system– Recall lecture 12: logistic process
• 47 deduced causal states
Crutchfield, The Calculi of Emergence, Physica D, 75, 1994
EME : 13
ε-machines and causal states
• From Crutchfield, ε-machines used to extract causal states from discrete time series
• Discrete measurements are approximate indicators of a hidden environment with finite accuracy, ε
• An ε-machine detects causal states by identifying pasts that predict the future– Based on computation theory and prediction of bit-strings
• Various algorithms for reconstructing an ε-machine • Shalizi and Crutchfield give example calculations
Crutchfield, The Calculi of Emergence, Physica D, 75, 1994Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 14
ε-machine: optimal model of complexity
• We cannot measure complexity directly– An ε-machine approximates the system’s information
processing
– An ε-machine is the smallest possible explanatory model• Ockham’s razor – include only what is needed
– Maximal accounting for structure • Basic tenet of science – obtain prediction of nature
• Achieve appropriate balance between order and randomness
• Compromise between:– Smallest model with huge error ε and little prediction
– Model with minimal error that differs minimally from system
Crutchfield, The Calculi of Emergence, Physica D, 75, 1994
EME : 15
Can we “calculate” emergence?
• It is hard to calculate causal states in general
• In Markovian behaviour (e.g. thermodynamics) any pre-state is a causal state– Standard entropy formulae in thermodynamics and
statistical mechanics allow calculation of predictive efficiency
– Not surprisingly, the predictive efficiency of macro-scale is considerably better than that of the micro-scale
• 1020 independent gas particles with almost no predictive power give rise to predictable emergent behaviour expressible in a small number of macro-variables
• So, by this measure, thermodynamics is emergentShalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 16
A note on calculations for thermodynamics
• Shalizi’s predictive efficiency calculation uses recognised facts and formulae of thermodynamics and statistical mechanics– Many assumptions about quantities, accuracy, etc.
• e.g. assumes macro-measurement error factor <10-15
– Finds sub-nano-second predictive efficiency of micro-scale is high, but rapidly reduces over time
• Most information in statistical mechanics is irrelevant to thermodynamic macro-state
– But the cumulated assumptions and errors make the figures at best debatable
EME : 17
Emergent structures and ε-machines
• An ε-machine summarises the dynamics of a process
• The ε-machine could be divided in to sub-machines and transitions among them– At each time step, causal and previous state may or may
not be in same sub-machine
• If successive states are in one sub-machine, this is an emergent process of the process approximated by the ε-machine – Because knowing sub-machine reduces statistical
complexityShalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 18
Shalizi’s definition of self-organisation
• An increase in statistical complexity is a necessary condition for self-organisation
• A more realistic definition, relying only on statistical complexity, not excess entropy, causal states, etc.
– Successive states in different sub-machines of an ε-machine – Optimal prediction requires more information
• There are more irreducible complex or random components
• For non-stationary processes, distribution of causal states changes over time– Statistical complexity is measured as a function of time, Cμ(t)
– A system that spontaneously moves from uniform to periodic behaviour exhibits an increase in Cμ(t)
Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 19
Thermodynamics: a box of gas
• In the micro-state, all particles are independent, and all behaviours equally likely– What happens in one time or spatial unit has no later effect
• Statistical complexity remains constant, and low
• Thermodynamic micro-process is not self-organising
• Note that, at least in this case, the definitions allow emergence without self-organisation– But thermodynamics is perhaps an extreme case
Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 20
Summarising the definitions
• Emergence – Informally: behaviour observed at one scale is not apparent
at other scales
– Formally: processes have better predictive efficiency than those from which they emerge
• Lower statistical complexity or greater explanatory power of the past
• Self-organisation– Informally: structures that emerge without systematic
external stimuli
– Formally: processes with an increase in statistical complexity over time
EME : 21
Intuitions on flocking
• Individual birds are probably not independent, but follow simple local rules– For a collection of birds, statistical complexity > 0 but no
where near 1
• If a bird could see the whole flock it would see complex dynamics– For a flock of birds, statistical complexity is closer to 1
• The flock is a self-organised collection of birds
• When we recognise self-organisation we label it as an emergent behaviour at a higher scale– “Flock” denotes an emergent pattern of behaviour
• Predictive efficiency of flock is better than that of a group of birds
EME : 22
Reconciling the definitions
• Natural systems can self-organise– Independent of observation
• Systems that self-organise are studied by “cognitively-limited observers”– Seeking descriptions that have good predictive ability
– Patterns are at a more abstract level than self-organising system elements
• An abstract description with enhanced prediction becomes an emergent process of the original behaviour
EME : 23
So does self-organisation imply emergence?
• Shalizi states that he knows of no reason against self-organising non-emergent systems, but …
• Emergence may be a precondition of detectable self-organisation – In practice, when humans recognise self-organisation, they
identify the abstract result at an emergent process
• At least some of what humans call “noise” may be unrecognisable complex self-organisation– We know that chaos has complex structure …
Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf
EME : 24
Self-organisation and emergence
Not self-organising
Self-organising
Not emergent Emergent
Not interesting Thermodynamics
Possibly verycomplex
self-organisation
EMER interest:Biology
Interesting physics …
Improved modeling and understanding
EME : 25
So where are we now?
• Statistical complexity gives us a nice definition of self-organisation– And an intuition for how self-organisation and emergence
are related
• Statistical complexity gives us a possible definition of emergence– In reality, predictive efficiency is hard to estimate with any
confidence• The definition does clarify features of emergence
• The definitions make assumptions about measurement and discrete spatial or temporal series
EME : 26
A note on discrete measurement
• All work on information theoretic definitions of emergence and self-organisation assumes discrete temporal or spatial observation
• Shalizi discusses using causal states with continuous trajectories:– No current mathematics of continuous conditional
probability
– Continuous entropy exists but depends on co-ordinates• Entropy changes if e.g., distance is measured in inches or
metres
– Reconstruction from data is hard, and seriously affected by measurement problems
• Not unlike problems of using PDEs for explanatory models of biological systems
EME : 27
Open questions
Observers and intrinsic emergence:• If emergence is a precondition of detectable self-
organisation, then it must be possible to observe the system
• In intrinsic emergence, the observer is a sub-process of the system
• monitors environment through sensors to construct an imperfect behavioural model
– Observer makes predictions of future • which, because internal, can interfere with that future
• But, are intrinsic observation and self-organisation compatible?– does observer have to be outside a self-organising system?
EME : 28
Open questions
How to determine agency in self-organisation:
• Organisation assumes that structure emerges over time
• Self-organisation assumes that there is no consistent external agency in the appearance of structure
• But, how can we distinguish self-organisation from organisation by external agency?– A complex system in a complex environment, where
external inputs may be stochastic, but might also be very complex
– Biological systems are usually in this category
EME : 29
And one last open question
• Informally, research has identified level, scope and resolution as relevant to emergence– Statistical complexity and mutual information assume that
processes or systems are well-defined– Statistical analysis (e.g., for causal states or PDEs) also
assumes that scope is known
• What happens if the chosen bounds exclude a key component of the emergence or self-organisation?
• What happens if slightly widening the bounds would reveal that emergence or self-organisation was an artifice of the scope?
• We have raised as many questions as we have solved