Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology
description
Transcript of Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology
Maximum Entropy,
Maximum Entropy Production
and their
Application to Physics and Biology
Roderick C. Dewar
Research School of Biological Sciences
The Australian National University
Part 1: Maximum Entropy (MaxEnt) – an overview
Part 2: Applying MaxEnt to ecology
Part 3: Maximum Entropy Production (MEP)
Part 4: Applying MEP to physics & biology
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
system
energy in
What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …
environment
matter in
many interacting degrees of freedom
energy out
matter out
opennon-equilibrium
Poleward heat transport
SW
LW
Latitudinal heat
transport H = ?
170 W m-2
300 W m-2
T
Cold plate, Tc
Hot plate, Th
Ra < 1760
conduction
T
Cold plate, Tc
Hot plate, Th
Ra > 1760
convection
H = ?
Turbulent heat flow (Raleigh-Bénard convection)
Fsw
Flw + H + E C, H20, O2, N
T,
Ecosystem energy & mass fluxes
system
energy in
What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …
environment
matter in
many interacting degrees of freedom
energy out
matter out
opennon-equilibrium
many degrees of freedom statistical mechanicsstatistical mechanics
Global Circulation Models, Dynamic Ecosystem Models ….
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
W(A) = number of microstates that give macrostate A
Microstate i1 Macrostate A = less detailed description
Ludwig Boltzmann (1844 - 1906)
The most probable macrostate A is the one with the largest W(A) (assume microstates are a priori equiprobable)
SB(A) = kBlog W(A) = Boltzmann entropy of macrostate A
The most probable macrostate is the one of maximum entropy
Boltzmann microstate counting
Microstate i2
Example: N independent distinguishable particles with fixed total energy E
Macrostate A = {nj particles are in state j}
Microstate i = {the mth particle is in state jm}ε3
ε2
ε1
j
jjj
jj
jj nnnnNN εβαloglogΦ
j
jn
NAW
!
!Nn
jj
Enj
jj ε: maximise S = kBlog W subject to
j
jj
jj
ZN
nβεexp
1
βεexp
βεexp
0
Φ
jn
(large N)
Boltzmann entropy Clausius entropy
β 1/kBT
Given E, Smax = kBlogWmax = kB(βE + NlogZ)
under δE = δQ, Smax changes by δSmax = kBβ(δQ)
cf. Clausius thermodynamic entropy δSTD = δQ/T
Smax STD
BUT: microstate counting only works for non-interacting particles
pi = probability that system is in microstate i Macroscopic predictions via
J Willard Gibbs (1839 - 1903)
Gibbs algorithm
i
iiQpQ
The Gibbs algorithm
(MaxEnt)
Maximise H = -i pi log pi with respect to {pi} subject to the
constraints (C) on the system
But how do we construct pi ?
‘minimise the index of probability of
phase’
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
• Closed, isolated
• Closed
• Open
Three applications of MaxEnt (equilibrium systems)
• Microcanonical
• Canonical
• Grand-canonical
System constraints (C) Distribution (pi)
Example 1: closed, isolated system in equilibrium
C: N and E fixed
Microstate i = any N-particle state with total energy Ei restricted to E
Precise description of i and Ei depends on microscopic physics (CAN include particle interactions)
1
1
NE
iip
,ΩMaximise
i
NE
ii ppH log
,Ω
1
subject to
NE
iii
NE
ii ppp
,Ω,ΩαlogΦ
11
NEpi ,Ω
10
ip
Φbasis for Boltzmann’s microstate counting
Example 2: closed system in equilibrium
Microstate i = any N-particle state (no restriction on Ei)
E
Maximise subject to
C: N and fixedE
iC
i EZ
p βexp 1
0
ip
Φ
β 1/kBT
N
iii ppH
Ωlog
1
1
1
N
iip
Ω
EEp
N
iii
Ω
1
N
iii
N
iii
N
ii Epppp
ΩΩΩβαlogΦ
111
Hmax STD
Example 3: open system in equilibrium
Microstate i = any physically allowed microscopic state (no restriction on Ei or Ni)
E N
Maximise
1i
ip
Ω
log1i
ii ppH subject to EEpi
ii
NNpi
ii
C: and fixedEN
iiGC
i NEZ
p γβexp 1
0
ip
Φ β 1/kBT
γ -μ/kBT
ΩΩΩΩ
γβαlogΦ1111 i
iii
iii
iii
i NpEpppp
Hmax STD
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
Frequency interpretation (Venn, Pearson, Fisher …)
System has Ω a priori equiprobable microstates
N independent identical systems, ni = no. of systems in state i
pi describes a physical property of the real world (frequency)
! ... !!
!
Ωnnn
N
21
W = no. of microstates giving {n1,n2 … nΩ}
pi = ni /N = frequency of microstate i
NHppWN i
ii as loglogΩ
1
1
MaxEnt coincides with large-N limit of Maximum Probability for multinomial W
• pi represents our state of knowledge of the real world
• basic axioms for uncertainty H associated with pi the unique uncertainty function is
• Applies to any discrete set of outcomes i
pi = 1/6 (i = 1…6) H = log 6maximum uncertainty :
pi = 0 (i = 1,2...5), p6 = 1 H = 0
minimum uncertainty :
Ω
1log
iii ppH
Information theory interpretation (Shannon 1948, Jaynes 1957 …)
Claude Shannon(1916-2001)
Jaynes (1957b, 1978)
Q (= ΣipiQi) reproducible under C
it is sufficient to encode only the information C into pi …
all information other than C is thrown away
… but this is precisely what MaxEnt does!
MaxEnt = max H subject to C
H = -i pi log pi = missing information about i
Behaviour that is experimentally reproducible under conditions C must be
theoretically predictable from C aloneEdwin Jaynes (1922-1998)
Assumed constraints C
experimental conditionsconservation laws
microstates (e.g. QM)
Reproducible behaviour Q
Max H subject to C pi
The prediction game
i
iiQpQ
Observed behaviour Qobs
Qobs Q missing constraint
MaxEnt
testC C'
ESSENTIAL PHYSICS PREDICTION
OBSERVATION
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
Edwin Jaynes (aged 14 months)
Information theory interpretation of MaxEnt
general algorithm for predicting reproducible behaviour
under given constraints
can be extended to non-equilibrium systems
(same principle, different constraints)
‘Maximum caliber principle’ (Jaynes 1980, 1996)
Γ micropaths
ΓΓ logmax ppH
cf. Feynman path integral formalism of QM!
A B
The second law in a nutshell
AB reproducible WB WA' = WA SB SA
WB
WA WA'
. .microscopic
path in phase-space
after Jaynes 1963, 1988
S = kBlog W
Liouville Theorem(Hamiltonian dynamics)
reproducible macroscopic change
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
Some general properties of MaxEnt distributions
λexpλ
1
kkiki f
Zp
1i
ip
m k,Ffpf ki
kiik 1
k
kk FλZH λlogmax
k
kZ
Fλ
λlog
kjkjkj
kjk
jjk Cffff
ZFC
λλ
λlog
λ
2
subject to m + 1 constraints C
i
ii ppH logmax
Fλ
k
k F
FHF
maxλ FHmax
i kkik fZ λexpλ
;
λ max2
kjkjj
kjk B
FF
FH
FB
1CB
Response-fluctuation & reciprocity relations:
Stability-convexity relation:
Constitutive relation: Orthogonality:
Partition function:
Summary of Lecture 1 …
The problem
to predict the behaviour of non-equilibrium systems with many degrees of freedom
The proposed solution
MaxEnt: a general information-theoretical algorithm for predicting reproducible behaviour under given constraints
Boltzmann
Gibbs
Shannon
Jaynes