The Combinatorial Basis of Entropy...
Transcript of The Combinatorial Basis of Entropy...
![Page 1: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/1.jpg)
The Combinatorial Basis of Entropy (“MaxProb”)
22nd Canberra International Physics Summer School ANU, Canberra
11 December 2008
by Robert K. Niven
Marie Curie Incoming International Fellow, 2007-2008 Niels Bohr Institute, University of Copenhagen, Denmark
School of Aerospace, Civil and Mechanical Engineering
The University of New South Wales at ADFA Canberra, ACT, Australia
![Page 2: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/2.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 2
Lectures
1. The Combinatorial Basis of Entropy (“MaxProb”)
2. Jaynes’ MaxEnt, Riemannian Metrics and the Principle of Least Action
![Page 3: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/3.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 3
Contents • Historical overview
- combinatorics - probability theory
• Combinatorial basis of entropy / MaxProb principle
generalised combinatorial definitions of entropy and cross-entropy
explanation of MaxEnt / MinXEnt
• Applications 1. Multinomial systems (asymptotic vs non-asymptotic) 2. (In)distinguishable particles or categories 3. “Neither independent nor identically distributed” sampling
• Future applications …
![Page 4: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/4.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 4
(Advertisement)
Courses at UNSW@ADFA, Canberra:
• Short course in “Maximum Entropy Analysis”, 14-15 May 2009
(fee paying $1270).
• Masters course: ZACM8327 Maximum Entropy Analysis, semester 2,
2009 (fee paying or UNSW@ADFA enrolled student)
- 3 hours of lectures + tutorials per week
- based on similar course at Niels Bohr Institute
![Page 5: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/5.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 5
Historical Overview
![Page 6: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/6.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 6
Combinatorics Knowledge is very old! (Edwards, 2002)
(a) Number-patterns
Pythagoras (500BC)
Egypt (300BC)
Theon of Smyrna, Nicomachus (100AD)
Higher dimensions: Tartaglia (1523, publ. 1556)
figurate numbers fk
![Page 7: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/7.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 7
(b) Binomial coefficients
= coefficients of (a + b)N
Al-Karaji (1007); Al-Samawal (1180); Al-Kashi (1429)
Chia Hsien (1100); Yang Hui (1261); Chu Shih-chieh (1303)
Cardano (1570), etc
- applied to solution of equations; finding roots; etc
Chu Shih-chieh (1303)
![Page 8: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/8.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 8
binomial coefficients
N
k= f
k , where =dimension
![Page 9: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/9.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 9
(c) Combinations + Permutations
Ancient; e.g.
- Susruta (600BC), Jains (300BC): combinations of 6 tastes
- Pingala (200BC): combinations of syllables
No. of permutations of N things = N !
- Hebrew Book of Creation (700); Bhaskara (1150)
No. of groups of N things, taken k at a time: - Mahavira (850); Bhaskara (1150); ben Gerson (1321)
Without replacement With replacement
Combinations
CkN
=N
k=
N!
k !(N k)!
wC
kN
=N + k 1
k=
(N + k 1)!
k !(N 1)!
Permutations PkN
= N(k) =N!
(N k)!
wPkN
= Nk
with
N
kk=0
N= 2
N
![Page 10: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/10.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 10
Pascal (1654) - equivalence of figurate numbers AND binomial coefficients AND
numbers of combinations without replacement
Multinomial weight - Bhaskara (1150); Mersenne (1636)
= no. of permutations of N objects, containing ni of each category
i = 1,...,s , is:
W =N !
n1!n
2! ... n
s!= N !
1
ni!
i=1
s
![Page 11: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/11.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 11
Probability Theory (a) Classical period (e.g. Cardano (1560s), Pascal, Fermat, Huygens, the Bernoullis,
Montmort, de Moivre, Laplace)
Probability =
No. of outcomes of interest
Total no. of outcomes
(b) “Frequentist” school (e.g. Venn, Pearson, Neyman, Fisher, von Mises, Feller)
- probability = measurable frequency, for an infinite number of repetitions of a “random experiment”
- attempt to define probabilities as certainties
- narrow applicability
![Page 12: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/12.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 12
(c) “Bayesian” or “Plausibility” school (Bayes, Laplace, Jeffreys, Polya, Cox, Jaynes, 1957; 2003)
- probability = “plausibility” = assignment based on what you know
- need not be a measurable frequency - manipulate using sum + product rules (Jaynes, 2003)
- “subjective” = “information-dependent”
- different observers, with different information, can assign different probabilities to the same event
- more useful; encompasses all frequentist situations
![Page 13: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/13.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 13
Probability Distributions “Measures of Central Tendency”
Continuous parameter x Discrete parameter x p(x) = probability density function (pdf) p(x) = pi = probability mass function
![Page 14: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/14.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 14
Combinatorial (or Probabilistic) Definition of Entropy
![Page 15: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/15.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 15
Definitions Entity = a discrete particle, object or agent, or an item in a sequence, which is
separate but not necessarily independent of other entities
Category = possible assignment of an entity
Probabilistic System = a set of entities K assigned to a
set of categories C by a discrete random variable : K C
e.g. physics: particles energy levels
gambling: die throws die sides
communications: signal bits letters of alphabet
![Page 16: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/16.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 16
Configuration = distinguishable permutation of entities amongst categories e.g.: physics: microstate; information theory: sequence
Realization = aggregated arrangement of entities amongst categories = set of configurations e.g.: physics: macrostate; information theory: type
Commonly define realizations by the no. of entities in each category {n
i}
![Page 17: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/17.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 17
MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)
- “A system can be represented by its most probable realization”
principle for probabilistic inference
- does not depend on asymptotic limits
- does not give certainty
BB GB BG GG
Superset of 2nd Law “A system tends towards its most probable realization”
- not just thermodynamics!
![Page 18: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/18.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 18
MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)
- “A system can be represented by its most probable realization”
principle for probabilistic inference
- does not depend on asymptotic limits
- does not give certainty
BB GB BG GG
Superset of 2nd Law “A system tends towards its most probable realization”
- not just thermodynamics!
Ergodicity
Inference
![Page 19: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/19.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 19
Multinomial Systems (Boltzmann, 1877)
N distinguishable balls (entities)
s distinguishable boxes (categories)
qi = source (“prior”) probability of ball falling in ith box
= normalised degeneracy gi / gii=1
s
Probability of a given realization {n
i} is given by the multinomial
distribution:
Pmult = N ! qi
ni
ni !i=1
s
qi =1/s
Pmult
=W
mult
sN
; Wmult
= N !1
ni!
i=1
s
If categories equiprobable, can use multinomial weight
![Page 20: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/20.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 20
MaxProb: want to maximise:
Easier to maximise:
lnPmult = lnN !+ ni lnqii=1
s
lnni !
i=1
s
Asymptotic limit for N (rigorously by Sanov (1957) theorem; crudely by Stirling’s approx. lnm! m lnm m ):
DKL = limN
lnPmult
N= pi ln
pi
qii=1
s
where pi =
ni
N.
If qi = 1/s = constant:
hSh= lim
N
lnWmult
N= pi lnpi
i=1
s
Shannon entropy
Kullback - Leibler cross - entropy
= directed divergence
= negative of relative entropy
Pmult = N ! qi
ni
ni !i=1
s
![Page 21: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/21.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 21
Summary
• Kullback-Leibler and Shannon functions are asymptotic forms of the
multinomial distribution P
mult
• If minimise D
KL (MinXEnt) or maximise
hSh (MaxEnt) of a multinomial
system, subject to constraints
obtain asymptotic MaxProb realization
Boltzmann principle:
Define entropy and cross-entropy by: h=
lnW
N,
D =
lnP
N
(compare S
total= SN = k lnW )
hence always consistent with MaxProb
![Page 22: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/22.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 22
Jaynes’ MaxEnt • Jaynes (1957)
- minimise D or maximise h , subject to constraints
“most uncertain” distribution = distribution which contains the least
information
• BUT how do we define uncertainty?
- Jaynes only considers D
KL or
hSh axiomatic basis of Shannon
(1948)
• However, a system: - need not be multinomial ! - need not be asymptotic !
Kullback-Leibler or Shannon functions will not give the MaxProb distribution
If know P P
mult or N , must include this information !
![Page 23: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/23.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 23
Application to Multinomial Systems (Asymptotic + Non-Asymptotic)
![Page 24: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/24.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 24
Why the Multinomial? (Niven, AIP Conf. Proc. 954 (2007) 133; Blower, pers. comm.)
Pmult = N ! qi
ni
ni !i=1
s
with pi =
ni
N
1. Frequentist approach
P
mult, {qi } = measurable frequencies
2. Bayesian approach
P
mult, {qi } = Bayesian probabilities
If ignorant about choice of model P , then all models equiprobable
must choose multinomial (“central model theorem”)
![Page 25: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/25.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 25
Asymptotic Analysis Jaynes’ (1957) Algorithm
Minimise
DKL = pi lnpi
qii=1
s
subject to
pii=1
s= 1 and
pii=1
sfri = fr , r = 1,...,R
Form Lagrangian, differentiate w.r.t. pi
pi*= qi e 0
'
r frir=1
R
=1
Zqi e r frir=1
R
Z = e 0
'
= qi e r frir=1
R
i=1
s
Boltzmann
distribution
with 0
'=
0+1
Jaynes’ (1957, 1963, 2003) analysis many more (generic) relations
![Page 26: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/26.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 26
Isolated Thermodynamic System:
Isolated from rest of universe
e.g. microcanonical ensemble
Natural:
pii=1
s= 1
Mean energy:
pii=1
si = U
pi
*= qi e 0
'
1 i =1
Zqi e 1 i
D *
h*=
0
'+
1U
compare S* = k lnZ +U
T
or F = kT lnZ = TS * + U
Hence 0
'= lnZ = and
1=
1
kT
![Page 27: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/27.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 27
Non-Asymptotic Analysis (Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)
Use raw multinomial:
Minimise
D(N)
=lnP
N=
1
NlnN !+ ni lnqi
i=1
s
lnni !
i=1
s
subject to
nii=1
s= N and
n
ii=1
sfri= F
r, r = 1,...,R
Form Lagrangian, differentiate w.r.t. n
i
pi#=
ni#
N=
1
N
1 lnN !
N+ lnqi 0
(N)r(N)fri
r =1
R
1
where ( )= digamma function
Non - asymptotic
distribution
![Page 28: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/28.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 28
Example 1:
Multinomial
n1,n2,n3
s=3
q =
1
2,3
8, 1
8
subject to
n
i= N
i=1
s
only
![Page 29: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/29.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 29
Example 2: Multinomial
n1,n2,n3
s=3
q =
1
14, 4
14, 9
14
subject to
n
i= N
i=1
s
n
i i= E
i=1
s
T
with
= 1,2,4[ ]
U =
ET
N=
5
3
![Page 30: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/30.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 30
Example 2 (cont’d):
For constant
U =
ET
N=
5
3,
obtain
0(N)
=(N)
1(N)
=1
kT(N)
![Page 31: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/31.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 31
Thermodynamic Double System:
System 1: N
1 particles; energy levels
i
System 2: N
2 particles; energy levels
j
Probs. of realizations {ni }, {nj } are:
Maximise P1P
2 subject to
n
i= N
1i=1
s,
njj=1
m= N
2 and
most probable distrib.:
pi#=
1
N1
1
lnN1!
N1
+ lnqi 0a
(N1)1 i 1
pj#=
1
N2
1 lnN2 !
N2
+ lnqj 0b
(N2)1 j 1
“Zeroth law” upheld
( 1 in common)
P1 = N1! qi
ni
ni !i=1
s
, P2 = N2 ! qj
nj
nj !j=1
m
ET = ni ii=1
s+ nj jj=1
m
![Page 32: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/32.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 32
Summary
- combinatorial approach straightforward analysis using system (not
ensemble) parameters
steps towards non-asymptotic thermodynamics
(without thermodynamic limit !)
Application to Information Theory
(Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)
Adopt Boltzmann principle as definition of information:
I =h
ln2=
log2 W
N or
I =
D
ln2=
log2 P
N (in bits)
non-asymptotic coding ?
non-asymptotic network theory ?
![Page 33: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/33.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 33
Application: (In)distinguishability
![Page 34: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/34.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 34
Statistics Consider role of distinguishability:
Disting. balls Indisting. balls
Disting.
boxes
Maxwell-Boltzmann
(Lynden-Bell)*
Bose-Einstein
(Fermi-Dirac)*
Indisting. boxes
? ?
* maximum of 1 ball per box
![Page 35: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/35.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 35
Statistics Consider role of distinguishability:
Disting. balls Indisting. balls
Disting.
boxes
Maxwell-Boltzmann
(Lynden-Bell)*
Bose-Einstein
(Fermi-Dirac)*
Indisting. boxes
D:I statistic I:I statistic
* maximum of 1 ball per box
![Page 36: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/36.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 36
(In)distinguishability Allocate students to PhD supervisors:
1. Disting. students disting. supervisors - consider personal interactions
2. Indisting. students disting. supervisors
- e.g. Dean
3. Disting. students indisting. supervisors
- e.g. student club
4. Indisting. students indisting. supervisors
- e.g. Government department Choice of statistic - and hence entropy - depends on purpose
Tseng & Caticha (2002): “Entropy is not a property of a system … [it] is a property of our description of a system.”
![Page 37: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/37.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 37
(a) Maxwell-Boltzmann
WMB = N ! gi
ni
ni !i=1
s
(b) Bose-Einstein
WBE =(gi + ni 1)!
ni !(gi 1)!i=1
s
(c) Fermi-Dirac
WFD =gi !
ni !(gi ni )!i=1
s
![Page 38: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/38.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 38
Entropy functions: Name Asymptotic Entropy Non-asymptotic Entropy
(Niven, 2005, 2006) MB
hMB= pi ln
pi
gii=1
s
hMB(N)
=1
Nln[(piN)!]
i=1
s
+1
Npi ln[N!]+ pi lngi
BE
hBE = ( i + pi )ln( i + pi )
i=1
s
i ln i pi lnpi
hBE(N)
=1
Nln ( iN + piN 1)!{
i=1
s
ln ( iN 1)! ln (piN)! }
FD
hFD = ( i pi )ln( i pi )
i=1
s
+ i ln i pi lnpi
hFD(N)
=1
Ni=1
s
ln ( iN piN)!{
+ ln ( iN)! ln (piN)! }
where i = gi / N = relative degeneracy
![Page 39: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/39.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 39
MaxProb: maximise h subject to
pii=1
s
= 1,
pi frii=1
s
= fr , r = 1,...,R
Name Asymptotic distribution Non-asymptotic distribution (Niven, 2005, 2006)
MB
pMB,i*
= gi e 0' r frir=1
R
pMB,i#
=1
N
1 ln[N!]
N+ lngi 0 ' r fri
r=1
R
1
BE
pBE,i*
=i
e 0+ r frir=1
R
1
pBE,i#
=
1
N
1( iN + pBE,i
# N) 0 r frir=1
R
1
FD
pFD,i*
=i
e 0+ r frir=1
R
+1
pFD,i#
=
1
N
1( iN pFD,i
# N +1) 0 r frir=1
R
1
where = digamma f’n; -1 =inverse digamma f’n
![Page 40: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/40.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 40
(d) D:I Statistic
(Niven, CTNEXT07)
- disting. entities
- indisting. categories, each with g indisting. subcategories
Can show
WD:I =N
n1,n2,...,nk ,0,...,0(g)
=N !
ni !
i=1
k
rj !
j=1
N
ni
g=1
min(g,ni )
i=1
k
where rj = no. of occurrences of j in
{n
i}
ni
g = Stirling no. of 2nd kind
Curious behaviour!
![Page 41: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/41.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 41
(e) I:I Statistic
(Niven, NEXT 07)
- indisting. entities
- indisting. categories, each with g indisting. subcategories
Can show
WI:I(g) =N
n1,n2,...,nk ,0,...,0(g)
= P ( j)
=1
min(g, j)rj
j=1
n1
where rj = no. of occurrences of j in
{n
i}
P ( j) = partition number
a + b + ...( )m
= Wronski aleph = “combinatorial polynomial”
![Page 42: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/42.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 42
Normal Polynomials Wronski (1811) alephs
(a + b)2 = a2+ 2ab + b
2
(a + b)3 = a3+ 3a
2b + 3ab
2+ b
3
(a + b)m =m
ta
tb
m t
t=0
m
(a + b)2 = a2+ ab + b
2
(a + b)3 = a3+ a
2b + ab
2+ b
3
(a + b)m = atb
m t
t=0
m
Hence
a
=1
m
= a1
t1
t1,t
2,...,t
a2
t2 ...a
t with
t = m
=1
![Page 43: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/43.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 43
Example: Non-Degenerate MB and BE statistics
![Page 44: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/44.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 44
Example: Non-Degenerate D:I statistic
![Page 45: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/45.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 45
![Page 46: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/46.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 46
Example: Non-Degenerate I:I statistic
![Page 47: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/47.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 47
Summary (non-degenerate, no moment constraints)
Disting. balls Indisting. balls
Disting. boxes
MB statistic
MaxProb; MeanProb
Highly symmetric
Strongly asymptotic
uniform distrib.
BE statistic
MeanProb only
Highly symmetric
Strongly asymptotic uniform
distrib.
Indisting. boxes
D:I statistic
MaxProb; MeanProb
Highly asymmetric
Slowly asymptotic, s N
Non-asymptotic, s N ?
I:I statistic
MeanProb only
Highly asymmetric
Non-asymptotic, s N
Monotonic asymptote for s N
![Page 48: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/48.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 48
Application:
Pólya Distribution
![Page 49: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/49.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 49
Pólya Distribution (Grendar & Niven, cond-mat/0612697)
- urn: M disting. balls, with mi of each
category,
mi= M
i=1
s
- sample: N balls, with ni of each category
- scheme: draw of ball of category i, return to urn + add c balls of same category to urn
“neither independent nor identically distributed” (ninid) sampling
Prob. of each realization {ni} is:
PPólya =M c + N 1
N
1 mi c + ni 1
nii=1
s
Multivariate
Polya
distribution
![Page 50: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/50.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 50
Pólya cross-entropy: put qi = mi/M, and =N/M: Name Asymptotic cross-entropy Non-asymptotic cross-entropy
(without Stirling approx.) Pólya (c>0)
DPólyax
=1
Npi ln
(N +1) ( Nc)
( Nc+ N)i=1
s
+ ln(qiN
c+ piN)
(piN +1) (qiN
c)
Pólya (c<0)
DPólyaSt pi
c( c +1)ln( c +1)
i=1
s
+1c
(qi + pi c)ln(qi + pi c)
1c
qi lnqi pi lnpi
DPólyax
=1
Npi ln
(N +1) ( Nc
N +1)
( N
FD
+1)i=1
s
+ ln(
qiN
c+1)
(piN +1) (qiN
cpiN +1)
![Page 51: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/51.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 51
Pólya MaxProb
Name Asymptotic distribution Non-asymptotic distribution Pólya (c>0)
pPólya,i#
=1
N
1F(N, c) + (
qiN
c+ pPólya,i
# N) 0 r frir=1
R
1
Pólya (c<0)
pPólya,i*
=qi
e 0+ r frir=1R
c
pPólya,i#
=1
N
1K(N, c) (
qiN
cpPólya,i
# N +1) 0 r frir=1
R
1
Compare Acharya-Swamy (1994) ansatz for “anyons”
pi* 1
e 0+
1xi
, with [ 1,1]
![Page 52: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/52.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 52
Future Applications: Graphs and Networks
![Page 53: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/53.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 53
Graph Entropy (Körner & Longo, 1973; Körner & Orlitsky, 1998)
- vertices = categories (alphabet)
- lines (edges) connect disting. categories
H(G,P) = limsupN
1
Nlog2 (G
P
N ) +1( )
where P = prob. distrib on vertex set
(GP
N ) = chromatic no. of graph G
P
N , for N-sequence
consider “heterogeneous” distinguishability of categories
BUT is asymptotic (does not consider entities)
Strong connection to networks + coding theory
![Page 54: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/54.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 54
Conclusions
![Page 55: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/55.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 55
Conclusions • MaxProb principle: choose realization of highest probability
principle of probabilistic inference
explanation for MaxEnt, MinXEnt
generalised definitions of D and h
• Non-asymptotic theory
- finite N thermodynamics (microcanonical) - other applications!
• Other statistics:
- MB, BE, FD - indisting. categories - Polya sampling (“ninid”)
• Strong connections to graphs, networks + coding
![Page 56: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/56.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 56
Acknowledgments:
Thanks to:
• The University of New South Wales, Australia
• The European Commission, for Marie Curie Incoming
International Fellowship at University of Copenhagen
• Dr Bjarne Andresen + Dr Flemming Topsøe
• COSNET, ANU and (Prof. R. Dewar)2 for opportunity to present
![Page 57: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/57.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 57
References Acharya, R., Narayana Swamy, P. (1994) J. Phys. A: Math. Gen., 27: 7247-7263. Boltzmann, L. (1872) Sitzungsberichte Akad. Wiss., Vienna, II, 66: 275-370; English transl.: Brush,
S.G. (1966) Kinetic Theory: Vol. 2 Irreversible Processes, Permagon Press, Oxford, 88-175. Boltzmann, L. (1877), Wien. Ber., 76: 373-435, English transl., Le Roux, J. (2002), 1-63,
http://www.essi.fr/~leroux/. Clausius, R. (1865) Poggendorfs Annalen 125: 335; English transl.: R.B. Lindsay, in J. Kestin (ed.)
(1976) The Second Law of Thermodynamics, Dowden, Hutchinson & Ross, PA, (1976) 162. Clausius, R. (1876) Die Mechanische Wärmetheorie (The Mechanical Theory of Heat), F. Vieweg,
Braunschwieg; English transl.: W.R. Browne (1879), Macmillan & Co., London. Edwards, A.W.F. (2002) Pascal’s Arithmetical Triangle: The Story of a Mathematical Idea, 2nd ed.,
John Hopkins U.P., Baltimore. Grendar, M., Grendar, M. (2001) What is the question that MaxEnt answers? A probabilistic
interpretation, in A. Mohammad-Djafari (ed.) Bayesian Inference and Maximum Entropy Methods in Science and Engineering, AIP (Melville), 83.
Grendar, M., Niven, R.K. (in submission), http://arxiv.org/abs/cond-mat/0612697. Jaynes, E.T. (1957), Physical Review, 106: 620-630. Jaynes, E.T. (Bretthorst, G.L., ed.) (2003) Probability Theory: The Logic of Science, Cambridge
U.P., Cambridge. Körner, J., Longo, G. (1973) IEEE Trans. Information Theory IT-19(6): 778. Körner, J., Orlitsky, A., (1998) IEEE Trans. Information Theory 44(6) 2207. Kullback, S., Leibler, R.A. (1951), Annals Math. Stat., 22: 79-86.
![Page 58: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/58.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 58
Lilly, S. (2002), A Practical Guide to Runes, Caxton Editions, London. Niven, R.K. (2005), Physics Letters A, 342(4): 286-293. Niven, R.K. (2006), Physica A, 365(1): 142-149. Niven, R.K. (in submission) CTNEXT07, 1-5 July 2007, Catania, Sicily, Italy, http://arxiv.org/
abs/0709.3124. Niven, R.K. (2005-07) Combinatorial information theory: I. Philosophical basis of cross-entropy
and entropy, cond-mat/0512017. Niven, R.K., Suyari, H. (in submission) Combinatorial basis and finite forms of the Tsallis entropy
function. Pascal, B. (1654), Traité du Triangle Arithmétique, Paris. Paxson, D.L. (2005) Taking Up the Runes, Red Wheel/Weiser, York Beach, ME, USA. Pennick, N. (2003) The Complete Illustrated Guide to Runes, HarperCollins, London. Planck, M. (1901) Annalen der Physik 4: 553. Sanov, I.N. (1957) Mat. Sb. 42, 11-44; English transl. Selected Transl. Math. Stat. Prob. 1 (1961),
213-224. Shannon, C.E. (1948), Bell System Technical Journal, 27: 379-423; 623-659. Suyari, H. (2006), Physica A 368(1): 63. Vincze, I, (1974) Progress in Statistics, 2: 869-895. Historical references prior to 1800AD are given in Edwards (2002).
![Page 59: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/59.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 59
Appendix 1: Runic alphabet: (Refs: Lilly, 2002; Pennick, 2003; Paxson, 2005; Wikipedia)
fuTarkgw hnijIpzs tbemlNod ...
f.u.th.a.r.k.g.w h.n.i.j.eo.p.z.s t.b.e.m.l.ng.o.d
- used across Germanic + central Europe, Britain + Scandinavia, 5th-10th cent.; in Sweden to 17th cent.
- derived from Etruscan alphabet (not Greek or Roman) - each rune has symbolic meaning Anglo-Saxon h (“Haegl”) = old German h (“Hagalaz”) = hail, hailstones - symbolic of destructive force of Nature, but melts and gives
new life - evokes need to accept what is inevitable; to “go with the flow”;
i.e. rune of transformation
![Page 60: The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis](https://reader035.fdocuments.us/reader035/viewer/2022070611/5b0aa3897f8b9ac7678cb0e8/html5/thumbnails/60.jpg)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 60