On some subadditivity inequalities of entropiesabesenyei.web.elte.hu/publications/entropy.pdf ·...

On some subadditivity inequalities of entropies

Adam BESENYEI

Dept. of Applied AnalysisEotvos Lorand University, Budapest, Hungary

[email protected]

April 12, 2014

A. Besenyei (ELTE Budapest) Subadditivity inequalities April 12, 2014 1 / 18

[email protected]

Outline

1 Entropy in probability• Shannon entropy• Tsallis entropy

}origins and their subadditivity

2 Quantum entropy• von Neumann entropy• quantum Tsallis entropy

}origins and their subadditivity


Entropy in probability Shannon entropy

Shannon entropy: definition and origins

Problem

Let X be a discrete random variable with possible values of outcome {x1, . . . , xm}and probability distribution p(xi ) = P(X = xi ).How to measure the “uncertainty” of the random variable?




Problem

Let X be a discrete random variable with possible values of outcome {x1, . . . , xm}and probability distribution p(xi ) = P(X = xi ).How to measure the “uncertainty” of the random variable?

Example

Coin toss (not fair): P(head) = p, P(tail) = 1− p.If p = 0 or p = 1, there is no uncertainty.As p increases from 0 to 1

2 , the uncertainty also increases.If p = 1

2 , then the uncertainty is maximal.




Definition

Let X be a discrete random variable with possible values of outcome {x1, . . . , xm}and probability distribution p(x) = P(X = x). Then, the Shannon entropy of thediscrete variable X is defined as

H(X ) = −m∑i=1

p(xi ) log p(xi )

with the convention that 0 log 0 = 0, since limx→0+

x log x = 0.

(The letter H is the Greek capital eta.)




History

Entropy (“turning towards”) in classical thermodynamics was introduced in 1865by Rudolph Clausius (1822–1888), later by Josiah Willard Gibbs (1839–1903), andby Ludwig Boltzmann (1844–1906) in a statistical point of view.

Rudolph Clausius Willard Gibbs Ludwig Boltzmann




History

The formula in information theory was introduced by Claude Shannon(1916–2001) in his seminal paper A Mathematical Theory of Communication in1948 while he was working at Bell Telephone.




History

The paper was reprinted in 1949 in the book The Mathematical Theory ofCommunication by Claude Shannon and Warren Weaver (1894–1978).




How to call it?Shannon visited John von Neumann in 1949. According to some sources,Neumann said:

“You should call it entropy, for two reasons. In the first place youruncertainty function has been used in statistical mechanics under thatname, so it already has a name. In the second place, and moreimportant, nobody knows what entropy really is, so in a debate you willalways have the advantage.”


Entropy in probability Subadditivity inequalities

Subadditivity of Shannon entropy

DefinitionIf Y is another discrete random variable with possible values of outcome{y1, . . . , yn} and probability distribution p(y), then the joint entropy of the pair(X ,Y ) with joint distribution p(x , y) is

H(X ,Y ) = −m∑i=1

n∑j=1

p(xi , yj) log p(xi , yj).

TheoremThe joint entropy is not greater than the sum of the individual entropies:

H(X ,Y ) ≤ H(X ) + H(Y ).

This is called the subadditivity of the entropy.

RemarkEquality holds if and only if X and Y are independent.



Strong subadditivity of Shannon entropy

TheoremIf X ,Y ,Z are discrete random variables, then

H(X ,Y ,Z ) + H(Y ) ≤ H(X ,Y ) + H(Y ,Z ).

This is called the strong subadditivity of the entropy.





H(X ,Y ,Z ) + H(Y ) ≤ H(X ,Y ) + H(Y ,Z ).


RemarkBy introducing the conditional entropies

H(Z |X ,Y ) = H(X ,Y ,Z )− H(X ,Y )

H(Z |Y ) = H(Y ,Z )− H(Y ),

the strong subadditivity inequality can be written as

H(Z |X ,Y ) ≤ H(Z |Y ).

Equality holds if and only if X ,Y ,Z form a Markov-triplet, i.e.,P(Z |X ,Y ) = P(Z |Y ) (X = “past”,Y = “present”,Z = “future”).





H(X ,Y ,Z ) + H(Y ) ≤ H(X ,Y ) + H(Y ,Z ).


Some notationLet us denote the joint distribution of X ,Y ,Z by

{pijk := p(xi , yj , zk) : 1 ≤ i ≤ m, 1 ≤ j ≤ n, 1 ≤ k ≤ r},and the marginal distributions p(xi , yj), p(yj , zk), p(yj) by

pij− :=r∑

k=1

pijk , p−jk :=m∑i=1

pijk , p−j− :=m∑i=1

r∑k=1

pijk .





H(X ,Y ,Z ) + H(Y ) ≤ H(X ,Y ) + H(Y ,Z ).


A reformulationThen the strong subadditivity has the form∑

ijk

−pijk log pijk −∑j

p−j− log p−j− ≤ −∑ij

pij− log pij− −∑jk

p−jk log p−jk ,

or equivalently∑ijk

pijk (log pijk + log p−j− − log pij− − log p−jk) ≥ 0.



Partial strong subadditivity of Shannon entropy

TheoremFor each fixed 1 ≤ j ≤ n and 1 ≤ k ≤ r the following inequality holds:

n∑i=1

pijk (log pijk + log p−j− − log pij− − log p−jk) ≥ 0.

Proof.The inequality is equivalent to∑

i

pijkp−jk

log

(pij−p−jkpijkp−j−

)≤ 0 where

∑i

pijkp−jk

= 1.

The concavity of the logarithm function implies∑i

pijkp−jk

log

(pij−p−jkpijkp−j−

)≤ log

(∑i

pijkp−jk

pij−p−jkpijkp−j−

)= log 1 = 0.


Entropy in probability Tsallis entropy

Tsallis entropy: definition and origins

DefinitionIf X is a discrete random variable and q 6= 1, then the Tsallis entropy or q-entropyis defined as

Sq(X ) =1−

∑mi=1 p(xi )

q

q − 1.




DefinitionIf X is a discrete random variable and q 6= 1, then the Tsallis entropy or q-entropyis defined as

Sq(X ) =1−

∑mi=1 p(xi )

q

q − 1.

RemarkBy introducing the q-logarithm function

logq x =xq−1 − 1

q − 1(q 6= 1),

the Tsallis entropy can be written as

Sq(X ) = −m∑i=1

p(xi ) logq p(xi ).

It is easily seen that limq→1

Sq = H.




History

This generalized entropy was introduced by Constantino Tsallis (1943–) in hispaper A possible generalization of Boltzmann–Gibbs statistics in 1988.




History

However, a similar entropy was already defined by Daroczy Zoltan (1938–) in hispaper Generalized information functions in 1970.



Subadditivity of Tsallis entropy

Theorem (Daroczy, 1970)

If q > 1, then the q-entropy is subadditive:

Sq(X ,Y ) ≤ Sq(X ) + Sq(Y ).

Equality holds if and only if q = 1 and X ,Y are independent.






Sq(X ,Y ) ≤ Sq(X ) + Sq(Y ).


A special case (proposed in KoMaL, 2013)

For m = n = 2, the subadditivity reduces to the nice inequality

(x + y)q + (z + v)q + (x + z)q + (y + v)q ≤ xq + yq + zq + vq + (x + y + z + v)q.






Sq(X ,Y ) ≤ Sq(X ) + Sq(Y ).







Sq(X ,Y ) ≤ Sq(X ) + Sq(Y ).


A similar inequality (proposed in Amer. Math. Monthly, 2014)




Proof (Case m = n = 2).

Let f (x) = x − xq, then f is non-negative and convex. The subadditivity is

f (x) + f (y) + f (z) + f (v) ≤ f (x + y) + f (z + v) + f (x + z) + f (y + v).

Observe

f (x) + f (y) + f (z) + f (v)− f (x + y)− f (z + v)

= (x + y)q[f

(x

x + y

)+ f

(y

x + y

)]+ (z + v)q

[f

(z

z + v

)+ f

(v

z + v

)].

Since (x + y) + (z + v) = 1 and (x + y)q ≤ x + y , (z + v)q ≤ z + v , thereforeJensen’s inequality implies

(x + y)qf

(x

x + y

)+ (z + v)qf

(z

z + v

)≤ f (x + z),

(x + y)qf

(y

x + y

)+ (z + v)qf

(v

z + v

)≤ f (y + v)

thus the subadditivity inequality follows.



Strong subadditivity of Tsallis entropy

Theorem (Shigeru Furuichi, 2004)

If q > 1, then the q-entropy is strongly subadditive:

Sq(X ,Y ,Z ) + Sq(Y ) ≤ Sq(X ,Y ) + Sq(Y ,Z ).

Equality holds if and only if q = 1 and X ,Y ,Z form a Markov triplet.



Strong subadditivity of Tsallis entropy

Theorem (Shigeru Furuichi, 2004)

If q > 1, then the q-entropy is strongly subadditive:

Sq(X ,Y ,Z ) + Sq(Y ) ≤ Sq(X ,Y ) + Sq(Y ,Z ).

Equality holds if and only if q = 1 and X ,Y ,Z form a Markov triplet.

RemarkAs q → 1+, we obtain the subadditivity inequalities of the Shannon-entropy.The strong subadditivity can be reformulated as∑

ijk

pijk(logq pijk + logq p−j− − logq pij− − logq p−jk

)≥ 0.



Partial strong subadditivity of Tsallis entropy

Theorem (B.–Petz, 2013)

For q > 1 and fixed 1 ≤ j ≤ n, 1 ≤ k ≤ r the following inequality holds:n∑

i=1


)≥ 0.






i=1


)≥ 0.

Lemma

Let ai , bi (i = 1, . . . ,m) be positive numbers and q ≥ 1. Then

m∑i=1

aibi (ai + bi )q−2 ≤

m∑i=1

ai ·m∑i=1

bi ·

(m∑i=1

(ai + bi )

)q−2

.






i=1


)≥ 0.

Lemma

Let ai , bi (i = 1, . . . ,m) be positive numbers and q ≥ 1. Then

m∑i=1

aibi (ai + bi )q−2 ≤

m∑i=1

ai ·m∑i=1

bi ·

(m∑i=1

(ai + bi )

)q−2

.

Remark

For q = 1, this is called Milne’s inequality (1925). (Edward Arthur Milne(1896–1950) British astrophysicist and mathematician.)




Proof.The inequality is equivalent to

n∑i=1

pijk(pq−1ij− − pq−1

ijk

)≤ p−jk

(pq−1−j− − pq−1

−jk

).

Denote ai = pijk , bi = pij− − pijk (i = 1, . . . ,m) and A =∑m

i=1 ai , B =∑m

i=1 bi .Then A = p−jk and A + B = p−j− so the inequality reduces to the form

m∑i=1

ai(

(ai + bi )q−1 − aq−1

i

)≤ A

((A + B)q−1 − Aq−1

),

or equivalently

(q − 1)

∫ 1

0

m∑i=1

aibi (ai + tbi )q−2 dt ≤ (q − 1)

∫ 1

0

AB(A + tB)q−2 dt,

which follows from the Lemma.


Quantum entropy von Neumann entropy

Generalization to density matrices

von Neumann entropy

Instead of a discrete probability distribution, we have a density matrix:

ρ ≥ 0, Tr ρ = 1.

Then the von Neumann entropy is defined as

S(ρ) = −Tr(ρ log ρ).




von Neumann entropy

Instead of a discrete probability distribution, we have a density matrix:

ρ ≥ 0, Tr ρ = 1.

Then the von Neumann entropy is defined as

S(ρ) = −Tr(ρ log ρ).

Remark

If ρ has eigenvalues λi (i = 1, . . . , n), then

Sq(ρ) = −n∑

i=1

λi log λi .




History

This formula was first studied by John von Neumann (1903–1957) in his paperThermodynamik quantenmechanischer Gesamtheiten in 1927 and later in hisfamous book Mathematische Grundlagen der Quantenmechanik in 1932.


Quantum entropy Subadditivity inequalities

Strong subadditivity of von Neumann entropy

Partial traces

Let X ∈Mm ⊗Mn, X =∑

i Ai ⊗ Bi . Then the partial traces Tr1 X ∈Mn andTr2 X ∈Mm are defined as

Tr1 X =∑i

Tr(Ai ) · Bi , Tr2 X =∑i

Tr(Bi ) · Ai .




Partial traces



Tr1 X =∑i


Tr(Bi ) · Ai .

RemarkThe partial trace Tr1 is the adjoint of the canonical embedding

Mn 3 B 7→ Im ⊗ B ∈Mm ⊗Mn.

Similarly, Tr2 is the adjoint of the canonical embedding

Mm 3 A 7→ A⊗ In ∈Mm ⊗Mn.

If Eij ∈Mm (i , j = 1, 2, . . . ,m) are the matrix units, then X =∑m

i,j=1 Eij ⊗Xij , so

Tr1 X =m∑i=1

Xii ∈Mn, Tr2 X = [TrXij ]mi,j=1 ∈Mm.




Partial traces



Tr1 X =∑i


Tr(Bi ) · Ai .

RemarkIf ρ12 ∈Mm ⊗Mn is a density matrix, then its reduced densities are

ρ1 = Tr2 ρ12 ∈Mm, ρ2 = Tr1 ρ12 ∈Mn.




TheoremIf ρ12 ∈Mm ⊗Mn is a density matrix with reduced densities ρ1 = Tr2 ρ12 ∈Mm,ρ2 = Tr1 ρ12 ∈Mn, then the subadditivity inequality holds true:

S(ρ12) ≤ S(ρ1) + S(ρ2).

Equality holds if and only if ρ12 = ρ1 ⊗ ρ2.




TheoremIf ρ12 ∈Mm ⊗Mn is a density matrix with reduced densities ρ1 = Tr2 ρ12 ∈Mm,ρ2 = Tr1 ρ12 ∈Mn, then the subadditivity inequality holds true:

S(ρ12) ≤ S(ρ1) + S(ρ2).

Equality holds if and only if ρ12 = ρ1 ⊗ ρ2.

Theorem (Lieb–Ruskai, 1973)

If ρ123 ∈Mm ⊗Mn ⊗Mr is a density matrix with reduced densitiesρ12 = Tr3 ρ123 ∈Mr , ρ23 = Tr1 ρ123 ∈Mm, ρ2 = Tr13 ρ123 ∈Mm ⊗Mr , then thestrong subadditivity inequality holds true:

S(ρ123) + S(ρ2) ≤ S(ρ12) + S(ρ23).




Elliott Hershel Lieb (1932–) Mary Beth Ruskai (1944–)



Partial subadditivity of von Neumann entropy

A reformulationThe strong subadditivity inequality can be written in the form

Tr(ρ123(log ρ123 + I1 ⊗ log ρ2 ⊗ I3 − log ρ12 ⊗ I3 − I1 ⊗ log ρ23)

)≥ 0.






)≥ 0.

Theorem (Isaac Kim, 2012, B.–Petz, 2013)

For a density matrix ρ123 ∈Mm ⊗Mn ⊗Mr the following partial subadditivityinequality holds true:

Tr12

(ρ123(log ρ123 + I1 ⊗ log ρ2 ⊗ I3 − log ρ12 ⊗ I3 − I1 ⊗ log ρ23)

)≥ 0,

i.e., the matrix on the left-hand side is positive semidefinite.






)≥ 0.

Theorem (Isaac Kim, 2012, B.–Petz, 2013)

For a density matrix ρ123 ∈Mm ⊗Mn ⊗Mr the following partial subadditivityinequality holds true:

Tr12

(ρ123(log ρ123 + I1 ⊗ log ρ2 ⊗ I3 − log ρ12 ⊗ I3 − I1 ⊗ log ρ23)

)≥ 0,

i.e., the matrix on the left-hand side is positive semidefinite.

Idea of the proof

Use the joint convexity of the quasi-entropy

SK− log(P‖Q) = Tr(K∗KQ logQ − KQK∗ logP).


Quantum entropy Tsallis entropy


Tsallis entropy

For a density matrix ρ the quantum Tsallis entropy is defined as

Sq(ρ) =1− Tr ρq

q − 1= −Tr(ρ logq ρ)

where the q-logarithm function for q 6= 1 is given by


q − 1(q 6= 1).


Quantum entropy Tsallis entropy


Tsallis entropy

For a density matrix ρ the quantum Tsallis entropy is defined as

Sq(ρ) =1− Tr ρq

q − 1= −Tr(ρ logq ρ)

where the q-logarithm function for q 6= 1 is given by


q − 1(q 6= 1).

Remark

If ρ has eigenvalues λi (i = 1, . . . , n), then

Sq(ρ) =1−

∑ni=1 λ

qi

q − 1.




Theorem (Koenraad Audenaert, 2007)

If ρ12 ∈Mm ⊗Mn is a density matrix with reduced densities ρ1 = Tr2 ρ12 ∈Mm

and ρ2 = Tr1 ρ12 ∈Mn, then for q > 1 the subadditivity inequality holds true:

Sq(ρ12) ≤ Sq(ρ1) + Sq(ρ2),

or equivalently

Tr ρq1 + Tr ρq2 ≤ Tr ρq12.

Equality holds if and only if ρ12 = ρ1 ⊗ ρ2 where min(rank ρ1, rank ρ2) = 1.




RemarkThe strong subadditivity is not true in general. For example, if

ρ123 =

0.02 0 0 0 0 0 0 00 0.02 0 0 0 0 0 00 0 0.4 0 0.12 0 0 00 0 0 0.06 0 0.12 0 00 0 0.12 0 0.06 0 0 00 0 0 0.12 0 0.4 0 00 0 0 0 0 0 0.02 00 0 0 0 0 0 0 0.02

,

then

S2(ρ12) + S2(ρ23)− S2(ρ123)− S2(ρ2) = −0.0108.




A nice special case

If q = 2 and

ρ12 =

[A BB∗ C

]∈M2 ⊗Mn,

then

ρ1 =

[TrA TrB

TrB∗ TrC

], ρ2 = A + C ,

therefore the subadditivity inequality reduces to the neat inequality

TrAC − TrB∗B ≤ TrATrC − TrB∗ TrB.




A nice special case

If q = 2 and

ρ12 =

[A BB∗ C

]∈M2 ⊗Mn,

then

ρ1 =

[TrA TrB

TrB∗ TrC

], ρ2 = A + C ,



Remark

Equality holds if and only if ρ12 = ρ1 ⊗ ρ2 such that min(rank ρ1, rank ρ2) ≤ 1.




A nice special case

If q = 2 and

ρ12 =

[A BB∗ C

]∈M2 ⊗Mn,

then

ρ1 =

[TrA TrB

TrB∗ TrC

], ρ2 = A + C ,



RemarkElementary proof given in the Bulletin of the International Linear Algebra Society.




Proof.Due to unitary invariance, we may assume that A is diagonal. Then the inequalityreduces to the form

2∑i>j

<(biibjj)−∑i 6=j

|bij |2 ≤∑i>j

(aiicjj + ajjcii ).

Since

[A BB∗ C

]≥ 0, therefore A ≥ 0, C ≥ 0 and the 2x2 principal

subdeterminants of X are also non-negative:· · · · ·· aii · bii ·· · · · ·· bii · cii ·· · · · ·

thus aiicii ≥ |bii |2 for i = 1, . . . , n. Therefore, by the AGM inequality,

aiicjj + ajjcii ≥ 2√aiiciiajjcjj ≥ 2

√|bii |2|bjj |2 ≥ 2<(biibjj).




Remark

A generalization was shown by Tsuyoshi Ando (1932–):[TrC · A + BB∗ −AC − TrB∗ · B−CA− TrB · B∗ TrA · C + B∗B

]≥ 0.



Partial subadditivity of Tsallis entropy

A reformulationThe subadditivity inequality for the quantum Tsallis entropy can be written as

Tr(ρ12(logq ρ12 − logq ρ1 ⊗ I2 − I1 ⊗ logq ρ2)

)≥ 0.






)≥ 0.

Open problem

For a density matrix ρ12 ∈Mm ⊗Mn the following partial subadditivity inequalitymight hold true:

Tr1

(ρ12(logq ρ12 − logq ρ1 ⊗ I2 − I1 ⊗ logq ρ2)

)≥ 0.






)≥ 0.

Open problem

For a density matrix ρ12 ∈Mm ⊗Mn the following partial subadditivity inequalitymight hold true:

Tr1

(ρ12(logq ρ12 − logq ρ1 ⊗ I2 − I1 ⊗ logq ρ2)

)≥ 0.

RemarkThe conjecture is true for

• ρ12 = ρ1 ⊗ ρ2 and q > 1,

• ρ12 ∈M2n ⊗M2n and q = 2.


References

T. Ando, Positivity of certain block-matrices, manuscript, 2013.

K. M. R. Audenaert, Subadditivity of q-entropies for q > 1, J. Math. Phys.,48 (2007), 083507.

Adam Besenyei, Denes Petz, Partial subadditivity of entropies, Linear AlgebraAppl., 439 (2013), 3297–3305.

Z. Daroczi, General information functions, Information and Control, 16(1970), 36–51.

S. Furuichi, Information theoretical properties of Tsallis entropies,J.Math.Phys., 47 (2006), 023302.

I. H. Kim, Operator extension of strong subadditivity of entropy, J. Math.Phys., 53 (2012), 122204.

E. H. Lieb, M. B. Ruskai, Proof of the strong subadditivity of quantummechanical entropy, J. Math. Phys., 14 (1973), 1938–1941.


The End

Thank you for your attention!


On some subadditivity inequalities of entropiesabesenyei.web.elte.hu/publications/entropy.pdf ·...

Documents

Transcript of On some subadditivity inequalities of entropiesabesenyei.web.elte.hu/publications/entropy.pdf ·...