On Contrastive Divergence Learning

Miguel A. Carreira-Perpinan∗ Geoffrey E. Hinton

Dept. of Computer Science, University of Toronto6 King’s College Road. Toronto, ON M5S 3H5, Canada

Email: {miguel,hinton}@cs.toronto.edu

Abstract

Maximum-likelihood (ML) learning ofMarkov random fields is challenging becauseit requires estimates of averages that have anexponential number of terms. Markov chainMonte Carlo methods typically take a longtime to converge on unbiased estimates, butHinton (2002) showed that if the Markovchain is only run for a few steps, the learningcan still work well and it approximatelyminimizes a different function called “con-trastive divergence” (CD). CD learning hasbeen successfully applied to various types ofrandom fields. Here, we study the propertiesof CD learning and show that it providesbiased estimates in general, but that the biasis typically very small. Fast CD learningcan therefore be used to get close to an MLsolution and slow ML learning can then beused to fine-tune the CD solution.

Consider a probability distribution over a vector x (as-sumed discrete w.l.o.g.) and with parameters W

p(x;W) =1

Z(W)e−E(x;W) (1)

where Z(W) =∑

x e−E(x;W) is a normalisation con-stant and E(x;W) is an energy function. This classof random-field distributions has found many practicalapplications (Li, 2001; Winkler, 2002; Teh et al., 2003;He et al., 2004). Maximum-likelihood (ML) learning ofthe parameters W given an iid sample X = {xn}N

n=1

can be done by gradient ascent:

W(τ+1) = W(τ) + η∂L(W;X )

∂W

∣

∣

∣

∣

W(τ)

∗Current address: Dept. of Computer Science & Elec-trical Eng., OGI School of Science & Engineering, OregonHealth & Science University. Email: miguel@cse.ogi.edu.

where the learning rate η need not be constant. Theaverage log-likelihood is:

L(W;X ) = 1N

∑Nn=1 log p(xn;W) = 〈log p(x;W)〉0

= −〈E(x;W)〉0 − log Z(W)

where 〈·〉0 denotes an average w.r.t. the data distribu-

tion p0(x) = 1N

∑Nn=1 δ(x − xn). A well-known diffi-

culty arises in the computation of the gradient

∂L(W;X )

∂W= −

⟨

∂E(x;W)

∂W

⟩

0

+

⟨

∂E(x;W)

∂W

⟩

∞

where 〈·〉∞ denotes an average with respect to themodel distribution p∞(x;W) = p(x;W). The average〈·〉0 is readily computed using the sample data X , butthe average 〈·〉∞ involves the normalisation constantZ(W), which cannot generally be computed efficiently(being a sum of an exponential number of terms). Thestandard approach is to approximate the average overthe distribution with an average over a sample fromp(x;W), obtained by setting up a Markov chain thatconverges to p(x;W) and running the chain to equilib-rium (for reviews, see Neal, 1993; Gilks et al., 1996).This Markov chain Monte Carlo (MCMC) approachhas the advantage of being readily applicable to manyclasses of distribution p(x;W). However, it is typicallyvery slow, since running the Markov chain to equilib-rium can require a very large number of steps, and nofoolproof method exists to determine whether equilib-rium has been reached. A further disadvantage is thelarge variance of the estimated gradient.

To avoid the difficulty in computing the log-likelihoodgradient, Hinton (2002) proposed the contrastive di-vergence (CD) method which approximately followsthe gradient of a different function. ML learning min-imises the Kullback-Leibler divergence

KL (p0‖p∞) =∑

x

p0(x) logp0(x)

p(x;W).

CD learning approximately follows the gradient of the

difference of two divergences (Hinton, 2002):

CDn = KL (p0‖p∞) − KL (pn‖p∞) .

In CD learning, we start the Markov chain at the datadistribution p0 and run the chain for a small numbern of steps (e.g. n = 1). This greatly reduces boththe computation per gradient step and the varianceof the estimated gradient, and experiments show thatit results in good parameter estimates (Hinton, 2002).CD has been applied effectively to various problems(Chen and Murray, 2003; Teh et al., 2003; He et al.,2004), using Gibbs sampling or hybrid Monte Carlo asthe transition operator for the Markov chain. How-ever, it is hard to know how good the parameter esti-mates really are, since no comparison was done withthe real ML estimates (which are impractical to com-pute). There has been a little theoretical investigationof the properties of contrastive divergence (MacKay,2001; Williams and Agakov, 2002; Yuille, 2004), butimportant questions remain unanswered: Does it con-verge? If so how fast, and how are its convergencepoints related to the true ML estimates?

In this paper we provide some theoretical and empir-ical evidence that contrastive divergence can, in fact,be the basis for a very effective approach for learn-ing random fields. We concentrate on Boltzmann ma-chines, though our results should be more generallyvalid. First, we show that CD provides biased esti-mates in general: for almost all data distributions, thefixed points of CD are not fixed points of ML, andvice versa (section 2). We then show, by comparingCD and ML in empirical tests, that this bias is small(sections 3–4) and that an effective approach is to useCD to perform most of the learning followed by a shortrun of ML to clean up the solution (section 5).

To eliminate sampling noise from our investigations,we use fairly small models (with e.g. 48 parameters)for which we can compute the exact model distributionand the exact distribution at each step of the Markovchain at each stage of learning. Throughout, we takeML learning to mean exact ML learning (i.e., with n →∞ in the Markov chain) and CDn learning to meanlearning using the exact distribution of the Markovchain after n steps. The sampling noise in real MCMCestimates would create an additional large advantagethat favours CD over ML learning, because CD hasmuch lower variance in its gradient estimates.

1 ML and CD learning for two types

of Boltzmann machine

We will concentrate on two types of Boltzmann ma-chine, which are a particular case of the model ofeq. (1). In Boltzmann machines, there are v visible

units x = (x1, . . . , xv)T that encode a data vector, andh hidden units y = (y1, . . . , yh)T ; all units are binaryand take values in {0, 1}.Fully visible Boltzmann machines have h = 0 andthe visible units are connected to each other. The en-ergy is then E(x;W) = − 1

2xT Wx, where W = (wij)

is a symmetric v× v matrix of real-valued weights (forsimplicity, we do not consider biases). We denote sucha machine by VBM(v). In VBMs, the log-likelihoodhas a unique optimum because its Hessian is negativedefinite. Since ∂E/∂wij = −xixj , ML learning takesthe form

w(τ+1)ij = w

(τ)ij + η

(

〈xixj〉0 − 〈xixj〉∞)

while CDn learning takes the form

w(τ+1)ij = w

(τ)ij + η

(

〈xixj〉0 − 〈xixj〉n)

.

Here, pn(x;W) = pn = Tnp0 is the nth-step distri-bution of the Markov chain with transition matrix1 T

started at the data distribution p0.

Restricted Boltzmann machines (Smolensky,1986; Freund and Haussler, 1992) have connectionsonly between a hidden and a visible unit, i.e.,they form a bipartite graph. The energy is thenE(x,y;W) = −yT Wx, where we have v visible unitsand h hidden units, and W = (wij) is an h × v ma-trix of real-valued weights. We denote such a machineby RBM(v, h). By making h large, an RBM can begiven far more representational power than a VBM,but the log-likelihood can have multiple maxima. Thelearning is simpler than in a general Boltzmann ma-chine because the visible units are conditionally inde-pendent given the hidden units, and the hidden unitsare conditionally independent given the visible units.One step of Gibbs sampling can therefore be carriedout in two half-steps: the first updates all the hid-den units and the second updates all the visible units.Equivalently, we can write T = TxTy where tx;j,i =p(x = j|y = i;W) and ty;i,j = p(y = i|x = j;W).

Since ∂E/∂wij = −yixj , ML learning takes the form

w(τ+1)ij = w

(τ)ij + η

(⟨

〈yixj〉p(y|x;W)

⟩

0− 〈yixj〉∞

)

while CDn learning takes the form

w(τ+1)ij = w

(τ)ij + η

(⟨

〈yixj〉p(y|x;W)

⟩

0− 〈yixj〉n

)

.

2 Analysis of the fixed points

A probability distribution over v units is a vector of2v real-valued components (from 0 to 2v − 1 in bi-

1Here and elsewhere we omit the dependence of T onthe parameter values W to simplify the notation.

nary notation) that lives in the 2v–dimensional sim-

plex ∆2v = {x ∈ R2v

: xi ≥ 0,∑2v

i=1 xi = 1}. Eachcoordinate axis of R

2v

corresponds to a state (binaryvector). We write such a distribution as p(·), when em-phasising that it is a function, or as a vector p, whenemphasising that it is a point in the simplex. We definea Markov chain through its transition operator, whichis a stochastic 2v × 2v matrix T. We use the Gibbssampler as the transition operator because of its sim-plicity and its wide applicability to many distributions.For Boltzmann machines with finite weights, the Gibbssampler converges to a stationary distribution whichis the model distribution p(x;W). For an initial dis-tribution p we then have pn = Tnp for n = 1, 2, . . . ;and p(x;W) = p∞ = T∞p for any p ∈ ∆2v , i.e.,T∞ = p∞1T where 1 is the vector of ones.

VBMs or RBMs define a manifold M within the sim-plex, parameterised by W, with wij ∈ R (we ignorethe case of infinite weights, corresponding to distribu-tions in the intersection of M and the simplex bound-ary). The learning (ML or CD) starts at a point inM and follows the (approximate) gradient of the log-likelihood, thus tracing a trajectory within M.

For ML with gradient learning, the fixed points are thezero-gradient points (maxima, minima and saddles),which satisfy 〈g〉0 = 〈g〉∞ where g = ∂E/∂W. Forn-step CD, the fixed points satisfy 〈g〉0 = 〈g〉n. In thissection we address the theoretical question of whetherthe fixed points of ML are fixed points of CD and viceversa. We show that, in general,

∃p0 : 〈g〉0 = 〈g〉∞ 6= 〈g〉1∃p0 : 〈g〉0 = 〈g〉1 6= 〈g〉∞ .

We give a brief explanation of a framework foranalysing the fixed points of ML and CD; full detailsappear in Carreira-Perpinan and Hinton (2004). Theidea is to fix a value of the weights and so a value of themoments (defined below), determine which data dis-tributions p0 have such moments (i.e., the opposite ofthe learning problem) and then determine under whatconditions ML and CD agree over those distributions.Call G the |W| × 2v matrix of energy derivatives, de-fined by

Gix = − ∂E

∂wi

(x;W)

where we consider W as a column vector with |W|elements and the state x takes values 0, 1, . . . , 2v − 1in the case of v binary variables. We can then writethe moments s =

⟨

− ∂E∂W

⟩

p= −〈g〉p of a distribution

p as s = Gp, i.e., s is a linear function of p. Call T

the transition matrix for the sampling operator withstationary distribution p∞ (so we have p∞ = Tp∞).In general, both G and T are functions of W.

PSfrag replacements

p = (0001)T

1000

0100

0010

14

14

14

14

13

13

130

w = 0

w = ∞

w = −∞

Figure 1: The simplex in 4 dimensions of the VBM(2).The model has a single parameter W = w ∈ (−∞,∞).The tetrahedron represents the simplex, i.e., the set{p ∈ R

4 : 1T p = 1, p ≥ 0}. The tetrahedron cornerscorrespond to the pure states, i.e., the distributionsthat assign all the probability to a single state. The redvertical segment is the manifold of VBMs, i.e., the dis-tributions reachable by a VBM(2) for w ∈ (−∞,∞).The ML estimate of a data distribution is its orthog-onal projection on the model manifold. The CD esti-mate only agrees with the ML one for data distribu-tions in the 3 shaded planes in the inset.

Now consider a fixed value of W and let p∞ be itsassociated model distribution. Thus its moments ares∞ = Gp∞. We define two sets P0 and P1 that dependon s∞. First, the set of data distributions p0 that havethose same moments s∞ is:

P0 =

p0 ∈ R2v

:Gp0 = s∞1T p0 = 1p0 ≥ 0

.

Each distribution in P0 gives a fixed point of ML. Like-wise, define the set of data distributions p0 whose dis-tribution p1 = Tp0 (first step in the Markov chain,i.e., what CD1 uses instead of p∞) has the same mo-ments as s∞:

P1 =

p0 ∈ R2v

:GTp0 = s∞1T p0 = 1p0 ≥ 0

.

Both P0 and P1 are nonempty since p∞ is in both.Now we can reformulate the problem in terms of thesets P0 and P1. For example, a distribution with p0 ∈P0 and p0 /∈ P1 satisfies 〈g〉0 = 〈g〉∞ 6= 〈g〉1 and thusgives a fixed point of ML but not of CD (that is, theCD learning rule would move away from such a p∞).

In general (and ignoring technical details regarding theinequality p0 ≥ 0), P0 and P1 are linear subspaces ofthe same dimension because G is full rank (the mo-ments are l.i.) and T is generally full rank. Thus wecannot generally expect P0 = P1, so points with CDbias are the rule; points in P0 ∩ P1 have no CD biasbut are the exception (the intersection being a lower-dimensional subspace). We can make the statementprecise for a given model. For example, for VBM(2)with Gibbs sampling we have G = (0 0 0 1) (G hap-pens to be independent of W for VBMs), we can com-pute T and we can work out the set P0 ∩ P1 for ev-ery s∞ value. The resulting set, which contains all thedata distributions for which ML and CD have the samefixed points (i.e., no bias), is the union (intersectedwith the simplex) of the 3 planes: p11 = 0; p11 =14 ; 3p01 + p11 = 1, where we write a distribution asa 4-dimensional vector p = (p00 p01 p10 p11)

T , corre-sponding to the probabilities of the 4 states 00, . . . , 11(see fig. 1). This set has measure zero in the simplex,so CD is biased for almost every data distribution.

A reachable distribution p0 ∈ M is a fixed point forboth CD and ML, as it is invariant under T (Hinton,2002). This is consistent with the above argument,as p0 = p∞ ∈ P0 ∩ P1. The distributions of practi-cal interest are typically unreachable because real dataare nearly always more complicated than our compu-tationally tractable model of it.

In summary, we expect that for almost every datadistribution p0, the fixed points of ML are not fixedpoints of CD and vice versa. This means that, in gen-eral, CD is a biased learning algorithm. Our argu-ment can be applied to models other than Boltzmannmachines, transition operators other than Gibbs sam-pling, and to n > 1 (writing Tn instead of T). Whatdetermines whether CD is biased are the hyperplanesdefined by the matrices G and GT. However, non-trivial models (i.e., defining a lower-dimensional man-ifold) may exist for which CD is not biased; an exam-ple is Gaussian Boltzmann machines (Williams andAgakov, 2002) and Gaussian distributions, at least in2D (Carreira-Perpinan and Hinton, 2004).

This analysis does not imply that CD learning con-verges (to a stable fixed point); at present, we do nothave a proof for this. But if CD does converge, asit appears to in practice and in all our experiments,it can only converge to a fixed point. Naturally, MLdoes converge to its stable fixed points (maxima) fromalmost everywhere, since it follows the exact gradientof an objective function; in the noisy sampling casethat is used in practice, it also converges providedthe learning rate η follows a Robbins-Monro sched-ule (Benveniste et al., 1990), since the rule performsstochastic gradient learning.

3 Experiments with fully visible BMs

Since CD is biased with respect to ML for almost alldata distributions, we now investigate empirically themagnitude of the bias. In all experiments in the pa-per, ML and CD are tested under exactly the sameconditions (unless otherwise stated). Both ML andCD learning use the same initial weight vectors, thesame constant learning rate η = 0.9 and the samemaximum of 10 000 iterations (which is rarely reachedfor VBMs), stopping when ‖e‖∞ < 10−7 (where e =〈xixj〉0 − 〈xixj〉∞ is the gradient vector for ML, ande = 〈xixj〉0 − 〈xixj〉1 is the approximate gradient forCD1). All the experiments use n = 1 step of Gibbssampling with fixed ordering of the variables for CDlearning, because this should produce the greatest bias(since CD −−−−→

n→∞ML). Although each of our simu-

lated models is necessarily small, our empirical resultshold for a range of model sizes and conditions, whichsuggests they may be more generally valid.

In this section we consider fully visible Boltzmannmachines, denoted VBM(v), which have a single MLoptimum. It appears that CD has a single conver-gence point too: for v = 2 we can prove this, andfor v ∈ {3, . . . , 10} we checked empirically by runningCD from many different initial weight vectors that italways converged to the same point (up to a smallnumerical error). Thus, we assume that CD has aunique convergence point for VBM(v). This allows usto characterise the bias for this model class by sam-pling many data distributions and computing the con-vergence point of ML and of CD.

For a given value of v we sampled a number (as largeas computationally feasible) of data distributions uni-formly distributed in the simplex in 2v variables (seeCarreira-Perpinan and Hinton, 2004 for details of howto generate these samples). Then we ran ML and CDstarting with W = 0 because small weights give fasterconvergence on average. The results for experimentsfor v ∈ {2, . . . , 10} were qualitatively similar. Forv = 2 it was feasible to sample 104 data distributionsand the results are summarised in figures 2–5.

The histograms in figs. 2–3 show the bias is very smallfor most distributions. Fig. 4 shows that the KL error(for both ML and CD) is small for data distributionsnear the simplex centre. For less vague data distribu-tions there is more variability, with some distributionshaving a low error and some having a much higherone. Generally speaking, the distributions having thehighest KL error for ML (i.e., the distributions thatare modelled worst by the VBM) are also the onesthat have the highest bias. Most of these lie near theboundaries of the simplex, particularly near the cor-ners. However, not all corners and boundaries are far

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

100

200

300

400ML

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

100

200

300

400CD

PSfrag replacements

KL (p0‖pML)

KL (p0‖pCD)

Figure 2: Histograms of KL (p0‖pML) andKL (p0‖pCD) for VBM(2) after learning on 104

data distributions, where pML and pCD are theconvergence points for ML and CD, respectively. Theperformance of CD is very close to ML on average.

from the model manifold; this depends on the geome-try of the model. In fig. 4 (for v = 2) we can discernthe geometry of the simplex in fig. 1. The discontinu-ity in the slope at a Euclidean distance ‖p0 − u‖ justless than 0.3 corresponds to the radius of the inscribedsphere. The branch in the scatterplot which has lowerror corresponds to the direction passing through thecentre and the simplex corner corresponding to thedelta distribution of the (1, 1) state (i.e., along theVBM manifold). The other branch which has higherror and more data points corresponds to the direc-tions passing through the centre and any of the other3 corners (i.e., away from the VBM manifold).

As v increases, most of the volume of the simplex con-centrates at a distance intermediate between the cor-ners and the centre, close to the radius of the inscribedhypersphere. Consequently, a finite uniform samplecontains essentially no points near the boundaries ofthe simplex, which produce the highest bias. For largev, CD has very small bias for nearly all randomly cho-sen data distributions. Only those rare distributionsnear the simplex boundaries produce a significant bias,but these are important in practice: real-world distri-butions are often near the boundaries (though not asfar as the corners) because large parts of the data spacehave negligible probability.

Fig. 5 shows some typical learning curves. Both CDand ML decrease in a similar way, converging at thesame rate (first-order), taking the same number of it-erations to converge to a given tolerance. CD yields a

0 0.002 0.004 0.006 0.008 0.01 0.012 0.0140

500

1000

1500

2000

2500

3000

3500

PSfrag replacements

Bias KL (pML, pCD)

Figure 3: Histogram of the symmetrised KL diver-gence for VBM(2) between the model distributionsfound by ML and CD for all 104 data distributions.This shows that the bias of CD is almost always verysmall (< 5% of the KL error obtained by ML for thesame distribution; data not shown). However, datadistributions do exist that have a relatively large bias.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

PSfrag replacements

KL

(p0‖·

)

Euclidean distance p0 ↔ simplex centre

Figure 4: KL error for ML KL (p0‖pML) (red ◦) andfor CD KL (p0‖pCD) (black +) vs Euclidean distance‖p0 − u‖ between the data distribution and the uni-form distribution (centre of the simplex). This Eu-clidean distance gives a linear ordering of the datadistributions (lowest Euclidean distance: p0 is the uni-form distribution, ‖p0 − u‖ = 0; highest: p0 is one ofthe corners of the simplex, ‖p0 − u‖ =

√1 − 2−v). For

clarity, not all 104 distributions are plotted.

higher KL error. In the lower example, the CD curveincreases slightly at the end, suggesting it came closeto the ML optimum but then moved away.

In summary, we find the CD bias to be very small formost distributions and to be highest (but still small)with real-world distributions (near the simplex bound-ary). This bias is small in relative terms (compared tothe KL error for ML) and absolute terms (comparedto the simplex dimensions). CD and ML converge atabout the same rate, but an ML iteration costs much

100

101

102

10−0.09

10−0.08

10−0.07

PSfrag replacements

CD

MLK

L(p

0‖·

)

100

101

10−0.51

10−0.5

PSfrag replacements

CD

ML

KL

(p0‖·

)

Number of iterations

Figure 5: Learning curves for ML and CD for 2 ran-domly chosen data distributions. Axes are in log scale.

more than a CD one in a MCMC implementation.

4 Experiments with restricted BMs

RBMs are practically more interesting than VBMs,since they have a higher representational power. Theyalso introduce a new element that complicates ourstudy: the existence of multiple local optima of MLand CD. This prevents the characterisation of the biasover a large number of data distributions. Instead, wecan only afford to select one data distribution (or afew) and try to characterise the set of all optima ofML and CD.

Given a data distribution, we generate a collection of60 random initial weight vectors W and compute allthe optima of ML and CD that are reachable from anyof the initial weight vectors or from the optima foundby the other learning method. This requires iterat-ing over the current set of optima with ML and CD,until no new optima are found. The result is a bipar-tite, self-consistent convergence graph where an arrowA → B indicates that ML optimum A converges to CDoptimum B under CD, or CD optimum A converges toML optimum B under ML. Using many different initialweight vectors should give a representative collection

of optima and a Good-Turing estimator (Good, 1953)can be used as a coarse indicator of how many op-tima we missed. The graph depends on how we decidewhether two very similar optima are really the same.The threshold and number of parameter updates haveto be carefully chosen so that truly different optimaare not confused but two discoveries of the same op-timum are not considered different. We found thatusing the symmetrised KL distance with a thresholdof 0.01 worked well with 105 parameter updates.

We ran experiments for various values of v and h andvarious data distributions. Fig. 6 summarises the re-sults for one representative case, corresponding to:v = 6, h = 4. The data distribution was generatedfrom a data set of 4 binary vectors by adding an ex-tra count of 0.1 to each possible binary vector andrenormalising (thus it is close to the simplex bound-ary). We used 105 iterations and 60 different initialweight vectors: 20 random N (0, σ = 0.1), 20 randomN (0, σ = 1) and 20 random N (0, σ = 10). We found27 ML optima and 28 CD optima, and missed about3 and 4, respectively (Good-Turing estimate).

Panel A shows a 2D visualisation of ML optima (red ◦)and CD optima (black +), i.e., the visible-unit distri-butions pML and pCD, and their convergence relations.The blue ✩ is the data distribution p0 (of the visi-ble variables). To avoid cluttering the plot, pairs ofarrows A � B are drawn as a single line without ar-rowheads A — B. Note that many such lines are tooshort to be distinguished. The 2D view was obtainedwith SNE (Hinton and Roweis, 2003) which tries topreserve the local distances. Using a perplexity of 3to determine the local neighborhood size, SNE givesa better visualisation than projecting onto the first 2principal components.

This panel shows an important and robust phenome-non: ML and CD optima typically come in pairs thatconverge to each other. The CD optimum always hasa greater or equal KL error than its associated ML op-timum but the difference is small. These pairs are tobe expected for CDn when n is large because CD be-comes ML as n → ∞. However they occur very ofteneven for n = 1, as shown. Panels B–C show that thechoice of initial weights has a much larger effect on theKL error than the CD bias.

5 Using CD to initialise ML

The previous experiments show that CD takes us closeto an ML optimum, but that a small bias remains. Anobvious way to eliminate this bias is to use increasingvalues of n as training progresses. In this section weexplore a crude version of this strategy: run CD untilit is close to convergence then use a short run of ML

A

1

2

3

4

5

6

7

8

9

10

11

12

13

14 15

16

17

18

19

20

21

22

23

24 25

26

27

1 2

3

4

5

6

7

8

9

10

11

12 13

14

15

16

17

18

19 20

21

22

23

24

25 26

27

28

B

0 0.1 0.2 0.3 0.4 0.5 0.60

0.1

0.2

0.3

0.4

0.5

0.6

PSfrag replacements

KL

(p0‖p

CD)

KL (p0‖pML)

C0 0.1 0.2 0.3 0.4 0.5 0.6

0

5

10

15

20

25

30ML

0 0.1 0.2 0.3 0.4 0.5 0.60

5

10

15

20

25

30CD

PSfrag replacements

KL (p0‖pML)

KL (p0‖pCD)

Figure 6: Empirical study of the convergence points ofML and CD for RBM(6, 4) with a single data distri-bution. A: 2D SNE visualization of the points (ML:red ◦; CD: black +), and convergence relations amongthem with ML and CD (a line without an arrowheadstands for two arrows �, to avoid clutter). B: KL er-ror of CD vs ML from the same initial weight vectors,for 60 random initial weight vectors. C: histograms ofthe KL error of ML and CD.

100

101

102

103

104

105

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.31.41.51.61.71.81.9

22.12.22.32.42.5

CDCD−MLML

PSfrag replacements

KL

(p0‖·

)

CPU cost

CD (104 its.)

ML (104 its.)

ML (103 its.)

Figure 7: Learning curves for CD-ML and ML, whereeach ML iteration is scaled to cost as much as 20 CDiterations. The axes have a log scale, so CD-ML is anorder of magnitude faster for about the same final KL.

to reduce the bias. We call this strategy CD-ML. Weuse a data distribution which is more representative ofa real problem. It is located on the simplex boundaryand derived from the statistics of all 3 × 3 patches in11000 16 × 16 images of handwritten digits from theUSPS dataset. The 256 intensity levels are thresholdedat 150 to produce 9-dimensional binary vectors (thusv = 9). p0 is the normalised counts of each of thesebinary vectors in the 2 156 000 patches.

We used 60 different initial weight vectors: 20 ran-dom N (0, σ = 1

3 ), 20 random N (0, σ = 1) and 20random N (0, σ = 3). For each starting condition, twotypes of learning were used: ML learning for 104 iter-ations; and CD learning for 104 iterations, followed bya shorter run of ML learning. We ran experiments forh ∈ {1, . . . , 8} and found that there is a unique MLoptimum and several CD optima of varying degrees ofbias. If CD learning was followed by 1 000 iterations ofML, all the CD optima converged to the ML optimum.

Fig. 7 shows the learning curves (i.e., the errorKL (p0‖·) as a function of estimated CPU time) forthe different methods with h = 8: CD (blue line), theshort ML run (1 000 iterations) following CD (greenline) and ML (red line), for a selected starting condi-tion. We assume that each ML iteration costs 20 timesas much as each CD iteration (a reasonable estimatefor the size of this RBM). CD-ML reaches the sameerror as ML but at a small fraction of the cost. Notehow sharply the CD-ML curve drops when we switchto ML, suggesting good performance can be achievedwith very few of the expensive ML iterations.

6 Conclusion

Our first result is negative: for two types of Boltz-mann machine we have shown that, in general, thefixed points of CD differ from those of ML, and thusCD is a biased algorithm. This might suggest thatCD is not a competitive method for ML estimation ofrandom fields. Our remaining, empirical results showotherwise: the bias is generally very small, at leastfor Gibbs sampling, since CD typically converges verynear an ML optimum. And this small bias can be elim-inated by running ML for a few iterations after CD,i.e., using CD as an initialisation strategy for ML, witha total computation time that is much smaller thanthat of full-fledged ML (which will also have slight biasbecause the Markov chain cannot be run forever).

The theoretical analysis of CD is difficult because ofthe complicated form that the p1 (or pn) distributiontakes; p1 is a moving target that changes with W in acomplicated way, and depends on the sampling schemeused (e.g. Gibbs sampling). As a result, very few the-oretical results about CD exist. MacKay (2001) gavesome examples of CD bias, but these used unusualsampling operators. Our analysis applies to any modeland operator (through the G and T matrices), in par-ticular generally applicable operators such as Gibbssampling. Williams and Agakov (2002) showed that,for 2D Gaussian Boltzmann machines, CD is unbiasedand typically decreases the variance of the estimates.Yuille (2004) gives a condition for CD to be unbiased,though this condition is difficult to apply in practice.

One open theoretical problem is whether the exact ver-sion of CD converges (we believe that it does). As-suming we can prove convergence for the exact case,the right tools to use to prove it in the noisy caseare probably those of stochastic approximation (Ben-veniste et al., 1990; Yuille, 2004).

Acknowledgements

This research was funded by NSERC and CFI. GEHis a fellow of CIAR and holds a CRC chair.

References

A. Benveniste, M. Metivier, and P. Priouret. Adap-

tive Algorithms and Stochastic Approximations.Springer-Verlag, 1990.

M. A. Carreira-Perpinan and G. E. Hinton. On con-trastive divergence (CD) learning. Technical report,Dept. of Computer Science, University of Toronto,2004. In preparation.

H. Chen and A. F. Murray. Continuous restrictedBoltzmann machine with an implementable train-

ing algorithm. IEE Proceedings: Vision, Image and

Signal Processing, 150(3):153–158, June 20 2003.

Y. Freund and D. Haussler. Unsupervised learningof distributions on binary vectors using 2-layer net-works. In NIPS, pages 912–919, 1992.

W. Gilks, S. Richardson, and D. J. Spiegelhalter, edi-tors. Markov Chain Monte Carlo in Practice. Chap-man & Hall, 1996.

I. J. Good. The population frequencies of speciesand the estimation of population parameters.Biometrika, 40(3/4):237–264, Dec. 1953.

X. He, R. S. Zemel, and M. A. Carreira-Perpinan. Mul-tiscale conditional random fields for image labeling.In CVPR, pages 695–702, 2004.

G. Hinton and S. T. Roweis. Stochastic neighbor em-bedding. In NIPS, pages 857–864, 2003.

G. E. Hinton. Training products of experts by mini-mizing contrastive divergence. Neural Computation,14(8):1771–1800, Aug. 2002.

S. Z. Li. Markov Random Field Modeling in Image

Analysis. Springer-Verlag, 2001.

D. J. C. MacKay. Failures of the one-steplearning algorithm. Available online athttp://www.inference.phy.cam.ac.uk/mackay/

abstracts/gbm.html, 2001.

R. M. Neal. Probabilistic inference using Markovchain Monte Carlo methods. Technical ReportCRG–TR–93–1, Dept. of Computer Science, Uni-versity of Toronto, Sept. 1993. Available on-line at ftp://ftp.cs.toronto.edu/pub/radford/review.ps.Z.

P. Smolensky. Information processing in dynamicalsystems: Foundations of harmony theory. In D. E.Rumelhart and J. L. MacClelland, editors, Par-

allel Distributed Computing: Explorations in the

Microstructure of Cognition. Vol. 1: Foundations,chapter 6. MIT Press, 1986.

Y. W. Teh, M. Welling, S. Osindero, and G. E. Hinton.Energy-based models for sparse overcomplete repre-sentations. Journal of Machine Learning Research,4:1235–1260, Dec. 2003.

C. K. I. Williams and F. V. Agakov. An anal-ysis of contrastive divergence learning in Gauss-ian Boltzmann machines. Technical ReportEDI–INF–RR–0120, Division of Informatics, Uni-versity of Edinburgh, May 2002.

G. Winkler. Image Analysis, Random Fields and

Markov Chain Monte Carlo Methods. Springer-Verlag, second edition, 2002.

A. Yuille. The convergence of contrastive divergences.To appear in NIPS, 2004.