Active Learning for Sparse Bayesian Multilabel Classiﬁcation€¦ ·...

Active Learning for Sparse Bayesian Multilabel

Classification

Deepak Vasisht

MIT

[email protected]

Andreas Damianou

University of Sheffield, UK

[email protected]

Manik Varma

Microsoft Research

[email protected]

Ashish Kapoor

Microsoft Research

[email protected]

ABSTRACTWe study the problem of active learning for multilabel clas-sification. We focus on the real-world scenario where theaverage number of positive (relevant) labels per data pointis small leading to positive label sparsity. Carrying out mu-tual information based near-optimal active learning in thissetting is a challenging task since the computational com-plexity involved is exponential in the total number of la-bels. We propose a novel inference algorithm for the sparseBayesian multilabel model of [17]. The benefit of this alter-nate inference scheme is that it enables a natural approxi-mation of the mutual information objective. We prove thatthe approximation leads to an identical solution to the exactoptimization problem but at a fraction of the optimizationcost. This allows us to carry out e�cient, non-myopic, andnear-optimal active learning for sparse multilabel classifica-tion. Extensive experiments reveal the e↵ectiveness of themethod.

Categories and Subject DescriptorsI.2.6 [Artificial Intelligence]: Learning

General TermsAlgorithms, Performance, Machine Learning, Optimization

KeywordsMulti-label Learning; Active Learning; Mutual Information

1. INTRODUCTIONThe goal in multilabel classification is to learn a classi-

fier which can automatically tag a data point with the mostrelevant set of labels. This is in contrast to multi-class clas-sification where only a single label needs to be predictedper data point. Our objective, in this paper, is to developan e�cient, non-myopic and near-optimal active learningalgorithm for multilabel classification employing a mutualinformation based data point selection criterion.

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for profit or commercial advantage and that copies bear this notice and the full citation

on the first page. Copyrights for components of this work owned by others than the

author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or

republish, to post on servers or to redistribute to lists, requires prior specific permission

and/or a fee. Request permissions from [email protected].

KDD’14, August 24–27, 2014, New York, NY, USA.

Copyright is held by the owner/author(s). Publication rights licensed to ACM.

ACM 978-1-4503-2956-9/14/08 ...$15.00.

http://dx.doi.org/10.1145/2623330.2623759 .

There has been much recent interest in sparse multilabellearning [2, 15, 17]. In this scenario, each data point ex-hibits positive label sparsity in that only a small fraction ofany point’s labels are ever marked as positive or relevant.Note that this is a typical setting in most real world ap-plications – the average number of positive labels per datapoint ranges from less than 5% on most UCI data sets toless than 0.001% on Wikipedia and other web data sets [2].Sparse multilabel learning therefore presents an important,real-world and novel challenge to active learning techniques.

Traditional active learning techniques for multilabel learn-ing do not take positive label sparsity into account [11, 13,20,21,31]. Furthermore, these techniques are myopic and ac-tively select only a single data point at a time for annotation.As a result, there is no provable guarantee that the final setof annotated points is optimal or even near-optimal. Whilemutual information based data sampling can provide near-optimal guarantees in certain regression settings [18,19], it isnon-trivial to extend those to the active multilabel setting.

In this paper, we first develop an alternate inference tech-nique for the sparse Bayesian multi-label graphical modelof [17] which allows us to carry out e�cient mutual infor-mation based active learning. We develop an approximationto the mutual information which is tightly coupled to our in-ference algorithm and which can be optimized much moree�ciently than the exact mutual information. This approxi-mation allows us to develop an active learning strategy thathas the following desirable properties:

• The set of annotated points selected by our algorithmis provably near optimal.

• The selection of annotated points can be done in timeindependent of the size of the label space if all labelsof the point are to be annotated, as opposed to theexisting methods which scale linearly (at best) withthe number of labels.

• The train and test times are considerably reduced dueto the reduction of the label vector to a low dimen-sional space using a compressed sensing matrix.

• The method can be easily adapted to be integratedwith di↵erent compressed sensing approaches in liter-ature [3, 5, 8, 10,12,15–17,28,29,32,33].

• The method takes into account label sparsity whichhas been proven to be a desirable characteristic in pastwork on multilabel classification [2, 15,17].

• The method allows us to generalize to other activelearning scenarios like active diagnosis (selecting whichlabels to annotate for a point) and generalized activelearning (selecting both the points and the labels toannotate).

Our main technical contribution is Theorem 1 where weprove that, at any given data point selection step, our ap-proximation leads to an identical solution as would havebeen obtained using exact mutual information in the limit-ing case, but at a fraction of the optimization cost. Further-more, by performing inference in a sparse Gaussian Processregression model, we can leverage the mutual informationguarantees to provably show that the entire subset of datapoints selected for annotation by our proposed algorithmis near-optimal. This enables us to carry out active sparsemultilabel learning in an e�cient, non-myopic and theoret-ically principled fashion. Finally, as a natural extension ofour method, the matrix inversions necessary for traditionalactive learning can be carried out in time that is independentof the size of the label vector for a point.

Extensive experiments reveal that our proposed activelearning strategy consistently outperforms the state-of-the-art approaches based on variance sampling, SVM based ac-tive learning proposed in [20] and random sampling base-lines. Furthermore, we demonstrate that our inference pro-cedure leads to a better estimate of the variance needed forBayesian active learning as compared to existing techniques.We also show that our method has significant gains (20 timesgain over the delicious dataset with 983 labels) in timeconsumption over the state-of-the-art [20] as the number oflabels increases. Finally, we demonstrate that our proposedapproach can also be used to sample both labels as wellas data points, thereby allowing learning at an even lowerannotation cost as compared to existing techniques whichhave so far focussed on all the labels being annotated for aselected data point.

Our main contributions are as follows: (a) a novel infer-ence procedure for the Bayesian sparse multilabel graphicalmodel of [17]; (b) an approximation to the mutual infor-mation which follows naturally from our proposed inferenceprocedure; (c) a proof that our proposed approximation se-lects the same data point as would have been chosen by op-timizing mutual information (in the limiting case); and (d)an extensive evaluation of the proposed scheme on a diverserange of real world datasets to prove the gains obtained byour algorithm. These contributions allow us to carry oute�cient mutual information based active sparse multilabellearning for the first time (to the best of our knowledge).

2. RELATED WORKPast work in active learning has primarily focused on sin-

gle label classification tasks [24]. However, multilabel clas-sification has received wide interest in recent times [2,3,5,8,10, 12, 15–17, 28–30, 32, 33]. Active learning is more advan-tageous in this setting as label acquisition costs are higherfor multilabel scenario. However, research in active learningfor multilabel classification is still in its preliminary stage.Current multilabel active learning strategies train a binarySVM classifier for each label and combine the SVM marginsusing di↵erent heuristics to select the training set from apool of available data for annotation [11, 21, 27]. [27] takesthe average of the SVM uncertainties obtained by Platt scal-

ing [23] and uses it as a selection criterion for greedily select-ing the set of points to be annotated. [21] uses Mean MaxLoss(MML) and Max Loss(ML) as two separate selectioncriteria for annotation of points. [31] uses logistic regressionto predict the number of positive labels for each point anduses another SVM based heuristic to minimize expected lossfor point selection. The state-of-the-art method [20] uses acombination of deviation from mean label cardinality andan SVM-based loss function for active learning. To the bestof our knowledge, all these works are myopic and do notprovide any theoretical guarantees on the optimality of theselected set.

On the other hand, mutual information has proved to bea very useful criterion for active learning with Gaussian Pro-cesses [18,19,26], both empirically and theoretically. In theseschemes, each point is selected from a pool of available unla-beled data, so as to maximize the information gain over theremaining unlabeled data. The approximate submodularityof mutual information ensures that a set of points selectedgreedily based on this criterion will be near-optimal [19].However, extending mutual information to multilabel clas-sification is non-trivial as a straightforward computation ofmutual information over the label space is exponential in thenumber of labels.

The Bayesian compressed sensing model proposed in [17]is closest to our work, in the sense that it models the multil-abel classification task as a Gaussian Process. However, theapproximations made in [17], for the inference procedure tobe tractable, do not preserve the covariance matrix acrosslabels and hence, render mutual information based activelearning infeasible. The inference procedure proposed in sec-tion 4, however, preserves this covariance matrix and allowsfor e�cient mutual information based active learning.

3. BACKGROUNDOur work builds upon a recently proposed Bayesian ar-

chitecture for multilabel classification [17]. We briefly sum-marize this framework here. Given a set of N data pointsX = {x

i

} and the corresponding sparse l-dimensional bi-nary label vectors Y = {y

i

}, the key idea was to first reducethe classification task to a lower multidimensional regressiontask via a random projection matrix �, and then during testtime, infer the labels for an unseen point via approximateBayesian inference.

Specifically, a factor graph corresponding to the modelconsists of three potential functions: (1) the first poten-

tial term fx

i

(W, zi

) = exp[� ||WT

x

i

�z

i

||22�

2 ] defines a regres-sion function from each input point x

i

to the k-dimensionallatent variable z

i

via W. (2) The second potential term

g�

(yi

, zi

) = exp[� ||�y

i

�z

i

||22�

2 ] corresponds to the transfor-mation of l-dimensional labels y

i

to the real-valued com-pressed space z

i

via a k ⇥ l, where k ⌧ l, random projec-tion matrix �. (3) Finally, the third potential h↵

i

(yi

) =Ql

j=1

N(yj

i

; 0, 1

↵

j

i

) enforces label sparsity by introducing a

zero mean-Gaussian prior on each of the labels with preci-sion ↵j

i

, which in turn are Gamma distributed. Formally,the joint probability distribution takes the following form:

p(Y,Z,W, [↵i

]Ni=1

|X,�)

=1Zp(W)

NY

i=1

fx

i

(w, zi

)g�

(yi

, zi

)h↵i

(yi

)p(↵i

). (1)

Here, Z is the partition function (normalization term), p(W) =QK

i=1

N(wi

, 0, I) is the spherical Gaussian prior on the lin-

ear regression functions, and p(↵i

) =Q

L

j=1

�(↵j

i

; a0

, b0

) isthe product of Gamma priors on each individual label. Intu-itively, the function f

x

i

aligns the latent variable zi

with theoutput of the linear regression functions, and the functiong�

favors compatibility with the compressed label vectorcorresponding to y

i

. Finally, h↵i

enforces sparsity over thelabels.

Using YL to denote labeled instances, inferring the exactposterior P (YU | YL,X,�) over labels YU for test pointsXU is prohibitive due to the product of Gaussian and non-Gaussian terms in the joint distribution. The approach in[17] resorts to variational inference by approximating theposterior over YU as a Gaussian distribution. While thisshowed good performance in terms of classification accuracy,there is a significant disadvantage of using the same inferenceprocedure for active learning tasks. The derived variationalinference equations lead to loss in information about thevariances between updates across the di↵erent layers of thegraphical model. Specifically, the procedure approximatesthe posterior over y

i

as a Gaussian, but the variance of therandom variable y

i

is completely independent of the varianceof the random variable z

i

. Similarly, the variance of zi

is inturn independent of the variance of the random variable W.The variance related to the Gaussian random variables iscentral to the use of mutual information in selective samplingtasks; hence, with the given inference procedure, mutualinformation cannot be optimally used as an active learningstrategy.

4. OUR APPROACHThere are two key ingredients to our framework: first is

an alternate approximate inference scheme that preservesvariances due to the regression part of the graphical model.Given this inference scheme, the second part focuses on de-riving an e�cient near-optimal selective sampling strategy.

4.1 An Alternate Inference ProcedureOur goal here is to derive an inference scheme that would

preserve the variances of the latent random variables andpropagate correct uncertainties to Y, thereby enabling ane↵ective active learning strategy. The key observation hereis the fact that such variances can in fact be preserved ifinstead of directly applying variational approximation, wefirst analytically integrate out the latent variables Z and Wfrom the joint distribution (Eq. 1). Such analytic integra-tion is feasible due to the form of potential functions f

x

i

, g�

and the Gaussian Process prior p(W). This results in a jointdistribution over Y and ↵ that is a product of a Gaussianand a Gamma distribution. Formally,

p(Y, [↵i

]Ni=1

|X,�) =

Z

Z,W

p(Y,Z,W, [↵i

]Ni=1

|X,�)

=1Ze

YT ⌃�1Y

Y

2

NY

i=1

h↵i

(yi

)p(↵i

).

Here the label vectorY is anNl-dimensional vector obtainedby appending all l-dimensional y0

i

s one after the other. Sim-ilarly, Z is an Nk-dimensional vector resulting from stackingall k-dimensional z0

i

s. Further, the term ⌃Y

takes the fol-

lowing form:

⌃Y

�1 = �T

(⌃Z

+ �2I)�1� (2)

where ⌃Z

=

0

BBB@

K11

Ik

K12

Ik

. K1N

Ik

K21

Ik

K22

Ik

. K2N

Ik

. . . .K

N1

Ik

KN2

Ik

. KNN

Ik

1

CCCA(3)

Here, the Nk⇥Nk-dimensional matrix ⌃Z

is a special blockmatrix, where K

ij

is the ith row and jth column entry ofK = (XTX+ �2I)1 and I

k

is the k ⇥ k identity matrix. Fi-nally, � is another Nk ⇥Nl block diagonal matrix with allthe N diagonal entries set to �. Further, we’d like to explic-itly point out that the matrix ⌃

Y

�1 is a precision matrixand is not full-rank. Note that such non-invertible precisionmatrices are in-line with use of improper un-normalizableGaussian distributions in Bayesian inference [9, 34] and donot pose any theoretical or practical problems.

The above mentioned marginalization preserves all the in-formation about uncertainty due to the Gaussian Process re-gression part of the graphical model and is succinctly repre-sented in the term ⌃

Y

. We propose to use variational infer-ence and obtain an approximate posterior distribution overthe label matrix Y that is a product of Gaussian terms andthe corresponding Gamma terms over ↵. If we denote theapproximation at the tth iteration as qt(Y) = N(µt

Y

,⌃t

Y

)and qt([↵]) =

Qi

�(at

i

, bti

), the resulting update rules are asfollowing:

Update for qt+1(Y): ⌃t+1

Y

= [diag(E(↵t)) +⌃�1

Y

]�1

Update for qt+1(↵i): at+1

i

= a0

i

+ 0.5,

bt+1

i

= b0i

+ 0.5[⌃t+1(i, i)]

Given observed labels and q(Y), the posterior distributionq(YU ) over unobserved labels is simply the conditional Gaus-sian distribution obtained by using standard Gaussian iden-tities. We’d like to point out that unlike the previous ap-proach [17], the cross co-variances across all the labels for allthe points are preserved in ⌃�1

Y

. Further note that this al-gorithm results in the final variance over the label vectors interms of the kernel function over the input features X. Thisis a direct consequence of the fact that uncertainty from theregression part of the network has already been propagatedto Y and sets a basis for an active learning scheme that ise↵ective.Selecting Hyperparameters a

0

and bo

: The choice ofthe hyperparameters a

0

and b0

is critical to induce sparsity,as arbitrary assignments can lead to non-sparse solutions.To see this, consider the joint distribution p(Y, [↵

i

]Ni=1

|X,�)and analytically marginalize over [↵

i

]Ni=1

. This results in adistribution of the following form:

Z

[↵i

]

N

i=1

p(Y, [↵i

]Ni=1

|X,�)

= e�YT ⌃�1

YY

2

NY

i=1

lY

j=1

(1 +y2

ij

2b0

)�a0 . (4)

The above marginalization leads to a very intuitive interpre-tation where the term⌃

Y

�1 arises due to the regression part

1Extension to non-linear kernel is straightforward by replac-ing XTX with an appropriate kernel function.

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

0.4

0.5

0.6

0.7

0.8

0.9

1

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20.992

0.993

0.994

0.995

0.996

0.997

0.998

0.999

1

1.001

𝑎 = 1𝑏 = 10

𝑎 = 1𝑏 = 1

𝑎 = 10𝑏 = 10

Convex Convex

−𝑎 = 10𝑏 = 10

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

𝑎 = 10𝑏 = 1

Figure 1: Plot of the sparsity inducing prior term

(1 + y

2

2b0)�a0 for di↵erent values of a

0

and b0

. Thedesirable shape of the prior term is shown in bottom-left corner (indicated by a green X) and occurs whena0

and b0

are set to 10�3 . The bottom-right figurecharacterizes the shape as a function of a

0

and b0

and highlights that the shape is undesirably concave

when �q

2b02a0+1

< y <q

2b02a0+1

.

of the framework and the terms (1+y

2ij

2b0)�a0 induce the spar-

sity in the solution. Note that in the sparsity term, highervalues of y

ij

lead to a lower value of the potential, therebypenalizing non-zero instantiations. However, the shape ofthe penalty term critically depends upon the values of a

0

and b0

. Figure 1 plots these penalty prior functions for dif-ferent values of a

0

and b0

. Figure 1 shows several shapescorresponding to di↵erent hyperparameter settings. Most ofthe shapes shown in the figure are undesirable for inducingsparsity due to their non-convexity localized around a largerange near zero (highlighted by a red ⇥). The bottom-rightpanel on the other hand shows what a desirable penaltyfunction should look like (a

0

= b0

= 10�3) (denoted by agreen X).

The bottom-right subpanel in figure 1 characterizes theshape of the function in terms of the hyperparameters. Inparticular, the function can decomposed into three regions:

(1) y < �q

2b02a0+1

, (2) y >q

2b02a0+1

, and (3) �q

2b02a0+1

<

y <q

2b02a0+1

. It is straightforward to show that the sec-

ond derivative of the function will always be greater thanzero for region (1) and (2), thereby implying a nice locallyconvex behavior leading to a desirable shape of the penaltyfunction. The region (3) on the other hand has the secondderivative less than zero, thus leading to a shape with a flat-ter penalty structure. Consequently, in order to induce anappropriate sparsity inducing prior it is desirable to mini-mize the range where region (3) occurs. This is achievedwhen either a

0

takes a very high value or when b0

tends tozero. However, setting a

0

to a high value will result in a verypeaky penalty function (see figure 1 bottom-middle panel).A peaky penalty function is undesirable as it will overridethe potential arising due to the regression term in the over-all mutilabel framework. Thus, a desirable prior can only beachieved when we select a very small value for both a

0

andb0

. It is no surprise that in prior research on Bayesian com-pressed sensing [6, 17, 25], a

0

and b0

were set to very smallvalues. Our analysis above validates such choices in order to

incorporate a reasonable sparsity inducing prior. This ob-servation about requiring a

0

and b0

to be close to zero willbe critical in the next section where we discuss non-myopicactive learning.

4.2 Non-Myopic Selective SamplingThe goal of active learning is to select a set of points A

to label from the available pool of unlabeled data U underbudget constraints, |A| = n, such that the resulting clas-sifier is most accurate. Intuitively, we want to sample then most informative points with respect to the classificationtask. Two potential criteria for the task are entropy and mu-tual information. Selecting points with maximum entropyboils down to choosing a set of points that jointly have themaximum uncertainty on their labels. The mutual infor-mation criterion [7], on the other hand, chooses A so thatthe uncertainty over the labels of remaining points is maxi-mally reduced after the labels of A are incorporated in thepredictive model.

The mutual information based criterion is the best of thetwo heuristics as the entropy criterion tends to select pointsthat are far apart from each other in the feature space andhence ends up selecting points on the boundary. Since apoint usually provides information about points in its nearbyregion, selecting points on the boundary wastes informationgathering e↵ort. In this paper, we present results in the con-text of mutual information as a criterion. Formally, if H(·)denotes the entropy, then we write the non-myopic subsetselection problem as:

A⇤ = argmaxA✓U,|A|=n

H(YU\A)�H(YU\A|YA). (5)

We call this problem non-myopic due to the fact that thegoal is to reason about the entire set A at once. This is dif-ferent from the other schemes for active classification wherea single point is chosen for querying a label, the model up-dated after observing the label and the process repeated fora total of n rounds. This non-myopic subset selection prob-lem is NP-complete as shown below:

Proposition 1. The subset selection problem defined in

equation 5 is NP-complete for the density p(Y, [↵i

]Ni=1

|X,�).

This proposition can easily be established by observing that,when the random variables ↵ are completely observed, thedistribution reduces to the form of a multivariate Gaussiandensity. The proof simply follows from the fact that thesubset selection problem for mutual information criterionhas been previously shown to be NP-complete [19].

Given the hardness of this problem, we resort to a greedyapproximation algorithm following recent ideas in submod-ular optimization. Specifically, if we can show that theobjective (eq. 5) being optimized is submodular and non-decreasing, then we can derive a greedy strategy that se-quentially selects data points based on marginal improve-ment of the objective. Note that the mutual informationcriterion is not submodular in general, but under some weakconditions, both submodularity and its non-decreasing prop-erty can be established. Formally, [18] have proved the fol-lowing proposition for random variables S and U in a graph-ical model:

Proposition 2. [18] Let S, U be disjoint subsets of

random variables such that the variables in S are indepen-

dent given U . Let information gain be F (A) = H(U) �

H(U\A|A), where A ✓ U . Then F is submodular and non-

decreasing on U , and F (;) = 0.

The following observation identifies the graphical model cor-responding to the joint distribution p(Y, [↵

i

]Ni=1

|X,�) as aspecial case of the above, thus establishing the submodular-ity and non-decreasing characteristic of our objective func-tion.

Corollary 1. Let S = [↵]ni=1

and U = YU , then due

to the form of h↵i

and p(↵i

), we have [↵i

]Ni=1

independent

given YU . Consequently, the objective in equation 5 is sub-

modular and non-decreasing.

Given this observation, we can now propose a greedy al-gorithm that repeatedly picks the points with the highestincrease in mutual information and adds them to the subsetA. The point x⇤ 2 U selected to be added to the subset Ais such that x⇤ = argmax

x

(MI(A [ x) �MI(A)). Here,MI(A) = H(YU\A)�H(YU\A|YA) and it has been provedearlier that this algorithm selects the subset A⇤ such thatthe value of the objective function will at least be (1 � 1

e

)times the optimal solution [19,22].

While the above mentioned corollary shows the existenceof a greedy algorithm that has a good approximation guaran-tee, it is still non-trivial to compute the mutual informationMI(A) for the multilabel classification model. Specifically,in order to compute mutual information, we need to solveexact inference, which itself is non-trivial in our model. Herealso, in order to derive an implementable solution, we needto establish a similar approximation MI to mutual informa-tion such that that the set of selected points A⇤ does notchange when the approximated mutual information is usedinstead of the exact one. Formally, we prove the followingtheorem which in turn will imply that such an approxima-tion is feasible:

Theorem 1. Let MI denote the mutual information of

any set A computed over the probability distribution, p(Y) /exp[YT⌃�1

Y

Y/2], where ⌃�1

Y

is as defined in eq. 3, then for

any x 2 U :lim

a0!0

b0!0

MI(A[ x)�MI(A) = MI(A[ x)� MI(A).

Proof Sketch: To prove this theorem, we first use thefact that MI(A [ x)�MI(A) can be written as H(x|A)�H(x|U\x) [19]. Next, we consider the joint distribution overY given in eq. (4) that arises after analytically marginalizingover [↵

i

]Ni=1

. It is easy to show that given A, the conditionalpredictive distribution of x takes a similar form:

p(x|A) / exp[� (x�mA)2

2�2

A](1 +

x2

2b0

)�a0

where: mA = ⌃x,A⌃�1

A,AYA

�2

A = ⌃x,x

�⌃x,A⌃�1

A,A⌃A,x

The above reduction follows from simple algebraic manipu-lation where the terms corresponding to the set A are col-lected by using matrix inversion lemma. A similar formalso can be derived for the conditional predictive distribu-tion of x given U\x. Let us denote the conditional en-tropy of x given A computed via the Gaussian distribu-tion as H(x|A). The proof then follows from the fact thatlim

a0!0

b0!0

H(x|A) = H(x|A) (similarly for H(x|U\x)). Proof

for the above statement is provided in appendix A.

The implication of the above theorem is that the greedyselection strategy described above will yield nearly the samesubset when MI is used instead of the true mutual infor-mation, as a

0

and b0

are set to very small values in ouralgorithm. This is due to the fact that the greedy proce-dure seeks to include x such that (MI(A [ x)�MI(A)) ismaximized at each step and the above theorem guaranteesthat the order of selecting the points will not change whenMI is replaced by MI. This result enables us to derive analgorithm based on the approximate inference procedure de-scribed earlier in the paper. Note that the MI is defined overa Gaussian Process; thus, prior work [19] on near-optimal se-lective sampling can be directly applied here. Also, note thatthe discussion above only focuses where the Gaussian termhas a zero mean for clarity purposes. It is fairly straightfor-ward to extend the analysis to non-zero mean cases.

One of the surprising by-products of our result is the factthat, similar to Gaussian Process regression models, the ob-served labels do not a↵ect the order in which the pointsshould be selected, which in turn allows us to do non-myopicselection of points. Intuitively this is feasible by realizingthat the mapping of the discrete label vectors to the con-tinuous space via the matrix � is almost 1-to-1 [15]. So,any subset selection via Gaussian Process regression on thecontinuous space, which is independent of observations, au-tomatically transfers to the discrete labeled space. We wishto point out explicitly that ⌃�1

Y

is not full-rank; however,the computation over Gaussian mutual information is stillfeasible in such cases [14].

The proposed compressed sensing framework compressesthe information about the labels, Y into the compressedspace, Z and then couples the Z in a single covariance ma-trix, hence allowing us to do mutual information based ac-tive learning in time that is independent of the number oflabels. The compressed sensing formulation also enables usto train lesser number of classifiers for the training processand reduce the training time significantly. Finally, note thatthe mutual information computation need not be done fromscratch after each iteration of the active learner. We uselazy evaluations proposed in [19] to drastically reduce thecomputational cost for selection of subsequent points to beannotated.

5. ACTIVE LEARNINGIt is interesting to note that selective sampling via MI can

accommodate many di↵erent scenarios of selective sampling.The most popular setting in literature considers revealing allthe labels for selected data points. Alternatively, in activediagnosis, di↵erent labels can be probed for one particulartest case only. Finally, in the most general case, individuallabels from di↵erent input points can be selected (general-ized active learning).

5.1 Traditional Active LearningIn traditional active learning, we select a subset of points

from the available data points for which all labels are re-vealed. Ideally, the goal is to choose a subset of size nthat leads to maximum decrease in entropy over the remain-ing unlabeled points as described in equation 5. However,proposition 2 and corollary 1 allow us to greedily selectpoints to be annotated so as to maximize the mutual in-formation at each step. Algorithm 1 shows the outline ofthe method that allows e�cient sampling of points based

Algorithm 1Mutual information based active learning overthe compressed space

Input: Input features X and budget nOutput: A: The set of points to be labeled

Compute C = ⌃Z

+ �2I using ⌃Z

defined in eq. 3A �for i 1 to n do

for x 2 X\A do

�x

|C(x,x)�C(x,A)C(A,A)

�1C(A,x)|

|C(x,x)�C(x,

¯A)C(

¯A,

¯A)

�1C(

¯A,x)|endx⇤ argmax

x

�x

A A [ x⇤

end

on mutual information. Note that, for this case, when allthe labels are revealed per data point, we can be far moree�cient in computation by directly using the Kernel matrixdefined over the points alone.

5.2 Active DiagnosisSuch an active learning scenario is more specific to multi-

label active learning, where obtaining each label for a pointhas significant cost associated with it. For example, in in-dustrial or medical settings, where each label may be ob-tained as a result of an expensive test, it is wiser to se-lect the labels which you want to be annotated for a se-lected point to decrease the overall cost, as opposed to an-notating all the labels. If we denote the label vector asy = {y

1

, y2

, y3

, ...yl

}, then the active diagnosis problem isto select a subset of labels, l0 ⇢ {1, 2, ...l}, which need tobe annotated. This scenario can be easily handled by ourmodel as we obtain the complete covariance matrix overthe labels and the mutual information criterion MI treatseach label independently. We can follow a greedy algorithmsimilar to the one followed in algorithm 1 to sample la-bels, wherein at each step, we select the label with index,i = argmax

i2U MI(yL[i

)� MI(yL) to be labeled, where Uand L denote the unlabeled and labeled indices respectively,as before. We present evaluation of our scheme for activediagnosis in the results section.

5.3 Generalized Active LearningA combination of the above active learning scenarios leads

to a general active learning problem, wherein the goal is toselect both the points and the labels to be annotated withina given active learning budget. Given the budget to obtainn annotations, we select n document-label pairs from the setof pairs {(1, 1), (1, 2), ....., (N, l)}, where N is the number ofdocuments with l labels each. To achieve this objective, weconsider the information gain computed using MI obtainedover the complete label matrix Y. Once again, we use thegreedy strategy to pick each pair. However, computing thismetric over the entire space of document-label pairs is ex-pensive and we resort to a two-way approach, wherein wefirst select the document to be annotated based on algorithm1 and then select, the label to be revealed similar to activediagnosis. Note that, both active diagnosis and generalizedactive learning are novel active learning scenarios which arespecific to the multilabel classification problem and can’t behandled by SVM based state-of-the-art methods.

6. RESULTSIn this section, we present empirical results to demon-

strate a) how the proposed approximate inference proce-dure compares to the prior work [17](denoted as BML-CS)in terms of accuracy as well as the capability to estimatevariances, b) the comparison of mutual information basedsampling with the state-of-the-art SVM based active learn-ing method proposed in [20], uncertainity sampling and ran-dom sampling baselines, and c) performance of the methodon novel active learning scenarios like active diagnosis andgeneralized active learning which are specific to multilabelclassification.

6.1 SetupDatasets: We used the datasets listed in Table 1 to eval-

Table 1: Datasets Used for ExperimentsDataset Type Instances Features Labels

Yeast Biology 2417 103 14

MSRC Image 591 1024 23

Medical Text 978 1449 45

Enron Text 1702 1001 53

Mediamill Video 43907 120 101

RCV1 Text 6000 47236 101

Bookmarks Text 87856 2150 208

Delicious Text 16105 500 983

uate our algorithm. As can be seen from the table, thedatasets exhibit a wide range in type, feature vector size andlabel vector size. For each of the datasets, 4000 randomlysampled points were used as the active learning pool fromwhich points had to be selected for training based on thedi↵erent active learning strategies compared. Another setof 2000 points was used as the test data. For datasets withfewer than 6000 points (enron, medical, MSRC and yeast),the entire set was selected as the active learning pool andtesting was done on the points not selected for training. Allthe results are reported after averaging over 5 such splits.For all the methods, an initial seed of 500 randomly sam-pled points was provided to the algorithms to start with (50points, in case of MSRC, and 200, in case of enron, medi-cal and yeast), as has been done in standard active learn-ing literature. For all our experiments, we used featuresdownloaded from the Mulan multilabel datasets [1]. ForMSRC, we used 1024 bit features generated using the Picodesscheme [4].

Baselines We compare the following methods for theevaluation of the proposed active learning strategy:

• MIML: Mutual Information for Multilabel classifica-tion as proposed in this paper

• Uncert: Uncertainty sampling on the model proposedin this paper. This method picks the point with thehighest entropy after each iteration.

• Rand: Random sampling baseline wherein each pointis randomly sampled.

• Li-Adaptive: SVM based adaptive active learningmethod proposed in [20], which is the state-of-the-artin multilabel active learning.

Parameters: For MIML and Uncert, the dimension ofthe compressed label space k was set to half of the number

−10 −5 0 5 10

−10

−8

−6

−4

−2

0

2

4

6

x

y

(a) Sample toy data, L = 5

5 10 15 20 25

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

# points selected

Prec

isio

n

MIMLUNCERTLi−AdaptiveRAND

(b) Precision at 1

5 10 15 20 250.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

# points selected

Prec

isio

n


(c) Precision at 2

5 10 15 20 25

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

# points selected

Prec

isio

n


(d) Precision at 3

Figure 2: Active Learning on Toy dataset:a)A sample toy dataset generated with 5 Gaussians. Points aregenerated from L = 5 Gaussians for this figure and are assigned labels based on the Euclidean distance fromthe Gaussian means. In our experiments, we use L = 35 Gaussians.b,c,d) The precision at 1,2 and 3 on the toydataset obtained after averaging over 50 di↵erent runs. This clearly shows how mutual information quicklypicks informative points to be annotated and hence, shows considerable increase in performance.

of labels for all datasets. The hyperparameters �2 and �2 inMIML and Uncert were selected via evidence maximizationand a

0

and b0

were set to 10�6, which lead to fairly un-informative priors. The Li-Adaptive mechanism proposedin [20] has a hyperparameter �, which was selected from aprefixed set of values 0.1, 0.2, ..., 0.9, 1 as suggested in [20].Cross validation was done for all other parameters.

Performance Metric: Typically, the goal in multilabelclassification is to predict the top k labels for each point.So, we use precision at 1, 2 and 3 to quantify the perfor-mance of di↵erent active learning schemes, which is in linewith previous work. Precision at k is defined as the frac-tion of true positives over the top k predicted labels foreach point. For novel active learning scenarios like gen-eralized active learning and active diagnosis, the ultimategoal is to recover the whole label set of each single testpoint. So, we quantify our results using the F-1 measurewhich depends on all labels, rather than on the few topones: F - measure = 2 precision ⇤ recall

precision+recall

.

6.2 Toy DatasetWe first consider a toy dataset before assessing the per-

formance of the di↵erent schemes on real-world datasets in6.4. The toy dataset is created as follows: We create L = 35two-dimensional Gaussians and assign a unique identifier toeach of them. Their mean and standard deviation is ran-domly selected from a fixed interval such that the contoursof the Gaussians, drawn for C standard deviation away fromthe mean, overlap with each other. Points are generatedfrom each of the L Gaussians. The feature vector is justthe 2-dimensional coordinate of the point and the labels arethe Gaussians whose mean is no more than C standard de-viations away from the point. For the experiments, C wasselected such that each point belongs to 7 classes on aver-age. To visualise the dataset, we plot the dataset generatedby setting L = 5, i.e. a dataset generated with 5 Gaussiansin Figure 2(a). For our experiments, we use L = 35 andstart with a randomly sampled set of 20 points. We, then,select 25 more points based on di↵erent schemes from a poolof 1000 data points. The performance is averaged over 50di↵erent runs and compared across all methods mentionedin section 6.1.

The precision at 1, precision at 2 and precision at 3 is

shown in figure 2. As can be seen in the figure, the mu-tual information criterion quickly fills the training set withpoints that are helpful to the model. Uncertainty samplingon our model outperforms Li-Adaptive as points are added.In fact, Li-adaptive baseline performs worse than the ran-dom sampling baseline in this case. Intuitively, MIML startsby picking points across di↵erent Gaussians in order to max-imize mutual information and hence leads to better predic-tion. On the other hand, the uncertainty sampling criterionpicks points closer to the boundaries, hence missing out oninformation.

6.3 Exploration of the Alternate Inference Pro-cedure:

In order to explore the properties of inference procedures(BML-CS [17] and proposed method(ML-OSS)), we com-pare the performance of both algorithms on the MSRC dataset,which has 23 labels. We used 1024-bit Picodes [4] image de-scriptors as features.

The results of the experiments are shown in Figure 4(a)and 4(b). Figure 4(a) plots the means of the labels inferredby both the methods for five randomly sampled test pointsand highlights that the means inferred by both the meth-ods are highly correlated indicating that they are equallycapable of modeling the means of the posterior distribution.However, similar correlation is not observed for variance es-timates. Figure 4(b) plots the variances for the first labelof all the test images inferred by the proposed method vsBML-CS on a negative log scale. Variances inferred by ourinference method have a much wider spread indicating thatBML-CS underestimates variances due to its limitation inpropagating variances across the graphical model.

6.4 Evaluation of Active Learning StrategyWe compared the performance of the proposed mutual

information (MIML) strategy with the strategy proposedin [20], uncertainty sampling (UNCERT) and random sam-pling (Rand). In these experiments, at every active learninground, all the labels corresponding to an input data pointwere revealed. Figure 3 shows the average of precision at1,2 and 3 achieved on di↵erent datasets. Note that this isa significant metric as all our datasets except delicious

have less than 3 positive labels per document on average.

0 50 100 150 200 2500.675

0.68

0.685

0.69

0.695

0.7

# points selected

Mea

n pr

ecis

ion

Mean precision for top 1,2,3 labels


(a) Yeast

0 50 100 150

0.32

0.33

0.34

0.35

0.36

0.37

0.38

0.39

0.4

0.41

0.42

# points selected

Mea

n pr

ecis

ion



(b) MSRC

0 50 100 150 2000.51

0.52

0.53

0.54

0.55

0.56

0.57

0.58

# points selected

Mea

n pr

ecis

ion



(c) Medical

0 50 100 150 200 2500.64

0.645

0.65

0.655

0.66

0.665

0.67

0.675

0.68

# points selected

Mea

n pr

ecis

ion



(d) Enron

0 50 100 150 200 250

0.731

0.732

0.733

0.734

0.735

0.736

0.737

0.738

0.739

# points selected

Mea

n pr

ecis

ion



(e) Mediamill

0 50 100 150 200

0.44

0.445

0.45

0.455

0.46

0.465

0.47

0.475

0.48

0.485

0.49

# points selected

Mea

n pr

ecis

ion



(f) RCV1

0 50 100 150 200 250

0.214

0.216

0.218

0.22

0.222

0.224

# points selectedM

ean

prei

cisi

on



(g) Bookmarks

0 50 100 150 200

0.522

0.524

0.526

0.528

0.53

0.532

0.534

0.536

# points selected

Mea

n pr

ecis

ion



(h) Delicious

Figure 3: Active Learning with Mutual Information (IG): The averages of precision at 1, 2 and 3 on di↵erentdatasets as points get added to the training set. Our method, MIML, comprehensively outperforms uncer-tainty sampling(UNCERT) and the random sampling baseline(Rand) as well as the state-of-the-art SVMbased active learning approach, Li-Adaptive [20]. None of the other methods shows such consistent perfor-mance across the diverse range of datasets. Every other method is outperformed by the random samplingbaseline in one or the other dataset.

Dataset Labels MIML Li-Adaptive

Yeast 14 3m 25s 1m 54sMediamill 101 41m 29s 54m 35sRCV1 101 30m 45s 37m 35s

Bookmarks 208 48m 58s 3h 57mDelicious 983 1h 11m 20h 15m

Table 2: Time complexity analysis: Time takento select 250 points from a pool of 4000 pointsfor di↵erent datasets for our method (MIML) andthe state-of-the-art SVM based approach [20]. ForYeast, the pool is 2000 points and other smallerdatasets have been excluded. MIML scales muchbetter than Li-Adaptive with the number of labelsand achieves nearly 20x gain for Delicious.

We observe that the MIML criterion proposed in this pa-per outperforms all the compared methods consistently, onall the datasets. No other method beats the other methodsacross all datasets. For instance, Li-adaptive performs worsethan random sampling on almost half of the datasets.

Evaluation of time complexity: Table 2 shows thetime taken by the proposed method (MIML) and Li-adaptive,to select 250 points to be annotated from a pool of 4000points. We do not include smaller datasets in this analy-sis. For Yeast, the pool is 2000 points as the dataset hasonly 2417 points. All these experiments were performed ona standard desktop PC with a 4 core, hyper-threaded IntelCore-i7 3.4GHz processor and 16GB RAM. As can be seen,

the SVM based approach scales badly (even though train-ing of binary classifiers per label was parallelized) with thenumber of labels and takes approximately 20 times the timetaken by MIML on delicious. The only overhead in MIMLwith increasing number of labels is because of the time takento learn the Gaussian process over the labels with an initialset of points. All other updates can be performed incremen-tally and hence, MIML is extremely time-e�cient.

Active Diagnosis and Generalized Active Learn-ing: Next, we demonstrate two active learning strategiesspecific to multilabel classification and show the benefitsof our mutual information based strategy in these scenar-ios. As described before, in active diagnosis, one label perround is revealed for the test set based on the selectioncriteria. This scenario cannot be straightforwardly tack-led by the method of [20], whereas it naturally fits in ourmodel, as the employed joint distribution correlates bothpoints and labels. Thus, we compare our results against theUNCERT and random sampling baselines. Since the ulti-mate goal is to recover the whole label set of each singletest point, we quantify our results using the F-1 measurewhich depends on all labels, rather than on the few topones: F - measure = 2 precision ⇤ recall

precision+recall

. We report the re-sults of this experiment on the RCV1 dataset; we randomlyselected n⇤ = 30 points collected in a test set Y⇤ for whichwe did active diagnosis by selectively sampling their labels.Each test point has an unknown label vector of size l = 101and the active diagnosis procedure was given a budget of

(a) Comparing means (b) Comparing Variances

0 5 10 15 20 25 30

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

# labels in active set

F−m

easu

re

MIMLUNCERTRAND

(c) Active Diagnosis

0 5 10 15 20 25 300.18

0.2

0.22

0.24

0.26

0.28

0.3

# points in active set

F−m

easu

re

MIMLUNCERTRAND

(d) Generalized AL

Figure 4: (a, b) Comparison of the inference procedure with BML-CS: Figure (a) is a plot of means propagatedby the proposed method (ML-OSS) vs means propagated by BML-CS for 5 randomly sampled test points.Mean values propagated by both the methods follow similar trends. Figure (b) plots variances inferred overthe test points by our method vs variances propagated by BML-CS. Both the axis have been convertedto negative log scale. Variances propagated by our approach have a wider spread and are higher. (c)Active Diagnosis: Plot of mean F-score against the number of active diagnosis rounds. Mutual information(MIML) outperforms uncertainity sampling (UNCERT) and random sampling (Rand). (d) GeneralizedActive Learning: Plot of mean F-score against the number of revealed points. Mutual information provesadvantageous over the other methods.

m = 30 labels to select per point. An initial fully observedtraining set of 100 points kick started the experiment.

Fig. 4(c) summarizes the results in a plot averaged over 35runs. We observe that the proposed criterion selects muchmore useful labels to reveal right from the start. The di↵er-ence in the performance compared to the baselines is evenlarger than for the traditional active learning task. This isdue to the fact that output labels within a data point are farmore correlated than across data instances. For example, atest data corresponding to an article tagged as “investment”is very likely to also fall into the category “banking”.

Generalized Active Learning is the scenario when both theinput point and the label to be queried are selected basedon the selection criteria. Since the goal here is to learn asmuch information as possible for the whole test set, we usethe following criterion to evaluate the performance: after se-lecting m = 30 labels for each of the chosen data points, wecompute the F-measure of the whole test set given the par-tially revealed label vectors of the current and all other datapoints already in the active set. We ran this experiment 35times and present the averaged results for F-measure withnumber of points in the active set in figure 4(d). Note that,both active diagnosis and generalized active learning are en-abled by the fact that our multilabel classification model cantrain on partially labelled datasets.

7. CONCLUSIONWe presented a novel mutual information based active

learning framework for multilabel classification, that enablesa theoretically principled and non-myopic approach. We ex-tensively evaluated our algorithm across various datasets intraditional active learning settings as well as active diagnosisand generalized active learning (which are specific to multi-label classification) and showed that it consistently outper-forms the state-of-the-art, both in terms of time-e�ciencyand precision. Possible future work includes integrationof the framework with other multilabel classification tech-niques.

8. REFERENCES[1] Mulan Multilabel Datasets.

http://mulan.sourceforge.net/datasets.html.[2] R. Agrawal, A. Gupta, Y. Prabhu, and M. Varma.

Multi-label learning with millions of labels:Recommending advertiser bid phrases for web pages.In WWW, 2013.

[3] K. Balasubramanian and G. Lebanon. The LandmarkSelection Method for Multiple Output Prediction. InICML, 2012.

[4] A. Bergamo, L. Torresani, and A. W. Fitzgibbon.PiCoDes: Learning a Compact Code forNovel-Category Recognition. In NIPS, 2011.

[5] W. Bi and J. T.-Y. Kwok. E�cient Multi-labelClassification with Many Labels. In ICML, pages405–413, 2013.

[6] C. M. Bishop and M. E. Tipping. VariationalRelevance Vector Machines. In UAI, 2000.

[7] W. Caselton and J. Zidek. Optimal monitoringnetwork designs . Statistics and Probability Letters,1984.

[8] Y.-N. Chen and H.-T. Lin. Feature-aware Label SpaceDimension Reduction for Multi-label Classification. InNIPS, pages 1538–1546, 2012.

[9] W. Chu, V. Sindhwani, Z. Ghahramani, andS. Keerthi. Relational Learning with GaussianProcesses. In NIPS, 2006.

[10] M. Cisse, N. Usunier, T. Artieres, and P. Gallinari.Robust Bloom Filters for Large MultiLabelClassification Tasks. In NIPS, pages 1851–1859, 2013.

[11] A. Esuli and F. Sebastiani. Active Learning Strategiesfor Multi-Label Text Classification. In ECIR, 2009.

[12] C.-S. Feng and H.-T. Lin. Multi-label Classificationwith Error-Correcting Codes. JMLR, pages 289–295,2011.

[13] A. Goldberg, X. Zhu, A. Furger, and J. Xu. OASIS:Online Active Semi-Supervised Learning. In AAAI,2011.

[14] A. Gretton, R. Herbrich, and A. Hyvarinen. Kernelmethods for measuring independence. JMLR, 2005.

[15] D. Hsu, S. Kakade, J. Langford, and T. Zhang.Multi-Label Prediction via Compressed Sensing. InNIPS, 2009.

[16] S. Ji, L. Tang, S. Yu, and J. Ye. Extracting SharedSubspace for Multi-label Classification. In KDD, pages381–389, 2008.

[17] A. Kapoor, R. Viswanathan, and P. Jain. MultilabelClassification using Bayesian Compressed Sensing. InNIPS, 2012.

[18] A. Krause and C. Guestrin. Near-optimal NonmyopicValue of Information in Graphical Models. In UAI,2005.

[19] A. Krause, A. Singh, and C. Guestrin. Near-OptimalSensor Placements in Gaussian Processes: Theory,E�cient Algorithms and Empirical Studies. JMLR,2008.

[20] X. Li and Y. Guo. Active Learning with Multi-labelSVM Classification. In IJCAI, 2013.

[21] X. Li, L. Wang, and E. Sung. Multi-label SVM ActiveLearning for Image Classification. In ICIP, 2004.

[22] G. Nemhauser, L. Wolsey, and M. Fisher. An analysisof approximations for maximizing submodular setfunctions. Mathematical Programming, 1978.

[23] J. C. Platt. Probabilistic Outputs for Support VectorMachines and Comparisons to Regularized LikelihoodMethods. In Advances In Large Margin Classifiers.MIT Press, 1999.

[24] B. Settles. Active learning literature survey. Technicalreport, 2010.

[25] Shihao, Y. Xue, and L. Carin. Bayesian CompressiveSensing , 2007.

[26] A. Singh, A. Krause, C. Guestrin, and W. J. Kaiser.E�cient Informative Sensing using Multiple Robots.

[27] M. Singh, E. Curran, and P. Cunningham. Activelearning for multi-label image annotation. Technicalreport, University College Dublin, 2009.

[28] F. Tai and H.-T. Lin. Multi-label Classification withPrincipal Label Space Transformation. In Workshop

proceedings of learning from multi-label data, 2010.[29] J. Weston, S. Bengio, and N. Usunier. Wsabie: Scaling

Up To Large Vocabulary Image Annotation. In IJCAI,2011.

[30] J. Weston, A. Makadia, and H. Yee. LabelPartitioning for Sublinear Ranking. In ICML, 2013.

[31] B. Yang, J. Sun, T. Wang, and Z. Chen. E↵ectivemulti-label active learning for text classification. InKDD, 2009.

[32] H.-F. Yu, P. Jain, and I. S. Dhillon. Large-scaleMulti-label Learning with Missing Labels. ICML,2014.

[33] Y. Zhang and J. G. Schneider. Multi-Label OutputCodes using Canonical Correlation Analysis. InAISTATS, pages 873–882, 2011.

[34] X. Zhu, J. La↵erty, and Z. Ghahramani.Semi-Supervised Learning: From Gaussian Fields toGaussian Processes. Technical report, School of CS,CMU, 2003.

APPENDIXAppendix A: Proof lim

a0!0

b0!0

H(x|A) = H(x|A)

By definition H(x|A) = EYA [H(x|YA)] (also for H). Here

EYA [·] is the expectation over the labels YA at the active

sites A under the distribution in eq. 4. Now,

ˆH(x|YA)�H(x|YA) =

1

ˆZ

Z

x

e

�(x�mA)2

2�2A

(x�mA)

2

2�2

A

�1

Z

Z

x

e

�(x�mA)2

2�2A

[1 +

x

2

2b0]

a0[

(x�mA)

2

2�2

A+ a

0

log(1 +

x2

2b0

)] + log

ˆZ

Z

Here, Z and Z are the corresponding normalizing constants.

Now as a0

! 0, the term a0

log(1 + x

2

2b0) vanishes. Further,

using the binomial expansion [1 + x

2

2b0]�a0 = 1 � a

0

· t +a0(a0�1)

2!

· t2 + .., where t = x

2

2b0+x

2 , it is straightforward toshow:

lim

a0!0

b0!0

ˆH(x|YA)�H(x|YA) =

ˆH(x|YA)[1�ˆZ

Z] + log

ˆZ

Z

The required proof follows directly by using the binomial

expansion and seeing that as a0

! 0, the expressionˆ

Z

Z

! 1and the above quantity evaluates to zero.

Active Learning for Sparse Bayesian Multilabel Classiﬁcation€¦ ·...

Documents

Transcript of Active Learning for Sparse Bayesian Multilabel Classiﬁcation€¦ ·...