Label Propagation in Video Sequences

Vijay Badrinarayanan†, Fabio Galasso† and Roberto CipollaUniversity of Cambridge, UK

vb292,fg257,cipolla@cam.ac.uk

Abstract

This paper proposes a probabilistic graphical model forthe problem of propagating labels in video sequences, alsotermed the label propagation problem. Given a limitedamount of hand labelled pixels, typically the start and endframes of a chunk of video, an EM based algorithm prop-agates labels through the rest of the frames of the videosequence. As a result, the user obtains pixelwise labelledvideo sequences along with the class probabilities at eachpixel. Our novel algorithm provides an essential tool to re-duce tedious hand labelling of video sequences, thus pro-ducing copious amounts of useable ground truth data. Anovel application of this algorithm is in semi-supervisedlearning of discriminative classifiers for video segmentationand scene parsing.

The label propagation scheme can be based on pixel-wise correspondences obtained from motion estimation, im-age patch based similarities as seen in epitomic models oreven the more recent, semantically consistent hierarchicalregions. We compare the abilities of each of these vari-ants, both via quantitative and qualitative studies againstground truth data. We then report studies on a state ofthe art Random forest classifier based video segmentationscheme, trained using fully ground truth data and with dataobtained from label propagation. The results of this studystrongly support and encourage the use of the proposed la-bel propagation algorithm.

1. Introduction

The problem of label propagation has received some at-tention from the machine learning community for the taskof semi-supervised learning using both labelled and unla-belled data points [14]. On the other hand researchers incomputer vision have paid only marginal attention to thisimportant problem, addressing it as “label transfer” acrosssimilar images in a database [9] or recognising objects (bylabelling corresponding pixels) using a trained image gen-

† indicates equal contribution

Figure 1. An illustration of label propagation in video sequences

erative model [7], both requiring large quantities of labelledtraining images. In any case to the best of our knowledgethis problem has not been considered for video sequencedata, possibly because it appears as a (deceptively) simpletask given the strong correlation between successive frames.In response, our contributions are two fold:1. We propose a probabilistic framework and algorithm forlabel propagation in video sequences;2. We demonstrate effective semi-supervised learning withlabel propagation for video segmentation and recognitionon new challenging sequences [11, 4].Our primary interest in this work is to transfer labels (road,pedestrians, cars and the like) from the two labelled ends ofa video sequence to the remaining unlabelled frames (SeeFig. 1 for an illustration). Additionally, we are also in-terested in obtaining a measure of how confident a labelassigned to a pixel is, or in familiar terms, the class dis-tribution at the pixel labels. In Section 3 we develop a prob-abilistic model to incorporate these requirements and per-form maximum likelihood based inference.The probabilistic model proposed in this paper, under differ-ent settings, allows one to compare naive methods for labelpropagation using optic flow estimates, for instance, along-side more sophisticated approaches based on image patches

1

[5] or extraction of semantically consistent regions [1]. Thefirst set of experiments in Section 4 perform a compara-tive study of these methods and demonstrates both quan-titatively and qualitatively that the proposed label propaga-tion methods are superior to naive solutions under variousexperimental settings and ground truth. The second set ofexperiments in Section 4 aims to convince the reader thatthe propagated labels from the proposed algorithms can in-deed be used to train state of art discriminative classifierslike Random Forests [11] for video sequence classificationwith minimal loss in accuracy of classification. We con-clude and discuss future prospects in Section 5.The following section presents a literature review.

2. Related WorkA well known method for label propagation in machine

learning literature is that of Zhu et al. [14]. They formulatetheir label propagation problem as a problem of assigningsoft labels to nodes of a fully connected graph with fewlabelled nodes. Weights are assigned to the links accordingto the proximity of the corresponding nodes in the graphand an iterative update is proposed for propagating labelsto the unlabelled nodes. This method is shown to performwell in assigning labels to a hand written digits database.Other such methods, based on exploiting the ”geometryof the data”, are also available in the machine learningliterature (see [3] for a detailed survey).In contrast, sequential data and their labelling are morenaturally modelled using directed graphs. Particularly, wedraw inspiration from the “static” epitome model for videosin [5]. We extend this to a “coupled” Hidden MarkovModel (HMM) which can employ pixels, image patchesor semantic regions. There is an implicit ”inpainting” ideabehind these models which is particularly useful for videosequence labelling where occlusions (disocclusions) arefrequent.Label propagation has also been proposed in the contextof multi-label learning [6], where data points are assignedtheir class label under the learnt influence of the correlationbetween the class labels. Although meaningful, thesemethods need copious amounts of labelled training data,which is difficult to gather in the first place. Instead, weavoid any sort of training for label propagation and rely ona generative model to perform accurate labelling.Only recently in the computer vision community someresearch has been devoted to label transfer from trainingimages in a set of similar (and labelled) images in adatabase [9] to a given test image. In particular, theauthors formulate their ”image matching” method in theSIFT-descriptor space using the traditional optic flow opti-mization setup. Hence, they argue that their method is theequivalent of optic flow correspondence for non-sequentialdata. However, some surprising and flawed matches are

obtained, inconsistent with the quantitative energy termthey evaluate. In this work, we demonstrate by comparisonsthat optic flow based labelling is less efficient than theproposed methods for label propagation.Elsewhere, [7] demonstrate the ability of their trained gen-erative Jigsaw model for images to transfer labels from alearnt jigsaw (source) to an image in the training (labelled)dataset. Similarly discriminative models like CRFs [8]also strive to perform joint segmentation and (pixelwise)recognition, but require a large dataset of carefully labelled(similar) images. At this juncture, straightforward exten-sions of such models for video sequences are not availableand for that purpose, one can definitely anticipate the needfor large amounts of labelled (video) data. The proposedalgorithms of this paper are exactly suited to this purpose.As a video labelling tool, the proposed algorithms can becontrasted with currently operative annotation tools. Arepresentative sample of these is LabelMe Video [12]. Herethe user draws a (rough) polygon around an object at thestart, some key frames and the end frame, and defines a richset of annotations (category, static or moving, occluded).Labelling is achieved via interpolation with a 3D motionmodel, while tags are used to learn the category priors andestimate the 3D structure of the scene. We base our “tool”on a generative model which jointly explains the observedframes and their annotations, with the resulting capabilityto obtain both pixelwise labels and their probabilities. Weemploy these results to train discriminative classifiers forpixelwise recognition.

3. Label Propagation

The proposed graphical model (see Fig. 2) is a cou-pled HMM for the joint generative modelling of image se-quences (continuous variables) and their annotation (dis-crete variables). While [5] learn a ”static” epitome modelwith space-time patches for video modelling (inpainting &video interpolation are their target applications), we employa time-series model for video annotation and avoid learn-ing a video epitome. However, in our model we incorpo-rate their key idea of inducing correlations between imagepatches in the inference stage. The elements of our modelare described below.

IMAGE MODELLING Shaded node variables I0:n repre-sent the observed sequence of images. Hidden variable Zkcan represent a set of mutually independent colour imagepixels, rectangular image patches or even semantic regions[1] (each colour channel is treated independently). For clar-ity’s sake we only describe the case of rectangular imagepatches here (See [2] for details for the other two cases).Conceptually, Ik−1 predicts the set of latent patches Zkwhich in turn are used to explain (generate) observation Ik

and so on along the Markov chain.Ik−1 → Zk : following [5] this link is defined as follows.

p(Zk|Ik−1, Tk) =

Ωk∏j=1

∏i∈jN (zk,j,i; Ik−1,Tk,j(i),Φk−1,Tk,j(i))

(1)where, index j runs over all the (overlapping) latent patchesZk = Zk,jΩk

j=1. zk,j,i is pixel i inside patch j at time k.Each patch j has an associated variable Tk,j which indexes(overlapping) patches 1, . . .Ωk in Ik−1. Tk = Tk,jΩk

j=1is the collection of patches. Note that the number, size andordering of patches are the same in Zk and Ik−1. Followingthis, Tk,j(i) indexes the corresponding pixel Ik−1,Tk,j(i) inpatch Ik−1,Tk,j

.N (zk,j,i; Ik−1,Tk,j(i),Φk−1,Tk,j(i)) is a normalized

Gaussian distribution over zk,j,i, with mean Ik−1,Tk,j(i)

and variance Φk−1,Tk,j(i). This distribution quantifies theability of image patches in Ik to predict latent patches inZk.Zk → Ik : this link is defined as follows.

p(Ik|Zk) =∏v∈VN (Ik,v;

1

Nv

Ωk∑j=1

s.t.j⊃v

zk,j,v,Ψk,v), (2)

where Ik,v denotes the intensity of pixel v in the pixel gridV . j indexes patches in Zk which overlap pixel v. Ψk,v isthe variance of the normalized Gaussian distribution.ANNOTATION MODELLING Let l = 1 . . . L index thedifferent object classes, where l = 1 corresponds to an un-labelled (void) class (See [2] for more details on void class).Hidden variableAk is an image sized grid. ak,v,lLl=1 are aset of positive real valued parameters at pixel v of this grid,which obey

∑Ll=1 ak,v,l = 1. Thus, ak,v,lLl=1 represent

the class distribution for the corresponding pixel in Ik. Theend variables of the chain, A0, An (shaded) are initializedusing Eqn. 12.

Hidden variable Zak can represent a set of mutuallyindependent “annotated” image pixels, rectangular imagepatches or semantic regions. Adjacent to each Zak on thechain are variables Ak−1 and Ak. As in the image model,Ak−1 predicts the set of annotated patches Zak , which inturn is used to predict Ak and so on along the bottomMarkov chain.Ak−1 → Za

k : this link is defined as follows.

p(Zak |Ak−1, Tk) =

Ωk∏j=1

∏i∈j

L∏l=1

azak,j,i,l

k−1,Tk,j(i),l, (3)

where the indices on the first two products are the same asin Eqn.1. The last term is the discrete class probability dis-tribution of the pixel zak,j,i in patch Zak,j .

I0

Ƭ1

A0 Za1

Imagelayer

Annotationlayer

I1

Za2 Za

nA1 An

Ƭ2 Ƭn

Z1 Z2 Zn In

Figure 2. Proposed graphical model for label propagation.

Zak → Ak : this link is defined as follows.

p(Ak|Zk) =∏v∈V

Γ(αv,0)

Γ(αv,1) · · · Γ(αv,L)

L∏l=1

aαv,l−1k,v,l , (4)

which is a Dirichlet prior on the (independent) parame-ters ak,vv∈V . Γ denotes the gamma function with pa-rameters αv,l = 1

Nv

∑Ωk

j=1s.t.j⊃v

zak,j,v,l for l = 1 . . . L and

αv,0 =∑Ll=1 αv,l. Note that j indexes patches in Zak which

overlap pixel index v in the pixel grid V .

3.1. Inference

Given I0:n, A0, An, we estimate the latent variablesΘ = Z1:n, Z

a1:n, A1:n−1, T1:n using the variational EM

algorithm. The log of the data likelihood is lower boundedas shown below.

log p(I0:n, A0, An) ≥∫

Θ

q(Z1:n, Za1:n, A0:n, T1:n)×

logp(Z1:n, Z

a1:n, A0:n, T1:n, I0:n)

q(Z1:n, Za1:n, A0:n, T1:n). (5)

For tractability, we assume the following form for the aux-iliary distribution (See [5]).

q(Z1:n, Za1:n, A0:n, T1:n) =

n∏k=1

q(Tk)×

δ(Zk − Z∗k)δ(Zak − Za∗k )δ(Ak −A∗k). (6)

The delta terms above imply that we infer the most probablehidden states.EXPECTATION STEP Fixing the latent variables toA∗0:n, Z

∗1:n, Z

a∗1:n (note A∗0 = A0, A

∗n = An) , the E-step

computes the posterior over the mapping variables:

q(Tk,j) = p(Tk,j |I0:n, A∗0:n, Z

∗1:n, Z

a∗1:n)

∝∏i∈jN (z∗k,j,i; Ik−1,Tk,j(i),Φk−1,Tk,j(i))

L∏l=1

aza∗k,j,i,l

k−1,Tk,j(i),l×

p(Tk,j). (7)

(a). Frame 1 (b). Frame 17 (c). Frame 35 (d). Frame 52 (e). Frame 70

Optic flow (OF) basedlabel propagation inthe proposed model -baseline method.

Notice the overall label”drag” effect causeddue to accumulationof flow error, clearlyvisible on the wheels ofthe car.

Proposed image patches(IP) based label propa-gation.

The overall labellingis ”cleaner” due to the”inpainting” abilitygained using both E andM steps.

Proposed semanticregions (SR) basedlabel propagation.

The ”drag” effect arisesas the region bound-aries do not exactlycorrespond to objectboundary.

Figure 3. Seq 1 - Frames 1 & 70 on the top row are user provided labels. Ground truth for frames 17, 35 & 52 are provided for comparisonon the top row. The proposed IP based method delivers the best labelling

Note that in this step the posterior of the mapping variablesis dependent on both the observed image data and thecurrent estimate of the labels and their probabilities.MAXIMIZATION STEP The lower bound under the E-stepdistributions is to be maximized wrt A∗1:n−1, Z

∗1:n, Z

a∗1:n.

Conditioned on the mapping variables, Tknk=1, theestimation of Z∗1:n and A∗1:n−1, Z

a∗1:n can be separated. This

leads to the following.

z∗k,v =

Ik,v

Ψ2k,v

+∑Ωk

j=1

∑Tk,j

q(Tk,j)Ik−1,Tk,j(v)

Φ2k−1,Tk,j(v)

1Ψ2

k,v+∑Ωk

j=1

∑Tk,j

q(Tk,j)

Φ2k−1,Tk,j(v)

,∀v ∈ V.

(8)For the remaining parameters we follow an alternation strat-egy to obtain their estimates. We fix A∗1:n and optimise thelower bound to get,

za∗k,v,l =

1 if ∇za∗k,v,l > ∇za∗k,v,l′ , l′ = 1, ..., L, l′ 6= l,

0 otherwise.(9)

Using the above estimate,

a∗k,v,l ∝ (10)

(za∗k,v,l − 1

)+

Ωk+1∑j=1

∑Tk+1,j

∑i

s.t.Tk+1,j(i)=v

q(Tk+1,j)za∗k+1,j,i,l

︸ ︷︷ ︸bk,v,l

,

which upon normalization delivers,

a∗k,v,l =bk,v,l∑Ll=1 bk,v,l

, l ∈ 1 : L. (11)

Eqns. 9, 10, 11 are alternated, in that order, to obtain theestimates of the hidden variables at convergence.INITIALIZATION The variables Za2:n−1 are all initializedto zero without affecting the iterations. Z1, Zn are clampedto the user provided labels and are not updated throughout.The parameters A0, An are initialized as follows.

ak,v,l =

1 if user provided class label is l,0 otherwise.

(12)

The variances of the normalized Gaussians in Eqns. 1, 2 arefixed to 5.0.

(a). Frame 1 (b). Frame 12 (c). Frame 25 (d). Frame 37 (e). Frame 50

Optic flow (OF) basedlabel propagation inthe proposed model -baseline method.

Aperture problemsand flow round offerror accumulationmanifest as ”frayed”and ”smudged” la-belling, noticeable onthe pedestrian.

Proposed image patches(IP) based label propa-gation.

The combination of Eand M steps with imagepatches performs stablelabelling. Border ef-fects cause sky misla-belling.

Proposed semanticregions (SR) basedlabel propagation.

A ”watercolor” effectis pronounced, as theregion boundaries donot exactly correspondto object boundaries.

Figure 4. Seq 2 - Frames 1 & 50 on the top row are user provided labels for label propagation. Ground truth for frames 12, 25 & 37 areprovided for comparison on the top row. The proposed IP method performs best

EVIDENCE PROPAGATION In this work we propagate thelabels in two full passes over all the hidden variables. Fur-ther details on the relation between the variational approxi-mation in Eqn 6 and its effects on label propagation can befound in [2].

4. Experiments

The first experiment compares the accuracy of propa-gated labels against known ground truth, using: pixel wisecorrespondences as delivered by an optic flow (OF) algo-rithm [13], rectangular image patch (IP) matches [5], andpre-optimised mappings of semantically consistent regions(SR) [1] (see Section 3). The results are relevant to ob-ject cutout in videos, extracting masks for alpha matting forcinema post-production and other interactive applications.Next, we study the test effects of training a state of theart Random Forest classifier [11] using the results of labelpropagation under different settings. However, due to spaceconstraints we only report the results of training the classi-fier using the IP mapping strategy.

DATASET DESCRIPTION We use a new and challengingpixelwise ground truthed video dataset. This dataset con-

sists of three outdoor driving video sequences (VGA reso-lution) captured using a camera placed on a car. The groundtruths are available for 70 frames for Seq 3 & Seq1, and98 frames of Seq 2. 14 different classes are labelled (sky,building, road, pavement, pedestrian, car and the like). Thisground truth is obtained via tedious hand labelling, costinga minimum of about 45-60 minutes per frame.

TESTING THE ACCURACY OF LABEL PROPAGATIONUsing only 1 iteration with 6 × 6 sized image patches and75% overlap between patches, the E-step (Eqn. 7) computesthe posterior probability of the mapping variables. A “flat”prior over a 30 × 40 search area surrounding the center ofthe patch is used. In case of OF and SR based mappings, theposterior over mappings is replaced with deterministicallyavailable approximate mappings, pre-computed optic flowand pre-computed tracks of semantically consistent regions([1]) obtained via dynamic programming (See Appendix fordetails) respectively. In particular, region track optimisa-tion can be bracketed in spirit to semi-dense particle flowcomputations [10]. Both use a forward backward strategyto deliver globally ”optimal trajectories” to handle occlu-sions (disocclusions) but region tracks also provide densematches. Probabilistic (IP) and deterministic (OF,SR) map-

(a). Frame 1 (b). Frame 6 (c). Frame 12 (d). Frame 18 (e). Frame 26

Optic flow (IP) basedlabel propagation inthe proposed model -baseline method.

Notice as the scenechanges several pointsare not labelled (inblack) due to lack ofcorrespondences.

Proposed image patches(IP) based label propa-gation.

The labelling is stableoverall. Pedestrians arelabelled well. Howeversmall structures areeroded.

Proposed semanticregions (SR) basedlabel propagation.

Optimised regiontrajectories producereasonable labelling.Small structures arelabelled well.

Figure 5. Seq 3 - Frames 1 & 25 on the top row are user provided labels. Ground truth for frames 6, 12 & 18 are provided for comparisonon the top row. The proposed methods propagate labels fairly accurately under a panning motion here

pings are evaluated over increasing width between the twoends of the video sequences, 25, 50 & 70 frames.

TRAINING RANDOM FOREST CLASSIFIERS This exper-iment is a study of semi-supervised learning using labelpropagation. We choose the image patch based mappingstrategy as the test label propagation method of choice (seeTable 1) and a Random forest [11] (with 15 trees and a max-imum depth of 10) as the classifier of choice. 98 frames ofSeq 1 is chosen as the test sequence, as maximum groundtruth is available for this sequence. Seq 1 and Seq 3 arechosen as training sequences. The classifier is trained un-der three different settings; under fully ground truthed Seq1 and Seq 3, using ground truth for Seq 1 and label propaga-tion for Seq 3, and finally, using label propagation for bothSeq 1 and Seq 3. These settings are also evaluated overvarying lengths of propagated labels (25, 50 and 70 frames)over the training sequences.

RESULTS AND DISCUSSIONS

Accuracy test Fig. 6 reproduces the quantitative results ofthe tests on Seq 1, 2 & 3. In Seq 1 (Fig. 3), image patch (IP)based mapping outperforms the other methods under occlu-sions (disocclusions) due to probabilistic mapping and its

inpainting ability. The qualitative result on Fig. 3 vindicatethese numerical results. In Seq2 (Fig. 4), the class averageaccuracies are highest for the OF mapping, this is primarilydue to the mislabelling of the ”sky” class in IP based map-ping (attributable to border effects). In contrast, the globalaccuracy and the qualitative result in Fig .4 clearly indicatethe superiority of the IP based mapping over the other two.It is interesting to note the slower degradation rate of bothglobal and class average accuracy for IP based mapping asthe width increases as compared to the other two. Of in-terest is also the fact that even with a slight lower accuracyon pedestrian and car class the qualitative effect is better forIP based mapping for Seq 2. Categories such as pavementsand road markings, which are very useful for driving ap-plications, are labelled better by IP based mapping. In Seq3 (Fig. 5), the scene changes quickly and new objects ap-pear (disappear). Here the SR based mapping which usessequence optimized trajectories demonstrates highest classaverage accuracy, closely followed by IP based mapping(for 25 and 51 frames). The IP based mapping degradesover the 70 frame test due to its inability to correctly ex-plain unseen (provided in ground truth) objects. The naiveOF based mapping leaves large amounts of data points un-labelled (this is not counted into the void class which is re-

0

20

40

60

80

100

Seq 1 − 25 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nBi

cycli

st Car

Road

mar

kRo

adBo

llard

Pave

men

tTr

eeOth

er

OF: G = 92.31, C = 70.35IP: G = 95.11, C = 73.13SR: G = 92.73, C = 70.44

0

20

40

60

80

100

Seq 1 − 50 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nBi

cycli

st Car

Road

mar

kRo

adBo

llard

Pave

men

tTr

eeOth

er

OF: G = 89.0, C = 61.56IP: G = 92.72, C = 64.45SR: G = 89.68, C = 62.20

0

20

40

60

80

100

Seq 1 − 70 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nBi

cycli

st Car

Road

mar

kRo

adBo

llard

Pave

men

tTr

eeOth

er

OF: G = 87.05, C = 57.35IP: G = 91.64, C = 63.09SR: G = 88.0, C = 57.99

0

20

40

60

80

100

Seq 2 − 25 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nCa

rRo

ad m

ark

Road

Bolla

rdPa

vem

ent

Tree

Other

OF: G = 93.51, C = 68.25IP: G = 94.29, C = 61.59SR: G = 93.41, C = 58.05

0

20

40

60

80

100

Seq 2 − 50 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nCa

rRo

ad m

ark

Road

Bolla

rdPa

vem

ent

Tree

Other

OF: G = 90.70, C = 67.18IP: G = 93.21, C = 59.72SR: G = 90.38,C = 53.70

0

20

40

60

80

100

Seq 2 − 69 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nCa

rRo

ad m

ark

Road

Bolla

rdPa

vem

ent

Tree

Other

OF: G = 87.97, C = 54.77IP: G = 91.51, C = 55.96SR: G = 87.89, C = 50.35

0

20

40

60

80

100

Seq 3 − 25 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nCa

rRo

ad m

ark

Road

Bolla

rdPa

vem

ent

Tree

Other

OF: G = 77.4, C = 62.42IP: G = 90.54, C = 67.31SR: G = 87.74, C = 70.06

0

20

40

60

80

100

Seq 3 − 51 frames

Void

Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nCa

rRo

ad m

ark

Road

Bolla

rdPa

vem

ent

Tree

Other

OF: G = 48.68, C = 46.51IP: G = 89.27, C= 60.59SR: G = 75.52, C = 59.55

0

20

40

60

80

100

Seq 3 − 70 frames

Void Sky

Build

ing

Tra

c sig

nTr

ac l

ight

Pole

Pede

stria

nCa

rRo

ad m

ark

Road

Bolla

rdPa

vem

ent

Tree

Other

OF: G = 37.57, C = 40.4IP: G = 78.37, C = 38.23SR: G = 68.19, C = 56.93

Figure 6. Quantitative comparison of OF(baseline), IP and SR label propagation methods. The reader is suggested to glance at the insetglobal(G) and class average(C) accuracies for a quick comparative summary

served for unlabelled pixels in the ground truth) as the cam-era pans. This is the cause of an apparent abberation (seethe OF results for 70 frames), in that, the class average ishigher than the global average. To summarise, the results inFigs. 3, 4, 5 & Fig. 6 indicate that at least one of the twoproposed methods (IP and SR based mappings) is superiorto optic flow based label propagation. From Fig. 6 it is clearthat OF based method is outperformed in 7 out of the 9 ex-periments by the proposed IP or SR based approaches.Semi supervised learning Table 1 reports results of videoclassification experiments with Random forests. The testaccuracy (on Seq 2) undergoes little degradation (class av-erage) as ground truth is progressively replaced by propa-gated labels. From the second row (50 frames) it appearsthat test accuracy is higher when trained with propagatedlabels as compared to ground truth based training, a factwhich can be attributed to randomization in classifier train-ing. Finally, as the width between end frames is increasedto 70 frames, the class average accuracy when trained withpropagated labels degrades only about 5%. These results

should encourage training of classifiers based on the pro-posed methods.

5. ConclusionWe have proposed a probabilistic generative model for

label propagation in video sequences. The inference mech-anism propagates labels to the unlabelled parts of the videosequence in a transductive (batch) setting. Over short se-quences naive optic flow based mapping under this modelperforms acceptable label propagation. More sophisticatedprobabilistic mappings using image patches or determin-istic pre-computed region trajectories provide accurate la-bel propagation even in longer and more challenging se-quences. By means of both qualitative and quantitative re-sults we have demonstrated that the proposed methods pro-vide a tool to extract large quantities of labelled groundtruth, which are useful for semi-supervised learning of dis-criminative classifiers.The variational approximation in Eqn 6 yields only the mostprobable (MP) value of the hidden states. This approxima-

Settings under which the Random Forest is trainedWidth Full ground truth (Seq1 & Seq3) Ground truth (Seq 1) + Propagated labels(Seq 3) Propagated labels (Seq1 & Seq 3)

25 frames G = 62.1%, C = 44.6% G = 61.6%, C = 44.6% G = 59.8%, C = 44.2%50 frames G = 60.5%, C = 45.5% G = 60.5%, C = 47.2% G = 60.7%, C = 46.1%70 frames G = 57.3%, C = 45.5% G = 58.7%, C = 42.2% G = 58.8% , C = 40.3%

Table 1. Test results of Random Forest based classification of Seq 2 trained under three different lengths of training sequences Seq 1and Seq 3 and three different training settings. The comparable test accuracy to training under ground truth provides support for trainingclassifiers using the proposed methods

tion leads to tractable inference of the MP values. The draw-back is that only an “instantaneous” notion of label uncer-tainty can be captured based on the MP values (see Eqns 9,10, 11). In the future we aim to introduce more complex ap-proximating distributions to propagate label uncertainties.We also aim to extend the model to employ discriminativeclassifiers for label propagation in longer sequences.

AcknowledgementsThe authors wish to acknowledge the helpful suggestions

of Ignas Budvytis, Tae-Kyun Kim and Yu Chen. Thanksalso to the funding agencies for their support.

References[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. From con-

tours to regions: An empirical evaluation. In CVPR, 2009.[2] V. Badrinarayanan, F. Galasso, and R. Cipolla. Label prop-

agation in video sequences. Technical report, CUED/F-INFENG/TR-643, March 2010.

[3] Y. Bengio, O. Delalleau, and N. Le Roux. Label propagationand quadratic criterion. In Semi-Supervised Learning, pages193–216. MIT Press, 2006.

[4] G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla. Seg-mentation and recognition using structure from motion pointclouds. In ECCV, 2008.

[5] V. Cheung, B. J. Frey, and N. Jojic. Video epitomes. InCVPR, 2005.

[6] F. Kang, R. Jin, and R. Sukthankar. Correlated label prop-agation with application to multi-label learning. In CVPR,pages 1719–1726, 2006.

[7] A. Kannan, J. Winn, and C. Rother. Clustering appearanceand shape by learning jigsaws. In NIPS, Volume 19., 2006.

[8] P. Kohli, L. Ladicky, and P. Torr. Robust higher order poten-tials for enforcing label consistency. IJCV, 82(3):302–324,2009.

[9] C. Liu, J. Yuen, and A. Torralba. Nonparametric scene pars-ing: Label transfer via dense scene alignment. In CVPR,2009.

[10] P. Sand and S. Teller. Particle video: Long-range motionestimation using point trajectories. IJCV, 80(1):72–91, 2008.

[11] J. Shotton, M. Johnson, and R. Cipolla. Semantic textonforests for image categorization and segmentation. In CVPR,2008.

[12] J. Yuen, B. C. Russell, C. Liu, and A. Torralba. Labelmevideo: building a video database with human annotations. InICCV, Kyoto, Japan, 2009.

[13] C. Zach, T. Pock, and H. Bischof. A duality based approachfor realtime tv-l1 optical flow. In Pattern Recognition (Proc.DAGM), pages 214–223, Heidelberg, Germany, 2007.

[14] X. Zhu and Z. Ghahramani. Learning from labeled and un-labeled data with label propagation. Technical report, CMU-CALD-02-107, Carnegie Mellon University., 2002.

Appendix: Extraction of semantic region tracks

The optic flow u(x) at a pixel x = (x, y) is estimatedusing the algorithm of [13] and smoothed using bilateralfiltering (as in [10]) to improve boundary sharpness:

u(x) =Σx′∈rf u(x′)w(x,x′)

Σx′∈rfw(x,x′), (13)

where rf is the region [1] at frame f and the functionw(x,x′) weighs the neighbouring pixels x′ of x accordingto spatial proximity and motion similarity;

w(x,x′) = N(x; ‖x−x′‖, σx)N(x; ‖u(x)−u(x′)‖, σm).(14)

Here we use σx = 6.9, σm = 3.6 and restrict x′ to liewithin 18 pixels from x′ but within the same region rf ,so as to only smooth the flow inside the same “object”.The filtered optic flow u serves to predict the mask m′rfat frame f + 1 of region rf . Hence the similarity betweenthe predicted and the masks of the N regions at frame

f+1,mrjf+1

Nj=1

, provides the cost of linking the regions:

srif−rjf+1

=m′rif·mrjf+1

m′rif

+mrjf+1

(15)

The hierarchical segmentation of [1] defines semantic re-gions over 255 coarse to fine levels. Dynamic programmingis employed to find all the possible region paths, over allframes and all levels. In particular, forward/backward dy-namic programming assures that a region has unique past-to-future links. This procedure results in multiple trajecto-ries of dense regions at different resolutions, e.g. the shirt,torso and silhouette of a pedestrian.