Part 4: Combined segmentation and recognitionby Rob Fergus (MIT)
AimGiven an image and object category, to segment the objectSegmentation should (ideally) be shaped like the object e.g. cow-like obtained efficiently in an unsupervised manner able to handle self-occlusionSegmentationObjectCategory ModelCow ImageSegmented CowSlide from Kumar 05
Feature-detector view
Examples of bottom-up segmentation Using Normalized Cuts, Shi & Malik, 1997Borenstein and Ullman, ECCV 2002
Jigsaw approach: Borenstein and Ullman, 2002
Perceptual and Sensory Augmented ComputingInterleaved Object Categorization and Segmentation
Implicit Shape Model - Liebe and Schiele, 2003Liebe and Schiele, 2003, 2005
Random Fields for segmentationI = Image pixels (observed)h = foreground/background labels (hidden) one label per pixel = ParametersPriorLikelihoodPosteriorJointGenerative approach models joint Markov random field (MRF)
2. Discriminative approach models posterior directly Conditional random field (CRF)
Generative Markov Random Field I (pixels)Image PlaneijPrior has no dependency on I
Conditional Random FieldLafferty, McCallum and Pereira 2001PairwiseUnary Dependency on I allows introduction of pairwise terms that make use of image.
For example, neighboring labels should be similar only if pixel colors are similar Contrast termDiscriminative approache.g Kumar and Hebert 2003
OBJCUT (shape parameter)Kumar, Torr & Zisserman 2005PairwiseUnary is a shape prior on the labels from a Layered Pictorial Structure (LPS) model
Segmentation by:
- Match LPS model to image (get number of samples, each with a different pose
Marginalize over the samples using a single graph cut [Boykov & Jolly, 2001]Label smoothnessContrastDistance from Color Likelihood
OBJCUT:Shape prior - - Layered Pictorial Structures (LPS)Generative modelComposition of parts + spatial layout
Layer 2Layer 1Parts in Layer 2 can occlude parts in Layer 1Spatial Layout(Pairwise Configuration)Kumar, et al. 2004, 2005
OBJCUT: ResultsIn the absence of a clear boundary between object and backgroundSegmentationImageUsing LPS Model for Cow
Levin & Weiss [ECCV 2006] Segmentation alignment with image edgesConsistency with fragments segmentation
Winn and Shotton 2006Layout Consistent Random Field
Layout consistencyNeighboring pixels(p,q)?(p,q+1)(p,q)(p+1,q+1)(p-1,q+1)Layout consistentWinn and Shotton 2006
Layout Consistent Random FieldWinn and Shotton 2006
Stability of part labellingPart color key
Object-Specific Figure-Ground SegregationStella X. Yu and Jianbo Shi, 2002
Image parsing: Tu, Zhu and Yuille 2003
Image parsing: Tu, Zhu and Yuille 2003
Segment out all the cars.fused tree model for carsUnseen imageTraining imagesSegmented CarsSegmentation TreesOverviewMultiscale Seg.Todorovic and Ahuja, CVPR 2006Slide from T. Wu
LOCUS modelDeformation field DPosition & size T Class shape Class edge sprite o,oEdge image eImageObject appearance 1Background appearance 0Mask mShared between imagesDifferent for each imageKannan, Jojic and Frey 2004Winn and Jojic, 2005
In this section: brief paper reviewsJigsaw approach: Borenstein & Ullman, 2001, 2002Concurrent recognition and segmentation: Yu and Shi, 2002Image parsing: Tu, Zhu & Yuille 2003 Interleaved segmentation: Liebe & Schiele, 2004, 2005OBJCUT: Kumar, Torr, Zisserman 2005LOCUS: Winn and Jojic, 2005LayoutCRF: Winn and Shotton, 2006Levin and Weiss, 2006Todorovic and Ahuja, 2006
SummaryStrengthExplains every pixel of the imageUseful for image editing, layering, etc.
IssuesInvariance issues(especially) scale, view-point variationsInference difficulties
Conditional Random Fields for SegmentationSegmentation map xImage ILow-level pairwise termHigh-level local termPixel-wise similarity
Object-Specific Figure-Ground SegregationSome segmentation/detection resultsYu and Shi, 2002
Multiscale Conditional Random Fields for Image LabelingXuming He Richard S. Zemel Miguel A . Carreira-PerpinanConditional Random Fields for ObjectRecognitionAriadna Quattoni Michael Collins Trevor Darrell
OBJCUTProbability of labelling in addition has Unary potential which depend on distance from (shape parameter)D (pixels)m (labels) (shape parameter)Image PlaneObject CategorySpecific MRFxymxmyUnary Potentialx(mx|)Kumar, et al. 2004, 2005
Localization using features
Levin and Weiss 2006Levin and Weiss, ECCV 2006
Results: horses
Results: horses
Cows: ResultsSegmentations from interest points
Single-frame recognition - No temporal continuity used!Liebe and Schiele, 2003, 2005
Examples of low-level image segmentationNormalized Cuts, Shi & Malik, 1997Borenstein & Ullman, ECCV 2002
Jigsaw approachEach patch has foreground/background mask
LayoutCRF
Segmentation
Interpretation of p(figure) mapper-pixel confidence in object hypothesisUse for hypothesis verificationLiebe and Schiele, 2003, 2005
Different occlusions preserves ordering, deformations preserve ordering
*Different occlusions preserves ordering, deformations preserve ordering
*Edge weight larger at image edges**Write down the contribution part of this paperEmphasise class model (shared) all other variables per-image. Emphasise LEARN EVERYTHING SIMULTANEOUSLY.
Top Related