Fast, Multiscale Image Fast, Multiscale Image Segmentation: From Pixels to Segmentation: From Pixels to
SemanticsSemantics
Ronen BasriThe Weizmann Institute of Science
Joint work with
Achi Brandt, Meirav Galun, Eitan Sharon
SSegmentation by egmentation by WWeighted eighted AAggregationggregation
A multiscale algorithm:• Optimizes a global measure• Returns a full hierarchy of segments• Linear complexity• Combines multiscale measurements:
– Texture– Boundary integrity
The Pixel GraphThe Pixel GraphCouplings (weights)
reflect intensity
similarity
Low contrast –strong couplingHigh contrast –weak coupling
i jI I
ijW e
Normalized-cut MeasureNormalized-cut Measure
( )( )
( )
Cut SE S
Int S
Minimize:
2( ) ( )ij i ji j
Cut S w u u
( ) ij i ji j
Int S w u u
Si
Siui 0
1
Saliency MeasureSaliency Measure
2( ) ( )
1( )
2
( ) 2
Tij i j
i j
Tij i j
i j
T
T
Cut S w u u u Lu
Int S w u u u Wu
u LuE S
u Wu
Lu WuMinimize:
Multiscale Computation Multiscale Computation of Ncutsof Ncuts
• Our objective is to rapidly find the segments (0-1 partitions) that optimize
• For single-node cuts we simply evaluate • For multiple-node cuts we perform “soft
contraction” using coarsening procedures from algebraic multigrid solvers of PDEs.
Coarsening the GraphCoarsening the Graph
• Suppose we can define a sparse mappingsuch that for all minimal states
: , ( / 2)N nP N n R R
11
22
.
..
Nn
Uu
P
uU
uU
Coarse EnergyCoarse Energy
• Then
• PTWP, PTLP define a new (smaller)
graph
( ) 2 2T T T
T T T
u Lu U P LPUE S
u Wu U P WPU
Recursive CoarseningRecursive Coarsening
11
22
.
..
Nn
Uu
P
uU
uU
For a salient segment :
( )n NP ,sparse interpolation matrix
iu julUkU
Weighted AggregationWeighted Aggregation
ijwi
jjlp
aggregate k aggregate l
[[ 1] ]s T sWW P P
klWikp
HierarchicHierarchical Graphal Graph
Pyramid of graphs Soft relations between levels Segments emerge as salient nodes at some level of the pyramid
Physical MotivationPhysical Motivation
• Our algorithm is motivated by algebraic multigrid solutions to heat or electric networks
• u - temperature/potential• a(x, y) – conductivity• At steady state largest temperature
differences are along the cuts• AMG coarsening is independent of f
( , ) ( , ) ( , )u u
a x y a x y f x yx x y y
A Chicken and Egg A Chicken and Egg Problem Problem
Problem:Coarse measurements mix neighboring statistics
Solution: Support of measurements is determined as the segmentation process proceeds
Hey, I was here first
Texture AggregationTexture Aggregation
• Aggregates assumed to capture texture elements
• Compare neighboring aggregates according to the following statistics:– Multiscale brightness measures– Multiscale shape measures– Filter responses
• Use statistics to modify couplings
Recursive Computation of Recursive Computation of MeasuresMeasures
• Given some measure of aggregates at a certain level (e.g., orientation)
• At every coarser level we take a weighted sum of this measure from previous level
• The result can be used to compute the average, variance or histogram of the measure
• Complexity is linear
Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements
Averaging
Our algorithm - SWA
Original
Geometric
Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements
Interpolation Geometric
Original
Our algorithm - SWA
Key DifferencesKey Differences
• Optimize a global measure(like Malik’s Ncuts)
• Hierarchy with soft relations(unlike agglomerative/graph contraction)
• Combine texture measurements while avoiding the “chicken and egg problem”
ComplexityComplexity
• Every level contains about half the nodes of the previous level:
Total #nodes 2 X #pixels• All connections are local, cleaning small
weights• Top-down sharpening: constant number
of levels• Linear complexity• Implementation: 5 seconds for 400x400
MS Lesion DetectionMS Lesion Detection
TaggedTagged Our resultsOur results
Data: FilippiData: Filippi
Top Related