Sections 10 1Sections 10.1 Edge linking 10.2.7 (very briefly) … · “How did Andy Warhol get his...
Transcript of Sections 10 1Sections 10.1 Edge linking 10.2.7 (very briefly) … · “How did Andy Warhol get his...
INF 4300 14.10.09Image segmentationImage segmentation
“How did Andy Warhol get his inspiration?”Sections 10 1Sections 10.1
Edge linking 10.2.7 (very briefly)10.410.5
10.6.1A S S lbAnne S. Solberg
Anne Schistad Solberg, Department of InformaticsAnne Schistad Solberg, Department of Informatics
Today
• Segmentation approaches
1. Region growing1. Region growing
2. Region split and merge
3. Watershed
4. Edge-based segmentation
5. Segmentation by motion
Assumed known:
1. Edge detection, point and line detection (10.2.1-10.2.6)1. Edge detection, point and line detection (10.2.1 10.2.6)
2. Segmentation by thresholding (10.3)
Anne Schistad Solberg, Department of Informatics
SegmentationSegmentation
Segmentation is separation of one or more regions orSegmentation is separation of one or more regions or objects in a image based on a similarity criterion
A region in an image can be defined by its border (edge) or its interior, and the two representations are equal.
If you know the interior, you can always define the border, and vice versa.and vice versa.
Because of this, image segmenation approaches cantypically be divided into two categories, edge and region based methodsbased methods.
Anne Schistad Solberg, Department of Informatics
What is segmentation?What is segmentation?
What if you wanted to group all the pixels corresponding to thecar in this image into one group Would that be easy?
Anne Schistad Solberg, Department of Informatics
car in this image into one group. Would that be easy?
What is a region?A region R of an image I is defined as a connected homogenousA region R of an image I is defined as a connected homogenous subset of the image with respect to some criterion such as gray level or texture
A segmentation of an image f is a partition of f into severalA segmentation of an image f is a partition of f into several homogeneous regions Ri, i=1,….m
An image f can be segmented into regions Ri such as:
P(Ri) is a logical predicate defined over all points in Ri. It must be true for all pixels inside the region and false for pixels in other regions.
T i R d R i hb if th i i f t d
Anne Schistad Solberg, Department of Informatics
Two regions Ri and Rj are neighbors if their union forms a connected component.
Segmentation approaches
• Pixel-based segmentation: each pixel is segmentedbased on grey-level values, no contextualg y ,information (information from other neighboringpixels). Example: thresholding.
• Region-based segmentation: takes into accountgrey-levels from neighboring pixels by eitheri l di i il i hb i i l ( i i )including similar neighboring pixels (region growing), split-and-merge, or watershed segmentation.
• Edge based segmentation: Detect edges and link• Edge-based segmentation: Detect edges and link the edges together to form contours.
Anne Schistad Solberg, Department of Informatics
Need repetition of thresholding?
• Read section 10.3 in Gonzalez and Woods.
• English foils on thresholding can be found atEnglish foils on thresholding can be found at
http://www.uio.no/studier/emner/matnat/ifi/INF3300/h06/undervisningsmateriale/thresholding 2006.pdf/undervisningsmateriale/thresholding_2006.pdf
Anne Schistad Solberg, Department of Informatics
R i d b d hRegion vs. edge-based approaches
• Region based methods are robust because• Region based methods are robust because– Regions cover more pixels than edges and thus you have more
information available in order to characterize your regionWhen detecting a region you could for instance use texture which is not– When detecting a region you could for instance use texture which is not easy when dealing with edges
– Region growing techniques are generally better in noisy images where edges are difficult to detectedges are difficult to detect
• The edge based method can be preferable because:– Algorithms are usually less complex– Edges are important features in a image to separate regions
• The edge of a region can often be hard to find because of noise or occlusions
Anne Schistad Solberg, Department of Informatics
occlusions• Combination of results may often be a good idea
Edge-based Segmentation• A large group of methods based on edge information.• We only mention the simplest methods here.
• Rely on edges found in an image by edge detecting operators (discontinuities in gray level, color, texture, etc.)
• Image resulting from edge detection cannot be used as a• Image resulting from edge detection cannot be used as a segmentation result
• Post processing steps must follow to combine edges into edge p g p g gchains to represent the region border
• The more prior information used in the segmentation process, th b tt th t ti lt b bt i dthe better the segmentation results can be obtained
• The most common problems of edge-based segmentation are:– edge presence in locations where there is no border
Anne Schistad Solberg, Department of Informatics
– edge presence in locations where there is no border – no edge presence where a real border exists
Edge-based Segmentation
1. Edge detection (Gradient, Laplacian, LoG, Canny filtering) g)
2. Edge linking – Local Processing
• strength of the response of the gradient
• direction of the gradient vector
• Edges in a predefined neighborhood is linkedEdges in a predefined neighborhood is linked
if both magnitude and direction criteria is satisfied
– Global Processing via Hough Transform (later)
Anne Schistad Solberg, Department of Informatics
Why is more than a gradient operatorWhy is more than a gradient operator needed for edge segmentation?
• Most images will produce a very complicated edge map under the Sobel filter.O l l ill th di t it d b• Only rarely will the gradient magnitude be zero.
• Calculating an approximation to the gradient vector in an image will generally not tell you were the salient edges are.
Anne Schistad Solberg, Department of Informatics
Method 1
Russian ground to air missiles near Havana, Cuba. The imagewas taken by an American U2 aircraft on the 29th of August 1962.
Anne Schistad Solberg, Department of Informatics
Method 1
Magnitude of gradient image, min=0, max=1011
Anne Schistad Solberg, Department of Informatics
Method 1
Threshold at 250
Anne Schistad Solberg, Department of Informatics
Method 1
Threshold at 350
Anne Schistad Solberg, Department of Informatics
Method 1
• What if we assume the following:– All gradient magnitudes above a strict threshold are assumed to g g
belong to a bona fide edge.
– All gradient magnitudes above an unstrict threshold and connected to a pixel resulting from the strict threshold are also assumed to p gbelong to real edges.
• Hysteresis thresholding – Canny’s edge detection ( INF 2310)(see INF 2310).
Anne Schistad Solberg, Department of Informatics
Method 1
Result of hysteresis, are we really impressed?
Anne Schistad Solberg, Department of Informatics
Method 2
• One evident problem is the thickening of edges.
• One simple method for reducing this is based onOne simple method for reducing this is based on using directional information provided by the edge detector.
• The method is described in the next slide.
Anne Schistad Solberg, Department of Informatics
Method 2
• Quantize the edge directions into eight (or four) directions.
• For each gradient magnitude pixel with nonzero magnitude, inspect its two neighboring pixels in the directions from point 1directions from point 1.
• If the edge magnitude of any of these neighbors are higher than that under consideration, mark it for g ,deletion.
• When all pixels have been scanned, delete those k d f d l timarked for deletion.
Anne Schistad Solberg, Department of Informatics
Edge based SegmentationEdge-based Segmentation exampleLeft: Input imageRight: G_y
gg
Left: G_xRight: Result after Right: Result after edge linking
Anne Schistad Solberg, Department of Informatics
Edge based segmentation
• Advantages– Similar approach as human segment an image
W k ll i i ith d t t b t bj t d– Works well in images with good contrast between object and background
• Disadvantages– Do not work well on images with smooth transitions and low
contrast– Sensitive to noise– Edge linking is not trivial
Anne Schistad Solberg, Department of Informatics
Region growing• Grow regions by recursively including the neighboring pixels that G o eg o s by ecu s e y c ud g t e e g bo g p e s t at
are similar and connected to the seed pixel.• Connectivity is needed not to connect pixels in different parts of
the image.
• Similarity measures - examples: – Difference in grey levels for regions with homogeneous grey levels– Texture features for textured regions
Similary: grey level differenceCannot be segmented using greylevel similarity
Anne Schistad Solberg, Department of Informatics
y g y level similarity.
Region growing• Starts with a set of seeds (starting pixels)Sta ts t a set o seeds (sta t g p e s)
– Predefined seeds– All pixels as seeds– Randomly chosen seedsy
• Region growing steps (bottom-up method)– Find starting points– Include neighboring pixels with similar features (grey-level, texture,
color) A similarity measure must be selected.Two variants: 1 Select seeds from the whole range of grey levels in the image Grow1. Select seeds from the whole range of grey levels in the image. Grow
regions until all pixels belong to a region.2. Select seed only from objects of interest (e.g. bright structures). Grow
regions only as long as the similarity criterion is fulfilled.g y g y
• Problems:– Not trivial to find good starting points
Anne Schistad Solberg, Department of Informatics
Not trivial to find good starting points– Need good criteria for similarity
Region growing example: WeldRegion growing example: Weld inspection
X-ray image of defective weld SeedsCriteria used:
• Seeds: f(x,y) = 255
X ray image of defective weld
↓Seeds
↓( ,y)
• P(R) = TRUE if – | seed gray level – new | g y
pixel gray level | < 65
and
– New pixel must be 8-connected with at least one pixel in the regionpixel in the region
Result of region growing →
Anne Schistad Solberg, Department of Informatics
Similarity measure
• Intensity difference within a region (from a pixel to a seed or the mean of a region)
• Within a Maximum and Minimum value
• Distance between mean value of the regions (specially for region merging or splitting)(specially for region merging or splitting)
• Variance or standard deviation within a region• Variance or standard deviation within a region
• Difference in another feature, e.g. texture
Anne Schistad Solberg, Department of Informatics
Difference in another feature, e.g. texture
Region merging techniques
• One region based type of segmentation methods is the so called region merging method.g g g
• Very simple method. Initialization is done as follows:– Start by giving all the pixels a unique label (a possible variation is
to give groups of 2 and 2 or 4 and 4 pixels a unique label).
– All pixels in the image are assigned to a region.
Anne Schistad Solberg, Department of Informatics
Region merging techniques
• The rest of the algorithm is as follows:– In some predefined order, examine the neighbor regions of all p g g
regions and decide if the predicate evaluates to true for all pairs of neighboring regions.
– If the predicate evaluates to true for a given pair of neighboring p g p g gregions then give these neighbors the same label. The predicate is the similarity measure (can be defined based on e.g. region mean values or region min/max etc.).
– Continue until no more mergings are possible.
– Upon termination all region criteria will be satisfied.
Anne Schistad Solberg, Department of Informatics
Region merging techniques
• An interesting case for region merging is the image to the rightto the right.
• The aim is to separate the apples from the background.Thi i f• This image poses more of a challenge than you might think.
Anne Schistad Solberg, Department of Informatics
Region merging techniques
• We run a standard region merging procedure where all i l i iti ll ipixels initially are given a
unique label.
• If neighboring regions have g g gmean values within 10 gray levels they are fused,
• Regions are considered• Regions are considered neighbors in 8-connectivity.
Anne Schistad Solberg, Department of Informatics
Region merging techniques
Anne Schistad Solberg, Department of Informatics
How Andy Warhol really did it…
Anne Schistad Solberg, Department of Informatics
Region merging techniques
• A caveat:
• Remember that initialization is critical segmentationRemember that initialization is critical, segmentation results will in general depend on the initialization.
• The order in which the regions are treated will alsoThe order in which the regions are treated will also influence the result, in the images in the next slide the right image was flipped upside down before it was fed to the merging algorithm.
Anne Schistad Solberg, Department of Informatics
Region merging techniques
The right image was flipped upside down before being fedto the region merging algorithm, notice the differences
b t th t
Anne Schistad Solberg, Department of Informatics
between the two.
Split and merge
• Separate the image in regions based on a given similarity measure. Then merge regions based on y g gthe same or a different similarity measure
• The method is also called "quad tree" division
1. Set up some criteria for what is a uniform area (ex mean, i t t t )variance, texture, etc)
2. Start with the full image and split it in to 4 sub-images.
3. Check each sub-image. If not uniform, divide into 4 new sub-images
4. After each iteration, compare neighboring regions end merge if uniform according to the similarity measure.
Anne Schistad Solberg, Department of Informatics
g y
Anne Schistad Solberg, Department of Informatics
Split and merge exampleSplit and merge example
Anne Schistad Solberg, Department of Informatics
W t h d t ti th idWatershed segmentation – the idea
L k t th i t hi f ith b th ll d• Look at the image as a topographic surface, with both valleys and mountains
• Assume that there is a hole in each minima and the surface is• Assume that there is a hole in each minima and the surface is immersed into a lake
• The water will enter through the holes at the minima and flood theThe water will enter through the holes at the minima and flood the surface
• To avoid the water coming from two different minima to meet, a dam gis build
• Final step: the only thing visible of the surface would be the dams
• The walls between the dams are called the watershed lines
Anne Schistad Solberg, Department of Informatics
Watershed segmentationD t t t d i ith i il l (i t it )• Detect connected regions with similar value (intensity )
• Every minimum correspond to a region
M tl b t h d(i )• Matlab: watershed(im, conn)
Anne Schistad Solberg, Department of Informatics
Watershed segmentation
Water graduallyrise
At the pointwhere tworegions
Negative distance transform
regions flood, dams are built
Watershed transform of D
Anne Schistad Solberg, Department of Informatics
Watershed segmentation
Original Original, Original image
Original, topographicview
Water rises Water nowand fill thedark background
Water nowfills one ofthe dark regions
Watershed segmentation
The twoThe twobasins areabout to meet, dam constructionco st uct ostarts
Final segmentationresult
Video: http://cmm.ensmp.fr/~beucher/wtshed.html
Watershed segmentation
• Can be used on images derived from:
– The intensity image
– Edge enhanced image
– Distance transformed image
• Thresholded image. From each foreground pixel, compute the distance to a background pixel.
– Gradient of the image
M t di t i• Most common: gradient image
Anne Schistad Solberg, Department of Informatics
Watershed algorithmL t ( ) b th i t i ( ft di t• Let g(x,y) be the input image (often a gradient image).
It has several local minima Let M M be the• It has several local minima. Let M1,…MR be thecoordinates of the regional minima.
• Let C(M ) be a set consisting of the coordinates of all• Let C(Mi) be a set consisting of the coordinates of all points belonging to the catchment basin associatedwith the regional mimimum Mi. g i
• Let T[n] be the set of coordinates (s,t) where g(s,t)<t{ }ntsgtsnT <= ),(|),(][
This is the set of coordinates lying below the plane g(x,y)=n
This is the candidate pixels for inclusion into the catchment basin, butwe must take care that the pixels do not belong to a different
{ }g ),(|),(][
Anne Schistad Solberg, Department of Informatics
we must take care that the pixels do not belong to a differentcatchment basin.
Watershed algorithm cont.
• The topography will be flooded with integer floodincrements from n=min-1 to n=max+1.
• Let Cn(Mi) be the set of coordinates of point in thecatchment basin associated with Mi that are floodedi
at stage n. This must be a connected componentand can be expressed as Cn(Mi) = C(Mi)∩T[n] (onlyth ti f T[ ] i t d ith b i M )the portion of T[n] associated with basin Mi)
• Let C[n] be the union of all flooded catchments at stage n: Rstage n:
)(]1[max
and
)(][1
i
R
in
R
i
MCC
MCnC
U
U=
=+
=
Anne Schistad Solberg, Department of Informatics
)(]1[max1
ii
MCC U=
+
C 1[M1] C [M ]Dam constructionStage n 1: two basins They form Step n-1C[n 1]
Cn-1[M1] Cn-1[M2]
Stage n-1: two basins. They form separate connected components.
To consider pixels for inclusion in
Step n 1C[n-1]
basin k in the next step (after flooding), they must be part of T[n], and also be part of the connected Step n
qp
component q of T[n] that Cn-1[k] is included in.
Use morphological dilation iteratively
p
Use morphological dilation iteratively
The dilation of C[n-1] is constrained to q
The dilation can not be performed on pixels that would cause two basins to be merged (form a single connected g ( gcomponent)
Watershed algorithm cont.I iti li ti l t C[ i 1] T[ i 1]• Initialization: let C[min+1]=T[min+1]
• Then recursively compute C[n] from C[n-1] as:L t Q b th t f t d t i T[ ]– Let Q be the set of connected components in T[n].
– For each component q in Q, there are three possibilities:
1. q∩C[n-1] is empty – new minimum
Combine q with C[n-1] to form C[n]
2. q∩C[n-1] contains one connected component of C[n-1]
q lies in the catchment basin of a regional minimumq lies in the catchment basin of a regional minimum
Combine q with C[n-1] to form C[n]
3. q∩C[n-1] contains more than one connected component of C[n-1]
q is then on a ridge between catchment basins, and a dam must be built to prevent overflow.
Construct a one-pixel dam by dilating q∩C[n-1] with a 3x3
Anne Schistad Solberg, Department of Informatics
structuring element, and constrain the dilation to q.
Watershed alg.
• Constrain n to only existing intensity values of g(x,y) (obtain them from the histogram)( g )
Anne Schistad Solberg, Department of Informatics
Watershed segmentation
Challenge: over-segmentation
Image IGradient magnitude image (g)image (g)
Watershed of g W t h d f of g Watershed of
smoothed g
Using the gradient image directly can cause over-segmentation because of noise and small irrelevant intensity changes
Improved by smoothing the gradient image or using markers
Solution: Watershed with markers
A marker is a connected component in the image A marker is a connected component in the image
Can be found by intensity, size, shape, texture etc
Internal markers are associated with the object (a region surrounded j ( gby bright point (of higher altitude))
External markers are associated with the background (watershed lines)lines)
Segment each sub-region by some segmentation algorithm
How to find markers
• Apply filtering to get a smoothed image
• Segment the smooth image to find the internal markers. – Look for a set of point surrounded by bright pixels. How this segmentation should be
done is not well defined. Many methods can be used.
• Segment the smooth image using watershed to find the externalmarkers, with the restriction that the internal markers are theonly allowed regional minima. The resulting watershed lines arethen used as external markers.
• We now know that each region inside an external marker consists of a single object and its background. Apply a segmentation algorithm (watershed region growing thresholdsegmentation algorithm (watershed, region growing, thresholdetc. ) only inside each watershed.
Anne Schistad Solberg, Department of Informatics
W t h d d d lWatershed – advanced example
Original Opening Threshold globalOriginal Opening Threshold, global
Matlab:
Imopen bwdist imimposemin imregionalmax
Watershed of inverse Second watershed to split Distance transform- Distance from a point in a
imregionalmax watershed
Anne Schistad Solberg, Department of Informatics
of distance transform cellsDistance from a point in a
region to the border of the region
Watershed – example 3
O i i l Opening Th h ldOriginal Opening Threshold
Matlab:i• imopen
• imimposemin• bwmorph
Find internal and Watershed
• watershed
Anne Schistad Solberg, Department of Informatics
external markers from gradient image
Watershed
• Advantages– Gives connected componentsp
– A priori information can be implemented in the method using markers
• Disadvantages :• Disadvantages :– Often needs preprocessing to work well
– Over-segmentation can be a problem
Anne Schistad Solberg, Department of Informatics
Segmentation by motion
• In video surveillance one of the main tasks is to segment foreground objects from the background g g j gscene to detect moving objects.
• The background may have a lot of natural movements and changes (moving trees, snow, sun, shadows etc)shadows etc)
Two main segmentation approaches are:• Two main segmentation approaches are:– Intensity difference between two frames are thresholded
– Background subtraction
Anne Schistad Solberg, Department of Informatics
g
Background subtraction
• Approach:– Estimate the background image (per pixel) by computing the mean g g (p p ) y p g
and variance of the n previous time frames at pixel (x,y).
– Subtract background image from current frame
– Differences above a threshold are interesting pixelsDifferences above a threshold are interesting pixels
– Note that the background is updated by only considering the n previous frames. This eliminated slow trends.
Anne Schistad Solberg, Department of Informatics
Other segmentation methods
• Active contour/snakes: Adjust a curve around the object. Initialize a curve and then change it based on a cost function (INF 5300)
• PDE/Level set: Describes the image as partial differential equations. Look at segmentation as a cut in 3D plan. Initialize a curve and then change it based on a cost function.change it based on a cost function.
• Graph-cutting methods.
• Statistical models (MRF), classification
• Neural network for classification of pixels• Neural network for classification of pixels• And many others…………
Anne Schistad Solberg, Department of Informatics