Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2...

72
Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ [email protected]

Transcript of Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2...

Page 1: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Computer Vision – Lecture 2

Binary Image Analysis

25.10.2012

Bastian LeibeRWTH Aachenhttp://www.vision.rwth-aachen.de/

[email protected]

Page 2: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Announcements

• Please subscribe to the lecture on the Campus system! Important to get email announcements and L2P

access! Bachelor students please also subscribe

2B. Leibe

Page 3: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Binary Images

• Just two pixel values• Foreground and background• Regions of interest

B. LeibeSlide credit: Kristen Grauman3

Page 4: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Uses: Industrial Inspection

B. LeibeR. Nagarajan et al. “A real time marking inspection schemefor semiconductor industries“, 2006

4

Page 5: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Uses: Document Analysis, Text Recognition

Source: Till Quack, Martin Renold

Natural text (after detection)

Handwritten digits

Scanned documents

5B. Leibe

Page 6: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Uses: Medical/Bio Data

B. Leibe

Source: D. Kim et al., Cytometry 35(1), 1999

6

Page 7: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Uses: Blob Tracking & Motion Analysis

B. Leibe

Frame Differencing

Source: Kristen Grauman

Background Subtraction

Source: Tobias Jäggli7

Page 8: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Uses: Shape Analysis, Free-Viewpoint Video

B. Leibe

Medial axis

Silhouette

Visual Hull Reconstruction

Blue-c project, ETH Zurich

8

Page 9: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Uses: Intensity Based Detection

• Looking for dark pixels…

9B. Leibe

fg_pix = find(im < 65);

Slide Credit: Kristen Grauman

Page 10: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Uses: Color Based Detection

• Looking for pixels within a certain color range…

10B. Leibe

fg_pix = find(hue > t1 & hue < t2);

Slide Credit: Kristen Grauman

Page 11: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Issues

• How to demarcate multiple regions of interest? Count objects Compute further features per object

• What to do with “noisy” binary outputs? Holes Extra small fragments

11B. LeibeSlide Credit: Kristen Grauman

Page 12: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Outline of Today’s Lecture

• Convert the image into binary form Thresholding

• Clean up the thresholded image Morphological operators

• Extract individual objects Connected Components Labeling

• Describe the objects Region properties

B. Leibe Image Source: D. Kim et al., Cytometry 35(1), 199912

Page 13: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Thresholding

• Grayscale image Binary mask• Different variants

One-sided

Two-sided

Set membership

B. Leibe

otherwise ,0

, if ,1,

TjiFjiFT

otherwise ,0

, if ,1, 21 TjiFT

jiFT

otherwise ,0

, if ,1,

ZjiFjiFT

Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/13

Page 14: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Selecting Thresholds

• Typical scenario Separate an object from a distinct background

• Try to separate the different grayvalue distributions Partition a bimodal histogram Fit a parametric distribution (e.g. Mixture of Gaussians) Dynamic or local thresholds

• In the following, I will present some simple methods.• We will see some more general methods in Lecture

6…

B. Leibe14

Page 15: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

A Nice Case: Bimodal Intensity Histograms

B. LeibeSource: Robyn Owens

Ideal histogram,light object on dark background

Actual observedhistogram with noise

15

Page 16: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Not so Nice Cases…

• How to separate those?

• Threshold selection is difficult in the general case Domain knowledge often helps E.g. Fraction of text on a document page ( histogram

quantile) E.g. Size of objects/structure elements

B. LeibeSource: Shapiro & Stockman

Multiple modesTwo distinct modesOverlapping modes

16

Page 17: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Global Binarization [Otsu’79]

• Search for the threshold T that minimizes the within-class variance within of the two classes

separated by T

where

• This is the same as maximizing the between-class variance between

B. Leibe17

¾2within (T) = n1(T)¾2

1 + n2(T)¾22(T)

n1(T) = j©I (x;y) < T

ªj; n2(T) = j

©I (x;y) ¸ T

ªj

¾2between (T) = ¾2 ¡ ¾2

within (T)

= n1(T)n2(T) [¹ 1(T) ¡ ¹ 2(T)]2

Page 18: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Algorithm

Precompute a cumulative grayvalue histogram h.

For each potential threshold T1.) Separate the pixels into two clusters according to T

2.) Look up n1, n2 in h and compute both cluster means3.) Compute

Choose

B. Leibe18

T¤ = argmaxT

£¾2

between (T)¤

¾2between

Page 19: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Local Binarization [Niblack’86]

• Estimate a local threshold within a small neighborhood window W

where k [-1,0] is a user-defined parameter.

B. Leibe

WWW kT

19

TW

Effect:

W

W

What is the hiddenassumption here?

Page 20: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Effects

B. Leibe21

Original image

Global threshold selection(Otsu)

Local threshold selection(Niblack)

Page 21: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Additional Improvements

• Document images often contain a smooth gradient

Try to fit that gradient with a polynomial function

B. LeibeSource: S. Lu & C. Tan, ICDAR’07

Original image

Binarized resultShading compensation

Fitted surface

22

Page 22: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Surface Fitting

• Polynomial surface of degree d

• Least-squares estimation, e.g. for d=3 (m=10)

B. Leibe

,0

( , )d

i ji j

i j

f x y b x y

IAb 2 3

0 0 0 0 0 0 0 02 3

1 1 1 1 1 1 1 1

2 3

1

1 ,

1 n n n n n n m n

x y x x y y b I

x y x x y y b I

x y x x y y b I

AIb \

Matlab (using SVD):

IAAAb TT 1)(

Solution withpseudo-inverse:

23

Page 23: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Surface Fitting

• Iterative Algorithm1.) Fit parametric surface to all points in region.2.) Subtract estimated surface.3.) Apply global threshold (e.g. with Otsu method)

4.) Fit surface to all background pixels in original region.5.) Subtract estimated surface.6.) Apply global threshold (Otsu)

7.) Iterate further if needed…

• The first pass also takes foreground pixels into account. This is corrected in the following passes. Basic assumption here: most pixels belong to the

background.

B. Leibe

Initialguess

Refinedguess

24

Page 24: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Result Comparison

B. Leibe

Original image Global (Otsu)

Polynomial + Global

Local (Niblack)

Source: S. Lu & C. Tan, ICDAR’0726

Page 25: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Outline of Today’s Lecture

• Convert the image into binary form Thresholding

• Clean up the thresholded image Morphological operators

• Extract individual objects Connected Components Labeling

• Describe the objects Region properties

B. Leibe Image Source: D. Kim et al., Cytometry 35(1), 199927

Page 26: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Cleaning the Binarized Results

• Results of thresholding often still contain noise

• Necessary cleaning operations Remove isolated points and small structures Fill holes

Morphological Operators

B. Leibe Image Source: D. Kim et al., Cytometry 35(1), 199928

Page 27: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Morphological Operators

• Basic idea Scan the image with a structuring element Perform set operations (intersection, union)

of image content with structuring element

• Two basic operations Dilation (Matlab: imdilate) Erosion (Matlab: imerode)

• Several important combinations Opening (Matlab: imopen) Closing (Matlab: imclose) Boundary extraction

B. Leibe Image Source: R.C. Gonzales & R.E. Woods29

Matlab:>> help strel

Page 28: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Dilation

• Definition “The dilation of A by B is the set

of all displacements z, such that and A overlap by at least one element”.

( is the mirrored version of B, shifted by z)

• Effects If current pixel z is foreground, set all

pixels under (B)z to foreground.

Expand connected components Grow features Fill holes

B. Leibe

A

B1

B2

A B1

A B2

Image Source: R.C. Gonzales & R.E. Woods

zB)ˆ(

zB)ˆ(

30

Page 29: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Erosion

• Definition “The erosion of A by B is the set

of all displacements z, such that is entirely contained in A”.

• Effects If not every pixel under (B)z is

foreground, set the current pixel zto background.

Erode connected components Shrink features Remove bridges, branches, noise

B. Leibe Image Source: R.C. Gonzales & R.E. Woods

A

B1

B2

1

2

zB)(

31

Page 30: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Effects

B. Leibe

Original image

Dilation with circularstructuring element

Erosion with circularstructuring element

Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/32

Page 31: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Effects

B. Leibe

Original image

Dilation with circularstructuring element

Erosion with circularstructuring element

Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/33

Page 32: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Opening

• Definition Sequence of Erosion and Dilation

• Effect is defined by the points that

are reached if B is rolled around inside A.

Remove small objects,keep original shape.

B. Leibe Image Source: R.C. Gonzales & R.E. Woods34

Page 33: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Effect of Opening

• Feature selection through sizeof structuring element

Original image

Opening with smallstructuring element

Thresholded

Opening with largerstructuring elementImage Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/

35B. Leibe

Page 34: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Effect of Opening

• Feature selection through shapeof structuring element

• How could we have extracted the lines?B. Leibe

Opening with circularstructuring element

Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/

Input Image

36

Page 35: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Closing

• Definition Sequence of Dilation and Erosion

• Effect is defined by the points that

are reached if B is rolled around on the outside of A.

Fill holes,keep original shape.

B. Leibe Image Source: R.C. Gonzales & R.E. Woods37

Page 36: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Effect of Closing

• Fill holes in thresholded image(e.g. due to specularities)

Original image Thresholded Closing with circularstructuring element

Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/

Size of structuringelement determineswhich structures areselectively filled.

38B. Leibe

Page 37: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Example Application: Opening + Closing

B. Leibe

Original image ClosingOpening

Dilated imageEroded image

Structuringelement

Source: R.C. Gonzales & R.E. Woods39

Page 38: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Application: Blob Tracking

Slide credit: K. Grauman

Absolute differences from frame to frame

40B. Leibe

Page 39: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Slide credit: K. Grauman

Thresholding

41B. Leibe

Page 40: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Slide credit: K. Grauman

Eroding

42B. Leibe

Page 41: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Morphological Boundary Extraction

• Definition First erode A by B, then subtract

the result from the original A.

• Effects If a 33 structuring element is used,

this results in a boundary that isexactly 1 pixel thick.

B. Leibe Source: R.C. Gonzales & R.E. Woods

)()( BAAA

43

Page 42: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Morphology Operators on Grayscale Images

• Dilation and erosion are typically performed on binary images.

• If image is grayscale: for dilation take the neighborhood max, for erosion take the min.

44B. Leibe

Original Dilated Eroded

Slide credit: Kristen Grauman

Page 43: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Outline of Today’s Lecture

• Convert the image into binary form Thresholding

• Clean up the thresholded image Morphological operators

• Extract individual objects Connected Components Labeling

• Describe the objects Region properties

B. Leibe Image Source: D. Kim et al., Cytometry 35(1), 199945

Page 44: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Connected Components Labeling

• Goal: Identify distinct regions

B. LeibeSources: Shapiro & Stockman, Chandra

Binary image Connected componentslabeling

46

Page 45: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Connected Components Examples

B. LeibeSource: Pinar Duygulu47

Page 46: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Connectedness

• Which pixels are considered neighbors?

B. Leibe

4-connected 8-connected

Source: Chaitanya Chandra48

Page 47: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Sequential Connected Components

B. LeibeSlide credit: J. Neira49

Page 48: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Sequential Connected Components (2)

• Process the image from left toright, top to bottom:1.) If the next pixel to process is 1 i.) If only one of its neighbors

(top or left) is 1, copy its label.

ii.) If both are 1 and have the same label, copy it.

iii.) If they have different labels Copy the label from the left. Update the equivalence table.

iv.) Otherwise, assign a new label.

• Re-label with the smallest of equivalentlabels

B. LeibeSlide credit: J. Neira50

Page 49: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Application: Segmentation of a Liver

B. LeibeSlide credit: Li Shen51

Page 50: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Outline of Today’s Lecture

• Convert the image into binary form Thresholding

• Clean up the thresholded image Morphological operators

• Extract individual objects Connected Components Labeling

• Describe the objects Region properties

B. Leibe Image Source: D. Kim et al., Cytometry 35(1), 199952

Page 51: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Region Properties

• From the previous steps, we can obtain separated objects.

• Some useful features can be extracted once we have connected components, including Area Centroid Extremal points, bounding box Circularity Spatial moments

B. Leibe53

Page 52: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Area and Centroid

• We denote the set of pixels in a region by R• Assuming square pixels, we obtain

Area:

Centroid:

B. LeibeSource: Shapiro & Stockman

Ryx

A),(

1

Ryx

A xx),(

1

Ryx

A yy),(

1

54

Page 53: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Circularity

• Measure the deviation from a perfect circle

Circularity:

where and are the mean and vari-ance of the distance from the centroid of the shape to the boundary pixels (xk,yk).

Mean radial distance:

Variance of radial distance:

B. LeibeSource: Shapiro & Stockman

R 2R

),( yx

1

0

1 ),(),(K

kkkKR yxyx

1

0

212 ),(),(K

kRkkKR yxyx

R

RC

55

Page 54: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Invariant Descriptors

• Often, we want features independent of location, orientation, scale.

B. LeibeSlide credit: Kristen Grauman

,,, 321 aaa ,,, 321 bbb Feature spacedistance

56

Page 55: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Central Moments

• S is a subset of pixels (region).

• Central (j,k)th moment defined as:

• Invariant to translation of S.

• Interpretation: 0th central moment: area 2nd central moment: variance 3rd central moment: skewness 4th central moment: kurtosis

B. Leibe

Syx

kjjk yyxx

),(

)()(

Slide credit: Kristen Grauman57

Page 56: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Moment Invariants (“Hu Moments”)

• Normalized central moments

• From those, a set of invariant moments can be defined for object description.

• Robust to translation, rotation & scaling, but don’t expect wonders (still summary statistics).

B. Leibe

,00

pq

pq 12

qp

02201 211

202202 4)(

20321

212303 )3()3(

20321

212304 )()(

58

Page 57: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Moment Invariants

B. Leibe

2

03212

123003210321

20321

21230123012305

)()(3))(3(

)(3)())(3(

))((4

)()()(

0321123011

20321

2123002206

2

03212

123003213012

20321

21230123003217

)()(3))(3(

)(3)())(3(

59

Often better to use log10(Ái) instead of Ái directly…

Page 58: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Axis of Least Second Moment

• Invariance to orientation? Need a common alignment

Compute Eigenvectors of 2nd moment matrix (Matlab: eig(A))

B. Leibe

Axis for which the squared distance to 2D object points is minimized(maximized).

T

T

vv

vv

vv

vvVDV

2221

1211

2

1

2222

1211

0211

1120

0

0

60

Page 59: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Summary: Binary Image Processing

• Pros Fast to compute, easy to store Simple processing techniques Can be very useful for constrained scenarios

• Cons Hard to get “clean” silhouettes Noise is common in realistic scenarios Can be too coarse a representation Cannot deal with 3D changes

B. Leibe61

Slide credit: Kristen Grauman

Page 60: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

References and Further Reading

• More on morphological operators can be found in R.C. Gonzales, R.E. Woods,

Digital Image Processing.Prentice Hall, 2001

• Online tutorial and Java demos available on http://homepages.inf.ed.ac.uk/rbf/HIPR2/

B. Leibe62

Page 61: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Questions ?

B. Leibe63

Page 62: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Demo “Haribo Classification”

64B. Leibe

Page 63: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

You Can Do It At Home…

Accessing a webcam in Matlab:

function out = webcam

% uses "Image Acquisition Toolbox„

adaptorName = 'winvideo';

vidFormat = 'I420_320x240';

vidObj1= videoinput(adaptorName, 1, vidFormat);

set(vidObj1, 'ReturnedColorSpace', 'rgb');

set(vidObj1, 'FramesPerTrigger', 1);

out = vidObj1 ;

cam = webcam();

img=getsnapshot(cam);

65B. Leibe

Page 64: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Matlab Intro: Everything is a matrix

66Tutorial adapted from W. Freeman, MIT

Page 65: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Matlab Intro: Matrix Index

67Tutorial adapted from W. Freeman, MIT

Page 66: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Matlab Intro: Manipulating Matrices

68Tutorial adapted from W. Freeman, MIT

Page 67: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Matlab Intro: Manipulating Matrices

69B. LeibeTutorial adapted from W. Freeman, MIT

Page 68: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Matlab Intro: Scripts and Functions

• Scripts are m-files containing MATLAB statements

• Functions are like any other m-file, but they accept arguments

• Name the function file the same as the function name

70B. LeibeTutorial adapted from W. Freeman, MIT

Page 69: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Matlab Intro: Try to Code in Matrix Ways

71B. Leibe

Page 70: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Matlab Intro: Important Commands

• whos List variables in workspace• help Get help for any command• lookfor Search for keywords• clear/clear x Erase a variable/all variables• save Save the workspace• load Load a saved workspace• keyboard Enter debugging mode (until dbquit)

72B. Leibe

Page 71: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Morphology in Matlab

• N = hist(Y,M)• IM2 = imerode(IM,SE);• IM2 = imdilate(IM,SE);• IM2 = imclose(IM, SE);• IM2 = imopen(IM, SE);• L = bwlabel(BW,N);• STATS = regionprops(L,PROPERTIES) ;

'Area' 'Centroid' 'BoundingBox' 'Orientation‘, …

Slide credit: Kristen Grauman

Page 72: Perceptual and Sensory Augmented Computing Computer Vision WS 12/13 Computer Vision – Lecture 2 Binary Image Analysis 25.10.2012 Bastian Leibe RWTH Aachen.

Perc

ep

tual an

d S

en

sory

Au

gm

en

ted

Com

pu

tin

gC

om

pu

ter

Vis

ion

WS

12

/13

Questions ?

B. Leibe74