1 Ellen L. Walker

Recognizing Objects in Computer Images

Ellen L. WalkerMathematical Sciences Dept

Hiram CollegeHiram, OH 44234

walkerel@hiram.eduhttp://hirame.hiram.edu/~walkerel

2 Ellen L. Walker

Why Recognize Objects?

“Put the clothing on the bed”

“Turn left at the apartment building”

? ??

?

3 Ellen L. Walker

Steps in Object Recognition

BLOCK

CYLINDER

MODELS

IMAGE FEATURES GROUPS

INTERPRETATION

segmentation grouping

matchingmatching

CYLINDER

BLOCK

4 Ellen L. Walker

Formation of a 2D Image

camera

image plane

image (2D on the image plane)

scene (3D)

5 Ellen L. Walker

Images are Composed of Pixels

Image has 138 x 255 pixels (35190 total)

Pixel values range from 0 (black) to 255 (white)

6 Ellen L. Walker

Edge Detection

• Use a technique called convolution to detect edges

• The 3x3 convolution of a pixel is the sum of the products of the pixel and its 8 neighbors with a 3x3 mask

• The convolution of the shaded cell with the mask isAbs(4x-1 + 3x0 + 2x1+ 4x-2 + 3x0 + 2x2 + 4x-1 + 3x0 + 2x1)

= 8

4 3 2 2

4 3 2 2

4 3 2 2

4 3 2 2

-1 0 12 0 2

-1 0 1

Image

Mask

7 Ellen L. Walker

Edge Detection with Convolution

0 0 0 0 0 0 0 0

0 0 1 1 1 1 1 1

0 1 4 3 2 2 2 2

0 1 4 3 2 2 3 2

0 1 4 3 2 2 3 3

0 1 4 3 2 2 3 2

0 1 4 3 2 2 3 2

0 0 0 0 0 0 0 0

0 6 12 12 9 8 8 0

0 5 9 8 5 5 6 0

0 0 0 0 0 1 3 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 1 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 6 4 2 1 0 0 0

0 13 7 6 3 1 0 0

0 16 8 8 4 3 1 0

0 16 8 8 4 4 2 0

0 16 8 8 4 4 1 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 12 16 14 10 8 8 0

0 18 16 14 8 6 6 0

0 16 8 8 4 4 4 0

0 16 8 8 4 4 2 0

0 16 8 8 4 4 2 0

0 0 0 0 0 0 0 0

horiz.conv.

vert.conv.

bothconvolutions

-1 0 12 0 2

-1 0 1

-1 2 -10 0 01 2 1

8 Ellen L. Walker

Edges from Test Image

After convolution and thresholding

This is still an image …

next find endpoints of straight segments

9 Ellen L. Walker

Fitting Line Segments to Edge Pixels

For each path of connected edge pixels:

draw a segment between the endpoints of the path

find the point farthest from the line

if it’s too far, then make that point an endpoint and recurse

10 Ellen L. Walker

Segments from Test Image

Circles denote endpoints

Note “broken” segments

11 Ellen L. Walker

Grouping Collinear Line Segments

Line segments are collinear if:

angle between them is small enough ()

distance between them is small enough (PD)

“gap” between them is small enough (G)

B

PD

G

collinearity

exact

not at all0 180

12 Ellen L. Walker

Grouped Segments from Test Image

Long vertical and horizontal segments were found

13 Ellen L. Walker

Higher Levels of Grouping

Junctions

Curves & Polygons (no T-Junctions allowed)L-Junction L-Junction

Both interior angles must be inthe same direction

IPDAB

A

B

IPDE = 0AB

L-Junction

A

B

IPDEAB

IPDAB

T-Junction

14 Ellen L. Walker

Groups from the Test Image?

After grouping curves and polygons, and combining polygons that share an edge and no T-junctions

Finally, we’re ready to recognize!

15 Ellen L. Walker

Types of Models

Exact Models — Describe single objects

e.g. CAD models for factory parts

the object is needed to construct the model!

Generic Models —Describe classes of objects

Exact models with parameters (e.g length, width, radius)

Structural constraints describe how the object is constructed

Functional constraints describe how the object is used

16 Ellen L. Walker

Matching Method Depends on Model Level

Image

when objects are hard to describe (e.g. faces)

match by correlation (like convolution)

Features, Groups

good for geometric objects

match by finding correspondences between image and model features (or groups)

Complete Objects

good for classes of objects

match by applying constraints from model to objects derived from the image

17 Ellen L. Walker

Sample Object Model (Box)

Constraints Adjacent faces are perpendicular (12 constraints) Adjacent edges are perpendicular (8 constraints) Opposite faces are parallel (3 constraints) Opposite edges are parallel (6 constraints)

top

left

front

top left back

top left

left front

left back

bottom left back

bottom left front bottom right front

right front

top right front

top right back

top right

top back

bottom front

bottom left

18 Ellen L. Walker

Finding Correspondences

Search the interpretation tree

each node is a correspondence

each path to a leaf is an interpretation

“prune” the tree based on constraints

B2 B6 B8 B1 B7 B3

A1 A2 A3 A4 A5 A6

…

C3 C7

D8 D4

AB

C

DE

F

Image Corners

Box Model

1 2

3

45

6 7

8

19 Ellen L. Walker

Structural Constraints

Buildings — structural constraints

Buildings have rectangular walls and flat horizontal roofs

Each outer wall of a building intersects exactly two other walls, forming a closed ring

Each edge of the roof is the perpendicular intersection of the roof and a wall

20 Ellen L. Walker

Functional Constraints

Beds — functional constraints

A bed has a surface to sleep on that is relatively flat.

A bed has physical means of supporting the sleeping surface at a comfortable height.

A bed has clearance above and to at least one side of the sleeping surface.

21 Ellen L. Walker

Considering Occlusion in the Interpretation

Include names & parameters of objects

Include hypotheses for hidden parts when feasible

Specify only what is guaranteed!

Image Interpretation

22 Ellen L. Walker

Toward Real World Solutions

Increasingly complex models

Large number of alternatives to choose from

Uncertainty at all levels

Recognizing partially-visible objects

Recognition must be fast enough

Generating models automatically (learning!)