Recognizing Surfaces using Three-Dimensional Textons

Thomas Leung and Jitendra Malik

Computer Science Division

University of California at Berkeley

ICCV '99, Corfu, Greece

Traditional Texture Recognition

• Assume texture to be planar;

• Assume constant illumination and viewing directions;

• Ignore 3D nature of natural materials, i.e. no shadowing, occlusions, etc…

• E.g. Puzicha et al, Jain et al, Greenspan et al, etc….

ICCV '99, Corfu, Greece

Example Natural Materials

Terrycloth Rough Plastic Plaster-b

Sponge Rug-a Painted Spheres

Columbia-Utrecht Database (http://www.cs.columbia.edu/CAVE)

ICCV '99, Corfu, Greece

Materials under different illumination and viewing directions

Differentilluminationand viewingdirections

Plaster-a CrumpledPaper

Concrete Plaster-b(zoomed)

ICCV '99, Corfu, Greece

TaskFelt?Polyester?Terrycloth?Rough Plaster?Leather?Plaster?Concrete?Crumpled Paper?Sponge?Limestone?Brick?

?

?

ICCV '99, Corfu, Greece

3D Texture Models

• Analytical models: – Simple parametric surface height distribution;– compute image statistics;– Dana & Nayar 97, 98, 99; Koenderink et al 96, 98;

Leung & Malik 97; Chantler et al 97, 98;

• Computer graphics models:– bump maps, displacement maps, point clouds, etc. – difficult to obtain for natural materials;

ICCV '99, Corfu, Greece

Problem Formulation

Image Database

Recognize

new sample

of different

light/view

Task

ICCV '99, Corfu, Greece

Main Idea

• Natural materials are made up of local features (geometric and photometric);

• There exists a universal set of local features for all materials;

• How these local features change appearance with different illumination and viewing directions determine how the materials look.

ICCV '99, Corfu, Greece

Outline• Learning the universal vocabulary of local

structures

• Material models

• Results

ICCV '99, Corfu, Greece

Outline• Learning the universal vocabulary of local

structures

– Introduce 2D textons for planar texture;

– Extend to 3D textons for natural materials;

• Material models

• Results

ICCV '99, Corfu, Greece

2D Textons

• Julesz suggests a universal vocabulary for such features --- textons [Julesz 81];

• crossings, line-ends, junctions, etc…

• Define textons for real images.

ICCV '99, Corfu, Greece

2D Textons• Goal: find canonical local features in a texture;

1) Filter image with linear filters:

2) Vector quantization on filter outputs;

3) Quantization centers are the textons.

• Spatial distribution of textons defines the texture;

ICCV '99, Corfu, Greece

2D Textons (cont’d)

ICCV '99, Corfu, Greece

3D Textons

• Consider textures with 3D features, e.g. bumps, grooves, ridges, etc…

• Want textons to capture local 3D geometric and photometric features;

• One image is ambiguous: different features can look the same under certain illumination and viewing conditions;

• More images will discriminate between the different cases.

ICCV '99, Corfu, Greece

Learning 3D Textons

Rough Plastic Concrete

Light/view 1

Light/view 2

Light/view N

3D textons

Texton 1

Texton K

Texton 2

ICCV '99, Corfu, Greece

Algorithm for Learning Vocabulary

• Register all 20 images for each material;

• Filter images with filter bank of 48 kernels;

• Concatenate filter responses of the 20 images;

• Each pixel becomes a 960 (20x48) dimensional feature vector;

• Apply K-means to the feature vectors of all materials together;

• Resulting centers are the 3D textons.

ICCV '99, Corfu, Greece

Algorithm for 3D Textons

ICCV '99, Corfu, Greece

Universal 3D Texton Vocabulary

• Columbia-Utrecht Database (60 materials, each with 205 images)

• Vocabulary of textons learned from 20 training materials;

• Use 20 different light/view images for each material.

ICCV '99, Corfu, Greece

Examples of 3D Textons

Texton 1

Texton 2

Texton 3

Texton 4

Texton 5

Texton 6

Texton 7

Different illuminationand viewing directions

ICCV '99, Corfu, Greece

Quantization Errors

• Reconstruct images after quantization;• SSD error within 5%.

ICCV '99, Corfu, Greece

Outline• Learning the universal vocabulary of local

structures;

• Material models;

– Image to texton representation;

– Material representation using textons;

• Results.

ICCV '99, Corfu, Greece

Texton Labeling

• Each pixel labeled to texton i (1 to K) which is most similar in appearance;

• Similarity measured by the Euclidean distance between the filter responses;

ICCV '99, Corfu, Greece

Material Representation

• Each material is now represented as a spatial arrangement of symbols from the texton vocabulary;

• Recognition --- ignore spatial arrangement, use histogram (K=100);

ICCV '99, Corfu, Greece

Histogram Models for Recognition

Terrycloth

Rough Plastic

Pebbles

Plaster-b

ICCV '99, Corfu, Greece

Similarity of materials

• Similarity between histograms measured using chi-square difference:

N

n nhnh

nhnhhh

1 21

221

212

)()(

))()((),(

ICCV '99, Corfu, Greece

Similarity Matrix

j) sample , i (materialSimilarity ije

Plaster-a Plaster-b

AluminumFoil

Cork

ICCV '99, Corfu, Greece

Outline• Learning the universal vocabulary of local

structures

• Material models

• Results– Material recognition from single image;– Synthesis of novel images.

ICCV '99, Corfu, Greece

Recognition from Single Image• 4 images to build histogram for model;

• 1 image of novel illumination and/or viewing directions to be recognized;

Image Database

?

Novel image

ICCV '99, Corfu, Greece

Novel Image from Material i?

• Build texton histogram for novel image.

• Compare with texton histogram for material i.

• However, texton labeling from 1 image is difficult, because in 1 light/view, several textons may have same appearance.

• Each pixel has N possible texton labels;

• Need to find the labeling that maximizes Similarity(novel image, material i)

ICCV '99, Corfu, Greece

Markov chain Monte Carlo for finding labeling

• Randomly label each pixel to one of N possibilities. Call this the initial state x(t),t=0

• Compute P(x(t)|material i);

• Obtain x’ by randomly changing M labels of x(t);

• Compute P(x’|material i);

• Compute

• If , the x’ is accepted, otherwise, accept with probability .

))((

)'(

txP

xP

1

ICCV '99, Corfu, Greece

P(detection) vs P(false alarm)

ICCV '99, Corfu, Greece

Synthesis of images with novel illumination and viewing directions

Map eachpixel totextons

Textons tellus how

appearancechanges

ICCV '99, Corfu, Greece

Synthesis of novel light/view images

• Keep exact spatial arrangement of textons

ICCV '99, Corfu, Greece

Synthesis Results

TextureMapping

GroundTruth

3D TextonModel

TextureMapping

GroundTruth

3D TextonModel

Plaster-a Concrete

ICCV '99, Corfu, Greece

Synthesis Results

TextureMapping

GroundTruth

3D TextonModel

TextureMapping

GroundTruth

3D TextonModel

Crumpled paper Plaster-b (zoomed)

ICCV '99, Corfu, Greece

Synthesis Results

TextureMapping

GroundTruth

3D TextonModel

TextureMapping

GroundTruth

3D TextonModel

Rough Plastic Sponge

ICCV '99, Corfu, Greece

Similarity to Appearance-based Object Recognition

• Object Recognition: objects are represented by a collection of images under different illumination and viewing conditions;

• Material Recognition: materials are represented by 3D textons, each of which is represented by the appearances under different illumination and viewing conditions.

ICCV '99, Corfu, Greece

Conclusions

• Model natural materials through images;

• Learn a universal vocabulary of 3D textons;

• Use the vocabulary to

– recognize materials from a single image of novel illumination and viewing directions;

– synthesize materials at novel illumination and viewing directions.