8/4/2019 Scene Understanding - Spatial Envelope

1/22

Modeling the Shape of aScene: Seeing the trees as a

forest

Scene Understanding Seminar20090203

8/4/2019 Scene Understanding - Spatial Envelope

2/22

Scene recognition

Images

objects: 1-2 meters

environments: > 5 meters

This paper

Scene representation

Scene statistics

8/4/2019 Scene Understanding - Spatial Envelope

3/22

Scene recognition

Scenes vs. collections of objects

Object information may be ignored Fast categorization

Low spatial frequencies

Change blindness, inattention blindness

Scene category provides context for

images Need holistic representation of a scene

8/4/2019 Scene Understanding - Spatial Envelope

4/22

Holistic scene representation

Finding a low-dimensional scene

space

Clustering by humans Split images into

groups

ignore objects,

categories

8/4/2019 Scene Understanding - Spatial Envelope

5/22

Spatial envelope properties

Naturalness natural vs. man-made environments

8/4/2019 Scene Understanding - Spatial Envelope

6/22

Spatial envelope properties

Openness decreases as number of boundary elements

increases

8/4/2019 Scene Understanding - Spatial Envelope

7/22

Spatial envelope properties

Roughness size of elements at each spatial scale, related to

fractal dimension

8/4/2019 Scene Understanding - Spatial Envelope

8/22

Spatial envelope properties

Expansion (man-made environments) depth gradient of the space

8/4/2019 Scene Understanding - Spatial Envelope

9/22

Spatial envelope properties

Ruggedness (natural environments) deviation of ground relative to horizon

8/4/2019 Scene Understanding - Spatial Envelope

10/22

Scene statistics

DFT (energy spectrum)

throw out phase function (represents localproperties)

Windowed DFT (spectrogram)

Coarse local information

8x8 grid for these results

8/4/2019 Scene Understanding - Spatial Envelope

11/22

Scene statistics

Dimensionality reduction via PCA

8/4/2019 Scene Understanding - Spatial Envelope

12/22

Scene classification fromstatistics

Different scene categories havedifferent spectral signatures

Amplitude captures roughness

Orientation captures dominant edges

8/4/2019 Scene Understanding - Spatial Envelope

13/22

Scene classification fromstatistics

Open environments have non-stationarysecond-order statistics support surfaces

Closed environments exhibit stationarysecond-order statistics

a) man-made open environmentsb) urban vertically structuredenvironmentsc) perspective views of streets

d) far view of city-center buildingse) close-up views of urbanstructuresf) natural open environmentsg) natural closed environmentsh) mountainous landscapesi) enclosed forests

j) close-up views of non-texturedscenes

8/4/2019 Scene Understanding - Spatial Envelope

14/22

Learning the spatial envelope

Use linear regression to learn

DST (discriminant spectral template)

WDST (windowed discriminant spectraltemplate)

Relate spectral representation to eachspatial envelope feature

8/4/2019 Scene Understanding - Spatial Envelope

15/22

Learning the spatial envelope

Primacy of Man-made vs. Naturaldistinction

Linear Discriminant analysis

93.5% correct classification

Role of spatial information

WDST not much better than DST

Loschky, et al., scene inversion

8/4/2019 Scene Understanding - Spatial Envelope

16/22

Learning the spatial envelope

Other properties calculated separatelyfor natural, man-made environments

8/4/2019 Scene Understanding - Spatial Envelope

17/22

Spatial envelope andcategories

Choose random scene and sevenneighbors in scene space

If >= 4 neighbors have same semanticcategory, image is correctly

recognized

WDST: 92%

DST: 86%

8/4/2019 Scene Understanding - Spatial Envelope

18/22

Applications

Depth Estimation (Torralba & Oliva)

8/4/2019 Scene Understanding - Spatial Envelope

19/22

Applications

Image Classification (Bosch &Zisserman)

Gist features used in object descriptions

8/4/2019 Scene Understanding - Spatial Envelope

20/22

Applications

Scene Completion (Hayes & Efros)

Use gist to find find possible matches

8/4/2019 Scene Understanding - Spatial Envelope

21/22

Applications

Robot Navigation (Siagian & Itti)

Different scene model used to recognizefamiliar environments

8/4/2019 Scene Understanding - Spatial Envelope

22/22

Questions?