1 Object-Oriented Database Languages Object Description Language Object Query Language.
EE 576 - Object - Scene...
Transcript of EE 576 - Object - Scene...
OutlineObject – Scene Analysis
EE 576 - Object - Scene Representation
H.I. Bozma
Electric Electronic Engineering
Bogazici University
May 22, 2013
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
Object – Scene AnalysisIntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Objects vs Scenes
◮ Objects – Single meaningful entities
◮ Scenes – Collection of objects
◮ Learning ⇔ Representation
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Problem
◮ Representation
◮ Scale invariance, occlusion
◮ Learning: A priori knowledge is essential
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Representation
◮ Levels of abstraction◮ Specific objects◮ Generic objects◮ Functional representations
◮ Properties of objects: Shape, color, texture, function
◮ Shape representation: Collection of points, boundaries,surfaces, volumes, collection of parts
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Complexity
◮ Object type◮ Type – Monocular, stereo, range◮ Nature – Natural, artificial
◮ Similar objects◮ Complex shapes◮ Occlusion◮ Illumination
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
General Approach
◮ Choose a representation model
◮ Find instances of known models in the scene
◮ Estimate their pose
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Basic Models
◮ 2D Features◮ Color◮ Shape - Area, moments, Fourier coefficients◮ Filter responses◮ Histograms◮ Corners, SIFT, SURF
◮ 3D Features◮ Geons (or geometric ions) - Simple volumes such as cubes,
spheres, cylinders, and wedges.
◮ Features and the relationships between them
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Visual Filters
◮ Robot at x ∈ X , ∀f ∈ F ,◮ Visual filters V with |V| = Nv (Nv = 56 used)
◮ Two color surfaces (a,b from CIELab)H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Difficulties
Projection and occlusion – create difficulties.Objects 3-D – images 2-D.Projected shapes – Depend on the viewpoint.Objects occlude each other → Only parts of objects visible →Problems in segmentation.Perceptual grouping
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Perceptual Grouping
Organization of image features into higher level feature
◮ Descriptive – Make geometrical structure explicit
◮ Hierarchical
◮ Invariant – if possible
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Perceptual grouping
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Complexity
◮ The number of models in the database
◮ Complexity of representation
◮ Complexity of matching
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Representation and Recognition Paradigm
◮ Could the image be a projection of the model?
◮ Match part of a scene with a model → Search for best fit◮ Estimate object pose◮ Project model and verify
◮ What types of features to match and how?◮ Feature matching◮ Structured description matching
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Matching Models
◮ Template matching◮ Correlate a template with the image - Invariant to translation,
sensitive to rotation, scaling and viewpoint◮ Using other metrics
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Feature Matching
◮ Match image and model features → Search for best fit
◮ Check for consistency using constraints that depend onrepresentation
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Feature Matching
Figure : Feature matchingH.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Representation By Parts
◮ One approach to structural shape representation is graphrepresentation
◮ A graph consists of two entities: nodes P and edges E (P)between nodes.
◮ Properties of nodes and edges in the model database.
Some examples are:
◮ Line drawings consisting of vertices (nodes) and lines (edges)
◮ Surfaces (nodes) and their relationships (edges)
◮ Complex shape represented by parts (nodes) and therelationships (edges)
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Recognition by Structure Matching
◮ Here, the model is based on a graph that represents theobjects’ parts and their relationships.
◮ Given an incoming image, a similar model is constructed andmatched with those in the model database.
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Graph Matching
Given G = (P ,E (P),R(P)) and G ′ = (P ′,E ′(P ′),R ′(P ′)
◮ Two nodes n ∈ P and n′ ∈ P ′ form an assignment ifR(n) = R ′(n′)
◮ Two assignments (n1, n′1) and (n2, n
′2) are compatible if
E (n1, n2) = E (n′1, n′2)
◮ Isomorphism: Compatibility for all nodes
◮ Isomorphism: Compabilility for a subset of nodes
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Search Tree: Breadth-First
Figure : Breath-first search of a match tree.H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Graph Isomorphism
Finding an isomorphism is NP-hard:
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Graph Isomorphism – Ex 2
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Aspect Graphs
Aspect graph – A series of two-dimensional views (nodes) of aknown object that are related to each other (edges) via 3Dtransformations.
◮ Generate the aspect graph
◮ Maintain them in some convenient representation in storage,
◮ Match one or more two-dimensional views of an unknownobject against the stored views of the known object,
Recognition problem → Transformed to a series of two-dimensionalmatching problems.
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Varying viewpoints
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Varying Viewpoints
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Aspect Graph
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Problem
Scene analyis (scene understanding) is the ability to make usefuldescriptions of the given scene.
◮ Identify objects and relationships between them
◮ Locate a certain object
◮ Make measurements for inspection
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Scene Representation
◮ Identify objects and relationships between them
◮ Locate a certain object
◮ Make measurements for inspection
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Scene Representation
◮ Bottom-up approaches: First find objects and then put themtogether
◮ Objects ⇒ A graph of objects
◮ Top-down – Holistic approaches◮ Image histogram◮ Bag-of-words◮ Spatial envelope◮ Hull census transform (HCT)◮ Bubble surfaces and descriptors
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Problems in Scene Analysis
◮ Representation
◮ Occlusion
◮ Image complexity
◮ Scene complexity
◮ A priori knowledge is essential
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Bag-of-Words
◮ Image representation - A collection of words, disregardinggrammar and even word order.
◮ Words – Obtain many samples and then cluster to get asmaller set
◮ Frequency of occurrence of each word - Used as a feature fortraining a classifier
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Spatial Envelope
naturalnessopennessroughnessexpansionruggednessSpectral Spatial domain
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Hull Census Transform (HCT)
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Bubble Space
Robot base
x = [c α]T
Base space X = R2 × S1
Pan-tilt movement:
f = [f1f2]T
Camera state space F ⊂ S2
Bubble space B = X × FRobot’s base + its viewing directions
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Bubble Surfaces
◮ Bubble surface Bi (x , t) : Im(h(x))× R≥0 → R≥0.
Bi (x , t) =
{[
f
ρi (b, t+)
]
| ∀f ∈ F and b = [x f ]T}
ρi : B × R≥0 → R≥0 - Encodes i th visual feature response
◮ Initialization: S2 sphere with radius ρ0 ∈ R≥0 – ρi (b, 0) = ρ0
◮
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Problems in Scene Analysis
◮ Representation
◮ Occlusion
◮ Image complexity
◮ Scene complexity
◮ A priori knowledge is essential
H.I. Bozma EE 576 - Object - Scene Representation
OutlineObject – Scene Analysis
IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding
Complexity
◮ Images are complex◮ Nature – Monocular, stereo, range◮ Quality – Indoors, outdoors,
◮ Scenes are complex◮ Illumination
H.I. Bozma EE 576 - Object - Scene Representation