Silhouette-based Object Phenotype Recognition using 3D Shape Priors

Yu Chen1 Tae-Kyun Kim2 Roberto Cipolla1

 University of Cambridge, Cambridge, UK1 Imperial College, London, UK2 

Problem Description

Task: To identify the phenotype class of deformable objects.

Given a gallery of canonical-posed silhouettes in different phenotype classes.

Can we find out

?

Problem DescriptionMotivation:

– Pose recognition is widely investigated;– Phenotype recognition is somehow overlooked;– Applications?

Difficulty: – Pose and camera viewpoint variations are more

dominant than the phenotype variation.

Problem Description 2D approaches hardly work in this

case.

Our strategy: make use of the 3D shape prior of deformable objects.

Shall we use a purely generative approach?

No! Too expensive to perform for a recognition task!

Solution: Two-Stage Model Main Ideas:

Discriminative + Generative Two stages:1. Hypothesising

– Discriminative;– Using random forests;

2. Shape Synthesis and Verification– Generative;– Synthesising 3D shapes

using shape priors;– Silhouette verification.

Recognition by a model selection process.

• Use 3 RFs to quickly hypothesize phenotype, pose, and camera parameters.

• Learned on synthetic silhouettes generated by the shape priors.

Parameter Hypothesizing

FA: Pose classifier

FC: Camera pose classifier

FS: Phenotype classifier

(canonical pose)

Examples of Tree Classifiers

The phenotype classifier

The pose classifier

Training RF Classifiers Random Features:

– Rectangle pairs with random sizes and locations.

– Difference of mean intensity values[Shotton et al. 09]

– Feature error compensation for phenotype classifier;

Criteria Function:– Similarity-aware diversity

index.

Shape Synthesis and Verification

Generate 3D shapes V – From candidate parameters

given by RFs.– Use GPLVM shape priors

[Chen et al.’10].

Compare the projection of V with the query silhouette Sq.

– Oriented Chamfer matching (OCM). [Stenger et al’03]

ExperimentsTesting data:

– Manually segmented silhouettes;

Current Datasets– Human jumping jack

(13 instances, 170 images);– Human walking

(16 instances, 184 images);– Shark swimming

(13 instances, 168 images).Phenotype Categorisation

Comparative Approaches:

Learn a single RF phenotype classifier; Histogram of Shape Context (HoSC)

– [Agarwal and Triggs, 2006] Inner-Distance Shape Context (IDSC)

– [Ling and Jacob, 2007] 2D Oriented Chamfer matching (OCM)

– [Stenger et al. 2006] Mixture of Experts for the shape reconstruction

– [Sigal et al. 2007].– Modified into a recognition algorithm

Comparative Approaches: Internal comparisons:

– Proposed method with both feature error modelling and similarity-aware criteria function (G+D);

– Proposed method w.o feature error modelling (G+D–E);

– Proposed method w.o similarity-aware criteria function (G+D–S)

Using standard diversity index instead.

Recognition Performance Cross-validation by splitting the dataset instances. 5 phenotype categories for every test. Selecting one instance from each category.

Recognition Performance How the parameters of RFs affect the

performance?– Max Tree Depth dmax

– Tree Number NT

Qualitative Results of SVR Left: Input image/silhouette; Centre: Using RF-hypothesizes;Right: Using the optimisation-based approach.

Qualitative Results of SVR

Take-Home MessagesPhenotype recognition is difficult but still

possible;

Combing discriminative and generative cues can greatly speed up the inference;

A divide-and-conquer strategy can help improve the recognition rate.

Future WorkExplore the application on more

complicated poses and more categories.– E.g. Boxing, gardening, other sports, etc.

Data collection;

Automate the silhouette extraction.– E.g. Kinect.

The End

Questions?