Transfer Learning of Object Classes: From Cartoons to Photographs
NIPS WorkshopInductive Transfer: 10 Years Later
Geremy HeitzGal Elidan
Daphne Koller
December 9th, 2005
Localization vs. Recognition
Traditional question:
“Is there an object of type X in this image?”
Airplane? NO
Human? YES
Dog? YESOur question:
“Where in this image is the object of type X?”
MAN
DOG
The man is walking the dog
Outline
Landmark-based shape model Localization as inference Transfer learning from cartoon
drawings Results
Shape Model
Set of landmarks Piecewise-linear contour between
neighbors Features of individual landmarks Features of pairs of landmarks
tail
nose
Outline
Landmark-based shape model Localization as inference Transfer learning from cartoon
drawings Results
Localization
Are local cues enough?
Need to jointly consider all cues (features)
“Correct” pixel is often not the best match!Markov Random Field
• Potentials = Functions of local and global features
Registration = Most Likely Assignment
Lnose Ltail
LunderLcockpit
Outline
Landmark-based shape model Localization as inference Transfer learning from cartoon
drawings Results
Bootstrap from simple instanceswhere outlining is easy = cartoons / drawings
Learning ChallengeHand Label Hidden Variables
Costly, and time-consuming
Where to start?Local optima problem
no confusing background
outline (shape) is easily recovered using snake
??
???
Phase I: Learning from Cartoons
Extract high resolution contour using snake Create shape-based model from training
contours Pairwise merging of models Selection of landmarks
Registration PyramidFinal Shape
Model
Training Set Selection
high score
low score
Phase II: Learning from Images
Correspond initial model to training images Select best correspondences as training instances Learn final shape- and appearance-based model
Cartoon PhaseModel
Natural ImageModel
Transfer
Outline
Landmark-based shape model Localization as inference Transfer learning from cartoon
drawings Results
Localization Results
0.84 0.75 0.84 0.72 0.18
0.81 0.81 0.66 0.77 0.40
sampletrainingcartoons
sampleregistration
Transfer of Object Shape
Transfer of shape speeds up learning
Benefit of shape
transfer
0 2 4 6 8 10
0
0.1
0.2
0.3
0.4
0.5
0.6
# images in phase II
Ave
rag
e o
verl
ap
transfer
no transfer
Learning Appearance
No Appearance
FG/BG Appearance
0 2 4 6 8 100.46
0.48
0.5
0.52
0.54
0.56
0.58
0.6
0.62
0.64
Ave
rag
e o
verl
ap
Shape template
shape + appearance
# images in phase II
Training Instance Selection
AUTO PICKED
0 2 4 6 8 10 120.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
AUTO
PICKED
HAND
Ave
rag
e o
verl
ap
# images in phase II
Summary and Future Work Flexible probabilistic shape model Effective registration to images Transfer
Shape from cartoons Appearance from real images
Develop a better appearance model Investigate self-training issues Transfer from one class to another
Cartoon vs. Hand Segmentation
0 1 2 3 4 5
Number of Training Instances
0.1
0.3
0.5
0.7
0.9
Mea
n O
verla
p S
core
Learned from Drawings
Hand Constructed
Human Inter-Observer
cartoon handsegmented
Learning shape from cartoons is competitive with hand segmentation!
Top Related