Roles of Appearance and Contextual Information
description
Transcript of Roles of Appearance and Contextual Information
![Page 1: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/1.jpg)
Roles of Appearance and Contextual Information
Devi Parikh, Larry Zitnick and Tsuhan Chen
![Page 2: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/2.jpg)
When?• High intra-class variance– Chairs
• Low inter-class variance– Lemon vs. tennis ball
• Occlusion
• Low resolution image– Image of a far away scene– Bad quality
[Rabinovich et al., ICCV 07]
[Bustos et.al., CSUR 2005]
![Page 3: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/3.jpg)
Popular motivations for context…
[Antonio Torralba]
![Page 4: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/4.jpg)
Popular motivations for context…
[Antonio Torralba]
![Page 5: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/5.jpg)
Popular motivations for context…
[Antonio Torralba]
![Page 6: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/6.jpg)
Popular motivations for context…
[Antonio Torralba]
![Page 7: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/7.jpg)
However…
Let’s look at it more carefully…
![Page 8: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/8.jpg)
Object appearance
![Page 9: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/9.jpg)
Appearance + Context
![Page 10: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/10.jpg)
Blind recognition
![Page 11: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/11.jpg)
Low resolution
![Page 12: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/12.jpg)
High resolution
![Page 13: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/13.jpg)
Blind recognition
![Page 14: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/14.jpg)
Appearance + Context
![Page 15: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/15.jpg)
When?
• Humans?
• Machines?
![Page 16: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/16.jpg)
Set-up
Low Resolution Appearance Machine
HumanContext
??
? ?
?
High Resolution
![Page 17: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/17.jpg)
Low resolution: Appearance
![Page 18: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/18.jpg)
Low resolution: Context
![Page 19: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/19.jpg)
Low resolution: Appearance + Context
![Page 20: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/20.jpg)
High resolution: Appearance
![Page 21: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/21.jpg)
High resolution: Context
![Page 22: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/22.jpg)
High resolution: Appearance + Context
11 subjects
2 sessions
3 scenarios
70 segments
![Page 23: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/23.jpg)
Related work
[Torralba et al., Tech Report 2007]
![Page 24: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/24.jpg)
Machines
N
jijiij
N
iii ccc
Z 1,1
,1)|P( Sc
??
? ?
?
AppearanceContext
Texture + Shape
[TextonBoost]*
Color
[GMM]
Neural Network
Co-occurrence**
Relative location
Relative scale
Inference:
Belief Propagation
*[Shotton et al., ECCV 2006] **[Rabinovich et al., ICCV 2007]
![Page 25: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/25.jpg)
Machines
Co-occurrence** Relative location Relative scale
**[Rabinovich 2007]
![Page 26: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/26.jpg)
Machines• MSRC dataset• Corel dataset
[Felzenshwalb and Huttenlocher IJCV 2004]
![Page 27: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/27.jpg)
Results
![Page 28: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/28.jpg)
Results
A: Appearance alone
C: Context alone
A+C: Appearance and context
Low High High resolution images do not benefit from
context
![Page 29: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/29.jpg)
Results
A: Appearance alone
C: Context alone
A+C: Appearance and context
Low High
Low resolution images NEED context
![Page 30: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/30.jpg)
Results
![Page 31: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/31.jpg)
Results (Machine)
Context & Appearance help each other
![Page 32: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/32.jpg)
Results (Machine)
Appearance hurts
![Page 33: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/33.jpg)
Results (Machine)
Context very weak
![Page 34: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/34.jpg)
Results (Machine)
Context makes no difference
![Page 35: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/35.jpg)
Results (Machine)
![Page 36: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/36.jpg)
Results (Machine)
Relative Location
Co-occurrence
Relative Scale
![Page 37: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/37.jpg)
Results (Machine)• Failure cases
![Page 38: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/38.jpg)
Results (Machine)
MSRC Corel
Existing (high) 75 1 81 2
Proposed (high)
91 93
Proposed (low) 83 86
1 [Yang et al., CVPR 2007] 2 [He et al., ECCV 2006]
![Page 39: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/39.jpg)
Results (Machine)
![Page 40: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/40.jpg)
Contributions
• Context is most useful when appearance information is weak
• Low resolution images are an appropriate venue for studying context
![Page 41: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/41.jpg)
Discussion• Improve appearance or context models to achieve human
performance?
• Need to improve both appearance and context models• In low resolution images, appearance information is similar for
humans and machines– Hence, appropriate venue for studying context
• Achieving human performances need not be the ultimate goal
Need to improve context models
Need to improve appearance models
![Page 42: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/42.jpg)
Follow up work (PAMI 2011)
![Page 43: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/43.jpg)
Results
A: Appearance alone
C: Context alone
A+C: Appearance and context
Low High
Machine do not leverage contextual
information as effectively as humans
Are machine missing a source of context?
![Page 44: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/44.jpg)
Different Sources of Context
![Page 45: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/45.jpg)
Different Sources of Context: None
![Page 46: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/46.jpg)
Different Sources of Context: Cooc
![Page 47: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/47.jpg)
Different Sources of Context: Rel-scale
![Page 48: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/48.jpg)
Different Sources of Context: Rel-loc
![Page 49: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/49.jpg)
Different Sources of Context: All
![Page 50: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/50.jpg)
Different Sources of Context: All
![Page 51: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/51.jpg)
Different Sources of Context: Blind
![Page 52: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/52.jpg)
Different Sources of Context: Image
![Page 53: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/53.jpg)
High Resolution Appearance
![Page 54: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/54.jpg)
PASCAL
“Natural” scenes
Bounding boxes
Common pixels
More void
![Page 55: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/55.jpg)
Results
MSRC PASCAL50556065707580859095 app
co-occrel-locrel-scaleall-explodedall-no-voidblindall
![Page 56: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/56.jpg)
Results
MSRC PASCAL50556065707580859095 app
co-occrel-locrel-scaleall-explodedall-no-voidblindall
Co-occurrence information helps in both datasets. Relative location does not help in PASCAL.
![Page 57: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/57.jpg)
MSRC Location Statistics
Building
TreeSh
eep
AeroplaneFa
ce
Bicycle Sig
nBook
RoadDog
Boat0
102030405060708090
100
BuildingGrass
TreeCow
SheepSky
AeroplaneWater
FaceCar
BicylceFlower
SignBird
BookChairRoad
CatDog
BodyBoat
![Page 58: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/58.jpg)
PASCAL Location Statistics
AerplaneBird
Bottle CarChair
Dining table
HorsePerso
nSh
eepTrain
0102030405060708090
100
AeroplaneBicycle
BirdBoat
BottleBusCarCat
ChairCow
Dining tableDog
HorseMotorbike
PersonPotted plant
SheepSofa
TrainTV / monitor
![Page 59: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/59.jpg)
Results
MSRC PASCAL50556065707580859095 app
co-occrel-locrel-scaleall-explodedall-no-voidblindall
Relative scale information does not help across the board.
![Page 60: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/60.jpg)
Results
MSRC PASCAL50556065707580859095 app
co-occrel-locrel-scaleall-explodedall-no-voidblindall
Relative scale information does not help across the board.
![Page 61: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/61.jpg)
Results
MSRC PASCAL50556065707580859095 app
co-occrel-locrel-scaleall-explodedall-no-voidblindall
Our choice of visualization does not affect performance in MSRC.
![Page 62: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/62.jpg)
Results
MSRC PASCAL50556065707580859095 app
co-occrel-locrel-scaleall-explodedall-no-voidblindall
There is information in the “void” (unlabeled) pixels!
We leverage this cue for object detectionCongcong Li, Devi Parikh and Tsuhan Chen.
Extracting Adaptive Contextual Cues from Unlabeled Regions, ICCV 2011
![Page 63: Roles of Appearance and Contextual Information](https://reader036.fdocuments.us/reader036/viewer/2022062310/568165c5550346895dd8d239/html5/thumbnails/63.jpg)
MUCH more interesting analysis and findings in the PAMI 2011 paper:
Exploring Tiny Images: The Roles of Appearance and Contextual Information for Machine and Human Object Recognition