Li- Jia Li, Richard Socher , Li Fei-Fei

Towards Total Scene Understanding:Classification, Annotation and

Segmentation in an Automatic Framework

Li-Jia Li, Richard Socher, Li Fei-Fei

City Travel

Pagoda

SunriseSunshine

City Travel

Pagoda

SunriseSunshine

Weber et al 00Fergus et al 03Felzenswalb et al 04Fei-Fei et al 05Sivic et al 05Bosch et al 06Oliva et al 01Lazebnik et al 06

Shi et al 00Felzenszwalb et al04Sali et al 99Winn et al 05Kumar et al 05Cao et al 07Russell et al 06Todorovic et al 06

Duygulu et al 02

Barnard et al 03

Blei et al 03Gupta et al 08

Alipr Li et al 03Sudderth et al 05

Segmentation

Classification

AnnotationRemark: Approaches in yellow will be used to compare withour model in later Experiments.

City Travel

Pagoda

SunriseSunshine

Weber et al 00Fergus et al 03Felzenswalb et al 04Fei-Fei et al 05Sivic et al 05Bosch et al 06Oliva et al 01Lazebnik et al 06

Shi et al 00Felzenszwalb et al04Sali et al 99Winn et al 05Kumar et al 05Cao et al 07Russell et al 06Todorovic et al 06

Duygulu et al 02

Barnard et al 03

Blei et al 03Gupta et al 08

Alipr Li et al 03Sudderth et al 05

Segmentation

Classification

Annotation

UTotal Scene Understandi

Application

Classification Annotation Segmentation

Mutually beneficial!

AthleteHorseGrassTreesSkySaddle

HorseHorse

HorseHorse

SkyTree

Athleteclass: Polo

HorseHorse

Related Work:

Tu et al 03

AnnotationSegmentation

HorseHorse

SkyTree

GrassHorse

Athlete

Li & Fei-Fei 07

AnnotationClassification

GrassHorse

AthleteHorse

HorseHorse

Class: Polo

ClassificationSegmentation

Heitz et al 08

Class: Polo

Learning

Recognition & Experiment

Outline

Classification

Annotation Segmentation

Visual

Joint distribution of random variable Visual Component

Text Component.

Visual

Color LocationTexture Shape

Text Component.

Visual

Visual

Text Component.

Visual

XAr ZNr Nt “Connector variable”

Text Component.

Visual

S AthleteHorseGrassTreesSkySaddle

VisibleNot visible

“Switch variable”

HorseHorse

Athlete

Visual

S AthleteHorseGrassTreesSkySaddle

VisibleNot visible

“Switch variable”

Visual Text C

TLearning

Outline

Learning

Exact Inference is Intractable !

Relationship of the random variables

Visual

Text C

Relationship of the random variables

Visual

Text C

Top-down force

Bottom-up force from visual information

Bottom-up force from text information

Collapsed Gibbs Sampling

(R. Neal, 2000)

Scene/Event imagesfrom the Internet

There is no object-text correspondence…

AthleteHorseGrassTree

Saddle

Our model builds the correspondence…

Saddle

AthleteHorseGrassBall

However, a big obstacle is: many objects always co-occur together

XAr Nr Z

One solution: some good initialization of O

Athlete

Initializing O: obtain internet images for each O Object images

Scene/Event images

XAr Nr Z

Any object

detection&

segmentation

Algorithm

Initializing O: train an object detector for each OObject imagesEvent/Scene images

Scene/Event images

…Black box

object detection& segmentation

Black box object detection& segmentation

XAr Nr Z

Initialize O in the scene image by the trained object detectors

Object imagesEvent/Scene images

Any object

detection&

segmentation

Algorithm

Scene/Event images

…Black box

object detection& segmentation

XAr Nr Z

Initialize O in the scene image by the trained object detectors

Cao & Fei-Fei, 2007

Our Model

Object imagesEvent/Scene images

XAr Nr Z

Auto-semi-supervised learning: Small # of initialized images + Large # of uninitialized images

Our Model + AthleteHorseGrassTree

SaddleWind

Small # of initialized images

AthleteRockGrassTree

SkyRope

AthleteSnow

TreeSky

SnowboardLarge # of uninitialized images

Scene/Event images

SaddleWind

SkyRope

AthleteSnow

TreeSky

Snowboard

Large # of uninitialized images

Visual Text C

Learning Model

Recognition & Experiment• Dataset• Learned Model• Results

OutlineSmall # of automatically initialized images

Badminton

Croquet

8 Event/Scene Classes

Remark: Tags are not used during testing

Rockclimbing

Rowing

Sailing

Snowboarding

8 Event/Scene Classes

Learned model: O

Athlete

Learned model: R

Learned model: S

8 way classification: 54%

Alipr: Li et al 03 Corr LDA: Blei et al 03

Effect of top-down class context

Model w/o top-down class Full Model

SaddleWind

SkyRope

AthleteSnow

TreeSky

Snowboard

Large # of uninitialized images

Small # of automatically initialized images

Visual Text C

AthleteTree

Mountain

Rock Class: Rock

climbingAthleteMountainTreeRockSkyAscent

Athlete

Tree sailboat

Class: SailingAthleteSailboatTreeWaterSkyWind

Learning Model

AthleteSnowboard

Class: Snowboarding

AthleteSnowboardTreeSnowSkyPowder

ThankProf. Silvio Savarese , Juan Carlos Niebles, Chong Wang, Barry Chai, Min Sun, Bangpeng Yao, Hao Su, Jia Deng, anonymous reviewers

And You

Li- Jia Li, Richard Socher , Li Fei-Fei

Documents

Transcript of Li- Jia Li, Richard Socher , Li Fei-Fei

Socher & Manning's Deep Learning for NLP tutorialnlp.stanford.edu/courses/NAACL2013/NAACL2013-Socher...Socher & Manning's Deep Learning for NLP tutorial

Fei-Fei Li & Justin Johnson & Serena Yeungcs231n.stanford.edu/slides/2019/cs231n_2019_lecture12.pdfFei-Fei Li & Justin Johnson & Serena Yeung Lecture 12 - 2 May 14, 2019 Administrative

FOIL it! Find One mismatch between Image and Language ......by using ‘neuraltalk’ (Karpathy and Fei-Fei, 2015) loss 11 Andrej Karpathy and Fei-Fei Li “Deep Visual-Semantic Alignments

Motion illusion, rotating snakes. Slide credit Fei Fei Li.

Fei-Fei Li & Andrej Karpathy & Justin Johnson …cs231n.stanford.edu/slides/2016/winter1516_lecture14.pdfFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 - 7 29 Feb 2016 Dense

Lecture’7:’’ Finding’Features’(part2/2)’vision.stanford.edu/teaching/cs131_fall1415/lectures/lecture7_DoG...Lecture 7 - !!! Fei-Fei Li! Aquickreview • Local’invariantfeatures’

Object-centric spatial pooling for image classification Olga Russakovsky, Yuanqing Lin, Kai Yu, Li Fei-Fei ECCV 2012.

Towards Total Scene Understanding: Classification, Annotation and Segmentation in an Automatic Framework Li-Jia Li, Richard Socher, Li Fei- Fei 1.

Fei-Fei Li & Justin Johnson & Serena Yeung

arXiv:1712.00559v2 [cs.CV] 24 Mar 2018 · Progressive Neural Architecture Search Chenxi Liu1;, Barret Zoph2, Maxim Neumann3, Jonathon Shlens2, Wei Hua3, Li-Jia Li 3, Li Fei-Fei;4,

ImageNet: A Large-Scale Hierarchical Image Database Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei Dept. of Computer Science, Princeton.

Lecture’12:’Clustering’and’ Segmentaon’ - Stanford …vision.stanford.edu/teaching/cs131_fall1314_nope/...Lecture 12 - !!! Fei-Fei Li! Whatwe’will’learn’today’ •

Object Recognition: History and Overview Slides adapted from Fei-Fei Li, Rob Fergus, Antonio Torralba, and Jean Ponce.

Cos 429: Face Detection (Part 2) Viola-Jones and AdaBoost Guest Instructor: Andras Ferencz (Your Regular Instructor: Fei-Fei Li) Thanks to Fei-Fei Li,

Lecture 12 - Stanford Universityvision.stanford.edu/.../cs231n/slides/2016/winter1516_lecture12.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 12 ... Fei-Fei Li & Andrej

Bangpeng Yao Li Fei-Fei Computer Science Department, Stanford University, USA.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 4 ...cs231n.stanford.edu/slides/2018/cs231n_2018_lecture04.pdf · Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 4 - April 12,

Lecture’12:’Clustering’and’ Segmentaon’ - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1415/lectures/... · 2014-10-28 · Lecture 12 - !!! Fei-Fei Li!

Lecture’6:’’ Finding’Features’(part1/2)vision.stanford.edu/teaching/cs131_fall1415/lectures/Lecture6... · Lecture 6 - !!! Fei-Fei Li! Requirements’ • Region’extracHon’needs’to’be’repeatable’and’accurate’

Lecture’4:’ Pixels’and’Filters - Stanford Computer …vision.stanford.edu/teaching/cs131_fall1617/lectures...Fei-Fei Li Lecture 4- Whatwe’will’learn’today? • Images’as’functions