ESTUARY WETLAND DETECTION IN SAR IMAGES Presented By Yu-Chang Tzeng.
Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 ›...
Transcript of Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 ›...
Adaptive)Deep)Learning)for)VisualUnderstanding
Kate)SaenkoBoston)University
Harvard&University,&January&22&2019
Boston&University Slideshow*Title*Goes*Here
Prof.*Kate*Saenko
Boston&University Slideshow*Title*Goes*Here
AIR :*AI*Research*at*BU*
AIR
Boston&University Slideshow*Title*Goes*Here
Prof.*Kate*Saenko
Boston&University Slideshow*Title*Goes*Here
Goal:*teach*machines*to*see,$talk,$actResearch*in*my*lab*********************************************************************************************************** Kate*Saenko
A baseball game in progress with the batter up to plate
A man is riding a bicycle
Find*“window*upper*right”
A: skateboard
Q:*What*is*the*child*standing*
on?
Find*the*moment*when*“girl*looks*
up*at*the*camera*and*smiles”
pick*up*what*you*see
A
I
Vision Lang&uage
Action
“Go*out*of*the*bedroom,*down*
the*stairs,*turn*left*and*stop*at*
the*dining*table”
deep$learning$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$human$learning
?
adaptiveexplainablemodular
train test
HUMAN
● learns/from/a/single/example● generalizes/knowledge
adaptive,/explainable,/modular
train test
SUPERVISED0DEEP0LEARNING
● learns0from010000’s0examples● fails0on0new0domains
adaptive,0explainable,0modular
imagenet.org
HUMAN
● can*explain*decisions● grounds*language*in*world
adaptive,*explainable,*modular
why*is*this*a*dog?tail,*four*legs,*...
DEEP$LEARNING
● cannot$explain$decisions● black$box
adaptive,$explainable,$modular
why$is$this$a$dog?
HUMAN
● disentangles0properties● compositional
adaptive,0explainable,0modular
dog0running cat0sitting cat0running
adaptive,)explainable,)modular
dog)running cat)sitting ??
DEEP)LEARNING
● entangles)properties● not)compositional
adaptiveexplainablemodular
this2talk
Kate%Saenko,%Boston%University
Domain'shift
“Dataset'Bias”“Domain'Shift”
What%your%net%is%trained%on What%it’s%asked%to%label
Kate%Saenko,%Boston%University
Example(shift
train test
from+simulation++++++++++++++++++++++++++++++++++++++++++++++++to+reality
Kate%Saenko,%Boston%University
Shift&from&simulation&to&reality
road
sidewalk
people
treesca
r
car
sidewalk
sidewalk
road
trees
Input1Image True1Segmentation
Model1Output
grass
Kate%Saenko,%Boston%University
Solution:)Domain)Adaptation
road
sidewalk
people
trees
car
car
sidewalk
sidewalk
road
trees
Adapted/Model/Output
Input/Image True/Segmentation
Model/Output
grass
Kate%Saenko,%Boston%University
Applications+of+Domain+AdaptationFrom+dataset+to+dataset
From+simulated+to+real+control
From+RGB+to+depth
From+CAD+models+to+real+images+
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Encoder'CNN Classifierloss
Adversarial*domain*adaptation
Classifier
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Adversarial*domain*adaptation
Classifier
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Adversarial*domain*adaptation
Adversarial'loss
Classifier
Discriminator
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Classifierloss
Adversarial*domain*adaptation
Adversarial'loss
ClassifierEncoder'CNN
Encoder'CNNDiscriminator
Classifier
Kate%Saenko,%Boston%University
Results'on'Digits'Classification
● Domain(Confusion(Loss((Tzeng(2015)
● Adversarial(Domain(Alignment((ADDA)((Tzeng(2017)
Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation." ICML 2015Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep Transfer Across Domains and Tasks” ICCV 2015Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." CVPR 2017.
Domain(Confusion(Loss((Tzeng(2015)GradReversal((Ganin(&(Lempitsky(2015)
ADDA((Tzeng(2017)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Encoder'CNN Classifierloss
Problem:)ambiguous)features
Classifier
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Problem:)ambiguous)features
Classifier
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Problem:)ambiguous)features
Adversarial'loss
Classifier
Discriminator
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Classifierloss
Problem:)ambiguous)features
Adversarial'loss
ClassifierEncoder'CNN
Encoder'CNNDiscriminator
Kate%Saenko,%Boston%University
Source' Target'
Before'Adaptation Adapted
Problem:)ambiguous)features
Kate%Saenko,%Boston%University
Source' Target'
Goal:&avoid&generating&ambiguous&featuresBefore'Adaptation Adapted
Adversarial'Dropout'Regularization,'Kuniaki'Saito, Yoshitaka'Ushiku, Tatsuya'Harada, Kate'Saenko,'ICLR'2018
Kate%Saenko,%Boston%University
Solution:)use)the)decision)boundaryTrain&a&critic&(C)&(=discriminator)&that&can&detect&target&samples&near&decision&boundaryTrain&a&generator&(G)&that&can&fool&the&critic
Slightly&change&the&boundary&and&measure&the&change&of&p(y|x)!&&(sensitivity)
Samples&near&the&boundary&have&larger&sensitivity
Original&Boundary
Adversarial&Dropout&Regularization,&Kuniaki&Saito, Yoshitaka&Ushiku, Tatsuya&Harada, Kate&Saenko,&ICLR&2018
Kate%Saenko,%Boston%University
Data FeatureClassifiers
Sampling2by2dropout
Predictions
Measure2Sensitivity
Fix G"and2train C"to2maximize2d(p1,p2)"for2target2samples.Train G"and C"to2minimize2Cross2Entropy2for2source2samples.For2k2=1:nFix2C"and2train G"to2minimize2d(p1,p2)"for"target.
Adversarial"Dropout"Regularization
Adversarial2Dropout2Regularization,2Kuniaki2Saito, Yoshitaka2Ushiku, Tatsuya2Harada, Kate2Saenko,2ICLR22018
Kate%Saenko,%Boston%University
Results'on'Digits'Classification
● Adversarial+Dropout+Regularization+(ADR)+(Saito+et+al+2018)
Adversarial+Dropout+Regularization,+Kuniaki+Saito, Yoshitaka+Ushiku, Tatsuya+Harada, Kate+Saenko,+ICLR+2018
Domain+Confusion+Loss+(Tzeng+2015)GradReversal+(Ganin+&+Lempitsky+2015)
ADDA+(Tzeng+2017)
Kate%Saenko,%Boston%University
Adaptation)for)semantic)segmentation
Adapted
Ground,TruthInput,image
Source5only,model
Kate%Saenko,%Boston%University
Adaptation)for)semantic)segmentation
Adapted
Ground,TruthInput,image
Source5only,model
Kate%Saenko,%Boston%University
Pixel&to&pixel*adaptation
Kate%Saenko,%Boston%University
Strong'weak,feature,alignment
Strong'Weak,Distribution,Alignment,for,Adaptive,Object,Detection,,Saito,,Ushiku,,Harada,,Saenko,,arxiv,2018
Kate%Saenko,%Boston%University
Strong'weak,feature,alignment
Strong'Weak,Distribution,Alignment,for,Adaptive,Object,Detection,,Saito,,Ushiku,,Harada,,Saenko,,arxiv,2018
Kate%Saenko,%Boston%University
ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline
Domain%shift:%from%PASCAL%VOC%to%CLIPART
Kate%Saenko,%Boston%University
Evidence(for(target(domain(label((GradCAM)(shows(that(the(feature(extractor(seems(to(deceive(the(domain(classifier(in(regions(with(car.
Domain(shift:(from(Grand(Theft(Auto(game(to(Cityscapes
target
source
Kate%Saenko,%Boston%University
ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline
Domain%shift:%from%Cityscapes%to%Foggy%Cityscapes
adaptiveexplainablemodular
CNNs learn to predict pneumonia by detecting hospital which took the image
Variable(generalization(performance(of(a(deep(learning(model(to(detect(pneumonia(in(chest(radiographs:(A(cross8sectional(study.(Zech%JR1,%Badgeley%MA2,%Liu%M2,%Costa%AB3,%Titano%JJ4,%Oermann%EK3.%https://www.ncbi.nlm.nih.gov/pubmed/30399157
● Study%on%detecting%pneumonia%using%158,323%chest%radiographs
● CNNs%robustly%identified%hospital%system%and%department%within%a%hospital
● CNN%has%learned%to%detect%a%metal%token%that%radiology%technicians%place%on%the%patient%in%the%corner%of%the%image%field%of%view%at%the%time%they%capture%the%image
RISE:&randomly&mask&input,&measure&output
42
P[shark]
shark
RISE:&Randomized&Input&Sampling&for&Explanation&of&Black:box&Models,&Petsiuk,&Das,&Saenko,&BMVC&2018
adaptiveexplainablemodular
Explainable*Neural*Computation*via*Stack*Neural*Module*Nets*Networksinput:*There*is*a*small*gray*block;*are*there*any*spheres*to*the*left*of*it?
Hu,$Andreas,$Darrell,$Saenko,$Explainable$Neural$Computation$via$Stack$Neural$Module$Networks,$ECCV’18
input*image
Disentangling*properties*via*generation
A"Two&Stream"Variational"Adversarial"Network"for"Video"Generation
generated"video
Disentangling*properties*via*generation
A*Two2Stream*Variational*Adversarial*Network*for*Video*Generation,*arxiv*2018
Disentangling*properties*via*generation
Disentangling*properties*for*domain*transfer
adaptiveexplainablemodular