Post on 22-Jan-2018
DRAWDeepRecurrentA-en.veWriter
KarolGregor,IvoDanihelka,AlexGraves,DaniloJimenezRezende,DaanWierstraGoogleDeepmind–(ICML2015)
TranQuocHoanPaperalert@2015-12-07
Imagegenera.ngwitha-en.on
Trainedgenera.ngsequen.alMNISTdigitwithouta)en,on
Trainedgenera.ngsequen.alMNISTdigitwitha)en,on
Introduc.on
• Addsana-en.onmechanismovertheinputtodefineasequen.alprocess
• Augmentstheencoderanddecoderwithrecurrentneuralnetworks
• Inferenceandgenera.ondefinedbyasequen.alprocess,evenfornon-sequen,aldata
4/25
Varia.onalAutoencoderRecap
• KingmaandWelling,Auto-EncodingVaria,onalBayes,(ICLR)2014
• Rezende,MohamedandWierstra,Stochas,cback-propaga,onandvaria,onalinferenceindeeplatentGaussianmodels,(ICML)2014
We’llintroduceVAEfirst!!!
5/25
VAEApproach• Leverageneuralnetworkstolearnalatentvariablemodel
Imagefrom:Ward,A.D.,Hamarneh,G.:3DSurfaceParameteriza,onUsingManifoldLearningforMedialShapeRepresenta,on,ConferenceonImageProcessing,Proc.ofSPIEMedicalImaging,2007
Latentvariable
6/25
Theinferenceproblem• Wheredoeszcomefrom?->Theposteriorp(z|x)isintractable
• VAEapproach:findinferencemachinethatlearnstoapproximatetheposteriorbyop.mizingthevaria.onallowerbound
7/25
Encoder Decoder
Trainingwithbackpropaga.on• Duetoreparametriza.ontrick,wecansimultaneously
trainboththegenera.vemodelandtheinferencemodelbyop.mizingthevaria.onalboundusinggradientbackpropaga.on
• Maximum
9/25
DRAWMNISTGenera.on
• Simplestinstan.a.onofDRAWiswithoutana-en.onmechanism
• En.reinputispassedtotheencoderatevery.me-step
• Decoderwritestoen.recanvasatevery.me-step
No.cehowthenetworkfirstgeneratesaveryblurryimagethatissubsequentlyrefined
21/25
DRAWA-en.onMechanism
• DRAWusesadifferen.ablea-en.onmechanism
• A-en.onusesrecurrence(viadecoder)toselectsubsetsofxforreadingandwri.ng
• A-en.oncontrolstheextractedpatchloca.on,scaleandblur
22/25
DRAW:Clu-eredMNISTClassifica.on
• Drawa-en.onbeingappliedtoaclassifica.ontask:clu-eredMNIST
• A-en.onlearnstofocusonthedigitinthescene
Time→24/25
DRAWMNISTGenera.onwithA-en.on
Figure6.GeneratedMNISTimages.AlldigitsweregeneratedbyDRAWexceptthoseintherightmostcolumn,whichshowsthetrainingsetimagesclosesttothoseinthecolumnsecondtoright.Notethatthenetworkwastrainedonthebinarysamples,whilethegeneratedimagesaremeanprobabili.es. 25/25
DRAWDeepRecurrentA-en.veWriter
KarolGregor,IvoDanihelka,AlexGraves,DaniloJimenezRezende,DaanWierstraGoogleDeepmind–(ICML2015)
TranQuocHoanPaperalert@2015-12-07
Manyslidesareborrowedfromh)p://videolectures.net/deeplearning2015_courville_autoencoder_extension/
h)p://mul,threaded.s,tchfix.com/blog/2015/09/17/deep-style/