Gettingcreativewithautoencoders

Tijmen Tieleman,CEO&CTOatminds.aiWiSSAP 2019,Trivandrum

Overview

• Thisisafairlygeneraltalk• Application-agnostic• Generalideas,ratherthanrecentresearch

• Thisisafairlyspecializedtalk• It’saboutDeepLearning• Morespecifically:representationlearning• Morespecifically:unsupervisedrepresentationlearning• Morespecifically:autoencoders

Autoencoders

• Trainingobjective:reconstructtheinput• Components:encoderanddecoder• Usualpurpose:representationlearning• Generativeview:variationalautoencoders• Components:encoderanddecoder generator

• Flexible• Getcreativewithencoderand/orgenerator• Interpretablecodescanbearranged

• Codeunitsareonlypartiallyhiddenunits

Clustering

• “kmeans”• Kmeansasan(odd)autoencoder• Encoder(notNN)• Generator(constantforeachcluster)

• Implicitassumption• Morepowerfulgenerator:non-trivialcovariance• Differentimplicitassumption

ClusteringautoencoderWithfreecodeunitsNaïve

Withfreecodeunitsanddiscretization

Intermission:afewminutesforquestionsaboutthepastfewslides.

Gettingcreativewithautoencoders

• Clusteringautoencoder• Ignoringautoencoder• Binarizingautoencoder• Biasedautoencoder• Computergraphicsautoencoder• Independentfeaturesautoencoder• Etc• Feelfreetocombine

Ignoringautoencoder

• Goal:codethatignoressomeinformation• E.g.ignoringspeakerinformation• E.g.ignoringthespokentext

• Application:focusingonessentials• Application:domainadaptation• Application:anti-discrimination• Application:reconstructingwithdifferentinformation• Method1:providethatinformationtothedecoder• Method2:insistthattheto-be-hiddeninformationcannotbereconstructed

Binarizingautoencoder

• Goal:binarycode• Application:”semantichashing”• Application:extremecompression(lossy)• Method:usediscretizationnoiseonabankoflogisticcodeunits

Biasedautoencoder

• Goal:codesthattendtobeofcertainkinds• E.g.asoftmax with2commonlyusedunitsand10rarelyusedones.• E.g.binarizingandpreferringsparsity

• Method:simplypushintheobjective

Intermission:afewminutesforquestionsaboutthepastfewslides.

Computergraphicsautoencoder

• Moregenerally:domain-specificgenerator• E.g.audiosynthesisautoencoder

• Goal:highlyinterpretablecodes• Goal:helptheencoderunderstandthenatureofimages• Application:reverserendering,for“imageparsing”oraspreparationforgeneratingimagevariants• Application:usingthebetterencoderasagoodfeaturizer• Method:replacethegenerator

Computergraphicsasgenerator

• Problem:ithastobedifferentiable• Solution1:implementareducedCGsysteminTensorFlow• Weakness:“reduced”isunfortunate• Nice:itcancontainlearnables

• Solution2:learnasecondNNtoapproximatearealCGsystem;backpropagatethroughtheapproximation• Weakness:approximationsareapain• Nice:ithasthepotentialforfullCG• Thecodesstillhavetobecontinuous

PoC:graphicsforMNIST

• Uses“capsules”• Learnedconstant:smallimagetemplates• Learnedfunction:freecodeunitvaluesà affinetransformationforthetemplates• Hard-codedfunction:applyaffinetransformationstothetemplates• Affinetransformedtemplatesareaddedtogethertocreatefinaloutput

PoC:graphicsforMNIST

• Uses“capsules”• Learnedconstant:smallimagetemplates• Learnedfunction:freecodeunitvaluesà affinetransformationforthetemplates• Hard-codedfunction:applyaffinetransformationstothetemplates• Affinetransformedtemplatesareaddedtogethertocreatefinaloutput

ClusteringMNIST

• Kmeansclusteringwouldn’tworkwell:thegeneratoristooprimitive• Weneedamorepowerfulgenerator• Ithastounderstandthatashiftedimageisstillaboutthesameimage• Let’susethiscomputergraphicsgenerator

• Unsupervisedclusteringinto25clusters• Goodclassificationaccuracywithjust25labeledcases

Independentfeaturesautoencoder

• Goal:codecomponentsthataresomewhatindependentlyinterpretable• Application:identifyingthepatternsinthedata• Application:generatingdatavariantswithahigherchanceofmeaningfulresults• Method:applydropouttothecodecomponents

Independentfeatures

• Thisfocusesononeimage(seethecentralcolumn)• Thisshowsdatavariantsbyvaryingonefeatureatatime• The8featuresareclearlysomewhatindependent• Thevariantsaremeaningful

ß Featurevalueà

ßFeatureindexà

Toolsforcraftingspecializedautoencoders

• Imposesomestructureonthecode• E.g.clusteringautoencoder,computergraphicsautoencoder

• Applydeterministiceditingtothecode• E.g.ignoringautoencoder

• Applyrandomeditingtothecode• E.g.binarizingautoencoder,clusteringautoencoder,independentfeaturesautoencoder

• Applyanobjectivefunctionadditiontothecode• E.g.sparsebinarizingautoencoder

• Hard-code(partof)thegenerator• E.g.computergraphicsautoencoder

Conclusion

• Creatingcustomspecializedautoencodersisdoable• Thereisacollectionofbasictoolsforthis• Thiscanimproveinterpretabilityandlearningpower

• NextstageforDL:combininglearnableandhard-codedcomponents• Learnablecomponentsprovide“intuition”• Hard-codedcomponentsprovidethepowerofcomputation• Eitheraloneisquitelimited• AlphaGohashard-codedasmain()andlearnableassubroutine• TheCGautoencoderhaslearnableasmain()andhard-codedassubroutine

Thankyou