ClassifyingRealEstateAdImageswithConvolutionalNeuralNetworksCASMachineIntelligence,DeepLearning,April2018

Introduction&Motivation• Onrealestateportalsoneoftenfindsprettybadimages;even

thefirstpictureisoftennotnice.• Weusedadatasetof2‘000realestateadimagestobuilda

classifier,whichmighthelptoautomaticallyselectanappropriateimage(e.g.exteriorviewforfirstpicture).

• ByextractingtheEXIF-Tagstheimagesweregroupedinto6classes(exteriorview,interiorview,kitchen,bathroom,floorplan,other).

Results• Overallthebestmodelachievedanaccuracyonthetestsetof

0.93(~200images).Howeverthismodelusedonlythreeofthe6classes.

• Onall6classesthepretrainedVGG16modelachievedthebestaccuracywith0.84.

• Classificationreportsprovidesfurtherdetailstotheperformanceofthedifferentclasses:

Models• Weplayedaroundandtrieddifferentapproaches(own

models,pretrainedfeaturesmodelVGG16)andworkedwiththekerasdatagenerator.

• ForallmodelsweusedBatchnorm- andDropout-Layers,andforoptimizationAdamwithcrossentropyloss.

• WedecidedtofurtherlookatModel4becausewelikedthegoodresultswithoutpretraining.

Insights• Thevisualizedfilters/featuremapsshowthebasicstructures

thatwerelearnedbythemodel;thelastlayerseemstomakemuchsenseandcanbevisuallylinkedtotheclasses.

• Byvisualizingimportantpixelsfortheclassesinteriorview,exteriorviewandfloorplan,wecanseethatit’snotjustoneareathat’srelevant,butpixelsalloverhaveanimpact.

Outlook• Theclassificationbyroom/viewcouldbecombinedwithother

imageclassifiers.• Forexamplethemostappealingimagecouldautomaticallybe

selectedtobedisplayedfirstinanadvertisement.• Anotherpossibilitywouldbetodetectobjectsintheimages

(e.g.dishwasher,fireplace).Thisinformationcouldbeusedtoaugmentanadwithadditionalinformation.

• Acombinationofallmodelscouldbeappliedinafullyautomatedonlinerealestateadprocess.

Conclusions• Modelswithfewerclassesshowedhigheraccuracyandseem

readyforrealworldapplication.• Moretrainingdataisneededforarobustmodelwithmore

classes.• Datacleansingshouldbeappliedtofurtherimprove

performance(misclassificationsrevealmislabeledtrainingdata).

• Imagesizecanbefairlysmallwithoutanylossinaccuracy,resultswerecomparableforinputimagesizesof224x224pxand100x100px.

• UsingaGPU(ongooglecolab)speedsuptrainingbyafactorof20(180s/epochvs.9s/epochtotrainModel5(VGG16)).

References:https://github.com/tensorchiefs,https://github.com/keras-team/keras,https://raghakot.github.io/keras-vis,https://github.com/experiencor/deep-viz-keras,https://github.com/marcotcr/lime,https://www.trulia.com/blog/Authors:RetoCamenzind,LukasStöcklin,JuliaSulc

Model Pretrained #Classes #Params #Conv.-Layers AccuracyModel1 No 6 6,378,314 5 0.75Model2 No 6 10,729,526 5 0.70Model3 No 5 (w/o other) 10,779,713 5 0.77Model4 No 3 (interior, exterior,

floorplan)6,333,219 5 0.93

Model 5 Yes(VGG16)

6 15,936,134(1’221’446trainable)

16 0.84

Model5:RetrainedVGG16(all6classes)Model4:5Conv.Layers(3classes)

380

224

11580

497

169