Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De...
-
Upload
papisio -
Category
Technology
-
view
259 -
download
0
Transcript of Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De...
![Page 1: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/1.jpg)
Shortening the time from analysis to deploymentwith ML-as-a-Service
TEVECSystems
LuizAugustoCanito Gallego deAndradeGabrieldeBodt Sivieri
![Page 2: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/2.jpg)
TimeSeriesForecasting
Brazil’s GDP
Indistrial Capacity
Sales
Whatwillsalesbelikeinthecomingperiods?
![Page 3: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/3.jpg)
TimeSeriesForecasting
Somestrategiestodealwiththeproblem
![Page 4: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/4.jpg)
TimeSeriesForecasting
Somestrategiestodealwiththeproblem
EmbeddingstrategyFeatureengineeringstrategy
![Page 5: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/5.jpg)
APICustomerStory
Thecustomerneedsinsightsabouthisdataandtobuildvalueuponitsdatabase 1
Thecustomeristhrilledwiththeresultsandeagerlywantstodeploythisnewacquiredknowledgeinhisbusinessprocesses
3
DataScienceteamscomesinthescenetocrunchdataanddeliverpowerfull modelsandinsightsaboutcustomerdata
2
Whataretherequirements?
4
CustomerSideConsultingSide
![Page 6: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/6.jpg)
APICustomerStory
APIServiceLevel CloudStandards Improvedaccuracyovertime
Freshinsightstoincreasevalue
CodeStandardsandreleaseworkflow
Newvariablesfrompublicsources
![Page 7: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/7.jpg)
APICustomerStory
APIServiceLevel CloudStandards Improvedaccuracyovertime
Freshinsightstoincreasevalue
CodeStandardsandreleaseworkflow
Newvariablesfrompublicsources
Some objectives/requirementsare extremely software related
![Page 8: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/8.jpg)
APICustomerStory
APIServiceLevel CloudStandards Improvedaccuracyovertime
Freshinsightstoincreasevalue
CodeStandardsandreleaseworkflow
Newvariablesfrompublicsources
Others are Data Science related
![Page 9: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/9.jpg)
MachinelearningasaService
FocusGroupsstrategies
FocusGroup1
Collaborationishard
Problemsaresolvedlocally
Problemoriented
Thereisnolongtermstrategy
FocusGroup2
FocusGroup4FocusGroup3
![Page 10: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/10.jpg)
MachinelearningasaService
ProductOrientedStrategy
LimitedAPIproblemrange
Softwareproblemsbecomefocus
”Distancefromdata”
”Onesizefitsall”
Softwareengineering
Customerservice
DataScience UserExperience
![Page 11: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/11.jpg)
Ourviewofthematter
Experimentationframework
CommonlyusedframeworksandAPIs
Model 1
Model2
Model3
Model4
Pipelines
Documentbased
database
Modelostreinados
ProductionStructure REST
ContinuousDataScience
![Page 12: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/12.jpg)
What’sapipeline?
NodeNode
Node
Node
Node
Target
Bycombiningeffectivesoftwarearchitectureandstate-of-the-artMLandDStoolsweareabletoquicklytestanddeployafreshpipelinesfordifferentproblems
![Page 13: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/13.jpg)
Experimenting(AgileDataScience)
MLengineeringRunAccuracyReport
DataScienceSubsamplesdatasetstofocusonanimprovement
DataScienceDesigningnewmodelsinsmall/mediumsizescale
testing
FocusonBusinessmetrics(MAPE,ROC).Secondaryuseof”math”metricssuchasRMSEorLogLoss
Accuracyisreportedbasedinproductionforecastsversusupdatedinformation
Clusteraccuracybydatasetthemeorkeystatisticalmetrics
UseofTEVEC’spipeliningframeworkforquickmodeldesign
Prototypeusingsmallscaletestinginaconsoleapplication(JupyterHub)
![Page 14: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/14.jpg)
Experimenting(AgileDataScience)
MLengineeringRunAccuracyReport
DataScienceSubsamplesdatasetstofocusonanimprovement
DataScienceDesigningnewmodelsinsmall/mediumsizescale
testing
DataScience/MLEngineeringLargescaletestingon
productionframeworkusingproductiondata
MLengineeringPushpipelinestoproductionandmonitoroperations
BusinessDecisionAnalyzetheaccuracyreport
anddecidetopushtoproduction
ExperimentingstructureisanactualdocumentinTEVEC’sODMdatastructure
Experimentconnectswithpipelinesandappliesittoasequenceofdatasets
A/BTestingcomparesperformanceinsameformatasAccuracyReport
Businesshasbusiness-likeinputstodecidecommunicateexpectedresultstocustomer
Thenewpipelinewasvalidatedthroughoutthewholeexperiment.Itissafetopushtoproduction.
![Page 15: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/15.jpg)
Experimenting(AgileDataScience)
MLengineeringRunAccuracyReport
DataScienceSubsamplesdatasetstofocusonanimprovement
DataScienceDesigningnewmodelsinsmall/mediumsizescale
testing
DataScience/MLEngineeringLargescaletestingon
productionframeworkusingproductiondata
MLengineeringPushpipelinestoproductionandmonitoroperations
BusinessDecisionAnalyzetheaccuracyreport
anddecidetopushtoproduction
Wetrytorepeatthecycleeveryweek
![Page 16: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/16.jpg)
Experimenting(AgileDataScience)
LargeScaleexperimentingisaninherentpartofthesystem.
![Page 17: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/17.jpg)
Conclusions
WeachievedprocessstabilityonceweseparatedourDataScienceteamfromtheProductionSoftwareEcosystem
ThroughacollaborationbetweenDataScienceteamandMLEngineerswewereabletodesignacontinuousexperimentationprocess
Tocareaboutstandardsandinterfaceinexperimentationstageistosavetimeindeployment.Thisalsoreducestheriskofunexpectederrorsinproduction
Pipelinestructureusesstate-of-the-artpackagesandframeworkswhileenforcinginterfacesandsoftwarearchitecture,notcodingstandards.ThissavestimetofocusonDataScience
Wearestilllearningfromthisnew”continuous”DSprocess,butsofarwehavehadexcellentresultsinteamgrowingandincrementallyimprovingoursoftware
![Page 18: Shortening the time from analysis to deployment with ml as-a-service — Luiz Andrade and Gabriel De Bodt Sivieri (tevec sistemas sa) @PAPIs Connect — São Paulo 2017](https://reader033.fdocuments.us/reader033/viewer/2022052915/5a67f3f77f8b9a8a068b46a1/html5/thumbnails/18.jpg)
Luiz Augusto Canito Gallego de Andrade+55 (11) 9 [email protected]
Gabriel Sivieri+55 (11) 9 [email protected]