6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for...
Transcript of 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for...
![Page 1: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/1.jpg)
Roadmap:OperatingPentahoatScaleJensBleuelSeniorProductManager,Pentaho
![Page 2: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/2.jpg)
Agenda– WorkerNodes
HearaboutnewupcomingcapabilitiesforscalingoutthePentahoplatforminlargeenterpriseoperations.Thiswillcover8.0androadmaptopics.
• WorkerNodes:OverviewandBusinessBenefits
• HowisthisdifferentfromAEL/HadoopMapReduce
• TypicalCustomerScenarios
• Architecture&CapabilitiesincludingMonitoring&Logging
• ImprovementsinRelatedAreas
• Demonstration
• Availability&Roadmap
![Page 3: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/3.jpg)
WorkerNodes– Overview
• WorkerNodescanscaleworkitemsacrossmultiplenodes(containers)like:
– PDIjobsandtransformations(in8.0)– Reportexecutions(notin8.0)– […]
• Itoperateseasilyandsecurelyacrossanelasticarchitecture,whichaddsadditionalmachineresourcesastheyarerequiredforprocessing
• WorkerNodescanoperateonpremiseorinthecloud
• UsesPopulartechnologiesunderthehoodsuchasDocker(ContainerPlatform),Chronos(Scheduler)andMesos/Marathon(ContainerOrchestration)
WorkerNode(a)
WorkerNode(b)
WorkerNode(c…)DistributeandScale
![Page 4: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/4.jpg)
WorkerNodes– BusinessBenefits
Largeenterprisesneedtheabilitytoseamlesslyandefficientlyspinupresourcestohandle100s+workitemsatdifferenttimes,withdifferentdependenciesandprocessingrequirements.WorkerNodesaddressestheseneedsanddelivers:• FastertimetovalueandreducedTCObecauseitenablescustomerstodeploytheirownscale-outprocesseswithoutrequiredservices• Managechangingworkloadsmoreefficientlybyspinningresourcesupanddownasneeded• Increasedbusinessagilitythankstocontainerization– whichenablesportabilityofprocessesacrosson-prem andcloudenvironmentswithouttheneedtore-engineerthem.– Eveninpureon-prem,WNprovideselasticityandresourceoptimization.
![Page 5: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/5.jpg)
HowIsThisDifferentfromAEL/HadoopMapReduce?
Thesetwoarchitecturescanalsobecombined:WithinaWorkerNode,aPDItransformationcanalsoscaleoutwithAELorMapReduce
SCALEOUTONDATA
SCALEOUTONPROCESSES(WORKITEMS)
AEL/HadoopMapReduce(simplified):• Dataisdistributedacrossnodes• Theprocessingtakesplaceatthenodelevel• Helpsinscaleoutdatavolume
WorkerNodes(simplified):• WorkItemslikePDIJobs,PDITransformationsgetdistributedacrossnodes– thisisabouttheprocessingandorchestration(incontrasttodistributingdata)
• HelpsinscaleoutPentahoprocesses
![Page 6: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/6.jpg)
TypicalCustomerScenarios
CustomerType TypicalNumberofWorkItems Scale-OutNeed
Small Upto10 No
Medium 10through100 Sometimes
Enterprisewithonedepartment +/- 100 Yes
Enterprisewithmultipledepartments Hundredsorthousands Yes
![Page 7: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/7.jpg)
TypicalCustomerExamples– SLA’sandTimeWindows
• NeedtomeetcustomerSLA’s– Datafromhundredsofsourcesneedtogetcollectedandaggregated– ThisisdonebyhundredsofPDIjobsandtransformations– Allthesejobsandtransformationsneedtobefinishedwithinadefinedtimewindow(forexamplebetween5amand7am)sothatthedataisavailableandaccurateforthetargetaudience
• WorkerNodesprovidesthetechnologytorunprocessesinparallelandscaleoutwhenneeded,forexampleatpeaktimes(endofmonth)
![Page 8: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/8.jpg)
TypicalCustomerExamples– SharedServices
Exampleofoneproject:
• 800dailybatchesfromdifferentdepartmentsinanenterprise
• Oneserverwith120GBmemoryandmanyCPUs
• ThismachinehostslotsofVMinparallel
Issue:Whenthereistoomuchworkload,onemachineisnotenough
• WorkerNodessolvesthisinscalingoutonacluster
![Page 9: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/9.jpg)
TypicalCustomerExamples– ScalableonDemand
• Needtosupportgrowingdatavolumesandcustomerrequirements
• WorkerNodesprovidesaflexibleandscalablearchitectureon-promiseorinthecloudforgrowingdemand
• Thisisseamlessanddoesnotneedtochangetheunderlyingarchitecture
WorkerNode(1)
WorkerNode(2)
WorkerNode(3)DistributeandScale
WorkerNode(1)
WorkerNode(2)
WorkerNode(3)DistributeandScale
WorkerNode(4)
WorkerNode(5)
BASETIMES PEAKTIMES
![Page 10: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/10.jpg)
WORKERNODES
OrchestrationFramework
ContainerFramework
WorkerNodes– Newin8.0
• Containerizedscale-out• PentahoPDI“workitems”
PentahoServerWN1e.g.KJB
WN2e.g.KTR
WN…n“Executor”
Orchestration(Scheduler,monitoring,security,etc.)
Controller
Master(Standby)
Master(Standby)
Master(Working)
PentahoRepository
PentahoClients
![Page 11: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/11.jpg)
WorkerNodesCapabilities
• Deployconsistentlyinphysical,virtual,andcloudenvironmentsAdaptstocustomerneeds(bare-metalvs.virtualizationvs.Cloud)andnoneedtomodifytheproductwhenthestrategychanges
• ScaleandloadbalanceservicesThishelpstodealwithpeaksandlimitedtime-windows,allocatetheresourcesthatareneeded.
• HybriddeploymentscanbeusedtodistributeloadEvenwhentheon-premise resourcesarenotsufficient,scalingoutintotheCloudispossibletoprovidemoreresources.
![Page 12: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/12.jpg)
MonitoringandLogging
![Page 13: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/13.jpg)
Monitoring– Overview
![Page 14: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/14.jpg)
Monitoring– WorkerNodeExample
![Page 15: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/15.jpg)
ImprovementsinRelatedAreasOpenandSaveDialogs
![Page 16: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/16.jpg)
• Wheneveryousaveanewtransformation/jobintotherepository,thedefaultfolderissettotheuser’shomefolder.
PainPoint:SaveaNewJob/Transformation
Inpreviousversions:Theuserwillneedtochangethefolderforeverytimetheysaveanewtransformationorjob.
![Page 17: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/17.jpg)
NewSaveDialogin8.0– Overview
• Remembersthelastopenedfolder!
• Justenterthefilename!(and/orchangethefolder)
• SimilartotheOpenDialogwithadditionalfunctionality(seenextslide).
![Page 18: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/18.jpg)
NewOpenDialogin8.0– Overview
Recents
Openshowsthelastopenedfolder.Thisisabigtimesaver!
Search
![Page 19: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/19.jpg)
ImprovementsinRelatedAreasRunConfigurations
![Page 20: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/20.jpg)
PainPoint:RemotePentahoServerExecutionbefore 8.0
ToexecuteonthePentahoServerbefore8.0,youneedtodefineaSlaveserverandgivethecredentials. ThenexecuteontheselectedServer.
![Page 21: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/21.jpg)
ExecuteonthePentahoServer
• ByselectingthePentahoserveroption,youdonotneedtodefineaSlaveserveranymorewhenyouwanttoexecuteremotely.
• Behindthescenes,thisoptionexecutesthetransformationorjobviatheScheduler.Thisisthesameasyouwoulddoa“ScheduleNow.”
Thisnewfunctionalityimprovestheeaseofuse,alsoforWorkerNodes
![Page 22: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/22.jpg)
RunConfigurationswithinJobEntries
• RunConfigurationcanbeusedintheRundialogandalsointhejobentriesthatcouldexecutejobsortransformationsremotelyandonWorkerNodes
7.1 Example
8.0
![Page 23: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/23.jpg)
Demonstration
![Page 24: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/24.jpg)
AvailabilityandRoadmap
![Page 25: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/25.jpg)
Availability
• WorkerNodesisEEonly
• Initially,8.0WorkerNodeswillbeLimitedAvailability– Fullysupported,productiondeployment– Distributiontoalimitednumberofcustomers
• Requiresadditionaldownloadandimplementationservices
![Page 26: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/26.jpg)
• PentahoServer&RepositoryasaServiceincludingHighAvailability
• ImprovedMonitoringandLogging
• ExtendtootherPentahoworkitemssuchasReports
• IntegratedwithotherHitachiVantara ServicesandProducts
Roadmap
ContainerFrameworkPentahoServer
WN1e.g.KJB
WN2e.g.KTR
WN…n“Executor”
PentahoRepository
![Page 27: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/27.jpg)
Summary
Whatwecoveredtoday:
• TheupcomingcapabilitiesforscalingoutthePentahoplatformandwhentousethem
• Howtousethenewwayofscalingoutworkitems(PentahoprocessessuchasPDIjobsandtransformations)acrossmultiplenodes
![Page 28: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/28.jpg)
NextStepsWanttolearnmore?
• Meet-the-Expert:– PedroTeixera
• Otherrecommendedbreakoutsessions:– MattHoward:Pentaho8.0andRoadmap– RakeshSaha andJensBleuel:Roadmap:ProcessingBigData– MattCasters:PDIBestArchitecturePractices– SteveSzabo:PDISizingOverviewandCaseStudy– JonathanJarvis:UnderstandingParallelismwithPDIandAdaptiveExecutionwithSpark– MarkBurnett:UnderstandingtheBigDataTechnologyEcosystem
![Page 29: 6 Roadmap Operating Pentaho at Scale - … –Worker Nodes Hear about new upcoming capabilities for scaling out the Pentaho platform in large enterprise operations. This will cover](https://reader033.fdocuments.us/reader033/viewer/2022051803/5b03307c7f8b9a89208bce27/html5/thumbnails/29.jpg)