1 ©HortonworksInc.2011–2016.AllRightsReserved1 ©HortonworksInc.2011–2017.AllRightsReserved
2 ©HortonworksInc.2011–2016.AllRightsReserved
How Customers are Optimizing their EDW for Fast, Secure, Cost Effective Actionable Insights
Wei Wang Sr. Director, Product Marketing
Paige RobertsBig Data Product Manager
Speakers:
3 ©HortonworksInc.2011–2016.AllRightsReserved
ActionableInsights
HumanConnections
BalancedSupplyChains
NewProducts&Services
OperationalEfficiencies
DataIsTheNewCurrency
4 ©HortonworksInc.2011–2016.AllRightsReserved
TheOldWayOperationalEfficiencies
Hierarchical
Historical
Highly Aggregated
One-size-fits-all
TheNewWayEngagementandInnovation
Multi-structured
Predictive
Agile &Real-time
Context Sensitive
TransformingTheEnterpriseDataWarehouse(EDW)
5 ©HortonworksInc.2011–2016.AllRightsReserved
HortonworksConnectedDataPlatformsandSolutions
HortonworksConnection
HortonworksSolutions
EnterpriseDataWarehouseOptimization
CyberSecurityandThreatManagement
InternetofThingsandStreamingAnalytics
HortonworksConnectionSubscriptionSupportSmartSense
PremierSupportEducationalServicesProfessionalServices
CommunityConnection
CloudHortonworks DataCloudAWS HDInsight
DataCenterHortonworks DataSuite
HDFHDP
6 ©HortonworksInc.2011–2016.AllRightsReserved
TheSolutionforEnterpriseDataWarehouseOptimization
DramaticCostReductionsThroughthePowerofOpenSourceandApacheHadoopReducecostofyourEDWImplementationbyaugmentationorreplacementofETLprocesses
DeployBusinessIntelligenceonHadoopEnablementBusinessusers(viaadesktopsolution)seeaggregatedata,drillup,drilldownthroughbusinessdatamodels.
EnrichwithUnstructuredDataSupportDeliversupportforphotos,video,textfilestoenrichyouranalytics
7 ©HortonworksInc.2011–2016.AllRightsReserved
LegacyEDWvs.EDWOptimizationSolutionwithConnectedDataPlatforms
910
920
930
940
950
960
970
980
990
1,000
Difficulty transforming data
into a suitable form for analysis.
Difficulty integrating
Big Data with existing
infrastructure.
Difficulty merging multiple, disparate
data sources.
Lack of skilled Big Data
practitioners.
Difficulty maintaining application
performance for large volume of
concurent users.
ImplementingtheModernDataArchitectureIsn’tEasy
8
Source:Wikibon BigDataAnalyticsAdoption Survey,2014-2015
Syncsort Confidential and Proprietary - do not copy or distribute
Oftheselectedtechnology-relatedbarrierstorealizingthefullvalueofyourBigDataAnalytics,pleaserankthetop3.
910
920
930
940
950
960
970
980
990
1,000
Difficulty transforming data
into a suitable form for analysis.
Difficulty integrating
Big Data with existing
infrastructure.
Difficulty merging multiple, disparate
data sources.
Lack of skilled Big Data
practitioners.
Difficulty maintaining application
performance for large volume of
concurent users.
ImplementingtheModernDataArchitectureIsn’tEasy
9
AdditionalChallenges• Long,drawn-outdevelopmentcycles
Source:Wikibon BigDataAnalyticsAdoption Survey,2014-2015
9Syncsort Confidential and Proprietary - do not copy or distribute
Oftheselectedtechnology-relatedbarrierstorealizingthefullvalueofyourBigDataAnalytics,pleaserankthetop3.
910
920
930
940
950
960
970
980
990
1,000
Difficulty transforming data
into a suitable form for analysis.
Difficulty integrating
Big Data with existing
infrastructure.
Difficulty merging multiple, disparate
data sources.
Lack of skilled Big Data
practitioners.
Difficulty maintaining application
performance for large volume of
concurent users.
ImplementingtheModernDataArchitectureIsn’tEasy
10
AdditionalChallenges• Long,drawn-outdevelopmentcycles
• Rapidlychangingtechnologypresentsamovingtarget,forcingconstantre-design
Source:Wikibon BigDataAnalyticsAdoption Survey,2014-2015
10Syncsort Confidential and Proprietary - do not copy or distribute
Oftheselectedtechnology-relatedbarrierstorealizingthefullvalueofyourBigDataAnalytics,pleaserankthetop3.
910
920
930
940
950
960
970
980
990
1,000
Difficulty transforming data
into a suitable form for analysis.
Difficulty integrating
Big Data with existing
infrastructure.
Difficulty merging multiple, disparate
data sources.
Lack of skilled Big Data
practitioners.
Difficulty maintaining application
performance for large volume of
concurent users.
ImplementingtheModernDataArchitectureIsn’tEasy
11
AdditionalChallenges• Long,drawn-outdevelopmentcycles
• Rapidlychangingtechnologypresentsamovingtarget,forcingconstantre-design
• Difficultyaccessingandintegrating legacyandnewdatasourcesincludingthemainframe
Source:Wikibon BigDataAnalyticsAdoption Survey,2014-2015
11Syncsort Confidential and Proprietary - do not copy or distribute
Oftheselectedtechnology-relatedbarrierstorealizingthefullvalueofyourBigDataAnalytics,pleaserankthetop3.
910
920
930
940
950
960
970
980
990
1,000
Difficulty transforming data
into a suitable form for analysis.
Difficulty integrating
Big Data with existing
infrastructure.
Difficulty merging multiple, disparate
data sources.
Lack of skilled Big Data
practitioners.
Difficulty maintaining application
performance for large volume of
concurent users.
ImplementingtheModernDataArchitectureIsn’tEasy
12
AdditionalChallenges• Long,drawn-outdevelopmentcycles
• Rapidlychangingtechnologypresentsamovingtarget,forcingconstantre-design
• Difficultyaccessingandintegrating legacyandnewdatasourcesincludingthemainframe
• Functionalitygapsinsecurity,governance,andcompliance
Source:Wikibon BigDataAnalyticsAdoption Survey,2014-2015
12Syncsort Confidential and Proprietary - do not copy or distribute
Oftheselectedtechnology-relatedbarrierstorealizingthefullvalueofyourBigDataAnalytics,pleaserankthetop3.
910
920
930
940
950
960
970
980
990
1,000
Difficulty transforming data
into a suitable form for analysis.
Difficulty integrating
Big Data with existing
infrastructure.
Difficulty merging multiple, disparate
data sources.
Lack of skilled Big Data
practitioners.
Difficulty maintaining application
performance for large volume of
concurent users.
ImplementingtheModernDataArchitectureIsn’tEasy
13
AdditionalChallenges• Long,drawn-outdevelopmentcycles
• Rapidlychangingtechnologypresentsamovingtarget,forcingconstantre-design
• Difficultyaccessingandintegrating legacyandnewdatasourcesincludingthemainframe
• Functionalitygapsinsecurity,governance,andcompliance
Source:Wikibon BigDataAnalyticsAdoption Survey,2014-2015
13Syncsort Confidential and Proprietary - do not copy or distribute
Bigdataintegrationiscomplicated!
Oftheselectedtechnology-relatedbarrierstorealizingthefullvalueofyourBigDataAnalytics,pleaserankthetop3.
14 ©HortonworksInc.2011–2016.AllRightsReserved
HortonworksEDWOptimizationSolutionComponents
HadoopScalableStorageandCompute
HiveLLAPHighPerformanceSQLDataMart
Fast,scalableSQLanalyticsIntelligentin-memorycaching
15 ©HortonworksInc.2011–2016.AllRightsReserved
HortonworksEDWOptimizationSolutionComponents
SyncsortHigh-PerformanceDataMovement
HadoopScalableStorageandCompute
HiveLLAPHighPerformanceSQLDataMart
SourceDataSystems
Fast,scalableSQLanalyticsIntelligentin-memorycaching
Highperformancedataimportfrom allmajorEDWplatforms
16 ©HortonworksInc.2011–2016.AllRightsReserved
HortonworksEDWOptimizationSolutionComponents
SyncsortHigh-PerformanceDataMovement
HadoopScalableStorageandCompute
HiveLLAPHighPerformanceSQLDataMart
AtScaleIntelligencePlatformOLAPCubesforHigherPerformance
SourceDataSystems
Fast,scalableSQLanalyticsIntelligentin-memorycaching
DefineOLAPcubesfor10xfasterqueriesUnifiedsemanticlayerforallBItools
Highperformancedataimportfrom allmajorEDWplatforms
Pre-aggregateddata
...Or,full-fidelitydata
17 ©HortonworksInc.2011–2016.AllRightsReserved
EDWOptimization:FastBIonHadoop
à TheProblem:– ProprietaryEDWsystemswereadoptedfor
FastBIanddeepslice-and-diceanalytics,butEDWpricesareunsustainablyhigh.
à TheSolution:– InteractiveSQLisarealityonHadooptoday.– AtScaleIntelligencePlatformaddsOLAP
capabilitiesfordeepdrilldownatscale.
à TheResult:– Queryterabytesofdatainseconds.– ConnectyourfavoriteBItoolslikeTableauand
ExcelthroughSQLandMDXinterfaces.– TheEDWOptimizationSolutionistailor-made
todeliverFastBIonHadoop.
ETL/ELT
DATAMART
DATALANDING&
DEEPARCHIVE
CUBEMART
ENDUSER
APPLICATIONS
APPLICATIONS
APPLICATIONS
ENDUSERSANDAPPS
EDWOPTIMIZATIONSOLUTION
18 ©HortonworksInc.2011–2016.AllRightsReserved
EDWOptimization:ETLOffload
à TheProblem:– EDWsconsumebetween50%and90%of
CPUjustonETL/ELTtasks.– Thesejobsinterferewithmorebusiness-
criticaltaskslikeBIandadvancedanalytics.
à TheSolution:– HiveandHDPdeliverETLthatscalesto
petabytes.– SyncsortDMX-hforsimpledrag-and-dropETL
workflows.– Economicalscale-outprocessingon
commodityservers.
à TheResult:– BetterSLAsformission-criticalanalytics.– LimitEDWexpansionorretireoldsystems.
ETL/ELT
DATAMART
DATALANDING&
DEEPARCHIVE
CUBEMART
ENDUSER
APPLICATIONS
APPLICATIONS
APPLICATIONS
ENDUSERSANDAPPS
EDWOPTIMIZATIONSOLUTION
19 ©HortonworksInc.2011–2016.AllRightsReserved
EDWOptimization:ActiveArchive
à TheProblem:– Increasingdatavolumesandcostpressure
forcedatatobearchivedtotape.– Archiveddatanotavailableforanalytics,or
mustberetrievedatgreatexpense.
à TheSolution:– AdoptingHadoopdeliverscostperterabyte
onparwithtapebackupsolutions.– DatainHadoopcanbeanalyzedbyallmajor
BItools,allowinganalyticsonarchivedata.
à TheResult:– Dataalwaysavailableforanalytics.– Storeyearsofdataratherthanmonths.
ETL/ELT
DATAMART
DATALANDING&
DEEPARCHIVE
CUBEMART
ENDUSER
APPLICATIONS
APPLICATIONS
APPLICATIONS
ENDUSERSANDAPPS
EDWOPTIMIZATIONSOLUTION
PoweringtheConnectedDataPlatformWithEDWOptimization
Paige RobertsBig Data Product Manager@RobertsPaige
GoalsoftheModernDataArchitecture
•Centralizeallyourdata
•Turnrawdataintoinsights
•Maintaingovernance,complianceandsecuritystandards
•EliminatecomplexitieswithinIT
21
SyncsortStrategicFocusonBigData&Hadoop
Lightfootprint
Self-tuningengine
Singleinstall.No3rd partydependencies
World-classdataprocessing,mainframeexpertise
JIRA:MAPREDUCE-2454MAPREDUCE-4807MAPREDUCE-4049MAPREDUCE-5455HIVE-8347SQOOP-1272PARQUET-134Spark-packagesand more!
|
22Syncsort Confidential and Proprietary - do not copy or distribute
OngoingContributionstotheOpenSourceCommunity1
LeverageSyncsortTechnologyInnovations&MainframeHeritage2
StrongPartnershipswithStrategicBigData&HadoopPlayers
1
3
22
à Connecttovirtualanydatasource,includingmainframeandMPPdatabases.
à MovedataintoandoutofHadoopupto6xfasterwithouttheneedformanualscripts.
à DevelopETLprocesseswithoutwritingcode.
à AutomaticallyoptimizeHadoopperformanceandscalabilityforETLoperations.
à FullycertifiedandintegratedwithHortonworksDataPlatformandHDF
à Secure– Kerberos,Ranger,Sentry
SyncsortBenefits
Syncsort: High Performance Import from Existing Sources
BringALLEnterpriseDataSecurelytotheDataLake
24
• Collectvirtuallyanydatafrommainframetorelational,cloudandNoSQLsources
• Access,re-formatandloaddatadirectlyintoHive&Hadoopfileformats.Nostagingrequired!
• Batch&streamingsources
• Pullhundredsoftablesatonceintoyourdatahub,wholeDBschemasinoneinvocation
24Syncsort Confidential and Proprietary - do not copy or distribute
•LoadmoredataintoHadoopinlesstime
GetYourDatabasedataintoHadoop,AtthePressofaButton
25
• Pullmultipledatasourcesandfunnelintoyourdatalake• ExtractandmapwholeDBschemasinoneinvocation• Extractfrommultipledatasources:DB2/z,Netezza,Oracle,Teradata,…
• One-stepdatamovement,auto-generatingjobs,auto-generatingHivetargettables,andupdateHivestatistics
• Processmultiplefunnelsinparallelonyouredgenodeor fromdatanodes‒ LeveragesDMX-hhighspeeddataengineviaDTL‒ GeneratedapplicationscanbeimportedintoGUI
• In-flighttransformations‒ Filtering,funneldependencyordering,mixedsource/target,datatypefiltering,tableexclusion/inclusion
25Syncsort Confidential and Proprietary - do not copy or distribute
DMXDataFunnel™
IntelligentExecutionLayer
DesignOnce,DeployAnywhere
26
IntelligentExecution- InsulateyourpeoplefromunderlyingcomplexitiesofHadoop.
Oneinterfacetodesignjobstorunon:SingleNode,ClusterMapReduce,Spark,FuturePlatformsWindows,Unix,LinuxOn-Premise,CloudBatch,Streaming
• UseexistingETLskills.• Noworriesaboutmappers,reducers,bigside,smallside,andsoon.• Automaticoptimizationforbestperformance,loadbalancing,etc.• Nochangesortuningrequired,evenifyouchangeexecutionframeworks• Future-proofjobdesignsforemergingcomputeframeworks,e.g.Spark
Insurance:EasyAccesstoALLDataforBetterAnalytics
27
• Challenge: Neededhard-to-accessoperationaldataforadvancedanalytics
• Solution:• Quicklyload~1000databasetablesintoHDPwiththeclickofa
button• Access&integratecomplexMainframeVSAMfiles,datafrom
DB2/z,Oracle&SQLServer• Trackchanges&keepdatauptodate
• Benefits:• Insight: Betterandfasteranalytics• Agility: Reclaimdevelopmenttime;singletooltoingest,detectchangesandpopulatethedatalake• Compliance: Buildaudittrails,keepHDPdatalakecurrent• Productivity:NoneedfordeepunderstandingofHadoop
HotelChain:EaseofUse,Timely&Up-to-DateReporting
28
• Challenge: Moretimelycollection&reportingonroomavailability,eventbookings,inventoryandotherhoteldatafrom4,000+propertiesglobally
• Solution:• Nearreal-timereporting• DMX-hconsumespropertyupdatesfromKafkaevery10s• DMX-hprocessesdataonHDP,loadingtoTDevery30min• DeployedonGoogleCloudPlatform
• Benefits:• TimetoValue:DMX-heaseofusedrasticallycutdevelopmenttime• Agility:Reportsupdatedevery30minutesvsevery24hours• Productivity:LeveragingETLteamforHadoop(Spark),visualunderstandingofdatapipeline• Insight: Up-to-datedata=betterbusinessdecisions=happiercustomers
LeadingMediaCompany:AccelerateNewBusinessInitiatives
29
• Challenge: Buildscalableplatformtosupportnewbusinessinitiatives&scalefordouble-digitdatagrowth,whilereducingescalatingEDW&ELTCosts
• Solution:• Shiftdatastorage&processingoutoftheEDWintoHadoop• Migrate500+SQLELTworkloadstoDMX-honHDP
• Benefits:• Agility: Scalablearchitecturetodeploynewbusinessinitiatives– analyzemoresettopboxdata,blend
websiteuseractivitydata,etc.• Cost:MillionsofdollarsinsavingsfromEDW,includingSQLtuning&maintenancecosts• Productivity:ETLdeveloperscanstopcoding&tuning,andgetup&runningonHadoopquickly
30 ©HortonworksInc.2011–2016.AllRightsReserved
Q/A
LearnMore:Ã EDWOptimizationwithHDP
– http://hortonworks.com/solutions/edw-optimization/– EDWOptimization7minvideo
à TrySyncsort DMX-h:syncsort.com/try– Yourexistingclusteror useourfullyfunctionalHortonworksSandbox– Getajump-startwithourlibraryofpre-builtjobs,includingUseCaseAcceleratorsforHadoop
ETLandHDFSExtract&Load
à Syncsort DMX-h– http://hortonworks.com/partner/syncsort/
31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You
Top Related