Where is Data Going? - RMDC Keynote
-
Upload
ted-dunning -
Category
Technology
-
view
117 -
download
0
Transcript of Where is Data Going? - RMDC Keynote
© 2016 MapR Technologies 1© 2014 MapR Technologies
Where is Data Going?
Ted Dunning Chief Application Architect
© 2016 MapR Technologies 2
My Background• University, Startups
– Aptex, MusicMatch, ID Analytics, Veoh– big data since before it was big
• Open source– even before the internet– Calcite, Datafu, Drill, Kylin, Mahout, Myriad, Samoa, Singa, Storm,
Zookeeper– bought the beer at first HUG
• VP Incubator at Apache Software Foundation• MapR Chief Application Architect
© 2016 MapR Technologies 3
When you see this
Real tech is coming shortly
© 2016 MapR Technologies 4
IT Spending at an Inflection Point
Source: IDC, Gartner; Analysis & Estimates: MapRNext-gen consists of cloud, big data, software and hardware related expenses
2013 2014 2015 2016 2017 2018 2019 2020
(100,000)
(50,000)
-
50,000
100,000
150,000 Next Gen vs. Legacy Shrinkage ($M)
$120
100
80
60
40
20
(20)
(40)
(60)
(80)
(100)
In Billions
Total $ Growth of IT Market Next-Gen Growth Legacy Market Growth/Shrink in $
© 2016 MapR Technologies 5
You Need to Make a Decision Now
In 4 years 90% of Data will be on next-gen technology
© 2016 MapR Technologies 6
MapR Converged Platform
Enterprise Hadoop + Storage2011-2012
2013
2014
2015
2016
+ NoSQL
+ Interactive SQL
+ Global Event Streaming
+ …
Top Ranked Hadoop
Top Ranked NoSQL
Top Ranked SQL
© 2016 MapR Technologies 7
Innovatio
n
$$
$
Reduce
Costs
© 2016 MapR Technologies 8
Wait a minute!
What’s really going on?
© 2016 MapR Technologies 9
Evolution of Data Storage
FunctionalityCompatibility
Scalability
LinuxPOSIX
Over decades of progress,Unix-based systems have set the standard for compatibility and functionality
© 2016 MapR Technologies 10
FunctionalityCompatibility
Scalability
LinuxPOSIX
HadoopHadoop achieves much higher scalability by trading away essentially all of this compatibility
Evolution of Data Storage
© 2016 MapR Technologies 11
Evolution of Data Storage
FunctionalityCompatibility
Scalability
LinuxPOSIX
Hadoop
MapR enhanced Apache Hadoop by restoring the compatibility while increasing scalability and performance
FunctionalityCompatibility
Scalability
POSIX
© 2016 MapR Technologies 12
FunctionalityCompatibility
Scalability
LinuxPOSIX
Hadoop
Evolution of Data Storage
Adding tables and streams enhances the functionality of the base file system
© 2016 MapR Technologies 13
What is Convergence?
Files StreamsTables
© 2016 MapR Technologies 14
© 2016 MapR Technologies 15
What is Convergence?
Files StreamsTables
© 2016 MapR Technologies 16
What is Convergence?
Files StreamsTables
© 2016 MapR Technologies 17
Basic System Design
© 2016 MapR Technologies 18
Basic System Design
© 2016 MapR Technologies 19
© 2016 MapR Technologies 20
Consumer Behavior
© 2016 MapR Technologies 21
Consumer Behavior
© 2016 MapR Technologies 22
© 2016 MapR Technologies 23
Reduce
Costs
$$$
Innovatio
n
© 2016 MapR Technologies 24PEOPLE
1.2BPEOPLE
Largest Biometric Database in the World
© 2016 MapR Technologies 25
4 data centers
Synchronous replication EW
Asynchronous replication NS
© 2016 MapR Technologies 26
EW distance is short
© 2016 MapR Technologies 27
NS separation is large
© 2016 MapR Technologies 28
Summary• Updates can happen anywhere, anytime
• Consistency maintained by insertion timestamp
• System is robust to link, power, data-center failure
© 2016 MapR Technologies 29
© 2016 MapR Technologies 30
Tenants Share Access to Data
© 2016 MapR Technologies 31
Tenants Share Access to Data
© 2016 MapR Technologies 32
© 2016 MapR Technologies 33
© 2016 MapR Technologies 34
Streaming and Micro-services
© 2016 MapR Technologies 35
Streaming and Micro-services
© 2016 MapR Technologies 36
And just a slight bit of future tense here
© 2016 MapR Technologies 37
Financial Services Use Case• For reference assume:
– 1000 - 10,000 unique senders and receivers– each bid or offer includes 10 recipients on average – bids and offers arrive at a rate of 300k messages / second
• What would you do?
© 2016 MapR Technologies 38
Financial Services: Our Solution
© 2016 MapR Technologies 39
Massive IoT• 100 million cars• 2kB / second
• Roaming across data centers
© 2016 MapR Technologies 40
© 2016 MapR Technologies 41
Streaming First for Micro-Services• How do applications need to change?
• How can we get micro-service benfits (this time around)?
• How does our development need to change?
© 2016 MapR Technologies 42
Traditional Solution
© 2016 MapR Technologies 43
What Happens Next?
© 2016 MapR Technologies 44
What Happens Next?
© 2016 MapR Technologies 45
How to Get Service Isolation
© 2016 MapR Technologies 46
New Uses of Data
© 2016 MapR Technologies 47
Data Platform
Integrated Analytics
Enterprise Grade
Scale
Legacy Support
Transformational Applications
Real Time
© 2016 MapR Technologies 48
Let’s Review the Options
S T O R A G E
StorageIcon D A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1
0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1
© 2016 MapR Technologies 49
H A D O O P
Let’s Review the Options
D T A
C O M P U T E
D A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1
© 2016 MapR Technologies 50
Let’s Review the Options
C O M P U T E
S P A R K
D A T A
© 2016 MapR Technologies 51
Let’s Review the Options
C A S S A N D R A
D T A
C O M P U T E
D A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1
© 2016 MapR Technologies 52
Let’s Review the Options
C O M P U T E
D T AD A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1
Write In:–– –– –– –
O T H E R
© 2016 MapR Technologies 53
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Data in MotionData at Rest
C O M P U T E
D T AD A T A 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 10 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1
© 2016 MapR Technologies 54
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Data in MotionData at Rest
© 2016 MapR Technologies 55
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
CloudOn Premise
© 2016 MapR Technologies 56
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Legacy Next Generation
© 2016 MapR Technologies 57
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Real TimeBatch
© 2016 MapR Technologies 58
Platform Appeal
Reduce CostsDrive Innovation
Graphic showing business users,
© 2016 MapR Technologies 59
Platform Appeal for Architects and Administrators
1. Simplifies administration
2. Avoids cluster sprawl
3. Unifies security, data protection, disaster recovery
© 2016 MapR Technologies 60
Platform Appeal for Developers
Flexibility Agility Simplicity Real Time
© 2016 MapR Technologies 61
Integration of data-in-motion and data-at-rest supports continuous, low latency processing
Converged Data Platform
mapr.com/appblueprint
© 2016 MapR Technologies 62
Q & A@mapr maprtech
Engage with us!
MapR
maprtech
mapr-technologies