Post on 13-Jan-2017
Big Data and the CloudShekhar Vemuri
#phxdataconference
ABOUT
• PRINCIPAL at CLAIRVOYANT
• PRODUCT, DATA, ANALYTICS and CLOUD
• large scale web and data systems
• simple, lightweight solutions
QUICK POLL
• HADOOP, HIVE, PIG
• PUBLIC CLOUD, IaaS, SaaS
• AMAZON AWS, EC2
• ELASTICITY
• S3, EMR, KINESIS
• IoT
WHAT WILL WE TALK ABOUT
BIG DATA
USE CASES
RISK MODELING PERSONALIZEDMEDICINE AD TARGETING
INTERNET OF THINGS
THREAT ANALYSIS
RECOMMENDATIONS
SURVEILLANCE RETENTION 360 CUSTOMERVIEW
DRIVING FACTORS
• variety in data
• not just transactional data
• potential for tremendous insight - when combining transactional data with additional data sources
• LinkedIn, Twitter, Facebook, Pinterest , Open Data
• Internet of Things
the CLOUD
the CLOUD
• IaaS, SaaS
• on demand subscription
• subscription vs owning
• tradeoff
• ease of adoption
• powering nextgen entrepreneurship
LANDSCAPE
DATA VALUE CHAIN
1010101011010101010101010101010101010101010101010101010101GENERATE STORE ANALYZE INSIGHTS
> > >
DATA VALUE CHAIN
ingest transform transform
BIG DATA + the CLOUD
LOG ANALYSIS
AMAZON S3
AMAZON EC2
LOG FILES
ReST CLIENTS
WEB APP, REST APIs
AMAZON EMR
AMAZON S3
AMAZON EC2
LOG FILES
ReST CLIENTS
WEB APP, REST APIs
AMAZON REDSHIFT
LOG FILES - STORED in S3
MAP-REDUCE, HIVE, PIG, CASCADING jobs
STORE summarized data
AMAZON EMR
AMAZON S3
AMAZON EC2
LOG FILES
ReST CLIENTS
WEB APP, REST APIs
LOG FILES - STORED in S3
MAP-REDUCE, HIVE, PIG, CASCADING jobs
CLOUDERA IMPALA
AMAZON S3
AMAZON KINESIS
AMAZON REDSHIFT AMAZON DYNAMODBAMAZON RDS
AMAZON EMR
AMAZON S3
DATA
AMAZON S3
INPUT
AMAZON EMR
AMAZON S3
INPUT
OUTPUT
AMAZON EMR
AMAZON S3
INPUT
OUTPUT
AMAZON EMR
AMAZON EMRWITH SPOT instances
BUILDING BLOCKS
• amazon AWS
• amazon EMR
• amazon S3
• kinesis
• redshift
• spot instances
HEADER
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud.
SUBHEADER
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
PROS
• like other cloud solutions - reduces the barrier to adoption
• especially if you are already in the cloud
• can provide ability to implement quick POCs
HEADER
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud.
SUBHEADER
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Category 4
Category 3
Category 2
Category 1
0 1.3 2.5 3.8 5 6.3
Series 1 Series 2Series 3
CONS
• depending on your current infrastructure - may end up continually replicating data
• data security, privacy
LEARNINGS
• Build platforms once the need is strongly felt
• Prepare to Fail fast, couple of times before the final version
• what you think will happen, will not
LEARNINGS
• COSTS can spiral out of control
• Leverage spot instances to reduce costs, especially for bursty workloads
• S3 Can be very slow to run and initialize large workloads
• especially in recovery scenarios
• but data resiliency is not an issue
www.clairvoyantsoft.com
@shekharvlinkedin.com/in/shekharvemuri