Scaling Wix with microservices architecture and multi-cloud platforms - Reversim Summit 2015
Big data (reversim)
-
Upload
nati-shalom -
Category
Documents
-
view
668 -
download
0
description
Transcript of Big data (reversim)
![Page 1: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/1.jpg)
Thank you
![Page 2: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/2.jpg)
Big Data In the
Cloud@natishalom
![Page 3: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/3.jpg)
3
About GigaSpaces
Managing Big Data on the Cloud
100’s of Enterprise Customers
![Page 4: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/4.jpg)
My Data Out of My
hands..
No Way!
![Page 5: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/5.jpg)
5
The Reality of Big Data..
2.7 ZB
0.5 Petabytes
66%
Global Digital Data
Two years tweets
Plan to use Big Data/Cloud
43% think that data
analytics could be improved in their organization if data analytics was part of
cloud services
![Page 6: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/6.jpg)
Large ISV Case Study
• Application– Call Center surveillance
• Background– Previously – voice data
• Goal for a new system– Monitor data & voice– Multiple data sources – Advanced correlations
![Page 7: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/7.jpg)
The Challenges..
Ever Growing Data
Deeper Correlation
Tight Performance
![Page 8: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/8.jpg)
A Classic Case for..
![Page 9: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/9.jpg)
A Typical Big Data System…
![Page 10: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/10.jpg)
The Challenge
Cost Business Impact
Lower Margins
Competiveness
Time to Market
Customer Satisfaction
Infrastructure
Operational
![Page 11: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/11.jpg)
The Solution Big Data
in the Cloud
![Page 12: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/12.jpg)
Big Data in the Cloud- 3 Reasons
• Skills– Do you really need/want this all in-
house?• Huge amounts of external data. – Does it make sense to move and
manage all this data behind your firewall?
• Focus on the value of your data– Instead of big data management.
Holger Kisker
![Page 13: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/13.jpg)
Managing Big Data on the
Cloud
• Auto start VMs• Install and configure
app components • Monitor • Repair • (Auto) Scale• Burst…
![Page 14: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/14.jpg)
Big Data in the Cloud..
Reduce the Infrastructure Cost
Choose the Right Cloud for the Job
Running Bare-Metal for high I/O workloads, Public cloud for sporadic workloads..
![Page 15: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/15.jpg)
Big Data in the Cloud ..
Reducing The Operational Complexity
• Consistent Management
• Automation Through the Entire Stack
![Page 16: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/16.jpg)
Lets Take a Closer Look …
![Page 17: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/17.jpg)
Before we Begin.. We’ll Need to Break
Some Common Myth’s on Portability
![Page 18: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/18.jpg)
Cloud Portability Myth #1
No one really needs cloud portability
![Page 19: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/19.jpg)
Cloud Portability
Facts
Zynga moved ~80% of their workload from Amazon to their private zCloud
“own the base, rent the spike”
http://code.zynga.com/2012/02/the-evolution-of-zcloud/
![Page 20: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/20.jpg)
Cloud Portability
Facts Started with Linode, then moved to RackSpace, then to AWS
http://code.mixpanel.com/2010/11/08/amazon-vs-rackspace/
![Page 21: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/21.jpg)
Cloud Portability
Facts
• You want the flexibility to choose what’s right for you, when it’s right for you
• Based on pricing, features, availability, performance, etc.
![Page 22: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/22.jpg)
Cloud Portability Myth #2
Cloud Portability ==
Cloud API Standardization
![Page 23: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/23.jpg)
Cloud APIs, Today
Standard APIs (?)OCCIVCloud
OSS FrameworksOpenStackCloudStackEucalyptus
Abstraction frameworksJCloudsDeltacloudFogLibvirt
![Page 24: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/24.jpg)
Cloud APIs, Today
Standard APIsNot practical in the foreseeable future
OSS Projects Need a couple more years to converge &
mature
Abstraction FrameworksProbably the only
practical (near-term) option
![Page 25: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/25.jpg)
Realization:
What You Really Care
about Is App
Portability
OS is the same on any cloud
Most clouds have compute & storage
Elasticity & scaling have same effects on the app, regardless of the cloud
![Page 26: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/26.jpg)
And now to a Closer
Look …
Consistent Management
Portability
Automation
![Page 27: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/27.jpg)
® Copyright 2012 GigaSpaces Ltd. All Rights Reserved
27
Consistent ManagementRecipes consistent description for running any app:
What middleware services to run Dependencies between services How to install services Where application and service binaries are When to spawn or terminate instances How to monitor each of the services.
![Page 28: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/28.jpg)
The Right Cloud for the Job (Cloud
Portability)
![Page 29: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/29.jpg)
® Copyright 2012 GigaSpaces Ltd. All Rights Reserved
29
Choosing the Right Cloud for the Jobcompute { template "SMALL_LINUX"}
SMALL_LINUX : template imageId "us-east-1/ami-76f0061f“ remoteDirectory "/home/ec2-user/gs-files“ machineMemoryMB 1600 hardwareId "m1.small" locationId "us-east-1" localDirectory "upload" keyFile "myKeyFile.pem"
options ([ "securityGroups" : ["default"]as
String[], "keyPair" : "myKeyFile"])
overrides (["jclouds.ec2.ami-query":"",
"jclouds.ec2.cc-ami-query":""])privileged true
}
SMALL_LINUX : template{ imageId "1234" machineMemoryMB 3200 hardwareId "103" remoteDirectory "/root/gs-files" localDirectory "upload" keyFile "gigaPGHP.pem" options ([ "openstack.securityGroup" : "default", "openstack.keyPair" : "gigaPGHP"
])privileged true
}
![Page 30: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/30.jpg)
Automation across the stack1 Upload your recipe.
2 Cloudify creates VM’s & installs agents
3 Agents install and manage your app
4 Cloudify automate the scaling
![Page 31: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/31.jpg)
Big Data Apps, on Any Cloud, Your Way
Open source (Apache2)
![Page 32: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/32.jpg)
32
Demo Time – Storm on Demand..
![Page 33: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/33.jpg)
Other Similar Solutions…
![Page 34: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/34.jpg)
RightScale
![Page 35: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/35.jpg)
Amazon Elastic Map Reduce
![Page 36: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/36.jpg)
Large ISV Case Study
• Application– Call Center surveillance system
• Background– Previously – voice data
• Goal for a new systemMonitor data & voiceMultiple data sources Advanced correlations Mission
Accomplished
![Page 37: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/37.jpg)
Additional Benefits
• True Cloud Economics
• One product -> any Customer Environment
• Increased Agility
![Page 38: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/38.jpg)
Try a simple Big Data Demo Yourself
The app
The Cloudify dashboardlaunch.cloudifysource.org/d
![Page 39: Big data (reversim)](https://reader034.fdocuments.us/reader034/viewer/2022042813/54b71ce64a795903798b4690/html5/thumbnails/39.jpg)
Thank You!
References: http://www.cloudifysource.org http://github.com/CloudifySource