Real-time data analytics with Cassandra at iland
-
Upload
julien-anguenot -
Category
Data & Analytics
-
view
470 -
download
0
description
Transcript of Real-time data analytics with Cassandra at iland
![Page 1: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/1.jpg)
Real-time data analytics with C* at iland
Cassandra Tech Day @ Houston, TX!October 14th, 2014!
Julien Anguenot (@anguenot) !
![Page 2: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/2.jpg)
Agenda
• iland, iland platform and why C*?!• real-time data & domain constraints!• quick overview of iland C* deployment
![Page 3: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/3.jpg)
iland platform
![Page 4: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/4.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
19 8
8 3
Years delivering Years of delivering IT
ServicesYears cloud
infrastructure & disaster recovery expertise
IISO 27001 & SSAE16 global
data centers
Cloud-based Specializations:
Production; Test & Dev; DR
iland Internet Solutions
44
![Page 5: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/5.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
iland platform essentially
• data warehouse running across multiple data-centers!• monitoring (resource consumption / performance)!• billing!• alerting!• predictive analytics!• cloud management!• cloud services (backups, DR, etc.)!• desktop and mobile applications (iland portal app)
5
![Page 6: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/6.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014 6
![Page 7: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/7.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014 7
![Page 8: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/8.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014 8
![Page 9: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/9.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014 9
![Page 10: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/10.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014 10
![Page 11: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/11.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
Why did we choose Cassandra?
• MySQL and MongoDB attempts been big fails!• write latency (constant-time writes)!• distributed nature (multi-data centers)!• scalability, reliability, performance, availability!• sharding makes things complicated!• no master - slave means no SPOF!• simplicity!
11
![Page 12: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/12.jpg)
real-time data & domain constraints
![Page 13: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/13.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
Constraints
• write latency!• precision (used for billing)!• availability!• multi-data center!• tens of thousands of VMs !• agent-less!• pull (vs push)
13
![Page 14: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/14.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
Pipeline
• collection of real-time data!• store!• aggregation!• rollups!• processing!• alerting!• reporting!• querying
14
![Page 15: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/15.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
Data sources
• VMware infrastructure stack (each location)!• vCloud Director, vCenter, vShield Manager!
• network statistics!• AMQP, Syslog-ng, Web Services!• Salesforce!• Veeam, Zerto, more …
15
![Page 16: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/16.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
Real-time metrics!
• 20 seconds samples!• dozens of performance counters per entity (VM, VNIC, etc.)!• time series!
• (timestamp, value)!• metadata!
• entity (network, vm, etc.)!• unit, etc.!
• then 1min, 1h, 1d, 1w and 1m historical rollups
16
![Page 17: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/17.jpg)
Overview of iland C* deployment
![Page 18: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/18.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
C* iland’s cluster (1/2)
• one (1) C* cluster !• 6 data centers - 25+ nodes - one (1) keyspace
18
![Page 19: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/19.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
C* iland’s cluster (2/2)• each DC (1 or 2 racks of 3 nodes)!
• 3 nodes per location for writes!• replication Factor (RF) = 3 !• 3 nodes per location for API reads (Dallas, TX, London, UK, Singapore)!
• each node!• VM (vCenter powered) - Ubuntu 14.04 LTS w/ Cassandra base dpkg!• 16GB of RAM / 16 vCPUs!• currently: ~ 1TB of data per node!• not using SSD (yet)
19
![Page 20: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/20.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014 20
Reston, VALA,CA
Dallas, TX
US
Singapore
Asia
London,UK
Manchester,UK
EU
Data centers and replication (1/2)
![Page 21: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/21.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
Data centers and replication (2/2)
21
C* W
iland ReST API
iland core platform iland core platform
iland ReST API
C* R C* RC* W
C* R only deployed in: Dallas, TX - London, UK - Singapore
![Page 22: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/22.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
Access to data (read / write)
22
US portal
https://my.ilandcloud.com/
Citrix Netscaler (US)
EU portal Asia portal
US API EU API Asia API
![Page 23: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/23.jpg)
Julien Anguenot @ Cassandra Tech Day Houston 2014
At the application level?
• History!• POC: C* 1.2 + Astyanax (2013/02)!• V1: C* 2.0 + Astyanax w/ CQL3 over thrift (2013/06) !• Current version: C* 2.0 + DataStax CQL Java driver!
• Java / JEE cluster (Wildfly AS)!• AMQP / RabbitMQ!• Redis to cache up “hot” data (read latency)!• Python & DataStax CQL driver for cluster / data upgrades
23
![Page 24: Real-time data analytics with Cassandra at iland](https://reader033.fdocuments.us/reader033/viewer/2022052907/558eada81a28ab83708b473e/html5/thumbnails/24.jpg)
Slides !@ !
http://www.slideshare.net/anguenot/cassandra-tech-dayhou2014
http://www.iland.com/enterprise-cloud-services-portal/!@ilandcloud!@anguenot