WSO2 Virtual Hackathon: Big Data in the Cloud Case Study

Lakmal WarusawithanaDirector - Cloud Architecture - WSO2 IncVice President, Apache Stratos

WSO2 Virtual Hackathon

● In celebration of our 10th year anniversary this year we have conducted WSO2’s first ever virtual hackathon.

● We had ten teams from around the world build a scalable solution that processes up to 4 billion real-time events within a short period of time, all on the cloud.

Big Data Set

● Used DEBS 2014 Grand Challenge Data set● The data set originated from 2125 smart plugs deployed

across 40 houses● Each household contained several smartplugs. Each smart

plug contained two sensors:○ A load sensor measuring current load with Watt as unit

○ A work sensor measuring total accumulated work since the start (or reset) with kWh as unit

● 4 Billion Events

Big Data Challenge

● Query 01: Load Prediction○ The goal of this query is to make load forecasts based on

current load measurements and those of recorded historical data

○ Tool - WSO2 CEP

● Query 02: Outlier○ The goal of this query is to find devices that have very high

data (outlying) readings○ Tool - WSO2 DAS

Cloud Challenges● Hosting scalable WSO2 CEP and DAS deployment in

the cloud● Hosting across 5 EC2 Region, 100 EC2 Instances● Automating deployment across 100 instances● Using 300 Docker Containers● Placing monitoring system on

○ cluster level○ node level○ container level

● Auto Healing and recovering

Hackathon Deployment Architecture

Technologies used for CEP Cluster

● WSO2 CEP● Apache Storm● Nimbus● Zookeeper

WSO2 CEP

WSO2 CEP and Apache Storm

Apache Storm Cluster

Distributed WSO2 CEP Deployment on WSO2 PPaaS

Technologies used for DAS Cluster

● WSO2 DAS● Apache Spark● Apache HBase● HDFS● Zookeeper

WSO2 DAS

Apache HBase Cluster

Distributed DAS Deployment on WSO2 PPaaS

Technologies used for Cloud deployment

● Apache Stratos/WSO2 PPaaS● EC2● Kubernetes● CoreOS● Flannel

WSO2 PPaaS

Kubernetes

Technologies used for Monitoring system

● Kube-UI● cAdvisor● Heapster● InfluxDB● Grafana

Monitoring K8S Cluster Usage - Kube UI

Monitoring Container’s Resource Usage - cAdvisor

Monitoring Aggregated Resource Usage - Grafana

Some Hackathon moments...

Winning Team!!

More info...

● http://virtual15.wso2hackathon.com/

● http://lakmalsview.blogspot.com/2015/10/wso2-virtual-hackathon-big-data-in.html

Thank You