RT Analytics with Storm and Redis @ Big Data Bucharest Meetup
19
RT Analytics with Data Streaming Felix Crisan / Co-Founder & CTO NETOPIA [mobilPay, web2sms, kartela.ro]
-
Upload
fixone -
Category
Technology
-
view
232 -
download
2
description
RT Analytics with Storm and Redis @ Big Data Bucharest Meetup
Transcript of RT Analytics with Storm and Redis @ Big Data Bucharest Meetup
RT Analytics with Data Streaming Felix Crisan / Co-Founder & CTO NETOPIA[mobilPay, web2sms, kartela.ro]
About Us
- Largest SMS Aggregator in RO (+100MM/year) - Largest mobile payments processor in RO (own wallet and SMS)- Second largest online card processing gateway in RO (+2M trx / year)
RT Analytics @ mobilPay
Is it BigData?
What do we use it for
- Operational Insights- Transaction Monitoring- Health Checks- Business Dashboards (internal for now, GA later this year)
Storm one-pager
- Streaming processor, Apache Incubator (http://storm.incubator.apache.org/)- Zookeeper/Nimbus(+ZeroMQ/Netty)- Scalable / No Data Loss (guaranteed)- Fault - Tolerant- Topologies (~ MR job) / Streams / Spouts (data sources) / Bolts (data processors)
Redis one-pager
- In-Memory (disk persistent) KV Store (http://redis.io)- Values can be complex types (strings, hashes, lists, sets, sorted sets)- Single-threaded (!) but high-performance (500k req/s)- Atomic operations
(Simplified) Data Model
Our Data Streaming setup
What Happens
- Transaction details are put in a RabbitMQ queue (state, errors, values etc)- A Storm Topology reads the details and aggregates
- per Merchant / Service - per Status / Error / Currency- per time interval (5min, H, D, M)
Why did we choose this setup?
So what’s the result?
Know Your Queries
Better insights
Clear(er) views
Better predictions
KYC
Summary
