July 2014 HUG : Pushing the limits of Realtime Analytics using Druid

Yahoo! Presentation, Confidential 16 July 2014

July 16, 2014

Pushing the limits of Realtime analytics with Druid

Reza IranmaneshSrikalyan Chandrashekar

By realtime we mean subsecond response, highly concurrent and realtime ingestion too


Agenda

1. What is Druid ?2. Fitting Druid into our Software Stack; Druid in the API layer3. The SQL 4 Druid suite (compiler, driver and client).4. Contrast with traditional RDBMS SQL.5. Features of SQL 4 Druid suite.6. Current state.7. Demo.8. Future plans.


What is Analytics?

Analytics The process of accessing, cleaning, transforming and modeling data with the goal of discovering information.

Business Intelligence analytics focused on business information.

Star Schema

Data Cubes / OLAP Systems


What is Druid?- A google dremel inspired, open source OLAP-like engine that

can do aggregate operations in sub second (most of them) on memory mapped data.

- de-normalized data

- Time-based segments

- Timeseries/GroupBy/TopN

- Plays well with Hadoop


What is Druid?Lambda Architecture in a box – a relatively self-contained system


Main ComponentsIndexing Service• Realtime ingestion• Hadoop batch ingestion• Local batch ingestion• Aggregates are defined at indexing stage• Final output: segments of data that will eventually live on deep store. Each segment

holds a timerange of data.

Coordinator Node• The Coordinator! Takes care of reading metadata from MySQL and looking at zookeeper to see who’s

there and putting segment distribution information for Historicals to pick, etc.

Broker Node• Forwards the queries to the nodes who have the segments that fall into the given time interval• Takes care of aggregating the partial aggregates from historical nodes

Historical Node• Loads segments of immutable indexed data that live on the deep store (usually Grid)• Each historical node

Realtime Node


Why Druid?Pros:• Horizontal Scalability with linear performance gains• Sub-second response time for most use cases• Native time zone support• Relatively self-contained (batch + real-time ingestion/query support,

distributed memcached support, multi-tier replication/load support)• Active and responsive community

Cons:• Limited query power compared with SQL/MDX• Lack of joins• Missing distinct count• Memory bounds for GroupBy Query


Horizontal Scalability


Some Numbers


Druid in our API LayerLimitation Solution

groupBy memory bounded - Replace with TopN- Not a real limit in Analytics world

Limited Query power - Extend Druid’s limits- Some extra work in the API

layer - Implement a SQL layer

Missing exact Distinct Count - Use HyperLogLog implementation

- Create pre-aggregates on the grid when we need exact values; create a multiplexer on the API side

The problem of mutable dimension data; supporting star schema

- Query pipeline in the API layer

Missing joins - SQL layer


GridRDBMS

Druid

Druid in our API LayerStar Schema

Solving the problem of mutable dimension tables with a Druid query pipeline and one-to-one joins:filter on pre-aggregates / join on post-aggregates

- uuid- name- url

- uuid- name- url- views- clicks

First Druid groupby

First result as input to the next Query

Select sum(uuid), ..From …Join …

Where …

API


Druid in our API LayerUnique Counts

Unique_count_1_hour Unique_count_7_dayUnique_count_24_hour Unique_count_14_day

Unique_count_30_day

use hyperLogLogaggregate

What is the granularity?hour? day? all?

Hybrid approach to take care of distinct counts:


SQL 4 Druid suite

1. JDBC driver 2. Command line client

the driver is powered by the DCompiler.


Demo


Sql features

1. GroupBy, Having clause supported.2. Post aggregation including javascript functions accepted.3. Order BY and LIMIT which essentially is Top N.4. Where clause translates into filters.5. Aggregators: count, double_sum, long_sum, unique, max, min etc.6. BREAK BY translates into granularity.7. HINT timeseries if dimension is timestamp.


Sql features continued

8. Specify micro and adhoc interval ranges.

9. Select-type query(no aggregation) just plain dimension (and/or) metrics retrieval.


Sample SQLs


Driver features.

1. Can do JOIN(Inner, Left and Right), can go only two level deep right now.2. Template parameters.


Driver features continued.

3. Map data to list .

4. Map data to bean.

5. Dynamic column type inference on select queries(Ex: dimension/metric is found from Coordinator)


Client Features.

1. GroupBy,TS and TopN, search queries.

2. Generate Bean source code based on previous SQL executed.

3. See all tables(data sources), schema of table etc.

4. Navigate through command history.

5. MySQL like pretty print.


We Are Hiring

[email protected]@yahoo-inc.com

Contact us:

mailto:[email protected]

mailto:[email protected]


Appendix ATimeseries with adhoc interval spec


Appendix B

Select Query on dimensions and metrics


Appendix CGroupBy With Join

July 2014 HUG : Pushing the limits of Realtime Analytics using Druid

Documents

Transcript of July 2014 HUG : Pushing the limits of Realtime Analytics using Druid