T I M E L I N E S E R V I C E N E X T G E N ( YA R N - 2 9 2 8 )

WHY NEXT GEN?

Scalability Single global instance of writer/reader v.1 uses a local-disk-based LevelDB storage instance

Usability Handle flows as first-class concept and model aggregation Elevate configuration and metrics to first-class members

Existing external tooling: hRaven, Finch, Dr. Elephant, etc.

KEY DESIGN POINTS

Distributed writer architecture Scalable storage backend (HBase) Reimagined object model API with flows built into it Separated reader instances Aggregation

DISTRIBUTED WRITERS & READERS

!melinereader

!melinereader

Storage

!melinereader

AM !melinewriter

NM

!meline reader pool

app metrics/events

container events/metrics

RM!meline writer

app/container events

user queries

STATUS

[DONE] timeline writers (per-app and per-node) as aux service [DONE] RM companion writer [DONE] first iteration of the object model API [DONE] file-based test writer [DONE] NM writing container events [DONE] RM writing app/container entities [DONE] AMs writing framework-specific events and metrics [DONE] first versions of Phoenix and HBase writer impls [DONE] performance benchmarking evaluation of writers

STATUS

[WIP] timeline readers [WIP] aggregation UI enhancements Stand-alone timeline writer (per-node and per-app) Finalize implementation of supported queries Security Migration/compatibility story …

TEAM

This is a true community collaboration! Sangjin, Vrushali and Joep (Twitter) Zhijie, Li, Junping and Vinod (Hortonworks) Naga and Varun (Huawei) Robert and Karthik (Cloudera) Input from LinkedIn, Yahoo! and Altiscale

QUESTIONS?