Lessons Learned from Building and Operating Scuba

13
Lessons learned from building and operating Scuba Ciprian Gerea Facebook

Transcript of Lessons Learned from Building and Operating Scuba

Page 1: Lessons Learned from Building and Operating Scuba

Lessons learned from building and operating Scuba

Ciprian GereaFacebook

Page 2: Lessons Learned from Building and Operating Scuba

ODS

Scuba

events

metrics

livehistorical

Presto & Hive

Page 3: Lessons Learned from Building and Operating Scuba

Demo time

• Getting started– Writing to Scuba

Page 4: Lessons Learned from Building and Operating Scuba

What is Scuba

• Database– Real time ingestion & queries– Simple query model: rollups, no joins– Simple data model, flexible schema

• UI platform• Service– Runs its own ETL– Demand control

• Retention• Queries

Page 5: Lessons Learned from Building and Operating Scuba

Scribelogsfrom

serversScuba GUI

`scuba` CLI

Scuba gauge ScriptAlerts

combinedlogs

for each scribe

category

Tupperware

Ptail

manage perfpipe

tailer

Tailer

Data storagerockfortexpress.wildcard

SMC tier in PRN1

Scuba backend

Root aggregator

Leaf

adddirectly to

leaf servers

queries

results

SparkleTable insertion counts

Scuba system architecture

valid

ation

dataswarm

HiveToScuba

Page 6: Lessons Learned from Building and Operating Scuba

Scuba DB

• Data lives in tables• Columns can be: int/string/vector<string>• Can change schema on the fly.• Shared nothing storage in memory & flash• Data sharded at random• Only support rollup queries:

sum/avg/percentile.• Best effort queries: skip bad nodes.

Page 7: Lessons Learned from Building and Operating Scuba

Demo time

• Let’s run some queries• Customize the UI• ETL control

Page 8: Lessons Learned from Building and Operating Scuba

How we keep it running

• ODS metrics for everything• Scuba data sets for queries & subsystems we’re

actively debugging• Dashboards– Cubism is king!– Unidash for niche cases

• Active management of demand– Table size quotas– CPU load -> push to stream processing

Page 9: Lessons Learned from Building and Operating Scuba

Root cause for outages

• Other systems– Scribe– Hosting layer– Deployment mechanism

• Media failures: high on disk, low on flash.• Queries of doom• High load– DOS workloads– Load shedding bugs

Page 10: Lessons Learned from Building and Operating Scuba

Why it is successful

• Scuba’s niche: – Easy to get started– Fast <50ms P50 wall time– Smooth learning curve– UI is customizable (~1k custom presenters!!!)– Its flaws are acceptable• Not everyone needs transactions from the beginning• Users are OK with retrying queries

• Other tools don’t serve this niche well

Page 11: Lessons Learned from Building and Operating Scuba

What could be better

• Customers ask– More space– More consistent results– More expressive queries

• Sharding• Better persistent storage• Better support for time series

Page 12: Lessons Learned from Building and Operating Scuba

Q & A

Page 13: Lessons Learned from Building and Operating Scuba

Cubism Intro

• Horizon charts


Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lessons learned after 190M lessons served

Lessons learned after 190M lessons served

Lessons Learned fromLessons Learned from Structural ... learned davy beicker.pdf · Lessons Learned fromLessons Learned from Structural Engggineers Learning ... reduced size of the

Lessons Learned fromLessons Learned from Structural ... learned davy beicker.pdf · Lessons Learned fromLessons Learned from Structural Engggineers Learning ... reduced size of the

Lessons Learned Handbook · 2011-10-12 · • Lessons Learned, an adjective, describes anything related to a Lessons Learned procedure. E.g. Lessons Learned process, Lessons Learned

Lessons Learned Handbook · 2011-10-12 · • Lessons Learned, an adjective, describes anything related to a Lessons Learned procedure. E.g. Lessons Learned process, Lessons Learned

ASSESSMENT & LESSONS LEARNED FROM AFRICAN DIASPORA MARKETPLACEpdf.usaid.gov/pdf_docs/PDACT256.pdf · ASSESSMENT & LESSONS LEARNED FROM AFRICAN ... ASSESSMENT & LESSONS LEARNED FROM

ASSESSMENT & LESSONS LEARNED FROM AFRICAN DIASPORA MARKETPLACEpdf.usaid.gov/pdf_docs/PDACT256.pdf · ASSESSMENT & LESSONS LEARNED FROM AFRICAN ... ASSESSMENT & LESSONS LEARNED FROM

Lessons Learned and Lessons to Be Learned: Investment Law & … · 2014. 4. 30. · 1 Lessons Learned and Lessons to Be Learned: Investment Law & Development for Developed Countries

Lessons Learned and Lessons to Be Learned: Investment Law & … · 2014. 4. 30. · 1 Lessons Learned and Lessons to Be Learned: Investment Law & Development for Developed Countries

Detecting Financial Reporting Fraud – Lessons learned · PDF fileDetecting Financial Reporting Fraud – Lessons learned by ... Detecting Financial Reporting Fraud – Lessons learned

Detecting Financial Reporting Fraud – Lessons learned · PDF fileDetecting Financial Reporting Fraud – Lessons learned by ... Detecting Financial Reporting Fraud – Lessons learned

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

The scuba sutras: Ten lessons from under the sea

The scuba sutras: Ten lessons from under the sea

Startup Lessons Learned

Startup Lessons Learned

LEARNED LESSONS:

LEARNED LESSONS:

Lessons Learned Remote PM+ Cohort Integrated K+S · Lessons Learned Below are key lessons learned on providing Remote PM+ Training of Helpers. These lessons learned are based off

Lessons Learned Remote PM+ Cohort Integrated K+S · Lessons Learned Below are key lessons learned on providing Remote PM+ Training of Helpers. These lessons learned are based off

Lessons learned

Lessons learned

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

Networking lessons learned

Networking lessons learned

Lessons Learned, Opportunities and Innovations in …pdf.usaid.gov/pdf_docs/PBAAD103.pdf · Lessons Learned, Opportunities and Innovations in Human ... Lessons Learned, Opportunities

Lessons Learned, Opportunities and Innovations in …pdf.usaid.gov/pdf_docs/PBAAD103.pdf · Lessons Learned, Opportunities and Innovations in Human ... Lessons Learned, Opportunities

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

Lessons Learned

Lessons Learned

DDM UPDATE LESSONS LEARNED & LESSONS SHARED …

DDM UPDATE LESSONS LEARNED & LESSONS SHARED …