Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large...
Transcript of Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large...
![Page 1: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/1.jpg)
Building/Administering Large DB ClustersLinuxCon Europe 2012
Who are Palomino?Bespoke Services: we work with and like you.Production Experienced: senior DBAs, admins, and engineers.24x7: globally-distributed on-call staff.No-lock-in contracts.Professional Services (DevOps):
➢ Chef,➢ Puppet,➢ Ansible.
Big Data Cluster Administration (OpsDev):➢ MySQL, PostgreSQL,➢ Cassandra, HBase,➢ MongoDB, Couchbase.
![Page 2: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/2.jpg)
Building/Administering Large DB ClustersLinuxCon Europe 2012
Who am I?Tim EllisCTO/Principal Architect, Palomino
Achievements:➢ Palomino Big Data Strategy.➢ Datawarehouse Cluster at Riot Games.➢ Back-end Storage Architecture for Firefox Sync.➢ Led DB teams at Digg for four years.➢ Harassed the Reddit team at one of their parties.
Ensured Successful Business for:➢ Digg, Friendster,➢ Riot Games,➢ Mozilla,➢ StumbleUpon.
![Page 3: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/3.jpg)
Questions?
Ask questions during presentation.No need to hold your questions until the end.
![Page 4: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/4.jpg)
Building/Administering Large DB ClustersWhat is this Talk?
Building a Large Database Cluster➢ Practical Concerns➢ The tools and choices
Monitoring the Cluster➢ How distributed DBs are different➢ Getting the data and acting on it
Rules of Thumb➢ How to size your cluster➢ Cluster architecture
![Page 5: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/5.jpg)
Prerequisite: Build a Large ClusterAllocating the Hardware
Getting Hardware – your own company's:➢ Can be politically-charged.➢ Get a small batch first.➢ Build small demonstration cluster.➢ Get everyone on-board with the demo.
Renting/Leasing Hardware – the Cloud:➢ Allocate hardware in EC2 or elsewhere.➢ Usually easier, but possibly harder admin:
➢ Hardware failure more common.➢ Hardware/network flakiness more common.
![Page 6: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/6.jpg)
Prerequisite: Build a Large ClusterBuilding the Cluster
Okay, I've got the hardware. What next?
![Page 7: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/7.jpg)
Prerequisite: Build a Large ClusterBuilding the Cluster
Configuring the Hardware. The old dilemma:➢ Spend days to install/configure DB software?
Subsequent management is painful.➢ Use SSH in “for” loops?
Rolling your own configuration management tools is a lot of work.
➢ Learn a configuration management tool?Obvious choice in 2012. Well-documented tools like Chef, Puppet, Ansible.
![Page 8: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/8.jpg)
Configuration Management ToolsMy Experience
Puppet: 6 years ago at Digg➢ Manage/Deploy of hundreds of servers.➢ Painful, but not as bad as hand-coding it all.
Chef: 2 years ago at Drawn to Scale and Riot➢ Manage/Deploy dozens of servers.➢ Learning Ruby is a “joy” of its own.
Ansible: 6 months ago at Palomino➢ Manage/Deploy dozens of servers.➢ First Palomino Cluster Tool subset built.
![Page 9: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/9.jpg)
Prerequisite: Build a Large ClusterConfiguration Management Options
Pick your Configuration Management:➢ Chef: Popular, use Ruby to “code your
infrastructure.” Must learn Ruby.➢ Puppet: Mature, use data structures to “define
your infrastructure.” Less coding.➢ Ansible: Tiny and modular, similar to Puppet,
but with ordering for deployment. Pragmatic.Write/Get Recipes, Manifests, Playbooks?➢ Writing is tedious. Can take >1 week.➢ Get from internet? Often incomplete.
![Page 10: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/10.jpg)
Prerequisite: Build a Large ClusterThe Palomino Cluster Tool
Palomino's tool for building large DB clusters:➢ Chef, Puppet, Ansible modules.➢ Open-source on Github.
➢ https://github.com/time-palominodb/PalominoClusterTool
➢ Google: “Palomino Cluster Tool.”➢ Will build a large cluster for you in hours:
➢ HA Master(s)/Slaves – hundreds as easy as two➢ HBase fully-distributed mode
➢ Previously this would take days.
![Page 11: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/11.jpg)
The Palomino Cluster ToolBuilding the Management Node
Cluster Management Node:➢ Will build the initial cluster.➢ Will do subsequent cluster management.
Tool for Initial Cluster Build:➢ Palomino Cluster Tool (Ansible subset).
Tools for Trending and Alerting:➢ Graphite or OpenTSDB➢ Nagios or Icinga
![Page 12: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/12.jpg)
The Palomino Cluster ToolBuilding the Management Node
Palomino Cluster Tool (Ansible subset).
Why Ansible?➢ No server to set up, simply uses SSH.➢ Easy-to-understand non-code Playbooks.➢ Use a language you know for modules.➢ For demo purposes, obvious choice.➢ Also production-worthy:
➢ Built by Michael DeHaan, long-time configuration management guru.
![Page 13: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/13.jpg)
The Palomino Cluster ToolBuilding the Management Node
Management node lives alongside your cluster.➢ We are building our cluster in EC2.➢ Thus management node in EC2.➢ This tutorial assumes Ubuntu 12.04.➢ t1.micro is fine for management node.
Install basic tools:➢ apt-get install git (for Ansible/P.C.T.)➢ apt-get install make python-jinja2 (for
Ansible)
![Page 14: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/14.jpg)
The Palomino Cluster ToolConfiguring the Management Node
Install Ansible:➢ git clone git://github.com/ansible/ansible.git➢ make install
Install Palomino Cluster Tool:➢ git clone git://github.com/time-
palominodb/PalominoClusterTool.git
I think we just finished the management node!
![Page 15: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/15.jpg)
Picking a Distributed DBMSThe Single Point of Failure?
Typical Reasons Clusters Fail:➢ Cascading failure (distributed fail)➢ Network failure (distributed fail)➢ Bad query executed (distributed fail)➢ NameNode failing? (single point of failure)
NameNode failure is not typical cause of cluster failure. Still, it's good to plan for it:➢ All critical filesystems RAID 1+0➢ Redundant PSUs and NICs
![Page 16: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/16.jpg)
Building an HBase Distributed DatabaseHardware and Architecture
NameNode as mentioned: highly redundant.
All other nodes: commodity hardware.➢ RAID-0 or, preferred, JBOD.➢ Spindles++: 8HDD in 1U good starting point.➢ 7200RPM SATA: nice, 15KRPM: overkill.➢ Many TB of storage. ←lots of this!➢ 8-24GB RAM.➢ Good/fast/multiple NICs.
Hadoop/HBase want lots of disk & network.
![Page 17: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/17.jpg)
Building a Distributed DBMSNetwork and Rack Considerations
Network Within the Rack (Top-of-Rack Switch)➢ Bandwidth for 30 machines going full-tilt.➢ Multiple TOR switches for redundancy.➢ Bridging on nodes.
Network Between Racks➢ Better than 2GB desireable.➢ Network instability causes cluster instability.
Enlist help of your in-house Networking Pros.
![Page 18: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/18.jpg)
Monitoring a Distributed DB ClusterPicking a Trending Tool
Tool must allow correlation of statistics.➢ Pick any N stats,➢ Put on a graph of log/linear scale,➢ Pick colours of each stat.
Tools that have these characteristics:➢ OpenTSDB,➢ Graphite,➢ Others?
![Page 19: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/19.jpg)
Monitoring a Distributed DB ClusterTrending
Which stats should I capture? In doubt?Graph all of them.➢ Every Hadoop statistic,➢ Every HBase statistic,➢ Every OS-level statistic.
How?➢ CollectD has JMX plugin.➢ HBase/Hadoop have Ganglia stats export.➢ Ganglia/gmond can store into Graphite.
![Page 20: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/20.jpg)
Monitoring a Distributed DB ClusterDistributed Databases are Different
Cross-node Correlation of Events:➢ Node X instability? Could be Node Y's fault.➢ ERRORs across all nodes?➢ Correlation of WARNINGs and ERRORs?➢ Log events correlate to graph anomolies?➢ Size of error logs change at new rate?
Outliers cause problems:➢ Slow nodes causing cascading failures.➢ Network instability causing cluster failure.
![Page 21: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/21.jpg)
Troubleshooting Distributed DB ClusterBy Scientific Method: Procedure
Problems on the cluster?
Formulate hypothesis from input:➢ Graphs➢ Logs
Test hypothesis (tweak config)
Check you're graphing everything and go to the start.
![Page 22: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/22.jpg)
Distributed DB Cluster TrendingGraphing your Logs
You need to graph everything. Are you graphing your logs?
➢ grep ERROR | cut [dt/hr part] | uniq -c
2012-07-29 06 156922012-07-29 07 304322012-07-29 08 769432012-07-29 09 549552012-07-29 10 15652
That's close, but what if it's hundreds of lines? Can use spreadsheet, but slows iteration cycle.
![Page 23: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/23.jpg)
Distributed DB Cluster TrendingGraphing your Logs
Graphing logs (terminal output) easier with Palomino's terminal tool “distribution,” OSS on Github:
# grep ERROR | cut <date/hour part> | distribution
2012-07-29 06|15692 ++++++++++ 2012-07-29 07|30432 +++++++++++++++++++ 2012-07-29 08|76943 ++++++++++++++++++++++++++++++++++++++++++++++++ 2012-07-29 09|54955 ++++++++++++++++++++++++++++++++++ 2012-07-29 10|15652 ++++++++++
On a quick iteration cycle in the terminal, this is very useful. For presentation to the suits later you can import the data into another prettier tool.
![Page 24: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/24.jpg)
Distributed DB Cluster TrendingGraphing your Logs
You want to characterise your logs.➢ How many ERRORs per hour?➢ How many WARNINGs per hour?➢ How many log lines per hour?
Look for patterns and ratios.➢ Alert on deltas.
System is imperfect, but it's a good start.➢ Good start >> Unstarted perfection.
![Page 25: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/25.jpg)
Distributed DB Cluster AlertingTools
The tools are already well-known:➢ Whatever you already use probably works.➢ Nagios/Icinga are very capable.
Alerting rules are simpler than RDBMS:➢ Daemon not responding on port?
And more complex:➢ Increased ERROR/WARNING frequency?➢ Different CPU/Network characteristics?
![Page 26: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/26.jpg)
Administering HadoopRules of Thumb
Don't let cluster get >70% full.➢ Disk throughput suffers.➢ HBase compactions slower or impossible.
Watch your network!➢ Network saturated? Perhaps reduce-heavy.➢ Disks saturated? Perhaps map-heavy.
Logs have WARNINGs and even ERRORs.➢ Act only if ERRORs translate to problems.
![Page 27: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/27.jpg)
Administering any Distributed DBMSRules of Thumb
The Cloud is flaky:➢ Dramatically variant performance (>30x!).➢ Cascading failures more common.➢ Cannot choose network topology.
EC2 Brazil is currently most stable.
EC2 US-East is currently most flaky.
![Page 28: Building/Administering Large DB Clusters LinuxCon Europe 2012€¦ · Building/Administering Large DB Clusters LinuxCon Europe 2012 Who are Palomino? Bespoke Services: we work with](https://reader033.fdocuments.us/reader033/viewer/2022053000/5f0466787e708231d40dc916/html5/thumbnails/28.jpg)
Building/Administering Large DB ClustersQ&A
Questions? Suggestions:➢ Interesting stuff. Got a job for me?➢ Well I got a job for you. Interested?➢ Average flight speed of a laden sparrow?➢ What's the meaning of Donnie Darko?
Thank you! Emails to domain palominodb, username time. LinuxCon Europe 2012 in Barcelona. Enjoy the rest of the show!