Post on 05-Dec-2014
description
Setting up New Relic's Infrastructure: A bunch of things we did wrong and a few we did right.DAVE PETERSON & NIC BENDERS2013-10-25
Wednesday, November 6, 13
David Peterson(dave@newrelic.com)
Nic Benders(nic@newrelic.com)
Wednesday, November 6, 13
[“star” graph of code base at launch]
Wednesday, November 6, 13
Wednesday, November 6, 13
100,000req/sec
2Gbps
200TB
Wednesday, November 6, 13
Who Does What?
Wednesday, November 6, 13
Site Operations != IT
Wednesday, November 6, 13
Wednesday, November 6, 13
You are not Batman
Do your work during the dayWednesday, November 6, 13
Lowest hour is still 60% of peak hour!
Wednesday, November 6, 13
Put your serverson the moon
Wednesday, November 6, 13
Put your serverson the moon
Wednesday, November 6, 13
Never open the box
Wednesday, November 6, 13
Domain names are cheap
Wednesday, November 6, 13
10.0.0.0/31 10.0.0.0-10.0.0.1 255.255.255.254 210.0.0.0/30 10.0.0.0-10.0.0.3 255.255.255.252 410.0.0.0/29 10.0.0.0-10.0.0.7 255.255.255.248 810.0.0.0/28 10.0.0.0-10.0.0.15 255.255.255.240 1610.0.0.0/27 10.0.0.0-10.0.0.31 255.255.255.224 3210.0.0.0/26 10.0.0.0-10.0.0.63 255.255.255.192 6410.0.0.0/25 10.0.0.0-10.0.0.127 255.255.255.128 12810.0.0.0/24 10.0.0.0-10.0.0.255 255.255.255.0 25610.0.0.0/23 10.0.0.0-10.0.1.255 255.255.254.0 51210.0.0.0/22 10.0.0.0-10.0.3.255 255.255.252.0 102410.0.0.0/21 10.0.0.0-10.0.7.255 255.255.248.0 204810.0.0.0/20 10.0.0.0-10.0.15.255 255.255.240.0 409610.0.0.0/19 10.0.0.0-10.0.31.255 255.255.224.0 819210.0.0.0/18 10.0.0.0-10.0.63.255 255.255.192.0 1638410.0.0.0/17 10.0.0.0-10.0.127.255 255.255.128.0 3276810.0.0.0/16 10.0.0.0-10.0.255.255 255.255.0.0 65536
Private IPsare FREE
Wednesday, November 6, 13
Hardware
Wednesday, November 6, 13
System Management
Wednesday, November 6, 13
X6550 E5-2450 E5-2643 E5-2690
Average response time
137
ms
167
ms
158
ms
189
ms
Wednesday, November 6, 13
25% performance improvement
2x 6-Core X6550@ 2.67 GHz
2x 8-Core E5-2690@ 2.90 GHz
Wednesday, November 6, 13
Storage
√
Wednesday, November 6, 13
Databases
Wednesday, November 6, 13
32-bits isn't enoughclass CreateTribbles < ActiveRecord::Migration def change create_table :tribbles do |t| t.boolean :hungry t.string :color t.references :parent, index: true end endend
Wednesday, November 6, 13
32-bits isn't enough`id` int(11) NOT NULL AUTO_INCREMENT
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT
`id` SERIAL
Wednesday, November 6, 13
Your database is not a queue
Wednesday, November 6, 13
Long running querieswill kill you
Wednesday, November 6, 13
And now, a thanks to the people who are out there every day fixing the mistakes that we made:
Wednesday, November 6, 13
Jonathan Owens
Jonathan Thurman
Aaron Bento
Jonathan OwensJonathan Owens
Sean Kane
Paul Showalter
Karl Matthias
Wednesday, November 6, 13