Capacity Planning for Web Operations - Web20 Expo 2008
-
Upload
john-allspaw -
Category
Technology
-
view
22.775 -
download
0
Transcript of Capacity Planning for Web Operations - Web20 Expo 2008
![Page 1: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/1.jpg)
Capacity Planning for Web Operations
John AllspawOperations Engineering
![Page 2: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/2.jpg)
?Do you know how many servers do
you have?
Are you tracking how your servers are
performing?
Do you know how much traffic can
your servers handle?
(without dying)
Are you tracking howyour application is being
used ?
![Page 3: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/3.jpg)
architecture
procurement
capex
deployment
monitoring
forecasting
testing
metrics
product planning
![Page 4: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/4.jpg)
architecture
procurement
capex
deploymentmonitoring
forecasting
testing
metrics
product planning
go see Adam Jacob’s talk!
![Page 5: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/5.jpg)
traditional capacity planning
![Page 6: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/6.jpg)
capacity planning for web
![Page 7: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/7.jpg)
* and network, datacenter space, power, etc.
Why capacity planning is important
(Cloudware costs $$, too)
Having too little is bad (!@#!!) -> ($$$)
Having too much is bad ($$$$!)
Hardware* costs $$
![Page 8: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/8.jpg)
Growth
“Normal”
“Instantaneous”
projectedplannedexpectedhoped for
spikesunexpectedexternal eventsdigg, etc.
(yay!)
(yay?)
(omg! wtf!)
![Page 9: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/9.jpg)
“Normal” growth at Flickrin a year....
4x increase in photo requests/sec
2.2x increase in uploads/day
3x increase in database queries/sec
![Page 10: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/10.jpg)
“Instantaneous”
Yahoo! FrontPage linkXMas lights
![Page 11: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/11.jpg)
“Instantaneous” coping
- Disabling “heavier” features on the site
- Cache aggressively or serve stale data
- Bake dynamic pages into static ones
![Page 12: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/12.jpg)
capacity != performance
Making something fast doesn’t necessarily make it
lastPerformance tuning = good, just
don’t count on it
Accept (for now) the performance you have, not the performance you wished you had, or you think you might have later
![Page 13: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/13.jpg)
Stewart: “Allspaw!!!! OMG!!!”
How many servers willwe need next year?!
(we need to tell finance by 2pm today)
![Page 14: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/14.jpg)
“Ah, just buy twice as much as we need”
2 x (how much we need) = ?
![Page 15: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/15.jpg)
![Page 16: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/16.jpg)
measurement
![Page 17: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/17.jpg)
Good capacity measurement tools can...
Easily measure and record any number that changes over time
Easily make graphs
Easily compare metrics to any other metrics from anywhere else (import/export)
![Page 18: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/18.jpg)
good tools are out there
cacti.net munin.projects.linpro.no
hyperic.com
ganglia.info
![Page 19: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/19.jpg)
good tools are out there
Flickr uses
cacti.net munin.projects.linpro.no
hyperic.com
ganglia.info
![Page 20: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/20.jpg)
application metrics
photouploads
viaemailper
minute
hour
![Page 21: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/21.jpg)
your stuff, not just system stuff
photos uploaded (and processed) per minuteaverage photo processing time per minute
average photo sizedisk space consumed per day
user registrations per dayetc etc etc
![Page 22: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/22.jpg)
your stuff, not just system stuff
photos uploaded (and processed) per minuteaverage photo processing time per minute
average photo sizedisk space consumed per day
user registrations per dayetc etc etc
![Page 23: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/23.jpg)
Tie application metrics to system metrics
Pretty!! But what doesthis mean?
![Page 24: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/24.jpg)
It means we can process
~120 images per minute
...and we can process them in
~3.5 seconds(on average)
It means that with about60% total CPU...
![Page 25: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/25.jpg)
Benchmarking
Great for comparing hardware platforms and configurations
BUTDoesn’t represent real workloads
![Page 26: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/26.jpg)
not exactly like a bike messenger
![Page 27: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/27.jpg)
Finding your ceilings
• Use real data, from production servers (if at all possible)
• No, really
![Page 28: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/28.jpg)
How many webservers can fail before we’re screwed?
How much traffic can each webserver take before it dies?
When should I add more webservers?
![Page 29: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/29.jpg)
people
webservers
(databases, etc.)
LBs
![Page 30: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/30.jpg)
people
webservers
(databases, etc.)
LBs
network
cpumemory
disk usage
disk i/o
![Page 31: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/31.jpg)
people
webservers
(databases, etc.)
LBs
network
cpumemory
disk usage
disk i/o
- comments/min- photos/min- videos/min- kittens/min- etc etc etc/min
![Page 32: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/32.jpg)
people
webservers
(databases, etc.)
LBs
![Page 33: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/33.jpg)
people
webservers
(databases, etc.)
LBs
![Page 34: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/34.jpg)
people
webservers
(databases, etc.)
LBs
![Page 35: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/35.jpg)
Ceiling = upper limit of “work” (and resources)
what happens here?
![Page 36: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/36.jpg)
Trends of peaks
Time
![Page 37: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/37.jpg)
Benchmarking
Might be your only option if you have a single server.
Siege http://www.joedog.org/JoeDog/Siege
httperf/autobench http://www.hpl.hp.com/research/linux/httperf/
http://www.xenoclast.org/autobench
some good benchmarking tools:
sysbench http://sysbench.sf.net
![Page 38: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/38.jpg)
Economics
![Page 39: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/39.jpg)
Time makes everything cheaper
(the Moore’s Law thing)
you don’t have a lot of time to wait around, do you?
BUT
![Page 40: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/40.jpg)
Vertical scaling
![Page 41: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/41.jpg)
Horizontal architectures
![Page 42: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/42.jpg)
Diagonal scaling
![Page 43: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/43.jpg)
Diagonal scaling
Replacing 67 dual-core webservers with 18 dual quads
![Page 44: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/44.jpg)
Diagonal scaling
more traffic from less machines
![Page 45: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/45.jpg)
Diagonal Scaling
servers CPUsper server
RAMper server
drivesper server
total power (W)@60% peak
67 2 4GB 1x80GB 8763.6
18 8 4GB 1x146GB 2332.8
~70% less power49U less rack space
![Page 46: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/46.jpg)
![Page 47: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/47.jpg)
Disclosure: We don’t use clouds at Flickr.
Utility Computing
(but we know folks who do)
![Page 48: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/48.jpg)
clouds
Still have to pay attention
Many people use the same forecastingmethods
Help with deployment timelinesHelp with procurement timelines
BUT
![Page 49: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/49.jpg)
Use Common Sense(tm)
Measure constantly, adapt constantly
Don’t pretend to know the exact future
Pay attention to the right metrics
Complex simulation and modeling is rarely worth it
Don’t expect tuning and tweaking will ever winyou any excess capacity
![Page 50: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/50.jpg)
Some more statsServing 32,000 photos per second at peak
Consuming 6-8TB per day
Consumed >34TB per day during Y!Photos migration
~3M uploads per day, 60 per second at peak
![Page 51: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/51.jpg)
20% off discount: “vel08js”
June 23-24, 2008
![Page 52: Capacity Planning for Web Operations - Web20 Expo 2008](https://reader033.fdocuments.us/reader033/viewer/2022052823/5556db1fd8b42a0f028b4890/html5/thumbnails/52.jpg)
We Are Hiring!(DBA, engineers)
http://flickr.com/photos/85013738@N00/542591121/http://flickr.com/photos/formatc1/2301500208/http://flickr.com/photos/mikefats/11546240/http://flickr.com/photos/kanaka/491064256/http://flickr.com/photos/randysonofrobert/1035003071/http://flickr.com/photos/halcyonsnow/446166047/http://flickr.com/photos/wwworks/2313927146/http://flickr.com/photos/sunxez/1392677065/http://flickr.com/photos/spacesuitcatalyst/536389937/http://flickr.com/photos/theklan/1276710183/