Inside Picnik: How We Built Picnik (and What We Learned Along the Way)
-
Upload
jjhuff -
Category
Technology
-
view
4.390 -
download
3
Transcript of Inside Picnik: How We Built Picnik (and What We Learned Along the Way)
![Page 1: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/1.jpg)
Inside Picnik: How we built Picnik and what
we learned along the way Mike Harrington
Justin Huff
Web2Expo,April3rd2009
![Page 2: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/2.jpg)
![Page 3: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/3.jpg)
Thanks! We couldn’t have done this without help!
![Page 4: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/4.jpg)
Traffic, show scale of our problem
Our happy problem
![Page 5: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/5.jpg)
Storage Render
MySQL
Apache
Python
Flex HTML
Data Center
Client
Amazon
![Page 6: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/6.jpg)
Fun with Flex
![Page 7: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/7.jpg)
Flex pros and cons
We Flash
• Browser and OS independence • Secure and Trusted • Huge installed base • Graphics and animation in your browser
![Page 8: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/8.jpg)
Flex pros and cons
We Flash
• Decent tool set • Decent image support • ActionScript
![Page 9: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/9.jpg)
Flex pros and cons
Flash isn’t Perfect
• Not Windows, Not Visual Studio • ActionScript is not C++ • SWF Size/Modularity • Hardware isn’t accessible • Adobe is our Frienemy • Missing support for jpg Compression
![Page 10: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/10.jpg)
Flex pros and cons
Flash is a control freak
![Page 11: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/11.jpg)
Forecast: Cloudy with a chance of CPU
![Page 12: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/12.jpg)
User Cloud
Your users are your free cloud!
• Free CPU & Storage! • Cuts down on server round trips • More responsive (usually) • 3rd Party APIs Directly from your client • Flash 10 enables direct load
![Page 13: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/13.jpg)
User Cloud
• Your user is a crappy sysadmin • Hard to debug problems you can’t see • Heterogeneous Hardware • Some OS Differences
You get what you pay for.
![Page 14: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/14.jpg)
User Cloud
Client Side Challenges
• Hard to debug problems you can’t see • Every 3rd party API has downtime • You get blamed for everything • Caching madness
![Page 15: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/15.jpg)
Dealing with nasty problems
• Logging • Ask customers directly • Find local customers • Test, test, test
![Page 16: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/16.jpg)
WouldweuseFlashagain?
Absolutely.
![Page 17: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/17.jpg)
LAMP
![Page 18: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/18.jpg)
Linux
YUP.
![Page 19: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/19.jpg)
Apache
Check.(staGcfiles)
![Page 20: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/20.jpg)
MySQL
Yeah,thattoo.
![Page 21: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/21.jpg)
P
IsforPython
![Page 22: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/22.jpg)
Python
90%ofPicnikisAPI.
![Page 23: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/23.jpg)
CherryPy
• WechoseCherryPy• RESTishAPI• CheetahforasmallamountoftemplaGng
![Page 24: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/24.jpg)
Ok,thatwasboring.
![Page 25: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/25.jpg)
Fun Stuff
![Page 26: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/26.jpg)
VirtualizaGon
• XEN• DualQuadcoremachines• 32GBRAM• Wevirtualizeeverything
– ExceptDBservers
![Page 27: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/27.jpg)
VirtualizaGonPros
• ParGGonfuncGons• MatchingCPUboundandIOboundVMs• HardwarestandardizaGon(intheory)
![Page 28: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/28.jpg)
VirtualizaGonCons
• Morecomplexity• UsesmoreRAM• IO/CPUinefficiencies
![Page 29: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/29.jpg)
Storage
• Westartedwithoneserver– Filesstoredlocally– Notscalable
![Page 30: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/30.jpg)
Storage• SwitchedtoMogileFS
– Workinggreat– Nodedicatedstoragemachines!
![Page 31: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/31.jpg)
Storage
• AddedS3intothemix
![Page 32: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/32.jpg)
Storage–S3
• S3isdeadsimpletoimplement.• Forourusagepaberns,it'sexpensive• Picnikgenerateslotsoftempfiles• Problemskeepingupwithdeletes• Madeadecisiontofocusonotherthingsbeforefixingdeletes
![Page 33: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/33.jpg)
LoadBalancers
• StartedusingPerlbal• BoughttwoBigIPs
– Expensive,butgood.• OutgrewtheBigIPs• WentwithA10Networks
– Aliblebumpy– They'vehadgreatsupport.
![Page 34: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/34.jpg)
CDNsRock
• WewentwithLevel3• Cutoutboundtrafficinhalf:
![Page 35: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/35.jpg)
Rendering
• Werenderimageswhenausersaves• Renderjobsgointoaqueue• Workerspullfromthequeue• WorkersinboththeDCandEC2
![Page 36: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/36.jpg)
Rendering• Managerprocessmaintainenoughidleworkers
• WorkersinDCareat~100%uGlizaGon
%$@#!!!
![Page 37: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/37.jpg)
EC2• OurprocessingiselasGcbetweenEC2andourownservers
• Webuyserversinbatches
![Page 38: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/38.jpg)
Monitoring
IfarackfallsintheDC,doesitmakeanysound?
![Page 39: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/39.jpg)
Monitoring
• NagiosforUp/Down– IntegratedwithautomaGctools– ~120hosts– ~1100servicechecks
• CacGforgeneralgraphing/trending– ~3700datasources– ~2800Graphs
• Smokeping– Networklatency
![Page 40: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/40.jpg)
Redundancy
• Ilikesleepingatnight• Makesmaintenancetaskseasier• Ittakessomeworktobuild• Webuiltearly(thanksFlickr!)
![Page 41: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/41.jpg)
WarStories
![Page 42: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/42.jpg)
ISPIssues
• We'rehappywithbothofourproviders.• But...
– Denialofserviceabacks– Routerproblems– Power
• Haveseveralproviders• Monitor&trend!
![Page 43: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/43.jpg)
ISPIssues
• Highlatencyonatransitlink:
• Cause:HighCPUusageontheiraggregaGonrouter
• SoluGon:Clearandreconfigourinterface
![Page 44: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/44.jpg)
AmazonWebServices
• AWSisprebysolid• But,it'snot100%• WhenAWSbreaks,we'redown
– Theworsttypeofoutagebecauseyoucan'tdoanything.
• Watchoutforissuesthatneitherpartycontrols– TheInternetisn'tasreliablepointtopoint
![Page 45: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/45.jpg)
EC2SafetyNet
• BugcausedrenderGmestoincrease
![Page 46: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/46.jpg)
EC2Safetynet
• ThatsOK!
![Page 47: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/47.jpg)
FlickrLaunch• VeryslowFlickrAPIcallspost‐launch• SpentanenGredayonphone/IMwithFlickrOps
• FinallydiscoveredanissuewithNATandTCPGmestamps
• Lessons:– Havetoolstobeabletodivedeep– Granularmonitoring
![Page 48: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/48.jpg)
Firewalls• WehadapairofWatchguardx1250efirewalls.
– Specs:“1.5Gbpsoffirewallthroughput”– Actualat100Mbps:
![Page 49: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/49.jpg)
authorize.net
• ConnecGvityissues• Supportwasuseless• EventuallygottotheirNOC• WereroutedviamyhomeDSL.• Lessons:
– Accesstotechnicalpeople– Handlefailureofservicesgracefully
![Page 50: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/50.jpg)
Theend.
![Page 52: Inside Picnik: How We Built Picnik (and What We Learned Along the Way)](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a7b9681a28ab226d8b471d/html5/thumbnails/52.jpg)
Credits/ResourcesTalksthatinfluencedus:hbp://Gnyurl.com/scalingwebsites
Images:hbp://www.flickr.com/photos/kimberlyfaye/2785383504/
hbp://www.flickr.com/photos/piez/574804017/sizes/o/hbp://www.flickr.com/photos/unclejojo/3224365412/sizes/o/hbp://www.flickr.com/photos/phoenixdailyphoto/1467681879/sizes/l/hbp://abducGonlamp.com/hbp://www.flickr.com/photos/msig/2716038191/hbp://www.flickr.com/photos/touringcyclist/303577344/hbp://www.flickr.com/photos/davidt2006/1407837008/