Inside Picnik: How we built Picnik and what we learned along the way
description
Transcript of Inside Picnik: How we built Picnik and what we learned along the way
![Page 1: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/1.jpg)
Inside Picnik:How we built Picnik and
what we learned along the way
Mike HarringtonJustin Huff
Web 2 Expo, April 3rd 2009
![Page 2: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/2.jpg)
![Page 3: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/3.jpg)
Thanks! We couldn’t have done this without help!
![Page 4: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/4.jpg)
Traffic, show scale of our problem
Our happy problem
![Page 5: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/5.jpg)
StorageRender
MySQL
Apache
Python
Flex HTML
DataCenter
Client
Amazon
![Page 6: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/6.jpg)
Fun with Flex
![Page 7: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/7.jpg)
Flex pros and cons
We Flash
• Browser and OS independence• Secure and Trusted• Huge installed base• Graphics and animation in your browser
![Page 8: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/8.jpg)
Flex pros and cons
We Flash
• Decent tool set• Decent image support• ActionScript
![Page 9: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/9.jpg)
Flex pros and cons
Flash isn’t Perfect
• Not Windows, Not Visual Studio• ActionScript is not C++• SWF Size/Modularity• Hardware isn’t accessible• Adobe is our Frienemy• Missing support for jpg Compression
![Page 10: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/10.jpg)
Flex pros and cons
Flash is a control freak
![Page 11: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/11.jpg)
Forecast: Cloudy with a chance of CPU
![Page 12: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/12.jpg)
User Cloud
Your users are your free cloud!
• Free CPU & Storage!• Cuts down on server round trips• More responsive (usually)• 3rd Party APIs Directly from your client• Flash 10 enables direct load
![Page 13: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/13.jpg)
User Cloud
• Your user is a crappy sysadmin• Hard to debug problems you can’t see• Heterogeneous Hardware• Some OS Differences
You get what you pay for.
![Page 14: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/14.jpg)
User Cloud
Client Side Challenges
• Hard to debug problems you can’t see• Every 3rd party API has downtime• You get blamed for everything • Caching madness
![Page 15: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/15.jpg)
Dealing with nasty problems
• Logging• Ask customers directly• Find local customers• Test, test, test
![Page 16: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/16.jpg)
Would we use Flash again?
Absolutely.
![Page 17: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/17.jpg)
LAMP
![Page 18: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/18.jpg)
Linux
YUP.
![Page 19: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/19.jpg)
Apache
Check.(static files)
![Page 20: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/20.jpg)
MySQL
Yeah, that too.
![Page 21: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/21.jpg)
P
Is for Python
![Page 22: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/22.jpg)
Python
90% of Picnik is API.
![Page 23: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/23.jpg)
CherryPy
• We chose CherryPy• RESTish API• Cheetah for a small amount of templating
![Page 24: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/24.jpg)
Ok, that was boring.
![Page 25: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/25.jpg)
Fun Stuff
![Page 26: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/26.jpg)
Virtualization
• XEN• Dual Quad core machines• 32GB RAM• We virtualize everything
– Except DB servers
![Page 27: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/27.jpg)
Virtualization Pros
• Partition functions• Matching CPU bound and IO bound VMs• Hardware standardization (in theory)
![Page 28: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/28.jpg)
Virtualization Cons
• More complexity• Uses more RAM• IO/CPU inefficiencies
![Page 29: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/29.jpg)
Storage
• We started with one server– Files stored locally– Not scalable
![Page 30: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/30.jpg)
Storage• Switched to MogileFS
– Working great– No dedicated storage
machines!
![Page 31: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/31.jpg)
Storage• Added S3 into the mix
![Page 32: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/32.jpg)
Storage – S3
• S3 is dead simple to implement.• For our usage patterns, it's expensive• Picnik generates lots of temp files• Problems keeping up with deletes• Made a decision to focus on other things
before fixing deletes
![Page 33: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/33.jpg)
Load Balancers
• Started using Perlbal• Bought two BigIPs
– Expensive, but good.• Outgrew the BigIPs• Went with A10 Networks
– A little bumpy– They've had great support.
![Page 34: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/34.jpg)
CDNs Rock
• We went with Level3• Cut outbound traffic in half:
![Page 35: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/35.jpg)
Rendering
• We render images when a user saves• Render jobs go into a queue• Workers pull from the queue• Workers in both the DC and EC2
![Page 36: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/36.jpg)
Rendering• Manager process maintain enough idle
workers• Workers in DC are at ~100% utilization
%$@#!!!
![Page 37: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/37.jpg)
EC2• Our processing is elastic between EC2 and our
own servers• We buy servers in batches
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
0
10
20
30
40
50
60
70
80
Render - EC2
Render - Local
Web
Time
![Page 38: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/38.jpg)
Monitoring
If a rack falls in the DC, does it make any sound?
![Page 39: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/39.jpg)
Monitoring• Nagios for Up/Down
– Integrated with automatic tools– ~120 hosts– ~1100 service checks
• Cacti for general graphing/trending– ~3700 data sources– ~2800 Graphs
• Smokeping– Network latency
![Page 40: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/40.jpg)
Redundancy
• I like sleeping at night• Makes maintenance tasks easier• It takes some work to build• We built early (thanks Flickr!)
![Page 41: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/41.jpg)
War Stories
![Page 42: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/42.jpg)
ISP Issues
• We're happy with both of our providers.• But...
– Denial of service attacks– Router problems– Power
• Have several providers• Monitor & trend!
![Page 43: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/43.jpg)
ISP Issues• High latency on a transit link:
• Cause: High CPU usage on their aggregation router
• Solution: Clear and reconfig our interface
![Page 44: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/44.jpg)
Amazon Web Services
• AWS is pretty solid• But, it's not 100%• When AWS breaks, we're down
– The worst type of outage because you can't do anything.
• Watch out for issues that neither party controls– The Internet isn't as reliable point to point
![Page 45: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/45.jpg)
EC2 Safety Net
• Bug caused render times to increase
![Page 46: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/46.jpg)
EC2 Safety net
• Thats OK!
![Page 47: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/47.jpg)
Flickr Launch• Very slow Flickr API calls post-launch• Spent an entire day on phone/IM with Flickr
Ops• Finally discovered an issue with NAT and TCP
timestamps• Lessons:
– Have tools to be able to dive deep– Granular monitoring
![Page 48: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/48.jpg)
Firewalls• We had a pair of Watchguard x1250e
firewalls.– Specs: “1.5Gbps of firewall throughput”– Actual at 100Mbps:
![Page 49: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/49.jpg)
authorize.net
• Connectivity issues• Support was useless• Eventually got to their NOC• We rerouted via my home DSL.• Lessons:
– Access to technical people– Handle failure of services gracefully
![Page 50: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/50.jpg)
The end.
![Page 52: Inside Picnik: How we built Picnik and what we learned along the way](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681461a550346895db32503/html5/thumbnails/52.jpg)
Credits/ResourcesTalks that influenced us:http://tinyurl.com/scalingwebsites
Images:http://www.flickr.com/photos/kimberlyfaye/2785383504/http://www.flickr.com/photos/piez/574804017/sizes/o/http://www.flickr.com/photos/unclejojo/3224365412/sizes/o/http://www.flickr.com/photos/phoenixdailyphoto/1467681879/sizes/
l/http://abductionlamp.com/http://www.flickr.com/photos/msig/2716038191/http://www.flickr.com/photos/touringcyclist/303577344/http://www.flickr.com/photos/davidt2006/1407837008/