Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML...
Transcript of Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML...
![Page 1: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/1.jpg)
Flickr and PHPCal Henderson
![Page 2: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/2.jpg)
What’s Flickr
• Photo sharing• Open APIs
![Page 3: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/3.jpg)
Logical Architecture
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
![Page 4: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/4.jpg)
Physical Architecture
Static Servers Database Servers Node Servers
Web Servers
Users
![Page 5: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/5.jpg)
Where is PHP?
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
![Page 6: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/6.jpg)
Other than PHP?
• Smarty for templating• PEAR for XML and Email parsing• Perl for controlling…• ImageMagick, for image processing• MySQL (4.0 / InnoDb)• Java, for the node service• Apache 2, Redhat, etc. etc.
![Page 7: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/7.jpg)
Big Application?
• One programmer, one designer, etc.• ~60,000 lines of PHP code• ~60,000 lines of templates• ~70 custom smarty functions/modifiers• ~25,000 DB transactions/second at peak• ~1000 pages per second at peak
![Page 8: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/8.jpg)
Thinking outside the web app
• Services– Atom/RSS/RDF Feeds– APIs
• SOAP• XML-RPC• REST• PEAR::XML::Tree
![Page 9: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/9.jpg)
More cool stuff
• Email interface– Postfix– PHP– PEAR::Mail::mimeDecode
• FTP• Uploading API• Authentication API• Unicode
![Page 10: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/10.jpg)
Even more stuff
• Real time application• Cool flash apps• Blogging
– Blogger API (1 & 2)– Metaweblog API– Atom– LiveJournal
![Page 11: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/11.jpg)
APIs are simple!
• Modeled on XML-RPC (Sort of)• Method calls with XML responses• SOAP, XML-RPC and REST are just transports• PHP endpoints mean we can use the same application
logic as the website
![Page 12: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/12.jpg)
XML isn’t simple :(
• PHP 4 doesn’t have good a XML parser• Expat is cool though (PEAR::XML::Parser)• Why doesn’t PEAR have XPath?
– Because PEAR is stupid!– PHP 4 sucks!
![Page 13: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/13.jpg)
I love XPath
if ($tree->root->name == 'methodResponse'){if (($tree->root->children[0]->name == 'params')&& ($tree->root->children[0]->children[0]->name == 'param')&& ($tree->root->children[0]->children[0]->children[0]->name == 'value')&& ($tree->root->children[0]->children[0]->children[0]->children[0]->name == 'array')&& ($tree->root->children[0]->children[0]->children[0]->children[0]->children[0]->name == 'data')){
$rsp = $tree->root->children[0]->children[0]->children[0]->children[0]->children[0];}if ($tree->root->children[0]->name == 'fault'){
$fault = $tree->root->children[0];return $fault;
}}
$nodes = $tree->select_nodes('/methodResponse/params/param[1]/value[1]/array[1]/data[1]/text()');
if (count($nodes)){$rsp = array_pop($nodes);
}else{list($fault) = $tree->select_nodes('/methodResponse/fault');return $fault;
}
![Page 14: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/14.jpg)
Creating API methods
• Stateless method-call APIs are easy to extend• Adding a method requires no knowledge of the transport• Adding a method once makes it available to all the
interfaces• Self documenting
![Page 15: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/15.jpg)
Red Hot Unicode Action
• UTF-8 pages• CJKV support• It’s really cool
![Page 16: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/16.jpg)
![Page 17: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/17.jpg)
Unicode for all
• It’s really easy– Don’t need PHP support– Don’t need MySQL support– Just need the right headers– UTF-8 is 7-bit transparent– (Just don’t mess with high characters)
• Don’t use HtmlEntities()!
• But bear in mind…• JavaScript has patchy Unicode support• People using your APIs might be stupid
![Page 18: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/18.jpg)
Scaling the beast
• Why PHP is great• MySQL scaling• Search scaling• Horizontal scaling
![Page 19: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/19.jpg)
Why PHP is great
• Stateless– We can bounce people around servers– Everything is stored in the database– Even the smarty cache– “Shared nothing”– (so long as we avoid PHP sessions)
![Page 20: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/20.jpg)
MySQL Scaling
• Our database server started to slow• Load of 200• Replication!
![Page 21: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/21.jpg)
MySQL Replication
• But it only gives you more SELECT’s• Else you need to partition vertically• Re-architecting sucks :(
![Page 22: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/22.jpg)
Looking at usage
• Snapshot of db1.flickr.com– SELECT’s 44,220,588– INSERT’s 1,349,234– UPDATE’s 1,755,503– DELETE’s 318,439– 13 SELECT’s per I/U/D
![Page 23: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/23.jpg)
Replication is really cool
• A bunch of slave servers handle all the SELECT’s• A single master just handles I/U/D’s• It can scale horizontally, at least for a while.
![Page 24: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/24.jpg)
Searching
• A simple text search• We were using RLIKE• Then switched to LIKE• Then disabled it all together
![Page 25: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/25.jpg)
FULLTEXT Indexes
• MySQL saves the day!• But they’re only supported my MyISAM tables• We use InnoDb, because it’s a lot faster• We’re doomed
![Page 26: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/26.jpg)
But wait!
• Partial replication saves the day• Replicate the portion of the database we want to search. • But change the table types on the slave to MyISAM• It can keep up because it’s only handling I/U/D’s on a
couple of tables• And we can reduce the I/U/D’s with a little bit of vertical
partitioning
![Page 27: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/27.jpg)
JOIN’s are slow
• Normalised data is for sissies• Keep multiple copies of data around• Makes searching faster• Have to ensure consistency in the application logic
![Page 28: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/28.jpg)
Our current setup
Slave Farm
DB1Master I/U/D’s
SELECT’s
Search Slave Farm
SearchSELECT’s
DB3Main Search
slave
DB2Main Slave
![Page 29: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/29.jpg)
Horizontal scaling
• At the core of our design• Just add hardware!• Inexpensive• Not exponential• Avoid redesigns
![Page 30: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/30.jpg)
Talking to the Node Service
• Everyone speaks XML (badly)• Just TCP/IP - fsockopen()• We’re issuing commands, not requesting data, so we
don’t bother to parse the response– Just substring search for state=“ok”
• Don’t rely on it!
![Page 31: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/31.jpg)
RSS / Atom / RDF
• Different formats• All quite bad• We’re generating a lot of different feeds• Abstract the difference away using templates• No good way to do private feeds. Why is nobody working
on this? (WSSE maybe?)
![Page 32: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/32.jpg)
Receiving email
• Want users to be able to email photos to Flickr• Just get postfix to pipe each mail to a PHP script• Parse the mail and find any photos• Cellular phone companies hate you• Lots of mailers are retarded
– Photos as text/plain attachments :/
![Page 33: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/33.jpg)
Upload via FTP
• PHP isn’t so great at being a daemon• Leaks memory like a sieve• No threads• Java to the rescue• Java just acts as an FTPd and passes all uploaded files
to PHP for processing• (This isn’t actually public)• Bricolage does this I think. Maybe Zope?
![Page 34: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/34.jpg)
Blogs
• Why does everyone loves blogs so much?• Only a few APIs really
– Blogger– Metaweblog– Blogger2– Movable Type– Atom– Live Journal
![Page 35: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/35.jpg)
It’s all broken
• Lots of blog software has broken interfaces• It’s a support nightmare• Manila is tricky• But it all works, more or less• Abstracted in the application logic• We just call blogs_post_message();
![Page 36: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/36.jpg)
Back to those APIs
• We opened up the Flickr APIs a few weeks ago• Programmers mainly build tools for other programmers• We have Perl, python, PHP, ActionScript, XMLHTTP and
.NET interface libraries• But also a few actual applications
![Page 37: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/37.jpg)
Flickr Rainbow
![Page 38: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/38.jpg)
Tag Wallpaper
![Page 39: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/39.jpg)
So what next?
• Much more scaling• PHP 5?• MySQL 5?• Taking over the world
![Page 40: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/40.jpg)
Flickr and PHPCal Henderson
![Page 41: Flickr and PHP · APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can](https://reader034.fdocuments.us/reader034/viewer/2022052005/6018d1cfcfa8040ae37f544b/html5/thumbnails/41.jpg)
Any Questions?