Distribute the workload, PHPTek, Amsterdam, 2011
-
Upload
helgi-bormar-borbjoernsson -
Category
Technology
-
view
2.515 -
download
1
description
Transcript of Distribute the workload, PHPTek, Amsterdam, 2011
![Page 1: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/1.jpg)
Distribute the workload
Helgi Þormar ÞorbjörnssonPHP Tek, Chicago, 26th May 2011
Sunday, 29 May 2011
![Page 2: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/2.jpg)
Who am I?
Sunday, 29 May 2011
![Page 3: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/3.jpg)
Helgi
Sunday, 29 May 2011
![Page 4: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/4.jpg)
VP of Engineering at Orchestra.io
Helgi
Sunday, 29 May 2011
![Page 5: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/5.jpg)
VP of Engineering at Orchestra.io
Developer at PEAR
Helgi
Sunday, 29 May 2011
![Page 6: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/6.jpg)
VP of Engineering at Orchestra.io
Developer at PEAR
From Iceland
Helgi
Sunday, 29 May 2011
![Page 7: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/7.jpg)
VP of Engineering at Orchestra.io
Developer at PEAR
From Iceland
@h on Twitter
Helgi
Sunday, 29 May 2011
![Page 8: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/8.jpg)
Why Distribute?
Sunday, 29 May 2011
![Page 9: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/9.jpg)
Why Distribute?
Efficiency
Sunday, 29 May 2011
![Page 10: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/10.jpg)
Why Distribute?
Budget
Efficiency
Sunday, 29 May 2011
![Page 11: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/11.jpg)
Why Distribute?
Budget
Efficiency
Perception
Sunday, 29 May 2011
![Page 12: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/12.jpg)
Efficiency
10 small servers > 1 big
Sunday, 29 May 2011
![Page 13: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/13.jpg)
Budget
Sunday, 29 May 2011
![Page 14: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/14.jpg)
Budget
Spend wisely
Sunday, 29 May 2011
![Page 15: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/15.jpg)
Budget
Spend wisely
Commodity servers
Sunday, 29 May 2011
![Page 16: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/16.jpg)
Budget
Spend wisely
Commodity servers
Cloud Computing (EC2)
Sunday, 29 May 2011
![Page 17: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/17.jpg)
Perception
Sunday, 29 May 2011
![Page 18: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/18.jpg)
Perception
Defer intensive processes
Sunday, 29 May 2011
![Page 19: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/19.jpg)
Perception
Defer intensive processes
Give instant feedback
Sunday, 29 May 2011
![Page 20: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/20.jpg)
Perception
Defer intensive processes
Give instant feedback
Users keep on browsing
Sunday, 29 May 2011
![Page 21: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/21.jpg)
Perception
Defer intensive processes
Give instant feedback
Users keep on browsing
Sunday, 29 May 2011
![Page 22: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/22.jpg)
Sunday, 29 May 2011
![Page 23: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/23.jpg)
Ant Colonies
Sunday, 29 May 2011
![Page 24: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/24.jpg)
Teamwork
When faced with a problem they will solve the problem as one.
Sunday, 29 May 2011
![Page 25: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/25.jpg)
Sunday, 29 May 2011
![Page 26: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/26.jpg)
Sunday, 29 May 2011
![Page 27: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/27.jpg)
Architect for Distribution
Sunday, 29 May 2011
![Page 28: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/28.jpg)
Characteristics
Sunday, 29 May 2011
![Page 29: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/29.jpg)
Characteristics
Decoupling
Sunday, 29 May 2011
![Page 30: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/30.jpg)
Characteristics
Decoupling
Elasticity
Sunday, 29 May 2011
![Page 31: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/31.jpg)
Characteristics
Decoupling
Elasticity
High Availability
Sunday, 29 May 2011
![Page 32: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/32.jpg)
Characteristics
Decoupling
Elasticity
High Availability
Concurrency
Sunday, 29 May 2011
![Page 33: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/33.jpg)
Decoupling
Sunday, 29 May 2011
![Page 34: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/34.jpg)
Application
DB API
Cache FE
Sunday, 29 May 2011
![Page 35: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/35.jpg)
Application
DB API
Cache FE
Sunday, 29 May 2011
![Page 36: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/36.jpg)
ApplicationDB API
Cache FE
Sunday, 29 May 2011
![Page 37: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/37.jpg)
ApplicationDB API
Cache FE
Cache
Sunday, 29 May 2011
![Page 38: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/38.jpg)
ApplicationDB API
Cache FE
Cache
API
Sunday, 29 May 2011
![Page 39: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/39.jpg)
ApplicationDB API
Cache FE
Cache
API
API
Sunday, 29 May 2011
![Page 40: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/40.jpg)
Elasticity
Sunday, 29 May 2011
![Page 41: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/41.jpg)
Cloud Computing
Sunday, 29 May 2011
![Page 42: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/42.jpg)
Load Balancing
Sunday, 29 May 2011
![Page 43: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/43.jpg)
HA Proxy
Nginx
My Favourite
Sunday, 29 May 2011
![Page 44: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/44.jpg)
Monitoring
Sunday, 29 May 2011
![Page 45: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/45.jpg)
When do I need more servers?
Sunday, 29 May 2011
![Page 46: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/46.jpg)
Needs to be around from the start!
Sunday, 29 May 2011
![Page 47: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/47.jpg)
Keep records
Sunday, 29 May 2011
![Page 48: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/48.jpg)
Spot trends
Sunday, 29 May 2011
![Page 49: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/49.jpg)
Different types
Sunday, 29 May 2011
![Page 50: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/50.jpg)
Different types
Hardware Performance
Sunday, 29 May 2011
![Page 51: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/51.jpg)
Different types
Hardware Performance
Software Performance
Sunday, 29 May 2011
![Page 52: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/52.jpg)
Different types
Hardware Performance
Software Performance
Availability
Sunday, 29 May 2011
![Page 53: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/53.jpg)
Different types
Hardware Performance
Software Performance
Availability
Resourcing
Sunday, 29 May 2011
![Page 54: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/54.jpg)
Different types
Hardware Performance
Software Performance
Availability
Resourcing
Sunday, 29 May 2011
![Page 55: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/55.jpg)
Applications
Sunday, 29 May 2011
![Page 56: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/56.jpg)
ApplicationsNew Relic
Sunday, 29 May 2011
![Page 57: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/57.jpg)
ApplicationsNew Relic
CloudKick
Sunday, 29 May 2011
![Page 58: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/58.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Sunday, 29 May 2011
![Page 59: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/59.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Nagios
Sunday, 29 May 2011
![Page 60: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/60.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Nagios
Cacti
Sunday, 29 May 2011
![Page 61: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/61.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Nagios
Cacti
Circonus
Sunday, 29 May 2011
![Page 62: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/62.jpg)
Automation
Sunday, 29 May 2011
![Page 63: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/63.jpg)
Plug into your monitoring
Sunday, 29 May 2011
![Page 64: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/64.jpg)
Bringing together Monitoring and Elastic behaviour into one
beautiful whole!
Sunday, 29 May 2011
![Page 65: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/65.jpg)
Add some intelligence to add / remove servers as needed based
on current information.
Sunday, 29 May 2011
![Page 66: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/66.jpg)
Just make sure it doesn’t turn into...
Sunday, 29 May 2011
![Page 67: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/67.jpg)
Skynet!!Sunday, 29 May 2011
![Page 68: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/68.jpg)
High Availability
Sunday, 29 May 2011
![Page 69: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/69.jpg)
Get a highly available and resilient setup by following a few
of those recommendations
Sunday, 29 May 2011
![Page 70: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/70.jpg)
Remember, even Google has outages
Sunday, 29 May 2011
![Page 71: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/71.jpg)
What to avoid
Sunday, 29 May 2011
![Page 72: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/72.jpg)
Local Sessions
Sunday, 29 May 2011
![Page 73: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/73.jpg)
Store sessions in DB / Memcache
Solution
Sunday, 29 May 2011
![Page 74: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/74.jpg)
Local Memory
Sunday, 29 May 2011
![Page 75: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/75.jpg)
Networked Memcache
Solution
Sunday, 29 May 2011
![Page 76: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/76.jpg)
Local Files
Sunday, 29 May 2011
![Page 77: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/77.jpg)
Local Uploads
Sunday, 29 May 2011
![Page 78: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/78.jpg)
Writing to /tmp
Sunday, 29 May 2011
![Page 79: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/79.jpg)
Store on S3 or a networked FS
Solution
Sunday, 29 May 2011
![Page 80: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/80.jpg)
Serve up static files from CDNs
Solution
Sunday, 29 May 2011
![Page 81: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/81.jpg)
Servers can vanish at any given time
Sunday, 29 May 2011
![Page 82: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/82.jpg)
Internal APIs
Sunday, 29 May 2011
![Page 83: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/83.jpg)
Application
S3GFS FS
Internal Storage API
Sunday, 29 May 2011
![Page 84: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/84.jpg)
Application
MySQLMongo Cache
Internal DB API
Sunday, 29 May 2011
![Page 85: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/85.jpg)
Eventually Consistent
Sunday, 29 May 2011
![Page 86: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/86.jpg)
CAP Therom
Sunday, 29 May 2011
![Page 87: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/87.jpg)
Consistency
Availability
Partition Tolerance
Sunday, 29 May 2011
![Page 88: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/88.jpg)
Consistency
All nodes see the same data at the same time
Sunday, 29 May 2011
![Page 89: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/89.jpg)
Availability
Node failures do not prevent survivors from continuing to
operate
Sunday, 29 May 2011
![Page 90: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/90.jpg)
Partition Tolerance
The system continues to operate despite arbitrary message loss
Sunday, 29 May 2011
![Page 91: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/91.jpg)
Consistency
Availability
Partition Tolerance
Sunday, 29 May 2011
![Page 92: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/92.jpg)
Queue Systems
Sunday, 29 May 2011
![Page 93: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/93.jpg)
Good for
Sunday, 29 May 2011
![Page 94: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/94.jpg)
Good forImage Processing
Sunday, 29 May 2011
![Page 95: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/95.jpg)
Good forImage Processing
Distributed Logs
Sunday, 29 May 2011
![Page 96: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/96.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Sunday, 29 May 2011
![Page 97: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/97.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Mass Emails
Sunday, 29 May 2011
![Page 98: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/98.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Mass Emails
Intensive transformation
Sunday, 29 May 2011
![Page 99: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/99.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Mass Emails
Intensive transformation
Search
Sunday, 29 May 2011
![Page 100: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/100.jpg)
Common Tools
Sunday, 29 May 2011
![Page 101: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/101.jpg)
Common Tools
Gearman
Sunday, 29 May 2011
![Page 102: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/102.jpg)
Common Tools
Gearman
Hadoop
Sunday, 29 May 2011
![Page 103: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/103.jpg)
Common Tools
Gearman
Hadoop
ZeroMQ
Sunday, 29 May 2011
![Page 104: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/104.jpg)
Common Tools
Gearman
Hadoop
ZeroMQ
RabbitMQ
Sunday, 29 May 2011
![Page 105: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/105.jpg)
Common Tools
Gearman
Hadoop
ZeroMQ
RabbitMQ
And many others!
Sunday, 29 May 2011
![Page 106: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/106.jpg)
Gearman
Sunday, 29 May 2011
![Page 107: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/107.jpg)
Your Client Code
Gearman Client API(C, PHP, Perl, MySQL UDF, ...)
Gearman Job Servergearmand
Gearman Worker API(C, PHP, Perl, Python, ...)
Your Worker Code
Your App Gearman
Sunday, 29 May 2011
![Page 108: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/108.jpg)
pear.php.net/net_gearman
Sunday, 29 May 2011
![Page 109: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/109.jpg)
A Story!
Sunday, 29 May 2011
![Page 110: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/110.jpg)
Financial Software
Sunday, 29 May 2011
![Page 111: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/111.jpg)
3000+ Clients
Sunday, 29 May 2011
![Page 112: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/112.jpg)
Each one has 5 external data sources
Sunday, 29 May 2011
![Page 113: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/113.jpg)
Each data source is a web service
Sunday, 29 May 2011
![Page 114: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/114.jpg)
Ran every 6 hours every day
Sunday, 29 May 2011
![Page 115: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/115.jpg)
Cron
Sunday, 29 May 2011
![Page 116: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/116.jpg)
Cron
Gearman
Sunday, 29 May 2011
![Page 117: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/117.jpg)
Cron
Gearman
Job 1
Sunday, 29 May 2011
![Page 118: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/118.jpg)
Cron
Gearman
Job 11
2
3
4
5
Web Services
Sunday, 29 May 2011
![Page 119: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/119.jpg)
Cron
Gearman
Job 11
2
3
4
5
Web Services
1
43
2
5
Processing
Sunday, 29 May 2011
![Page 120: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/120.jpg)
But! That wasn’t enough
Sunday, 29 May 2011
![Page 121: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/121.jpg)
Job kicked off on login
Sunday, 29 May 2011
![Page 122: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/122.jpg)
Supervisord
Sunday, 29 May 2011
![Page 123: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/123.jpg)
phpadvent.org/2009/daemonize-your-php-by-sean-coates
Sunday, 29 May 2011
![Page 124: Distribute the workload, PHPTek, Amsterdam, 2011](https://reader035.fdocuments.us/reader035/viewer/2022081403/555214a1b4c90520548b4877/html5/thumbnails/124.jpg)
Questions?
Joind.in: http://joind.in/3433
Sunday, 29 May 2011