Gearmam, from the_worker's_perspective copy
-
Upload
brian-aker -
Category
Technology
-
view
762 -
download
0
description
Transcript of Gearmam, from the_worker's_perspective copy
![Page 1: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/1.jpg)
GearmanFrom the Worker's Perspective
![Page 2: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/2.jpg)
/usr/bin/whoami /usr/bin/whoami
Brian AkerBrian Aker
HP FellowHP Fellow
Previously MySQL, Slashdot, Sun Previously MySQL, Slashdot, Sun MicrosystemsMicrosystems
![Page 3: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/3.jpg)
What is Gearman?
![Page 4: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/4.jpg)
“The way I like to think of Gearman is a massively
distributed fork mechanism” -Joe Stump, Digg
“The Not Mechanical Turk” -Don MacAskill, SmugMug
![Page 5: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/5.jpg)
![Page 6: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/6.jpg)
do do (“resize_image(“resize_image”)”)
resize_image()resize_image() {{ … … return $image;return $image;}}
![Page 7: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/7.jpg)
(Livejournal)
![Page 8: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/8.jpg)
![Page 9: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/9.jpg)
![Page 10: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/10.jpg)
![Page 11: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/11.jpg)
![Page 12: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/12.jpg)
![Page 13: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/13.jpg)
Server Provides Asynchronous and Synchronous Provides Asynchronous and Synchronous
RequestsRequests
Restarts WorkRestarts Work
Durable Requests (MySQL, Postgres,...)Durable Requests (MySQL, Postgres,...)
Gearman Protocol/HTTPGearman Protocol/HTTP
Epoch SchedulingEpoch Scheduling
LoggingLogging
(also available, native Servers in Java, (also available, native Servers in Java, Erlang, and Perl)Erlang, and Perl)
![Page 14: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/14.jpg)
Client
# Create our client object.$gmclient= new GearmanClient();# Add default server (localhost).$gmclient->addServer();
$result= $gmclient->do("reverse", "Hello!");echo "Success: $result\n";
![Page 15: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/15.jpg)
Worker
# Create our worker object.$gmw= new GearmanWorker();# Add default server (localhost).$gmw->addServer();
$gmw->addFunction("reverse", "reverse_fn");while ($gmworker->work()) {…}
![Page 16: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/16.jpg)
Worker Function
function reverse_fn($job){ $workload= $job->workload(); $result= strrev($workload); return $result;}
![Page 17: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/17.jpg)
Lots of functions...
$gmw->addFunction("resize", "resize_fn");$gmw->addFunction("grep", "grep_fn");
$gmw->addFunction("fetch_url", "fetch_url");
![Page 18: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/18.jpg)
gearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);
… gearman_job_send_data(job, chunk, sizeofchunk);
… gearman_job_send_status(job, 50,100);
… if (issue_warning) gearman_job_warning(job, “I'm sorry, Dave. I'm afraid I can't do that.”, size); return GEARMAN_SUCCESS;}
Function
![Page 19: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/19.jpg)
GEARMAN_SUCCESSGEARMAN_SUCCESS
GEARMAN_FATALGEARMAN_FATAL
GEARMAN_ERRORGEARMAN_ERROR
GEARMAN_SHUTDOWNGEARMAN_SHUTDOWN
Worker Return
![Page 20: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/20.jpg)
Client APIClient API(C, PHP, Perl, Python, (C, PHP, Perl, Python,
Java,Drizzle, ...)Java,Drizzle, ...)
Worker APIWorker API(C, PHP, Perl, Python, Java, ...)(C, PHP, Perl, Python, Java, ...)
ServerServer
Your Client Your Client request()request()
Your WorkerYour Workerfunction()function()
Network,Highly Available,Fault Tolerant
![Page 21: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/21.jpg)
Client APIClient API(C, PHP, Perl, Python, (C, PHP, Perl, Python,
Java,Drizzle, ...)Java,Drizzle, ...)
Worker APIWorker API(C, PHP, Perl, Python, Java, ...)(C, PHP, Perl, Python, Java, ...)
ServerServer Provided by Provided by GearmanGearman
Your Client Your Client request()request()
Your WorkerYour Workerfunction()function()
resize_imageresize_image(“…”);(“…”);
resize_image()resize_image(){{ … …;; return resized;return resized;}}
Network,Highly Available,Fault Tolerant
![Page 22: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/22.jpg)
CPU?event()
![Page 23: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/23.jpg)
![Page 24: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/24.jpg)
multiple languages
![Page 25: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/25.jpg)
Connectors?
•C/C++
•PHP
•Python
•Java
•MySQL and Postgres
•....
![Page 26: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/26.jpg)
Other items of interest?
•Work status requests.
•Chunked Data.
•Exception Handling.
•Up to 4gig message sizes.
•Threaded server.
•Coalescence (the stealth killer feature)
![Page 27: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/27.jpg)
NamespacesFoo::resize_image()Foo::resize_image() {{ … … return $image;return $image;}}
Acme::resize_image()Acme::resize_image() {{ … … return $image;return $image;}}
![Page 28: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/28.jpg)
Better Map Reduce?
![Page 29: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/29.jpg)
map(list[…], map(list[…], reduce());reduce());
map() map() {…}{…}
reduce() reduce() {…}{…}
reduce() reduce() {…}{…}
reduce() reduce() {…}{…}
![Page 30: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/30.jpg)
Map @#$@# ?
![Page 31: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/31.jpg)
Partitioning
{A...K}
{L...Q}
{R...Z}
find(find())
find(find())
find(find())
![Page 32: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/32.jpg)
gearman_return_t split_worker(gearman_job_st *job, void* /* context */){ const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ' ') { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } } return GEARMAN_SUCCESS;}
Partitioning
![Page 33: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/33.jpg)
Aggregation
+ result
+ result
+ result$res$resultult
$res$resultult
$res$resultult
= sum result
![Page 34: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/34.jpg)
Aggregation
gearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::string string_value; do { gearman_result_st *result_ptr= gearman_task_result(task); string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task))); gearman_result_store_value(result, string_value.c_str(), string_value.size()); return GEARMAN_SUCCESS;}
![Page 35: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/35.jpg)
Do we have to partition?
(What other tricks exist!)
![Page 36: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/36.jpg)
Pipeline
Store()Store() Resize()Resize() Publish()Publish()
![Page 37: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/37.jpg)
Future
0.32 Released0.32 Released
Custom Logging PluginsCustom Logging Plugins
Client/Worker Configuration Client/Worker Configuration
Extended Administrative CommandsExtended Administrative Commands
SSLSSL
Status lookup via Unique IdentifierStatus lookup via Unique Identifier
Job Result CacheJob Result Cache
Uplift!Uplift!
![Page 38: Gearmam, from the_worker's_perspective copy](https://reader036.fdocuments.us/reader036/viewer/2022062617/54b76a304a795971038b462c/html5/thumbnails/38.jpg)
•Gearman.org (...)
•http://launchpad.net/gearmand/
•twitter: brianaker
•blog: blog.krow.net
Gearman.info