Dispatching jobs with Gearman

or

How to farm out jobs

or

Using Gearmanin Perl

Using Gearmanin Perl

gearman.org

The name "Gearman" was chosen as an

anagram for "Manager," "since it dispatches

jobs to be done, but does not do anything

useful itself"

Originally written in Perl

Rewritten in C

Clients in Perl, PHP, Phython,

Java, C# (.NET)

Clients in Perl, PHP, Phython,

Java, C# (.NET),even MySQL and PostgreSQL

WTF?

Architecture

Part I

Application

Job

Application

Job (or task)

Application

Job

gearmand

Application

Job

gearmand

Worker

Job

Application

Job

gearmand

Worker

Job Response

Application

gearmand

Worker

Worker

Worker

Application

gearmand

Worker

Worker

gearmand

Application

gearmand

Worker

Worker

gearmand

Application

Workers are scalable

Run any number you need

Run on remote servers

Application does not care

Application only talks with gearmand server

Application only talks with gearmand server

(one or more)

One or more application

One or more application

Applications throw jobs

One or more job servers

One or more job servers

Job servers dispatch jobs

One or more workers

One or more workers(clones or different)

One or more workers

Workers do jobs

One or more workers

Workers do jobsand may issue new jobs

Scalable also means redundant

Scalable also means redundant and reliable

Scalable also means redundant and reliable*

* when it works properly

Differentapproaches

to the architecture

Synchronousor

asynchronous

Worker either generates a response or works silently

Application

Job

gearmand

Worker

Job Response

Response

Synchronous

Application

Job

gearmand

Worker

Job Response

Response

Asynchronous

Application

Job

gearmand

Worker

Job Response

Response

With a response

Application

Job

gearmand

Worker

Job

With no response

“I’m done”

Application

Job

gearmand

Worker

Job

With no response

“I’m done”

DatabaseResult

Installation

Installation

1. Job server2. Client libraries3. Worker libraries

Installation

1. Job server C2. Client libraries Perl3. Worker libraries

Installation

1. Job server C2. Client libraries Perl XS3. Worker libraries

https://launchpad.net/gearmand

$./configure

$make

#makeinstall

Installing job server

#cpanGearman

Installing Perl modules

#cpanGearman::Client

#cpanGearman::Worker

or

#cpanGearman

Installing Perl modules

#cpanGearman::XS

or

Starting gearmand deamon

$gearmand‐d

Starting gearmand deamon

$gearmand‐d

or$gearmand‐d\

‐L127.0.0.1\

‐p4730

Starting worker(s)

$./worker.pl&

Starting worker(s)

$./worker.pl&

or$./workerA.pl&

$./workerB.pl&

Starting worker(s)

$./worker.pl&

or$./workerA.pl&

$./workerB.pl&

$./workerA.pl&

$./workerB.pl&

Running client

$./client.pl

Worker example#!/usr/bin/perl

usev5.10;

usestrict;

useGearman::Worker;

my$worker=newGearman::Worker;

$worker‐>job_servers('127.0.0.1:4730');

$worker‐>register_function(echo=>\&echo);

$worker‐>workwhile1;

subecho{

my$job=shift;

say'+1';

return$job‐>arg+1;

}

Worker example

my$worker=newGearman::Worker;

$worker‐>job_servers('127.0.0.1:4730');

$worker‐>register_function(echo=>\&echo);

$worker‐>workwhile1;

Worker example

my$worker=newGearman::Worker;

$worker‐>job_servers('127.0.0.1:4730');

$worker‐>register_function(echo=>\&echo);

$worker‐>workwhile1;

Connect to the server

Worker example

my$worker=newGearman::Worker;

$worker‐>job_servers('127.0.0.1:4730');

$worker‐>register_function(echo=>\&echo);

$worker‐>workwhile1;

Register worker method

Worker example

my$worker=newGearman::Worker;

$worker‐>job_servers('127.0.0.1:4730');

$worker‐>register_function(echo=>\&echo);

$worker‐>workwhile1;

Wait for incoming jobs

Worker example#!/usr/bin/perl

usev5.10;

usestrict;

useGearman::Worker;

my$worker=newGearman::Worker;

$worker‐>job_servers('127.0.0.1:4730');

$worker‐>register_function(echo=>\&echo);

$worker‐>workwhile1;

subecho{

my$job=shift;

say'+1';

return$job‐>arg+1;

}

Job method

subecho{my$job=shift;say'+1';return$job‐>arg+1;}

Receive data

Job method

subecho{my$job=shift;say'+1';return$job‐>arg+1;}

Evaluate response

Client example#!/usr/bin/perl

use v5.10;use strict;use Gearman::Client;

my $client = new Gearman::Client;$client->job_servers('127.0.0.1:4730');

for (1..10000) {    my $result_ref = $client->do_task('echo', 10);    say $$result_ref;}

Client example

for (1..10000) {    my $result_ref = $client->do_task('echo', 10);    say $$result_ref;}

Issue jobs

Asynchronous client exampleuse v5.10;use strict;use Gearman::Client;

my $client = new Gearman::Client;$client->job_servers('example.com:4730');

my $tasks = $client->new_task_set;$tasks->add_task(    'echo' => 10,    {        on_complete => \&completed    });

$tasks->wait;

sub completed {    my $result = shift;    say $$result;}

my $tasks = $client->new_task_set;$tasks->add_task(    'echo' => 10,    {        on_complete => \&completed    });$tasks->wait;

Get task set placeholder

Asynchronous client example

my $tasks = $client->new_task_set;$tasks->add_task(    'echo' => 10,    {        on_complete => \&completed    });$tasks->wait;

Place the task

Asynchronous client example

my $tasks = $client->new_task_set;$tasks->add_task(    'echo' => 10,    {        on_complete => \&completed    });$tasks->wait;

and register callback method

Asynchronous client example

my $tasks = $client->new_task_set;$tasks->add_task(    'echo' => 10,    {        on_complete => \&completed    });$tasks->wait;

Wait until the job is done

Asynchronous client example

sub completed {    my $result = shift;    say $$result;}

Use result somehow

Asynchronous client example

XS worker exampleuse v5.10;

use strict;

use Gearman::XS::Worker;

my $worker = new Gearman::XS::Worker;

$worker->add_server('127.0.0.1', 4730);

$worker->add_function('echo', 0, \&echo, undef);

$worker->work() while 1;

sub echo {

    my $job = shift;

    return $job->workload() + 1;

}

XS client exampleuse v5.10;

use strict;

use Gearman::XS::Client;

my $client = new Gearman::XS::Client;

$client->add_server('127.0.0.1', 4730);

my ($ret, $result) = $client->do('echo', '15'); # arg as string!

say $result;

Async XS client exampleuse v5.10;

use strict;

use Gearman::XS::Client;

my $client = new Gearman::XS::Client;

$client->add_server('127.0.0.1', 4730);

my ($ret1, $task1) = $client->add_task('echo', '15');

my ($ret2, $task2) = $client->add_task('echo', '51');

$client->run_tasks();

Monitoring job server

TCP protocol

Connect with telnet

$ telnet localhost 4730

$ telnet localhost 4730Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.

$ telnet localhost 4730Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.status

$ telnet localhost 4730Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.statusecho2   1      1       2echo    5      2       5.

echo2   1      1       2echo    5      2       5.

echo2   1      1       2echo    5      2       5.

Known worker methods

echo2   1      1       2echo    5      2       5.

Jobs running

echo2   1      1       2echo    5      2       5.

Jobs queued

echo2   1      1       2echo    5      2       5.

Workers available

Step by step example

Part III

for(...){...}

worker.pl

worker.pl

worker.pl

worker.pl

Live demo

__END__

Andrew Shitov

andy@shitov.ru