High gear PHP with Gearman

PHPBenelux meeting - 13/10/2009Felix De Vliegher

High gear PHP with Gearman

zondag 11 oktober 2009

Contents

• What is Gearman?

• Distributed work

• Once upon a time

• Advantages

• Gearman 101

• PHP interface

• Examples!

• Application areas

• Other bits

2


What is Gearman?And why should I care about it?

3


4


Quote

The way I like to think of Gearmanis as a massively distributed,

massively fault tolerant fork mechanism.

- Joe Stump, Digg

5

“”


A grain of truth

The name is an anagram for “Manager,”since it dispatches jobs to be done,

but does not do anything useful itself.

- From Gearman website

6


What is Gearman?

It’s an

Application frameworkto

distribute work

7


Some keywords

• Open source• Multi-language• Parallel• Asynchronous• Multi threaded -> blazingly fast• Persistent queues• Decentralized• Fault tolerant• tcp 4730• multiple client api’s (c, php, perl, drizzle, ...)

8


Distributed processing

9

Or: how to make others do all the work


“Distributed computing”

are

“autonomous computational entries”

(computers or nodes)

which are communicating by

message passing

10



• Actual work is distributed

• Handled by a number of “computational entities”

• Some sort of RPC

• Distributed parallel processing

• Shared nothing

• Decentralized

11



• Actual work is distributed

• Handled by a number of “computational entities”

• Some sort of RPC

• Distributed parallel processing

• Shared nothing

• Decentralized

11

Gearman!


Once upon a timeGearman history

12


Once upon a time

• 2005: http://brad.livejournal.com/2106943.html

• Originally a Perl implementation

• Created by Danga Interactive

• LiveJournal

• SixApart

• Guys behind Memcache and MogileFS

13


Once upon a time

• 2008: Rewrite in C by Brian Aker

• PHP Extension by James Luedke

• Gearman powers some of the largest sites around:

• Digg: 45+ servers, 400K jobs/day

• Yahoo: 60+ servers, 6M jobs/day

• Xing.com

• Maybe Netlog? ;-)

14


Advantages

15


Advantages

• Speed up work

• Load balance work

• Parallel and asynchronous work

• Scales well

• Architecture-based workload distributing

• Call functionality in other programming languages

• Doesn’t block your apache processes!

16


Gearman 101Terminology and architecture

17


Terminology

18


Terminology

18

Client Create jobs to be run and send them to a job server

Worker Register with a job server and grab jobs to run

Job Server Coordinates assignment from clients to workers, handles restarts


Tasks vs Jobs

• A job is always a task

• A task is not always a job

• Single interface: jobs

• Concurrent interface: tasks

• Clients deal with tasks

• Workers deal with jobs

19


Architecture

20

Client application

Gearman Job server(gearmand)

Gearman Client API

Gearman Worker API

Worker application

Gearman Worker API

Worker application

Gearman Worker API

Worker application


Installing Gearman

• Job Server: gearmand

• Get it from https://launchpad.net/gearmand/• extract, configure, make, make install:

21

dev:/usr/local/src# wget http://launchpad.net/gearmand/trunk/0.10/+download/gearmand-0.10.tar.gz

dev:/usr/local/src# tar -xzvf gearmand-0.10.tar.gz

dev:/usr/local/src# cd gearmand-0.10/

dev:/usr/local/src/gearmand-0.10# ./configure --prefix=/usr/local/

dev:/usr/local/src/gearmand-0.10# make

dev:/usr/local/src/gearmand-0.10# make install

dev:/usr/local/src/gearmand-0.10# /usr/local/sbin/gearmand --help


Gearmand usage

/usr/local/sbin/gearmand -d -u <user> -L 127.0.0.1 -p 7003

22

-d Start as daemon in background

-u <user> Run as the specified user

-L <host> Only listen on the specified host or IP

-p <port> Listen on the specified port


Commandline gearman

• Client mode• ls | gearman -f function

• gearman -f function < file

• gearman -f function “foo data”

• Worker mode• gearman -w -f function -- wc -l

• gearman -w -f function ./script.sh

• Example:

23

dev:~/gearman# gearman -w -f foo -- grep GearmanClient &dev:~/gearman# cat client.php | gearman -p 7003 -f foo$client= new GearmanClient();


PHP Interface

24


PHP: Worker API

Possible to add multiple servers:

25


PHP: Worker API

Registering functions and any valid callback:

Pass application data to the functions:

26

# php -q client.php Sending jobCount: 1: HELLO!Count: 2: WORLD!


PHP: Worker API

27

Put it all together:


PHP: Job API

Status notifications:

GearmanJob::sendStatus(int $numerator, int $denominator)

GearmanJob::sendWarning(string $warning)

GearmanJob::sendComplete(string $result)

GearmanJob::sendFail(void)

GearmanJob::sendException(string $exception)

28


PHP: Callbacks

Provide functionality on different moments in the process:

29

GearmanClient::setDataCallback

GearmanClient::setCompleteCallback

GearmanClient::setCreatedCallback

GearmanClient::setExceptionCallback

GearmanClient::setFailCallback

GearmanClient::setStatusCallback

GearmanClient::setWorkloadCallback


http://docs.php.net/manual/en/gearmanclient.setdatacallback.php

http://docs.php.net/manual/en/gearmanclient.setdatacallback.php

http://docs.php.net/manual/en/gearmanclient.setcompletecallback.php

http://docs.php.net/manual/en/gearmanclient.setcompletecallback.php

http://docs.php.net/manual/en/gearmanclient.setcreatedcallback.php

http://docs.php.net/manual/en/gearmanclient.setcreatedcallback.php

http://docs.php.net/manual/en/gearmanclient.setexceptioncallback.php

http://docs.php.net/manual/en/gearmanclient.setexceptioncallback.php

http://docs.php.net/manual/en/gearmanclient.setfailcallback.php

http://docs.php.net/manual/en/gearmanclient.setfailcallback.php

http://docs.php.net/manual/en/gearmanclient.setstatuscallback.php

http://docs.php.net/manual/en/gearmanclient.setstatuscallback.php

http://docs.php.net/manual/en/gearmanclient.setworkloadcallback.php

http://docs.php.net/manual/en/gearmanclient.setworkloadcallback.php

PHP: Job API

Client receiving the status notifications:

30

dev:~/gearman# php -q client.php Running: true, numerator: 0, denomintor: 2Running: true, numerator: 1, denomintor: 2Running: false, numerator: 0, denomintor: 0


PHP: Client API

Setting up the client:

31


PHP: Client API

Job priorities and synchronous vs asynchronous:

32

$status returns an array containing status information for the job corresponding to the supplied job handle.

The first array element is a boolean indicating whether the job is even known, the second is a boolean indicating whether the job is still running, and the third and fourth elements correspond to the numerator and denominator of the fractional completion percentage, respectively.


Tasks vs Jobs

33


Application areas

34


Application areas

• Map/Reduce

• Log analysis and aggregation

• Image resizing

• Asynchronous queues

• URL processing

• Cache warm-up

• Database jobs

• Cloud services and applications

35


Map / Reduce

• Client sends request to intermediate job server

• Map / reduce worker splits up job (MAP)

• Each part is processed by workers

• Response is sent back to map / reduce worker

• Results are collected and aggregated (REDUCE)

• Client receives one reduced response

36


Map / Reduce

37

Client

Gearman Job server

Map/Reduce Worker

Client Client Client

Gearman Job server

Worker Worker Worker Worker Worker


Image resizing: worker

38


Image resizing: Client

39


Image resizing: output

$ php -q resize_worker.php &

$ php -q resize_client.php image.jpg

H:dev:39 - creating: small_image.jpg x:200 y:

Status: 1/1

$ ls *.jpg

image.jpg small_image.jpg

40


Other bits

41


Other bits

GearUp: Gearman based monitoring service

• No code yet, but looks promising

• http://launchpad.net/gearup

Apache logging through Gearman:

42


Other bits

Gearman and MySQL with Drizzle

• http://krow.livejournal.com/628025.html• todo

43


Links & sourcesCredits:- Red gears: http://lineymachine.googlepages.com/- Distributed computing: http://www.theleadblog.com/2009/06/lead-distribution-creating-right-sales.html- Old gears: http://decorate.pebblez.com

Gearman online:- http://www.danga.com/gearman/- http://gearman.org/- http://pecl.php.net/package/gearman/

44


Questions ?

45


Thank you!

Contact details:

Email: [email protected]: felixdv


mailto:[email protected]

mailto:[email protected]

High gear PHP with Gearman

Technology

Transcript of High gear PHP with Gearman