High gear PHP with Gearman

PHPBenelux meeting - 13/10/2009Felix De Vliegher

zondag 11 oktober 2009

Contents

• What is Gearman?

• Distributed work

• Once upon a time

• Advantages

• Gearman 101

• PHP interface

• Examples!

• Application areas

• Other bits

What is Gearman?And why should I care about it?

The way I like to think of Gearmanis as a massively distributed,

massively fault tolerant fork mechanism.

- Joe Stump, Digg

“”

A grain of truth

The name is an anagram for “Manager,”since it dispatches jobs to be done,

but does not do anything useful itself.

- From Gearman website

What is Gearman?

It’s an

Application frameworkto

distribute work

Some keywords

• Open source• Multi-language• Parallel• Asynchronous• Multi threaded -> blazingly fast• Persistent queues• Decentralized• Fault tolerant• tcp 4730• multiple client api’s (c, php, perl, drizzle, ...)

Distributed processing

Or: how to make others do all the work

“Distributed computing”

“autonomous computational entries”

(computers or nodes)

which are communicating by

message passing

• Actual work is distributed

• Handled by a number of “computational entities”

• Some sort of RPC

• Distributed parallel processing

• Shared nothing

• Decentralized

• Actual work is distributed

• Handled by a number of “computational entities”

• Some sort of RPC

• Distributed parallel processing

• Shared nothing

• Decentralized

Gearman!

Once upon a timeGearman history

Once upon a time

• 2005: http://brad.livejournal.com/2106943.html

• Originally a Perl implementation

• Created by Danga Interactive

• LiveJournal

• SixApart

• Guys behind Memcache and MogileFS

Once upon a time

• 2008: Rewrite in C by Brian Aker

• PHP Extension by James Luedke

• Gearman powers some of the largest sites around:

• Digg: 45+ servers, 400K jobs/day

• Yahoo: 60+ servers, 6M jobs/day

• Xing.com

• Maybe Netlog? ;-)

Advantages

• Speed up work

• Load balance work

• Parallel and asynchronous work

• Scales well

• Architecture-based workload distributing

• Call functionality in other programming languages

• Doesn’t block your apache processes!

Gearman 101Terminology and architecture

Terminology

Client Create jobs to be run and send them to a job server

Worker Register with a job server and grab jobs to run

Job Server Coordinates assignment from clients to workers, handles restarts

Terminology

Tasks vs Jobs

• A job is always a task

• A task is not always a job

• Single interface: jobs

• Concurrent interface: tasks

• Clients deal with tasks

• Workers deal with jobs

Architecture

Client application

Gearman Job server(gearmand)

Gearman Client API

Gearman Worker API

Worker application

Gearman Worker API

Worker application

Gearman Worker API

Worker application

Installing Gearman

• Job Server: gearmand

• Get it from https://launchpad.net/gearmand/• extract, configure, make, make install:

dev:/usr/local/src# wget http://launchpad.net/gearmand/trunk/0.10/+download/gearmand-0.10.tar.gz

dev:/usr/local/src# tar -xzvf gearmand-0.10.tar.gz

dev:/usr/local/src# cd gearmand-0.10/

dev:/usr/local/src/gearmand-0.10# ./configure --prefix=/usr/local/

dev:/usr/local/src/gearmand-0.10# make

dev:/usr/local/src/gearmand-0.10# make install

dev:/usr/local/src/gearmand-0.10# /usr/local/sbin/gearmand --help

Gearmand usage

/usr/local/sbin/gearmand -d -u <user> -L 127.0.0.1 -p 7003

-d Start as daemon in background

-u <user> Run as the specified user

-L <host> Only listen on the specified host or IP

-p <port> Listen on the specified port

Commandline gearman

• Client mode• ls | gearman -f function

• gearman -f function < file

• gearman -f function “foo data”

• Worker mode• gearman -w -f function -- wc -l

• gearman -w -f function ./script.sh

• Example:

dev:~/gearman# gearman -w -f foo -- grep GearmanClient &dev:~/gearman# cat client.php | gearman -p 7003 -f foo$client= new GearmanClient();

PHP Interface

PHP: Worker API

Possible to add multiple servers:

PHP: Worker API

Registering functions and any valid callback:

Pass application data to the functions:

# php -q client.php Sending jobCount: 1: HELLO!Count: 2: WORLD!

PHP: Worker API

Put it all together:

PHP: Job API

Status notifications:

GearmanJob::sendStatus(int $numerator, int $denominator)

GearmanJob::sendWarning(string $warning)

GearmanJob::sendComplete(string $result)

GearmanJob::sendFail(void)

GearmanJob::sendException(string $exception)

PHP: Callbacks

Provide functionality on different moments in the process:

GearmanClient::setDataCallback

GearmanClient::setCompleteCallback

GearmanClient::setCreatedCallback

GearmanClient::setExceptionCallback

GearmanClient::setFailCallback

GearmanClient::setStatusCallback

GearmanClient::setWorkloadCallback

PHP: Job API

Client receiving the status notifications:

dev:~/gearman# php -q client.php Running: true, numerator: 0, denomintor: 2Running: true, numerator: 1, denomintor: 2Running: false, numerator: 0, denomintor: 0

PHP: Client API

Setting up the client:

PHP: Client API

Job priorities and synchronous vs asynchronous:

$status returns an array containing status information for the job corresponding to the supplied job handle.

The first array element is a boolean indicating whether the job is even known, the second is a boolean indicating whether the job is still running, and the third and fourth elements correspond to the numerator and denominator of the fractional completion percentage, respectively.

Tasks vs Jobs

Application areas

• Map/Reduce

• Log analysis and aggregation

• Image resizing

• Asynchronous queues

• URL processing

• Cache warm-up

• Database jobs

• Cloud services and applications

Map / Reduce

• Client sends request to intermediate job server

• Map / reduce worker splits up job (MAP)

• Each part is processed by workers

• Response is sent back to map / reduce worker

• Results are collected and aggregated (REDUCE)

• Client receives one reduced response

Map / Reduce

Client

Gearman Job server

Map/Reduce Worker

Client Client Client

Gearman Job server

Worker Worker Worker Worker Worker

Image resizing: worker

Image resizing: Client

Image resizing: output

$ php -q resize_worker.php &

$ php -q resize_client.php image.jpg

H:dev:39 - creating: small_image.jpg x:200 y:

Status: 1/1

$ ls *.jpg

image.jpg small_image.jpg

Other bits

GearUp: Gearman based monitoring service

• No code yet, but looks promising

• http://launchpad.net/gearup

Apache logging through Gearman:

Other bits

Gearman and MySQL with Drizzle

• http://krow.livejournal.com/628025.html• todo

Links & sourcesCredits:- Red gears: http://lineymachine.googlepages.com/- Distributed computing: http://www.theleadblog.com/2009/06/lead-distribution-creating-right-sales.html- Old gears: http://decorate.pebblez.com

Gearman online:- http://www.danga.com/gearman/- http://gearman.org/- http://pecl.php.net/package/gearman/

Questions ?

Thank you!

Contact details:

Email: felix@phpbenelux.euTwitter: felixdv

High gear PHP with Gearman

Technology

Transcript of High gear PHP with Gearman

Gearman bundle, Warszawa 2013 edition

PHP Development - Introductionmhughes/Lectures/PHPDevelopment1.pdfPHP Development - Introduction . Php Hypertext Processor • PHP stands for PHP: Hypertext Preprocessor • PHP is

High gear PHP with Gearman (phpDay 2010)

Gearman and asynchronous processing in PHP applications

Gearman and Perl

COMP519: Web Programming Autumn 2010 PHP Basics: Introduction to PHP a PHP file, PHP workings, running PHP. Basic PHP syntax variables, operators,

Gearman for MySQL

Distributed Queue System using Gearman

Distributed Applications with Perl & Gearman

Egearmand: an Erlang Gearman daemon

Nagios and Mod-Gearman In a Large-Scale Environment Jason Cook 8/28/2012.

PHP Certified training Coimbatore|PHP institutes Coimbatore|PHP certification

1 Open Source Programming. -Introduction to PHP -PHP installation /wamp server installation for PHP environment -PHP syntax -PHP variables -PHP Strings.

Nagios Conference 2011 - Mike Weber - Training: Reducing Nagios Server Load With Mod-Gearman

MapReduce Using Perl and Gearman

Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011

Energy Regulatory Commission · Prtce Amount PhP Amount Amount Amount PhP PhP PhP PhP COMMON OFFICE SUPPLIES PhP PhP Ball IntPen fioe int black 5.00 500 2500.00. 1300 ... Portable

Gearman 200907 Oscon Tutorial

Gearman, Supervisor and PHP - Job Management with Sanity!

Gearman - Job Queue