Low Latency Loggingwith RabbitMQ

BrnoPHP Conference 2014

James Titcumbwww.jamestitcumb.comwww.protected.co.ukwww.phphants.co.uk@asgrim

Who is this guy?

Let’s go back to basics...

Errors

source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/

error_reporting(0);

They look rubbish!

@

Log to file only

Ways Around// Will raise E_WARNING

fopen($somefile, 'r');

Ways Around// No error! :)

if (file_exists($somefile)) {

fopen($somefile, 'r');

} else {

// nice handling...

// maybe throw exception...

}

Exceptions

Jargon Buster

● throwTriggering

● tryTry to run

● catchHandle it

● finallyRun code after try/catch

source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/

C♯/C++

Catchable

Turn into fatal errorsif not caught

Classy

Descriptive exceptions

Example (exception class)class DivisionByZeroException

extends LogicException

{

}

Example (throw)class Mathematics {

public function divide($a, $b) {

if ($b == 0) {

$msg = sprintf(“Divide %s by zero”, $a);

throw new DivisionByZeroException($msg);

}

return ($a / $b);

}

}

Example (catch)$maths = new Mathematics();

try

{

$result = $maths->divide(5, 0);

}

catch (DivisionByZeroException $exception)

{

$logger->warning($exception->getMessage());

}

Throw Descriptively

Logging

What is “logging”?

What is “logging”?Keeping a record of all events...

What is “logging”?Keeping a record of all events...

exceptions, errors, warnings, info, debug

Paper Trail

Log like you just don’t care

Log like you just don’t care

Log wherever you like

What about Apache’s error_log?

source: http://up-ship.com/blog/?p=20903

Doin’ it rightwrong… // Magic file that makes your entire project work perfectly

@ini_set('display_errors', 0);

@error_reporting(0);

function __globalErrorHandler() {

return true;

}

@set_error_handler('__globalErrorHandler');

@set_exception_handler('__globalErrorHandler');

@register_shutdown_function(function() {

if(error_get_last())

echo "Script executed successfully!";

}); https://github.com/webarto/boostrap.php

Fire & Forget

Low Latency

Highly Available

PSR-3

● monolog (PSR-3)● Drupal - PSR-3 Watchdog● phpconsole● log4php● RavenPHP + Sentry● FirePHP (dev environment)● Roll your own

Logging Options

source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html

Capturing Messages

Capturing Messagesset_error_handler()

Capturing Messagesset_exception_handler()

Capturing Messagesregister_shutdown_function()

Capturing MessagesUsing the Logger Directly

Adapter / Handler

Capture Method

Logger (PSR-3)

Handler / Adapter

Data Storage

Typical PSR-3 Compatible Design

Monolog\ErrorHandler->handleException()

Monolog\Logger->log()

Monolog\Handler->handle()

Monolog

Low Latency Logging

Make it easy

Fire & Forget

Minimum Impact

Slow Logging

ApplicationBrowser Log Server

HTTP request

Send log message to log server

Error!

Acknowledge message

HTTP response to client

Zero Latency Logging (ideal)

ApplicationBrowser Log Server

HTTP request

Send log message to log server

Error!

HTTP response to client

UDP?

We need a balance.

Low Latency Logging (balance)

ApplicationBrowser Log Server

HTTP request

Send log message to log server

Error!

HTTP response to client

…so how?

Disclaimer...

Say hello to RabbitMQ

RabbitMQ - Basic

source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html

Producer Consumer

test_queue

1 2 3 4 5

RabbitMQ - Exchanges

source: http://www.rabbitmq.com/tutorials/tutorial-three-php.html

Exchange

Consumertest_queue

1 2 3 4 5

Consumer

Producer

Producer

Producer

Exchanges are flexible

RabbitMQ === Fast!

Ed\Log\Handler\ErrorHandler->handleException()

Ed\Log\Logger->log()

Ed\Log\Publisher\AmqpPublisher->publish()

Logging Server

Low Latency (using AMQP)

RabbitMQ JSON payload

Fetch message

Low Latency Logging (with AMQP)

ApplicationBrowser Log Server

HTTP request

JSON via AMQP

Error!

HTTP response

RabbitMQ

Why bother?● Scalability

RabbitMQ

Application A

Application B

Application C

Log Worker

Log Worker

Log Worker

Log Worker

Single Point of Failure...

RabbitMQNode 1

RabbitMQNode 3

RabbitMQNode 2

RabbitMQNode 4

RabbitMQNode 5

RabbitMQNode 6

To recap...

Scalable.

Easy.

Fast.

Future proof.

Questions?

James Titcumb@asgrim

Thanks for watching!