Logging with Perl and Fluentd

Logging and Analytics Using Perl and Fluentd

Jason Crome Technical Lead, All Around the World

June 21, 2016 Copyright 2016, Jason A. Crome

Who’s this for?

• People who want to know more about enterprise logging (beginner and intermediate)

• People who want to know about structured logging

• People who want to know a bit about fluentd

• Some code, some setup, some discussion.

Copyright 2016, Jason A. CromeJune 21, 2016

Who Am I?• Project manager for Veure at AAW

• Formerly spent 18 years writing software for local government

• Dancer core team member

• Student pilot and hockey player (probably not relevant but fun to mention anyhow)


Veure’s Building Blocks• Catalyst

• Template Toolkit

• DBIx::Class

• Moose

• PostgreSQL


– Colin Eberhardt, The Art of Logging

“Logging is the process of recording application actions and state to a secondary

interface.”


What is Logging?

Why Do We Log?

• Prevent cheating

• Help determine effectiveness of game content

• Find out what players are spending money on and why

• Help us adjust difficulty of the game


Why Fluentd?• Log collector/aggregator (“Unified Logging Layer”)

• Log from any layer in the stack

• Asynchronous

• Cluster-able

• Expandable


Why Fluentd?• Supports structured logging

• Self-hosted

• Affordable/free

• Memory-friendly

• Built-in reliability

• Super easy to set upCopyright 2016, Jason A. CromeJune 21, 2016

Why not Fluentd?

• Ruby :/

• Overkill for many projects


Alternatives

• logstash

• Splunk

• Loggly

• Papertrail


Why Log4Perl?

• Does what we need

• Has everything and the kitchen sink

• Easily integrates with Catalyst

• It already talks to Fluentd (so we don’t have to build this ourselves)


Where Do We Log From?

• DBIx::Class

• Other Moose-based classes

• Catalyst

• TT (this is rare)

• Stack (future)


What Do We Log?


What Do We Log?

Copyright 2016, Jason A. Crome

Everything

June 21, 2016

What Do We Log?


Everything

(seriously)

June 21, 2016

What Do We Log?

• With an MMO of any sort, you must log all sorts of things you might not otherwise care about

• None of us are veteran game designers (some of us aren’t even game players!), and need all the information we can get from our game


• Logins

• Access log

• Errors

• Warnings

• Character events

• Character actions

• Randomness

• Courses taken

• URL Spoofing

• Chat hacking

• Combat

• Item Transfers

• Credit Transfers

• Auction house results

• Real-World Purchases

• Monetization events

• Security

• A/B test results (future)


What Do We Log?

June 21, 2016

Randomness• Really?

• Things left up to chance (combat success/failure) should be measured for effectiveness

• Veure::Role::Random is used wherever randomness is needed

• Right now, only collecting information. Insufficient data to get meaningful analysis.


Randomness (cont.)

• Allows us to easily modify the difficulty of all rolls

• Allows us to easily change the random number generator used

• Running all random checks through one spot makes logging easy


How Do We Log?

• Everything runs through log4perl

• Log4perl uses the Fluentd appender

• Log methods in Catalyst routed to Fluentd

• Logs are structured data. Turned into JSON or stringified.


The Code!

• Based on Log::Message::Structured

• L::M::S automatically logs all attributes

• L::M::S does not let attributes contain objects

• Both of these are a problem for us


The Solution

• Veure::Role::LogEvent

• Along with a trait, allows us to pass in an object and ignore what we don’t want

• Individual attributes can be set to only log at a specific verbosity level


Veure::Role::LogEvent::Character

has character => ( is => 'ro', isa => 'Veure::Schema::Result::Character', );

has ch => ( traits => ['Logged'], is => 'ro', isa => 'Str', lazy => 1, default => sub { my $self = shift; if ( my $char = $self->character ) { return $char->slug; } }, );


Randomnesssub attempt_to { my ($self, $type, $chance) = @_;

my $rand = rand(); my $success = $rand < $chance ? 1 : 0;

$self->log_event('RandomAttempt', type => $type, success => $success, chance => $chance, value => $rand, ) if $type; return $success; }


Randomnesspackage Veure::LogEvent::RandomAttempt; use Moose; use namespace::autoclean;

with 'Veure::Role::LogEvent';

use Veure::LogEvent::Trait::Logged;

has type => ( traits => ['Logged'], is => 'ro', isa => 'Str', );

has [qw(chance value)] => ( traits => ['Logged'], is => 'ro', isa => 'Num', );

has success => ( traits => ['Logged'], is => 'ro', isa => 'Bool', );

__PACKAGE__->meta->make_immutable; 1;


Exchanges my $exchange = $self->exchange( action => 'purchase_visa', messages => { success => 'You purchased a visa', failure => 'You could not purchase a visa', }, steps => [ [ Wallet => remove => $visa_price ], [ Visa => add => $affiliation->affiliation_id =>

$visa_dur ], ], ); return $exchange->attempt;


Exchangespackage Veure::LogEvent::Exchange;use Moose;use namespace::autoclean;

with qw( Veure::Role::LogEvent Veure::Role::LogEvent::Character Veure::Role::LogEvent::CharacterLocation Veure::Role::LogEvent::CharacterVitals);

use Veure::LogEvent::Trait::Logged;

has exchange => ( is => 'ro', isa => 'Veure::Economy::Exchange', required => 1,);

has '+character' => ( lazy => 1, default => sub { shift->exchange->character },);

has steps => ( traits => ['Logged'], is => 'ro', lazy => 1, default => sub { shift->exchange->steps },);


Exchangeshas action => ( traits => ['Logged'], is => 'ro', lazy => 1, default => sub { shift->exchange->action }, );

has counterparty => ( traits => ['Logged'], is => 'ro', isa => 'Str', default => 'GAME', );

has success => ( traits => ['Logged'], is => 'ro', lazy => 1, default => sub { shift->exchange->success }, );

has details => ( traits => ['Logged'], is => 'ro', lazy => 1, default => sub { shift->exchange->_messages }, );



Access Logging

# End action of Veure::Controller::Root if ($self->log->is_info && ($ENV{TEST_ACCESSLOG} || !$ENV{TESTING_VEURE})) { my $req = $c->request; my $elapsed = $c->use_stats ? $c->stats->elapsed : '??'; $self->log_event( 'Accesslog', uri => $req->uri->as_string, ua => $req->user_agent, method => $req->method, args => $req->uri->query, timing => $elapsed, ); }


Access Loggingpackage Veure::LogEvent::Accesslog; use Moose; use namespace::autoclean;

extends 'Veure::LogEvent';

has [qw(uri method args timing)] => ( is => 'ro', isa => 'Maybe[Str]', );

has ua => ( traits => ['Log'], log_level => 2, is => 'ro', isa => 'Maybe[Str]', );



Verbosity

• Overall Verbosity vs. Individual Verbosity

• Can adjust the overall verbosity on different subsystems within the game

• With individual verbosity, we can log everything a player does (future)


Verbosity

# Set to 0 to disable logging for that LogEvent <LogEvent> Accesslog 2 Exchange 1 RandomAttempt 1 RandomValue 1 Scavenge 0 Travel 1 </LogEvent>


Where do the Logs Go?• For a developer: local filesystem

• For a development or production server: Postgres

• Developers can also install local instances of fluentd if so inclined and do what they want with the log output

• Fluentd lets you easily configure multiple outputs


Stringifying Output

• Use Tie::IxHash to order hashes

• Log::Message::Structured::Stringify:: AsJSON does the rest


Stringifying Outputpackage Veure::Role::LogEvent; use Moose::Role; use namespace::autoclean;

with qw( Log::Message::Structured Log::Message::Structured::Stringify::AsJSON );

use MooseX::ClassAttribute; use Veure::LogEvent::Trait::Logged; use Tie::IxHash;

# …snip…

sub as_hash { my ($self) = @_; my %hash; tie( %hash, 'Tie::IxHash' ); %hash = map { $_ => $self->$_ } @{ $self->logged_attrs }; return \%hash; }

1;


PostgreSQL (JSON)# Easiest <match Veure.**> type pgjson host localhost port 5432 sslmode require database veure table fluentd user veure_user password abcdefg time_col time tag_col tag # Classname record_col record </match>


# Maps to this… CREATE TABLE fluentd ( tag Text ,time Timestamptz ,record Json );


PostgreSQL (JSON)

Text Output


# Developer

2016-04-28 14:37:16 INFO Veure::Role::WithLogging:184 127.0.0.1 cromedome> {"args":null,"class":"Veure::LogEvent::Accesslog","method":"GET","timing":0.044844,"ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36”,"uri":"http://localhost:5000/combat"}

2016-04-28 14:37:16 INFO Veure::Role::WithLogging:184 127.0.0.1 cromedome> {"chance":0.9,"class":"Veure::LogEvent::RandomAttempt","success":1,"type":"Flee","value":0.743257306080512}

2016-04-28 14:37:16 INFO Veure::Role::WithLogging:184 127.0.0.1 cromedome> {"class":"Veure::LogEvent::RandomValue","max":5,"min":0,"step":null,"type":"Area","value":2}

# Fluentd

2016-03-18T09:22:55-04:00 Veure.Model.DB.Character {"success":0,"stamina":0.9,"level":1,"class":"Veure::LogEvent::Scavenge","item":"","character":"cees","reason":"Nothing found"}

June 21, 2016

Testing

• Log messages easily testable, but log must be turned on to be tested

• Test::Role::LogTesting used for testing log output

• Look for specific hash values in log message


Problem (cont.)

• Couldn’t you just test the individual log events rather than sending them through Test::Log4perl? Yup.

• So why didn’t you? Time.


Problem• Test::Log::Log4perl only lets us run one

regex against a log entry

• To solve this, we build up a big regex in parts and test for all hash entries we care about

• Test::Role::LogTesting does the dirty work for us

• It’s not pretty.


Code# By default access logging is disabled in the test suite local $ENV{TEST_ACCESSLOG} = 1;

# With Accesslog = 1 we should see log record with no ua local config->{LogEvent}->{Accesslog} = 1; $test->log_expect('Veure.LogEvent.Accesslog', info => $test->log_event_check({ "args" => undef, "class" => "Veure::LogEvent::Accesslog", "method" => "GET", "timing" => \"anything", "ua" => \"missing", "uri" => “http://localhost/travel/area/ruins" }) ); $mech->get('/travel/area/ruins'); $test->logs_ok;


How Do We Analyze?

• For the easy cases (logins, access log, etc), SQL select record->>'distance' as distance from fluentd where record->>'class' = ‘Veure::LogEvent::Travel';

select record->>'distance' as distance from fluentd where record::jsonb @> '{"class" : "Veure::LogEvent::Travel", "distance" : "1.74"}';


Game Balance/Log Analysis

• For game balancing and other analysis, more is needed. Using randomness as an example…

• For each type of roll, track chance to succeed, roll, and success

• Periodically (daily? weekly?) determine average roll, % of success, standard deviation, other stats

• Are we too high? Too low?



• If we need to adjust, we have easy/medium/hard settings to tweak the success rate.

• These settings adjust the success rate by a multiple of the standard deviation.

• Still haven’t accumulated enough data for this.



• Leveling subject to this too - intended to be slow, but not too slow

• Tracking experience gain, look at daily progression over time. Determine average gain, standard deviation, etc.

• Based on the information we have collected, increase or decrease experience gain using standard deviation as a guide.


Economic Balance• Economy = transfer of one resource (time, money,

item) for another (course, item, skill)

• Log every type of transfer in game and absolute amount of each resource

• Gives us both state and the flow of all resources in game.

• Visualizing flow through Sankey diagrams and other tools


Economic Balance


Log Analysis - Future

• Redshift (Amazon AWS - based on Postgres)

• keen.io


http://keen.io

Log Retention

• Having lots of data is nice, where to keep it all?

• Current: archive after 1 month

• Future: archive weekly?

• With an agile release cycle, anything older than one month becomes much less relevant


The Future

• Overall Verbosity vs. Individual Verbosity

• A/B testing results

• Stack logging via fluentd


Final Thoughts

• Logging is a process. Don’t ever expect to get it right.

• Make something flexible enough to expand later.

• Once you show people what information they can get, they will constantly ask you for more.


Credits

• Cees Hek (AAW, Toronto.pm)

• Noel Maddy (AAW)

• https://bost.ocks.org/mike/sankey/ (Sankey diagram)


https://bost.ocks.org/mike/sankey/

Questions?


Bonus Slides! (?)


What’s Veure?• Catalan verb for “to see”.

• Pet project/dream of Ovid

• Massively Multiplayer Online Bulletin Board Game (MMOBBG)

• Set in dystopian future where humanity has colonized the galaxy, and eventually pushed to the brink of extinction

• Think Mad Max meets FireflyCopyright 2016, Jason A. CromeJune 21, 2016

What’s Veure?


Logging with Perl and Fluentd

Software

Transcript of Logging with Perl and Fluentd