Maximizing Revenue Through Logging Albert Ho

About Rumble Funded by: Mission: ●  Bring AAA quality to F2P ●  Play together cross-device/platform

Rumble’s Platform ●  Played in >200 countries ●  Six data centers worldwide ●  Hybrid infrastructure ●  Play together on:

What this talk is about…

User Acquisition

Funnel Loss Retention Loss

Your Game Paid Users

Free Users

“Logging” Distinctions ●  User behavior

●  Game design, retention, virality, monetization

●  System behavior ●  Maximize potential ●  Lower CPA

Thesis “You really aren’t maximizing revenue

unless you can measure system behavior”

Challenges ●  24/7 operations ●  Game teams ship code daily ●  Being “up” is not good enough

The Long Tail of Revenue Lost: Daily Site Problems Affect Weeks of Revenue

Outage

Problem

Recovery

New Installs/Sales

Revenue

Days

How to Detect Problems

●  Solving is Harder than it sounds Signal to noise ratio + Indexing and search + Multiple game teams shipping code daily = BIG Headache and world’s biggest chore

The answers are in the logs!

Rumble Logging 1.0 ●  Graylog, Elasticsearch ftw?

●  1 day to setup ●  Collecting for 2 game servers ●  Few hundred MB a day / 7-day retention time ●  Scaling is easy… right?

- Bigger hard disks, more ES nodes

Rumble Logging 1.5 ●  7 days to setup ES cluster ●  > 20 servers ●  Logs @ TB a day / 3-day retention time ●  Log noise incomprehensible ●  Very expensive!!! ●  Logging cluster broke every day

Key Learnings Log Management is really… “Undifferentiated Heavy Lifting”

We were spending more on LOGGING than HOSTING

There must be an expert that does it better, faster, and cheaper

Rumble Logging 2.0 Requirements ●  Must support JSON ●  Easy to integrate ●  Can scale up and meet retention time goals ●  Cost-effective

Enter…….

●  No more agents! ●  REST API – 21st century please/no syslogs ●  Scalable and Responsive ●  Superb partner relationship

Sample Log

Loggly Use Cases ●  Log clean-up ●  Release monitoring and debugging ●  Customer service requests ●  Trend analysis ●  3rd-party developers

Log Clean-up ●  1000’s errors per minute noise ●  DEBUG, INFO, WARN, ERROR? ●  Reduce TB’s of logs to GB’s

Release Monitoring ●  Log snapshots before and after

Release Monitoring ●  Game teams own

process ●  All logs in one place ●  Easy for PM’s

Customer Service Reactive problem solving ●  “Why is customer x slow?” ●  “Why did I not get my gems?” !

Proactive debugging ●  “These 20 customers were affected by

today’s issue.”

Trend Analysis Game performance analysis over time

Client Lag

Days

3rd Party Development ●  Bake logging into games at the beginning

●  Including 3rd party ones…

Loggly Live Demo

Loggly Before And After Before After

Log Retention 3 days 15 days

Log Availability 50% uptime >99.9%

Debugging Speed Days Minutes

Happiness Factor

Questions? albert.ho@rumbleentertainment.com KingsRoad http://kingsroad.com http://apps.facebook.com/kingsroadgame

Ballistic https://apps.facebook.com/ballistic