Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging
28
Maximizing Revenue Through Logging Albert Ho
description
How do you balance the dynamic needs of millions of concurrent users, support hundreds of developers, optimize a distributed cloud environment, push code releases daily, and stay sane? It starts with visibility into all your data and real-time insight from your log files. In this session at the 2014 Game Developers Conference, Albert Ho provided a step-by-step outline of how Rumble Entertainment maximizes revenue in online games by: Instrumenting its games and platform to facilitate global troubleshooting Gathering insight from log files to identify, quantify and solve operational issues Conducting transactional tracing starting from player feedback on open social networks to isolate the source, server and all users affected by the issue for remediation Baking Loggly into the DNA of its games to serve as a single source of truth and extended it to how we work with partners Correlating real-time visibility directly to revenue to speed decision making and planning
Transcript of Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging
Maximizing Revenue Through Logging Albert Ho
About Rumble Funded by: Mission: ● Bring AAA quality to F2P ● Play together cross-device/platform
Rumble’s Platform ● Played in >200 countries ● Six data centers worldwide ● Hybrid infrastructure ● Play together on:
What this talk is about…
User Acquisition
Funnel Loss Retention Loss
Your Game Paid Users
Free Users
“Logging” Distinctions ● User behavior
● Game design, retention, virality, monetization
● System behavior ● Maximize potential ● Lower CPA
Thesis “You really aren’t maximizing revenue
unless you can measure system behavior”
Challenges ● 24/7 operations ● Game teams ship code daily ● Being “up” is not good enough
The Long Tail of Revenue Lost: Daily Site Problems Affect Weeks of Revenue
Outage
Problem
Recovery
New Installs/Sales
Revenue
Days
How to Detect Problems
● Solving is Harder than it sounds Signal to noise ratio + Indexing and search + Multiple game teams shipping code daily = BIG Headache and world’s biggest chore
The answers are in the logs!
Rumble Logging 1.0 ● Graylog, Elasticsearch ftw?
● 1 day to setup ● Collecting for 2 game servers ● Few hundred MB a day / 7-day retention time ● Scaling is easy… right?
- Bigger hard disks, more ES nodes
Rumble Logging 1.5 ● 7 days to setup ES cluster ● > 20 servers ● Logs @ TB a day / 3-day retention time ● Log noise incomprehensible ● Very expensive!!! ● Logging cluster broke every day
Key Learnings Log Management is really… “Undifferentiated Heavy Lifting”
We were spending more on LOGGING than HOSTING
There must be an expert that does it better, faster, and cheaper
Rumble Logging 2.0 Requirements ● Must support JSON ● Easy to integrate ● Can scale up and meet retention time goals ● Cost-effective
Enter…….
● No more agents! ● REST API – 21st century please/no syslogs ● Scalable and Responsive ● Superb partner relationship
Sample Log
Loggly Use Cases ● Log clean-up ● Release monitoring and debugging ● Customer service requests ● Trend analysis ● 3rd-party developers
Log Clean-up ● 1000’s errors per minute noise ● DEBUG, INFO, WARN, ERROR? ● Reduce TB’s of logs to GB’s
Release Monitoring ● Log snapshots before and after
Release Monitoring ● Game teams own
process ● All logs in one place ● Easy for PM’s
Customer Service Reactive problem solving ● “Why is customer x slow?” ● “Why did I not get my gems?” !
Proactive debugging ● “These 20 customers were affected by
today’s issue.”
Trend Analysis Game performance analysis over time
Client Lag
Days
3rd Party Development ● Bake logging into games at the beginning
● Including 3rd party ones…
Loggly Live Demo
Loggly Before And After Before After
Log Retention 3 days 15 days
Log Availability 50% uptime >99.9%
Debugging Speed Days Minutes
Happiness Factor
Questions? [email protected] KingsRoad http://kingsroad.com http://apps.facebook.com/kingsroadgame
Ballistic https://apps.facebook.com/ballistic
