Holding Effective Data Meetings Ben Weber

Director of BI & Analytics

April 30, 2015

Analytics Summit Goals

Learn how to build an effective data team

Discover new techniques and tools for mining data insights

Learn how to effectively interact with game teams to act on data insights

Questions

What data to share with game teams?

When and how to share data?

How to operationalize your data?

Barriers to Using Data

Little communication between the data and game teams

Automated reports not providing actionable data

Data team is not working on actionable issues

Team structure separates game developers from analysts

Daybreak’s Approach

Integrate analysts into game teams

Develop consistent reporting across the portfolio and teach teams how to use the data

Frequently meet with game teams to identify areas of opportunity

Overview

Team Structure

Sharing Data

Meetings

Acting on Data

BI Team Structure

Centralized Roles – Director of BI & Analytics – Data Architect – Ecommerce Analyst

Embedded Analysts – H1Z1 – PlanetSide 2 – DC Universe Online

Database Operations

Centralized Analyst Role

Performs data analysis across the portfolio Supports the marketing and exec teams by building

portfolio-wide KPIs and reports Types of Analysis

– User acquisition funnels – Email campaign ROI – Ad-hoc analysis for centralized teams

Embedded Analyst Role

Co-located and considered part of the game team Supports the producer and creative director with game

design and marketplace insights Types of Analysis

– Game balance – In-game retention funnels – Ad-hoc analysis for the game team

What Data to Share?

All game-specific KPIs are provided to all members of the team

Ad-hoc analysis is first shared with the leads and communicated to relevant team members

Portfolio KPIs are shared across the company

Reports

Automated Reports – Daily KPIs

– Hourly KPIs

– Monthly KPIs

– Self-Service Portal

Data Pipeline – Vertica, Tableau Server, and Excel

Meetings

Analytics 101

Data Scrum Meetings

Data Insights Meetings

Monthly Business Reviews

Analytics 101

Goal – Explain our automated reports to game teams

Attendees

– Everyone!

Agenda

– Define all of those fun acronyms – Provide context for each of the metrics being tracked – Team members ask questions and provide feedback

Data Scrum Meetings

Goal – Meet with game team weekly to track KPIs

Attendees

– Game team leads, brand manager, eComm team

Agenda

– Discuss KPI trends and targets – Identify ad-hoc analysis to perform – Evaluate the impact of promotional events

Data Insights Meetings

Goal – Share insights from portfolio or game-specific analysis – Scheduled as necessary – Example: item drop rates in a game are too high

Attendees – Game team leads

Agenda – Explain the results of the analysis – Indentify how to respond to the findings

Monthly Business Reviews

Goal – Monthly meeting to review game performance

Attendees

– Senior management and game team leads

Agenda

– Review KPIs for the past and current months – Present substantial data insights – Try to get executives to become data evangelists

Acting on Data Insights

How can we get game teams to act on data insights?

Interaction Approaches

– Hand-off the results

– Own a data model

– Share a data model

– Show ROI results

The Hand-Off Approach

Overview – The data team performs an analysis and hands-off the results to

the game team – The game team decides how to use the data

Examples

– DCUO Character Traits – PlanetSide 2 Starter bundles – PlanetSide 2 Implant Drops

Improving Hand-Offs

Hand-offs can be improved by incorporating the game team earlier and following up

Approach

– Share preliminary data insights – Schedule a “Data Insights” meeting – Perform additional analysis – Deploy the change – Schedule a follow up “Data Insights” meeting

Owning a Data Model

Overview – The data team develops a model that provides input to an

in-game mechanism

– On a daily or weekly basis, the game team ingests a targeted user list

Examples – DCUO Nudge System

– PlanetSide 2 Tutorial Targeting

Improving Owned Models

Issues – Manual data hand-off

– Multiple failure points

– Model is a black box to the game team

Recommendation – Automate as much of the process as possible and allow

game teams to run competing approaches

Sharing a Data Model

Overview

– The data team builds and validates the models

– The game team implements the models in-game

Examples

– EverQuest Landmark Recommendation System

– PS2 Nudge System

Landmark’s Recommendation System

Improving Shared Models

Work with the game team to make lots of parameters available for the models

Implement fall-back approaches in case the models fail

Use a modular approach for implementing the model rules and parameters

Showing ROI Results

Overview – The data team performs analysis that does not require work

from the game team

– The results are shared with the game team and used to plan for future campaigns

Examples – A/B testing email campaigns

– Advertising on PSN

Summary

Meeting Types – Analytics 101 – Data Scrum Meetings – Data Insights Meetings – Monthly Business Reviews

Interaction Approaches – Hand-off the results – Own a data model – Share a data model – Show ROI results

Questions

What data to share with game teams? – All the KPIs and more detailed analysis as needed

When and how to share data?

– Regularly through automated reports, scrum meetings, and data insights meetings

How to operationalize your data?

– Meet early with preliminary data, develop an action plan, and follow up with more data

Thank You

Ben Weber

Director of BI & Analytics

Daybreak Game Company

DaybreakGames.com

@bgweber

Holding Effective Data Meetings

Ben Weber Daybreak Game Company