It's in the game: The path to micro-services at Electronic Arts with Docker

Andrew HatelyIBM CTO Cloud Architecture

Scott PorterEA Cloudcell Server LeadMelbourne Firemonkeys

GAMING MARKET GROWING ACROSS ALL PLATFORMS AND REGIONS

Mobile PC/F2P Console

NORTH AMERICA LATAM EUROPE ASIA/ROW

GROWTH YoY 12%

REVENUES 2015est.

21BGROWTH YoY 19%

REVENUES 2015est.

4BGROWTH YoY 7%

REVENUES 2015est.

17BGROWTH YoY 9%

REVENUES 2015est.

40B

17% 46% 37%41% 16% 43%

195M 182M 553M 817M

33% 24% 43% 51% 44% 5%

2015 est. Revenues Consist of Software Gaming Market

MOBILE FUELING INDUSTRY GROWTH

2009 2010 2011 2012 2013 2014 2015$3.3 $4.3 $6.1

$9.9

$17.0

$27.8

$34.0

Mobile games industry has grown 10x in less than a decade

Mobile Gaming Revenue, WW $B

Mobile Game Server architecture/requirements• Mobile Gaming Back-end is a very typical Web application• Frequent updates to game content - infrequent updates to game

engine• Daily peak users 150,000 to 250,000 simultaneous users and 1-2

million daily • Need to scale-up/down around release and content pack releases• Stateless front-ends with memory caching and both SQL and no-

SQL DB• 50TB per day of data - Increasing use of analytics on user

experience, recorded game telemetry, replays, etc.

Firew

all

Load

Ba

lanc

er

NGNI

X

Game Server REDIS

Memcache

MySQL cluster

PHPPHPPHPPHPPHP

Object Storage

Demo – Docker for Gaming

• Our current Ops complexity

• Partnership with IBM

• Why we believe Docker can help

• How we integrated Docker

Scott PorterCloudcell Server LeadEA Melbourne Firemonkeys

scott@ea.com@sl_porter

EA MELBOURNE FIREMONKEYS

• Moble studio based in Melbourne, Australia

• Formed in 2012 from two EA Mobile studios, Iron Monkey and Firemint

• Focused on developing the highest quality mobile titles

• Firemonkeys in house Client/Server Tech Stack

• Mobile focused connected features

• Originally developed by Firemint for Flight Control

Integrated into all Firemonkeys live titles

• Live Service Games

• Bi monthly client releases

• No downtime

What do we do?

• 13.5k Peak Requests/s

• ~900M Requests/day

• ~430k Peak Concurrent Players

• 5M – 8M Daily Active Players

• > 100 TB Binary Data

• 170 Hosts (60 Live Hosts)

• Team of 7

Current Scale

• Baremetal Hosts

• Nginx, PHP, Memcache, MySQL, Redis

• Batched client requests, short-lived, stateless

• Requests can touch any datastore

Architecture

Game Logic Data StorageLoad Balancing

Architecture

External Services

StatsD / Diamond / Graphite / Graphana

Object Storage

Datadog

• Automated Development / Manual Live process

• In house database migration and code deployment system

• Bash & rsync

• Rollbacks are handled via code duplication

Current Deployment Methodology

• Asynchronous processes (workers) manually configured ( services, cron jobs )

• Brittle.

• SaltStack is awesome!

• But still slow, complex, no HA

Current Orchestration Methodology

• Moved binary data to SoftLayer Object Storage (Open Stack Swift)

• Currently provision for Peak

• Shard live environments per game title

• Interested in transitioning to microservices

Current Scalability

• Improved deployments

• Growing ecosystem

• Scalable

• Easy transition, run side by side with existing stack

Why Docker?

• Focused on the Game Logic Layer

• Decomposition• Nginx• Admin• Application• Workers

Packaging Cloudcell

• Generic Containers

• Run multiple containers, not processes

• Tradeoff considerations

Packaging Cloudcell

Code

Building Images

Jenkins

Docker Registry

Base Images

nginx

php-fpm

php-worker

Built Images

php-fpm-nfs-0001

php-fpm-nfs-0002

php-worker-a-1

nfs-0002.tar

worker-a-1.tar

nfs-0001.tar

Dev Env

Jenkins

Application Code (SVN)

• Fleet

• Unit files per service• Nginx• Admin• Application• Each worker type

Orchestration

Load Testing

Host

Baremetal

nginx

Application

Worker A Load Generation

LTG LTG

LTG LTG

LTG

LTG

Host

Docker

Docker

nginx

php-fpm-nfs-0001

php-worker-a-1

etcd

Fleet

• Overall 5 – 10% overhead running Load Test with Docker

Load Testing Results

PLACEHOLDER NEED A REAL GRAPH HERE!

• Docker Private Registry

• Jenkins• Studio Jenkins ships code to build machine• Build machine builds and commits images to

Registry• Generates Fleet configuration from templates• Executes Fleet deployments

Deployments

• Config Editor• Bridges Cloudcell Application deployments to

environment state• Provides a simple UI for building, deployments,

and state via Jenkins and Fleet

Deployments

Deployments

Docker Build Machine Host

Docker

etcd

Jenkins

Docker Registry

Config Editor nginx

php-fpm-nfs-0001

php-worker-a-1

Fleet

Docker enhanced architecture on SoftLayerCentOS 7 host

Discovery/Config

Fleet managed Docker services

nginx-confd.service

NGINX-confd container

Orchestration Docker

php-fpm@1.servic

eea/php-fpm:latest

1

php-fpm@2.servic

eea/php-fpm:latest

2php-fpm-discovery

@2.service

php-worker-A@2.service

ea/php-worker:latest

2

php-worker-B@1.service

ea/php-worker:latest

1

php-worker-A-discovery@2.service

php-worker-B-discovery@1.service

php-worker-A@1.service

ea/php-worker:latest

1php-worker-A-discovery

@1.service

php-fpm-discovery@1.service

Redi

sEx

tern

al d

ata

serv

ices

MyS

QLM

emca

ched

Load

Bal

ance

r

Configuration, orchestration and deployment

host

service

service

service

service

host

service

service

service

service

host

service

service

service

service

host

service

service

service

service

Configure topology, maintain state.

Build images with code, deploy to or scale target environments.

Run containers in staging, load test, and production. Capture

metrics.

• Centralised Logging

• Monitoring - processes inside containers

Future Work

— Scott Porter

“Using IBM and Docker container solutions, we have been able to build a gaming platform that is massively scalable, highly performant and meets our deployment velocity needs.”

33

Thank you!@cloudtrollhately@us.ibm.com

Andrew HatelyScott Porter@sl_portersporter@ea.com