Video Games “Level Up” NetworkingVideo Games “Level Up” NetworkingBy: Harlan Tytus BeverlyBy: Harlan Tytus BeverlyCo-Founder of Bigfoot NetworksCo-Founder of Bigfoot Networks

Networking and Computing

Requirements for Video Games: Trends &

Solutions.

About HarlanAbout Harlan

Harlan Tytus BeverlyFounder of Bigfoot NetworksIEEE Member since 1995Hardcore GamerProgrammer (Networking)Chip Designer/Architect

Favorite Games: Lord of the Rings Online, Duels.com, Plants vs. Zombies, CS: S, BF Heroes

Beverly

• Founded Bigfoot Networks & Designed “Killer NIC.” Raised over $13MM in funding.

• Filed over 30 patents, including 16 about Online Game Design.

• Designed “World's first” 10-Gig Ethernet Adapter at Intel.

• Architected & helped write Bigfoot's LLR2 Network Protocol (x-platform networking SDK)

• Wrote several games including “Online” Othello for Linux

“It’s boom time for an industry now even bigger than Hollywood” – San Francisco Chronicle, 2004

Gaming is Bigger than Hollywood

Gaming PCs, $5B

PC SW + Sub, $6B

Midware/Hosting, $2B

Gaming Accessory, $10B

Discrete Graphics, $21B

Mobile $6B

Console HW, $12B

Console SW, $11B

Gaming Market: $72B+ in 2008

Source: DFC Intelligence + Others

Online Gaming ExplosionTotal MMOG Active Subscriptions

R2 = 0.9791

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

1999 2000 2001 2002 2003 2004 2005 2006

0%

20%

40%

60%

80%

100%

2002 2003 2004 2005 2006

Online Games (% Total)

•2.7M Gaming PCs/Yr• 39% Y/Y Growth

•Big OEMs Entering Market: • Dell, HP, Acer, Lenovo

What is fueling gaming?Answer: Technology & Culture.

Technology: Leap-frogs in technology are leading to ever-more immersive (and addictive) games. Graphics are leading the charge, networking right behind driven by broadband penetration.

Culture: Gamers are coming of age. Nintendo generation are in their 30s… spending power has shifted to gamers… so has business.

An Industry of LeapfrogsPC Shooters have advanced state of the art

Graphics for 25 years.

Online games have advanced networking for 15 years.

Holy Grail of Online GamesThe Holy Grail of online games is defined as:

Unlimited number of players (infinitely scalable).

With real-time & rich interaction (twitch, voice, video).

And zero Lag.

Lag is Waiting.Lag is Waiting.

Ever been stuck in a traffic jam?

Lag is defined as the feeling of frustration while waiting for something online.

“Over 40% of gamers have quit an MMO because of Lag!” -2005 MW Research

If 40% of WoW users continued their subscription an extra 12 months that would be $700,000,000 more revenue.

Lag Example VideoLag Example Video

http://www.youtube.com/watch?v=rgsxZbv0rrM

The 3 Causes of LagThe 3 Causes of Lag

P

AG

E

• 35%* Game Servers (Overloaded, Costly)• 35%* Game Clients (Un-trusted, Multitasking)• 30%* The Network (Latency, Bandwidth)

*Source: 10,000 LagMeter Submissions

Client Network Server

• Bursty Graphics (CPU/GPU Interaction)

• Bursty Application (CPU/Memory)

• Hard-Drive accesses• Memory

Management/Swapping• Background apps/tasks

• Client Bandwidth (Voice/etc.)• Long Routes (hops)• High Latency (variety)• Packet Loss• Faulty Equipment• ISP Congestion• Backbone congestion• Bursty datagrams

• Per-User Calculations• Per-Object Scoping + Message

Duplication (Exploding/Output Processing)

• Validation of User Actions (Input processing)

• Application: simulation, DB, etc• Server Network Bandwidth• Backend Interaction (Database)• Congestion Recovery (bursts)

35% (+/-15%) 30% (+/-10%) 35% (+/-25%)

LagMeter • LagMeter: Free “viral” tool to measure Lags-per-Minute (end-to-end responsiveness)

• 2007 Study: 10,000 Unique LagMeter Data Submissions

Latency vs. Throughput Latency vs. Throughput (Seriously)(Seriously)

• Pipes are a ‘better analogy’ than tubes.

• Pipes have “Diameter” and carry water from A to B at a given “rate”.

• A 10-Gig Internet Pipe carries 10-Gb per second of data.

• A DSL Pipe carries 2Mbps per second of data.

• Which has “lower latency?”

P

AG

E

Latency vs. Throughput Latency vs. Throughput (Seriously) Answer(Seriously) Answer

• Which is lower latency?• DEPENDS on the latency &

packet size (e.g. flow rate of pipe, not diameter)!

• What if the 10Gig only updates once per second??? (“rate”)?

• Latency = 1000ms!!!!• What if the DSL updates at

2,000,000/second?• Latency = 0.5ns

P

AG

E

.

Its like a Train w/TONS of cargo, vs a Plane

with a smaller cargo…

Latency in GamesLatency in Games• For Games, Latency *USUALLY* Matters MUCH more than throughput.

• Most games use LESS than 64kbps of throughput.

• HOWEVER, games are interactive, so latency is very important, since you want to react ASAP!

• FPS games can’t usually tolerate more than 100ms.

• MMO games can’t usually tolerate more than 500ms.

• Exception to the rule: Bandwidth Waste:

• Constrained bandwidth can CAUSE backup buffer over-run, & packet-loss…

• Leading to Excessive Latencies due to retransmission & deep packet queues.

P

AG

E

The Fight = InnovationThe Fight against Lag has lead to dozens of

innovations in networking.Dealing with Dial-up bandwidth.

Applicable today with cellular bandwidth.Dealing with Scalability.

Today called “the cloud”.Dealing with crowding.

Applicable today with social networks & still a problem for games.

Dealing with “time-synchronization”Applicable today for dozens of industries.

Dial-up BandwidthLAN gaming was born first. (Doom)Dealing with Dial-up was a major

challenge for games, blocking the road to the holy grail by limiting the number of users and making a ‘simulation’ required.

Solving Dial-up meant having “imperfect” simulation on the client… only tiny state updates could be sent.

John Carmack’s solution has become an industry norm:Delta systems.

Ultima Online introduced a new concept:Interest systems.

Delta Systems *Source: Book of Hook

• With Quake3, John Carmack dropped the notion of a reliable packet altogether … the server sends sequenced game state updates that are delta compressed from the last acknowledged game state the client received.

• The client's receive logic boils down to:

P

AG

E

if ( newState.sequence < lastState.sequence )

{ //discard packet }

else if ( newState.sequence > lastState.sequence )

{ lastState = deltaUncompress( lastState, newState ); ackServer( lastState.sequence ); }

Interest SystemsAn interest system manages WHAT relevant

state data you are sent.This reduces bandwidth needs.And processing time by Al.

Things “close” to Al… sent.Things “not close” to Al… not sent.

Al

Joe

Al

Bob

Joe

“Bob + Joe interested in Al.”

“Bob moved.”

ScalabilityMMOG – Massively Multiplayer Online GameBut how massive?

Build too few: epic disaster as people “wait in line” to play.

Build too many: way too expensive capital overhead/user.

Gaming Solutions:Latency HidingShardingVirtualizationTask Based Clusters

Latency & Sharding: Solving the Scalability Problem.

• WoW: Hide the Latency! Animations, Can or Can't Move While Attacking, “Recharge” Rates.– Allows ‘variable’ latencies & heavily loaded servers

to tolerate (hide) up to 500ms+ of latency!

• Ultima Online, WoW, nearly every MMO ever….

• Sharding. Create multiple copies of the “game world” and get people to try to average themselves out with wait queues.

P

AG

E

Modern Approaches to ScalabilityRun it in the cloud! *or virtualization*

Can improve bandwidth & server processing problems, but…

Still problems with the persistent database.Many game companies looking at 10-Gig & infiniband for

solutions to database & server interconnect.

Task-based game server design.Don’t run servers based on users or location or AI, etc.Run them based on a distributed task system

(clustering).Still problems with the persistent database &

interconnect.10-Gig/Iband.

Complexity problem exponential.Use Instancing (see next section)

P

AG

E

Ultima Online: Localization Problem (Crowding)

• One of the VERY first to have 'localization' or 'crowding' problem. (gamers grouping at the bank).

Causes server processing and bandwidth get overloaded…

Ultima Online: Localization Solution

• Game Design Solutions for “Localization”/”Crowding” Problem:– Increased locations for most common grouping

issues (e.g. more Dragons, more Banks, etc.)– Sharding: create more shards and therefor, fewer

users will be ‘able’ to crowd, and encourage people to use those other shards.

– Instancing: where crowding might happen, simply make it an instance and limit the instance to a fixed size. (see Guild Wars)

P

AG

E

Time SynchronizationSynchronizing the “views” of millions of

clients is a daunting challenge.Similar to the challenge of ‘data sync’

between two real-time systems.

Games have evolved very unique solutions:Server AuthorityLatency CompensationInterpolationDead Reckoning

Server AuthorityVery early on it became obvious… the

simplest way to maintain data synchronization was to keep it all in one place.

One authority: the server.The problem: latency varies by user. (see

Latency Compensation).Other problems: (see scalability)

P

AG

E

CS & CS: Source: “Twitch” Internet Gaming• “Internet” Latencies VARY wildly between players (Some on dial-up, some on LAN, some on Broadband)!• Game Design Solution(s):

•Player Hosted Servers

•Internet Server Browser, sort by latency!

•Moderated Rules “kicking” players above certain Ping (Latency)

Latency Compensation *Source: Valve Software Wiki

• Basic Idea: Use Latency Information about each Client & Position & Velocity to compensate for Lag & Latency

P

AG

E

Example showing “Hit

Box” of 200ms Lagged

Shooter.

Latency Compensation *Source: Valve Software Wiki

1. The server estimates at what time the command was created

• Command Execution Time = Current Server Time - Packet Round Trip-Time - Client View Interpolation

• Note: Interpolation is Defined Next…(think view latency caused by 'smoothing')

2. Then the server moves all other players back to where they were at the command execution time.

3. The user command is executed and the hit is detected correctly.

PA

GE

Interpolation*Sources: Valve & OpenTNL.org

• Basics: Interpolation is used to smoothly move an object from where the client thinks it is to where the server declares it to be over some short period of time.

• Uses history of states to smooth out the Animations on the Client…

• E.g. without it, EVERYTHING IS JERKY!!! (LAGGY!)

P

AG

E

Interpolation Advanced*Sources: Valve & OpenTNL.org

• Interpolation exacerbates the time difference between the client and the server, because the client is spending even more time than the one-way message time to move the object from its current position to the known server position.

• E.g. Interpolation Delay adds to Total Latency.

• Extrapolation – Guessing of new state by velocity (for REALLY bad Latency)

• Client-Side Prediction – Better guessing of OWN state, based on ow user input. (e.g. doesn’t wait for server ACK)

P

AG

E

Dead Reckoning *Source: Gamasutra.com

• Interpolation, Extrapolation, Client-Prediction, Delta System ALL TOGETHER… within ‘limits’ or Threshold of ‘rightness’. Broken Limits = New ‘Full-State’ Message..

P

AG

E

New Approaches to “the holy grail” of Lag-Free gaming.Peer-to-Peer (security & time-synch

problems abound).Asynchronous games.Massively single player.

WarRock – No Servers? Free-to-Play FPS?

• Genre: “Free FPS”

• Problem: If it is free, who pays for servers?

• Solution: P2P Game Design, Lobbies do Matchmaking.

• (sidenote: Cheating is a HUGE problem in WarRock)

P

AG

E

.

Duels (Challenge Games) – Fight Other Players…asynchronously?

• Genre: Web Based Strategy & Collectible Persistent Game

• Problem: Challenging Players that aren’t online?

• Solution: Asynchronous Challenge System (Message Center)

– Player-to-Player Latency irrelevant.

– ‘Slow’ Web Content, just fine since Asynchronous.

P

AG

E

.

Spore – User Content for Single Player Online?

• Genre: Strategy Game, Massively Single Player Online

• Problem: Players at different ‘stages’ of game interacting. Bragging/Showing your User Generated Content (UGC) big part of fun.

• Solution: Loads Dynamic UGC at startup (if available), and auto-uploads UGC to Sporopedia (Spore Wiki) for bragging rights.

PA

GE

Trackmania – International Competition Online?

• Genre: Online Racing Simulation

• Massively Single-player

• Problem: International Competition in “Twitch” Racing Game?

• Solution: Cars do NOT collide! No/Limited User-User Interaction. Client based Simulation.

P

AG

E

Game Software DevelopmentNetworking developers are RARE and in

great need in the gaming industry.Skills coding in Sockets and knowledge of

networking & network infrastructure are in high demand.

Tools of the trade?Network libraries Server software architectureNetwork hardware & configuration

P

AG

E

Network Libraries•FPS Style

– Licensed Engine (UT3, Gamebryo, Source)

– Free/Semi-Free Engine (MS XNA, Torque, others)

– Truly Free (and very good) Net Code (Rakknet, HawkNL)

•MMO Style– Engine: (Emergent, Hero,

Multiverse, Sun Darkstar)

•RTS or Casual Style– MS .net, SmartFox, Unity, XNA

• GameDev.net and Gamasutra.com are great resources.

• Also see http://developer.valvesoftware.com/wiki

Testing an MMOOnline game testing is 2x

as hard as offline.What is different?

Testing for different latencies & packet loss.

Testing different bandwidths.

Tools?Monkey clients (fake

clients doing automated stuff).

Massive open and closed alpha & beta programs.

Post-launch patches & bug-fixes GALORE.

BackupEmail me or connect on Linked-in.

Harlan T. Beverlybeverly@ieee.org