LagLag, The Barrier to Innovation in Online , The Barrier to Innovation in Online GamingGamingHarlan “Tytus” BeverlyHarlan “Tytus” BeverlyCo-Founder of Bigfoot NetworksCo-Founder of Bigfoot Networks

About HarlanAbout Harlan

Harlan Tytus BeverlyFounder of Bigfoot NetworksHardcore GamerProgrammer (Networking)Chip Designer/Architect

Favorite Games: Lord of the Rings Online, Duels.com, Plants vs. Zombies, CS: S, BF Heroes

Beverly

• Founded Bigfoot Networks & Designed “Killer NIC.” Raised over $13MM in funding.

• Filed over 30 patents, including 16 about Online Game Design.

• Designed “World's first” 10-Gig Ethernet Adapter at Intel.

• Architected & helped write Bigfoot's LLR2 Network Protocol (x-platform networking SDK)

• Wrote several games including “Online” Othello for Linux

Lag is Waiting.Lag is Waiting.

Ever been stuck in a traffic jam?

Lag is defined as the feeling of frustration while waiting for something online.

“Over 40% of gamers have quit an MMO because of Lag!” -2005 MW Research

If 40% of WoW users continued their subscription an extra 12 months that would be $700,000,000 more revenue.

Lag Example VideosLag Example Videos

http://www.youtube.com/watch?v=rgsxZbv0rrM

300ms+ = “Game Freezing”

<16ms = “Lag-Free”

33-100ms = “Sluggish Responses, Stuttery”

Humans can easily perceive less than 16ms of delay. (60fps=60hz=16ms)

100-200ms = “Rubber-banding, etc.”

The 3 Causes of LagThe 3 Causes of Lag

P

AG

E

• 35%* Game Servers (Overloaded, Costly)• 35%* Game Clients (Un-trusted, Multitasking)• 30%* The Network (Latency, Bandwidth)

*Source: 10,000 LagMeter Submissions

Duration of Percieved Lag

0

50

100

150

200

250

300

350

400

450

Broadband Normal Packet Loss Event Client Slow Event Server Slow Event

La

ten

cy

(m

s)

m

s

Server/Peer Delay (ms)

Packet Loss Delay (ms)

Client Delay (ms)

ISP Delay (ms)

The 3 Causes of Lag create Perceivable Lag Events.

Client Network Server

• Bursty Graphics (CPU/GPU Interaction)

• Bursty Application (CPU/Memory)

• Hard-Drive accesses• Memory

Management/Swapping• Background apps/tasks

• Client Bandwidth (Voice/etc.)• Long Routes (hops)• High Latency (variety)• Packet Loss• Faulty Equipment• ISP Congestion• Backbone congestion• Bursty datagrams

• Per-User Calculations• Per-Object Scoping + Message

Duplication (Exploding/Output Processing)

• Validation of User Actions (Input processing)

• Application: simulation, DB, etc• Server Network Bandwidth• Backend Interaction (Database)• Congestion Recovery (bursts)

35% (+/-15%) 30% (+/-10%) 35% (+/-25%)

LagMeter • LagMeter: Free “viral” tool to measure Lags-per-Minute (end-to-end responsiveness)

• 2007 Study: 10,000 Unique LagMeter Data Submissions

Latency vs. ThroughputLatency vs. Throughput

Ted Stevens, former Alaskan Senator,

referred to the Internet as "a series of tubes"

Latency vs. Throughput Latency vs. Throughput (Seriously)(Seriously)

• Pipes are a ‘better analogy’ than tubes.

• Pipes have “Diameter” and carry water from A to B at a given “rate”.

• A 10-Gig Internet Pipe carries 10-Gb per second of data.

• A DSL Pipe carries 2Mbps per second of data.

• Which has “lower latency?”

P

AG

E

Latency vs. Throughput Latency vs. Throughput (Seriously) Answer(Seriously) Answer

• Which is lower latency?• DEPENDS on the latency &

packet size (e.g. flow rate of pipe, not diameter)!

• What if the 10Gig only updates once per second??? (“rate”)?

• Latency = 1000ms!!!!• What if the DSL updates at

2,000,000/second?• Latency = 0.5ns

P

AG

E

.

Its like a Train w/TONS of cargo, vs a Plane

with a smaller cargo…

Latency in GamesLatency in Games• For Games, Latency *USUALLY* Matters MUCH more than throughput.

• Most games use LESS than 64kbps of throughput.

• HOWEVER, games are interactive, so latency is very important, since you want to react ASAP!

• FPS games can’t usually tolerate more than 100ms.

• MMO games can’t usually tolerate more than 500ms.

• Exception to the rule: Bandwidth Waste:

• Constrained bandwidth can CAUSE backup , buffer over-run, & packet-loss…

• Leading to Excessive Latencies due to retransmission & deep packet queues.

P

AG

E

ICMP vs. In-Game PingICMP vs. In-Game Ping• ICMP: Internet Control Message Protocol

• Part of “Ping” and “Tracert” commands.

• ONLY uses “IP” and DOES NOT use TCP or UDP.

• Does a fine job of measuring the pipes (but not the fittings).

• In-Game Ping:

• Measures Both Pipes & the fittings (Client, Server, Routers)• Windows Network Stack• ‘Network & Graphics’ interaction on Client.• Server’s ability to process and respond.

• Uses UDP or TCP as well as IP

• Does a better job of measuring responsiveness in the game. P

AG

E

Fixing Lag...Fixing Lag...• http://www.endlagnow.org/eln/TakeAction_TopTips.aspx

What Gamers Do:1. Improve their ISP

• (FiOS = more bandwidth AND lower latency)

2. Baby, Baby, Upgrade!!!!!

• (get faster hardware!)

3. Find better servers (closer, faster, less load)

4. Quit... and play a different game!!! EEEEK!!!

Game Designers:

Solve Lag by design! (See Next Section)

P

AG

E

What can Developers Do?• Fix the Internet? No.

• Ensure Gamers have Decent systems? No.

• Instead, we have to work with what we can control!– Game Design (See Next Section)– Net Code & Protocol (See Part 3)– Proper Server Infrastructure– Scalable Graphics Systems/Low System

Requirements

P

AG

E

P

AG

E

Ultima Online: Localization Problem

•Game Type: MMOG

•The Problem: One of the VERY first to have 'localization' or 'crowding' problem. (gamers grouping at the bank).

Causes server and/or bandwidth to get overloaded…

Ultima Online: Localization Solution

• Game Design Solutions for “Localization”/”Crowding” Problem:– Increased locations for most common grouping

issues (e.g. more Dragons, more Banks, etc.)– Sharding: create more shards and therefor, fewer

users will be ‘able’ to crowd, and encourage people to use those other shards.

– Instancing: where crowding might happen, simply make it an instance and limit the instance to a fixed size. (see Guild Wars)

– (NEW technical solution) Load Balancing: non-location based servers & networking protocols.

P

AG

E

P

AG

E

World of Warcraft: The “Too Many Players” problem

• Game Type: MMOG• Problem(s):

•Intense 3D Graphics Could Slow Systems

•Lots and Lots of Users expecting seamless experience regardless of Load.

P

AG

E

World of Warcraft: The “Too Many Players” Solution(s)

•Beautiful LAST-GEN 3D Graphics Made WoW have very low system requirements.•http://www.gamasutra.com/view/feature/3403/building_scalable_3d_games_for_the_.php

•Lots and Lots of Users was solved by “BRUTE FORCE”... by building LOTS AND LOTS OF SERVERS... then Sharding (not ideal).

•WoW also tolerates INCREDIBLE 1000ms+ latencies by “hiding” Lag extremely effectively... (see later slide)

World of Warcraft PC System RequirementsOS: Windows XP / Windows Vista (with latest Service Packs)

Processor Minimum: Intel Pentium 4 1.3 GHz or AMD Athlon XP 1500+Memory Minimum: 512 MB RAM (1GB for Vista users)Video Minimum: 3D graphics processor with Hardware Transform and Lighting with 32 MB VRAM Such as an ATI Radeon 7200 or NVIDIA GeForce 2 class card or better

Sound: DirectX-compatible sound card or motherboard sound capability

Guild Wars: Free MMO? “How Many Players?” Problem

• Game Type: MMO• Problem:

– Unknown # of Players (and no recurring revenue).

– Expectation of reasonable performance.

PA

GE

WoW & Guild Wars: Solving the “SOOO Many” Problem.

• WoW: Hide the Latency! Animations, Can or Can't Move While Attacking, “Recharge” Rates.– Allows ‘variable’ latencies & heavily loaded servers

to tolerate (hide) up to 500ms+ of latency!

• Guild Wars: Instancing!– Instancing is a form of load balancing.– Guild Wars ‘forces’ instancing (even in public areas),

with “n” copies of the world.

• Detecting System Capabilities:– “Hard” minimum with default set to minimum.– Pre-run Profiling– Profiling at Run-Time

P

AG

E

P

AG

E

CS & CS: Source: “Twitch” Internet Gaming• Genre: FPS “Twitch” • Problem: “Internet” Latencies VARY wildly between players!• Game Design Solution(s):

•Player Hosted Servers

•Internet Server Browser, sort by latency!

•Moderated Rules “kicking” players above certain Ping (Latency)

(note) Also innovated using ‘Delta’ and Latency Compensation System (see later)

P

AG

E

Battlefield 2: “Twitch” Internet & Persistence!

•Genre: “Twitch” FPS (w/Persistence)

•Problem: How to have persistence of character, between servers w/low Lag.

•Solution: Login, loads character data (persistence), THEN server browser like normal “Twitch”.

•Also, EA hosts many of its own hosted ‘twitch’ servers.

COH/COV – an Action MMO?• Genre: MMO/Action

• Problem: Action Gameplay Mechanics in MMO Worlds = Lots of Servers?

• Game Design Solution:– Approximation: Let the

user ‘think’ he “aimed” that gun...

– Back-end is highly latency tolerant

– Calculations are stats & vector based (not just vector based.)

P

AG

E

WarRock – No Servers? Free-to-Play FPS?

• Genre: “Free FPS”

• Problem: If it is free, who pays for servers?

• Solution: P2P Game Design, Lobbies do Matchmaking.

• (sidenote: Cheating is a HUGE problem in WarRock)

P

AG

E

.

Duels (Challenge Games) – Fight Other Players…asynchronously?

• Genre: Web Based Strategy & Collectible Persistent Game

• Problem: Challenging Players that aren’t online?

• Solution: Asynchronous Challenge System (Message Center)

– Player-to-Player Latency irrelevant.

– ‘Slow’ Web Content, just fine since Asynchronous.

P

AG

E

.

Spore – User Content for Single Player Online?

• Genre: Strategy Game, Massively Single Player Online

• Problem: Players at different ‘stages’ of game interacting. Bragging/Showing your User Generated Content (UGC) big part of fun.

• Solution: Loads Dynamic UGC at startup (if available), and auto-uploads UGC to Sporopedia (Spore Wiki) for bragging rights.

PA

GE

Trackmania – International Competition Online?

• Genre: Online Racing Simulation

• Problem: International Competition in “Twitch” Racing Game?

• Solution: Cars do NOT collide! No/Limited User-User Interaction. Client based Simulation.

P

AG

E

P

AG

E

Yahoo Internet Poker – Users with Lost Connections & Varied Latencies

•Genre: Casual Card Game

•Problem: Detecting “Dropped Connections” but Keeping game fast enough to be fun.

•Solution: 5-10 second timeout for players to respond… allows for SLOW latencies, drop & reconnect in most cases, but also keeps game moving.

Summary: Fight Lag w/Game Design

1. Hide Lag & Latency w/Animation

2. Restrict # of Players Interacting

3. Minimize System Requirements

4. Asynchronous Gameplay

5. Spread out the Fun & Redundant Resources

6. Instancing/Sharding

7. Limit Player-Player Interactions (Single-Player Online)

8. Use Lots of Servers

9. Use Peer-to-Peer (Carefully)

P

AG

E

Delta Systems *Source: Book of Hook

• With Quake3, John Carmack dropped the notion of a reliable packet altogether … the server sends sequenced game state updates that are delta compressed from the last acknowledged game state the client received.

• The client's receive logic boils down to:

P

AG

E

if ( newState.sequence < lastState.sequence )

{ //discard packet }

else if ( newState.sequence > lastState.sequence )

{ lastState = deltaUncompress( lastState, newState ); ackServer( lastState.sequence ); }

Latency Compensation *Source: Valve Software Wiki

• Basic Idea: Use Latency Information about each Client & Position & Velocity to compensate for Lag & Latency

P

AG

E

Example showing “Hit

Box” of 200ms Lagged

Shooter.

Latency Compensation *Source: Valve Software Wiki

1. The server estimates at what time the command was created

• Command Execution Time = Current Server Time - Packet Round Trip-Time - Client View Interpolation

• Note: Interpolation is Defined Next…(think view latency caused by 'smoothing')

2. Then the server moves all other players back to where they were at the command execution time.

3. The user command is executed and the hit is detected correctly.

PA

GE

Interpolation*Sources: Valve & OpenTNL.org

• Basics: Interpolation is used to smoothly move an object from where the client thinks it is to where the server declares it to be over some short period of time.

• Uses history of states to smooth out the Animations on the Client…

• E.g. without it, EVERYTHING IS JERKY!!! (LAGGY!)

P

AG

E

Interpolation Advanced*Sources: Valve & OpenTNL.org

• Interpolation exacerbates the time difference between the client and the server, because the client is spending even more time than the one-way message time to move the object from its current position to the known server position.

• E.g. Interpolation Delay adds to Total Latency.

• Extrapolation – Guessing of new state by velocity (for REALLY bad Latency)

• Client-Side Prediction – Better guessing of OWN state, based on ow user input. (e.g. doesn’t wait for server ACK)

P

AG

E

Dead Reckoning *Source: Gamasutra.com

• Interpolation, Extrapolation, Client-Prediction, Delta System ALL TOGETHER… within ‘limits’ or Threshold of ‘rightness’. Broken Limits = New ‘Full-State’ Message..

P

AG

E

P

AG

E

Net Code Tools & Systems•FPS Style

– Licensed Engine (UT3, Gamebryo, Source)

– Free/Semi-Free Engine (MS XNA, Torque, others)

– Truly Free (and very good) Net Code (Rakknet, HawkNL)

•MMO Style– Engine: (Emergent, Hero,

Multiverse, Sun Darkstar)

•RTS or Casual Style– MS .net, SmartFox, Unity, XNA

• GameDev.net and Gamasutra.com are great resources.

• Also see http://developer.valvesoftware.com/wiki

Summary: Fight Lag with Better Net Code

• Use Delta Systems• Use Latency Compensation• Use Interpolation• Use Dead Reckoning• Use Network Libraries

P

AG

E