Tapping the Collective Creativityof Your Team

Marcin ChadyTechnical Director, Radical Entertainment(Activision Blizzard)

Acknowledgment – George Mawle

Paradox

●We expect team members to be “passionate” about games● Not just animation or physics programming or lighting

●We give them highly specialised roles● With little influence on other areas

●Now, what do we actually want them to do with this passion?● When creativity and passion breach the role boundaries● “They are smart, passionate and creative. Let them figure it out.”

Economics of Making a Pitch

The value of a new idea(as subjectively perceived)

vs

Cost of pitching it(preparation, figuring out who to talk to, getting their attention)

vs

Risk of it being dismissed(because it’s lame, impractical, or because someone already thought of it)

Disenfranchisement

●If we don’t give them enough to make an informed calculation, e.g.

● Who to talk to?

● Has this been already discussed?

●They exaggerate the cost or the risk*

●The calculation comes out negative

● “Not worth making fuss over such a small detail”

● “It’s not my job to worry about this, after all”

* That’s human nature: see e.g. Stephen Pinker, “How the Mind Works”

Should We Care?

Design by Committee

• Listen to the team• Distraction• Lateral thinking• Might get the lowest common

denominator

Ivory Tower Syndrome

• Don’t listen to the team• Stay focused• Tunnel vision• Missed opportunities• Might get a polished turd

?

The Goal

●Retain independent creative decision making●Help team members make informed assessments of new ideas●A way to give and receive feedback that is:

- Easy- Accessible- Public- Unintimidating

- Unimposing- Impersonal- Structured- Constructive

Other Goals

●A home for suggestions- The awkward, unwanted bastard sibling of bugs- Rarely tracked in a bug database- Usually forgotten, unless they have a champion

●Engage QA more in game-making process- Quality is more than being bug-free- Testers are hypersensitive to issues

●An indicator of game quality

Prior Art

●Encouragement

●Q&A sessions

●Suggestion boxes

●Surveys

●All of them are manual:

● Cost time

● The first to go when deadlines loom

General Idea

• Hierarchical feature break-down

• Comments

• Ratings

• Subscriptions

• Database-backed

– Ability to update one’s comment or rating

• Agree/disagree option

Game (86)

•Art (88)•World (90)•Props (86)•Effects (88)

•Audio (85)•Music (90)•Dialogue (81)•Effects (84)

•Gameplay (84)•Combat (86)•Powers (90)•Blade (90)•Claws (90)

•Enemies (82)•Hunter (80)•Hydra (84)

•Missions (83)•Story (82)•Intro (76)•A1 (88)

•Open World (84)

Implementation

●About 5 days’ worth work●Embedded in the build tool

• As opposed to a separate tool or a website●Feature breakdown and subscriptions stored in a simple YAML file

• Feature breakdown owned by QA (a major headache solved)• People can manage their own subscriptions• Version controlled

●Control spam• One score/comment per feature per person• One score, multiple comments per feature per person

Scoring Scheme

• 0 to 100 percent or 1 to 5 stars?

• Is mediocre 50 or 70?

• No clear answer

• Our choice: 0 – 100 scale with 5 point increments

0 - C

an’t

be

any

worse

5 10 15 20 25 30 35 40 45

50 -

Meh

55 60 65 70 75 80 85 90 95

100

– Can

’t

be a

ny

bette

r

Policy

●Build trust that feedback will be read and considered- Producer role critical

●Feature owners free to decide whether to act on feedback- But prodded by producers to at least respond to new concerns

Roll-Out

●Initial resistance- “Unnecessary distraction”- “Leave it to professionals”- “Too technocratic – what we need is a change in attitudes”- Concerns about the name: BetterCritic → Feedback Tool

●Need buy-in and commitment from the top- The ratings feature!

●Devolve ownership quickly- QA engagement- Users to control subscriptions

Results

●Between Oct 2010 and  Oct 2011 we collected:

- 4000 operations (adding or modifying feedback or agreeing/disagreeing with it)

- 10 per day, 30 per person- 3275 comments, 9 per day, 26 per person- 2561 scores, 7 per day, 20 per person- All comments resulted in an email sent to 10+ people- Most followed up by at least one email replied to all

Contributions per User

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 1210

50

100

150

200

250

300

350

400

450

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Individual

Contrib

ution

s

Contributions per FeatureIntro 1 Story L1 Intro 2 Targeting Story C1 Story C2 Story L2 Story A2 Story B2 Collateral Damage Gliding Story F1 Feedback ToolStory C3 Story A1 Standard (gz_re_01, yz_re_01) Camera Game ideas Story A3 Stockpile Story D3 Story F2 Story G2 Story M1 Story D2 DisguiseStory E1 Regular Story B3 Pulse Story L3 Story M2 Chopper Race Story H1 Consume Lair (yz_it_02) Story G1 Story E2 HellerDisguise Story E3 Map Story F3 Multiple Locations (yz_sc_02) Recovery Race Rampage Standard (yz_fi_01) Hydra Story D1 Effects Brawler Black BoxChopper (gz_vc_01) Skeet Shooting Chain of Command Mutations Story I3 Multiple Infected (yz_it_01) Unarmed Tendrils Other Final Boss Story H2 Thai StealthRadnet VS The Behemoth Behemoth Power Wheel Pedestrians Interface Blacknet Button Callouts Story I1 Motor Pool (gz_fi_01, yz_fi_02) Round Up (rz_sc_02) Collectibles Assault RifleChopper (gz_sc_01) Viral Sniffers Rain Maker (rz_re_01) Story G3 Power Switching gz_la_04 Standard (yz_sc_01) Blackwatch Gunship Heli Hammer Fist Shield Story H3 Evolved (gz_sc_02) Green ZoneGrenade Launcher Blackwatch Soldier Strike Team (gz_vc_02) Jumping Story Arcs Biobomb Super Soldier Objective Callouts Cars Generic Evolved Indian Italian Red ZoneEvolved (rz_vc_02) VS First Target World Props Claw Blackwatch Tank VS The Evolved Alert Level AI In-game Music SteamRollers VS The Lair Multiple Commanders (yz_vc_01) Blackwatch APC Hidden Scientist (rz_sc_01)Blackwatch Gunship Heli Weapons In-game SFX KFC Strike Team Story M3 Main Standard (gz_it_01) Mission Evolved Air Conditioning Flyer Facility in Trouble (gz_fi_02) Lair (yz_vc_02) ThrowingJapanese Gym Evolved (gz_re_02) Main Menu NIS Yellow Zone Attacking yz_cd_04 Kamikaze Persian rz_sp_01 Pillar Death Squads Blade Black BoxArmor Opening Video gz_sp_01 Pit General Infected 1 Missile Launcher gz_ra_01 Freefall TOW Missile Launcher Behemoth (rz_it_02) Achievement Ideas Heller Heavy Fire (gz_re_03, rz_re_02) Millitary Base Exteriorsgz_la_01 In-game Speech yz_cd_02 Crosshair Devastators Story I2 Pizza Infected 3 TOW Missile Launcher Whip Fist Chinese yz_la_01 TOW Missile Launcher GreekThrown Objects Framer Cams yz_cd_01 Blast Radius Blackwatch Transport Heli Close Up Vignette rz_rr_02 Ruins Controller Health Bar Radar Gattling Evader Gattling yz_cd_03 Grand SlamPack Leader Evolution Points gz_la_02 Lab Interiors rz_ra_01 Battle Royal APC rz_cd_01 Corkscrew Stopped by Evolved (rz_it_01) Apache Smartnodes Evolved (rz_fi_02) GenTek Scientist rz_cd_02 Disaster ZoneCamera Rotation Savory Island Sandwiches & Potato Salad yz_ra_01 City Salvage HUD yz_cr_02 Gorge Test Bunker Interiors Airbridge Screen rz_la_05 Front End Black Hawk gz_rr_01 The Final Shootout Facilities gz_vc_01gz_la_03 UAV yz_cr_01 Flyover FMV VS Vehicle Commander 3D Markers GenTek Facility Exteriors Minimap markers ITeam Mission Complete Screen General Mission Props Infected PedestriansMap Screen Blackwatch UAV Disguise Gattling yz_sp_01 Gauntlet Generic Meter Scientist Consume Player Death Screen Mission Complete Competitive Stats Lair New Tech Suggestions Infected Destoyer (rz_vc_01)Blood Tox (rz_fi_01) rz_rr_01 Survival of the Fittest Apache Loading Screen Code Editor gz_sp_02 Crossfire Mission Start Volumes Civilian Evolved (gz_it_02) Gun Turrets Blackwatch APC Weapons Open World Mission Complete DLC JuggernautTank Pustule Black Hawk Thermobaric rz_la_01 Lair Interiors Charts Challenges rz_la_02 Scripting Rendering rz_la_03 Tankrz_la_04 Blackwatch Helicopter Pilot Pause Menu Iconography Autobuilder APC rz_la_06 Video Minimap Mass Meter Blackwatch Officer General Thermobaric HellfireMuscle Mass SAW Crash Handler Audio yz_ra_02 Rooftop Recon Thermobaric yz_rr_01 Outbreak Leadership Animation Pure3D Viewer Mercer Infected Transport New Tool SuggestionsMilitary Soldier Training General

Goals Achieved

• Most of the team has used it• Lots of positive feedback on the tool itself

• Feature owners appreciated getting regular feedback• Some used it to discern patterns and trends

● E.g. polarised opinions often indicate an interesting gameplay mechanic

•Catalyst for brainstorming• Translates directly to the increased quality of the game

• Targeting• Gliding• Camera

• Story• Helicopter controls• Profanity

A Life of Its Own

●Ideas for the next game

●Ideas for tools and tech improvements

●Feedback on the studio, e.g.‒ Overtime food‒ Air conditioning

Other Benefits

●Water cooler effect●Catalyst effect●Critical mass effect●Calling out unsung heroes●Taking of team’s temperature●Repository of evidence

- “Is this an issue? It clearly is.”- “Does this mission suck? Apparently not.”

What Didn’t Work

●Reading feedback- UI left much to be desired- Feature owners often relied entirely on email notifications

●Many people didn’t like the scoring part- Onerous – “Why do I have to give a score?”- Unclear – “Not sure how to score”- Threatening – “Not fair that someone rated my mission 3/10”- Generally unreliable

In Defence of Mandatory Ratings

●Part of the package●Not meant to be absolute

● Is game score going up?

● What areas are dragging it down?

● Are our effort in area X paying off?

●Encourages people to give the feature full consideration

●Makes feedback more readable and easier to organise- E.g. when faced with 200 comments, start with the highest and lowest-rating ones

Future Work

●“Feed-forward” to close the loop, i.e.– to pre-empt suggestions which have been already dismissed– to pre-empt feedback which is already being addressed– can take form of a special comment from the feature owner

●Better UI●Data-mining tools

– Charts and reports– Search

● Binary ratings