A Game Usability StudyMore Actions than Inputs

By: Chris Roe & Don W. Lail, Jr.

Team Members & Roles

Design of Experiment – Don W. Lail, Jr. & Chris Roe

Data Collection – Don W. Lail, Jr. (Based on on-campus availability.)

Data Analysis – Don W. Lail, Jr. (Quantitative), Chris Roe (Qualitative)

Presentation & Review – Don W. Lail, Jr. & Chris Roe

Users/Contexts Evaluated

Out of 25 respondents to the original survey, 15 made appointments to set up a trial and actually play the game. Subjects were all given the same introduction and were allowed the same five minutes of free-play time before their ten-minute trial.

The context of the trial included an area that encompassed a village and hill that were both guarded by enemy forces. In order to complete the objectives, the subject had to move through the environment and deal with/hide from enemies.

Tasks Performed

Practice Time (Subjects were given 5 minutes to free-play in this environment. They were allowed to move through the environment in any way and experiment with the controls/actions of the game. Should they reach the end of the test area, the practice time would end.)

Diagnostic Objectives (achieved during practice time):1. Lie down and change camouflage.2. Equip the Solid Eye and change to NV and then back to normal.3. Change between Mk2 pistol and stun knife (and back).

Tasks Performed

Play-Test(During this time, subjects were asked to complete 9 objectives within 10 minutes. They were given the opportunity to see someone else play through once before doing it themselves.)

Test Objectives (achieved during the play-test):1. Shoot an enemy soldier at range.2. Take down an enemy soldier in close-quarters-combat (fist or knife).3. Purchase a special bullet for the Mk2 in the shop and shoot someone with it.4. Reach a difficult to get item by grabbing a ledge while rolling (two choices).5. Move a dead, sleeping, or knocked-out body to an inconspicuous location.6. Throw an empty magazine to make a guard look the other way.7. Sneak around/past an active enemy.8. Move to the next area without causing an alert.9. Make it to the end of the level or time limit without dying.

System Requirements

All subjects were required to use the same hardware/software to complete the trial. A Playstation 3 with a Dualshock 3 wireless controller were used in conjunction with a 50-inch HDTV which the subjects used to play the introduction to the second chapter of the game Metal Gear Solid 4: Guns of the Patriots.

Subjects were screened during the survey process to ensure that they were currently a Purdue Calumet student.

Problem

• There are more actions available in Metal Gear Solid than buttons/inputs available on the controller (input device).

• Metal Gear Solid 4 utilizes context and button/input combinations to provide 70+ actions on a 17-button controller.

• Is this effective?

• Can players be reasonably expected to adapt to and understand this solution?

• Does this solution apply well to the target audience?

• Who is the target audience?

Definitions

Genre: Metal Gear Solid 4 is commonly considered an action/adventure game.Target Market: Action Gamers

Adventure GamersShooter GamersRole-Playing Gamers

*Metal Gear Solid defines itself as “Stealth Action”, or more specifically “Tactical Espionage Action”.So, we are comparing usability between: (Source: MobyGames.com)

Genre: Action GamesPerspective: 1st-Person Perspective, 3rd-Person PerspectiveThemes: Stealth, Shooter, Spy/Espionage

So, First-and-Third-Person, Non-Shooter Action Games and First-Person, Shooter Action Games.Category 1: First/Third Person, Non-Shooter Action GamesCategory 2: First-Person, Shooter Action Games

MethodologyAll subjects must identify themselves by their experience with action and FPS games.

• Subjects affiliated strongly with FPS (more than two-thirds of Category 1, less than one third of Category 2) will be grouped into the FPS trials.

• Subjects affiliated strongly with Action (more than two-thirds of Category 2, less than one third of Category 1) will be grouped into the Action trials.

• Subjects that do not affiliate with either category (less than one third of both categories) will be grouped into the Inexperienced trials.

Subjects were surveyed before the trial to determine their past experiences.

The full trial consisted of: An outline of trial objectives for the subject’s reviewA live demonstration by another playerA five-minute free-play time before their ten-minute trial

Methodology

The definitions of action-game-player and FPS-player changed due to the fact that none of the 25 survey respondents or the 15 participants in the trial fit in the action or FPS categories as defined. When looking at the data in the review of the experiment, we looked at these factors as ratios/percentages instead of distinct categories.

Hypotheses

In order to prove the effectiveness of Metal Gear Solid 4’s solution to the action/input problem, we will make the following hypotheses:

• HO : Previous game experience does not affect performance on MGS4 trials.• HA1 : Previous game experience negatively affects performance on MGS4 trials.• HA2 : Previous game experience positively affects performance on MGS4 trials.

More specifically, we could define variables as:

• SP : Shooter Performance (Average number of objectives accomplished per FPS trial)• AP : Action Performance (Average number of objectives accomplished per action trial)• UP : Un-Experienced Performance (AVG # of objectives accomplished per other trial)

HypothesesUnfortunately, it is impossible to directly compare these values in our hypotheses. Therefore, it is necessary to compare these to a pre-established, arbitrary number. It would be preferred to use the same value across hypotheses in order to make comparisons easier in recommending further study, or deducing implications of the data.

• HO1 : UP ≤ 60% (Un-experienced players will accomplish less than 60% of the objectives.)

• HA1 : UP > 60% (It would be interesting to find that un-experienced players can accomplish over 60% of the objectives.)

• HO2 : SP ≥ 60% (FPS players will accomplish more than 60% of the objectives.)• HA2 : SP < 60% (It would be interesting to find that FPS players can accomplish

under 60% of the objectives.)

• HO3 : AP ≥ 60% (Action game players will accomplish more than 60% of the objectives.)

• HA3 : AP < 60% (It would be interesting to find that action game players can accomplish under 60% of the objectives.)

AffordanceThe look and placement afford behaviors other than those intended.

Does Metal Gear Solid 4 afford behaviors other than those intended?Absolutely.

This is ultimately the point of the game.

Let’s look at an example:

AffordanceThe look and placement afford behaviors other than those intended.

You are approaching a guard from behind. If he spots you, he will raise an alarm. He hasn’t noticed you yet, but as you get closer your footsteps have a higher

chance of giving you away. He stands about five feet in front of you and acts as if he is going to turn around. Which of these actions do you perform?

AffordanceThe look and placement afford behaviors other than those intended.

• Shooting• Punching• Accessing an item menu• Accessing a weapon menu• Calling for advice over a radio• Kneeling• Crawling• Climbing over/onto objects• Crawling into/through small spaces in first-person view• A "punch, punch, kick" style melee system• Grabbing a guard to throw them• Grabbing a guard from behind to choke them• Looking around in first person view• Shooting while kneeling, crawling, or running• Knocking on walls (used to just be punching toward a wall)• Leaning on walls (while standing or crouching)• Moving along walls (while standing or crouching)• Seeing around corners while pressed against a wall• Accessing elevators, computers, or other devices• Shooting while riding in a vehicle• Hanging over a ledge• Doing a chin-up while hanging on a ledge• Shooting in first person view• Leaning in first person view• Leaning out from a corner while pressed against a wall• Jumping out from a corner to shoot

• Executing a forward roll• Holding up a guard• Grabbing a guard from behind and to drag them• Grabbing a guard from the front• Grabbing a guard and disarming them• Grabbing a guard and using them as a shield• Grabbing a guard and interrogating them• Grabbing a guard and cutting their throat with a knife• Throwing a guard on the ground and holding them up as they lay down• Shooting while hanging from a ledge (in first or third person mode)• A system for selecting up to 8 weapons at a time to use in a sub-menu• A system for selecting up to 8 items at a time to use in a sub-menu• A camouflage system that requires the use of face paint and clothing to

blend in to the environment• Slow "stealth" walking and crawling• Using a knife• Moving while aiming in first-person view• Rolling in a drum/barrel• Tapping someone laying down on the back to retrieve items or wake

them up• Controlling a small robot that can move and attack• Crawling on your stomach or back (at multiple speeds)• Shooting while laying on your back• Stabbing an enemy with a sensor to track their movements• Walking while crouched

Stereotypes/ExpectanciesDoes not fit to what was expected. (Example: Proximity to similar functions.)

Players consistently found it surprising that they could not climb over waist-high obstacles.

Players had difficulty understanding that an additional button needed to be pressed to grab onto a ledge while falling.

Players had difficulty understanding why equipping an item required three separate actions. (Much confusion over which menu did what.)

Stimulus/ResponseWhat you do and what you get look/feel similar.

Everyone achieved the objective to shoot a soldier at range. The L1+R1 button combination felt appropriate enough for everyone to figure it out easily.

Because of the low difficulty level, a soldier would fall asleep after being hit by a sleeping dart at any spot on the body. Most subjects felt satisfied and that the

response was appropriate, but some high-level players found it odd.

All subjects were able to tell immediately when they had been spotted, though most of the time they didn’t seem to be sure why.

CognitionPlanning and decision-making under stress.

Subjects found the looming time limit to be difficult, and were consistently forgetting objectives altogether because of the pressure.

After attempting an objective once or twice, players would sometimes skip to the next objective on the list and never return (even if they had time left or had much

better opportunity than before).

Visual (Light/Receptors) Ability to perceive color, detail, distance.

Subjects were shown how to use a night-vision item to see in low-light areas. Several locations in the game were so dark that no detail could be seen. Players

often made mistakes in these locations because they couldn’t clearly tell the distance between the player character and an enemy, or the direction either was

facing.

A single color scheme was used for the whole environment, making everything appear to be a shade of blue/green. Some subjects had a hard time spotting enemies moving at a distance because they blended in to the background.

Auditory (Sound/Noise)Alarms, false alarms, occupational noise.

The alert noise did an excellent job of letting subjects know when they had been spotted.

False alarms: Subjects often heard enemies saying things like “enemy spotted” in reaction to non-player characters on the map. Sometimes subjects took this as a

sign that they had been spotted.

Gunshots from an ongoing battle between enemies and other non-player characters often caused inexperienced subjects to immediately take cover and

wait for the fighting to end, even though it did not have anything to do with them.

PerceptionTop-down processing vs. bottom-up processing. Depth cues.

Top-down processing – Subjects familiar with first-person shooter games felt accomplished and closer to completing objectives when they shot everyone in the

area. Only after doing so did they realize that some of the objectives could no longer be completed.

Bottom-up processing – It was easy for most subjects to see the relation between weapon and item menus because the item menu (on the left) was accessed with the left trigger, and the weapon menu (on the right) was accessed with the right

trigger. Once one was learned, both were known and remembered.

Search & DetectionPop-out effect. Gestalt organizing principles: proximity, closure, continuity,

symmetry, similarity.

Pop-out effect – Soldiers were sometimes difficult to differentiate from the background. Subjects who preferred shooting often left night-vision active to help

distinguish soldiers from the environment.

Symmetry – The weapon/item menus were easy to understand.

Proximity – The process of equipping an item was difficult because the buttons and menus were disconnected/discontinuous.

Similarity – The pause menu offered an item menu, weapon menu, and shop which all looked remarkably similar. This confused subjects to the point where

many gave up using them to achieve an objective. Also, enemy and friendly soldiers were nearly impossible for subjects to differentiate due to their similar

appearance.

MemoryStorage of information; short (30-90 seconds) and long (nearly permanent) term.

Many subjects forgot objectives along the way even after having reviewed them on paper and seen them performed. Some even forgot after having attempted an

objective and failing.

Many subjects forgot how to use the shop and weapon selection inside the pause menu, while they did not experience difficulty using the on-screen item/weapon

menus.

In the second half of the level, subjects could never remember the appropriate path to scale the hill at the end. Some subjects experimented with the same

incorrect route multiple times.

Decision MakingSelecting one choice from a number of choices involving some level of uncertainty.

Most subjects attempted the objectives in order.

Almost all subjects attempted to grab a ledge in the same location, even though there were two choices.

Subjects had the opportunity to select from four different special bullets to achieve the 3rd objective. The bullets accomplished generally the same thing,

though in different ways.

HeuristicsShortcuts that aren’t guaranteed to lead to the best answer, but are more efficient.

Subjects often chose to start shooting all the enemy soldiers they could find because they felt it would make their objectives easier (despite the fact that some

objectives required the soldiers to be alive).

None of the subjects attempted to use any of the other items or weapons provided.

Situation AwarenessSkilled behavior which can extract, integrate, assess, and act upon stimuli.

One piece of equipment—the Solid Eye—which was shown to subjects during their free-play time showed a motion-radar in the top corner of the screen which helps players identify the locations of nearby enemies. No one showed that they noticed

it at all.

During each trial in which the subject was spotted and caused an alert, there was an instance where an enemy that the subject saw before had now moved to a location that the subject did not expect. No one seemed to be keeping track of

enemy movements.

Many subjects did not notice when enemy soldiers woke each other up (after being incapacitated), even when there was dialogue heard and displayed on-

screen.

DataThe following is the most relevant data collected during our project.