The Science of Game-based Training Effectiveness 29 March 2012 Krista Langkamer Ratwani

© 2012 Aptima, Inc.

The Science of Game-based Training Effectiveness

29 March 2012

Krista Langkamer RatwaniKara L. Orvis

© 2012 Aptima, Inc. 2

“It is fair to say that we currently ask young infantry [personnel] to function reliably across more domains than we ever have in the past, and the defense community has responded by inundating our forces with new technology and training to meet this challenge. Although this focus stems from the admirable desire to ensure that our forces receive the latest and greatest training and technology prior to combat, it has resulted in a patchwork of training efforts and technologies that have been implemented without demonstrating substantial proof of effectiveness.” - Marine Corps Gazette (2012)

Background


Current State of Game-Based Training

Deployment of games making leaps and bounds– 2006: $125M spent on game-based training (Blunt, 2007)

Research making baby steps– Little evidence that games lead to learning (Blunt, 2007)

Research focused on whether serious games were motivating– Training effectiveness studies are slowly leaking in but literature

still disparate with mixed results (e.g. O’Neil and Perez, 2008) Studying game-based training “in the wild” is hard Results are published across many domains and sources

– There continue to be many research questions regarding computer games and instruction (Tobias et al, 2011)

Little understanding of “why” serious games work


Contributors to Training Effectiveness

Campbell & Kuncel, 2001; Colquitt, et al., 2000; Mathieu, et al., 1992; Salas & Cannon-Bowers, 2001

The Training Program

The Trainee

The Situational Context


The Training Program: Things to Consider

What game features are most useful for targeting specific learning objectives? (Wilson et al., 2009)

Feedback– How often and what type of feedback should be provided?

Scaffolding– How can scaffolding approaches be incorporated into game-

based training?

Learning Objectives… Should Drive…

Design Decisions!


The Training Program – Empirical Evidence Example

Participants: 157 students playing a game that manipulated the level of Game Fantasy (low vs. high fantasy conditions)

Fantasy significantly impacted Declarative Knowledge and Application

F(1, 64) = 4.58, p = .04

F(1, 65) = 6.51, p = .013

Orvis et al., 2009


The Trainee: Things to Consider

General training rule: Training should be difficult enough to increase motivation and performance, but not beyond the trainee’s capability– Attribute by treatment (ATI) interactions have been a focus within

the general training research (e.g., Gully et al., 2002) Examples of Trainee Variables to Consider

– Personality Prior research in military settings has shown a positive relationship between

personality variables and training effectiveness (e.g., Driskell et al., 1994) – Prior videogame experience

Research has shown that less than 50% of college age students play video games on a regular basis (e.g. Orvis, Orvis, Belanich, and Mullin, 2005)

Both general and genre specific game experience impacts how students react to the game (e.g. Orvis, et al., 2005)


The Trainee – Empirical Evidence Example

Participants: 139 students playing versions of the game G-DIS which manipulated task difficulty across 6 missions

1 2 3 4 5 61

2

3

4

5

6

7Adaptive Difficulty Condition

low opennesshigh openness

Mission

Perf

orm

ance

1 2 3 4 5 61234567

Static Difficulty Condition

low opennesshigh openness

MissionPe

rfor

man

ce

Bauer et al., in press

β14 = -.15, t(826) = -2.15, p = .032


The Situational Context: Things to Consider

Training environment impacts effectiveness of training program (e.g., the game)

Organizational analysis is a critical component to effective training – even for serious games– Organizational or unit commitment to the training– Resources (e.g., time)– Involvement of the leader– Whether units are training together– KSA transfer environment– Available pre-training


The Situational Context – Empirical Evidence Example

Participants: 165 Soldiers (representing 9 platoons) playing VBS2

Amount of Preparation prior to playing game and Amount of Leader Involvement during the game significantly impacted Unit Performance variables

F(1, 126) = 10.23, p < .01

2.53

3.54

3.19 3.30

3.90

Level of Leader Involvement During Training

Uni

t Pro

cess

F(2, 137) = 4.02, p < .05Ratwani et al., 2010


The Importance of Evaluation

Evaluation is needed to actually assess learning Common methods of assessing training effectiveness

(Marine Corps Gazette, 2012)– Performance on the battlefield– Subjective assessments by trainees

Best practices– Longitudinal– Pre- and post-evaluation (assess skill level before and after

training) – Comparisons to other training methods (e.g., live training;

classroom training)– Good measurement

Include measures of the 3 factors just discussed Identify and develop standards of performance


Continue to test the effectiveness of serious games

Understand how and why games lead to learning– Design characteristics– Learner characteristics– Context

Use scientific principles to build games that teach

Implications and Recommendations

The Science of Game-based Training Effectiveness 29 March 2012 Krista Langkamer Ratwani

Documents

Transcript of The Science of Game-based Training Effectiveness 29 March 2012 Krista Langkamer Ratwani