Valerie ShuteFlorida State University

Games for Learning @ G4C (June 22, 2011)

Stealth Assessment Games, Learning, and

Games, Learning, Assessment

Claim 1

Good games can act as transformative environments to support skill development and deep, meaningful learning.

Claim 2

Learning is best when it is active, interesting, goal-oriented, and contextualized (i.e., features of good games).

Claim 3

Stealth assessmentcan collect dynamic evidence of learning in real-time, at various grain sizes (and use info to support learning).

+ + +

Define 21st century competencies & good games

Overview evidence-centered design

Describe stealth assessment

Illustrate approach in a worked example

Game Plan

The world has changed a lot in the past 100 years. Education has not.

Classroom photo, 1910 Classroom photo, 2010

It’s Time for a Change

Today’s kids need new

skills to be successful

and productive

(e.g., creativity, collabor

ation, communication).

The world in which we

live is complex and

interconnected

(especially compared to

the really old days).

21st Century Skills

Curiosity

Critical Thinking

Collaboration

Grit

Systems Thinking

Empathy

Creativity

One problem with embracing new skills is the lack of valid & reliable assessments for them.

Old ways of testing won’t work. Learning and succeeding in our complex world can’t be optimally be measured by MC tests.

Re-think assessment!

Challenges

Games

Good Game Elements

1. Interactive problem solving

2. Specific goals/rules

3. Adaptive challenges

4. Control

5. Ongoing feedback

6. Uncertainty

7. Sensory Stimuli

Interactivity

Goals

Challenges

ControlFeedback

Uncertainty

Stimuli

Gestalt of Games

(e.g., Mislevy, Steinberg, & Almond, 2003)

Evidence-Centered Design

Competency Model

What do you want to say about the

person?

Evidence Model

What observations would provide best

evidence for what you want to say?

Task/Action Model

Model

What kinds of tasks let you make the

necessary observations?

Assessment Design

Design & Diagnosis

Stealth Assessment

Accurate & Rich Learner Models

Seamless & Ubiquitous Assessment

Aligned along 3 dimensions: vertically, horizontally, and temporally

Stealth Assessment Features

Advances in measurement let us administer evidence-based assessments to:

Extract ongoing information from a learner

Make accurate inferences of competencies

React in immediate and helpful ways.

Accomplished via automated scoring and machine-based reasoning techniques.

When assessment is so seamlessly woven into the fabric of the learning environment that it’s invisible, this is stealth assessment.

Stealth Assessment

Elder Scrolls IV

Oblivion

First person 3D RPG set in a medieval world

Can be one of many characters (e.g., knight, mage, elf), each who has (or can obtain) various weapons, spells, and tools

Primary goal—gain rank & complete quests

Quests may include locating a person to obtain info, figuring out a clue for future quests, etc.

Multiple mini quests along the way, and a major quest that results in winning the game (100s of hr of game play)

Players have the freedom to complete quests in any order

Elder Scrolls IV: Oblivion

Success in Oblivion

Cognitive Dispositions

Background

Knowledge

Reasoning

Skills

Working

Memory

Reading

Comp

Problem

Solving

ReflectionExploratory

Behavior

PersistenceCreativity

Creative

Problem

Solving

Attention Efficiency Novelty

Competency Model

Efficiency Novelty

Scene 2Scene 1

Evidence Model

Competency Model

Action Model

Creative Problem Solving

Scene 1

Action Indicators Scene 2

Unobservables

Observables

The Glue

Example ECD Models

Problem Solving Creativity

Relevant* Action Novelty Efficiency

Swim across the river n = 0.12 e = 0.22

Levitate over the river n = 0.33 e = 0.70

Freeze water with a spell and slide across n = 0.76 e = 0.80

Find a bridge over the river n = 0.66 e = 0.24

Dig a tunnel under the river n = 0.78 e = 0.20

* Relevant refers to any action included in a successful solution.

Problem: Cross river filled with dangerous fish to get to the cave on the other side.

Action Model w/Indicators

Novelty: 1 – frequency

Efficiency: Inverse fn (resources, time)

Action: Find a bridge over the river

Indicators: Novelty = 1 - 0.34 = 0.66

Efficiency = 1 / [(3 0.4) + (5 0.6)] = 0.24

• Resources Used = Weapon (1, fight monster w/sword) + Health (1, damage from monster) + Object (1, magic potion) = 3 resources (weight = 0.4)

• Time expended = 5 minutes (weight = 0.6)

Indicators Per Action

CreativeProblemSolving

LowHigh

0.600.40

Creativity

LowHigh

0.110.89

ProblemSolving

LowHigh

0.640.36

ObservedNovelty

0 to 0.250.25 to 0.50.5 to 0.750.75 to 1

0001

0.78 0.07

Novelty

LowHigh

0.020.98

Efficiency

LowHigh

0.860.14

ObservedEfficiency

0 to 0.250.25 to 0.50.5 to 0.750.75 to 1

1000

0.20 0.07

Dig a tunnel under the river: e = 0.20; n = 0.78

Bayes Model—Case 1

Bayes nets can be used in various ways to improve learning and performance.

Continuously gather evidence for accurate, real-time estimates of comp’s.

Info on competencies used by (a) teachers(to adjust instruction & give good feedback), (b) system (to select new gaming experiences), and/or (c) students (to reflect on how they’re doing).

Current estimates of competency levels can be integrated into the game and displayed as progress indicators.

This elevates valued competencies to the same level as health and weapons!

Wrapping it Up

To address educational challenges and harness potential of immersive games, I presented an ECD-inspired idea re:

• Specifying competencies to be acquired from the game

• Defining EMs that link game behaviors to competencies

• Updating the learner model regularly

Using ECD, stealth assessment, and automated data collection and analysis tools is meant to collect valid evidence of students’ emerging competencies, and reduce teachers’ workload allowing them to focus on fostering student learning.

Next steps: adapt content (feedback, difficulty levels, scaffolding, etc.) in game to fit current needs of player, test plug ’n play of models, evaluate learning effects, etc.

Summary

Thank you!

Questions?

Email: vshute@fsu.eduWebsite: http://www.myweb.fsu.edu/vshute

In the Stone Age, kids

learned how to start a fire

with sticks. Hunting skills

were important too.

The world in which they

lived was fairly simple and

flat.

Really Old Days

Return