Eben Myers & Santosh Mathan - Building Robot Factory: A Game Design and Brain Science Collaboration

BUILDING

A Game Design& Brain ScienceCollaboration

Eben Myers Santosh MathanSerious Play ConferenceJuly 22, 2015

2

Eben MyersVP, DesignSimcoach Games

• BA in Studio Art from Swarthmore College

• Masters in Entertainment Technology from CMU

• EPO for NASA• 10+ years designing games

Introductions

Santosh MathanPrincipal ScientistHoneywell

• Research at the intersection of Neurotechnology and HCI

• Interest in enhancing training, performance, and safety in aviation

• PhD in HCI from CMU• Prior roles at NASA Ames and

Microsoft

FAST: Flexible, Adaptive, Synergistic

TrainingHarvard Site Visit

4

IARPA’s SHARP Program

5

• EF moderate complex tasks (Elliott, 2003) :

- Solving novel problems

- Modifying behaviors in light of new information

- Generating strategies

- Sequencing complex actions

• ARP crucially tied to EF- overlap in core computational properties (Diamond, 2013)

- Neural correlates (e.g. DLPFC) (Duncan et al., 2000; Barbey et al, 2012)

Executive Functions and ARP

6

• Updating: updating and monitoring working memory representations• Switching: shifting between tasks or mental sets • Inhibition: suppression of stimuli and responses that are unrelated to the task

Executive Functions: Component Processes

Diamond, 2013

Miyake, 2000

7

• Training and practice have robust and predictable impact

- Skills improve – following power law function

- ARP can be strengthened through training (Carlson, 1990; Mackey, 2011)

• Limited generalization:

- Acquired skill highly specific

- Limited evidence for near or far transfer

• By nature ARP must be domain general and widely applicable

Enhancing Executive Functions: Promise, Pitfalls

8

• Train component processes as cognitive skill

• Combine processes, in novel ways, with range of operations (logical relational), and cognitive domains (symbolic, visual)

• Create rich, but systematically mapped training space

• Promote flexible encoding by changing instructions every few trials

- Rapidly encode instructions (Salthouse & Pink, 2008)

- Flexibly configure cognitive resources (Duncan et al., 2012)

- Strong activation of regions implicated with ARP (Dumontheil et al., 2011)

FAST Training: Implications

9

• Train component processes as cognitive skill

• Combine processes, in novel ways, with range of operations (logical relational), and cognitive domains (symbolic, visual)

• Create rich, but systematically mapped training space

• Promote flexible encoding by changing instructions every few trials

- Rapidly encode instructions (Salthouse & Pink, 2008)

- Flexibly configure cognitive resources (Duncan et al., 2012)

- Strong activation of regions implicated with ARP (Dumontheil et al., 2011)

Intervention 1 - FAST Training: Implications

10

Collaborators

11

FAST - Intervention

12

FAST: Level 1

13

FAST: Level 2

14

FAST: Level 3

15

FAST: Level 4A

16

FAST: Level 4B

17

• Engaging isomorphs of FAST

• Everyday processing demands

• Immediate feedback

• Goal-directed practice

• Self-motivating demands

Gamification

18

• Augment activity of cortical substrate supporting EF using tES

• Distributed fronto-parietal network: DLPFC, ACC, LPC

- DLPFC: connectivity with sensory motor regions, maintaining rules for action, and guiding response selection and inhibition

- ACC: detect processing conflict, engaging DLPFC

- LPC : Stimuli salience, stimulus-response pairings

• tES to act synergistically with training to boost core ARP processes

tES

19

• Augment activity of cortical substrate supporting EF using tES

• Distributed fronto-parietal network: DLPFC, ACC, LPC

- DLPFC: connectivity with sensory motor regions, maintaining rules for action, and guiding response selection and inhibition

- ACC: detect processing conflict, engaging DLPFC

- LPC : Stimuli salience, stimulus-response pairings

• tES to act synergistically with training to boost core ARP processes

Intervention 2: tES

20

• Learning is the development of new neural connections

• Connections are formed through synchronized recruitment

• Anodal stim has depolarizing effect of resting potential of neurons – making it easier for neural cells to fire

• Synergistic interaction with training to boost efficacy of core ARP processes

Causal Mechanism

21

Pilot Study

1

• Assess game implementation of FAST• Test procedures

Pilot Study 2

• Test expanded FAST game• Assess stimulation types

Pre T&E

Evaluation

• Demonstrate efficacy of intervention (FAST + stimulation)

Empirical Validation – Stages and Objectives

22

1A Demographics

23

Pre T&E Fluid Intelligence Results

Gf MeasuretRNS+FAST

b p-value Cohen’s d

BOMAT 1.417 .051 .424

Sandia 1.400 .275 .164

Raven 0.473 .448 .136

24

• https://www.youtube.com/watch?v=qdhKR0g3bhc

Video Demo

Process:What Worked & What Could Have

Been Better

26

• Iterative design and development process• Communication• Personnel

What Worked

27

• Iterative design - from research to 3 concepts to one

Iterative design & development process

28

• Regular, frequent releases of game


Stage Build # Date

Project Kick-Off 1/29/14

High Level Design Complete 2/18/14

First Playable Proof-of-Concept 1 3/3/14

29

Proof-of-Concept

30

• Regular, frequent releases of game


Stage Build # Date

Project Kick-Off 1/29/14

High Level Design Complete 2/18/14

First Playable Proof-of-Concept 1 3/3/14

On-Going Weekly Builds4 Major Releases (for Experimental Data Collection)

1A-1 9 4/18/14

1A-2 20 6/25/14

1A-3 25 9/23/14

1B 34 3/9/15

31

Mature Game

32

Communication

33

• 2 weekly meetings

Communication

34

• Structuring of the abstract games

Communication

35


Communication

36


Communication

37


Communication

38


Communication

39

• GitHub releases and integrated issue tracking

Communication

40

• Attitude:

- Respectful

- Collaborative

- Honest and critical

Personnel

41

• Inclusion of research engineer on Simcoach dev team

Personnel

42

• Building the airplane while flying

What Could Have Been Better

43

• Building the airplane while flying• Building the experimental rocket while launching into space…


44

• More focus on QA


45

• More time to respond to feature requests


46

• Iterative design and development process• Communication• Personnel• Planning

Summary of Takeaways

47

Questions?

@santoshmathan@honeywellnow

@ebenmyers@simcoachgames

Eben Myers & Santosh Mathan - Building Robot Factory: A Game Design and Brain Science Collaboration

Science

Transcript of Eben Myers & Santosh Mathan - Building Robot Factory: A Game Design and Brain Science Collaboration