Post on 07-Feb-2018
1
PROGRAMA DE EVALUACIÓN DE COMPETENCIAS
PARA EL APRENDIZAJE DE LAS CIENCIAS
NASA Center for Educational Technologies
Wheeling Jesuit University
Debbie Reese
2
[ _________]:
The Greatest Shows on Earth
3
4
Successful art tears away the veil and allows you to
see the world with lapidary clarity; successful art
pulls you apart and puts you back together again,
often against your will, and in the process reminds
you in a visceral way of your limitations, your
vulnerabilities, makes you in effect more human. . . .
We have the technology, the narrative sophistication
and an audience willing to take any number of wild
illuminating rides as long as they're couched in the
grammar of spectacular addictive gameplay. And the
fact that such a game is possible, even if it hasn't yet
materialized, is [a cause for celebration].
-Junot Díaz
5
Aspirations:
What we can do for entertainment--
We can do for education.
6
Cyberlearning
Game-based
Metaphor-Enhanced
***
1. Make hard concepts intuitive.
2. Translate expert knowledge into
gameplay.
3. Provide experiential learning.
4. Assess learning and flow.
~CyGaMEs
7
Cyberlearing
Game-based
Metaphor-Enhanced
***
1. Make hard concepts intuitive.
2. Translate expert knowledge into gameplay.
3. Provide experiential learning.
4.Assess learning & flow.
~CyGaMEs
8
~Intersection
Game Design
Instructional
Design
Subject/
Pedagogy
Expertise
Single teacher
in classroom
does not create
instructional games
Instructional
game is not play.
Metaphorist
Game
design/development is
expensive
Game
design/development is
tough!
9
~Intersection
Game Design
Instructional
Design
Subject/
Pedagogy
Expertise
Single teacher
in classroom
does not create
instructional games
Instructional
game is not play.
Metaphorist
Game
design/development is
expensive
Game
design/development is
tough!
10
~The Team
• NASA eEducation
• Dr. Debbie Denise Reese - Center for Educational Technologies, Wheeling Jesuit University (WJU)
• Dr. Charles Wood –Center for Educational Technologies, WJU
• Dr. Ian Bogost – Persuasive Games & Georgia Tech
• Dr. Ben Hitt – Center for Informatics Sciences, WJU
• Andrew Harrison – Wheeling Jesuit University
• James Oliverio –Digital Worlds Institute, University of Florida
• The Georgia Tech and Center for Educational Technologies Team Members (listed alphabetically)– Dr. James Coffield, Dr. Karen Chen, Katy Cox, Justin Erfort, Matt Gilbert, Will Hankinson, Andrew Harrison, Chris Kreger, Ron Magers, Lisa McFarland, Don Watson.
• David P. Nichols –SPSS master statistician
11
A Growing Team
• Dr. Agustin Tristan – FAMILIA DE PROGRAMAS
KALT.
• Dr. Virginia Diehl – Western Illinois University
• Dr. Beverly Carter – Wheeling Jesuit University
• Dr. Barbara Tabachnick - California State University,
Northridge
• Storm Conaway - Wheeling Jesuit University
• Michael Phillips - Wheeling Jesuit University
• Ralph Seward – Wheeling Jesuit University
• Lisa McFarland – Wheeling Jesuit University
12
~Theory
• Structure mapping (Gentner, 1980+)
• GaME design (Reese, 2003+)
13
~Approach
CORE Content
Gameplay
Game system
Game mechanics
Game goals
14
~ 3 Tools
Flowometer
Assessments
1
3
2
Timed Report Gesture
15
~Continuum
Assessments
16
~Design
From “First Steps and Beyond: Serious Games as Preparation for Future Learning” by Debbie
Denise Reese, 2007, Journal of Educational Media and Hypermedia, in press. Copyright 2007
Association for the Advancement of Computing in Education.
17
~Design
Adapted from Double Transfer Paradigm (Schwartz & Martin, 2004).
Myset 4 Myset 3 Myset 1 Myset 2
Play
Instruction
Watch
Watch Watch
Play
P l a y
18 Image: Bill Hartmann
Chuck Wood
Lunar Scientist
19
Proof-of-Concept
• Embodied.
• Relevant.
• Interest thru
knowledge.
© William K. Hartmann
20
~Domain
21
22
~Assessment
Embedded
• Perceived experience – Flowometer
• Learning – Timed Report
• Learning – Gestures
• Time
External
• Timeline
• Mutual Alignment
23
~Assessment
Accretion
0. Slingshot
1. Scale
Surface Features
2. Select tool
3. Aim
4. Launch
5. Impact
6. Garden
7. Place vent
8. Aim basin-forming asteroid
9. Launch basin-forming asteroid
10. Basin-forming impact
24
~Assessment
Embedded
• Perceived experience – Flowometer
• Learning – Timed Report
• Learning – Gestures
• Time
External
• Timeline
• Mutual Alignment
25
• Pragmatic constraints (Holyoak,1980+)
• Learning goal = Game goal
• 1 variable
~3 Scales: 1.1
26
• Pragmatic constraints (Holyoak,1980+)
• Learning goal = Game goal
• 3 variables
~3 Scales: 1.2
27
• Pragmatic constraints (Holyoak,1980+)
• Learning goal = Game goal
• 5 variables
~3 Scales : 1.3
28
Flow
• Action/awareness
• Concentration
• Loss of self-
consciousness
• Goals
• Feedback
• Paradox of control
29
30
~ESM
Experience Sampling Method Form
31
~Flowometer
32
~Omnibus
• 4 x 7 x 2
• Between-within-within ANOVA
• IV – Condition (4, between): 1-PIP, 2-PPI, 3-
WIP, 4-WPI.
– Segment (7 within)
– Flow (2 within): Challenge and Skill
• DV: Rating (on scale 0-100)
• Outliers moved to next lowest
33
~Omnibus
• Between-within-within ANOVA
• N=96
• nPIP=20, nPPI=9, nWIP=35, nWPI=32
• Univariate, SPSS 15 GLM repeated
• Sphericity-Huynh-Feldt adjustment
34
~Omnibus
Flow: NS
Flow x Condition
• F(3,92)=6.55, p<.001, partial η2=18
Segment
• F(4.40,404.83)=3.98, p<.01, partial η2=.04
Segment x Condition
• NS
Flow x Segment
• F(3.67,337.62)=55.39, p<.001, partial η2=.38
Flow x Segment x Condition
• F(11.01,337.62)=5.30, p<.001, partial η2=.15
35
Myset 1
61.58
39.96
36.29
76.16 75.42
30.68
26.00
39.87
68.87 69.68
34.68
24.89
75.61 76.16
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Play R1. Watch R2. Play
Skill
Challenge
W P W P
36
Myset 1
61.58
39.96
36.29
76.16 75.42
30.68
26.00
39.87
68.87 69.68
34.68
24.89
75.61 76.16
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Play R1. Watch R2. Play
Skill
Challenge
W P W P
37
61.58
39.96
36.29
76.16 75.42
30.68
26.00
39.87
68.87 69.68
34.68
24.89
75.61 76.16
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Play R1. Watch R2. Play
Skill
Challenge53.41
59.67
40.86
45.52
36.82
83.41
71.82
51.09
67.96
74.41
77.72 77.36
32.27
39.82
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. Iacc 9. ISF
R1. Watch R1. Play R2. Play R2. Watch
Skill
Challenge
51.38
64.24
72.14 71.22
74.72
33.53
27.33
39.18
31.85
22.67 22.78 22.81
75.63
78.97
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Watch R1. Watch R2. Play
Skill
Challenge50.10
64.94
69.65
50.73
40.81
71.94 72.81
51.53
40.77 41.61
67.63
73.84
37.84
32.48
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. I-Acc 9. I-SF
R1. Watch R1. Watch R2. Play R2. Watch
Skill
Challenge
Myset 1 Myset 2
Myset 3 Myset 4
38
61.58
39.96
36.29
76.16 75.42
30.68
26.00
39.87
68.87 69.68
34.68
24.89
75.61 76.16
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Play R1. Watch R2. Play
Skill
Challenge53.41
59.67
40.86
45.52
36.82
83.41
71.82
51.09
67.96
74.41
77.72 77.36
32.27
39.82
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. Iacc 9. ISF
R1. Watch R1. Play R2. Play R2. Watch
Skill
Challenge
51.38
64.24
72.14 71.22
74.72
33.53
27.33
39.18
31.85
22.67 22.78 22.81
75.63
78.97
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Watch R1. Watch R2. Play
Skill
Challenge50.10
64.94
69.65
50.73
40.81
71.94 72.81
51.53
40.77 41.61
67.63
73.84
37.84
32.48
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. I-Acc 9. I-SF
R1. Watch R1. Watch R2. Play R2. Watch
Skill
Challenge
Myset 1 Myset 2
Myset 3 Myset 4
39
61.58
39.96
36.29
76.16 75.42
30.68
26.00
39.87
68.87 69.68
34.68
24.89
75.61 76.16
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Play R1. Watch R2. Play
Skill
Challenge53.41
59.67
40.86
45.52
36.82
83.41
71.82
51.09
67.96
74.41
77.72 77.36
32.27
39.82
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. Iacc 9. ISF
R1. Watch R1. Play R2. Play R2. Watch
Skill
Challenge
51.38
64.24
72.14 71.22
74.72
33.53
27.33
39.18
31.85
22.67 22.78 22.81
75.63
78.97
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Watch R1. Watch R2. Play
Skill
Challenge50.10
64.94
69.65
50.73
40.81
71.94 72.81
51.53
40.77 41.61
67.63
73.84
37.84
32.48
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. I-Acc 9. I-SF
R1. Watch R1. Watch R2. Play R2. Watch
Skill
Challenge
Myset 1 Myset 2
Myset 3 Myset 4
40
61.58
39.96
36.29
76.16 75.42
30.68
26.00
39.87
68.87 69.68
34.68
24.89
75.61 76.16
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Play R1. Watch R2. Play
Skill
Challenge53.41
59.67
40.86
45.52
36.82
83.41
71.82
51.09
67.96
74.41
77.72 77.36
32.27
39.82
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. Iacc 9. ISF
R1. Watch R1. Play R2. Play R2. Watch
Skill
Challenge
51.38
64.24
72.14 71.22
74.72
33.53
27.33
39.18
31.85
22.67 22.78 22.81
75.63
78.97
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 4. IAcc 5. I. SF 6. Acc 7. SF
R1. Watch R1. Watch R1. Watch R2. Play
Skill
Challenge50.10
64.94
69.65
50.73
40.81
71.94 72.81
51.53
40.77 41.61
67.63
73.84
37.84
32.48
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
1. SSAcc 2. Acc 3. SF 6. Acc 7. SF 8. I-Acc 9. I-SF
R1. Watch R1. Watch R2. Play R2. Watch
Skill
Challenge
Myset 1 Myset 2
Myset 3 Myset 4
41
~Difference Trend
-80
-60
-40
-20
0
20
40
60
80
1 2 3 4 5 6 7
Time
Dif
fere
nce (
Ch
all
en
ge -
Skil
l)
1 PIP
2 PPI
3 WIP
4 WPI
p
PLAY
WATCH
p
p
pp
p
p
p
p
pp
I
I II
I
I
II
42
~Flow Linear Trend
43
~Flow Linear Trend
44
~Means what?
Myset 1
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100Skill
Ch
all
en
ge
1. r1w
2. r1p
3. r1p
4. r1wI
5. r1wI
6. r2p7. r2p
Myset 2
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100Skill
Ch
all
en
ge
1.r1w
2. r1p
3.r1p
8.r2wI
9. r2wI
6. r2p7.r2p
Myset 3
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100Skill
Ch
all
en
ge
1.r1w
4.r1I
5.r1I
3.r1w
2. r1w
6.r2p
7. r2p
Myset 4
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100Skill
Ch
alle
ng
e
1. r1w
2. r1w
3. r1w
8. r1wI
9. r1wI
6. r2p7. r2p
45
~Screening
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Skill
Ch
all
en
ge
Instructional Movie
Watch Solar System Accretion
Gameplay
Apathy Relaxation
Boredom
Worry
Anxiety Flow
Arousal
Control I
I
W
I
W
46
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 10
0
Skill
Ch
allen
ge
1. r1w
2. r1p
3. r1p
4. r1wI
5. r1wI
6. r2p
7. r2p
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Skill
Ch
all
en
ge
Instructional Movie
Watch Solar System Accretion
Gameplay
Apathy Relaxation
Boredom
Worry
Anxiety Flow
Arousal
Control I
I
W
I
W
Skill
100
Player 1714 Trace Mean Myset 1 Trace
47
~Flow PID 1714
0
20
40
60
80
100
1201.
DS
SA
cc2.
GM
Ar1
s13.
GM
Ar1
s34.
GM
Ar1
s35.
GM
Ar1
s36.
GM
Ar1
s37.
GS
Fr1
S2
8.
GS
Fr1
s3
9. I
MO
10.
IV
11.
GM
Ar2
s312
.
GM
Ar2
s313
.
GS
Fr2
s2
R1.
Watch
R1. Play R1.
Watch
R2. Play
PID 1714 •Challenge
•Skill
Myset 1Mean •Challenge
•Skill
48
~Timed Report
• Every 10 seconds of gameplay.
• Scored
– -1 – 0
– +1
• X by
– Segment (4 or 2)
– Subsegment (12 or 6)
49
~TR Trace
0
10
20
30
40
50
60
70
80
90
100ssa
ccs/t
accR
1s/t
1
accR
1s/t
2
accR
1s/t
2
accR
1s/t
3
accR
1s/t
3
accR
1s/t
3
sfR
1s/t
2
sfR
1s/t
3
accR
2s/t
2
accR
2s/t
2
accR
2s/t
3
sfR
2s/t
1
Timed Report
skill
challenge
10
-1
IVm
agR
1
IVvolR
1
50
~TR Trace
0
10
20
30
40
50
60
70
80
90
100
ssa
ccs/
t
acc
R1
s/t1
acc
R1
s/t2
acc
R1
s/t2
acc
R1
s/t3
acc
R1
s/t3
acc
R1
s/t3
sfR
1s/
t2
sfR
1s/
t3
acc
R2
s/t2
acc
R2
s/t2
acc
R2
s/t3
sfR
2s/
t1
summed TR
skill
challenge
IVm
ag
R1
IVvo
lR1
51
~Timed Report
• 2 x 12
• Between-within ANOVA
• IV
– Condition (2, between): 1-PIP, 2-PPI.
– Subsegment (12 within)
• DV: TR Progress (scale: -1 to 1)
• Outliers not yet considered
52
~Omnibus
Subsegment
• F(8.50,229.62)=19.85, p<.001, partial
η2=.42
Subsegment x Condition
• NS
Condition
• NS
53
~Learning @5.1
54
~TR Post hoc
Sub-
Seg.
Mean*
Std.
Error
95% CI
LCI UCI
1) 1.1 .65 4, 6, 7, 9, 12 .05 .54 .76
2) 1.2 .68 3, 4, 6, 7, 9, 10, 12 .03 .63 .74
4) 2.1 .38 1, 2, 7,8
.38 .30 .45
7) 5.1 .94 1, 2,3,4,5,6,8,9,10,11,12
.02 .91 .97
8) 5.2 .68 3, 4, 6, 7, 9, 10, 12
.03 .62 .75
11) 6.2 .54 6, 7 .06 .43 .66
*Bonferroni adjustment
for multiple comparisons of
alpha=.05
Mysets PIP & PPI
55
~TR Post hoc
Subsegment (Huynt-Feldt), alpha=.0125
• F(1,28)=19.85, p<.001, partial η2=.54
.66 .95
56
~Timed Reports
57
~Timed Report
• 4 x 6
• Between-within ANOVA
• IV
– Condition (4, between): 1-PIP, 2-PPI, 3-
WIP, 4-WPI.
– Segment (6 within)
• DV: TR Progress (scale: -1 to 1)
• Outliers not yet considered
58
~Omnibus
Subsegment (Huynh-Feldt correction)
• F(3.90,355.24)=56.22, p<.001, partial
η2=.38
Subsegment x Condition
• NS
Condition
• NS
59
~All TR
60
~TR Post hoc
Sub-
Seg.
Mean*
Std.
Error
95% CI
LCI UCI
1) 5.1 .90 2, 3, 4, 5, 6 .02 .87 .93
2) 5.2 .66 1, 3, 4, 5, 6 .02 .63 .69
3) 5.3 .41 1, 2
.02 .37 .45
4) 6.1 .42 1, 2
.04 .34 .49
5) 6.2 .51 1, 2, 6
.04 .43 .59
6) 6.3 .33 1, 2, 6 .04 .25 .41
*Bonferroni adjustment
for multiple comparisons
61
~Omnibus
• Between-within-within ANOVA
• N=96
• nPIP=20, nPPI=9, nWIP=35, nWPI=32
• Univariate, SPSS 15 GLM repeated
• Sphericity-Huynh-Feldt adjustment
62
~Omnibus
Flow: NS
Flow x Condition
• F(3,92)=6.55, p<.001, partial η2=.18
Segment
• F(4.40,404.83)=3.98, p<.01, partial η2=.04
Segment x Condition
• NS
Flow x Segment
• F(3.67,337.62)=55.39, p<.001, partial η2=.38
Flow x Segment x Condition
• F(11.01,337.62)=5.30, p<.001, partial η2=.15
64
~Concepts
• Define targeted domain relational structure.
• Domain structure=game structure.
• Flow supports domain structure.
• Goal: Requires players to master targeted learning.
• Component of events of instruction.
http://selene.cet.edu
65
~CyGaMEs
Csikszentmihaly and Larson (1980) wrote: “the main goal of a truly
civilized education is in fact to teach children to experience flow in
settings that are not harmful to self and others. Again this is the
goal Plato established for his own educational system: To train
youths in how to find pleasure in action which strengthens the
bonds of human solidarity rather than set them against each other”
(p. 186).
CyGaME’s ultimate accomplishment may be to contribute to a truly
civilized education by teaching people to experience flow in
settings that are beneficial to both self and others.
Games as preparation for Future
Learning
65
66
[ _________]:
The Greatest Shows on Earth
67
~Questions?
debbie@cet.edu
selene@cet.edu
304-243-4327
CyGaMEs PI Selene project manager
Senior educational researcher
Center for Educational Technologies
NASA-sponsored Classroom of the Future
Wheeling Jesuit University
Wheeling, WV 26003
http://selene.cet.edu
68
Dr. Debbie Denise Reese is the senior educational researcher at the NASA-
sponsored Classroom of the Future (COTF) within Wheeling Jesuit University’s
Center for Educational Technologies in Wheeling, WV. She specializes in the
application of cognitive theories to the design of educational technologies and
environments. Over the past 10 years she developed a method for the design,
development, and evaluation of metaphor-enhanced, computer-mediated learning
objects through applied cognitive science metaphor theory. When NASA
eEducation established its roadmap to enhance the nation’s learning and practice
of science through videogames and synthetic worlds, COTF appointed Reese to
lead the research effort into learning and assessment in videogames. In that
capacity Reese refined her earlier work into a method for game-based, metaphor-
enhanced (GaME) instructional design and assessment. In 2007 she implemented
GaME design to produce a cyber-enabled research environment. The resultant
proof-of-concept, Selene: A Lunar Construction GaME, is the subject of national
research study in formal and informal education venues. This research is now
funded by NSF. Reese was the principal researcher and project manager for
NASA’s COTF Inspiration project, an earlier nationwide research study that
developed and tested instructional technology tools for enhancing flow, self-
efficacy, and identity. Selene extends flow research into Game environments.
Dr. Debbie
Denise Reese
CURRICULUM