Slide 1

Serious Play Conference Los Angeles, CA July 22, 2014 Girlie C. Delacruz Games as Assessment: Validating Patterns of Gameplay Via Cluster Analysis

Slide 2

ENGAGE Team Regents of the University of California Eva Baker Principal Investigator Girlie Delacruz Project Director Greg Chung Project Director Noelle Griffin Associate Director Ayesha Madni Senior Researcher Markus Iseli Senior Researcher Alan Koenig Senior Researcher

Slide 3

Typical Metrics

Slide 4

Determine Proficiency

Slide 5

Diagnostic Assessment

Slide 6

Games As Formative Assessment Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967).

Slide 7

Games As Formative Assessment Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967).

Slide 8

Qualitative Metrics

Slide 9

Slide 10

ENGAGE

Slide 11

Project Goals

Slide 12

ENGAGE Games

Slide 13

Go Vector Go

Slide 14

Current Study: Validating Patterns of Gameplay

Slide 15

Research Questions 1.What patterns of gameplay are identified using cluster analysis? 2.What is the association between identified clusters and performance on the related item administered the pretest?

Slide 16

Force and Motion: Constant Velocity Big Idea A change in the position and motion of an object is a result of the manipulation of forces acting on an object. Example Learning Outcome Manipulate friction, mass, force magnitude, force direction, and slope to cause an object to move at constant velocity.

Slide 17

Pretest Item

Slide 18

Diagnostic Options 1. No changes in forces. 2. Vector know what to do now. 3. A lot of forces. 4. A lot of energy. Prompt Why does Vectors speed not change?

Slide 19

Cluster Analysis

Slide 20

Validity Evidence for Gameplay Patterns

Slide 21

Choosing Clustering Variables LevelVector isNeed toBy Initial Force Magnitude Initial Surface Friction 4 Not moving Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt)

Slide 22

Results What patterns of behavior are identified using cluster analysis? Gameplay pattern of friction values chosen on first attempt (correct friction value should be 0) Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster 178--00 Cluster 246--03 Cluster 363--33

Slide 23

Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be 0) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster 178--00 Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster 246--03 Cluster 363--33 Students in this cluster were over 3 times more like to choose the incorrect answer option Vector knows what to do now.

Slide 24

Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be 0) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster 178--00 Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster 246--03 Cluster 363--33 Students in this cluster were over 3 times more like to choose the incorrect answer option Vector knows what to do now.

Slide 25

Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be 0) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster 178--00 Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster 246--03 Cluster 363--33 Students in this cluster were over 3 times more like to choose the incorrect answer option Vector knows what to do now.

Slide 26

What Does This Mean? LevelVector isNeed toBy Initial Force Magnitude Initial Surface Friction 4 Not moving Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt)

Slide 27

Implications: Validity Evidence for Gameplay Patterns

Slide 28

Next Steps

Slide 29

THANK YOU!!! Regents of the University of California gdelacruz@cse.ucla.edu www.cresst.org

Slide 30

Gameplay Patterns: Levels 4 - 10 - 18

Slide 31

Gameplay Patterns: Levels 10 - 18

Slide 32

Gameplay Patterns: Levels 4 - 10 - 18

Slide 33

Gameplay Patterns: Levels 10 - 18

Slide 34

Association of Cluster Membership to Pretest Performance

Slide 35

Association of Cluster Membership to Incorrect Answer Options Chosen 2. Vector know what to do now.