Post on 15-Aug-2015
Developing Number Sense with Play: Lessons in Data AnalysisRyan Carlson
Overview• New iPad game to practice & develop
number sense
• In-school user study & pilot study addressing non-cognitive factors
• Discussion of instrumentation & evaluation of our game
Outline• Some Definitions• NumberACEr• Designing a configurable game• Instrumentation• User Study• Pilot Study• Data Analysis• Lessons Learned
Who Are We?• Founded 1998• Develop K-12 mathematics blended
curricula• 600k students / year across >2,000 schools• Cognitive Tutor
• Model student knowledge• Adaptively present new material
What is Number Sense?
“fluidity and flexibility with numbers, the sense of what numbers
mean and ability to perform mental mathematics to look at the world and
make comparisons”(Gersten & Chard, 1999)
• Foundation for further mathematics• Early number sense predicts later
mathematics ability (Geary et. al., 2013)
What is Number Sense?
What is Number Sense?
What is Number Sense?
What is Number Sense?
Personalization
CognitiveWhat you know
(positive results)
• Knowledge Tracing
Non-CognitiveEverything Else
(less well understood)
• Self-Efficacy• Grit• Learning
Orientation
NumerACEr
NumerACEr
NumerACEr
Configurability• Content• Game Parameters• Level Layout• Scripting System
• Dynamic runtime engine• React to game events• Add text / images
For non-programmers
Configurability
Configurability
Instrumentation
Boring But Important• Generate usage data & plan analyses
before deployment• Logging infrastructure & API
• Developers annotate code• Log messages sent to central storage
• Log everything!• Level started?• Speed changes?• iPad tilt?• Device suspend?
User Study• December 2014 in Raleigh, West Virginia
• 6th graders (36)• 8th graders (7)
• Played for ~10 min• Exit feedback (verbal + written)
User Study
User Study
Aggressive
Neutral(High Accuracy)
Neutral(Low Accuracy)
Cautious
User Study
Aggressive
Neutral(High Accuracy)
Neutral(Low Accuracy)
Cautious
Pilot Study• May 2015 in Savannah, Georgia
• 6th graders (200)
• Remote Deployment• 15 minute play requested• Non-Cognitive survey administered prior to
play• Self-efficacy• Grit• Mastery learning orientation
Pilot Study: Interventions
Self-Efficacy• Low SE Motivation
Pilot Study: Interventions
Grit• High Grit Increased Difficulty Earning
Brains
Learning Orientation• Low Mastery Coach Towards Mastery
Pilot Study: Interventions
Pilot Study: Interventions
Game Strategy• Aggressive Slowdown
• Reduce maximum speed
• Cautious Speedup• Increase minimum speed
• Neutral (low acc.) Instruction• Provide in-game feedback on mistakes
Pilot Study: Interventions
Pilot Study: Interventions
Pilot Study: Game Usage
Pilot Study: Game Usage
Pilot Study: Game Usage
Pilot Study: Game Effectiveness
Pilot Study: Intervention Effectiveness
Relationship b/w Self-Efficacy and how much students play the game
Self-Efficacy & Grit interact in an interesting way
Pilot Study: Intervention Effectiveness
F(1,105)=3.81 p=0.05
Pilot Study: Intervention Effectiveness
χ2 test, p<.01
Pilot Study: Intervention Effectiveness
F(1,176)=3.46p<0.10
Pilot Study: Intervention Effectiveness
F(1,176)=0.05p>.10
Pilot Study: Intervention Effectiveness
F(1,176)=4.50p<.05
Pilot Study: Intervention Effectiveness
F(1,176)=0.03p>.10
Lessons Learned & Limitations
• Promising directions for adapting to cognitive + non-cognitive factors into games
• Gamification serves a purpose• Countdown timer adds pressure• Speed forces rapid comparisons• “pretty fun for a school game”
• Desire for common game features• Customization, currency
• Assumption: the game is a valid measure of number sense
References• Geary DC, Hoard MK, Nugent L, Bailey DH (2013) “Adolescents’
Functional Numeracy Is Predicted by Their School Entry Number System Knowledge”. PLoS ONE 8(1): e54651.
• Gersten, R., and D. Chard. “Number Sense: Rethinking Arithmetic Instruction for Students with Mathematical Disabilities.” The Journal of Special Education 33.1 (1999): 18-28
Thank you!Questions?
rcarlson@carnegielearning.com