Observing programming

novicesNeil BrownUniversity of Kent@twistedsq

Digibury, 13 Nov 2013

Neil Brown, University of Kent, @twistedsq

How do people learn to program?(and how can we help them?)

How can we find this outat a large scale?

Neil Brown, University of Kent, @twistedsq

What We Make

BlueJ Greenfoot

Neil Brown, University of Kent, @twistedsq

What We Make

Neil Brown, University of Kent, @twistedsq

What We Make

Neil Brown, University of Kent, @twistedsq

What We Make

BlueJ Greenfoot

2.5 million usersannually

0.4 million usersannually

Neil Brown, University of Kent, @twistedsq

What We Make

BlueJ Greenfoot

2.5 million usersannually

0.4 million usersannually

What Are They All Doing?

Neil Brown, University of Kent, @twistedsq

Some Small-Scale StudiesAn Exploration of Novice Compilation Behaviour in BlueJ, Matt Jadud, 2007

“Many students write significant amounts of code (10+ lines) at a time, and then attempt to eliminate all the syntactic errors that exist in the code”

Neil Brown, University of Kent, @twistedsq

Some Small-Scale StudiesAn Exploration of Novice Compilation Behaviour in BlueJ, Matt Jadud, 2007

“Many students write significant amounts of code (10+ lines) at a time, and then attempt to eliminate all the syntactic errors that exist in the code”

Study Size: 62 students

Neil Brown, University of Kent, @twistedsq

BIG DATAAdd recording to all BlueJ instances

(With explicit opt-in)

Neil Brown, University of Kent, @twistedsq

BIG DATAAdd recording to all BlueJ instances

(With explicit opt-in)

MEDIUM

Neil Brown, University of Kent, @twistedsq

How Much Data?

20,000 users per day≈ 25% opt-in?≈ 100KB data per user per day≈ 0.5GB per day≈ 200GB per year

Neil Brown, University of Kent, @twistedsq

How Much Data?

20,000 users per day≈ 25% opt-in?≈ 100KB data per user per day≈ 0.5GB per day≈ 200GB per year

✓✗ 40%

✓✗ ≈ 1 GB✗ 300-400 GB

Neil Brown, University of Kent, @twistedsq

Headline statistics so far (5 months in)

140,000 opted-in users

600,000 projects

5,100,000 successful compilations

4,700,000 unsuccessful compilations

Neil Brown, University of Kent, @twistedsq

Hardware Specs2 machines (1 for recording, 1 for analysis)24 core 2.5Ghz Xeon, 32GB RAM, 5TB RAID

Neil Brown, University of Kent, @twistedsq

Most common compile errorsUnknown variable 17%

Semi-colon expected 10%

Unknown method 7%

Bracket expected 7%

Unknown class 5%

Illegal start of expression 4%

Neil Brown, University of Kent, @twistedsq

Most common compile errorsUnknown variable 17%

Semi-colon expected 10%

Unknown method 7%

Bracket expected 7%

Unknown class 5%

Illegal start of expression 4%

Do they change during the term?

Neil Brown, University of Kent, @twistedsq

Compile errors over time

Neil Brown, University of Kent, @twistedsq

Rarer compile errors

65th most common compilation error:

非法的表�式开始

Neil Brown, University of Kent, @twistedsq

Rarer compile errors

65th most common compilation error:

非法的表�式开始

Neil Brown, University of Kent, @twistedsq

Problematic if statements

What does this code do?

if (x >= 6 && x <= 9){ x = 0;}

Neil Brown, University of Kent, @twistedsq

Problematic if statements

What does this code do?

if (x*x >= 36 && x*x <= 81);{ x = 0;}

Neil Brown, University of Kent, @twistedsq

Problematic if statements

How prevalent is this mistake?

How long does it take before people fix it?

Appeared in 0.15% of source files

Later fixed in half of them...

Neil Brown, University of Kent, @twistedsq

Problematic if statements

Neil Brown, University of Kent, @twistedsq

ChallengesA lot of data -- and a lot of method questions, e.g.

- How do you measure error difficulty? - What is a frequent error? (what is worth caring about?) - How much can you get from this kind of data-set?

Scaling the analysis (already maxing out 24 cores)

Questions?