Mining student repositories to gain learninganalytics

An experience report

Gregorio Robles, Jesus M. Gonzalez Barahona

{grex,jgb}@gsyc.urjc.esGSyC/LibreSoft, Universidad Rey Juan Carlos, Madrid, Spain

Berlin, Germany, March 14th, 2013

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

c©2013 Gregorio Robles, Jesus M. Gonzalez-Barahona

All figures are ours, except when the original source is specified.

Some rights reserved. This presentation is distributed under the“Attribution-ShareAlike 3.0” license, by Creative Commons, available at

http://creativecommons.org/licenses/by-sa/3.0/

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

Summary: What this talk is about

Engineering students often have to deliver small computerprograms in many engineering courses

Instructors have to evaluate these assignments according tothe learning goals and their quality, but ensure as well thatthere is no plagiarism

We report the experience of using mining software repositoriestechniques

Related efforts (please, see paper)ProcedureToolsLinks and ideas

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

Context

3rd-year Telecommunication Engineering students at URJC

Course on multimedia networks

Assignments: small programs that exchange multimediaset-up and content information using standardized networkprotocols: SIP, SDP, RTP, UDP, IP...

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

Technologies used by students

Python

git: distributed versioning system

pep8 (a script that checks if the code follows codingguidelines)

wireshark: network protocol analyzer

Scope. The assignment includes:

Program with communication among clients and servers usingstandardized protocols

A live capture with the result of a scenario

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

2. Preprocess

Cloning of the repository

Checking if the files with the assignment exist are have beencorrectly named

Checking if the style guide has been followed (with pep8)

Evaluating the quality of the code (in our case, Pylint)

Retrieving of the git log and analysis (analyzed withCVSAnalY)

Analysis of the wireshark network exchange capture

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

3. Plagiarism detection

A note about ”plagiarism”

We use the non-open-sourcefree web service MOSS(from Stanford University)

Figure: Plagiarism

4. Functional assessment

Black box testing

Domain-specific

In computer networks:standards-oriented

Figure: Black-box testing. Source:goo.gl/3e1Fq

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

5. Post-process

Final grades for theassignment are calculated

Creates file with feedbackfor the student, with inputinformation from all thesteps

Instructors get a report ofthe whole process, includingassignments suspicious ofplagiarism, and errors

6. Personalized exam

Three type of questions:

Code snippets (own andexternal)

Black box questions

Questions about specificscenarios

Personalized exams take from 10to 20 minutes and can be donesimultaneously by many students.

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

6. Personalized exam (example question)

Figure: Personalized exams

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

Experience report

Students

Welcome feedback andbecome better in followingassignments

Automatizing allows tobetter understand standards

Instructors

We promised ourselvescomplete automatizing...really not possible

We have a lot of data... butmaybe not enough of theone we want!

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

Conclusions/summary

Not fully automatable, but scalable (continuous evaluationpossible)

Offers large and good feedback to students

Domain of the program is very important!

Group assessment could be easily introduced

Discussion: we have a wealth of data, but not always the onewe would really like

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics

Mining student repositories to gain learninganalytics

An experience report

Gregorio Robles, Jesus M. Gonzalez Barahona

{grex,jgb}@gsyc.urjc.esGSyC/LibreSoft, Universidad Rey Juan Carlos, Madrid, Spain

Berlin, Germany, March 14th, 2013

Gregorio Robles, Jesus M. Gonzalez Barahona Mining student repositories to gain learning analytics