Data Wranglers: Human data interpreters to close the feedback loop

Doug Clow Institute of Educational Technology The Open University, UK

The Open University

• Largest university in the UK

• 200,000 students

• 7,000 tutors

• 1,000 academic staff

• Supported open learning

Milton Keynes

At scale, each year

• 600 courses• > 200,000 students

• > 1 million assignments• > 1 billion views of OU/BBC coproductions

• > 3 million Moodle transactions per day

Photo (cc) Marieke IJsendoorn-Kuijpers http://www.flickr.com/photos/mape_s/333862026//

data wrangling

“Learning is a complex social activity” (Siemens, 2012)

• Lots of data• Lots of tools

• Humans to make sense

Photo CC (BY) Tobias Lindman on Flickr: http://www.flickr.com/photos/cowb0y2000/6243008944/

Data Wrangling• Institutional capacity

building• 8 Wranglers• Faculty link• Regular reports

Data

• Delivery data • Completion,

pass rates, demographics

Pelophylax perezi Photo CC (BY-SA) Paulo Brandão on Flickr: http://www.flickr.com/photos/paulobrandao/3567716506/

• Student feedback • Activity data

(Moodle)

Photo CC (BY) Preneur d’Image on Flickr: http://www.flickr.com/photos/surbykids/9368298544/

Report Process

• Briefing discussion• Data analysis and report writing• Draft discussion• Final report

Photo (cc)-BY-NC nataliej on Flickr: http://www.flickr.com/photos/nataliejohnson/2122722198/

• Scrutiny in draft• Standard document template• Executive summary• Recommendations

Quality assurance

Responsive ‘quick reports’

Photo (CC)-BY Mark Dumont on Flickr: http://www.flickr.com/photos/wcdumonts/9256493975/

Doing extras

reaction

users data

users data

users data

examples

Unique visits per week to VLE components for one course

Course use of Elluminate Students using Elluminate

None 18%

Informal 27%

General student support 35%

Specified activities not assessed 49%

Specified activities referenced in assessment 95%

Usage of Elluminate broken down by course use of Elluminate for one Faculty between 2011-2012.

conclusions

Photo CC (BY-NC-SA) quas on Flickr http://www.flickr.com/photos/quas/1703493/

Evaluation:• Data quality• Data quantity• Unevenness

• High cost• Slow• Bottom up

Photo CC (BY-NC) Alexey Kljatov on Flickr http://www.flickr.com/photos/chaoticmind75/9516811453/

It is only through the detailed process of engagement and dialogue between analysts, stakeholders and the data that insight and organisational change are developed.

• Gill Kirkup, Mary Thorpe, Alison Ashby & her team– esp. Vicky Marsh and Jim Peard

• Oliver Millard• Evaghn DeSouza• My fellow Data Wranglers

Thanks to …

Doug Clow@dougclowdougclow.orgdoug.clow@open.ac.uk

This work is licensed under a Creative Commons Attribution 3.0 Unported License

cc licensed ( BY ) flickr photo by David Goehring: http://flickr.com/photos/carbonnyc/33413040/

Doing extras• 60 points vs 30 points

Photo (CC)-BY Steve Snodgrass on Flickr: http://www.flickr.com/photos/stevensnodgrass/4034636727/

cc licensed ( BY ) flickr photo by zzpza: http://www.flickr.com/photos/zzpza/3269784239/

Workshops

• Integrated/joint Learning Design/Data Wrangling• Module production workshops

– Intensive all-week workshop– Many presentations (LTS, other relevant modules)– IET: Data Wrangling and Learning Design

(CC) Eoin Gardner on Flickr http://www.flickr.com/photos/18091975@N00/5310045271/

The impersonal is political

• Data used as weaponry in ongoing battles

Data

• Delivery data • Completion,

pass rates, demographics

Photo CC (BY-NC) ChaoticMind75 on Flickr: http://www.flickr.com/photos/chaoticmind75/8629790711/

• Student feedback • Activity data

(Moodle)