Riding the tiger: dealing with complexity in the implementation of institutional strategy for learning analytics

Kevin Mayles, Head of Analytics, Open University

The Open University Mission

3

Student profile

Nearly 30% of new OU undergraduates are under 25

The average age of our new undergraduate students is 30

Only 9% of our new students are over 50

42% new undergraduates have 1 A-Level or lower on entry

Over 17,400 OU students have disabilities

11,000 OU students are studying at postgraduate level

p.5

A clear vision statement has been developed to galvanise effort across the institution on the focused use of analytics

Analytics for student success vision

VisionTo use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals

MissionThis needs to be achieved at :● a macro level to aggregate information about the student learning experience at an

institutional level to inform strategic priorities that will improve student retention and progression

● a micro level to use analytics to drive short, medium and long-term interventions

Vision in action

The OU recognises that three equally important strengths are required for the effective deployment of analytics

Underpinning organisational strengths

Adapted from Barton and Court (2012)

The OU recognised three equally important strengths are required for the effective deployment of analytics

Underpinning organisational strengths

We need to ensure we have the right architecture and processes for collecting the right data and making them accessible for analytics – we need a ‘big data’ mind-set

The OU recognised three equally important strengths are required for the effective deployment of analytics

Underpinning organisational strengths

The university needs world class capability in data science to continually mine the data and build rapid prototypes of simple tools, and a clear pipeline for the outputs to be mainstreamed into operations

The OU recognises that three equally important strengths are required for the effective deployment of analytics

Underpinning organisational strengths

Benefits will be realised through existing business processes impacting on

students directly and through enhancement of the student learning

experience – we will develop an ‘analytics mind-set’ in

these areas

For/in/on-action adapted from Schön (1987)

The OU is developing its capabilities in 10 key areas that

build the underpinning strengths required for the effective deployment of analytics

Analytics enhancement strategy

12

Development of predictive indicators

Application of a student number forecasting model to trigger interventions with vulnerable students

Calvert (2014)

13

Development of predictive indicators

The 30 variables identified associated with success vary in their importance at each milestone

Student

(Demographic)

Student – previous study/motivation

Student progress in previous OU

study

Student – moduleQualification /

module of study

Calvert (2014)

14

Application of predictive indicators

15

Application of predictive indicators

16

Application of predictive indicators

Technology showcase

4.30pm Thursday

17

Learning design link to success

Rienties et al (2015)

18

Learning design link to success

Rienties et al (2015)

Concurrent session 6C

2.45pm Thursday

19

Implementation at the OU

© Transport for London

p.20

The complexity challenge

What is project complexity?

● “Complicated”: e.g. a Swiss watch● “Complex”: from the Latin ‘complexus’ (braided together). Nonlinear and

unpredictable.●Like quality – it is hard to quantify and is something that is experienced

● Language: an analogy – not based in complexity science / complex adaptive systems theory

● Subjective not objective● Complexity is art not science

Maylor et al (2013)

p.21

Complexities

● Structural complexity●Number, size, financial scale, interdependencies, variety, pace, technology, breadth

of scope, number of specialties, multiple locations/time zones● Socio-political complexity●People, politics, stakeholder / sponsor commitment, resistance, shared

understanding, fit, hidden agendas, conflicting priorities, transparency● Emergent complexity●Technology and commercial maturity and novelty, clarity of vision / goals, clear

success criteria / benefits, previous experience, availability of information, unidentified stakeholders

● Assessed through the ‘Complexity Assessment Tool’

Maylor et al (2013)

p.22

How complex is the OU Analytics project?

Structural

Socio-politicalEmergent

OU Analytics Project Complexity

H

M

L

p.23

Responding to complexities

Complexity Response

Structural Socio-political Emergent

Plan and control

Plan comms (inc. clear visualisation); isolate key

tasks; create project board of stakeholders

Co-location; use PMO as point of control; scenario planning; change control

RelationalPrioritise communication with stakeholders; reach

out to others Socialise changes; revisit

assumptions; increase formal communication

Flexibility (Risk and change)

Anticipate refinement and testing; change

control; parallel developments

Manage expectations of change; revisit benefits regularly; ‘look-ahead’

with client

Maylor et al (2013)

p.24

Complexities faced at the OU

Structural Socio-political Emergent

Benefits - clarity

Unfamiliar technology

Supply chain not in place

Skills shortage

Integration of technical disciplines

Dependencies

Pace

Experience of staff

Culture change needed

Impact of organisational change

External stakeholder alignment and understanding

Benefits and success measures will become

clear

Technology will become familiar and change

Scope, schedule and resource availability likely

to change

Stakeholder engagement will improve

p.25

What have we done, what have we learned?

Structural Socio-political Emergent

Effective projectmanagement controls in

place

Agile method – early delivery and iterate

You can never do enough communicating

Revisited benefits regularly

Project board – wide representation – including

the doubters

High profile amongst senior leadership

Spend time on key (loud) stakeholders

Direct control of resources – small dedicated team

leading the way

Get small pilots going and people come on board

Change control – use it!

p.26

You cannot control the complexity…

Thank you…

Are there any questions?

For further details please contact:● Kevin Mayles – kevin.mayles@open.ac.uk● @kevinmayles

References:BARTON, D. and COURT, D., 2012. Making Advanced Analytics Work For You. Harvard business review, 90(10), pp. 78-83. CALVERT, C.E., 2014. Developing a model and applications for probabilities of student success: a case study of predictive analytics. Open Learning: The Journal of Open, Distance and e-Learning.MAYLOR, H.R., TURNER, N.W. and MURRAY-WEBSTER, R., 2013. How Hard Can It Be? Research Technology Management, 56(4), pp. 45-51. RIENTIES, B., TOETENEL, L. and BRYAN, A., 2015. “Scaling up” learning design: impact of learning design activities on LMS behaviour and performance. Proceedings of the 5th Learning Analytics and Knowledge Conference 2015.SCHÖN, D.A., 1987. Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco, CA, US: Jossey-Bass.