Vince Kellen - Tales from the Frontiers of Higher Education Analytics

38
Tales from the Frontier of Higher Education Analytics Vince Kellen, Ph.D. Senior Vice Provost Analytics and Technologies University of Kentucky [email protected] March 4 th , 2015

Transcript of Vince Kellen - Tales from the Frontiers of Higher Education Analytics

Tales from

the Frontier

of Higher

Education

Analytics

Vince Kellen, Ph.D.Senior Vice Provost

Analytics and Technologies

University of Kentucky

[email protected]

March 4th, 2015

Challenges to higher education

Questions

• What is the purpose of higher education?

– Job training and economic development? An informed citizenry and an antidote to tyranny? Create ‘good’

people? Increase knowledge in civilization? Provide a space for young adults to grow?

• What is the role of government in higher education?

– Is it a private good with access being managed by the market? Is it a public good with access provided by the

people for the people?

Facts

• Pay gaps are widening, skills are hollowing out

– The gap in economic prospects between those without college, those with an undergraduate degree and those

with advanced degrees are diverging.

– Growth in technology is placing new demands on job skills needed in the future

• The price of education, as felt by the public, is increasing

– Increased regulation and student support requirements, diminishing funds from government for education and

research, stagnant wage growth for middle-income families

– Some public colleges are considering ‘going private’ or are nearly there

2

Higher education has a ‘last mile’ problem

Education in any form is

struggling to address families

and communities with

economic and other readiness

problems

Free or low-cost educational

content does not easily solve

readiness problems which have

a multitude of causal factors

For profit models rightfully

struggle with ‘last-mile’

problems. Public policy

matters!

3

What would Abraham Lincoln think of eLearning?

Abraham Lincoln

• Autodidactic

• Books, books, books

• Became a skilled military strategist

• Penchant for poetry, Shakespeare,

politics and history

My nephew

• Not an autodidact

• Good worker, smart kid, but…

• It takes a village

• After a few low-security colleges

and much money borrowed

• He has found an intellectual home

• And a good occupation

4

Volume operations versus complex systems

Excluding the late 20th

century, universities

have been largely

complex systems,

delivering niche and

customizable

interactions F2F

settings. Large

lectures were added

to increase output

while reducing costs

Personalization

technology and e-

Learning approaches

can begin to handle

both high-volume and

specialty classes

5

Geoffrey Moore (2005). Dealing With Darwin

Given all the challenges in higher

education, nearly all roads to

improvement lead through

information technology

6

How technology can help in student success

7

High

effectiveness

Low

effectiveness

Low volume High volume

Small F2F class

Broadcast class

Current eLearning approach

PT + F2F

F2F = Face-to-face

PT = Personalization technology, adaptive learning technology

Some of our student analytic models

8

Model Description

Enrollment Enrollment in a class, midterm and final grades, credit hours attempted and earned, instructor teaching the class,

class level, room building, time and meeting pattern

Student retention and

graduation

Student demographics and cohort identification (e.g., John Doe is in the 2009 entering first-year student cohort),

fall-to-spring, 1, 2, 3, 4, 5 year retention rates; 4, 5, 6-year graduation rates, incremental retention rates

Student demographics Demographics, such as age, high school GPA, entrance test scores (SAT, ACT) and subcomponent scores. Also,

in a secure location, additional personally identifiable demographic details such as name, address, email, financial

aid and unmet need

Student performance Present the enrollment data in such a way as to easily show the student’s performance for each term, including

credit hours earned, term GPA, cumulative GPA for that term, etc., midterm and final grades, academic progress

Student academic career Keep a list of the majors and minors for each student and degrees awarded. Also, include details on students who

transfer in and out, including transfer institution, credit hours transferred in, etc.

Productivity The room utilization model contains every building, every potential classroom and lets users analyze the room

capacity and enrollments for the class or event in the room at five minute intervals, how far students walk to/from

classes. The faculty stats per term model pulls together the number of students and sections taught per term and

will contain other important data such as research expenditures per term and grant proposals submitted and won

Micro-surveys Capture questions and answers from the My UK Mobile micro-survey feature

Student involvement Interaction history with various student software including the learning management system, clickers, course

capture and playback, academic alerts, student organizations, advising interactions, mobile usage

Analytics at the University of Kentucky

Our goal was to utilize high speed analytics to help us improve student retention and

graduation rates

We had to alter our own thoughts about analytics, data warehouse construction and in-

memory computing. We spent one year in DENIAL about the capabilities of the tool

Our approach utilized ‘democratizing’ the data, engaging and fostering a community of

analysts, outside of central IT and institutional research

We developed a ‘single source of truth’

We focused on high-speed analytics to make the analyst experience enjoyable, allowing

aggregation at the highest level with drill-down into the lowest level quickly (within a

second)

After two years, we have collected nearly all data from our current systems and we expect

to gather the rest in the next year9

What does if feel like having high-

speed analytics for nearly all data

available that is easy-to-analyze,

easily sharable, quick to alter or

adjust and of sufficient quality?

10

11

Mobile & micro-surveys

We connected analytics to our

mobile app, envisioning

personalized messaging

We deliver simple 1-question

surveys with the ability to

trigger message responses

and interactions

In 17 months, we have gotten

162,891 surveys, averaging

about 45% response rates

12

13

Micro-surveys

14

Course “curves” (aka, Arnie’s S-Curve)

Size and growth of departments

15

Share of credit hours by section size

16

17

Iteratively query any/all fields of your

choosing, linking in an AND or OR

fashion

Combine different lists using SET

manipulations

Refresh lists regularly (nightly or

otherwise)

Apply the set name as a filter on ALL

models

This provides advanced filtering and

combining that works regardless of

the user interface

Our AA team can build and maintain

Lists easily. So can some users

Since lists are refreshed nightly, we

can keep track of each time a student

(or other entity) is added or removed

from a list

We can develop workflow apps using

this. Backend, front-end agnostic

List builder

List Builder

18

Found all students who take a lot of

classes at one point in their career

and then took less classes at

another point in their career.

Interpretation: These students start

with a bang but many fade at the

finish

How long did this analysis take?

Start to finish with this visualization:

25 minutes

Student readiness and frequency of the use of “The Study” and

2nd year retention rates

19

K-Feed

Intelligent,

personalized

alerts, news,

reminders

20

Targeted student interactions

In addition to our work on

difficult student cases, we

needed to find a way to reach a

‘murky middle’ group of

students

We identified students who are

just as likely to come back as

they are not

The predicted reenrollment was

about 50%

After interventions, the actual

enrollment was about 65%

Analytics in higher

education: Déjà vu?

The “long tail” concept applies

We have classes than enroll large

numbers of student, usually with

dozens of sections and frequently

with very large lectures

We also have many smaller

classes representing interesting

knowledge niches

These niches are one of the

reasons students globally come to

U.S. higher education

How can technology help students

and instructors in these classes?

22

http://www.thelongtail.com/conceptual.jpg

MOOCs

Large lectures

PHI 698

???

Taxonomy? Automatic metadata? Automatic atomic metadata?

Let learners more easily navigate an audio/visual

stream based on keywords

Let the system learn what are top terms. Let the

system map terms to concepts. Let instructional

designers lightly ‘bump’ the taxonomy, post

production

Record student engagement with and mastery over

specific terms / concepts

Deliver personalized messages to students,

detecting frustration, interest

Guide students with recommendations and

prompting, promoting active learning

23

See http://p.uky.edu

Can we help learners build proper mental models?

24

Natural language processing

(NLP) and neural-networks. Can

these classes of technology:

Extract concepts from audio

and text?

Find highly relevant articles,

scientific papers and

discussions regarding these

concepts?

Help learners construct and

validate their knowledge?

What will instructors do?

University of Kentucky

Personalized messages

in learning systems

Many classes integrate different

content within one experience

The following features are

becoming more common

• Use of video to replace or

augment lectures

• Backchannel ‘tweeting’ for live

student questions

• Use of mobile apps to answer

simple questions in class

• Students sharing notes, rating

content effectiveness

With high-speed analytics, we now

have a window into the mind of the

student, real-time

Analyze this clickstream to deliver

personalized messages

Confusion detector

In the middle of working online, a student searches for

“dorsolateral prefrontal cortex” in the online course content,

replays the associated segment in the video lecture and has

gotten 4 out of 5 related quiz questions incorrect.

1. Through content analysis, real-time analytics detect the

confusion regardless of the type of content and notifies the

peer tutor.

2. The real-time analytics service lets the student know that a

peer tutor is available and will be calling

3. The peer-tutor and the student discuss the concept

“dorsolateral prefrontal cortex” and arrange to meet later to

explain

4. The real-time analytics service places a entry in the

students advising log and notifies the faculty

RAMCaliper

“Uber Tutor”

Person-to-person

Confused student

Real-time analytics Peer tutor

How can we personalize? Let me count the ways…

General cognitive attributes, such as the composite ACT score or subcomponent math and verbal scores, high

school GPA, performance in prior classes

Economic attributes such as level of unmet financial need, level of tuition, prior tuition payment timeliness

Engagement levels such as the learner’s interactivity in the class relative to their peers

Non-cognitive factors, such as their level of conscientiousness, effort-regulation, planning, which affect

performance in classes

Class-level engagement and mastery including concept-level engagement and mastery, mastery over early

tests (including midterms) and participation in discussions, blogs, etc.

Co-curricular engagement including level of activity in student clubs, events on campus, advising services

Physical behavior, including how far they have to walk between classes, what dorm do they live in, etc.

Social network, including who else participates in clubs, discussions and classes with the student

General survey data, including mobile surveys

27

RAMCaliper: Universities defining real-time clickstream events

Purpose

• To allow the university to deliver real-time personalized messages in response to specific

learning system events to the learning system(s) environment(s) via a personalized messaging

region and to any other interaction channel the university utilizes including but not limited to

mobile applications, SMS messaging, email and delivery of messages in the university portal.

• To allow the university to capture data regarding student engagement with the learning

system(s) and add this data to a real-time analytic infrastructure.

• Partners: IMS Global, APLU PLC, about 15 universities

Specifics• API will need to send messages, ideally through web‐based modern streaming approaches. Specifically, messages should be available to

subscribers within 100‐1500 milliseconds, in what is commonly referred to as a publish/subscribe system. A web‐based RESTful "pull"

API, should be used to control the publish/subscribe system. The API need not make available the entire clickstream. It only needs to

make available the "terminal" action coinciding with completion of the important student and faculty events. As described, this results in a

few events per minute per user. This will allow institutional back‐end system to submit timely messages either to the learning system

environment or any other student interaction interface available in response to events originating in the learning platform. The events do

not need to persist indefinitely. Ideally, events will persist for 48 hours and will then be removed from originating streaming event system.

Our intent is to persist the messages within the institutional analytic infrastructure.

28

Is there an opportunity for a real-time event stream broker?

Technical value: Do we need a value-added network (a provider) to make it easy for everyone to connect?

Contractual value: Can this provider enforce contractual obligations regarding data privacy and sharing?

29

RAMCaliper

Hub

Vendor

B

Vendor

A

Vendor

D

Vendor

C

Institutions

The whole enchilada

Personalize learning, learning analytics and IPAS analytics

into one real-time architecture

• Real-time personalized interactions• Target on-demand peer tutoring based on student’s profile

• Deliver micro-surveys and assessments to capture additional information needed to improve personalization

• Give students academic health indicators that tell students where they can improve in study, engagement,

support, etc.

• Let students opt their parents in to this information so the family can support the student

• Tailor and target reminder services, avoid over messaging, enable timing of message delivery based on user

temporal proclivities, mix and match messages across learning, support and progression areas

• Allow for open personalized learning• How content gets matched to students is psychologically complex

• Several theories of how humans learn give many insights

• Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning, cognitive

reflection, need for sensation, need for cognition, various verbal abilities, confidence, persistence, prospective

memory, etc.

• We need an open architecture to promote rapid experimentation, testing and sharing of what works and what

doesn’tUniversity of Kentucky

What we will be working on this year: Five ‘Easy’ Use Cases

1. Simple, event-driven, real-time personalized messages in classes

• We give recommendations to learners in response to real-time comments, searches, and navigation through educational

content and online discussions, to their system of origination or any other channel to which we have access (e.g., mobile)

2. Guiding learners as they “map” out concepts in order to master them

• In the middle of students actively ‘drawing’ relationships between terms, we can reduce their search time for relevant

information while giving them feedback on how close or far they are from understanding the relationships between key

concepts in the class

3. Peer-to-peer ‘Uber’ tutoring service

• We can let the system detect student difficulties with specific skills in class and automatically pair them up with more

advanced students for just-in-time, mobile-phone assistance 7X24

4. Real-time performance and engagement analysis

• We can move beyond ‘point-in-time’ learning analytics and instead provide analysis of student engagement with and

mastery over critical class concepts across multiple interaction channels

5. Enhanced face-to-face advising interactions

• We can provide human experts with richer information about the learner to help the expert in their F2F advice

31

What will faculty do?

First, we can stop giving passive lectures, which

is educational malpractice

We can do more of the following:

• Monitor student engagement with the material

• Analyze student mastery in real-time

• Provide for more small group interactions

• Give academic career advice

• Connect students to other faculty and disciplines

• Engage students in research opportunities

• Instead of “sage on the stage” become “guide on

the side”

• Meet 1:1 to explain difficult material, probe the

student’s ability, give face-to-face feedback

• Motivate and inspire students

Return to our roots!

32

Organizing the IT unit

Our organizational model makes a

big difference. Other organizations

fail to take advantage of new tools

like this for mostly political reasons

Making key data transparent to all

does not help those who made their

living being the data ‘go to’ person

We had to merge two units, losing 1/3

of the staff. This let us hire three data

scientists with different analytic

backgrounds

The tool let the staff transition their

skills easily

33

Herding cats

We shared with everyone that we are

building the bridge as we walked on it

We established a community of practice and

rules of analysis etiquette

We built tailored objects for colleges, let

users choose their own front end tool

We relied on word-of-mouth adoption and

some teasing-revealing

Guess what happened?

34

Analytics Community of Practice (CoP) Principles

1) Be safe and secure. Respect the acceptable use of information policies and guidelines the university has in place. Please

have good passwords and secure your laptop, desktop and other devices appropriately. Treat private student and UK

information appropriately.

2) Be collegial. University data is a community asset and a community of people steward the data. Use and share the data with

the best interests of the university community in mind. Since parts of our data analysis environment is designed to allow for

greater transparency, analysis will potentially be able to see other unit data. While we will make private to a unit what

absolutely needs to be private, the way the university runs its business often involves multiple colleges and units at the same

time requiring broad data access. Don't use your access to take unfair advantage of another unit.

3) Help improve data quality. If you see data that doesn't appear to be correct, let someone know. We have a team of staff

dedicated to helping improve data quality. This team can work with colleges and units on any data entry and data

management processes that might need to be changed to improve data quality.

4) Be open-minded and inquisitive. Data can be represented in multiple ways at the same time. While the teams are taking

great care to enable multiple views of the data to support the community, you might have a valid and unique perspective. In

time, we can accommodate more ways of looking at the same data while not interfering with other views or taxonomies.

5) Share. The main benefit from open analytics is the power of a community of analysts learning from each other rather than a

few select individuals hoarding knowledge or access. As the community improves its knowledge and skill with the data, the

university can improve accordingly.

35

Analytic maturity scale: Rate your organization’s abilities

36

Top-down versus bottom-up

Doing this top down is like pushing water

uphill. Its harder than pushing a rock

uphill

The great leader is one who the people

say “We did this ourselves”

Consider analytics to be a process of

self-discovery. Each person has to go

through the stages of analytic maturity

Paradoxically, this also requires strong

top-down commitment and action!

Difficult organizational changes are

[often] required37

Questions?

38