Learning Analytics & Educational Research

Leveraging Big Data in Powerful WaysAlyssa Friend Wise

Image Credit: Graham Cook via Flickr (CC BY 2.0), adapted

Associate Professor, Simon Fraser University Educational Technology & Learning Design

A F W 3 @ S F U .C A A L Y W I S E

Image Credit: Dakotilla via Flickr (CC BY 2.0)

Impassive clock! Terrifying, sinister god, Whose finger threatens us and says: Remember! …Three thousand six hundred times an hour, Second whispers: Remember!

(Baudelaire, 1857)

The clock is a powerful machinery that creates the product of “seconds” and “minutes” (Mumford, 1934)

Image Credit: Dakotilla via Flickr (CC BY 2.0)

The clock is a powerful machinery that creates the product of “seconds” and “minutes” and thus changes our relationship to time

Was a critical tool for navigation originally in calculating ship’s longitude, today for GPS

Is important in communication, scheduling connections of people within and across geographic space

Creates an awareness of allocation that can lead to greater efficiencies or value-based decision-making

(Mumford, 1934)

Time’s Quantification & Standardization

IF BEING ABLE TO TRACK (AND KEEP TRACK OF)

SOMETHING CHANGES OUR RELATIONSHIP

TO IT, WHAT IS IT THAT BIG DATA IS CHANGING OUR

RELATIONSHIP TO IN THE REALM OF EDUCATION?

Image Credit: UGA College of Agriculture and Environmental Sciences via Flickr (CC BY 2.0), adapted

Image Credit: UGA College of Agriculture and Environmental Sciences via Flickr (CC BY 2.0), adapted

BIG DATA IS CHANGING OUR RELATIONSHIP TO

THE PROCESS OF LEARNING

EPHEMERAL FLOW OF EXPERIENCE

GLOBAL “PROCESS” VARIABLES

ONLY RECENTLY HAVE TRUE PROCESS

DATA & ANALYSES BECOME

AVAILABLE

INNOVATION / CHARACTERISTIC OUTCOMESLEARNING

PROCESS

EMERGING FORMS OF PROCESS DATA IN

EDUCATION LOG-FILE DATA FROM ONLINE COURSES (INC. MOOCS)

CLICKSTREAM DATA FROM LMSS USED IN F2F COURSES

USAGE DATA FROM DIGITAL TEXTBOOKS AND NEW SPECIALIZED CONTENT BROWSERS (E.G. NSTUDY)

DATA FROM PHYSICAL SPACE (MOVEMENT, EYETRACKS…)

SELF-CONTRIBUTED DATA (POLLING, HABITS, GOALTRACKS)

VISIONS OF WHAT WE CAN DO WITH THIS

DATACONTRIBUTE TO BROAD BASED UNDERSTANDING OF LEARNING IDENTIFY GLOBAL PATTERNS ACROSS STUDENTS FIND MEANINGFUL DISTINCTIONS -> SUBSETS OF STUDENTS

INFORM US AT THE “FRONT LINES” OF HIGHER EDUCATION CREATE ACTIONABLE LOCAL INTELLIGENCE

FOR INSTRUCTORS TO PRACTICE RESPONSIVE TEACHING FOR STUDENTS TO BECOME ACTIVE AGENTS OF THEIR OWN LEARNING - “N = ME” (WINNE, IN PRESS)

L E A R N I N G A N A LY T I C S

LEARNING ANALYTICS

THE COLLECTION AND ANALYS IS OF DATA TRACES RELATED TO LEARNING IN ORDER TO INFORM AND IMPROVE THE PROCESS AND/OR ITS OUTCOMES

(S I EMENS ET AL . , 20 11)

00101110100101110101000110011010110

0101001010110100101100110010110

11010100011001010101001010011010

1011110110011011010010101001101011

001011101001011101010001100101101

0101001010110100101100110010

11010100011001010101001010

1011110110011011010010101100101010101

0010111010010111010100011

0101001010110100101100110

1101010001100101010100100

10111101100110110100101011001

001011101001011101010001100

01010010101101001011001101010

11010100011001010101001011

1011110110011011010010110

00101110100101110101000110101

0101001010110100101100110

110101000110010101010010111

1011110110011011010010101

1101010001100101010100101

001011101001011101010001100

01010010101101001011001101

11010100011001010101001010

1011110110011011010010101

0010111010010111010100011

010100101011010010110011001

1101010001100101010100101

10111101100110110100101011

10111101100110110100101011010100

00101110100101110101000110101101101

010100101011010010110011001101

1101010001100101010100101

0010111010010111010100010

1011110110011011010010101

01010010101101001011001101

0010111010010111010100011010

0010111010010111010100011

0010111010010111010100011010011010

001011101001011101010001101001001101

0101001010110100101100111

1101010001100101010100100

001011101001011101010001100

0010111010010111010100011

0101001010110100101101110001

00101110100101110101000110

0101001010110100101100110111001

0101001010110100101011001110110

11010100011001010101001010110010

110101000110010101010010110110101010

00101110100101110101000110011010110

010100101011010010110011001011

11010100011001010101001010011010

1011110110011011010010101001101011

001011101001011101010001100101101

0101001010110100101100110010

110101000110010101010010101

10111101100110110100101011010100

00101110100101110101000110101101101

010100101011010010110011001101

0010111010010111010100011010011010

001011101001011101010001101001001101

0101001010110100101100110111001

0101001010110100101011001110110

11010100011001010101001010110010

110101000110010101010010110110101010

10111101100110110100101001

001011101001011101010001100

01010010101101001011001101

11010100011001010101001010

1101010001100101010100101

10111101100110110100101011

Sub UniquePostsRead()

For k = 1 To MaxUser Step 1RowCount = Range("A1").CurrentRegion.Rows.CountFor w = 1 to MaxWeek Step 1

StartTime = Sheets("Week").Cells(w + 1, 2)EndTime = Sheets("Week").Cells(w + 1, 3) PostNum = 0PostsIndex = 0

Do While Cells(i, datestamp) <= EndTime And i <= RowCountIf Cells(i, Source) = “Read" Then

If Cells(i, Message_Author) <> Val(ActiveSheet.Name) And Cells(i, Scan) <> "X" Then

flag = 0For j = 1 To PostsIndex Step 1

If Posts(j) = Cells(i, Message_Id) Then flag = 1 j = PostsIndexEnd If

Next jIf flag = 0 Then PostsIndex = PostsIndex + 1

Posts(PostsIndex) = Cells(i, Message_Id)End If

End IfEnd If

Sheets(“Stats").Cells(Line, 22) = PostsIndexNext w

Next k

End Sub

PercentPostsRead =SUniquePostsRead TotalPostNumber

Image Credit: Joshua Rothass via Flickr (CC BY 2.0), adapted

Image Credit: US Department of Education via Flickr (CC BY 2.0), adapted

LEARNING ANALYTICS

USES COMPUTATIONAL METHODS TO GENERATE INS IGHT INTO LEARNI NG PROCESSES THAT CAN BE USED TO INFORM HUMAN-DECIS ION MAKING WHILE LEARNING EVENTS ARE ST ILL IN PROCESS

(WISE , I N PREPA RATI ON)

HOW DO WE HELP LEARNING ANALYTICS

BE AN INNOVATION THAT MAKES A REAL

IMPACT ON TEACHING AND

LEARNING ?and maybe

even

revolutionizes

higher education!

WE NEED TO DESIGN FOR WAYS IN WHICH

ANALYTICS CAN USEFULLY

REFLECT & INFORM THE TEACHING AND

LEARNING PRACTICES OF INSTRUCTORS AND

STUDENTS

PART 1: WORKING WITH INDICATORS

THAT MEANINGFUL REFLECT LEARNING

PROCESSES

HOW DO WE DEVELOP

RICH INDICATORS THAT CAN BE MEANINGFUL

TO TEACHERS AND STUDENTS AS

REFLECTIONS OF THEIR PARTICULAR

PRACTICES OF TEACHING AND

LEARNING?

WHAT’S THE LEARNING MODEL?

Image Credit: Torley via Flickr (CC BY 2.0)

MORE IS

BETTER

Image Credit: Torley via Flickr (CC BY 2.0)

WE CAN DO

BETTER!

Image Credit: Torley via Flickr (CC BY 2.0)

ONLINE DISCUSSION LEARNING MODEL

Externalizing one’s ideas by contributing

posts to an online discussion

Taking in the externalizations of others by accessing

existing posts

• Social constructivist perspective - online discussions as a forum for learning through conversation

• Students learn as they articulate their ideas, are exposed to the ideas of others, and negotiate differences in perspective

• Focus on how students contribute comments (“speak”) and attend to other’s messages (“listen”)

Speaking Mechanism for sharing ideas

Value in speaking that is Recurring, responsive , rationaled Distributed temporally and

conversationally Moderately portioned

While “speaking” is visible, not all qualities are salient in the system (esp. as related to time)

Post quality info valuable, but complex to assess

Listening Attending to the ideas of others is critical,

but “invisible”

Value in listening that is Broad yet Deep (to consider multiple

ideas; predicts posts’ content quality) Integrated (so comments are informed by

others’ views) Recurrent (to provide context for

discussion flow; predicts responsiveness)

Early research suggested universally poor behaviors, but recent work shows students listen in very distinct ways E.g. Disregardful, Coverage, Focused,

Thorough

ONLINE DISCUSSIONLEARNING MODEL

ONLINE DISCUSSIONLEARNING MODEL ANALYTICS

Criteria Metric Definition

Temporal Distribution

Percent of sessions with posts

Number of sessions in which a student made a post, divided by their total of number sessions

Speaking Quantity

Number of posts Total number of posts a student contributed to the discussion

Average post length Total number of words posted by a student divided by the number of posts they made to the discussion

Listening Breadth

Percent of posts viewed

Number of unique posts that a student viewed divided by the total number of posts in the discussion

Percent of posts read Number of unique posts that a student read divided by the total number of posts in the discussion

Listening Recurrance

Number of reviews of own / others’ posts

Number of times a student revisited posts that they had made / viewed previously in the discussion

Conversational Distribution

Posts made / viewed throughout discussion

Dispersion or concentration of posts made / viewed by a student in the discussion space

ONLINE DISCUSSIONLEARNING MODEL ANALYTICS

PART 2: USING DATA TRACES TO INFORM OUR

TEACHING AND LEARNING ACTIVITIES

HOW DO WE CONSIDER AND DESIGN FOR WAYS

IN WHICH ANALYTICS CAN PLAY A PART IN

THE LARGER

ACTIVITY PATTERNS OF

INSTRUCTORS AND STUDENTS?

A M O D E L F O R T E A C H E R S –

C O N N E C T T H E U S E O FL E A R N I N G A N A LY T I C S T O T H E P R A C T I C E O F

L E A R N I N G D E S I G N

(LOCKYER, HEATHCOTE, & DAWSON, 20 13)

KEY CONCEPTUAL QUESTIONS

1. W H AT A R E T H E G O A L S O F T H E E D U C AT I O N A L A C T I V I T Y ? [ W H AT I S T H E P O I N T ? ]

2. W H AT D O P R O D U C T I V E & U N P R O D U C T I V E L E A R N I N G P R O C E S S E S T O M E E T T H E S E G O A L S L O O K L I K E ? [ W H AT I S T H E P R O C E S S ? ]

3. H O W C A N T H E AV A I L A B L E A N A LY T I C S S E R V E A S I N D I C AT O R S O F T H E S E ?

[ W H AT I S T H E P R O X Y ? ]

• Purpose of engaging in [online discussions]

• Expectations for a productive process of engaging in [online discussions]

• How the learning analytics provide a proxy for [this]

L INKING LEARNING ANALYTICS & LEARNING DESIGN

articulating one’s ideas, being exposed to the ideas of others, negotiating differences in perspective

attending deeply to a spectrum of others’ ideas, and contributing comments that are responsive and rationaled,

percent of posts read introduced is a metric that has clear meaning in the context of the activity

Metric Student 1(Week X)

Student 2(Week X)

Class Average (Week X)

Range of participation 2 days 6 days 5 days

# of sessions 8 3 11

Average session length 13 min 48 min 39 min

% of sessions with posts 35% 67% 49%

# of posts made 9 4 7

Average post length 126 words 386 words 216 words

% of posts read 42% 87% 75%

#of reviews of own posts 2 22 13

#of reviews of others’ posts 3 12 8

ONLINE DISCUSSIONLEARNING ANALYTICS

ONLINE DISCUSSIONLEARNING ANALYTICS

A M O D E L F O R S T U D E N T S

– C O N N E C T T H E U S E O FL E A R N I N G A N A LY T I C S T O T H E P R A C T I C E S O F

S E L F - R E G U L AT E D L E A R N I N G (WISE , 2 014 )

WHY FOCUS ON STUDENTS AS USERS OF LEARNI NG ANALYTICS?

E N G A G E T H E M A S A C T I V E PA R T N E R S I N L E A R N I N G

A B I L I T Y TO M A K E I M M E D I AT E L O C A L C H A N G E S

A C T I VAT E M E TA C O G N I T I V E P R O C E S S E S

E M P O W E R M E N T N O T E N S L AV E M E N T

D E M O C R AT I Z E A C C E S S TO D ATA

O N E - TO - O N E R AT I O AT A N Y S C A L E

STUDENT TUNING MODEL

ALIGN DESIGN FRAMEWORK

Integration (technological and pedagogical) made analytics a coherent part of the learning process

Students embraced agency in setting (often recurring) personal goals and evaluating their progress, no “big brother” issues

Individual, peer, and instructor reference frames were important for making sense of the data; reactions were both cognitive and emotional

Reflection on data a powerful starting placeConcrete and proximal goal-setting is harderChange happens slowly, isn’t always intentional, requires support!

STUDENT ANALYTICS USE INIT IAL F INDINGS ( ! )

DIST INC T CHANGE PROFILES

HOW CAN YOU USE ANALYTIC DATA TO

USEFULLY REFLECT & INFORM

YOUR TEACHING AND THE

LEARNING PRACTICES OF YOUR STUDENTS

?

Image Credit: enjosmith via Flickr (CC BY 2.0), adapted

Learning Analytics & Educational Research

Leveraging Big Data in Powerful Ways

A LY S S A F R I E N D W I S ES I M O N F R A S E R U N I V E R S I T Y

A F W 3 @ S F U .C A A L Y W I S E