Tracking and monitoring learners in MOOCs

Social surveys in EMMA

Chiara M Ferrari, Group Director, social research department at

Ipsos Public Affairs, Milanopart of the Ipsos Group

Participants Profiling: HOW IT WORKS

REGISTRATION FORM

Email __________Password_________Age__________Gender_________________Please complete your registration answering to a short questionnaire:

http://research.ipsosinteractive.com/mrIWeb/mrIWeb.dll?I.Project=S1036981&Id=101&i.User1=39253324&i.User2=CIBeCCBd

INDIVIDUAL Link matched to the users ID , which redirects to the

IPSOS platform

REGISTRATION FORM

Thank you!Your registration is completed.

The registration data are recorded in the EMMA SERVER . Each

student/batch of data gets an individual and unique ID which will

allow to connect the data from different sources: surveys, tracking and

learning analytics

Data on expectations will be stored in the IPSOS

SERVER and will receive the same individual and unique ID, to allow connection and

subsequent analysis

QUESTIONNAIRE ON EXPECTATIONS

•Email•Gender•Age•Employment status:

• Working full time• Working part time• Looking for a work• In full time education• Retired• Other “don’t work”

•Education Level: (ISCED LEVEL)

What is the highest level of education that you have completed?Level 0 - [Pre-primary education] Level 1 - [Primary education or first stage of basic education] Level 2 - [Lower secondary or second stage of basic education] Level 3 - [(Upper) secondary education] Level 4 - [Post-secondary non-tertiary education] Level 5 - [First stage of tertiary education] University/collegeLevel 6 - [Second stage of tertiary education] Doctorate level

• Country of birth/Country of residence• Mother tongue/language used in studies/language of

choice for EMMA

Participants Profiling: WHAT WE COLLECT

Expectation questionnaireoptional and one-off, a red toolbar which is there during the entire user navigation and reminds the student he /she should complete the survey

Specific questions about the EMMA offer

how did they hear about EMMA

what are the main reasons to participate, e.g..

• experience a MOOC

• look and experience the design of the MOOC

• gain knowledge in a new domain

• keep up to date with my domain

• want a certificate

personal learning objectives single course vs. wider path of studies (e.g. several courses in connection) vs. focus on a specific topic / theme vs. general info to gain general knowledge on a topic / theme vs. …

Previous experience with e-learning (as a learner)

How much time they expect to invest in this MOOC

In which of the learning activities they expect to take part

In which of the interactions they expect to participate

How much time they have available for this MOOC

From what location (home, university, work, library, ...)

On which support (PC, laptop, tablet, mobile…)

Expectations from EMMA users

Gender AVG 36.5

Age

Education

59% 41%

Occupation

SOCIODEMOGRAPHIC PROFILE

Was this in context of your formal education? Have you had any previous experience with e-learning?

What is your overall level of satisfaction with e-learning so far?

90% 10%

Very satisfied = 29%

E-learning experience

What are the reasons why you decided to register on EMMA?.

Reasons for visiting EMMA

Mini questionnaire -> optional one for each MOOC, it is entered freely by the teacher as a simple unit (see the picture below). Teachers can decide when and where they submit it

1414

OUNL_E-learningMOOC QUESTIONNAIRE REPORT Milan, March 2016

Ipsos – II round of MOOCs

© 2016 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may not be disclosed or reproduced without the prior written consent of Ipsos.

15

INDEX

Advice

Your experience with the MOOC E-learning

Tasks and assignments

Translation functionality

Your point of view

16

YOUR EXPERIENCE WITH THE MOOC E-LEARNING

The course in general

The audiovisual material

Recordings of expert interviews

Masterclasses on specific topics

Interaction with other learners

Interaction with teacher/tutor

Learning activities and assignments

The opportunity to keep up with the domain “e-learning” in a flexible manner

12%

38%

25%

12%

38%

38%

12%

25%

25%

38%

25%

38%

12%

12%

12%

12%

12%

12%

38%

50%

50%

50%

62%

62%

38%

50%

Strongly like Like Neutral Dislike Strongly dislike Didn't use/ does not apply

Base casi: totale rispondenti

Please rate from "strongly dislike" to "strongly like" how you would rate your experience with this MOOC

17

50%

38%

12%

Fully in English

Fully in Dutch

Partly in English and partly inDutch

TRANSLATION FUNCTIONALITY

Base casi: totale rispondenti

a) Let’s now discuss the translation functionality. Did you follow the course…?

b) How much do you use the translation functionality? I used it…

25%

75%

Always

Sometimes

Not at all

c) How much do you agree or disagree with the following statements about the translation functionality?

It’s effective

It needs more languages to be reallyeffective

It’s totally new and different

It enhances the overall value of theEMMA platform

It makes EMMA a truly Europeanexperience

50%

50%

50%

100%

50%

50%

50%

50%

50%

Completely agree

Fairly agree

Neither agree nor disagree

Fairly disagree

Completely disagree

18

TASKS AND ASSIGNMENTS

Base casi: totale rispondenti

How much do you agree or disagree with the following statements about the tasks and the assignments which were requested?

They are a good way to experience the course

They are too time consuming

I found them educational

They are engaging

They motivated me

They are a good way to create a collaborative activity

They are a good way to improve individual learning

I received the expected feedback

I liked receiving feedback from other participants

I liked giving feedback to other participants

If I was left behind with the tasks, I was still motivated to continue with the course

If I was left behind with the tasks, I wanted to quit the course

12%

12%

12%

12%

12%

12%

12%

25%

25%

38%

38%

38%

25%

50%

12%

12%

25%

12%

25%

12%

12%

12%

25%

12%25%

12%

38%

12%

12%

12%

12%

12%

25%

25%

12%

12%

12%

12%

12%

38%

25%

38%

38%

25%

25%

25%

62%

62%

50%

38%

38%

Completely agree Fairly agree Neither agree nor disagree Fairly disagree Completely disagree Does not apply

19

ADVICE

Verbatim

I engaged in this MOOC but there is no subtitle in English.

Part of the course was in Dutch and not in English as expected. The very first video about what is e-learning was in Dutch with no subtitles. There were too many videos in Dutch so even if you activate the subtitles, it is not easy to follow as there is too much text and what is being said is not well translated. There were some bugs with the activities so after the week 2/3, I just left it as for me, if the technology supporting the course is not reliable, then, it is a bit pointless. I guess that this platform needs a lot of improvements as well as the course design. From the moment you accept any participant, just make sure that all the material is accessible in proper English and that the activities can be made without any technical hassle. Overall, navigating into the platform is not intuitive, it takes too long. I also felt that there was too much text and that some resources was out of date. Another thing, the platform is not supported well by all the browsers

When I started the course, the automatic translation was not working well and some functions were not working well. This was commented by the participants, but the facilitators were not reaction on this in the first days. After two days it was still not working, I somehow stopped the course, because I simply forget to check in ....

cursus eigenlijk niet gevolgd (andere prioriteiten)

so bad because mi mother tongue is spanish for that reason i dont understood nothing

The course is not available after the end date which is a significant weakness for those pushed for time and who can\'t complete the course in the timescale advised. I wasn't able to complete the course and the platform doesn't tell you this.

Timing was niet ideaal. Start tijdens paasvakantie, daarna een weekje conge en dan was het moeilijk om nog op te volgen!

Een enkele expert zou moeten werken aan haar presentatietechniek. Zij komt ondanks haar hoog opleidingsniveau tamelijk moeizaam aan een vlotte redenering.

In your opinion, which are the weaker aspects of the course? Why?

20

YOUR POINT OF VIEW

Is there anything you would like to report or highlight, after having analyzed the data?

Should you re-run this same MOOC, is there anything you would do differently? Please explain

_____________________________________________________________________________________________________________________________________

_____________________________________________________________________________________________________________________________________

_____________________________________________________________________________________________________________________________________

_____________________________________________________________________________________________________________________________________

Exit questionnaire -> optional and and one-off, automatically embedded at the end of each slide in all units of the last lesson of each course

65% 66% 57% 61% 73% 71%

It's engaging It's immediate

It's self-explaining It's rich in offer

It offers cultural diversity It offers flexibility in learning paths

disagree9% 9% 22% 13%

7% 11%

The EMMA platform is … (% of agreement)

35%

46%

13%

5% 1% Extremely good

Fairly good

Neither good not bad

Fairly bad

Extremely bad

Evaluation of the experience

1%3% 3%3%

3%3%

4%

4%

4%

4%

6%

9%9%

11%

11%

21%

Sport

Arts

Education

Law and commerce

Marketing

Creative writing

Social Science

Sciences

Literature

Languages

Economy/Finances/taxes

Computer /web programming

Communication culture

History

Health/medical school

Food and wine

Areas of interest for the future

Learning Analytics in EMMA

Kairit Tammets, PhDSenior researcher, head of the studies

Centre for Educational TechnologySchool of Digital Technologies

Tallinn University

Why Learning Analytics?

• LA allows to understand:

– how MOOC participants learn, engage with the content and with the other participants;

– how MOOC designers and facilitators can improve their course designs and pedagogies

• LA was implemented in EMMA as an additional evaluation tool;

Why Learning Analytics? (2)

• Current approaches of learning analytics in MOOCs focus on:– Calculation of drop-out rates;– Prediction of drop-out;– Clustering on participants based on their online

behavior

• EMMA approach:– To establish feedback loop between facilitators and

participants in the form of dashboard;– To encourage evidence-based course design and

pedagogical interventions

Learning Analytics methodology

• What are the research questions?

• What data is needed for answering research questions?

• How this data can be collected?

• How to meaningfully present data?

• How to design feedback loop?

• How to learn from the process?

How Learning Analytics work?

• Tracking tool – tracks and collects users traces;

• Learning Record Store – stores learning experiences (xAPI statements) and sends them to dashboard application (Learning Locker);

• Dashboard – visualization of the interactions

How Learning Analytics work?

•

•

•

•

•

•

EMMA dashboards

• Learner view:

– Progress based on access of materials and submissions of assignments;

– Overview of usage of the learning resources;

– Social network analysis based on comments and posts;

– Activity stream;

– Participants – enrolled MOOC participants

EMMA dashboards

• Facilitator view:

– Overview of different interactions – accessed materials, time spent on activities, submissions and posts

– Overview of the usage of the learning resources;

– Social network analysis based on comments and posts;

– Activity stream

– Participants – enrolled MOOC participants

Addition to dashboard

• Additional analysis:

– Cluster of the participants:

• Enrolled;

• Observers;

• Contributors;

• Active

– In-depth overview of usage of the course content

– Additional inquiries by the MOOC facilitators

What next?

• Beyond the dashboards – what happens after using the dashboards, how MOOC designers improve their pedagogies, technologies and media

• Combining survey data and LA data;

• Translation of the numbers for contributing to European MOOC model