CS-411 : Digital Education & Learning Analytics

ÉC OLE POL Y TEC H NIQ U EFÉ DÉRA LE D E LA USANNE

Pierre Dillenbourg, Patrick Jermann & Łukasz Kidziński

Chapter 3:

Massive Open Online Courses

The “tsunami” Massive Open Online Courses (2008)

Technology-enhanced learning (2004) Virtual Campus (2000)

Learning Management Systems (1999)

Virtual University (1999) Open Learning (1995)

e-Learning (1993) Online Education (1993)

Computer-Mediated Learning (1990)

Educational telematics (1988) Computer-Assisted Learning (1985)

Computer-Based Learning (1980) Computer-Assisted Instruction (1960)

4 millions exercises per day

George SIEMENS produced the word «MOOC » in 2008, while running a « connectivist » course

« Connectivism » •  Learning and knowledge rests in

diversity of opinions. •  Learning is a process of connecting

specialized nodes or information sources.

•  .Capacity to know more is more critical than what is currently known

•  Nurturing and maintaining connections is needed to facilitate continual learning.

•  Ability to see connections between fields, ideas, and concepts is a core skill.

•  Decision-making is itself a learning process.

C-MOOC

In 2011, Sebatian Thrun & Peter Norvig (Stanford) launched an AI course that attracted 160’000 participansts

X-MOOC

h2p://blog.pathgather.com/blog/

Financial Times

1.  Digital became natural: better devices, better

connectivity, social networks,…

2.  A link with top universities

3.  A rigid timing (to end soon)

4. The web business models

5.  The debt of US students (1,2 trillion)

6.  It’s not just on-line courses, but a deeper change

of the academic landscape

Why this hype ?

Will small universities disappear ?

HUMO

What is the future ?

The Guru The Veteran The Pessimist

2012

-01

2012

-05

2012

-07

2012

-08

2012

-09

2012

-10

2012

-11

2012

-12

2013

-01

2013

-02

2013

-03

2013

-04

2013

-05

2013

-06

2013

-07

2013

-08

2013

-09

2013

-10

2013

-11

2013

-12

2014

-01

2014

-02

2014

-03

2014

-04

2014

-05

2014

-06

2014

-07

2014

-08

2014

-09

2014

-10

2014

-11

2014

-12

2015

-01

2015

-02

2015

-03

2015

-04

2015

-05

2015

-06

2015

-07

2015

-08

2015

-09

0K

200K

400K

600K

800K

1000K

1200K

Accu

mul

ated

Num

ber o

f Reg

istra

tions

100,000

200,000

400,000

300,000

500,000

600,000

700,000

800,000

900,000 1,000,000

French MOOCs339,086

English MOOCs686,541

Accumulated Number of RegistrationsTimeline of the accumulated number of registrations for all courses combined

_Language

French MOOCsEnglish MOOCs

1,027,896 (sept 21 2015)

EPFL ?

Mécanique de Newton Jean-Philippe Ansermet Ready Mécanique du Point matériel Jean-Philippe Ansermet In production Launching New Ventures Chris Tucci Ready Plasma Physics and Applications Ambrogio Fasoli Ready SES Swiss-Energyscope Daniel Favrat Ready Fonctions Trigonométriques, Logarithmiques et Exponentielles Hans-Jorg Ruppen Ready Cellular Mechanisms of Brain Function Carl Petersen Ready L'Art des Structures II - Structures en treillis, poutres et cadres Olivier Burdet Ready Introduction à l'Astrophysique Frédéric Courbin Ready Planning and Design of Sanitation Systems and Technologies Christoph Luethi Ready Electrotechnique I Yves Perriard Ready Introduction to Household Water Treatment and Safe Storage Richard Johnston Ready Mécanique des Fluides Christophe Ancey Ready Introduction à la programmation orientée objet (en C++) Jean-Cédric Chappelier Ready Introduction à la programmation orientée objet (en Java) Jean-Cédric Chappelier Ready Electrotechnique II Yves Perriard Ready Villes Africaines - Introduction à la planification urbaine Jérôme Chenal Ready Éléments de Géomatique Pierre-Yves Gilliéron Ready Mécanique II Jean-Philippe Ansermet Ready Principles of Reactive Programming Martin Odersky Ready Comprendre les Microcontrôleurs Jean-Daniel Nicoud Ready

Neuronal Dynamics - Computational Neuroscience of Single Neurons Wulfram Gerstner Ready Mécanique I Jean-Philippe Ansermet Retired L'Art des Structures I - Câbles et arcs Olivier Burdet Ready Analyse numérique pour ingénieurs Marco Picasso Ready Digital Signal Processing Paolo Prandoni Ready Linear and Discrete Optimization Friedrich Eisenbrand Ready Functional Programming Principles in Scala Martin Odersky Ready Initiation à la programmation en C++ Jean-Cédric Chappelier Ready Initiation à la programmation en Java Jean-Cédric Chappelier Ready

h2p://moocs.epfl.ch/

Synchrotrons and Free electron lasers Leonid Rivkin In production Chimie générale Jacques-Edouard Moser In production MEMs & Nanofabriction Jürgen Brugger In production Credit risk Pierre Collin-Dufresne In production Global Macroeconomic Environment Luisa Lambertini In production Interest Rate Models Damir Filipovic In production Technologies for disaster risk reduction Silvia Hostettler In production Fundamentals of Biomedical Imaging Rolf Gruetter Planned Ecologies du Savoir Dominique Foray Planned Underground architecture Dominique Perrault In production Introduction aux Systèmes d’Information Géographique Stéphane Joost In production Villes africaines 3 - Agriculture urbaine Jérôme Chenal In production Solid Waste Management in Developing Countries Christian Zurbrügg In production Economie du sol et de l'immobilier Philippe Thalmann In production Water Quality and the Biogeochemical Engine Rizlan Bernier-Latmani In production Engineering Economy Weber In production History of Housing Luca Ortelli In production Management of Urban infrastructures Matthias Finger In production Matlab & octave for beginners Simone Deparis In production Space Mission design and operations Claude Nicolier In production Introduction aux Systèmes d’Information Géographique Stéphane Joost In production Villes Africaines II Jérôme Chenal In production Transmission Electron Microscopy for Materials Sciences Cécile Hébert In production Innovation and Techonology Development for Poverty Reduction Schoneberger In production Mobilité et Urbanisme Vincent Kaufmann In production Horizontal Metropolis - A radical project Paola Vignano In production Thermodynamique Jean-Philippe Ansermet In production Data analysis Heather Miller In production Parallel programming Aleksandar Prokopec In production Villes africaines 3 - Transport et Mobilité Jérôme Chenal In production Villes africaines 3 - Environnement urbain Jérôme Chenal In production Enseignes et afficheurs à LEDs Pierre-Yves Rochat In production Functional programming 2 Martin Odersky In production Gestion des aires protégées en Afrique Geoffroy Mauvais In production Villes africaines 3 - Quartiers Précaires Jérôme Chenal In production Mécanique du Solide indéformable Jean-Philippe Ansermet In production Mécanique Lagrangienne Jean-Philippe Ansermet In production Control Systems Colin Jones In production Electronique I Maher Kayal In production Functional programming 1 Martin Odersky In production Exploring Humans’ Space: An Introduction to Geographicity Jacques Levy In production Basic Steps in Magnetic Resonance Geoffrey Bodenhausen In production Introduction à la Sciences des Matériaux Michel Rappaz In production Algèbre Linéaire (Partie 1) Donna Testerman In production

1.  Share knowledge outside campus

2.  Increase EPFL visibility

3.  Better pedagogy on campus

4.  Support French speaking Africa

5.  Boost Continous training

6.  Generate revenues

Why does EPFL produce MOOCs ?

EPFL

World

EPFL

World

EPFL Africa World

USP BA Africa

WHY ? 1. Knowledge outside campus

2. EPFL visibility

3. Better pedagogy on campus

4. French speaking Africa

5. Continous training

6. Generating revenues

1. Knowledge outside campus

2. EPFL visibility

3. Better pedagogy on campus

4. French speaking Africa

5. Continous training

6. Generating revenues

Why does EPFL do MOOCs ?

MOOCs are not the « McDonaldisation » of European universities

Figure 17. Users’ level of engagement for each MOOC. The users are classified as actives (solved at least one exercise), viewers (watched a video, but didn’t solve any exercise), and inactives (never watched a video nor solved an exercise). The 21 bars on the left represent all registered users for which the data is available [N = 645,455]; the 2 bars on the right represent only those users who’ve paid $50 for a verified Coursera certificate [N = 2,847].

Figure 18. Effects of personal commitment at the time of registration [N = 159'287]. Asking users to commit to either audit a course or complete it increases the percentage of users who complete the course with a statement of achievement. Each point represents a MOOC session where this survey was done.

7.3. User Achievement One of the biggest factors that negatively affects MOOCs’ reputation among critics is the low percentage of registered users who complete the courses. The average figure for MOOCs in the world is somewhere below 10%, which is extremely low compared to courses in a traditional academic program. These figures are, however, misleading.

Figure 17. Users’ level of engagement for each MOOC. The users are classified as actives (solved at least one exercise), viewers (watched a video, but didn’t solve any exercise), and inactives (never watched a video nor solved an exercise). The 21 bars on the left represent all registered users for which the data is available [N = 645,455]; the 2 bars on the right represent only those users who’ve paid $50 for a verified Coursera certificate [N = 2,847].

Figure 18. Effects of personal commitment at the time of registration [N = 159'287]. Asking users to commit to either audit a course or complete it increases the percentage of users who complete the course with a statement of achievement. Each point represents a MOOC session where this survey was done.

7.3. User Achievement One of the biggest factors that negatively affects MOOCs’ reputation among critics is the low percentage of registered users who complete the courses. The average figure for MOOCs in the world is somewhere below 10%, which is extremely low compared to courses in a traditional academic program. These figures are, however, misleading.

Figure 19. Users’ level of achievement for each MOOC. The users are classified as having passed (obtained the minimum grade specified by the teacher) or failed (didn’t obtain the minimum grade). The top 21 bars represent all registered users for which the data is available [N = 655,453]; the middle 21 bars represent only those users who were classified as active [N = 120,985]; the bottom 2 bars represent only those users who’ve paid $50 for a verified Coursera certificate [N = 2,847].

7.4. Users in Developing Countries As mentioned earlier, most of our users come from countries in the western world, including Europe, North America, and Oceania. This would naturally lead us to assume that the results in Figure 17 and 19 are biased by the fact that western countries are highly literate (and relatively rich) compared to the rest of the world. This is a false assumption, as our data demonstrates. In fact, users in developing countries (particularly in Africa) tend to be as engaged in the course material as users from the rest of the world, and nearly as successful in completing the courses. Figures 20 and 21 illustrate this effect for a variety of African countries.

Figure 14. Percentage of users who are currently studying full­time, part­time, or not studying at all [N = 62,837]. The distribution is shown per MOOC by ascending order of non­students, as well as per MOOC language, level, and target audience.

Figure 15. Employment status of users who declared to be non­students [N = 41,485]. The distribution is shown per MOOC by ascending order of employed users, as well as per MOOC language, level, and target audience. Equally important to defining the strategy of content creation and promotion of EPFL’s MOOCs, particularly in the context of continuous education, both nationally and internationally, is understanding the background of our user base. Figure 16 illustrates the percentage of users who have studied or are currently working on a particular class of subject areas. Note that some users specify more than one area of expertise in the questionnaire, which is why the percentage blocks add up to more than 100%.

15

2/3 non-respondents who actually earned a certificate, resulting in 19%. Taking 57% as an upper bound, assuming perfect generalization from respondents to non-respondents, we can approximate that 19%-57% of participants plan on earning a certificate.

Teaching Teachers

The two survey items relating to teaching are legacy items were intended to capture the broadest sense of teacher and instructor identity:

Table 4 shows frequencies and percentages by category, excluding the relatively small unsure categories. We report a higher percentage of teachers than Seaton et al. (2015)—they find 28%, and we find 39%. Institution-level and course-level differences explain the discrepancy, where fully half (50%) of respondents from the 6 HarvardX courses identify as teachers or instructors, and the remaining 15 MITx courses have 28%, the same percentage as that reported by Seaton et al. (2015). One course at HarvardX has respondents who self-identify as teachers at a 67% rate; perhaps unsurprisingly, this is Leaders of Learning, a course from the Harvard Graduate School of Education that sought to have an impact not only on but through teachers. The other five HarvardX courses still have high teacher percentages from 41-47%.

Category Percentage # Participants % Explored % Certified Teacher (is or has been) 39% 33,228 42% 20% Not a teacher (has never been) 61% 51,127 34% 15% Teaches this topic 21% 6,122 46% 21% Teaches another topic 79% 22,915 43% 20% Responded 21% 84,355 38% 17% Did not respond or unsure 79% 310,485 13% 4% Surveyed 27% 394,840 19% 7% Not surveyed 73% 1,077,305 16% 8%

Table 4. Categories of response and nonresponse to a survey item about whether a participant is or has been a teacher, whether of the course topic or not. Participants in an additional 43 courses were not surveyed. If we hypothesize that an additional mechanism for MOOC impact is through teachers, it is teachers of the topic rather than teachers in general who are most likely to be able to use the content or pedagogy directly on their own students. Table 4 shows that 6,122 respondents identify as teachers of the topic across the 21 courses, a small percentage but a large number. Almost half of them explore and 20% are certified. As Seaton et al. (2015) argue, residential

Are you currently or have you ever identified yourself as an instructor/teacher? Yes/No/Unsure

Are you or have you ever taught material related to this course? Yes/No/Unsure

15

2/3 non-respondents who actually earned a certificate, resulting in 19%. Taking 57% as an upper bound, assuming perfect generalization from respondents to non-respondents, we can approximate that 19%-57% of participants plan on earning a certificate.

Teaching Teachers

The two survey items relating to teaching are legacy items were intended to capture the broadest sense of teacher and instructor identity:

Table 4 shows frequencies and percentages by category, excluding the relatively small unsure categories. We report a higher percentage of teachers than Seaton et al. (2015)—they find 28%, and we find 39%. Institution-level and course-level differences explain the discrepancy, where fully half (50%) of respondents from the 6 HarvardX courses identify as teachers or instructors, and the remaining 15 MITx courses have 28%, the same percentage as that reported by Seaton et al. (2015). One course at HarvardX has respondents who self-identify as teachers at a 67% rate; perhaps unsurprisingly, this is Leaders of Learning, a course from the Harvard Graduate School of Education that sought to have an impact not only on but through teachers. The other five HarvardX courses still have high teacher percentages from 41-47%.

Category Percentage # Participants % Explored % Certified Teacher (is or has been) 39% 33,228 42% 20% Not a teacher (has never been) 61% 51,127 34% 15% Teaches this topic 21% 6,122 46% 21% Teaches another topic 79% 22,915 43% 20% Responded 21% 84,355 38% 17% Did not respond or unsure 79% 310,485 13% 4% Surveyed 27% 394,840 19% 7% Not surveyed 73% 1,077,305 16% 8%

Table 4. Categories of response and nonresponse to a survey item about whether a participant is or has been a teacher, whether of the course topic or not. Participants in an additional 43 courses were not surveyed. If we hypothesize that an additional mechanism for MOOC impact is through teachers, it is teachers of the topic rather than teachers in general who are most likely to be able to use the content or pedagogy directly on their own students. Table 4 shows that 6,122 respondents identify as teachers of the topic across the 21 courses, a small percentage but a large number. Almost half of them explore and 20% are certified. As Seaton et al. (2015) argue, residential

Are you currently or have you ever identified yourself as an instructor/teacher? Yes/No/Unsure

Are you or have you ever taught material related to this course? Yes/No/Unsure

Knoweldge outside campus

EPFL visibility

Better pedagogy on campus

French speaking Africa

Continous training

Generating revenues

Why does EPFL do MOOCs ?

Knoweldge outside campus

EPFL visibility

? Better pedagogy on campus

French speaking Africa

Continous training

Generating revenues

Why does EPFL do MOOCs ?

« Flipped Class »

Standard Lecture

MOOC Physique Générale EPFL, J.-Ph. Ansermet

Introduction à la programmation, J.-C. Chappelier

Pierre

can you prove that MOOCs

are courses in lectures

rooms ?

Good MOOCs

are (in general) better than

bad MOOCS

What is a

« Good MOOC »

?

A MOOC

with rich activities

h2p://128.178.27.98:8082/LHE1.html

Fluid Dynamics (Gallaire & Ancey)

StaYcs (Mu2oni & Burdet)

Figure 3. Microcontrollers remote experiment. Users send their code to a system composed of two microcontrollers, which execute the code and send back a video of the output.

(4) http://128.178.27.98:8082/LHE1.html

3.5. Innovations in Evaluation Methods

One of the critical questions raised by the MOOC’s model is how to evaluate and grade thousands of assignments every week in a reliable manner. Quizzes and 1­answer exercises provide a fast (and subtle) way of testing users’ understanding of the course material. Automatically graded assignments, such as programming language assignments, which are typically graded by another program running on the back­end, are particularly useful and appreciated by users—especially when the system offers multiple attempts to submit the right answer. For more complex assignments, where human judgement is needed to grade the thousands of answers, and where quizzes and auto­graded exercises won’t do the trick, a more complex solution is needed. One such solution is called peer grading. In a peer grading assignments, the answers submitted by users are randomly distributed among other participating students, who then are required to grade each other. Peer grading turns out to be an effective way of grading users. Although common sense would say that teachers are better qualified to grade assignments, the data actually shows that the grades provided by peer users are well correlated to the grades provided by teachers, provided that the teacher has defined a clear grading rubric. This correlation is described in more detail in Section 9.

Watching MOOCs together

MOOCs are very social

Knoweldge outside campus

EPFL visibility

? Better pedagogy on campus

French speaking Africa

Continous training

Generating revenues

Why does EPFL do MOOCs ?

Teaching became a highsta ke activity

Teachers spend more time

preparing their course

Knoweldge outside campus

EPFL visibility

? Better pedagogy on campus

French speaking Africa

Continous training

Generating revenues

Why does EPFL do MOOCs ?

anticipated until the end of 2017 need an additional fund raising of approximately CHF 800,000. The total of CHF 4.3 million is dedicated to financing the capacity­building activities in higher education—described in more detail in Section 6.3. EPFL contributes with an additional CHF 2.7 million, covering essentially part of the cost of the manpower associated with the production of the MOOCs, as well as the salaries of the programme management team. Overall, the total budget allocated to the MOOCs for Africa programme, for the period between mid­2013 and the end of 2017, amounts to CHF 7 million. Impact

With the MOOCs for Africa programme, we expect to improve the quality of higher education in Africa (and emerging countries) in a significant and measurable way, while, at the same time, helping Africa address the crucial issue of the massification of higher education. Ultimately, we expect to promote the employability of graduates and the creation of new ventures—in particular, new technology­driven businesses.

6.2. Institutional Network In order to achieve our goals, we work with our existing partners from the RESCIF—out of which 4 are located in Africa. In addition, we’ve developed new partnerships with a range of carefully chosen African universities, in order to develop an offer of French­speaking (though not exclusively) MOOCs for the Bachelor, Master, and Continuing Education levels.

Figure 7. Map of African countries we are currently targeting.

MOOCs in Africa ?

63% of all African users are from North African countries (mostly Morocco, Algeria, Tunisia, and Egypt), followed by 23% from Western African countries (mostly Senegal, Cameroon, and Cote d'Ivoire).

Figure 5. Distribution of users across continents for each MOOC [N = 418,070]. The shades of blue represent countries in the western world (Europe, North America, and Oceania). The English and French flags indicate the instruction language of the course. The effective reach of each MOOC in terms of total number of registrations, illustrated in Figure 6, also reveals sharp differences between the subjects offered. By far, our most popular MOOCs are those created by Martin Odersky, “Functional Programming Principles in Scala” and “Principles of Reactive Programming,” which have been consistently successful over several sessions. The second most popular MOOC is “Digital Signal Processing” by Martin Vetterli and Paolo Prandoni, which has also been consistently successful in every session. English courses are in general more popular than French courses, accounting for 68% of all the registrations. In fact, our top 3 courses account for an overwhelming 56% of the total 787,320 registrations. In the French­speaking niche, programming courses are the most popular, accounting for 66% of the 225,478 users who registered for one of our French MOOCs.

Language Matters

Figure 4. Number of registrations in each country. The area of each bubble (equivalent to a log scale) is directly proportional to the number of localized users in that country [N = 418,070]. The Swiss flag bubble represents the number of registrations in Switzerland, and its size is scaled accordingly. Each bubble is centered on the average coordinate of all users in a country. Of course, not all MOOCs are created equal. Certain courses have a higher prevalence in some continents than others. Figure 5 shows the distribution of users across continents for each of the MOOCs listed in Table 1. As mentioned, our MOOCs are mostly taken by users from Europe and North America. However, different MOOCs tend to reach different “niche continents.” For instance, those offered in English language are widely popular in Asia, whereas those offered in French language are more popular in Africa. Users from North America also tend to register for our English courses (note the sharp reduction from English to French courses in Figure 5), whereas users from Europe tend to go for our French courses. This trend, however, is biased by the fact that almost 70% of these European users are from French­speaking countries (France, Switzerland, Belgium, Monaco, Luxembourg, and Andorra). In contrast, only 15% of European users who take our English courses are from French­speaking countries. Other interesting facts:

36% of all Asian users are from India, followed by 14% from China and an equal 14% from Russia.

49% of all South American users are from Brazil, followed by 14% from Colombia as a distant second.

Figure 8. Number of registrations per 1,000 internet users in each country. The radius of each bubble (equivalent to a linear scale) is directly proportional to the number of localized users in that country [N = 418,070]. The red bubble represents the number of registrations in Switzerland, and its radius is scaled accordingly. Each bubble is centered on the average coordinate of all users in a country. In addition to the countries that we are currently targeting, there are several other countries that show enormous potential, where a high number of internet users sign up for our MOOCs—examples such as Madagascar and Mauritius. The number of registrations per internet user in each African country is detailed in Figure 9.

Figure 20. Users’ level of engagement in African countries [N = 23,707] compared to the rest of the world [N = 342,877].

Figure 21. Active users’ level of achievement in African countries [N = 5,147] compared to the rest of the world [N = 91,414].

8. The EPFL MOOC Structures

8.1. Editorial Process

EPFL has set up the appropriate structures that allow us to shape and operate our MOOC strategy. These structures consist of: (1) a steering committee, who translates the EPFL board policy into strategic decisions, and (2) an editorial board, which selects the MOOC proposals submitted by teachers (twice a year) and who allocates the sources of funding accordingly. At the current rate of production, about 10­15 EPFL courses are offered as a MOOC every year. Indeed, our objective is not to “moocify” EPFL’s course catalog indiscriminately, but

!!! Raw numbers are low

Nan Li, Lukasz Kidzinski; 175,621 video sessions

An increase of 0.25 video speed results in an average decrease of perceived difficulty by 16% (p < .0001)

Learning Analytics

MOOCs in Corporate TRaining

Employees

Sales Network

Customers & Users

X-M

OO

Cs

vs C

-MO

OC

s

The Gartner Cycle of Technology Adop4on

Is this hand useful ?

Kshitij Sharma, Patrick Jermann, Pierre Dillenbourg EPFL Center for Digital Education

0.1 0.35

Kshitij Sharma, Patrick Jermann, Pierre Dillenbourg EPFL Center for Digital Education

« withmeness »