University of Toronto November 2014design. How UofT instructors think about their own teaching at...

Running Head: University of Toronto Faculty Members’ Experiences 1

University of Toronto Faculty Members’ Experiences with Developing MOOCs

Authors’ names withheld

University of Toronto

November 2014

DRAFT IN PROGRESS. PLEASE DO NOT CITE

University of Toronto Faculty Experiences


2

Introduction

This paper presents the findings of a study that investigated the experiences of University

of Toronto’s faculty members and instructors who have offered MOOCs on Coursra or edX

between 2012 and 2014. In July 2012, the University of Toronto initiated an institutional

partnership with Coursera and agreed to deliver a number of courses on their new platform.

Coursera (coursera.org) is a commercial company located in the US that has developed a web-

based environment for large open courses (known as MOOCs – Massive Open Online Courses)

for informal and continuing non-degree learning. As of February 2013, the University of Toronto

also commenced an institutional partnership with EdX (edx.org), a non-profit organization also

supporting development and instruction of large open courses. The new MOOC activity occurred

under the umbrella of Open UToronto, a broader institutional initiative established in the spring

of 2012 to promote discovery, use, creation and sharing of open access literature, shared content

and educational resources, as well as open courses.

One of the chief motivations for the University of Toronto to enter into partnership with

Coursera and EdX, and invest in course development, is to strengthen UofT’s role as a Canadian

leader in higher education pedagogy, and better understand the affordances and challenges of

organizing courses on a large, international scale. In addition to previous research carried out

related to the University of Toronto MOOC offerings (Campbell, Horton, Craig, & Gries, 2014;

Gibbs, 2014; Restoule, 2013; Soman, Evans, & Federico, 2013), the current study focuses on

expanding this inquiry by exploring the experiences of instructors participating in the design and

delivery of MOOCs.



3

The overarching objective of this study was to explore UofT faculty members and

instructors' experience of MOOC development and delivery. Faculty members' experience

producing online courses has been studied before, both from the perspectives of faculty members

(Haklev, 2011), and from the perspectives of their supporting instructional designers (Power,

2009). While descriptive self reports about MOOC design process and faculty members'

experience have been published (Robinson, & Ash, 2014; Severance, 2013) there is a paucity of

systematic analysis of faculty members’ experiences with creating MOOCs, as highlighted in a

review of MOOC literature published between 2008 and 2012 (Liyanagunawardena, Adams, &

Williams, 2013). While contributing to the growing body of literature on MOOC design,

development, and offering, results of the present research also help UofT better support faculty

members involved in the pedagogical design process of MOOCs, or similar courses in the future.

The purposes of this study were to investigate:

The motivations as to why UofT instructors chose to participate in these MOOC projects.

How UofT instructors understood and conceptualized MOOC development projects.

The processes underlying conception to completion, including the design, development,

and the actual delivery of the MOOCs.

How the intense course-design process in a new medium, with the help of a

multidisciplinary team, influenced UofT instructors' ideas about pedagogy and learning

design.

How UofT instructors think about their own teaching at the University of Toronto based

on their MOOC experiences.



4

Review of Related Research

By 2014, more than 90 universities from across the globe partnered with Coursera and

Edx to offer just above 1000 MOOCs. With various formats of MOOCs being conceptualized,

including "on demand" and "signature track", examining faculty members' experience with the

design and delivery of MOOCs and the implications of their experiences on their teaching, is

timely. A deeper understanding of faculty members' experiences and the potential challenges they

face during the MOOC design and development process can inform universities in developing

relevant instructional design support, an issue that has been the topic of formal online education

research (Tallent-Runnels et al., 2006). Empirical literature on faculty members and instructors’

MOOC design and delivery experience is still in its infancy. Thus, we also drew on online

learning literature to understand outstanding issues that faculty members recount regarding their

motivation to teach in a different context–face to face vs. online –, instructional redesign, and

pedagogical implications.

Motivation to Teach in a New Context

Faculty members could be personally driven, or institutionally encouraged to engage in

online instruction (Power, 2009). Personal interest in teaching with technology and treating

online teaching as a professional challenge, were among intrinsic motivators for faculty members

engaging in online instruction in Maguire's (2005) review of thirteen studies. The potential for

increased student access to education was another motivator for faculty members' participation in

online instruction. Haklev (2011) interviewed 5 faculty members and two administrative staff in

two Chinese universities that participated in the Top Level Quality Project that particularly

focused on improving undergraduate level education through engaging faculty members in



5

designing Open Educational Resources. Motivation to apply for the program varied among these

faculty members and included the fit between their course and application requirements, more

institutional recognition, and using the application process to improve course design and student

learning. An institutional benefit is inherent in these cases.

Within the context of credit courses, institutional requirements and mandates may play an

important role in motivating faculty members to participate in course redesign and online

teaching. Motivation for designing and teaching MOOCs might be different, nevertheless. In a

MOOC context, providing learning opportunities on a global scale, with no entry barrier, is one

motivating factor for many faculty members. Johns Hopkins Bloomberg School of Public Health,

for instance, offered public health related MOOCs to achieve this purpose (Gooding, Klaas,

Yager, & Kanchanaraksa, 2013). Similarly, a MOOC focused on dementia provided a research

center with the opportunity to impact learners' understanding of the disease informed by the

center's expertise (Kelder, King, Carew, & O’Reilly, 2013). Holland and Tirthali (2014)

interviewed 83 individuals, including administrators, faculty members, and researchers, from

universities and research centers active in MOOC initiatives to explore institutional goals for

engaging in such initiatives. Increasing global access to education through an outreach beyond

physical and geographical boundaries, in general, and enabling broad access to unique faculty

expertise, in particular, were among stated goals for deciding to offer or use MOOCs. It is

important to note that Holland and Tirthali (2014) do not distinguish between institutional

objectives and faculty members' motivations.



6

Instructional Design Considerations

Teaching in an online environment requires university faculty members to develop new

pedagogical practices and technological skills as lack of such skills or limited experience with

online instruction could be inhibiting factors for offering online courses (Bruner, 2007; Maguire,

2005; Nelson, & Thompson, 2005; Shea, 2007). In their review of nine qualitative papers,

DeGagne and Walters (2009) note that technological concerns may overshadow pedagogical ones

as faculty members with less technological competence may rely on support staff to build their

courses.Therefore, institutional support for faculty members to develop their technological

competencies; learn about instructional design and pedagogical approaches conducive to online

learning; and master the platform where they will teach their online courses, would facilitate the

shift from face-to-face to online instruction. Institutional support has been identified as a

motivator for university faculty members to engage in teach online courses (Tabata, & Johnsrud,

2008).

Instructional design frameworks that promote a systematic design approach guided by

desired learning outcomes could facilitate the design of online courses. Understanding by Design

(Wiggins, & McTighe, 2005), Integrated Course Design (Fink, 2013), and Horizontal Course

Syllabus Grid (Power, 2009) are three well-documented frameworks. Examples of design

frameworks specifically developed for MOOCs also exist. MOOC Canvas (Alario-Hoyos, Pérez-

Sanagustín, Cormier, & Kloos, 2014) is a visual and participatory framework that guides the

instructional team through MOOC design in two high level steps: 1) available resources and 2)

design decisions. Several design issues in the form of questions are embedded in each of the

steps. Providing the design team with a shared representation of design decisions regarding

pedagogical, technological, and logistical aspects of learning design is noted as one advantage of



7

MOOC Canvas. In the first step, the instructional team assesses and identifies available human,

intellectual, production equipment, and MOOC platform resources. Informed by the available

resources, the team then defines the focus of the MOOC and its target audience, decides on

pedagogical approaches and learning objectives, produces content and designs assessment, and

decides on using complementary technology tools external to the selected MOOC platform.

A team based approach (Hixson, 2008) increasingly characterizes instructional support for

online learning where, in addition to instructional designers and technological support staff,

sometimes graduate students collaborate with faculty members to redesign their courses (Alvarez,

Blair, Monske, & Wolf, 2005; Koehler, Mishra, Hershey, & Peruski, 2004). A team approach to

course design and sharing of course material may spark opportunities for peer review and

reflection among faculty members that are not possible when instructors work on their own

(Haklev, 2011). MOOCs, too, are mostly developed by teams consisting of faculty members,

administrators, technological support staff, and instructional designers (Alario-Hoyos et al., 2013;

Holland, & Trithali, 2014). In some cases, MOOCs are co-designed and co-taught by several

faculty members (Arnold, Kumar, Thilosen, & Ebner, 2014). Opportunities for reflection may

need to be explicitly integrated in the design process as faculty members may design parts of a

MOOC individually and with little interaction with each other due to practicality issues. Five

faculty members who co-taught a MOOC shared their concerns about inconsistencies within the

teaching team in managing social interactions across different modules each led by an individual

instructor (Arnold et al., 2014).

Teaching online credit courses and MOOCs also requires detailed instructional planning

before the course commences, as well as a continuous presence to answer students' questions and

to provide feedback while the course is underway (Arnold et al., 2014; Conceicao, 2006; De



8

Gangne, & Walters, 2009; Mitchell, 2009; Power, 2009). The anticipation of increased upfront

workload during course redesign may also discourage faculty members new to online instruction

(Maguire, 2005; Nelson, & Thompson, 2005; Shea, 2007). As for MOOCs, time commitment and

resource allocation have been reported to be higher than credit courses and would manifest in two

phases: First, when faculty members develop MOOCs and create learning and assessment

components almost from scratch; and later, when they attempt to re-design their for-credit

courses (Holland, & Trithali, 2014).

Implications of Online Instruction

Designing and teaching online courses may affect faculty members’ perception of their

role and how they approach teaching and learning in their face-to-face classes. In a review of nine

qualitative papers, DeGagne and Walters (2009) found that faculty members who had

transitioned their instruction to an online context experienced a qualitative change in their role as

they moved away from knowledge dissemination towards facilitating a learning community

among students. Ten faculty members with varied levels of online teaching experience who

participated in a phenomenological study (Conceicao, 2006) identified four facets to their

student-centered online teaching approaches: instructional designer, facilitator, learner, and

motivator of discussions.

Online teaching allows faculty members to use technology to revise their face-to-face

instruction (Maguire, 2005). The 913 faculty members who participated in Shea, Picket, and Li’s

(2005) study, filled out a survey that probed, in part, the implication of online course design and

teaching on their pedagogical assumptions and classroom teaching. Results of a regression

analysis suggested opportunities for learning, including alternate means of instruction and



9

assessment, as factors significantly related to participants' satisfaction with online instruction.

Opportunities to use new technologies and develop new competencies were also among the

motivating factors for 142 full-time and sessional faculty members and instructors who

participated in a study that investigated motivators and incentives for teaching online courses

(Chapman, 2011). Also, a three-round action research study following seven faculty member new

to online teaching as they received professional development and designed online courses,

suggested that opportunities for reflection could encourage them to consider alternate views of

teaching and learning (McQuiggan, 2012). Evidence for the positive impact of online course

design on faculty members' teaching has been reported outside North America as well (Haklev,

2011). In this qualitative study faculty members reported improvements in their pedagogical

knowledge, knowledge of course design, and quality of their instruction. For example, they

redesigned their lectures to include discussion, allowed students to watch lectures when needed,

or added social media components to their courses to communicate with students.

University faculty members' self-reports and observations by researchers and

administrators suggest that developing MOOCs could encourage the revision of pedagogical

practices and lead to course re-design. Course re-design may entail transforming a course in to a

flipped format using MOOC material or, alternatively, integrating frequent feedback, discussion,

and peer assessment within the curriculum. Yet rigorous research, beyond anecdotal accounts, is

lacking evidence on improved student learning outcomes as a result of such transformation

(Holland, & Tirthali, 2014).

While faculty members' experience with designing and teaching online credit courses is

well-documented, similar studies with regard to MOOCs as non-credit courses offered to a

broader student audience, are sparse in the literature. Informed by existing research, in this study



10

we examined UofT faculty members' experience with developing and teaching MOOCs on

Coursera and edX. The following research questions guided this study:

What motivated UofT instructors to offer MOOCs?

How did UofT faculty instructors design, develop, and deliver their MOOCs?

What are the implications of MOOC instruction for faculty members as researchers and

instructors?

Method

Research Context

As of September 2014, UofT faculty members and instructors, referred to as instructors

here, from various departments and schools have taught 12 distinct MOOCs on Coursera and

edX. A number of these MOOCs have been offered more than once. Detailed information about

UofT MOOCs can be found in first year and second year reports of UofT MOOC activity

(http://www.ocw.utoronto.ca/open-utoronto-mooc-initiative/).

Several classifications of MOOCs have been proposed to distinguish among courses that

fall under the general description of being massive, online, and open. Such classifications are

based on the underlying learning theory, pedagogical focus, and the degree of openness in

instructional design elements (Rosselle, Caron, & Heutte, 2014). The distinction between

cMOOCs and xMOOCs, reflecting connectivist and cognitive behaviourist theories of learning,

respectively, recurs more frequently in MOOC literature (Rodriguez, 2012) . xMOOCs follow the

structure and instructional design approach of large university lectures with the instructor

defining and designing learning goals, learning activities, content, and assessment. cMOOCs, on

http://www.ocw.utoronto.ca/open-utoronto-mooc-initiative/



11

the other hand, de-emphasize the central role of the instructor, and hold learners responsible to

define their learning goals and form networks of peers and experts to construct and share their

knowledge. Here, we use this distinction to characterize the MOOCs UofT faculty members and

instructors who participated in this study offered between 2012 and 2014. UofT faculty members

and instructors identified learning outcomes, created multimedia content, and designed learning

activities and assessments for their MOOCs in advance, making their Massive Open Online

Courses fall under the category of xMOOCs.

Participants

A call for voluntary participation was sent to all University of Toronto continuing

appointment instructors who had offered MOOCs on either Coursera or edX. Out of twelve, eight

instructors agreed to participate in this exploratory study. Participants of this study were

primarily award-winning, tenure stream or teaching stream instructors. Six participants had

received teaching awards and the other two were recipients of research awards. The identity of

the participating instructors is kept strictly confidential and, thus, no reference is made in this

paper to their name, MOOC title and platform, exact time period of MOOC offering, or distinct

research project or dissemination that would reveal the identity of the participants. In the rest of

the paper, "UofT MOOCs" refers to the MOOCs whose instructors participated in this study,

unless otherwise stated. We also use the terms "instructor", "MOOC instructor", or "UofT

MOOC instructor" interchangeably to refer to our research participants. The term "learner" refers

to individuals who have registered for a University of Toronto MOOC.



12

MOOC Instructional Design Support at UofT

Since 2012, the Online Learning Strategies (OLS) portfolio at UofT has been providing

instructional support for MOOC development. More specifically, OLS supports MOOC

instructors in the following ways:

Advising on evolving institutional strategies and new developments

Liaising with senior leadership on MOOC-related initiatives and funded projects

Delineating institutional MOOC development workflow

Outlining critical success factors and consulting on resourcing implications

Conducting team-based MOOC design workshops, offered in 2013 and 2014 to the

second and third cohorts of UofT MOOC instructors, to systematically facilitate an

instructional design process that targets:

o Setting clear weekly learning outcomes

o Designing assessments that map onto learning outcomes

o Developing activities and resources that allow learners to develop knowledge and

competency as relevant to learning outcomes

Organizing symposium and round table sessions to disseminate findings from faculty

MOOC research and raise awareness about the latest developments in UofT MOOCs



13

OLS also provides on-demand technical and instructional coaching and collaborates with

MOOC platform providers to prepare necessary technical support for MOOC instructional teams

and solve technical issues as they may arise.

Data Collection and Data Analysis

Semi-structured, hour long, recorded interviews with participants were conducted in

person or via video conferencing between August 2013 and August 2014. Interviewers were

UofT researchers affiliated with OLS. Interviews were guided by the protocol included in

Appendix I. Specific questions would be added, or existing questions would be modified for each

interview to allow the interviewers to follow-up distinct issues that were raised during an

interview. Other sources of data included syllabi, announcements, and curricular material of

UofT MOOCs in their latest edition available on Coursera or edX. Additionally, we drew on

documentations from OLS. An inductive approach was used to analyze interview data (Creswell,

2012; Thomas, 2006) as indicated in the following steps:

1. Listening to each interview and taking notes, in the form of short phrases, of what seemed

relevant to the purpose of the study.

2. Assigning codes informed by notes and transcribing code segments relevant to research

questions.

3. Collapsing codes into emergent themes and categories.

4. Concurrently, corroborating interview data with curriculum documents, UofT documents

and reports, and, if needed, perform member check.

5. Preparing descriptive account of major and minor themes from data.



14

Results

We have structured the findings in 6 sections: MOOC instructors' motivations to offer a

MOOC, MOOC design processes, MOOC development and offering, professional implications

of MOOC instruction, MOOC development support, and measures for MOOC success.

1- Motivation to Offer a MOOC

1-1- Contributing to open educational resources. During the first two years, 2012-

2013, all UofT MOOCs were offered free of charge. University of Toronto’s positioning of

MOOCs as being accessible to a broad audience as open courses was the main reason for

instructors’ interest in offering a MOOC. Thus, by designing and delivering a MOOC, UofT

instructors would contribute to open educational resources that are available to the public and, as

an instructor pointed out, also showcase the quality of teaching and learning that takes place at

UofT: "When I heard about MOOC I also thought that UofT is a big mysterious place for people

who walk by and they may think some holy mysterious learning is happening here. I think there

is wonderful learning happening at UofT. Always thought we could do more to open UofT to the

public. MOOC was a tangible way to do that."

In the same vein, all UofT MOOCs have been left on their respective platform as archived

MOOCs upon their conclusion Providing even more flexibility for learners through on-demand

learning options was suggested to be another step forward in the public availability of such open

educational resources. The structured course-like format of MOOCs was cited by one instructor

as an enabler for learning about a topic by covering interconnected concepts in a progressive

manner.



15

1-2- Academic relevance. According to instructors, with global learners signing up for

MOOCs in massive numbers, their teaching and its impact would transcend beyond the limits of

their face-to-face or online UofT credit course and into an international public realm. Beyond

being a long-term professional desire for some instructors, such international instructional

exposure would allow the instructors to examine the instructional potential of MOOCs, and in a

broader sense online learning, with large number of learners.

Experiencing, firsthand, the complications of teaching in a MOOC-like context and

assessing learners within the constraints of current MOOC platforms were frequently cited

motivations for offering a MOOC. Two instructors further pointed out the anticipated

pedagogical and academic implications of MOOCs as their motive. From a pedagogical

perspective, understanding how to foster deep learning in MOOCs would inform the instructional

design of large on-campus credit courses that may have become limited to lectures and recall-

type assessment, suggested one instructor: "As there is a larger student-faculty ratio, often people

talk about wanting small classes. In small classes students get to practice more skills. As the class

size grew, the number of skills practiced in class decreased and a lot of courses have become

lecture and multiple choice questions. So I thought MOOCs were interesting because if you could

show that at that scale you could bring in the skills, then people who teach 200 students classes

will not have an excuse. It could be a driver for a change." From a broader academic perceptive,

MOOCs could facilitate scaling up of knowledge within new fields of study by sharing current

knowledge and enabling the exchange of ideas that would lead to inter-institutional research

partnerships. For three instructors, their previous decision to flip their credit courses contributed

to the decision to offering MOOC. As a practical motivating factor, the videos they would create

for their MOOCs could then also be used in a flipped classroom context.



16

2- MOOC Design

2-1 Instructional team. MOOC design, development, and delivery was a team effort

where MOOC instructors collaborated with and supervised teaching assistants, instructional

designers, and technical staff to design content, create a pool of questions for assessment,

populate the course on its respective platform, and monitor discussion forums while the MOOC

was running. In most cases, however, the instructors themselves developed the bulk of the

MOOC content. In addition to instructors' personal preference, availability of funding and

existing departmental provisions impacted the extent to which technical staff were involved in the

design and development phases of a MOOC project.

2-2 Learning outcomes. MOOC design was mainly driven by current successful

pedagogical approaches, i.e. those proven to be effective in credit courses, related to the

respective discipline. The overarching design goal for UofT MOOCs that motivated the design of

MOOC learning and assessment components ranged from providing individual learners as many

opportunities as possible to practice what they learned, to deliberately foster a sense of

community, and to maximize engagement in discussions. Regardless of the discipline, however,

UofT MOOC instructors stressed the importance of learning relevance to MOOC participants and

envisioned opportunities for self-reflection.

All instructors stated clear learning goals for their MOOCs and described their vision for

what learners would take away from their learning experience. To satisfy the anonymity

condition of this study, however, we synthesized high level desired learning outcomes for UofT

MOOCs learners into the following cross-cutting themes:



17

Understand the fundamental concepts of a discipline or field of study and be able to

transfer their knowledge to other contexts and to apply those concepts to relevant

situations in order to improve existing practice.

Become familiar with methods of scientific inquiry, problem solving, and reasoning

relevant to a discipline or field of study and be able to apply those methods to their own

inquiry and develop the competency to assess the quality of existing work in their relevant

context.

Develop knowledge of the discourse and vocabulary of a discipline or field of study and

be able to use both in their own learning/work/personal context.

The instructors also described the learning outcomes for students taking their credit

course as:

Being able to apply knowledge to new situations; thus being able to transfer knowledge.

Being able to articulate fundamental concepts of the discipline and explain them to their

peers.

Develop critical thinking and inquiry skills and be able to locate relevant resources to

enrich their learning without being highly dependent on an instructor.

Develop metacognitive and self regulative skills and to be able to identify and improve

their shortcomings while building on their strengths.

UofT MOOC instructors' identified learning outcomes for their MOOCs and credit

courses show considerable overlap.



18

2-3 MOOC learners. When asked about their target audience, instructors expressed

initial uncertainty about their actual MOOC learners and their academic or professional

backgrounds. However, they designed the MOOCs for a perceived group of learners that would

relate to the given MOOC’s learning outcomes. The target audience for UofT MOOCs would

range, informed by relevant learning outcomes, from learners with little or no previous

knowledge or experience to professionals who desired to expand their knowledge on a specific

topic relevant to their practice. One instructor indicated how the diversity of students in a credit

course, with regards to knowledge of course subject, guided the perception of MOOC learners "I

wanted to plan a course accordingly for people who had little knowledge to people who could

knew more than me and could teach the course. It is the balance of start where you are in terms of

knowing and push yourself to go deeper. Make this an opportunity to deepen your knowledge

about something, whatever that is."

In general, learners who signed up for UofT MOOCs were internationally distributed.

Based on instructor’s observations, actual learners of more specialized MOOCs, turned out to be

close to perceived learners, although learners with little or no background were also present in the

course in smaller numbers. On the other hand, MOOCs that were targeted towards learners with

no or little knowledge or experience in the field attracted highly educated learners. Reflecting on

the reasons why learners with postgraduate degrees would sign up for a MOOC that covered

fundamental concepts of a discipline, an instructor stated: “From the discussion forum, I realized

that a lot of people, even if they had taken [courses relevant to this subject] before, never really

learned those learning goals or did not learn them very well and many of them found it

productive to go back and revisit them. Perhaps, they have taken more advanced courses or need

it for a job.”



19

3- MOOC Development and Implementation

Instructors took different approaches to plan and develop UofT MOOCs. Owing to the

short time available between the approval of some of the MOOCs and their launch date, a number

of the instructors produced the latter parts of their MOOCs while the MOOC was running. The

instructor of an earlier UofT MOOC described the tight timeline as: "We agreed to do the MOOC

[within three months from the agreement the MOOC was offered]. Timeline was incredibly tight.

We were developing the MOOC as it was being offered. We started two weeks ahead so the first

two weeks were done and then we were just go go go. Not a timeline that anyone should ever

attempt again" . Other examples existed where the instructors completely developed a MOOC

before its launch date when the approval date and the launch date were further apart. In either

case, according to the instructors, the process of MOOC planning and development including

determination of learning outcomes, content creation, and assessment design was time consuming

and labor intensive.

Two specific planning and production strategies deemed effective by the instructors were:

Production line approach, and intensive plan-ahead approach. A production line approach was

described by an instructor as follows: "I first created lectures, then somebody edited those,

somebody else generated questions on the video, and somebody else uploaded the video lecture

with the embedded question. This system was effective. Each person was ahead of the next

person." In an intensive plan ahead approach, on the other hand, all components of the MOOC

were completely planned and scripted before the production started.



20

Among the instructors, some had already taught online courses for a number of years.

However, we could not find any evidence that instructors with no online teaching experience

were at a disadvantage while planning and offering a MOOC.

Here, we studied how UofT instructors developed their MOOCs. To facilitate the

structural examination of a MOOC, we informally divided it into three components: Learning

components, assessment components, and communicative components that were present in all

UofT MOOCs. In the following sub-sections, we provide a working definition for each

component and present relevant findings.

3-1 MOOC learning components: Keeping content concise. Learning components

included, but were not limited to, video lectures, suggested or required readings, guest speakers,

and complementary external links. All UofT MOOCs were self contained with no required

textbook. Depending on common practice within a discipline, a number of UofT MOOC

instructors would integrate, or intended to integrate, additional technological environments and

tools to enrich learners’ experience. Integration of external tools demanded close collaboration

between UofT MOOC instructors and the technical staff of their MOOC platform provider. Such

integration was not always feasible due to time constraints or technical complications. However,

instances of successful integration existed where, for example, MOOC learners could engage in

hands-on systematic hypotheses testing and theorizing using community-generated data.

According to the instructors, video lectures were a central learning component in UofT

MOOCs. Below, we examine the instructors’ experience and the issues they might have faced

while planning and producing video lectures.



21

3-1-1 Planning and producing short video lectures. A common theme regarding MOOC

video lectures were their much shorter length compared to lectures in credit courses. Although a

challenging medium to convey concepts thoroughly, short video lectures would suit perceived

learning purposes of MOOC learners. As one instructor stated: “MOOC students are not usually

young. They have busy lives. For them, 15 minute bites can work in a day. So for purposes of

people being able to keep up with a MOOC, this would work.” Observing the constraints of

MOOC learning, the instructors planned their lecture videos to be short, precise, and engaging,

with opportunities for learners to reflect on what they watched, for example, by including in-

video quizzes. This work was time consuming and required advance planning and in many cases,

scripting. We will return to the issue of clarity and precision in MOOC instruction later in the

findings.

One instructor, however, avoided instructor-dominated video lectures and instead

interviewed other subject matter experts: "Others had said that they didn’t like the talking

head.....I also thought of the possibility with the MOOC is that so many people can be part of the

teaching and updating material. So, one of the first thoughts I had was we can include other

professors that do [name of the filed] research and I invited them all to do a lecture. In turned out

we did interviews instead to make it more comfortable...I thought if I am inviting people to do

guest lectures, I could do interviews and ask them the questions that I am hoping the students

would ask the same questions." The variety in video production, however, was largely affected

by the discipline and by the desired learning outcome of a MOOC. Interview with experts, as

mentioned here, were not relevant or applicable to all MOOCs.

UofT MOOC instructors approached video production in different ways. Some instructors

recorded their videos in a Do-it-yourself (DIY) format, others had technicians to produce



22

professional videos, and yet others had partial technical assistance. Decision making regarding

the process of video production depended on various factors including personal preference,

technical knowledge, time constraints, departmental facilities, and to some extent, available

funding.

3-2 MOOC assessment components: Scaffolding complexity. Assessment components

allowed learners to apply what they had learned in practice, receive feedback in a variety of

formats and, if desired, collect a grade that would count towards obtaining a certificate of

completion. Note that, for most of their MOOCs, Coursera and edX offer a free certificate to

learners who successfully finish a course, i.e. achieve at least a minimum assessment grade

specified in the syllabus. Generally, the free certificate, Statement of Accomplishment in

Coursera and Certificate of Accomplishment in edX, does not equate to a university credit course

(http://help.coursera.org/customer/portal/articles/1164817-what-are-the-differences-between-a-

statement-of-accomplishment-and-a-verified-certificate-; https://www.edx.org/student-faq).

Assessment design closely followed the learning outcomes of a UofT MOOC, explained

previously, and thus emphasized application of knowledge and reflection over information recall.

While acknowledging apparent limitations of assessing learning in MOOCs, the instructors

explained their strategies to leverage available assessment tools relevant to their learning goals.

One instructor stated: “Quite happy with the set of questions we made. Managed to find other

ways of asking questions within the constraints and getting students to think and get them to

apply knowledge.” Another instructor explained why social sciences MOOCs may face further

challenges regarding assessment design: "Social science courses required a different approach for

assessment. For previous MOOCs you could design tests that were completely computer graded.

There were peer assessment pieces but usually it was a very digital thing."



23

At an information recall level, instructors used multiple choice questions to gauge

learner’s understanding of essential concepts and, in some cases, would allow learners to make

several attempts at such quizzes to encourage mastery. Depending on the MOOC, learners would

have seen a new set of questions or would have answered the same questions again. Sometimes,

non-graded in-video quizzes were used for a similar objective and to increase engagement.

Progressing to knowledge application, instructors would use self-graded assessments, including

self-reflections, and/or discussion questions. A carefully crafted rubric, the instructors

emphasized, would enable learners to self-assess their submissions.

Increasing in complexity and the degree of learner responsibility, peer assessment was

perhaps one of the most logistically challenging but cognitively promising types of assessment

that every one of the instructors intended to include in their MOOCs. Peer assessment allowed

learners to receive scaffolded feedback from their peers and also review samples of other

learners' works. The quality of feedback provided to peers, learners' potential tendency to be

lenient towards their peers, and language proficiency related issues concerned some of the

instructors. In practice, learners took the responsibility seriously and even in the face of

unforeseen technical difficulties, managed to provide feedback to their peers. Again, instructors

from different disciplines believed that precise marking schemes and grading rubrics, which were

also time consuming to develop, were important facilitators of peer assessment processes.

3-3 MOOC Communicative component: Strategizing instructional presence. A

Communicative component was the platform-based discussion forum for all MOOCs considered

in this study, allowed learners to ask questions, discuss discussion topics, or seek help.

Discussion forums also served as sources of information for the instructors to address occasional

technical or content-related problems as identified by the learners.



24

UofT MOOC instructors organized the discussion forum in a way that would ease posting

comments or questions to relevant discussion threads. Dedicating a thread to weekly discussion

questions and video lectures proved effective although, as the instructors commented, some

learners would post their comments in an irrelevant thread. Sometimes organization of the

discussion forum partially followed common practice of the discipline or subject of study.

Reasons for intervention and providing feedback mainly included controversial discussions,

strong disagreement, instances of misconceptions, and comments posted to irrelevant discussions.

One of the instructors commented: “My heuristic of being involved was we plant the question, or

idea, or example [deleted for anonymity purposes] and I would sit back unless one of two things

happened. There were people that would vehemently disagree with the point or there were people

who were not getting it.”

3- 4 Credit course vs. MOOC instruction. Instructors who answered a question about

differences between the quality of learning experience in MOOCs vs. credit courses agreed that

finishing a UofT MOOC did not equate to completing a UofT credit course on the same topic.

First, while the content of a UofT MOOC could be informed by existing UofT credit courses,

practically speaking, they could not be as deep or as comprehensive as their counterpart credit

courses, the instructors emphasized, due to the shorter time frame of MOOCs and perceived

MOOC learners’ goals. For example, an instructor mentioned: “The question was how to take

[credit course] and break it into 10, 15 minute bites that have a beginning, middle and end and

covers something. It was a real re-envisioning of the course I had taught for so many years.” The

difference in depth and breadth was also highlighted in assessment design. However, at least two

instructors intended to encourage peer interaction as they practiced in their face to face or online

credit courses. Comparing his expectations from MOOC learners and students taking seminar-



25

style credit courses, one instructor mentioned: "So that there was learning that was happening

horizontally, student to student, rather than all the information coming top down. And that’s the

way I approach my in-person classroom. A lot of it was about how to translate what I do in the

classroom to this online environment. How to make the same kinds of things happen and address

the challenges that will come from expanding it from 20 participants in a seminar to 23000

participants."

A second factor that differentiated MOOC learning from learning in credit courses was

the variety in form and the extent of feedback available to MOOC learners vs. UofT students.

Students in face-to-face or hybrid classes have the opportunity to interject and ask questions to

receive immediate feedback, for instance. Teaching assistants can also encourage students to

think about alternative ways of approaching a problem or an issue. Due to a far larger proportion

of learners to MOOC instructors, co-instructors, and teaching assistants, providing feedback to

every learner seemed impossible, even if the instructional team vigilantly monitored the

discussion forums to address learners’ questions. Designing deliberate opportunities for soliciting

learners’ understanding of a concept and providing an aggregate feedback that also

acknowledged high quality learner contributions was an effective strategy for one of the

instructors.

4- Professional Implications of Teaching MOOCs

4-1 Pedagogical implications. The process of designing and teaching MOOCs entailed

two main pedagogical implications for the instructors: Enhancing the clarity of content, e.g.

discipline specific terminology, and restructuring the flow of teaching credit courses. Instructors

paid specific attention to making content or assessment criteria clear in their MOOCs. Explaining



26

the process of rubric development for peer assessment, an instructor noted: "In terms of the

challenges with a MOOC, it is so hard to explain something extremely clearly and not have it be

taken or interpreted in multiple ways... We had seven boxes where [MOOC learners] had to

check and be ultimately marked out of seven... We wanted to make sure each of those points

were really clear." Often times, MOOC learners would candidly, compared to UofT students,

share their opinion about the clarity of materials or learning outcomes, which the instructors

generally found constructive. Some of the instructors pointed out that they now had new ways of

explicitly stressing the clarity and precision of their learning outcomes and the presentation of

concepts in their credit courses. Such attention to clarity and shared understating is captured in

the following comment of one of the instructors: “I have become much much more clear about

my learning objectives and articulating them and making sure that they are articulated in a

language that students can understand – and I revisit them with the students. I think it [deleted]

makes the students more aware of what they are supposed to be learning.”

The second implication, restructuring credit course instruction, manifested in instructors

making MOOC lectures available to students and replacing a portion of their lecture time with

opportunities for active learning. Using MOOC material in credit courses happened either as a

pre-planned strategy where instructors intended to flip their classroom or as the opportunity

arose. Note that some of the instructors did not integrate MOOC material in their teaching due to

lack of practical relevance to their credit courses. An expected affordance of flipped classrooms

was described by an instructor as: “if you get it (the concept) in the first five minutes, you still

have to listen to all the questions and repetitions. Also in lectures there is a lot of pauses and

examples that these students who got it in the first 5 minutes do not need.” Flipping a course

appeared a fulfilling experience for the instructors who participated in this study as their students



27

could come to the class more prepared, spend more in class time applying their knowledge to

classroom projects, or engage in peer instruction in small-group or whole-class format. Peer

instruction, an instructor believed would reinforce students' learning as they would have to

articulate concepts, in their own language to other students.

4-2 Recognition and Dissemination. Although challenging and resource-intensive, the

instructors in general expressed their professional satisfaction of teaching a MOOC and

appreciated the wide range of recognition they received for their efforts. The instructors

highlighted three sources of recognition and acknowledgement for their MOOC instruction: UofT

top level administration including the Office of the President and the Office of the Vice President

and Provost; Academic community including affiliated UofT department and faculty members

from other universities; and MOOC learners. MOOC development and delivery also resulted in

opportunities for broader dissemination of their scholarship and teaching. Four instructors had

already attended international conferences or presented and published papers based on their

experience, thus gaining international academic exposure.

5- Evolution of UofT MOOC Support Services

Based on UofT MOOC instructors’ account of available MOOC support and resources, OLS

support targeted two aspects of MOOC development and implementation: pedagogical and

technical. UofT MOOC support capacity evolved and improved since 2012 and was evident from

instructors’ description of the nature and source of pedagogical and technological support,

presented below. One instructor specifically emphasized that how OLS team adjusted and

improved their instructional support strategies since earlier UofT MOOCs.



28

An early UofT MOOC instructor stated: “It was early and people did not have the

support. The [platform’s] bug supports were sent to a UofT bug support list that included [OLS].

But nobody from UofT knew how to answer it. [deleted] but only people from [MOOC platform]

could answer the question. UofT people watched and learned but they couldn’t support us at that

time because everything was new.” For the first cohort of UofT MOOCs, late 2012 and early

2013, support was rather limited to technical aspects of MOOC design and development provided

mainly by MOOC platform providers. Other instructors from the same cohort added that UofT's

OLS would follow up to make sure issues of concern would be properly addressed.

The second cohort of UofT MOOC instructors, mid 2013 to 2014, received both

pedagogical and technological support from OLS. At the time of data collection for this study,

and according to OLS documentation, planning and support provided research-informed

instructional guidance regarding goal setting, content development, and assessment design

through course design institutes, workshops and consultations. The change in the level of

institutional support for the second cohort of UofT MOOCs is evident in this comment by one of

the instructors: “OLS was fantastic in terms of the planning process…planning was key. One of

the things that OLS folks particularly pressed on me was the idea to have modules with themes,

learning objectives for each module, and to also plan every little lecture bit. So that’s what we

did.” Technical support services addressed MOOC platform-related issues. Another instructor

described an example of technical support provided by OLS as: “[OLS learning strategist] helped

a lot to sort out technical issues with [MOOC platform] and also he went in everyday and looked

at participants' postings about technical problems.” These instructors explained how following a

backward design-inspired approach (as encouraged through UofT support and training

opportunities) helped them systematically plan their MOOC and include as much detail in



29

planning as to what would each video lecture contain in relation to weekly and overall learning

outcomes. One of the instructors shared the story board their instructional team had developed for

each week.

6- Measures of MOOC Success

Completion rate, percentage of learners who obtain a certificate of completion was

deemed an unsuitable and rather simplistic measure of MOOC success as the instructors believed

learners’ goals would impact their level of engagement with learning, assessment, and

communicative components of a MOOC. An instructor explained: “There were many people in

the course who did not want the certificate. Some learners would say that I am very happy. I am

watching the videos and learning a lot. I am not doing assessment because I don’t care about

completion. So I realized that there could be engaged people who don’t fall into that number for

different reasons.” Another instructor emphasized the importance of the relation between learner

motivation and learner activity when measuring success in MOOC context. For one instructor,

developing a high quality learning resource took precedence over completion rate: "I wanted the

MOOC to be high quality because my name was sitting next to it. I Don’t care about attrition

rate. The thing is I achieved producing a high quality MOOC and that’s success. I was pleased to

see people enjoy it and engage with it and find the resources stimulating, clear and helpful for

them."

The quality, and not the quantity, of posts in a MOOC discussion forum was regarded as a

more relevant measure of success. High quality contributions to forums partly showed that

discussion questions and learning materials aroused learners’ interest and encouraged them to

engage in knowledge-based discussions with their peers. The quantity of discussion forum



30

participation could partly be a function of learners' intent to engage in such activity and, as

mentioned before, not a thorough success indicator.

Discussion and Conlusion

In this study, we interviewed seven UofT instructors who offered MOOCs either in

Coursera or edX platforms to gain a better understanding of their experience in MOOC design,

development, and implementation, motivation to offer MOOCs, and potential impact on their

credit course instruction. MOOC development was a resource intensive endeavor for all

participating instructors in design, development, and implementation stages. During design and

development, time was spent planning the MOOC and creating content and assessment. In this

study, MOOC content was either based on existing credit courses, or was developed from scratch

and in both cases instructors reported spending considerable time to develop content. In the

absence of detailed records of the amount of time spent on content development, we could not

assess the actual time required for content adaptation or content creation; however, we do know

from the instructors interviewed, that the time was deemed to be considerable.

Motivational Factors: Broader Outreach and Pedagogical Promise

A network of factors impacted UofT instructors' motivation to offer a MOOC. First,

identifying MOOCs as a means to reach out to a broader international public and to share a

glimpse of UofT teaching and learning approaches was a main motivation. Geographical

distribution of UofT MOOC learners with respect to instructors' intended audience seemed

satisfactory and sometimes beyond expectations for the instructors. Thus, establishing a broader

outreach, was a motivation that was realized. As of July 2013, according to OLS statistics, the



31

geographical distribution of UofT MOOC learners based on the number of registrants from six

regions was as follows, confirming the instructor's perception of student diversity:

Australia: 1268

Africa: 1610

South America: 3244

Asia: 9938

Europe: 15655

North America: 16894

However, this motivation did not function in isolation and was contextualized as well

within academically driven motives such as examining ways to push the limits of pedagogy in a

large online classroom context, to devise instructional approaches that leverage active learning

within that context, and to conceptualize alternative learning opportunities and targeted programs.

Instructors' emphasis on learners' ability to apply their learning, as demonstrated in the learning

outcomes section of findings, resonates this desire. Alternatively, open courses would engage an

academic audience beyond course boundaries by showcasing the latest research developments at

UofT and inviting research partnerships. The opportunity to develop and run a MOOC, for some

instructors, coincided with their decision to flip their credit course. The possibility to connect a

MOOC to current teaching and research interests justified the time and effort needed to design

and implement MOOCs.



32

MOOC Learning Outcomes: Moving beyond Knowledge Transmission

Owing to the openness of MOOCs, characterizing MOOC learners in terms of their

background, learning goals, and commitment level is near impossible (DeBoer, Ho, Stump, &

Breslow, 2014). While UofT MOOC instructors echoed such concern, they did not compromise

on MOOC learning outcomes. Instructors' MOOCs and credit course learning outcomes, as

shown in the findings, converge in many ways, specifically with regards to knowledge transfer

and application. Mapping the desired learning outcomes to Fink's (2003) Taxonomy of

Significant Learning, instructors kept the bar high for MOOC learners by moving beyond a focus

on foundational knowledge to fostering application and integration of knowledge. As relevant to

one MOOC, the instructor included "caring" (Fink, 2003) as one of the desired learning

outcomes. However, higher level learning outcomes, such as "learning to learn" (Fink, 2003)

were desired more in credit courses, which could be due to the time it takes for learners to

develop skills of metacognition and self-regulation. One instructor expressed his goal for on-

campus students to develop such competencies upon completing their undergraduate studies.

Learning outcomes of a MOOC are articulated in relation to potential MOOC learners.

However, with virtually no barrier to entry, MOOCs may attract learners who may not have been

perceived as a target audience but find the topic professionally or academically useful or are just

curious and sign up opportunistically. For credit courses, an existing knowledge of students'

characteristics facilitates the articulation of learning outcomes. MOOC instructors, on the other

hand, may have to make assumptions about their learners and plan ahead without extensive

knowledge their actual learners. This has raised concerns about MOOCs, specifically, xMOOCs,

being teacher-centric and perpetuating traditional education (Eisenberg, & Fischer, 2014).

However, our findings show that MOOC instructors can engage learners in more than knowledge



33

acquisition. Instructional design that demands active learning, even within the constraints of

MOOCs, can offset the initial teacher-centered design.

Working within Assessment Constraints

UofT MOOC instructors used three types of formative and summative assessment

approaches relative to their learning outcomes and the nature of MOOC subject matter:

knowledge/understanding (recall of information), self-assessment, and peer-assessment. The

complexity of design, development, and implementation of these type of assessments increases

from machine-graded to peer-graded assessments. The choice of assessment format depended

highly on the MOOC platform and whether a specific form of assessment was functional during

the implementation of a MOOC. None of the instructors relied exclusively on information recall,

multiple choice, or machine graded quizzes to assess learners' progress. Rather, they aimed for

educative assessment to measure if students were making progress regarding learning outcomes

focused on application and integration.

To be effective, self and peer assessment need carefully designed rubrics that enable

learners to judge the level of their own or their peers' achievement of leaning outcomes. The

importance of rubric design is highlighted in the instructional design literature (Fink, 2003).

Within a MOOC context, Kulkarni et al. (2013) emphasize the importance of creating readable

rubrics and avoiding ambiguous terminology to facilitate peer feedback. UofT MOOC

instructors pointed out the importance of content clarity to ensure shared understanding. Also,

given the sheer number of MOOC learners makes it impossible for the instructional team to

check the accuracy of peer feedback, strategies for scaffolding peer assessment and self

assessment facilitates their scalability. Such scaffolding can be achieved through calibration



34

exercises and corroboration with staff-graded assignments. In this study, one instructor

mentioned employing calibration in the MOOC's peer-assessment assignment. However, we did

not collect detailed data on the instructors' approaches to designing rubrics and their use of peer-

assessment calibration or training, which could be further studied in the future.

Learners who enrolled in archived MOOCs, nevertheless, would not be able to benefit

from peer assessment. While on-demand MOOCs are currently in a development phase at UofT,

examining the practicality of future peer-assessment approaches, would be an area of further

investigation.

Multi-disciplinary MOOC Development Team

MOOC development requires a team of instructors, instructional designers, teaching

assistants, technical staff, and MOOC platform providers to work together at various stages

between design and implementation. Although the configuration of such teams may vary, our

findings were consistent with previous work on the importance of adopting a team approach to

MOOC development (Belanger, & Thornton, 2013; Alario-Hoyos et al., 2014). Similar to

Arnold et al. (2014), sometimes MOOC instructors adopted a "divide and conquer approach" to

content development, although time constraint, rather than making the task manageable, was the

main reason for such choice.

UofT MOOC Instructors who participated in this study expressed their desire to closely

follow discussions to address issues of controversy or to provide feedback. The high volume of

interaction during the offering of a MOOC may become overwhelming for the instructional team

to follow (Robinson, & Ash, 2014) and demand informal instructional support, one source being

community Teaching Assistants (community TAs). Community TAs are learners who



35

successfully finish a MOOC and demonstrate exceptional performance. In the occasion that the

same MOOC is offered again, these learners are invited to provide peer feedback in discussion

forums threads. Community TAs in a Coursera Python MOOC, for instance, would address the

bulk of learners' questions (Severance, 2013). One UofT MOOC that was offered twice also

enlisted the help of community TAs in their instructional team. Another instructor mentioned the

anticipated involvement of community TAs in the second offering of a MOOC. Community TAs

should have usually excelled in a MOOC and are formally invited back for future iterations of

that MOOC which may present a limitation for first time running MOOCs. Strategies such as

inviting on campus students to informally assist the instructor, as one UofT MOOC instructor

did, may help extend the instructional team for such MOOCs.

Systematic and On-demand MOOC Development Support

At this time, central MOOC development support at UofT exists through OLS, while

instructors are dependent on departmental facilities for technical support and equipment required

for content development. The structure and centrality of support varies in different universities

mainly based on existing infrastructure prior to MOOC initiatives (Gooding et al., 2013).

Potential affordance of centralized vs. distributed support systems are beyond the scope of this

paper, yet an important topic for further research as the instructors suggested the benefits of

consistency in presentation and technical support if UofT plans to offer more MOOCs. Here, we

focus our attention to the ways OLS has progressed at UofT and expanded its services to MOOC

instructors since 2012.

Considering the variety of disciplines anticipated learners, and learning outcomes of UofT

MOOCs (UofT MOOC Year 2 report, 2014), since 2012, OLS has gradually developed a



36

common instructional design support system for all UofT MOOCs, while also attending to

specific requirements of each MOOC. In two round-table sessions offered by OLS, instructors

reviewed design considerations for online learning and engaged in hands-on instructional design

tasks based on backward design and understanding by design frameworks (Wiggins & McTighe,

2005). These workshops attempted to highlight the importance of the alignment between learning

outcomes, assignments and activities, and assessment. Instructors received suggested templates to

visualize such alignments and had a chance to share one set of learning outcome, activities, and

assessment with other workshop participants. The two instructors who participated in these

workshops provided evidence of the practicality and effectiveness of such workshops. In

comparison to an existing model such as MOOC Canvas (Alario-Hoyos et al., 2014), OLS

MOOC development support system emphasizes a systematic and research-based instructional

design approach that directly addresses the alignment of learning outcomes, instructional

activities, and assessment.

Following the suggestion of OLS, one instructor fully developed a MOOC a month before

it opened for registration. The other instructor, while facing unanticipated challenges in the

process of design and development, reflected positively about a systematic approach to MOOC

planning and development. The importance of planning ahead in MOOCs has been emphasized

by other MOOC instructors (Arnold et al., 2014). More recently, OLS has developed a

comprehensive resource requirement guide that outlines, in detail, steps of developing a MOOC,

time and resource allocation for each step, and an explanation of what each step would entail.

Often times, faculty members undertake a MOOC on top of their current teaching and research

responsibilities. Thus, pragmatic estimates of time and resource requirements further facilitates

their planning in relation to their existing work load. The total time required for the instructional



37

team to develop and teach a MOOC varies considerably in existing reports from an average of 75

hours (Gooding et al., 2013) to 620 hours (Belanger, & Thornton, 2013). Contextual and

institutional factors impacts the time estimate, thus increasing the accuracy of the time

commitment proposed based on existing UofT MOOC experience.

Pedagogical Implications of Teaching a MOOC

UofT MOOC instructors aimed for significant learning and deliberately provided

opportunities for learners to engage with course material and with their peers, Including but not

limited to debates, discussion questions, and increased complexity of assignments. However, they

did not equate the potential depth of learning that could take place in MOOCs with their credit

courses for reasons such as shorter length of MOOCs, broader audience, and amount and the

quality of feedback available to MOOC learners vs. credit course students. For instructors who

had never taught an online credit course, MOOC development also allowed them to, either with

previous intention or as the opportunity arose, flip their credit course and integrate more

opportunities for active learning.

MOOC development may have further stressed the importance of instructional design and

detailed upfront planning regarding the alignment of learning outcomes, learning activities, and

assessment. The emphasis on systematic upfront planning was more evident for the second cohort

of UofT MOOC instructors as they had participated in OLS MOOC development workshops.

Comparing preparation for credit courses and MOOCs an instructor commented: "I don’t think I

ever spent as much time planning out any course. I obviously spend time thinking through how

much time to devote to each section, but in terms of going down to every 5 minute chunk and

what should I communicate by the end of that, I had never done that and so, that was a big issue



38

and a big difference." As proposed by Arnold et al. (2014), upfront planning and clarification of

roles and responsibilities is specifically important if two or more instructors are teaching a

MOOC. Importance of upfront and detailed planning is also stressed in distance education.

Reflecting on the structure of courses that was designed systematically following Horizontal

Course Syllabus instructional design model, a full professor and associate professors appreciated

working closely with an instructional designer and favored the explicit design of the courses

which attended to all details and left little room for ambiguity (Powers, 2009).

Characterizing MOOC Success: Thinking Beyond Numbers

Although apparent attrition rates in MOOCs and the small proportion of learners who

complete all assessment requirements of MOOCs, compared to the number of enrolled learners,

has raised considerable concern (Belanger, & Thornton 2013; Breslow et al., 2013; Chen et al.,

2012), UofT instructors agreed that such raw numbers taken out of learner intent context do not

reflect the success of a MOOC. More recently, learner engagement with regards to their intention

and commitment level when enrolling in MOOCs has emerged as a lens to propose an alternate

interpretation of MOOC completion or achievement (DeBoer et al., 2014; Kizilcec, Piece, &

Schneider, 2013). Knowledge of MOOC learner intentions, nevertheless, is affected by learners'

decisions to answer survey questions designed to collect such information.

Our participants noted the quality of learner participation and use of MOOC resources in

light of learner intention, as an alternative and relevant indicator of MOOC success. One

instructor identified investigating strategies to keep learners engaged as a research opportunity in

the MOOC context: "In a MOOC it is so easy to stop going either because you are bored or

because your life is too busy. It is still a glorious situation for looking at factors that enhance



39

engagement. So if you can hold the students in that context where it is most easy to leave, then

you are really doing something. Different techniques could be used and then compared.

Engagement doesn’t get a lot of research." Such research would have further implications for

MOOC design and instructional design support required to include fostering engagement with

regard to intention.

Implications and Future Research

In this qualitative study, we examined motivations, experiences, and reflection of eight UofT

MOOC instructors in the process of designing and delivering MOOC. By focusing on less

studied aspects of MOOCs as learning environments, MOOC design considerations and

instructors' experience, our study addresses the paucity of research that could eventually impact a

MOOC learner's experience. We acknowledge the limitation of our study specifically with

regards to the small number of instructor participants and the confidentiality consideration that

prevented us from comparing and contrasting instructional designs between MOOCs, or our

ability to deeply contextualize our findings, where relevant, with examples from specific

MOOCs. That said, three major findings from our research provide an important contribution to

the literature on MOOCs and can be summarized as:

Instructors' motivation to offer a MOOC were to provide quality learning experiences to

broader groups of learners and to contribute to open educational material.

MOOC experience led to integration of or increasing the opportunity for active learning in

credit courses for four instructors.

Systematic, research-based instructional support during MOOC design facilitated

instructors' planning.



40

Informed by our findings and in light of the current state of MOOC literature, we propose

five areas of future research:

Design considerations and learning implications of self and peer-assessment approaches

Impact of the organization of discussion forum and various arrangements of instructional

team presence on learner participation in subject matter related discussions

Role of community TAs in facilitating learner engagement. Influence of MOOC planning

and implementation on subsequent credit course design

Understanding Inhibiting or motivating factors to offer MOOCs beyond instructors who

have already offered MOOCs.

UofT MOOC instructors were also interested in designing new learning experiences for

specific group of learners inspired by the MOOC concept. Target learners varied from learners

who could not enroll in a MOOC and complete it while the MOOC was running, known group of

professionals who were geographically dispersed in remote areas , and organizations who were

seeking alternative modes of professional development for their staff. On-demand MOOCs are

already in the design phase at UofT. Whether current or future UofT MOOC instructors would

experiment designing MOOCs with different learning tracks to correspond to learning intentions

remains an open question. Currently, a comprehensive research study is underway at UofT that

examines high level relations between learner intent, learner behaviour, and learner outcome in

multiple UofT MOOCs.



41

References

Alario-Hoyos, C., Pérez-Sanagustín, M., Cormier, D., & Kloos, C. D. (2014). Proposal for a

conceptual framework for educators to describe and design MOOCs. Journal of

Universal Computer Science, 20(1), 6–23.

Alario-Hoyos, C., Pérez-Sanagustín, M., Delgado-Kloos, C., Parada G., H., Muñoz-Organero,

M., & Rodríguez-de-las-Heras, A. (2013). Analysing the impact of built-in and external

social tools in a MOOC on educational technologies. In D. Hernández-Leo, T. Ley, R.

Klamma, & A. Harrer (Eds.), Scaling up Learning for Sustained Impact , 8095,( pp. 5–

18). Springer Berlin Heidelberg. Retrieved from http://dx.doi.org/10.1007/978-3-642-

40814-4_2

Alvarez, D. M., Blair, K., Monske, E., & Wolf, A. (2005). Team model in course development: A

unit specific approach. Educational Technology & Society, 8(3), 176-186.

Arnold, P., Kumar, S., Thillosen, A. & Ebner, M. (2014). Offering cMOOCs collaboratively: The

COER13 experience from the convenor’s perspective. In U. Cress, & C.D. Kloos (Eds.),

Proceedings of the European MOOC Stakeholder Summit 2014, pp. 184-188.

Belanger, Y., & Thornton, J. (2013). Bioelectricity: A quantitative approach Duke University’s

first MOOC.

http://dukespace.lib.duke.edu/dspace/bitstream/handle/10161/6216/Duke_Bioelectricity_

MOOC_Fall2012.pdf

Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013).

Studying learning in the worldwide classroom: Research into edX’s first MOOC.



42

Research & Practice in Assessment, 8, 13-25. Retrieved from

http://www.rpajournal.com/dev/wp-content/uploads/2013/05/SF2.pdf

Bruner, J. (2007). Factors motivating and inhibiting faculty in offer their course via distance

education. Online Journal of Distance Education Administration, 10(2).1-24.

Campbell, J., Horton, D., Craig, M., & Gries, P. (2014). Evaluating an Inverted CS1. In

Proceedings of the 45th ACM Technical Symposium on Computer Science Education

(pp. 307–312). New York, NY, USA: ACM. doi:10.1145/2538862.2538943

Chapman, D. D. (2011). Contingent and tenured/tenure-track faculty: Motivations and incentives

to teach distance education courses. Online Journal of Distance Learning Administration,

14(3), 1-12.

Chen, Z., Cheng, J., Chia, D., & Koh, P. W. (2012). Engagement, interaction, and retention in

online Classes. CS224W Final Report.

http://www.stanford.edu/class/cs224w/upload/cs224w-003-final.pdf

Conceição, S. C. (2006). Faculty lived experiences in the online environment. Adult Education

Quarterly, 57(1), 26-45.

Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative

and qualitative research (4rd ed.). Upper Saddle River, NJ: Pearson.

DeBoer, J., Ho, A. D., Stump, G. S., & Breslow, L. (2014). Changing “course”:

Reconceptualizing educational variables for massive open online courses. Educational

Researcher, 43(2), 74–84.

De Gagne, J. C., & Walters, K. (2009). Online teaching experience: A qualitative metasynthesis



43

(QMS) study. Journal of Online Learning and Teaching, 5(4), 577-589.

Eisenberg, M. & Fischer, G. (2014) "MOOCs: A perspective from the learning sciences" in J. L.

Polman et al. (Eds.), Learning and Becoming in Practice: 11th International Conference of

the Learning Sciences 2014, Boulder, pp. 190-197.

Fink, L. D. (2003). Creating significant learning experiences: An integrated approach to

designing college courses. San Francisco: Jossey-Bass

Gibbs, A. (2014). Experiences teaching an introductory statistics MOOC. In Proceedings of the

Ninth International Conference on Teaching Statistics (ICOTS9, July, 2014). Voorburg,

The Netherlands: International Statistical Institute.

Gooding, I., Klaas, B., Yager, J. D., & Kanchanaraksa, S. (2013). Massive open online courses in

public health. Frontiers in Public Health, 1, 59.

Haklev, S. (2011). The Chinese national top level courses project: Using open educational

resources to promote quality in undergraduate teaching (Unpublished master’s Thesis).

University of Toronto, ON, Canada.

Hixon, E. (2008). Team-based online course development: A case study of collaboration

models. Online Journal of Distance Learning Administration, 11(4).

Hollands, F. M. & Tirthali, D. (May, 2014). MOOCs: Expectations and reality. Full report.

Center for Benefit-Cost Studies of Education, Teachers College Columbia University.

Retrieved from http://cbcse.org/wordpress/wp-

content/uploads/2014/05/MOOCs_Expectations_and_Reality.pdf

Kelder, J-A., King, C., Carew, T. and O’Reilly, J. (2013). Evaluation of a MOOC pilot: impacts



44

on pedagogical and technical design and dementia education research. In H. Carter, M.

Gosper and J. Hedberg (Eds.), Electric Dreams. Proceedings ascilite 2013 Sydney.

(pp.456-460)

Kizilcec, R., Piece, C. & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner

subpopulations in massive open online courses. The 3rd Proceedings of the Learning

Analytics & Knowledge Conference. Leuven, Belgium.

http://www.stanford.edu/~cpiech/bio/papers/deconstructingDisengagement.pdf

Koehler, M. J., Mishra, P., Hershey, K., & Peruski, L. (2004). With a little help from your

students: A new model for faculty development and online course design. Journal of

Technology and Teacher Education, 12(1), 25-55.

Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., … Klemmer, S. R.

(2013). Peer and self assessment in massive online classes. ACM Transactions on

Computer-Human Interaction, 20(6), 1–31.

Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic

study of the published literature 2008-2012. The International Review of Research in

Open and Distance Learning, 14(3). Retrieved from

http://www.irrodl.org/index.php/irrodl/article/view/1455/2531

Maguire, L. (2005). Literature review-faculty participation in online distance education: Barriers

and motivators. Online Journal of Distance Learning Administration, 8(1), 1-15.

McQuiggan, C. A. (2012). Faculty Development for Online Teaching as a Catalyst for Change.

Journal of Asynchronous Learning Networks, 16(2), 27-61.



45

Mitchell, R. L. G. (2009). Online education and organizational change. Community College

Review, 37(1), 81-101.

Nelson, S. J., & Thompson, G. W. (2005). Barriers perceived by administrators and faculty

regarding the use of distance education technologies in pre-service programs for

secondary agricultural education teachers. Journal of Agricultural Education, 46(4), 36-

48.

Power, M. (2009). A designer’s log: Case studies in instructional design. Athabasca University

Press. Retrieved from http://www.aupress.ca/index.php/books/120161

Robinson, D., & Ash, P. (2014, May). Developing a pedagogical model for a massive open

online course (MOOC). Paper presented at Frontiers in Mathematics and Science

Education Research Conference. Famagusta, Cyprus. Retrieved from

http://scimath.net/fiser2014/presentations/David%20Robinson.pdf

Restoule, J.P. (2013) . Massive Open Online Courses and the Future of Adult Education. In C.

Kawalilak & J. Groen (Eds.) Proceedings from Canadian Association for the Study of

Adult Education (CASAE) 2013, 514-520.

Robinson, D., & Ash, P. (2014). Developing a pedagogical model for a massive open online

course (MOOC). In proceedings of the Frontiers in Mathematics and Science Education

Research Conference. (pp. 131-135). Famagusta, North Cyprus.

Rodriguez, C. O. (2012). MOOCs and the AI-Stanford like courses: Two successful and distinct

course formats for Massive Open Online Courses. European Journal of Open, Distance

and E-Learning.

http://www.aupress.ca/index.php/books/120161

http://scimath.net/fiser2014/presentations/David%20Robinson.pdf



46

Rosselle, M., Caron, P. A., & Heutte, J. (2014). A typology and dimensions of a description

framework for MOOCs. Proceedings of the European MOOC Stakeholder Summit 2014,

130-139.

Severance, C. (2013). MOOCs: An Insider’s View. Computer, 46(10), 93–96.

Shea, P. (2007). Bridges and barriers to teaching online college courses: A study of experienced

online faculty in thirty six colleges. Journal of Asynchronous Learning Networks, 11(2),

73-128.

Shea, P., Pickett, A., & Li, C. S. (2005). Increasing Access to Higher Education: A study of the

diffusion of online teaching among 913 college faculty. The International Review of

Research in Open and Distance Learning; 6(2). Retrieved from

http://www.irrodl.org/index.php/irrodl/article/view/238/493

Soman, D., Evans, R., & Federico, C. (2013, December). Secondary School Students and

MOOC’s: A Comparison between Independent MOOC Participation and Blended

Learning. Paper presented at the MOOC Research Conference, University of Texas,

Arlington.

Tabata, L. N., & Johnsrud, L. K. (2008). The impact of faculty attitudes toward technology,

distance education, and innovation. Research in Higher Education, 49(7), 625-646.

Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data.

American journal of evaluation, 27(2), 237-246.

Tallent-Runnels, M. K., Thomas, J. A., Lan, W. Y., Cooper, S., Ahern, T. C., Shaw, S. M., &

Liu, X. (2006). Teaching courses online: A review of the research. Review of educational



47

research, 76(1), 93-135.

Wiggins, G. P., & McTighe, J. (2005). Understanding by design. Ascd.

University of Toronto November 2014design. How UofT instructors think about their own teaching at...

Documents

Transcript of University of Toronto November 2014design. How UofT instructors think about their own teaching at...