jaln guidelines - Online Learning Consortium · grading as a learning assessment method in the MOOC...

The purpose of the Online Learning Consortium (OLC) is to help learning organizations continually improve the quality, scale, and breadth of online programs according to their own distinctive missions, so that education will become a part of everyday life, accessible and affordable for anyone, anywhere, at any time, in a wide variety of disciplines.

This publication contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the authors and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use.

The Journal of Asynchronous Learning Networks is published under the terms of the Creative Commons Attribution 3.0 Unported (CC BY 3.0) License.

Attributions should be made with a full citation including the link to the article or issue available by search at the JALN webpage, http://onlinelearningconsortium.org/publications/jaln_main. Please direct all inquiries to [email protected].

To learn more about submission guidelines for authors and subscription information, please visit: http://onlinelearningconsortium.org/publications/jaln_guidelines

Publications assistants:

Michael Mastroianni, Damien Bilka: Copyeditors

Beth Meigs: Layout Editor

Cover design by Leighton Ige

Copyright ©2014 by OLC® Published 2014

Printed in the United States of America 0 9 8 7 6 5 4 3 2 1

Standard Book Number 978-1-934505-11-3-1-934505-11-0 (print) International Standard

Book Number 978-1-934505-12-0-1-934505-12-9 (online)

Journal of Asynchronous Learning Networks – Vol. 18. No. 2 (2014)

Dr. Peter Shea

Beth Meigs

Dr. Gregory W. Hislop

Dr. Stephanie J. Jones

Dr. Katrina Meyer

Dr. Anthony G. Picciano

Dr. Don Spicer

Dr. Karen Swan

The Online Learning Consortium, Inc. is a consortium of higher-education providers sharing the common bonds of understanding, supporting and delivering education via asynchronous learning networks (ALNs). With the mission of providing learning to anyone anywhere seeks to provide new levels of learning capability to people seeking higher and continuing education. For

information about OLC


Journal of Asynchronous Learning Networks

Volume 18 Issue 2 - July 2014INTRODUCTION

Dr. Peter Shea, Editor-in-Chief

SECTION I: MASSIVE OPEN ONLINE COURSE (MOOC) RESEARCH

Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects Heng Luo, Anthony C. Robinson, Jae-Young Park

SECTION II: ONLINE DISCUSSION ANALYSES

Good Quality Discussion is Necessary but Not Sufficient in Asynchronous Tuition:A Brief Narrative William James Fear, Andrew Erikson-Brown

An Exploration of Metacognition in Asynchronous Student-Led Discussions: A Qualitative InquiryMartha M. Snyder, Laurie P. Dringus

The Effect of Structured Divergent Prompts on Knowledge Construction Ginger Sue Howell, Autumn Sutherlin, Usen Akpanudo, Laura James, Mengyi Chen

Differences in Classroom Versus Online Exam Performance Due to Asynchronous Discussion Dr. Robert Jorczak, Danielle N. Dupuis

SECTION III: MOBILE LEARNING

The SAMR Model as a Framework for Evaluating mLearning Danae Romrell, Lisa C. Kidder, Emma Wood

SECTION IV: QUALITATIVE PERSPECTIVES

Teaching Presence: Co-Creating a Multi-National Online Learning Community in an Asynchronous Classroom Leanne Dzubinski


IntroductionOn behalf of the Online Learning Consortium and our Editorial Board I invite you to read and enjoy the final issue of the journal under the name Journal of Asynchronous Networks (JALN). As you have probably noticed, Sloan-C , our sponsoring organization, very recently underwent a name change to become the Online Learning Consortium (OLC). With this organizational transition we also decided to take the opportunity to re-envision the identity of the flagship publication of OLC. Much has changed since the founding of Sloan-C and JALN but our focus on advancing the scholarship, theory, and practice of online learning remains. For those reasons and with the support of our Editorial Board and the board of OLC we are pleased to announce our new name: “Online Learning.” Stay tuned for additional updates and news with the release of our next issue.

This final edition of JALN features a collection of articles on topics critical to advancing our understanding of online education. These papers apply a variety of methods to investigateimportant issues that shape how learning occurs online. The articles in this issue examine new environments such as massive open online courses, as well as traditional online and mobile learning formats and provide guidance on critical issues such as assessment, the design of onlinediscussions, and the overall quality of online course dynamics.

As interest in new forms of online education continues to grow we are very pleased to present acutting edge paper by Heng Luo, Anthony Robinson, and Kae-Young Park that investigates important questions around peer grading in massive open online courses. This article is one of the first of its kind to provide empirical evidence in relation to reliability, validity, and student attitudes toward peer-grading in MOOCs. Results are contrary to some of the popular perceptions that peer grading in MOOCs is unpopular or ineffective. Based on an analysis of more than 1800 peer graded assignments the authors conclude that the students in the course they investigated were able to provide feedback that was relatively reliable and consistent with that of the instructor. They also found that a majority of students felt that peer grading was fair and should continue to be used. The authors conclude with essential recommendations for the design and implementation of peer assessment in massive open online courses. This article will be of great value to researchers, MOOC instructors, as well as instructional designer working in these environments.

Understanding online learning hinges on better knowledge of the discursive practices that occur in mediated environments. A set of four articles provides us with new insights into the definition, roles, effects, and limitations of online discussion. William James Fear and Andrew Erikson-Brown contribute a narrative review that identifies areas of consensus within the literature on the key factors for successful asynchronous discussion concluding that the facilitation of peer-peer discussion is considered the key element with the caveat that online discussions are necessary but not sufficient to online education. Delving deeper into this topic

Journal of Asynchronous Learning Networks – Vol. 18. No. 2 (2014) 1

Martha Snyder and Laurie Dringus provide a multifaceted analysis of metacognition as evidenced in threaded discussion and conclude that student -led discussion facilitates the development of core components of meta-cognition when appropriately managed by online instructors. This article represents hard-won progress in our understanding of metacognition and how to support it in online environments.

Also in this line of inquiry Robert Jorczak and Danielle Dupuis investigated the effects of online discussion on learning outcomes and include findings that suggest asynchronous peer-to-peer discussion can be more effective than traditional classroom lecture-discussion. Looking more deeply into the design of such instructional dialogue Ginger Sue Howell, Autumn Sutherlin, Usen Akpanudo, Laura James, and Mengyi Chen examined the effects of structured divergent prompts on knowledge construction based in online discussion. They extend the general results that online discussion may be more effective and explain how certain kinds of prompts (focal and brainstorm) lead to higher levels of knowledge construction. Together, this collection of articles provides a useful contribution to the literature on dialogic approaches to learning in online environments.

Stepping back from the focus on discussion Leanne Dzubinski provides a qualitative analysis of a multi-national, multi-ethnic, and multi-cultural online course in which she seeks to understand the elements of teaching presence necessary to support a diverse student population. As do Snyder and Dringus, Dzubinski also references the Community of Inquiry (and my own research into this analytic model) in an investigation that utilizes in-depth interviews to provide a richer and more nuanced understanding of course dynamics. Moving beyond evidence reflected in discussions activities alone the author provides an analysis of multiple forms of interaction and concludes that supporting student confidence, affirming student voice, and the strategic use of groups can help create a climate of safety conducive to effective learning. Dzubinski also provides a brief summary of effective instructor techniques that will be helpful to othersdesigning, supporting, or teaching online in diverse cultural settings.

As mobile devices become increasing ubiquitous it is inevitable that online learning will be carried out “on the go” and with the affordances and constraints of these devices. Or final article in this issue presents an evaluative framework for considering mobile online learning anddiscusses the components of substitution, augmentation, modification, and redefinition (SAMR). The goal here is to advance our conceptual thinking and the article concludes with useful suggestions for instructional designers as they consider the technical, pedagogical, and management issues related to m-learning.

Once again we hope and trust that reader will find these articles of significant value in their efforts to build, further develop, and sustain high quality online educational environments. Enjoy!


SECTION I: Massive Open Online Course (MOOC) Research

Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects

Heng Luo, Anthony C. Robinson, Jae-Young Park


Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects

Heng Luo

John A. Dutton E-Education Institute, Pennsylvania State University Anthony C. Robinson

Pennsylvania State University Jae-Young Park

Pennsylvania State University

ABSTRACTPeer grading offers a scalable and sustainable way of providing assessment and feedback to a massive student population. However, currently there is little empirical evidence to support the credentials of peer grading as a learning assessment method in the MOOC context. To address this research need, this study examined 1,825 peer grading assignments collected from a Coursera MOOC with the purpose of investigating the reliability and validity of peer grading, as well as its perceived effects on students’ MOOC learning experience. The empirical findings provide evidence that the aggregate of student graders can provide peer grading scores fairly consistent and highly similar to instructor grading scores. Student survey responses also indicate peer grading activities to be well received by a majority of MOOC students, whobelieve it was fair, useful, beneficial, and would recommend it to be included in future MOOC offerings. Based on the empirical results, this study concludes with a set of principles for designing and implementing peer grading activities in the MOOC context.

I. INTRODUCTIONThe recent development of Massive Open Online Courses (MOOCs) has provided instructors with exciting opportunities to teach to a massive and diverse student population through learning platforms such as Coursera, EdX, and Udacity. However, the large-scale participation and open access nature of MOOCs also present many pedagogical problems. One major problem relates to providing MOOC students with timely, accurate, and meaningful assessment of their course assignments since enrollment in a MOOC can be as large as hundreds of thousands of students (Pappano, 2012; Piech, et al., 2013), exceeding the grading capacity of a single instructor or teaching assistant. While automated grading software, like the one used by EdX, provide a potential solution for this problem, many MOOC assignments, such as design projects, art works, and essays, can be too complex to be graded by computers at this point in time. In an attempt to solve this assessment problem, Coursera has incorporated a peer review system in its learning platform that guides students in using grading rubrics to evaluate and provide feedback for each other’s work. While Coursera’s peer review system is informed by literature on peer review and crowd-sourcing (Coursera, n.d.), its reliability and validity as a learning assessment method in a MOOC


environment has yet to be verified, and little is known regarding how the peer grading practice can affect students’ MOOC learning experience. To address this research need, this study systematically investigates the peer grading process and results from a Coursera-based MOOC offered by The Pennsylvania State University (PSU) in 2013. Findings from this study provide empirical evidence on the reliability, validity, and perceived effects of MOOC-scale peer grading.

II. LITERATURE REVIEWA. Overview of Peer GradingPeer grading, also known as peer assessment, is defined by Topping (2009) as “an arrangement for learners to consider and specify the level, value, or quality of a product or performance of other equal-status learners” (pp. 20-21). Peer grading has been practiced in a wide range of subject domains, including natural sciences (Billington, 1997; Butcher et al., 1995), medicine (Hammond & Kern, 1959; Magin, 1993), social sciences (Falchikov, 1994; Orpen, 1982), engineering (Fry, 1990; Oldfield & Macalpine, 1995), and business (Freeman, 1995; Kaimann, 1974). Peer grading results typically take the form of a numeric rating, or a written comment—in many cases, a combination of both (Lu & Law, 2012, Strijbos, Narciss, & Dünnebier, 2010). In addition to reducing instructors’ workloads, peer grading is also believed to bring many potential benefits to student learning, including a sense of ownership and autonomy (Brown, Race, & Rust, 1995; Race, 1998), increased motivation (Bostock, 2000; Vu & Dall'Alba, 2007), enhanced social presence (Strijbos & Sluijsmans, 2010; Topping et al., 2000), and the development of higher-order thinking and metacognition skills (Brown, Rust, & Gibbs, 1994; Mok, 2011; Topping, 2009; Wen, Tsai, & Chang, 2006). Despite the potential benefits, peer grading still faces resistance from both students and instructors (Cho, Schunn, & Wilson, 2006; Magin, 2001; Stefani, 1994). Pre-conceived notions of low reliability and validity of peer grading is found to be one of the main reasons for such resistance (Falchikov & Goldfinch, 2000; McGarr & Clifford, 2013).

B. Reliability and ValidityThe reliability and validity of peer grading have been researched primarily in the context of face-to-face higher education (Cheng & Warren, 1999; Cho et al., 2006; Falchikov & Goldfinch, 2000; Stefani, 1994; Zhang, Johnston, & Kilic, 2008). Reliability is usually measured by the consistency of scores given by multiple student graders, and validity is commonly calculated as the correlation coefficient between student-assigned scores and instructor-assigned scores, assuming that instructors can provide fair and accurate grading results. In other words, reliability and validity discussed in this literature review should be considered as inter-rater reliability and convergent validity. Peer grading appears to be a valid learning assessment method, as many studies have reported a high correlation between student and instructor grading results. For example, Falchikov and Goldfinch (2000) conducted a meta-analysis of 56 studies on peer grading published between 1959 and 1999 and found a significant overall correlation between student-assigned scores and instructor-assigned scores (r = 0.69). Bouzidi and Jaillet (2009) and Sadler and Good (2006) further investigated peer grading in the contexts of online instruction and secondary education and found its validity to be high in both contexts (r = 0.88-0.91 and r = 0.91-0.94). However, contradictory evidence can also be found in the literature as incidences of low validity were reported in a few studies (Cheng & Warren, 1999; Korman & Stubblefield, 1971; Mowl & Pain, 1995). Contrary to the extensive body of literature on peer grading validity, there are few studies which calculated the inter-rater reliability of peer grading. The absence of such measurements undermine research findingsregarding peer grading validity because a valid assessment should almost always be reliable (Gay & Airasian, 2003), it also makes the interpretation of individual peer grading scores more difficult. Furthermore, researchers sometimes failed to differentiate the two concepts and misreported validity (i.e., students can provide accurate grading) as reliability (i.e., students can provide consistent grading)


(Topping, 1998). Based on patterns of how students grade each other’s work (e.g., the number of graders, the method of selection), researchers have used different metrics to calculate peer grading reliability, including Pearson product-moment correlation (Haaga, 1993), proportion of variance (Marcoulides & Simkin, 1995), and intraclass correlation (Cho et al., 2006; Miller, 2003). Statistical results show students can produce consistent and reliable grading scores. However, more empirical research is needed for any generalizable claims to be made about peer grading reliability. Factors influencing peer grading validity and reliability were also examined in the literature. Falchikov and Goldfinch (2000) found that using a single composite score to assess academic products based on given criteria improved the agreement between student graders and instructors, and thus identified grade structure, assignment type, and grading criteria as the factors affecting validity. Cho et al (2006) considered the number of student graders to be a key factor for reliability as the consistency of student-assigned scores can be significantly improved with the introduction of more graders. On the other hand, factors such as subject domain, course level, and student attitude were found to have limited impact on peer grading validity and reliability (Falchikov & Goldfinch, 2000; McGarr & Clifford, 2013). In summary, research findings in general support the legitimacy of peer grading and have identified a list of factors that might affect its reliability and validity. However, it is important to note such findings are mainly based on the context of traditional college degree courses with small or moderate enrollments and relatively homogenous student populations, and thus their applicability in the MOOC context remains largely unknown and in need of further research.

C. Peer Grading in MOOCsThe concept of crowd-sourcing grading activities to MOOC students has garnered a fair amount of attention from interested parties. Many educators and scholars have described their experiences with MOOC-scale peer grading from the perspective of either an instructor or a student, and there are ongoing conversations discussing its validity and effects in the popular press and on personal blogs (McEwen, 2013; Morrison, 2013; Neidlinger, 2013; Rees, 2013; Watters, 2012). Mixed findings regarding the fairness of peer grading in MOOCs have been reported. For example, Rees (2013) described her learning experience in a MOOC on world history. She admitted that she tended to get high grades for those assignments she worked hard on, and commented, “I think my peers graded my essays just right” (para 5). On the other hand, Neidlinger (2013) described the frustration felt by many MOOC students who believed their peers were not qualified to evaluate their assignments as they “don’t grade according to the rubric but according to their opinion” (para 5). McEwen (2013) and Watters (2012) further discussed additional problems facing peer grading in MOOCs, such as the varying quality of feedback, little sense of reciprocity and community, and lack of supervision and moderation. However, these assumptions about MOOC-scale peer grading have not been empirically verified, as none of them were based on the examination of real peer grading data.

III. RESEARCH CONTEXT AND QUESTIONSA. The Peer Grading AssignmentThe peer grading assignment examined in this study is the final assignment for a Coursera MOOCnamed Maps and the Geospatial Revolution (MGR) (www.coursera.org/course/maps), a 5-weekintroductory course on mapping and geospatial analysis offered by The Pennsylvania State University in 2013. MGR aims to teach students the key concepts in cartography, geographic information systems, and spatial thinking by having students work with contemporary mapping and analysis software to solve real-world geographic problems. There were 48,984 students who registered for the course, with 8,707 students remaining active in the last week of the course. According to self-reporteddemographic data for 7,551 of the MGR students, 70% of students were male and 30% were female. The average age of students was 36.5, and over 80% of students held post-secondary degrees (33.8% Bachelor’s degree, 39.1% Master’s degree, and 8% Ph.D. degree). About 61% of students reported working full-time, and roughly 30% of students resided in the United States at the time of the course. A total of 3,064 students passed the course (assignment completion rate over 70%), and 1,211 passed with distinction (assignment completion rate


over 90%). The instructor of MGR is the second author of this paper, and developed most of the course’s content, activities, and assignments, including the peer grading assignment in the final week. In this assignment, each student was required to select a mapping technology platform (e.g. ArcGIS Online, QGIS, and GRASS) and design a map that tells a story about a chosen topic. A possible story topic could be highlights from a recent travel experience, identification of good locations for favorite restaurants, or changes that occurred in a hometown in relation to its neighboring towns. The peer grading assignment accounted for 20% of the overall grade and was evaluated with a grading rubric consisting of four criteria: clarity of presentation, convincingness of the story, quality of cartography (e.g. color, symbology, and layout organization), and aesthetics of design. Each criterion was rated using a 4-point scale from 0 to 3, with the sum of the four criterion scores as the overall peer grading score. As a result, the score for the mapping assignment ranged from 0 to 12. Each student in MGR was required by the syllabus to grade three mapping assignments submitted by their peers. Many students, however, chose to grade more than three. The MGR instructor also required students to evaluate their own mapping assignments using the same rubric and provide a self-grading score. It is important to note Coursera uses median rather than mean to determine the final peer grading score for an assignment, which is calculated by the sum of all median scores of the rubric criteria (Coursera, 2014).

B. Research QuestionsTo extend our understanding of peer grading to the MOOC context, this study investigated the peer grading results and processes in MGR. More specifically, the following three research questions guided our research agenda:

Q1. Can peer grading provide a reliable and valid assessment of student assignments in a Coursera MOOC?

Q2. Does the use of median score provide a more valid assessment than the use of mean score when calculating the final peer grading scores?

Q3. What are the perceived effects of peer grading on students’ MOOC learning experience?

IV. METHODSA. Data SourceThe primary data source in this study is the relational database used internally by Coursera containing all of the instructor-provided and student-generated content in MGR, including website content, copies of submitted assignments, peer grading scores and feedback, public forum data, and logs of learning activities. Upon exporting data from the database, personally identifiable information was removed and an anonymized 40-character hexadecimal identifier was assigned to identify each student. The portion of data on peer grading was organized in the structure shown in Figure 1. The submission_metadata contains the most important information regarding the mapping assignment, such as the submission ID linking back to the actual student work, the final peer grading score (reported in overall_evaluation_metadata), and the five individual peer grading scores (reported in evaluation_metadata). Additional information such as submission time and completion status can be found in peer_grading_set_metadata. Students’ self-grading results (total score and four criterion scores) arestored in self_grading_metadata. Since MGR did not include a peer grading training session, there is no data in training_set_metadata. The instructor required each student to grade at least three assignments, with many students opting to grade more than three. As a result, there are a total of 1,825 assignment submissions that each received five peer grades. Only fully graded assignments with five peer grading scores (N=1,825) were selected for data analysis in this study, and the assignments with missing peer grading scores were excluded (N=919).


Figure 1. The structure of peer grading assignment data in Coursera’s database

Besides the secondary data exported from Coursera’s database, the MGR instructor also manually graded a5% sample of peer graded assignments (N=93) randomly selected from the 1,825 submissions. The instructor used the assignment submission IDs to identify the actual student works and evaluated them using the same grading rubric. The instructor assigned a score for each criterion, with the sum of the four criterion scores the final grading score. As a result, a selected mapping assignment (Xassign) has the following attributes: five individual peer grading scores (Xpeer1-5), one final peer grading score using median (Xmedian), one final peer grading score using mean (Xmean), one instructor grading score (Xinst), and one self-grading score (Xself), as shown in Figure 2.

Figure 2. Attributes of a submitted peer grading assignment

Another data source for this study is the end-of-course survey which asked students to rate their MOOC learning experience. Seven 5-point Likert-scale questions were built into the survey to collect students’ opinions about the fairness, usefulness, and potential benefits (e.g. learner engagement, social presence, and higher-order thinking) of the peer grading activities in MGR. The default survey tool of Coursera was not used in this study since all Coursera-based survey data are stored in an unstructured key value store in the database and thus could pose great difficulty for data extraction. Instead, this study employed an external online survey tool named Qualtrics to develop and administer the end-of-course survey, with Coursera user IDs having been passed into the Qualtrics-based survey.


r2

22

/2/

//

)N

)X(X)(N

)X(X(

NXXXX

instinst

meanmedianmeanmedian

instmeanmedianinstmeanmedian

B. Data AnalysisIn order to answer the three research questions proposed in this study, the data analysis focused on the following three aspects: (1) calculating the inter-rater reliability of peer grading scores submitted to Coursera’s peer review system, (2) calculating and comparing the convergent validity of peer grading scores based on the median and the mean, and (3) examining the perceived effects of peer grading activities on students’ MOOC learning experience. The reliability of peer grading in this study is inter-rater reliability, measured by the general agreement among the student graders assigned to grade the same assignment. Because the mapping assignment was graded by five randomly selected students for the given student pool, case1 intraclass correlation coefficient [ICC (1)] was selected as the appropriate statistical model to calculate the rater agreement (absolute agreement) in this situation (Shrout & Fleiss, 1979). Mathematically, this model can be formulated as:

In this model, ICC (1) is used to estimate the reliability of MOOC-scale peer grading, and the variance among student graders and the grader-assignment interaction are viewed as the measurement errors. The calculation of peer grading reliability was conducted using SPSS, as ICC (1) is known as one-way random agreement measure in SPSS for reliability analysis. The validity of peer grading in this study is convergent validity, measured by the similarity between the final peer grading scores and the instructor grading scores, which is calculated as Pearson product-moment correlation coefficient (r). Two types of final peer grading score were examined: the final score determined by the median (as used in Coursera’s peer review system) and the final score determined by the mean (as calculated in this study). The following is the mathematical model for computing peer grading validity (Pearson’s r) for both types. The computation was executed in SPSS by selecting two-tailed Pearson correlation coefficient for bivariate correlation. By comparing the computation results using the median score and the mean score, this study is able to determine which type of peer grading score has yielded higher validity as an assessment.

Students’ responses to the seven survey questions were downloaded from Qualtrics and were imported into SPSS for descriptive analysis. The descriptive statistics (e.g. mean, frequency) of the survey data provide atallied summary of students’ overall attitude towards the peer grading assignment in the MOOC and their perceptions of whether peer grading activities have positively influenced their MOOC learning experience in terms of engagement, social presence, and higher-order thinking, as suggested by the literature.

assign

assignpeerpeerpeerassignpeerpeerassign

assignpeerassign

NMSMSN

MSdfMS

MSMSICC )(

)()1(


V. RESULTS A. Inter-Rater Reliability The inter-rater reliability of peer grading scores was calculated using ICC [1] and the statistical results are presented in Table 1. The Single Measures ICC estimates the inter-rater agreement among the five randomly selected student graders when grading the same assignment. The coefficient value of .262 is considered to be low in strength, indicating peer grading scores tend to vary greatly among individual students and a single student’s grading score is not very reliable. Compared to the Single Measures, the Average Measures ICC (.64) shows moderate strength, suggesting the reliability of peer grading scores can be enhanced if the mean of the five individual scores is used as an index of measurement. To further determine the source of error, the random criteria scenario (i.e., 5 random nested raters and 4 random crossed criteria) was conducted. Results show that the generalizability coefficient remains the same (.64) and the standard error of measurement increases only slightly from .272 to .276. Therefore, there is little room for improvement on the rubrics and scoring criteria, and the source of error is basically student graders.

Table 1. Intraclass Correlation Coefficient (Case 1) for Peer Grading Scores (N=1825)

Intraclass Correlation

95% Confidence Interval F Test with True Value 0Lower bound

Upper bound

Value df1 df2 Sig

Single Measures .262 .240 .284 2.774 1824 7300 .000Average Measures .640 .613 .665 2.774 1824 7300 .000

This study also calculates the ICCs for sub-scores assigned to the four grading criteria with the purpose of finding out how reliability measurements might differ due to the varying complexity of grading tasks.These statistical results are presented in Table 2. As can be seen, the Single Measures ICCs are low for all four grading criteria, and the biggest grading disagreement is on the evaluation of cartography quality. The Single Measures ICC for this criterion is only .176. Using the mean score rather than the individual scores can greatly increase the reliability of peer grading scores assigned to a specific criterion, as the Average Measures ICCs for the four criterion scores are between .516 and .579, a significant improvement to the Single Measures ICCs.

Table 2. Intraclass Correlation Coefficients (Case 1) for the Four Criterion Scores (N=1825)

Clarity of the Presentation

Convincingness of the Story

Quality of the Cartography

Aesthetics of the Design

Single Measures .216 .215 .176 .210Average Measures .579 .578 .516 .571

In order to examine how many peer grading scores are needed to generate a composite score with acceptable inter-rater reliability, this study also examines ICCs based on the varying numbers of studentgraders selected for calculation (2-5 graders). As shown in Table 3, the number of student graders has a large effect on Average Measures ICCs, and an increase in graders can generate more reliable grading results. On the contrary, the impact of total graders on Single Measures ICCs is quite limited. Dancey and Reidy (2002) suggested that correlation coefficient between .40 and .69 should be considered as being moderate in strength. As a result, it seems at least three student graders are necessary to produce a composite score with moderate inter-rater reliability (correlation coefficient > .40), whereas peer grading scores based on only two graders tend to be less reliable.


Table 3. Intraclass Correlation Coefficients (Case 1) for Different Number of Student Graders (N=1825)

5 graders 4 graders 3 graders 2 gradersSingle Measures .262 .256 .256 .241Average Measures .640 .580 .508 .389

B. Convergent ValidityThis study assumes the MGR instructor can provide the true score for a submitted assignment; therefore, the validity of peer-grading scores and self-grading scores can be determined by their similarity to instructor-assigned scores, measured by the strength of bivariate correlation. As shown in Table 4, there is a strong, positive correlation (r = .619) between the instructor grading scores and the median-based peer grading scores, indicating Coursera’s peer review system can provide similar scores to those assigned by the course instructor. This correlation coefficient is slightly increased (r = .662) when mean scores ratherthan median scores are used to calculate the final peer grading scores. However, the difference in correlation coefficient between median-based and mean-based peer grading scores is inconsequential, and the two types of peer grading scores are also highly correlated with each other (r = .952). Compared to the two types of peer grading scores, students’ self-grading scores seem to be a less valid assessment of the mapping assignment, as the correlation between the self-grading scores and the instructor grading scores is found to be only moderate (r = .341). The descriptive analysis also reveals that the mean of self-grading scores ( =10.02) is higher than the means of instructor grading scores ( =8.68), median-based peer grading scores ( =9.194), and mean-based peer grading scores ( =9.103). This result shows that students tend to give higher scores when evaluating their own assignments, and the scores given by the MOOC students in general are higher than those given by the instructor.

Table 4.Pearson’s Correlation Coefficient between Instructor, Peer, and Self Grading Scores (N=93)

instructor_grading peer_grading_median peer_grading_mean self_gradinginstructor_grading 1 .619** .662** .341**peer_grading_median 1 .952** .279**peer_grading_mean 1 .464**self_grading 1** Correlation is significant at the 0.01 level (2-tailed)

C. Perceived EffectsTable 5 summarizes students’ ratings of the seven survey questions regarding the peer grading activity in MGR. Missing responses for each survey question were excluded from the descriptive analysis. As shown in Table 5, about 63% of students believed the peer grading activity was helpful in developing their spatial thinking competencies, which was the main instructional goal of the course. The majority of students feltthey received fair grades (62%) and useful feedback (61%) from their peers. Consistent with what the literature suggests, students in general agreed that the peer grading activity benefited their MOOC learning experience due to enhanced learner engagement (63%), an increased sense of social presence (57%), and the added opportunity of higher-order thinking (72%). As a result, about 70% of the students stated they would recommend the peer grading assignment to be included in future offerings of MGR.


Table 5. Students’ Ratings of the Seven Statements Regarding the Peer Grading Activity in MGRQuestion Statements (1-5 from strongly disagree to strongly agree)

N 1 2 3 4 5 Positive* Mean

The peer review process helped me to grow and improve as a spatial thinker.

2121 83 137 562 950 389 63% 3.67

I received fair grades on my Lesson 5 Mapping Assignment from my peers.

1694 56 90 505 607 436 62% 3.75

The feedback my peers gave me on my Lesson 5 Mapping Assignment was useful.

1719 59 106 509 677 368 61% 3.69

The peer grading activity in this course made me feel more engaged in the course.

2027 97 165 489 780 496 63% 3.70

The peer grading activity in this course made me feel more connected with other students.

2039 111 191 572 760 405 57% 3.57

The peer grading activity provided me with opportunities to review and/or reflect on course content.

2044 75 90 413 898 568 72% 3.88

I would recommend keeping the peer grading assignment in future offerings of this course.

2185 111 126 420 732 796 70% 3.90

* Agreed or strongly agreed responses from the students are considered as positive responses

VI. DISCUSSIONA. Research Question One DiscussionTo answer Research Question One, the inter-rater reliability of peer grading scores assigned by individual MOOC students was found to be rather low, and large variance among peer-assigned scores should be expected. It is not surprising to find that the source of error is individual student graders rather than the grading criteria, considering MOOC students can be from different backgrounds and vary greatly in terms of knowledge and skills needed for providing accurate evaluation, and no training on grading is typically provided to the students. In this study, all selected assignments were graded by five students, and we found that the reliability of peer grading results can be largely improved when all five grading scores were averaged to create a composite score, as Average Measures ICC is much higher than Single Measures ICC. One easy way to increase the peer grading reliability is to assign more student graders to the assignments, as we found the number of graders to be positively correlated with the reliability measurement (Average Measures ICC), and the use of at least three graders to generate moderately reliable grading scores. Thisfinding is also consistent with what Cho et al. (2006) suggested in their study: the use of multiple graders (4-6 graders) is necessary to achieve satisfactory levels of reliability. It is also interesting to note that the grading criterion with the lowest inter-rater agreement is quality of cartography. One possible explanation is that the evaluation of cartography quality is more closely related to the specific course content taught in the MOOC (e.g. color selection, layout design, symbolization, and data classification), and thus is more likely to suffer from MOOC students’ varying levels of prior knowledge and learning outcomes. The empirical findings in this study also support the validity of Coursera’s peer review system as an assignment assessment tool. Coursera has taken into consideration MOOC students’ diversity and unpredictability, and attempts to counter the influence of outlier scores by using median values as the final score. The .619 correlation coefficient between Coursera’s final peer grading scores and the scores assigned by the MOOC instructor shows that the peer review system in general can provide grading results similar towhat an instructor would provide. It is also interesting to find that using the mean score as the final peer grading score can provide equally valid assessment in this study. The choice of mean or median as peer grading scores is discussed in detail in the next section. While Coursera’s peer grading results might never be as accurate as instructor grading, they yielded much higher validity than simply having MOOC students evaluate their own works since self-grading scores are found to only moderately correlate with the instructor grading scores (r = .341) and tend to get inflated.Therefore it is unwise to dismiss the validity of peer grading simply because of MOOC students’ unverified


credentials and to substitute peer grading with self-grading when designing a MOOC.

B. Research Question Two DiscussionThe results of this study actually show that the use of mean scores would yield slightly higher validity than the use of median scores when calculating the peer grading results, as the validity calculations for mean-based and median-based peer grading scores are .662 and .619 respectively. Statistically, median is more effective in dealing with extreme scores, which are expected to occur more often in the MOOC context due to MOOC students’ varying capability and/or motivation to grade. However, median is also a less representative average, as it is calculated based on only one or two students’ grading scores rather than all five, therefore two completely different score distributions might have the same median. In other words,mean is probably a better measurement of average than median when there are not many outlier scores, which happened to be the case in this study. Figure 3 shows how individual peer grading scores differ from the instructor grading scores based on the 5% randomly selected mapping assignments (N=93). It turns out most scoring differences are within 3 points, and outlier scores (5 points or more difference) were minimal.This might explain why mean-based peer grading score turns out to be a slightly better assessment for this MOOC. However, such finding should not be over-generalized as other MOOCs might have more frequent outlier scores where the use of median would be better.To explore the possible causes for the outlier scores, we selected four assignment submissions with the largest scoring differences for further examination. These assignments are submissions No.17, No.23, No.64, and No.84 as shown in Figure 3. For submissions No.23 and No.64, the assignments were no longer viewable to the instructor at the time of grading (a few months after the course) and the instructor had to assign zero scores to both of them. The fact that those two submissions received fairly consistent high scores from all five student graders made us believe that the authors had posted their works to onlineplaces that no longer exist or they removed their submissions intentionally after the MOOC ended. Submissions No.17 and No.84 revealed a different situation: In both cases, the instructor-assigned score issimilar to the scores assigned by the student graders 2 points), except for one student who gave extremely higher or lower scores. It is not surprising to find this type of grader in a MOOC, who always assigns high, low, or random scores regardless of the assignment quality, since there is no way to hold students accountable for the quality of their peer grading services in the course. The statistical solution to this problem is the use of median to counter the influence of outliers, but a more effective solution might be educating MOOC students to be more responsible and/or establishing a mechanism to review students’ grading performances.

Figure 3. Difference between scores assigned by student graders and the instructor


C. Research Question Three DiscussionTable 5 shows that the MGR students’ general perception towards the peer grading activity in the course ispositive. The majority of MOOC students (70%) wanted to keep the peer grading assignment in futurecourse offerings, compared to only 11% who wanted it removed. Although in the news media there aremany MOOC instructors and students expressing their concerns about the fairness of peer grading and the quality of peer feedback (McEwen, 2013; Morrison, 2013; Neidlinger, 2013; Watters, 2012), the survey results in this study told a different story: the percentage of students who complained about the fairness and usefulness of the peer grading assignment was below 10%, and over 60% of students indicated they had received fair grades and useful feedback from their peers. It is possible that students who had negative experiences with peer grading are more vocal in expressing their complaints, but such complaints might not be representative of the overall MOOC student population. The survey responses also supported the additional benefits of peer grading activities—the most recognized benefit being the opportunity for students to review and/or reflect course content. This finding can be justified using Bloom's Taxonomy of Learning Domains (Bloom, 1956), as peer grading is a form of evaluation, which is considered a higher level cognitive activity in the taxonomical hierarchy that promotes meaningful learning.

VII. CONCLUSIONSOur work here has shown a variety of ways to evaluate the reliability (i.e., inter-rater reliability), validity(i.e., convergent validity), and perceived effects of peer assessment in the context of map design projects in a MOOC. These results suggest that in general, the joint efforts of multiple student graders can produce fairly consistent grading results using Coursera’ peer review system. There were also high levels of agreement between student-assigned scores and instructor-assigned scores measured by the correlation coefficients, which support the validity of peer grading in the MOOC context. The post-course surveyresponses reveal students in general consider the peer grading activity to be a positive learning experience and would recommend keeping this component of the course in future offerings. MOOC students especially appreciate the review and reflection learning experiences associated with the peer grading activity. Based on the empirical findings in this study, we proposed a set of principles for designing and implementing peer grading activities in MOOCs. While the design principles are context-specific and should not be over-generalized, we expect them to offer insights to MOOC instructors and designers and inform future MOOC design practices. The proposed principles are listed below:

1. Peer grading should not be replaced by self-grading, as peer grading results tend to be more validthan self-grading results.

2. It is advisable to use Coursera’s default peer review system, as it can provide valid peer gradingresults and reduce the influence of outlier scores. However, when outlier scores are rare,mean-based peer grading scores might be a better alternative.

3. The instructor/designer should try to assign a sufficient number of student graders to eachsubmission to increase the reliability of peer grading scores. A good rule of thumb is 3 to 5 studentgraders.

4. In order to increase the reliability of peer grading scores, proper training on assignment evaluationshould be provided to MOOC students prior to the grading activity, since student graders are themain source of error for peer grading.


VIII. REFERENCESBillington, H. L. (1997). Poster presentations and peer assessment: novel forms of evaluation and

assessment. Journal of Biological Education, 31(3), 218-220.

Bloom, B. S. (1956). Taxonomy of educational objectives: Vol. 1. Cognitive domain. New York: McKay.

Bostock, S. (2000). Student peer assessment. Keele, Staffordshire: Centre for Learning Technology, Keele University. Retrieved fromhttp://www.reading.ac.uk/web/FILES/engageinassessment/Student_peer_assessment_-_Stephen_Bostock.pdf

Bouzidi, L., & Jaillet, A. (2009). Can Online Peer Assessment be Trusted? Educational Technology & Society, 12 (4), 257–268.

Brown, S., Race, P., & Rust, C. (1995). Using and experiencing assessment, in P. Knight (Ed.) Assessment for Learning in Higher Education (pp.75-85). London: Kogan Page/SEDA.

Brown, S., Rust, C., & Gibbs, G. (1994). Strategies for diversifying assessment in higher education.Oxford: Oxford Centre for Staff Development.

Butcher, A. C., Stefani, L. A. J., & Tariq, V. N. (1995). Analysis of peer-, self- and staff-assessment in group project work. Assessment in Education, 2(2), 165-185.

Cheng, W., & Warren, M. (1999). Peer and teacher assessment of the oral and written tasks of a group project. Assessment & Evaluation in Higher Education, 24, 301–314.

Cho, K., Schunn, C., & Wilson, R. (2006). Validity and Reliability of Scaffolded Peer Assessment of Writing from Instructor and Student Perspectives. Journal of Educational Psychology, 98(4), 891-901.

Coursera. (n.d.). Pedagogical Foundations. Retrieved from https://www.coursera.org/about/pedagogy

Coursera. (2014, March 12) How will my grade be determined? Retrieved from. http://help.coursera.org/customer/portal/articles/1163304-how-will-my-grade-be-determined-

Dancey, C. P., & Reidy, J. (2002). Statistics without maths for psychology (2nd ed). London: Prentice Hall.

Falchikov, N. (1994). Learning from peer feedback marking: student and teacher perspectives. In H. C. Foot, C. J. Howe, A. Anderson, A. K. Tolmie, & D. A. Warden (Eds.), Group and interactive learning (pp. 411-416). Southampton and Boston: Computational Mechanics Publications.

Falchikov, N., & Goldfinch, J. (2000). Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks. Review of Educational Research, 70(3), 287-322.

Freeman, M. (1995). Peer assessment by groups of group work, Assessment and Evaluation in Higher Education, 20(3), 289-300.

Fry, S. A. (1990). Implementation and evaluation of peer marking in higher education. Assessment and Evaluation in Higher Education, 15(3), 177-189.

Gay, L. R., & Airasian, P. (2003). Educational research: Competencies for analysis and application (7th ed.). Columbus, OH: Merrill, Prentice Hall.

Haaga, D. A. F. (1993). Peer review of term papers in graduate psychology courses. Teaching of Psychology, 20(1), 28–32.

Hammond, K. R., & Kern, F. (1959). Teaching comprehensive medical care: a psychological study of a change in medical education. Cambridge, MA: Harvard University Press.


Kaimann, R. A. (1974). The coincidence of student evaluation by professor and peer group using rank correlation. The Journal of Educational Research, 68(4), 152-153.

Korman, M., & Stubblefield, R. L. (1971). Medical school evaluation and internship performance. Journal of Medical Education, 46, 670-673.

Lu, R., & Bol, L. (2007). A comparison of anonymous versus identifiable e-peer review on college student writing performance and the extent of critical feedback. Journal of Interactive Online Learning, 6(2), 100-115.

Lu, J., & Law, N. (2012). Online peer assessment: effects of cognitive and affective feedback. Instructional Science, 40(2), 257-275.

Magin, D. (1993). Should student peer ratings be used as part of summative assessment? Higher Education Research and Development, 16, 537-542.

Magin, D. (2001). Reciprocity as a source of bias in multiple peer assessment of group work. Studies in Higher Education, 26(1), 53–63.

Marcoulides, G. A., & Simkin, M. G. (1995). The consistency of peer review in student writing projects. Journal of Education for Business, 70, 220–223.

McEwen, K. (2013, January 7). Getting to Know Coursera: Peer Assessments. Retrieved from http://cft.vanderbilt.edu/2013/01/getting-to-know-coursera-peer-assessments/

McGarr, O., & Clifford, A. M. (2013). ‘Just enough to make you take it seriously’: exploring students’ attitudes towards peer assessment. Higher education, 65(6), 677-693.

Miller, P. J. (2003). The effect of scoring criteria specificity on peer and self-assessment. Assessment & Evaluation in Higher Education, 28(4), 383-394.

Mok, J. (2011). A case study of students' perceptions of peer assessment in Hong Kong. ELT journal, 65(3), 230-239.

Morrison, D. (2013, March 9). Why and When Peer Grading is Effective for Open and Online Learning.Retrieved fromhttp://onlinelearninginsights.wordpress.com/2013/03/09/why-and-when-peer-grading-is-effective-for-open-and-online-learning/

Mowl, G., & Pain, R. (1995). Using self and peer assessment to improve students’ essay writing—A case study from geography. Innovations in Education and Training International, 32, 324–335.

Neidlinger, J. (2013, May 13). Does peer grading of essays really work in a Coursera online class?Retrieved from http://loneprairie.net/peer-grading-coursera/

Oldfield, K. A., & Macalpine, M. K. (1995). Peer and self-assessment at tertiary level - an experimental report. Assessment and Evaluation in Higher Education, 20(1), 125-131.

Orpen, C. (1982). Student versus lecturer assessment of learning: a research note. Higher Education, 11,567-572.

Pappano, L. (2012, November 2). The Year of the MOOC. The New York Times. Retrieved from http://www.nytimes.com/2012/11/04/education/edlife/massive-open-oline-courses-are-multiplying-at-a-rapid-pace.html?pagewanted=all&_r=0

Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013). Tuned Models of Peer Assessment in MOOCs. Retrieved from http://www.stanford.edu/~jhuang11/research/pubs/edm13/edm13.pdf

Race, P. (1998). Practical pointers on peer-assessment. In S. Brown (Ed.) Peer Assessment in Practice (SEDA Paper 102) (pp.113-122). Birmingham: SEDA.


Rees, J. (2013, March 5). Peer Grading Can't Work. Retrieved from http://www.insidehighered.com/views/2013/03/05/essays-flaws-peer-grading-moocs#ixzz2MiKxNP7b

Sadler, P., & Good, E. (2006). The impact of self- and peer-grading on student learning. Educational Assessment, 11 (1), 1-31.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428.

Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education,19(1), 69–75.

Strijbos, J. W., Narciss, S., & Dünnebier, K. (2010). Peer feedback content and sender's competence level in academic writing revision tasks: are they critical for feedback perceptions and efficiency? Learning and instruction, 20(4), 291-303.

Strijbos, J. W., & Sluijsmans, D. (2010). Unravelling peer assessment: Methodological, functional, and conceptual developments. Learning and Instruction, 20(4), 265-269.

Topping, K. J. (2009). Peer assessment. Theory into Practice, 48

Topping, K. J., Smith, E. F., Swanson, I., & Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25(2), 149-169.

Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249-276.

Vu, T. T., & Dall’Alba, G. (2007). Students’ experience of peer assessment in a professional course. Assessment & Evaluation in Higher Education, 32(5), 541-556.

Watters, A. (2012, August 27). The Problems with Peer Grading in Coursera. Retrieved from http://www.insidehighered.com/blogs/hack-higher-education/problems-peer-grading-coursera

Wen, M. L., Tsai, C. C., & Chang, C. Y. (2006). Attitudes towards peer assessment: A comparison of the perspectives of pre-service and in-service teachers. Innovations in Education and Teaching International, 43(1), 83–92.

Zhang, B., Johnston, L., & Kilic, G. B. (2008). Assessing the reliability of self-and peer rating in student group work. Assessment & Evaluation in Higher Education, 33(3), 329-340.


SECTION II: Online Discussion Analyses

Good Quality Discussion is Necessary but Not Sufficient in Asynchronous Tuition: A Brief Narrative

William James Fear, Andrew Erikson-Brown

An Exploration of Metacognition in Asynchronous Student-Led Discussions:A Qualitative Inquiry

Martha M. Snyder, Laurie P. Dringus

The Effect of Structured Divergent Prompts on Knowledge Construction

Ginger Sue Howell, Autumn Sutherlin, Usen Akpanudo, Laura James, Mengyi Chen

Differences in Classroom Versus Online Exam PerformanceDue to Asynchronous Discussion

Dr. Robert Jorczak, Danielle N. Dupuis


Good Quality Discussion is Necessary But Not Sufficient in Asynchronous Tuition: a Brief Narrative

Review of the Literature

William James FearDepartment of Organizational Psychology Birkbeck College University of London

Andrew Erikson-BrownDepartment of Organizational Psychology Birkbeck College University of London

ABSTRACTThe growth of online learning within education has corresponded to an increase in use of asynchronous discussion. Asynchronous discussion is a form of interaction that is mediated rather than directed, and is characterized by a time lag in the interactions between discussants. In this paper we conducted a brief narrative review of the literature on asynchronous discussion. We argue, initially, that discussion is necessary, but not sufficient, for successful pedagogic outcomes—especially in the case of online learning. We identified areas of agreement within the literature on what can be considered the key factors for successful asynchronous discussion.

I. INTRODUCTIONThe use of the Internet to expand distance learning and create non-traditional pathways to Higher Education is expanding rapidly. It is possible to achieve a large range of undergraduate and post-graduate degrees through distance learning that places a high level of reliance on web-based interaction with content, tutors, and peers (fellow students). While higher rated and more strongly accredited qualifications tend to use blended learning, most rely strongly on asynchronous discussion. In this paper we consider the role of web-based asynchronous discussion as a critical factor of online learning within Higher Education (Cantor, 1992; Dennen, 2005, 2008; Henri, 1995; Kanuka & Anderson, 1998; Salmon, 2000). We take the position that high quality discussion is a necessary, but not sufficient, factor in producing high level pedagogical outcomes. The ‘problem’ with asynchronous discussion is precisely that it is asynchronous and lacks the affective immediacy of face-to-face interaction. The question posed to guide the present review is how instructors can best manage asynchronous discussion. For example: is a high level of lecturer engagement better or worse; how directive should the lecturer be in the discussion; does asynchronous discussion favor the ‘sage on the stage’ or the ‘guide on the side’; and so on (Andresen, 2009; York & Richardson, 2012). Given the breadth and general nature of the question, we conducted a narrative literature analysis (Baumeister & Leary, 1997). To select articles, we focused specifically on those directly related to asynchronous discussion. However, we did not want to be drawn into comparisons and discussion relating to, for example, blended learning and considerations such as the use of additional technologies such as


Skype and other means of web-based interaction (Brown, 2012; Junk, Deringer & Junk, 2011; Lim, 2004; McGee & Reis, 2012). We will discuss the methodology for the review briefly, and then consider the context and other factors related to our question. We will make the point that the single difference between asynchronous discussion and traditional forms of discussion is the lag in time between interactions and the use of written text—written by students and instructors—rather than verbal components. We then discuss what we found to be the key factors for successful asynchronous discussion noted in the literature. By successful we mean leading to recognizable pedagogical outcomes (Andresen, 2009; Brown, 2012; Crawley & Fetzner, 2013; Fetzner, 2013; Hew & Cheung, 2003). In reviewing the literature we began by reading widely on the topic and learning to identify and select papers that were germane to our questions either in part or whole. We selected reading from mainstream journals that cover asynchronous learning and searched databases such as EBSCO, ERIC, and PSYCHLIT for the term asynchronous discussion/learning. We also sought direction from credible web-based sources such as the Community of Inquiry group, the Sloan Consortium, and the Higher Education Academy. We read until we reached the point where no new information was forthcoming (similar to qualitative saturation), although we recognized that details and nuances continued to emerge. We selected a small body of papers (c. 55 papers excluding grey literature and web-based material, circa 70 including grey literature; not all of these are cited here. Note: Grey literature is defined by the New York Academy of Medicine as "That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers." It is commonly referred to in narrative reviews and includes sources such as include institutional websites, professional level reports, policy documents.) which we felt presented the clearest and most concise answer to our question: what are the core considerations for conducting asynchronous discussion that produces high level pedagogical outcomes? Selection was a matter of personal and professional judgment based on prior reading. This seemed appropriate given we were seeking a ‘broad brush answer’ to an interesting question and were not attempting to develop theory. Of course, there is the matter of how to recognize good quality discussion. The literature recognizes success factors such as retention, good grades, high levels of interaction, quality of student interaction (using specific metrics), and so on (Andresen, 2009; Brown, 2012; Crawley & Fetzner, 2013; Fetzner, 2013; Hew & Cheung, 2003). This provided a benchmark against which we exercised professional judgment. Papers reporting the means to deliver good tuition were benchmarked against papers that simultaneously report both the means to deliver good tuition as well as success factors. In addition, we selected papers using theories and models to generalize findings (Armstrong & Thornton, 2012; Kirby, Moore & Schofield, 1998; Moule, 2007). Once we had a small set of papers with consistency we collated key reported factors and grouped factors accordingly. Once the factors were grouped, they were then summarized. We note that following our approach the factors emerged naturally in much the same way as themes emerge in a thematic analysis or grounded theory approach. The difference here was that we did not need to make a judgment about selecting the factors or naming the factors as they were already identified and named in the literature. Following this it became apparent that there was an unexpected theme in the literature, particularly for papers in JALN, regarding students’ perspectives (Crawley & Fetzner, 2013; Fetzner, 2013; Parisio, 2011; Vonderwell, 2003). A selection of papers was further culled, and the same process followed, in order to identify the necessary and sufficient factors from students’ perspectives. Student success factors for asynchronous discussion tend to be the same as student success factors in face-to-face situations with one exception: in the asynchronous setting there is no way to catch up. Thus we were able to combine theory driven observation of factors for successful asynchronous discussion in the literature with: a) reported observations of student behavior during successful asynchronous tuition; and b) reported observations from institutions with a record of success in asynchronous tuition (an unexpected finding that emerged during the reading of the papers on student perspectives and behaviors observed in colleges with high retention rates).


A. Context and other factorsMany of the principles of asynchronous discussion, the methods of pedagogy, are no different than those for face-to-face discussion (Andresen, 2009; Huang & Hsiao, 2012; Parisio, 2011; Tu & Corry, 2003; Vonderwell, 2003) —for example, the use of case studies and collaborative assignments. That is, from a constructivist perspective, we need to provide the structures, activities and guidelines that facilitate learning through student-student interaction. The difference with asynchronous learning is four-fold: 1) clearly, it is not face-to-face; 2) there is a time lag in the interactions; 3) interactions take place through the medium of text rather than verbal discourse; and 4) students may be in distant and separate geographical locations. These factors, when combined, create subtle and nuanced distinctions that may not apply, for example, to blended learning scenarios where students have the opportunity, albeitsomewhat limited, to interact with faculty and other students. Equally, there are predictable success factors in the form of, for example, institutional commitment to student success which can be enacted across all facilities and services throughout the institution and owned by key players (Moore & Fetzner, 2004). Similarly, knowing your students is critical in terms of the characteristics of students attending the institution rather than on a single course. Some general principles emerged relating specifically to the facilitation of asynchronous discussion in a teaching environment. There are some contradictory findings, such as the debate between keeping the discussion focused and allowing for, and even encouraging, divergent discussion (Beaudin, 1999; Cantor, 1992; Jorczak, 2011; Ugoretz, 2005; Winiecki & Chyung, 1998), and the number of times either the tutor or student should post each week (Pelz, 2004; Zingaro & Oztok, 2012). We have not engaged with these debates but have grouped and summarized key factors identified in relation to facilitating discussion as a non-traditional lecturer (NTL). Arguably the purpose of facilitated discussion in Higher Education is to create a community of inquiry as a means for entry into a community of practice (Kear, 2004). Membership within a community of practice requires familiarity with, and practice in the use of, the artifacts of that community.One of the key artifacts is vocabulary. Vocabulary allows participants access to shared concepts, meanings, and understandings. Practice with the vocabulary is developed through appropriate discussion and the application of that vocabulary in authentic situations (Kear, 2004; Kirby, Moore & Schofield, 1998; Moore & Fetzner, 2004). The NTL introduces students into the community of practice by facilitating student learning of language, vocabulary, and constructs used by the community of practice. This learning is developed by designing and structuring the space in which a community of inquiry can develop through facilitated peer-peer interaction and task oriented collaboration. A necessary factor for interaction and collaboration is discussion. Intuitively it would seem likely that discussion is potentially more difficult in asynchronous settings because of time delays, lack of access to guardians and gatekeepers, and so on. However, we did not find anything to support this intuition. Design, strategy, structure and so on all match those for face-to-face with some slight differences largely due to time lag and the physical logistic realities (e.g. you can’t get everybody together in the same location at the same time and have a synchronous discussion etc.). Courses and activities should be designed to maximize interaction using a high number of collaborative activities. Emphasis should be placed on their relevance to everyday activity and interaction with other external resources such as websites, YouTube, Twitter, and so on. Problem solving tasks related to course objectives should be used to focus activities.

II. FACTORS FOR SUCCESSFUL ASYNCHRONOUS DISCUSSIONA. PresenceThe presence of the NTL is a critical factor in asynchronous discussion, but that presence also needs to be


restrained (Kear, 2004). This is the same distinction as in face-to-face environments between the ‘sage on the stage’ and the ‘guide on the side’ heuristics. Creating an appropriate online presence is balancing act—there is the requirement for a social presence, a cognitive presence, and a teaching presence (Dennen, 2005; Garrison, Cleveland-Innes & Fung, 2010; Kanuka & Anderson, 1998; Lee, 2014; Sloane Consortium; Wan, 2008). Developing a social presence requires the NTL to interact, to bring cohesion to the group, and to express and respond appropriately to expressed emotion. A cognitive presence is developed through the demonstration of factual, theoretical and conceptual knowledge. The teaching presence is probably the most familiar. The NTL facilitates discussion, identifies agreement and disagreement, promotes consensus and understanding, and encourages discussion by acknowledging contributions and drawing in participants. However, a somewhat counterintuitive and uncomfortable position for the NTL is that they should not be drawn into settling debates. Rather, they should set the boundaries, guidelines and rules for discussion but need to allow students opportunities to reach agreement and consensus on their own terms(Andresen, 2009).

B. Threaded postsAsynchronous discussion typically takes the form of threaded posts (a topic which has been the subject of some debate). The principles of using threaded posts are the least controversial and most straightforwardand the technique of requiring a meaning summary in the subject line is widely supported (Winiecki & Chyung, 1998). While regular posting is crucial, quality should be encouraged over quantity. That having been said, it has been observed that earlier posts generate more discussion than later posts, but there is no reason to assume this pattern is unique to asynchronous discussion.

C. Quality postsQuality posts are considered those that address course matters, discuss and reflect critically on content, and respond explicitly to comments by other students (Henri, 1995; Salmon, 2000; Zingaro & Oztok, 2012). Simple, useful rules of posting such as requiring comments to introduce new material and creating a subject field that conveys the essence of the main point have been applied successfully.Quality posts may also present personal experience relevant to the topic, present examples, and introduce divergence and digression into conversations. The value of divergence is based upon findings that effective peer-to-peer discussion—i.e. leading to learning—requires conceptual conflict and divergence in order to reach consensus (Cantor, 1992; Ugoretz, 2005). This is not unique to asynchronous discussion but is perhaps more visible in such formats. Furthermore, peer-to-peer discussion is more effective than instructor-learner discussion. There is broad agreement that discussion should take the form of student-led conversational dialogue that encourages, and allows for, divergence and digression.

D. DiscussionAsynchronous discussion seems to benefit from a conversational style that leads to dialogue, with a high degree of student control including the construction of open-ended questions and discussion topics by students (Kanuka & Anderson, 1998). That is, the discussion benefits from being student-led. Similarly to all group work the NTL should initiate discussion to promote student reflection and to initiate and develop relationships within groups and progress to deeper cognitive discussion. This takes place over the duration of the course rather than being the format for each topic. The NTL needs to consider group development and learning across the whole of the course, not just within each discussion. There is some debate on the benefits, or lack thereof, of the NTL attempting to manage and moderate discussion. One risk is that the discussion becomes tutor-led. The converse is that the discussion strays off-topic. However, there is some agreement that if appropriate rules and instructions are in place, and if the pattern of communication is set and modeled early by the NTL, then students benefit from controlling the discussion and incorporating their own experience and goals (Moore & Fetzner, 2004). A common and accepted finding is that setting quantitative measures of engagement limits discussion (Andresen,


2009; Denne, 2005; Tu & Corry, 2003; Zingaro & Oztok, 2012). Some caution is required in the interpretation of the latter point. For example, setting a goal of two posts per week is not the same as saying students must make ‘X’ number of contributions. This is the distinction between quantitative measures of engagement and goal setting.An important and perhaps somewhat neglected consideration is the need to integrate discussion with other activities. This can be achieved through clear goal setting where goals include: interrogating the material; interaction with peers; discussions of a specific object (e.g. a paper/report/case); and the formation of critiques and ‘hypotheses.’ When the NTL posts a comment/discussion and/or poses questions, it’shelpful to state whether an individual or group response is required. This opens up the possibility for public student-NTL debate and student-student debate (Aleksic-Maslac, Korican & Njavro, 2007; Hirumi, 2002a, 2002b). This is similar in function to the face-to-face lecturer engaging in debate and discussion during synchronous discussion.

E. Conversational styleThe NTL’s use of a conversational approach to students and within online interaction fosters high quality student contributions (Kanuka & Anderson, 1998; York & Richardson, 2012; Salmon, 2000) This means writing in conversational form and style; using personal anecdotes and affective verbal immediacy; the expression of appropriate emotion through the use of capitals, bold, italics; emoticons; and so on. The NTL should aim to set a tone of ‘we’re in this together’ and be an active part of the community. Early communications should make use of plain language and over time the NTL should introduce the appropriate vocabulary for the course.

F. FeedbackFeedback tends to improve dialogue, especially when students know posts are being read by the NTL. This is true even if the NTL does not respond to each and every post. The level of dialogue seems to be higher when the NTL is involved. However, when the NTL leads the discussion student-student interaction decreases (Kanuka & Anderson, 1998). Thus far, no best fixed form of feedback has been identified in the literature. The key principles remainfor timely and relevant feedback that meets students’ communication needs and treats them as individuals. It should be personal, frequent and regular, and allow students to measure their progress.

G. QuestionsThe use of questions, including posing problems for solving, is a well-recognized learning tool and forms the spine of asynchronous discussion (Kanucka & Anderson, 1998; York & Richardson, 2012; Moore & Fetzner, 2004; Salmon, 2000). There is agreement on several points in the literature. Perhaps the most difficult aspect for the NTL is allowing students to develop and design questions, including exam questions. That is not to say the students should design actual exam questions, but should be encouraged to knowingly construct exam type questions. Open-ended questions that ask students to evaluate and make connections, and which have multiple possible answers, develop higher cognitive functions(Andresen, 2009; Armstrong & Thornton, 2012) They should be related to learning objectives and concepts and ideas in course reading as these questions generate more complex interaction between learners. When presenting situations, scenarios, case studies, and similar, questions should be designed that ask students what to do in such a situation, not what they think of the situation. The use of bi- and tri-level discussion questions provides a useful structure for conceptualizing anddesigning questions (York & Richardson, 2012). Level 1 questions are those where the answer can be found in course materials. Level 2 questions require students to relate materials to a personally relevant answer (i.e. one based on experience). Level 3 questions require students to find connections between course materials and broader historical, social, or cultural contexts. As the community of inquiry develops it is suggested that, as with the development of the language of the


community of practice, Level 1 questions are used early in the course discussion progressing to Level 3 questions later in the course.

III. CONCLUSIONSThe purpose of this paper was to address the question: as a tutor in an on-line asynchronous environment what necessary and sufficient behaviors can one enact to deliver high quality tuition? All other things being equal the facilitation of peer-peer discussion is considered the key factor. Discussion may be the lynchpin of asynchronous tuition, but discussion on its own is not enough.Discussion is best considered as part of the collaboration process and production of artifacts in relation to the achievement of pedagogical outcomes. That is, discussion is necessary but is it not sufficient. Other factors, such as course design and goal setting, play an equally important role. However, with regard to these factors, researchers may benefit more from looking at the similarities between web-based tuition and face-to-face tuition than at the differences. This was the main reason for focusing on discussion in this short review. That is, we found such a high degree of common-sense similarity between most of the design and structural elements of face-to-face tuition and asynchronous tuition that our conclusion was to defer to the existing literature on optimal design and structure. Overall we could find nothing to indicate distinct differences between course design and structure for asynchronous and synchronous learning. This included the reported finding that the reasons students drop out of asynchronous courses mirror the reasons students give when dropping out of synchronous courses. It is similarly logical to assume that problems such as non-participation and absenteeism are mirrored across both environments. That having been said, the importance of providing a social space was recognized as was the importance of introductions and engagement with and between students, as well asthe importance and value of affective verbal immediacy—i.e., engaging with the text using a conversational style and with appropriate expression of emotion. We suggest that discussion is a key factor in asynchronous learning, perhaps the key factor in producing high level pedagogical outcomes (Andresen, 2009; Brown, 2102; Dennen, 2005, 2008; Hew & Cheung, 2003; Kear, 2004; Parisio, 2011; Aleksic-Maslac, Korican & Njavro, 2007). However, it must be combined with the shared production of artifacts that relate directly to course objectives. The shared production of artifacts that relate directly to course objectives rely heavily on facilitated peer-peer discussion. The role of the NTL is critical in facilitating peer-peer discussion but fulfilling this role to its potential depends upon good design and course structure. We acknowledge there are wider and deeper debates in the literature regarding the necessary and sufficient factors for successful implementation. The matter is not without controversy—the resolution of which depends largely on one’s pedagogical philosophy. For example, is there a benefit in keeping the discussion controlled and focused on ‘the topic’ or is there a benefit in allowing and encouraging digression? Is there a benefit in bringing in personal experience or should the discussion be kept focused on the conceptual and cognitive elements? And so on. These elements are acknowledged and some consideration has been given to these above.


IV. REFERENCES

Andresen, M. A. (2009). Asynchronous discussion forums: success factors, outcomes, assessments, and limitations. Educational Technology & Society, 12(1): 249–257.

Armstrong, A. and Thornton, N. (2012). Incorporating Brookfield’s discussion techniques synchronously into asynchronous online courses. The Quarterly Review of Distance Education,13(1): 1–9.

Baumeister, R. F. and Leary, M. R. (1997). Writing Narrative Literature Reviews. Review of General Psychology, 1(3): 311-320.

Beaudin, B. P. (1999). Keeping online asynchronous discussions on topic. Journal of Asynchronous Learning Networks, 3(2): 41-53. from http://www.editlib.org/p/107297.

Brown, J. L. M. (2012). Online learning: A comparison of web-based and land-based courses. The Quarterly Review of Distance Education, 13(1): 39–42.

Cantor, J. A. (1992). Delivering Instruction to Adult Learners. Toronto: Wall & Emerson.Community Of Inquiry. Retrieved July 2013 from https://coi.athabascau.ca/Crawley, A. and Fetzner, M. (2013). Providing service innovations to the student inside and outside of

the online classroom: Focusing on student success. Journal of Asynchronous Learning Networks,17(1): 7-12.

Dennen, P. V. (2005). From message posting to learning dialogues: Factors affecting learner participation in asynchronous discussion. Distance Education, 26(1): 127–148.

Dennen, P. V. (2008). Looking for evidence of learning: Assessment and analysis methods for online discourse. Computers in Human Behavior, 24: 205–219.

Fetzner, M. (2013). What do unsuccessful online students want us to know? Journal of Asynchronous Learning Networks, 17(1): 13-27.

Garrison, D. R., Clevenland-Innes, M. and Fung, T. S. (2010). Exploring causal relationships among teaching, cognitive and social presence: Student perceptions of the community of inquiry framework. The Internet and Higher Education, 13(1-2): 31–36.

Henri, F. (1995). Distance learning and computer-mediated communication: Interactive, quasi-interactive or monologue? In C. O’Malley (Ed.), Computer supported collaborative learning (pp. 145-161). New York: Springer-Verlag.

Hew, K. F. and Cheung, W. S. (2003). Models to evaluate online learning communities of asynchronous discussion forums. Australian Journal of Educational Technology, 19(2): 241-259.

Higher Education Academy. Retrieved July 2013 from http://www.heacademy.ac.uk/Hirumi, A. (2002a). A framework for analyzing, designing and sequencing planned e- learning

interactions, Quarterly Review of Distance Education, 3(2): 141–60. Hirumi, A. (2002b). The design and sequencing of e-learning interactions: a grounded approach,

International Journal on E-Learning, 1(1): 19–27.Aleksic-Maslac, K., Korican, M. and Njavro, D. (2007). Important role of asynchronous discussion in e-learning system. International Conference on Engineering Education & Research December 3-7, Melbourne, Australia.Retrieved June 2013 from http://bib.irb.hr/datoteka/468834.169.pdf.

Huang, X. and Hsiao. E-L. (2012). Synchronous and asynchronous communication in an online environment. The Quarterly Review of Distance Education, 13(1): 15–30.

Junk, V., Deringer, N, and Junk, W. ( 2011). Techniques to engage the online learner. Research in Higher Education Journal, 10: 1–15.

Jorczak, R. L (2011). An information processing perspective on divergence and convergence in collaborative learning. Computer-Supported Collaborative Learning 6(2): 207-221.

Kanuka, H. and Anderson, T. (1998). Online social interchange, discord, and knowledge construction. Journal of Distance Education, 13(1): 57-74.

Kear, K. (2004). Peer learning using asynchronous discussion systems in distance education. Open Learning, 19(2): 151-164.


Kirby, J. R., Moore, P. J. and Schofield, N. J. (1998). Verbal and visual learning styles. Contemporary educational psychology, 13: 169-184.

Lee, S-M. (2014). The relationships between higher order thinking skills, cognitive density, and social presence in online learning. The Internet and Higher Education, 21: 41–52.

Lim, C. P. (2004). Engaging learners in online learning environments. TechTrends, 48(4): 16-23. McGee, P. and Reis, A. (2012). Blended course design: A synthesis of best practice. Journal of

Asynchronous Learning Networks, 16(4): 7-22. Moore, J. C. and Fetzner, M. J. (2004). The road to retention: A closer look at institutions that achieve

high course completion rates. Journal of Asynchronous Learning Networks, 13(3): 3-2. Moule, P. (2007). Challenging the five-stage model for e-learning: a new approach. ALT-J, Research in

Learning Technology, 15(1): 42–50. Parisio, M. (2011). Engaging students in learning through online discussion: A phenomenographic

study. Proceedings ascilite Hobart: Concise Paper. Retrieved June 2013 from http://www.ascilite.org.au/conferences/hobart11/downloads/papers/Parisio-concise.pdfPelz, B. (2004). (My) three principles of effective online pedagogy. Journal of Asynchronous Learning

Networks, 8(3): 33-4. Salmon, G. (2000). E-moderating: the key to teaching and learning online. London, Kogan Page. Tu, C-H. and Corry, M. (2003). Designs, management tactics, and strategies in asynchronous learning

discussions. The Quarterly Review of Distance Education, 4(3): 303-315. Sloane Consortium. Retrieved July 2013 from http://sloanconsortium.org/Ugoretz, J. (2005). Two roads diverged in a wood: Productive digression in asynchronous

discussion. Innovate: Journal of Online Education, 1 (3). Retrieved June 2013Vonderwell, S. (2003). An examination of asynchronous communication experiences and perspectives

of students in an online course: a case study. Internet and Higher Education, 6: 77–90. Wan, Y-m. (2008). Essential elements in designing online discussions to promote cognitive presence – a

practical experience. Journal of Asynchronous Learning Networks, 12(3-4): 157-177. Winiecki, D. J. and Chyung, Y. (1998). Keeping the thread: Helping distance students and instructors

keep track of asynchronous discussions. Distance Learning '98. Proceedings of the Annual Conference on Distance Teaching & Learning. Retrieved June 2013 from http://files.eric.ed.gov/fulltext/ED422886.pdf.

York, C. S. and Richardson, J. C. (2012). Interpersonal interaction in online learning: Experienced online instructors’ perceptions of influencing factors. Journal of Asynchronous Learning Networks, 16(4): 83-98.

Zingaro, D. and Oztok, M. (2012). Interaction in an asynchronous online course: A synthesis of quantitative predictors. Journal of Asynchronous Learning Networks, 16(4): 71-82.


An Exploration of Metacognition in Asynchronous Student-Led Discussions: A Qualitative Inquiry

Martha M. Snyder Laurie P. Dringus Graduate School of Computer and Information Sciences Nova Southeastern University

ABSTRACTResearch is limited on how metacognition is facilitated and manifested in socially situated online learning environments such as online discussion forums. We approached metacognition as the phenomenon of interest with a methodological objective to evaluate the relevance of a metacognition construct. We also had a content objective to study student-led facilitation of discussions as a strategy in promoting metacognition. The purpose of this study was to explore: (1) where metacognition is evident in student-led online discussions and (2) how students’ experiences leading and participating in student-led online discussions relate to their awareness of learning. Content analysis was used to analyze discussion forum transcripts and interpretative phenomenological analysis (IPA) was used to analyze responses to an open-ended questionnaire. We concluded the metacognition construct was useful in helping us understand and organize the data and student-led online discussions can be an effective strategy for helping students develop dimensions of metacognition including knowledge, monitoring, and regulation. However, in order for students to use these skills effectively, instruction, motivation, and guidance are needed—particularly in relationship to the regulation of metacognition and co-construction of meaning.

I. INTRODUCTIONThe study of metacognition gained popularity in the late seventies and early eighties as a promising area of inquiry to help develop methods of teaching children and adults to “comprehend and learn better in formal educational settings” (Flavell, 1979, p. 910). Metacognition is generally defined as how we monitor and control our own cognition (Flavell, 1979; Young & Fry, 2008). Hart’s (1965) seminal work about the feeling-of-knowing experience and its use to indicate what is stored in long-term memory, paved the way for further studies focused on metacognition. Flavell (1979) presented a conceptual model of cognitive monitoring and encouraged educational researchers to develop interventions to increase cognitive monitoring given his conviction that more cognitive monitoring would lead to better learning—especially for children. Garner (1990) advised of the need to consider context when using various strategies to enhance learning. Metacognition within the context of an online learning community is the focus of our present study.Akyol and Garrison (2011) defined metacognition in an online learning community as “a set of higher knowledge and skills to monitor and regulate manifest cognitive processes of self and others” (p. 184). This definition implies metacognition is not necessarily developed individually but can be co-constructed


within an online social context. For example, Akyol and Garrison explored metacognition as a centralconstruct to describe an awareness of one’s learning and the ability to control and construct meaning in an online community of inquiry (CoI) (situated in an online course). They suggested that “metacognition mediates between reflection and action” (p. 186), an important aspect of understanding the degrees to which students are able to be critical thinkers and inquirers. Akyol and Garrison (2011) emphasized that sharing cognitive experiences is important to learning; metacognition is “seen to mediate between internal knowledge construction and collaborative learning activities” (p. 185). In this regard, there is an aspect of “community and sharing thinking” (p. 189) in knowledge acquisition that also suggests learning to be a shared process. In metacognition, there is a collaborative aspect or “co-construction” of three dimensions of cognition including knowledge, monitoring, and regulation. Co-construction is an aspect that is often applied to online learning environments and is consistent with social constructivism theory (McInerney, 2005; Palincsar, 1998) and with the Community of Inquiry (CoI) model in online learning (Garrison & Arbaugh, 2007). Researchers who have studied metacognition in traditional learning environments (Mayer, 1998; McCabe, 2011; Young & Fry 2008) and in online learning environments (Akyol & Garrison, 2011; Choi, Land, & Turgeon, 2005) suggest that the more adept learners are at using metacognitive skills, the more academically successful they will be. In two studies that explored undergraduate students’ metacognitive awareness in traditional learning environments, McCabe (2011) found that students’ awareness of metacognitive skills was low to non-existent and metacognitive awareness may be improved through targeted educational interventions. Choi et al. (2005) studied the effectiveness of peer-questioning strategies to facilitate metacognition during online small group discussions. While the authors found the scaffolding of questions to be helpful in encouraging other students to ask questions, they did not find any differences in the quality of questions or learning outcomes. However, their study suggested that there is value in the implementation of instructional strategies to facilitate “learner’s reflection and knowledge reconstruction in online small group discussion” (p. 506). Their results also emphasized the value of testing and revising instructional strategies aimed at facilitating metacognition. With an understanding of how to assess metacognition in online discussion forums coupled with the awareness of instructional strategies that are effective in developing metacognitive skills, improvements can be made to the design of online discussions that foster successful learning outcomes.

II. THEORETICAL FRAMEWORKThe metacognition construct is a fast-growing trend of discovery—in both theory and practice—in online learning, and interest in the construct extends the popular Community of Inquiry (CoI) model (Garrison & Arbaugh, 2007). In a CoI, it is assumed learning takes place as a result of the interaction of three elements: cognitive presence, social presence, and teaching presence (Garrison, Anderson, & Archer, 2000). The CoI has been accepted as a useful framework for research and practice in online learning environments (Garrison & Arbaugh, 2007) and was used as the theoretical framework for the initial development of Akyol and Garrison’s (2011) metacognition construct as it applies to the learning process in a CoI. Akyol and Garrison situated their metacognition construct at the intersection of cognitive presence (representing the inquiry process) and teaching presence (representing the shared teaching roles and responsibilities of all participants in a community of inquiry). They suggested these two presences were “essential to better understand and assess metacognitive knowledge and skills of learning in an online community of inquiry” (p. 186). This present study is based upon Akyol and Garrison’s (2011) work, which revealed that individuals apply knowledge and skills to monitor and regulate cognition in themselves and others. Little, however, is actually known about this phenomenon in online discussions specifically. As a result, this study examines the metacognition construct and its dimensional scale (see Figure 1) for its theoretical and practical significance in online discussions.


We employed student-led facilitation (Baran & Correia, 2009)—also known as student-led discussions—as an instructional strategy to learn to what extent students are able to “co-construct” meaning whenparticipating in discussions with peers. In student-led discussions, the instructor maintains a minimal facilitative role in managing the online discussions while students take turns leading the discussion. Therefore, we examined both the metacognition construct as it manifested in student-led online discussions, and we examined the effectiveness of student-led online discussions as an instructional strategy to promote metacognition. Guided by pre-established dimensions and indicators of the metacognition construct proposed by Akyol and Garrison (2011), our study describes what we observed in the discussion forum transcripts in terms of metacognition patterns, and what we interpreted from students’ self-reporting of their experience as a facilitator and participant in the student-led discussion activity. In determining the effectiveness of student-led discussions, we sought to capture how students described their experiences and how those experiences reflected metacognition processes.

Metacognition in a Community of InquiryKnowledge of Cognition(KC)(Entering Knowledge/Motivation)

Monitoring of Cognition(MC)(Assessment/Task Knowledge)

Regulation of Cognition(RC)(Planning/Strategies)

Pre-Task ReflectionKnowledge of the inquiry processKnowledge of critical thinking and problem solvingKnowledge of factors that influence inquiry and thinkingKnowledge of self as a learnerEntering motivational stateKnowledge of disciplineKnowledge of previous experiencesExpectancy of success

Reflection on ActionDeclarative; judgingCommenting on task, problem or discussion threadAsking questions for confirmation of understanding Commenting about self’s andothers’ understandingMaking judgments about validity of contentCommenting on or making judgments about the strategy appliedAsking questions about progression or stallingExpressing emotions during learningAssessing motivational state and effort required

Reflection in ActionProcedural; planningSetting goalsApplying strategiesProviding/asking for supportChallenging self or othersAsking questions to deepen thinkingAsking for clarificationRequest informationSelf questioningQuestioning progression, successTaking control of motivation and effortFacilitating/directing inquiry

Figure 1. The metacognition construct. From “Assessing Metacognition in an Online Community of Inquiry” by Akyol, Z. and Garrison, R.D. (2011). The Internet and Higher Education, 14, (p. 185). Copyright [2011] by Elsevier Inc. Reprinted

with permission.


A. Justification: Gaps in the Literature on Metacognition in Online Discussions

Since the study of metacognition in online learning is a recent trend, we identified three research gaps that inspired and extended our study of Akyol and Garrison’s (2011) theory. The first gap involved a lack of understanding of how metacognition is manifested in online discussions, given that asynchronous online discussions are used predominantly for supporting teaching, learning, and collaborative activities in online learning. The purpose of Akyol and Garrison’s study was to develop and validate a metacognition construct to assess three dimensions of metacognition (see Fig. 1) based on indicators adapted from the CoI framework. The validation of their metacognition construct was limited by assessing one online graduate course, suggesting that more studies and data are needed. We sought to extend this effort in validating the metacognition construct. A second gap was in the identification of effective instructional strategies that facilitate metacognition in online discussions. Akyol and Garrison (2011) identified the need to develop new strategies for supporting the metacognition process in an online community of inquiry (CoI). In particular, we were interested in whether student-led facilitation (2009) was an effective instructional strategy for supporting the development of metacognition in online contexts. A third gap was the limited methodological choices that have been applied in the research to examine metacognition in online discussions. Akyol and Garrison (2011) applied content analysis to quantitative data derived from archived discussion transcripts, but they recommended a qualitative approach (i.e., interviewing) could “further verify the indicators of each dimension of metacognition” (p. 189). We interpreted this gap to mean that content analysis of discussion postings was limited in the scope of exploring meaning and identifying the essences of online discussions. Hull and Saxon (2009) also suggested a need to move beyond the counting of posts to a more qualitative approach—that is, exploring the nature of the interactions within an asynchronous discussion forum.We sought to extend our understanding of Akyol and Garrison’s (2011) work by using these gaps as a strategic approach to frame new discovery about how metacognition is manifested in online discussions. We approached Akyol and Garrison’s work as a starting point for pragmatic orientation. But we also sought to learn what was known and not known about metacognition in online discussions by not narrowing or limiting our scope strictly to an a priori approach (Chenail, Cooper, & Desir, 2010). Therefore, we considered novel methods for qualitative research analysis to explore the qualitative nature of metacognition in online discussions.

III. PURPOSE AND RESEARCH QUESTIONSWe approached metacognition as the phenomenon of interest with a methodological objective of evaluating the relevance of Akyol and Garrison’s (2011) construct. We also had a content objective to study student-led facilitation of discussions as a strategy in promoting metacognition. Therefore, the purpose was to explore: (1) where metacognition is evident in student-led online discussions and (2) how students’ experiences leading and participating in student-led online discussions relate to their awareness of learning. Four research questions guided our inquiry: What indicators of metacognition in a community of inquiry (CoI) are evident in student-led online discussions? What are students’ perceptions about leading and participating in student-led online discussions? How do students’ perceptions of their student-led discussion experiences reflect metacognition processes? What are the implications on the design of online student-led activities?


IV. RESEARCH DESIGNWe took a qualitative approach to analyzing the data and employed two qualitative analysis methods: content analysis (Sandelowski, 2000; Smith, 2000) to analyze discussion forum transcripts, and interpretative phenomenological analysis (IPA) (Smith, Flowers, & Larkin, 2009) to analyze student-led discussion questionnaire responses.

A. Site and ParticipantsThe study took place in the context of an online master’s course titled “Communities of Practice (CoP)” in a learning technology graduate program. The first author taught the course. The course was fully online and included four assignments: a reflection paper (20 points); mini research paper (30 points); online learning CoP plan and prototype (30 points); as well as student discussion forum participation and facilitation (i.e., student-led discussions) (20 points). In the course syllabus, guidelines were given about the student-led discussions including expectations, directions, and format. A list of bi-weekly topics and readings was given to students at the beginning of the semester. During the first week of class, students emailed the professor indicating their first, second, and third choices of bi-weekly segments to facilitate. The professor organized the schedule based on choices and distributed the updated schedule via Blackboard’s Course Announcements. The professor modeled the facilitation process during the second and third weeks of the term. Specifically, student-facilitators were instructed to: read the assigned readings in advance of their facilitation dates, introduce the discussion topic/readings, provide guiding questions for the discussion, encourage participation, keep the discussion focused, encourage multiple viewpoints of the same issue(s), and bring the discussion to an end by summarizing highlights (1 or 2 short paragraphs). Twelve students participated in the online discussions and six students completed a questionnaire at the end of the term regarding their experiences in the student-led discussions. After receiving approval from the university’s Institutional Review Board (IRB), data collection and analysis commenced.

B. Data SourcesThe two primary data sources were the archived transcripts from discussion postings, and the anonymous student text responses to a Web-based questionnaire given at the end of the term. Similar to Akyol and Garrison’s (2011) method of selecting transcripts from the first, fifth, and ninth weeks of a nine-week course for analysis, we selected three distinct two-week-long student-led discussion segments from the 14-week semester. Cases (A, B, and C) were selected from the beginning, middle, and end of the student-led discussions. In doing so, we used Miles and Huberman’s (as cited in Marshall & Rossman, 2011) definition of a typical case as these cases highlighted discussions that were “normal or average” (p. 111). The student-led discussion questionnaire consisted of eight questions regarding students’ opinions about the student-led discussion activity. The questions were open-ended and designed to gain feedback about 1) how the activity facilitated metacognition (e.g., how the activity helped them learn the course content),2) how the activity was designed (e.g., bi-weekly discussion timeline, one or more facilitators runningdifferent discussions during the same time period), and 3) students’ overall perceptions about the value of the student-led discussions.

C. Data AnalysisIn exploring the first research question, “What indicators of metacognition in a community of inquiry (CoI) are evident in student-led online discussions?” we analyzed discussion transcripts in two ways. First, we used a priori codes derived from Akyol and Garrison’s (2011) metacognition construct and Garrison and Akyol’s (2013) metacognition questionnaire. Then, we used an inductive approach (Sandelowski, 2000) to develop codes “generated from the data” (p. 338). It is important to distinguish between content analysis in quantitative studies and qualitative studies. Sandelowski (2000) explained that both types of studies involve counting responses, however “in qualitative content analysis, counting


is a means to an end, not the end itself” (p. 338). She further distinguished “the end result of counting is not a quasi-statistical rendering of the data, but rather a description of the patterns or regularities in that data that have, in part, been discovered and then confirmed by counting” (p. 338). While we sought to evaluate the relevance of Akyol and Garrison’s (2011) construct, our perspective of the phenomenon of interest was drawn from the descriptive indicators that were provided for each of the three dimensions. However, we were aware that the application of the metacognition construct in assessing metacognition in online discussions has not been validated fully. In addition, the metacognitive descriptors were open to interpretation, meaning they were not defined concretely. This is an important position statement on our part that we intended to review and identify what we saw in the data to determine if and how Akyol and Garrison’s construct was relevant and evident in the context of asynchronous student-led discussions. Therefore, we remained open to the discovery of meaning of the data throughout the analysis.We used interpretative phenomenological analysis (IPA) (Smith, et al., 2009) to analyze responses to the student-led discussion questionnaires. As an interpretative theory and method, Smith and Osborn (2003) indicated that IPA involves a two-stage interpretation process, or “a double hermeneutic,” in that the participant is trying to make sense of their experience and the researcher is trying to make sense of the participant’s experience (p. 53). The written responses of students’ perceptions of their student-led discussion experiences were coded into categories of events and meaning of events (Smith, et al., 2009). By examining the content of online discussions and student responses to the questionnaire using these exploratory and descriptive methods, we were able to gain a better understanding how metacognition can be facilitated and manifested in online discussions.

1. Analysis of Discussion Forum Transcripts Using A Priori Coding ApproachAkyol and Garrison (2011) performed content analysis of forum transcripts by contextually coding the data (i.e., forum postings as the unit of analysis) in identifying instances of specific indicators from the three metacognitive dimensions presented in their construct. For example, indicators for monitoring of cognition (MC) include commenting on a task, asking questions for confirmation of understanding, and expressing emotions during learning (See Figure 1). They coded each posting by identifying the evidence of MC indicators as well as the other dimensions and related indicators from the construct. We developed a coding guide (See Appendices) using Akyol and Garrison’s (2011) construct andmetacognition questionnaire (Garrison & Akyol, 2013). This guide was used to help us identify instances of metacognition and organize the data. One discussion post was the unit of analysis. While in some cases, we coded several examples of one or more of the dimensions in one post, we did not count all instances. Instead, we followed the approach used by Akyol and Garrison and counted the number of messages in each case that showed at least one or more instances of each of the dimensions. The first author analyzed and coded the transcripts from weeks 4-5 (Case A), 8-9 (Case B) and 12-13 (Case C) and identified indicators in the text that reflected each of the three dimensions. The second author reviewed the coded transcripts so that we could discuss the usefulness of the metacognition construct in assessing metacognition in the transcripts and also to ensure that the codes represented each dimension (Smith, 2000).

2. Analysis of Transcripts Using Inductive Coding ApproachAfter we analyzed the cases using a priori codes, the first author analyzed the transcripts again without the codes. The purpose for conducting this second analysis was to gain new insights about the data that might shed light on how metacognition is manifested in the student-led discussions. To begin this process, the first researcher read through the transcripts making notes and memos reflected her thinking. Next, she read and re-read the transcripts and developed a set of open codes. These open codes were then categorized into themes.


3. Analysis of Student-Led Discussion QuestionnaireWe applied interpretative phenomenological analysis (IPA) (Smith, et al., 2009) to our investigation of the student-led discussion questionnaire responses. The questionnaire consisted of eight open-ended questions about students’ opinions of the student-led discussion activity (see Appendices). This analysis addresses research question two: “What are students’ perceptions about leading and participating in student-led online discussions?” question three: “How do students’ perceptions of their student-led discussion experiences reflect metacognition processes?” and question four: “What are the implications on the design of online student-led activities?” Smith et al. (2009) suggested using the following six-step process for analyzing interview data: 1) immerse oneself in the data; 2) take notes including descriptive, linguistic, and conceptual comments; 3) develop emergent themes; 4) search for connections acrossemergent themes; 5) move to next case; and 6) look for patterns across cases.

V. RESULTS AND INTERPRETATIONS A. Results: Analysis of Discussion Forum Transcripts Using A Priori CodesThree key findings resulted from this analysis. First, students’ metacognitive behaviors were observable in online learning environments. This finding is consistent with Akyol and Garrison’s (2011) and Garrison and Akyol’s (2013) research. Second, instances of all three indicators (KC, MC, and RC) were observed. However, there was overlap and it was difficult during the coding process to separate statements into these three dimensions. Third, in looking across all three cases, we observed more instances of MC than KC and RC. Garrison and Akyol (2013) stated, “MC reflects the individual perspective of an educational community of inquiry” (p. 86). We found most statements to reflect MC indicators. Garrison and Akyol also stated, “regulation of cognition contextualizes the learning dynamic consistent with a purposeful learning community” (p. 86). We coded the least amount of statements as those reflecting the RC dimensions. Table 1 shows the percentages of dimensions of metacognition in student-led online discussions.

Table 1Percentages of Dimensions of Metacognition in Student-led Online Discussions

Case # of Messages Knowledge Monitoring Regulation

CASE A (Weeks 4&5)

13 (3) 23.0% (12) 92.3% (5) 38.4%

CASE B (Weeks 8&9)

20 (7) 35.0% (18) 90.0% (6) 35.0%

CASE C (Weeks 13&14)

25 (7) (28.0% (20) 80.0% (9) 36.0%


Table 2 shows examples of quotes from the transcripts and how they were coded using Akyol and Garrison’s (2011) metacognition construct and Garrison and Akyol’s (2013) metacognition questionnaire items.

Table 2Coding Examples from Case B Using A Priori Codes

Metacognition Dimension

Quotes and References to Akyol and Garrison’sMetacognition Construct (2011) and Garrison and Akyol’s (2013) Metacognition Questionnaire

Knowledge of Cognition (KC)

“Not only am I more interested in reading the posts of my classmates, I also feel a sense of community with them.” Construct: Entering motivational stateQuestionnaire: “I know my motivational state at the beginning of the learning process.”

Monitoring of Cognition (MC)

“I am much more engaged in a class where I feel some connection with the instructor, or that the instructor cares if I understand.”Construct: Assessing motivational state and effort requiredQuestionnaire: “I realize I need confirmation of my understanding.”

Regulation of Cognition (RC)

“What activities did the instructor present that made students want to engage with one another and experience reflective thought beyond a simple question in a discussion post?” Construct: Asking questions to deepen thinking Questionnaire: “I ask questions to deepen thinking.”

B. Results: Analysis of Discussion Transcripts Using Inductive ApproachThe inductive analysis approach resulted in 51 open codes which were categorized into the following six themes: Acknowledges Others (e.g., greets students, refers to another student’s post, agrees/disagrees) Engages Others (e.g., encourages participation, asks specific questions directed toward classmates, asks follow-up questions) Self-Discloses (e.g., expresses emotion, shares personal beliefs, shares personal experiences)Self-Reflects (e.g., self-questions and “I’ve thought about this…) Declares Knowledge (e.g., extends discussion, states opinion, makes an inference, references literature, synthesizes information, summarizes discussion)Interacts (e.g., student-to-student(s)/class, student-to-facilitator, facilitator-to-student(s)/class)Figure 2 is a visual representation of the inductive analysis of the discussion forum transcripts. The five themes: acknowledges others, engages others, self-discloses, self-reflects, and declares knowledge represent how students made sense of the discussion topics both individually and collaboratively. The sixth theme, interacts, is represented as the roots given these interactions supported the growth and development of the discussion. Students not only constructed knowledge internally through self-reflection and declarative knowledge but also helped to facilitate learning through their interactions with others by acknowledging classmates’ contributions, engaging them in dialogue, and self-disclosing. The double-headed arrow in between self-regulation (constructing meaning) and co-regulation (facilitating learning) represents the back and forth between internal knowledge construction and co-construction of meaning as described by Garrison and Akyol (2013).


Figure 2. Visual Representation of Inductive Coding of Transcripts from a Student-led Online Discussion

C. Results and Interpretations of the Analysis of Student-Led Discussion Questionnaire

How students perceive their participation in student-led discussions and how their perceptions reflect metacognitive processes were categorized into the following nine themes:PreparationPerceived role as facilitatorPerceived responsibilities as facilitatorEffects of facilitator rolePerceived role as participantPerceived helpfulness of student-led discussionsDifficulties in shifting rolesGeneral perceptions about student-led discussion activityLogistics – student-led discussion activity designEach of the nine themes is presented here along with their associated events and meanings. In this section the “I” and the “P” in interpretative phenomenological analysis (IPA) are emphasized. Smith, et al. (2009) reminded researchers, “At each stage the analysis does indeed take you further away from the participant and includes more of you. However, the ‘you’ is closely involved with the lived experiences of the participant—and the resulting analysis will be a product of both of your collaborative efforts” (p. 92). We


approached this inquiry as seasoned online instructors who have expertise in instructional design and online learning environments. We attribute the design of this course and inclusion of SLDs to our social-constructivist philosophy. Although we attempted to bracket our preconceptions and stay open to the data, one is never free of all bias. As such, we acted as researchers and participants and engaged in a double hermeneutic loop in which we attempted to make sense of the learner’s, who were making sense of their facilitation experiences (Smith et al., 2009).

1. Theme One: Preparationa. EventsWhen describing their experience of preparing to lead their two-week segment of the online discussion, students noted the following:“I prepared by reading the chapters.”“I read the content a few times and did some brainstorming on discussion topics/key points.” “I read the chapter and thought about the nine orientations of a CoP identified by the authors. Since [our class] uses on Blackboard as our orientation, I thought it would be interesting to see what other orientations the class thought would facilitate our learning together.” The amount of time students reported needing to prepare to lead a discussion varied from two hours to one week, with an average time of about three hours.

b. MeaningsBased on these findings, it seems learners take their role as facilitator seriously and want to appear prepared and knowledgeable about the topic when leading a discussion with their peers. How can this preparation activity be designed to leverage metacognitive skills? That is, are preparation activities such as the ones reported by students (i.e., reading, re-reading, brainstorming, and thinking about the readings within a specific framework) the most effective preparation strategies? Perhaps if we could teach preparation strategies that are research-based as opposed to relying on the learner’s use of “improvised strategies” (McCabe, 2011, p. 463) we could promote explicit metacognitive awareness (Schraw, 1998)early on in the facilitation process.

2. Theme Two: Perceived role as facilitatora. EventsStudents identified various roles they played as facilitator including:Discussion forum guide – “I read the chapter giving thought to how I would facilitate a discussion on the topics covered.” Content expert – “I felt that I had to study the course content more closely.” “I felt as if I had to know that material more authoritatively than my peers who would be answering my questions so that I could properly respond.” Content guide – “I had to choose one or two things to focus the discussion on so I wanted to make sure I understood each aspect first.”Discussion prompter – “I would try to come up with more engaging questions.” Crafter of “good questions” – “I would attempt to craft questions which were closed-ended enough to get my peers to think about the chapter but open-ended enough that they could take ownership of their response.”Motivator – “I would try to motivate others with more thoughtful responses.”

b. MeaningsStudents readily accepted the role of facilitator. In that role, none of the students described his/her role as the authority figure. While one student indicated he/she felt the need to understand the material “more


authoritatively than my peers,” there was not an indication that students felt the need to be experts. This self-perceived role of serving as a guide rather than an authority figure is consistent with constructivistpedagogy (Harasim, 2012). Students perceive their role in the learning process not as a passive studentbut one who is willing and capable of guiding others.

3. Theme Three: Perceived responsibilities as a facilitatora. EventsWith regard to students’ perceptions about leading and participating in student-led online discussions, the majority of comments pertained to their role as facilitator. Activities students engaged in as a facilitator included: Preparing – “I spent the week prior readying for my discussion questions.” Reading – “I read through the materials and tried to identify the crux of the chapter.” Brainstorming – “I read the content a few times and did some brainstorming on discussion topics/key points.” Crafting questions – “I would attempt to craft questions which were closed-ended enough to get my peers to think about the chapter but open-ended enough that they could take ownership of their response.” Monitoring – “I also would have tried to log in every day to at least check in on how the topic was progressing. If I missed a day I felt like I was behind and had to catch up on what was happening.” Motivating others – “I would try to motivate others with more thoughtful responses.”

b. MeaningsThese findings indicate student facilitators were aware of a responsibility to the group to lead andfacilitate discussion in a way that fosters deep learning. These activities reflect metacognitive dimensions identified by Akyol and Garrison (2011). For example, the first three bullet points reflect indicators of KC including knowledge of the inquiry process, knowledge of self as learner, and entering motivational state. These activities seem to be focused on helping the student-facilitator acquire and comprehend information in order to facilitate the discussion effectively. The last three bullet points reflect indicators of RC including applying strategies, procedural/planning, and taking control of motivation and effort. These types of regulatory skills, however, seem to be focused outward toward the student-facilitator’s classmates. That is, it appears these regulatory skills were applied to help facilitate the learning of the group, rather than the learning of the individual.

4. Theme Four: Effects of facilitator rolea. EventsBased on students’ descriptions of their experience as a student facilitator, it appears they applied many of the regulation of cognition (RC) planning techniques and strategies outlined by Akyol and Garrison (2011). For example, some of the comments that reflect “procedural/planning” (Akyol & Garrison, 2011, p. 185) include:“I read the chapter giving thought to how I would facilitate a discussion on the topics covered.” “I had to choose one or two things to focus the discussion so I wanted to make sure I understood each aspect first.” “I felt that I had to study the course content more closely.”Comments that reflect application of strategies such as “providing/asking for support; challenging self or others; asking questions to deepen thinking; asking for clarification; request information; self-questioning)” (Akyol & Garrison, 2011, p. 185) include: “I would attempt to craft questions which were closed-ended enough to get my peers to think about the chapter but open-ended enough that they could take ownership of their response.” (Strategy: Asking questions to deepen thinking)


“Being the facilitator made me look at the content from a different perspective.” (Strategy: Self-questioning) A comment that reflects “questioning progression/success” (Akyol & Garrison, 2011, p. 185) is: “…but facilitating helped me by making me look for deeper meaning among the participants’ comments and continue [to] review the reading assignment a few times with a more critical eye.”Comments that reflect “taking control of motivation and effort” and/or “facilitating/directing inquiry” (Akyol & Garrison, p. 185) are: “I felt as if I had to know the material more authoritatively than my peers who would be answering my questions so that I could properly respond.” “I would try to come up with more engaging questions…” “I would try to motivate others with more thoughtful responses.”

b. MeaningsThese comments reflect the indicators that Akyol and Garrison (2011) identify under regulation of cognition (RC) almost exclusively. The student facilitator is not only making sense of the task individually but is also aiming to deepen one’s understanding through the process of facilitating discussion among his/her peers. Taking on the role of student-facilitator required the student to apply these strategies more so than as a participant, simply because of the nature of the task. With regard to this type of shared agency – when all students share the role of learner and facilitator, Martin (2013) stated, “When this capacity shifts from one central figure to multiple participants, all community members acquire both a right and a duty to influence the community” (p. 150). Thus, through this responsibility to the class, the student-facilitator is applying strategies that could also help him/her develop metacognitive skills.

5. Theme Five: Perceived role as participanta. EventsWhen asked to reflect on their role as a participant in student-led discussions, students identified themselves as active participants and co-constructors of knowledge. For example, one student commented: “I was a fellow-traveler who was being asked to co-construct/formulate an understanding of the materials being presented. I didn’t have to be 100% and it was ok to ask questions and make suppositions in the dialogue which took place.”Another student noted, “The expectations of my role were to be an active participant.”

b. MeaningsThe student as a participant in the online discussions is a collaborator—one who is co-constructing meaning along with the student facilitator and the rest of the class. One might question whether the element of student-facilitation had any impact on the perceived role as participant. That is, because the student was leading the discussion instead of the instructor, were students more comfortable asking questions and making mistakes? Also, for those who took the role of student-facilitator early on, how did the role of student-facilitator impact their behavior as a participant later on in the semester? Did this experience help the student become more metacognitively aware?

6. Theme Six: Perceived helpfulness of student-led discussionsa. EventsWhile it seemed that the perceived role of student as facilitator reflected indicators of regulation of cognition (RC), when students were asked what they found most beneficial about the student-led discussions, responses seem to fall into the category of monitoring of cognition (MC). Table 4 shows


students’ comments about perceived helpfulness of student-led discussions with corresponding indicatorsfrom Garrison and Akyol’s (2013) metacognition questionnaire. Akyol and Garrison (2011) stated, “…the monitoring of cognition (MC) dimension specifically includes the awareness and willingness to reflect upon the learning process” (p. 184). Table 4 includes student comments that reflect monitoring of cognition (MC) indicators.

Table 4Students’ Comments about Perceived Helpfulness of Student-Led Discussions and Corresponding Indicators from Garrison

and Akyol’s (2013) Metacognition Questionnaire

Students’ Comments about Perceived Helpfulness of Student-Led Discussions

Corresponding Indicators from Garrison and Akyol’s (2013, p. 86) Metacognition Questionnaire

“[The discussions] helped me understand the readings and get different perspectives from other students.”

MC - I realize I need confirmation of my understanding.

“[The discussions] helped me gain a deeper understanding of the content we were reading.”

MC - I consciously assess my understanding during the learning process.

“I do think that the discussions helped me understand some content that may have been confusing. By reading others’ responses, it was easy to see another point of view.”

MC - I pay attention to other course participants’ ideas / understandings /comments.

“I thought [student-led discussions] wereespecially useful because it gave us an opportunity to be a steward/facilitator of our own mini-CoP – we had the opportunity to see how the shoe fit.”

MC – I think about how we are approaching a task.

b. MeaningsPerhaps these comments reflect MC more than RC because students are reflecting upon how their involvement in facilitating the discussions helped them personally, thus internalizing the learning gained from the shared online discussions. Akyol and Garrison (2009) described how “Taking responsibility for teaching presence enables students to reflect on each other’s contributions and their contribution to the developmental progress toward the intended goals while engaged in discourse” (p. 184). However, it is uncertain whether and how student-led discussions improve knowledge construction. For example, De Wever, Van Winckel, and Valcke (2008) found no significant difference in the level of knowledge construction between student-facilitated and instructor-facilitated discussions. However, they found significant differences between these two types of discussions when a student-facilitated discussion included another student role called, “developer of alternatives.” That is, when these two student roles were assigned within a discussion (i.e., facilitator and developer) higher levels of knowledge construction were achieved. These results have implications on the effective design of student-led discussion suggesting that additional student roles may need to be assigned.

7. Theme Seven: Difficulties in shifting rolesa. EventsStudents were asked what they found to be the most difficult aspect of the student-led discussion assignment. The emergent theme from their responses was difficulties in shifting roles. This theme was drawn from statements like the following:


“To me the most difficult part was knowing when to chime in as a facilitator. I know it is a fine line, I wanted to answer everyone but also didn’t want to dominate the discussion.” Another student said, “I also found that I didn’t always know when to not comment during my facilitation week. I didn’t want to scare anyone off, but I didn’t want to bombard and/or prod everyone who spoke up.” A couple of students expressed feelings of uncertainty and uneasiness about leading a discussion but found in doing so, they felt more confident. One student said, “Commenting in a discussion thread, much less leading the discussion is not something I am super-comfortable with. However, I do think that it helped me to face the fact that I hesitate in posting.” Another student commented, “Online facilitation is not something I had experienced before and I now have more confidence (and some knowledge of what worked and didn’t work) to facilitate if asked again outside of this class.”

b. MeaningsStudents describe difficulties that instructors also voice. As online instructors, we struggle with the same concerns regarding effective facilitation (e.g., what types of questions to ask, how often to participate, how much to guide the discussion, how to give good feedback, how to best summarize the discussion, and how to guide critical inquiry). These are issues that should be addressed when preparing students to facilitate online discussions.

8. Theme Eight: General perceptions about student-led discussion activitya. EventsWhen asked about students’ overall perceptions of student-led discussions, students described the experience as helpful, useful, valuable, enjoyable, safe, and comfortable. One student noted that he/she preferred instructor-led discussions. Two comments that reflect the overall general perceptions include: “I like how each opinion was valued. I felt like I could explain what I was thinking freely and get feedback.”“At times there was a lot to process, but overall I enjoyed the discussions.”

b. MeaningsThe comments reveal general acceptance of student-led discussions as an enjoyable activity; however, there remains uncertainty (as expressed by the one student) that student-led discussions are less valued than instructor-led discussions.

9. Theme Nine: Logistics – student-led discussion activity designa. EventsThe final question asked for additional comments about the student-led discussion activity. These comments provided helpful insight regarding the logistics and design of student-led discussions. For example, students were generally okay with the bi-weekly timeline; however some felt more time was needed to process information. For example, on student noted: “Conversations need time to develop. I wish there was more time to respond. I would have loved to have gone back after reading things and answered questions from previous weeks.”Students also commented on the variety of the discussion topics and the ability to collaborate with other students who were leading their discussion during the same timeframe. One student noted his/her frustration with students waiting until the end of the time period to post a comment.

b. MeaningsPerhaps the student-led discussions could be set up in a way that provided a general timeline but also allowed enough flexibility for students to revisit discussion threads. Further, it seems it would be


beneficial to allow time before the end of the semester where there is not guided discussion but rather an opportunity for students to reflect on and share their experiences. The issue of waiting until the end of the time period to post could be addressed by acknowledging students who maintain a steady presence in the discussion from beginning to end.

VI. DISCUSSIONWe found that Akyol and Garrison’s (2009) metacognition construct was useful in exploring and examining deep instances of metacognition in online discussion forums. The results from the analysis of discussion forum transcripts using an inductive content analysis approach sheds more light on what happens in student-led discussions, and this information can aid researchers and practitioners to better understand and assess metacognition and how it is manifested in online discussions. With regard to our first research question: “What indicators of metacognition in a community of inquiry (CoI) are evident in student-led online discussions?” we found instances of all three indicators of metacognition, knowledge, monitoring, and regulation. However, most comments reflected instances of monitoring of cognition with the majority of those instances reflecting declarative/judging statements. While metacognition can be co-regulated, there was little evidence that students used the full range of metacognitive skills to help co-construct meaning. However, these interpretations should be taken with caution given the inherent difficulty in observing metacognition. Regarding metacognitive activities, De Wever et al. (2008) warned, “Concluding that students do not perform any kind of metacognitive activity might be wrong, as the absence of metacognitive statements might be caused by the fact that students do not communicate explicitly about these activities” (p. 38). Concerning research questions two: “What are students’ perceptions about leading and participating in student-led online discussions?” and three: “How do students’ perceptions of their student-led discussion experiences reflect metacognition processes?” we learned that students take the role of facilitator seriously and use metacognitive processes to help them prepare and act in that role. However, students expressed a feeling of tentativeness in their facilitator role and may lack the necessary skills to be effective. In a CoI, there is an expectation that aspects of teaching presence, such as facilitation of discourse, are shared among instructors and students. At the same time, it is taken for granted that students have the knowledge and skills to be successful in this role. Most students have limited knowledge and experience on how to be an effective online facilitator. While students may have some expertise facilitating a face-to-face discussion, they may be less likely to be able to facilitate a purely text-based discussion. These results imply the need to inform, educate, and motivate students in their roles as facilitators and teach them online facilitation strategies including how to use metacognitive strategies effectively. For example, informational resources and instructional modules on facilitation, as well as, metacognitive skills (i.e., assessing metacognitive awareness, questioning skills, feedback skills, making summaries, guiding critical inquiry) could be built into the design of the course or added as stand-alone instructional modules. When students demonstrate these skills within the course, instructors can privately or publicly (depending on the situation) acknowledge these students. As for our fourth research question: “What are the implications on the design of online student-led activities?” we learned that student-led online discussions can be used as an effective instructional activity to promote metacognition. However, instruction and guidance are needed not only to help students become aware of metacognitive skills but also to learn how to use them effectively in an online discussion. In addition, assignment of student roles such as a developer of alternatives in addition to student as facilitator and student as participant in the online discussion may support the development of knowledge construction (DeWever, et al., 2008). We acknowledge that this study has limitations. As a qualitative inquiry, we realize while our methods helped us examine metacognition in student-led online discussions in detail, results are limited to our personal interpretations of the data. However, we have attempted to provide transparency of our research process by providing clear and explicit details of our methods. A mixed methods study could help to offset the limitations of a purely qualitative study and present a more balanced portrayal of this


phenomenon (Creswell, 2014). Also, the data represent one class of master’s students within a specific course in a learning technology program. Future research in other knowledge domains could extend the generalization of these findings.

VII. CONCLUSIONSAkyol and Garrison (2011) suggested teachers and learners share roles and responsibilities in a CoI, and that these shared roles and responsibilities are inherent in the philosophical underpinnings of the CoI as a social-constructivist framework. However, Shea et al. (2012) noted while it is generally understood the teacher and the student to share roles within the context of teaching presence in a CoI, there are different motivations for teachers and students due to the nature of the outcomes of the educational experience. For example, students may engage in a CoI to acquire new knowledge and earn grades and/or credit, while teachers may engage in a CoI primarily to share knowledge and guide student learning experiences through effective instructional design and facilitation. These “real world dynamics that shape and constrain much of learning in practice” (p. 93) are important to recognize in order to extrapolate and address specific aspects about teaching presence within the context of those who participate in the CoI. One of these aspects is the facilitation of online discussions and how members (e.g., learners and teachers) in a CoI can use online discussions to achieve deeper learning. Metacognition (i.e., when one thinks about his/her learning) is a precursor to critical thinking and deep learning. Researchers have already acknowledged that metacognitive skills can be taught (Choi et al., 2005; Schraw, 1998) and that some metacognitive activities are more effective than others (Haller, Child, & Walberg, 1998; McCabe, 2011). However, additional studies are needed that investigate the use of instructional strategies whichfacilitate the development of metacognitive awareness and skills (See Choi et al., 2005 for an example) and the ways teachers and learners can apply metacognitive skills effectively. If we can assess metacognition in online learning environments, in which learners are largely self-regulated, we can integrate design strategies and learning activities that support metacognition (e.g., sharing of cognitive experiences) in asynchronous discussion forums.Taking an exploratory approach to data analysis helped us identify certain challenges in understanding how metacognition is manifested and facilitated in online discussions and why this understanding is important to know and to articulate to online learning practitioners. Future research could include the use of data visualization strategies to more effectively analyze the data and track patterns of metacognition dimensions as well as design-based studies that integrate information about metacognition and instructional strategies for developing metacognitive awareness and skills in an online learning environment.


VIII. REFERENCESAkyol, Z., & Garrison, D.R. (2011). Assessing metacognition in an online community of inquiry. The

Internet and Higher Education, 14, 183-190.

Baran, E., & Correia, A. P. (2009). Student-led facilitation strategies in online discussions. Distance Education, 30(3), 339-361.

Chenail, R. J., Cooper, R., & Desir, C. (2010). Strategically reviewing the research literature in qualitative research. Journal of Ethnographic & Qualitative Research, 4, 88-94.

Choi, I., Land, S., Turgeon, A.J. (2005). Scaffolding peer questioning strategies to facilitate metacognition during online small group discussion. Instructional Science, 33, 483-511.

De Wever, B., Van Winckel, M., & Valcke, M. (2008). Discussing patient management online: The impact of roles on knowledge construction for students interning at the paediatric ward. Advances in Health Sciences Education, 13(1), 25-42.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906-911.

Garner, R. (1990). When children and adults do not use learning strategies: Toward a theory of settings. Review of Educational Research, 60(4), 517-529.

Garrison, D.R. & Akyol, Z. (2013). Toward the development of a metacognition construct for communities of inquiry. The Internet and Higher Education, 17, 84-89.

Garrison, D.R., Anderson, T. & Archer, T. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education 2(2-3), 87-105.

Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework: Review, issues, and future directions. The Internet and Higher Education, 10, 157-172.

Harasim, L. (2012). Learning theory and online technologies. New York, NY: Taylor & Francis Group.

Hart, J.T. (1965). Memory and the feeling-of-knowing experience. Journal of Educational Psychology, 56(4), 208-216.

Hull, D. M., & Saxon, T. F. (2009). Negotiation of meaning and co-construction of knowledge: An experimental analysis of asynchronous online instruction. Computers & Education, 52, 624-639.

Marshall, C. & Rossman, G.B. (2011). Designing qualitative research (5th ed.). Thousand Oaks, CA: Sage Publications, Inc.

Martin, K.H. (2013). Leveraging disinhibition to increase student authority in asynchronous online discussion. Journal of Asynchronous Learning Networks, 17(3), 149.164.

Mayer, R.E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving. Instructional Science, 26(1/2), 49-63.

McCabe, J. (2011). Metacognitive awareness of learning strategies in undergraduates. Memory and Cognition, 39(3), 462-476.

McInerney, D. M. (2005). Educational psychology – theory, research and teaching: A 25-year retrospective. Educational Psychology, 25(6), 585-599.

Palincsar, A. S. (1998). Social constructivist perspectives on teaching and learning. Annual Review of Psychology, 49, 345-375.

Sandelowski, M. (2000). Whatever happened to qualitative description? Research in Nursing & Health, 23, 334-340.


Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26, 113-125.

Smith, C.P. (2000). Chapter twelve: Content analysis and narrative analysis. In Reis, H.T. and Judd, C.M. (Eds.), Handbook of research methods in social and personality psychology (pp. 313-335). New York, NY: Cambridge University Press.

Smith, J. A., Flowers, P., & Larkin, M. (2009). Interpretative phenomenological analysis: Theory Method and research. London: Sage.

Smith, J. A., & Osborn, M. (2003). Interpretative phenomenological analysis. In J.A. Smith (Ed.), Qualitative psychology: A practical guide to methods. London: Sage.

Young, A., & Fry, J. D. (2008). Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 8(2), 1-10.

IX. APPENDICESA. Appendix A: Student-Led Discussion QuestionnaireThank you for participating in the student-led discussions for this course. I am interested in your opinions about this activity. Please take a few minutes to answer the following eight questions. Although completion of the questionnaire is purely optional, I value your input and your responses will remain completely anonymous. As the facilitator of the discussion, how did you go about preparing for your particular discussion topic? What resources did you use? How much time did you spend preparing for your topic? How did having to facilitate the discussion help you to learn the course content for your assigned period? Looking back, what would you do differently before, during, or after your assigned discussion? Why? As a participant in the student-led discussion, what were your expectations of the role you played? Did the discussions help you communicate, explain, or justify your thinking? How? Was the bi-weekly discussion timeline with two or more facilitators reasonable? Why? Why not?Overall, what did you find most beneficial about the student-led discussion assignment?Overall, what did you find most difficult about the student-led discussion assignment?Please provide any additional comments related to the student-led discussion assignment that you wish to share.

B. Appendix B: Coding GuideIn addition to using the descriptors of knowledge of cognition, monitoring of cognition, and regulation of cognition from Akyol and Garrison’s (2011) metacognition construct (See Figure 1), we used the following information to help us analyze the discussion forum transcripts.

1. Knowledge of Cognition (KC) “Pre-Task Dimension”The “pre-task metacognitive state…Examples of KC are students’ assessment of how they learn best, what they know or do not know about the subject matter, or how they feel with regard to the task or their ability” (Akyol & Garrison, 2011, p. 184). “Taking on the responsibility of teaching presence, enables students to reflect on each other’s contributions and their contribution to the developmental progress toward the intended goals while they are engaged in discourse” (Akyol & Garrison, 2011, p. 184). Look for evidence in the discussion forum post that seem to show evidence of the following statements. (Statements from Garrison & Akyol, 2013.) I know my strengths as a learner. I know my weaknesses as a learner. I have good critical thinking skills


I have good problem solving skills. I know what factors may enhance my thinking and learning. I know my motivational state at the beginning of the learning process. I am clear on my opportunities for success.I know my existing knowledge and experiences related to the learning task.

2. Monitoring of Cognition (MC) – “Reflective Dimension”“The monitoring dimension specifically includes the awareness and willingness to reflect upon the learning process” (Akyol & Garrison, 2011, p. 184). Look for evidence in the discussion forum post that seem to show evidence of the following statements. (Statements from Garrison & Akyol, 2013.) I make judgments about the difficulty of the tasks.I am aware of my effort during the learning process. I am aware of my level of thinking during the learning process. I constantly monitor my feelings during the learning process. I consciously assess my understanding during the learning process. I realize I need confirmation of my understanding. I pay attention to other course participants’ ideas/understandings/comments.I think about how we are approaching the task.

3. Regulation of Cognition (RC) – “Active Dimension”Regulation of cognition is the “action dimension of the learning experience. It is the enactment and control of the learning process through the employment of strategies to achieve meaningful learning outcomes” (Akyol & Garrison, 2011, p. 184). Look for evidence in the discussion forum post that seem to show evidence of the following statements. (Statements from Garrison & Akyol, 2013.) I set goals to achieve a high level of learning.I modify my approach to enhance my effort. I ask questions or request information to deepen my thinking. I challenge myself and other course participants.I make suggestions to other course participants to help their learning. I apply specific strategies to enhance my understanding.I ask for help when I encounter difficulty. I modify my goals or strategies when I encounter difficulty in understanding. I change my strategy depending on the tasks.I try to control my anxiety to enhance my understanding.


The Effect of Structured Divergent Prompts on Knowledge Construction

Ginger S. HowellUsenime AkapnudoMengyi ChenHarding University

Autumn L. SutherlinLaura E. JamesAbilene Christian University

ABSTRACTDiscussion forums are a widely used activity in online courses. However, knowledge construction within online discussion rarely stimulates high levels of thinking. Therefore, it is important to understand which aspects of online discussion encourage learning and increase knowledge construction. This paper investigates the effect three Structured Divergent prompts (playground prompts, brainstorm prompts, and focal prompts) have on knowledge construction as compared to Convergent prompts. Students (N = 58) were required to participate in an online discussion during a graduate education course at a private university. The Interaction Analysis Model was used to determine the levels of knowledge construction demonstrated within students’ posts. The posts were given a score using the following codes: 0-no post/no understandable post; 1-sharing information, 2-disagreeing; 3-negotiation of meaning; 4-testing co-construction; 5-agreement of the constructed meaning. The analysis revealed two of the three Structured Divergent prompts (focal and brainstorm) yielded significantly higher levels of knowledge construction as compared to Convergent prompts.

I. INTRODUCTIONAccording to Gagne (1970), various internal and external conditions are required for learning. The underlying idea of Gagne’s work is that external conditions affect the learner internally. As Gagneexplained, “[a] learning event … takes place when the stimulus situation affects the learner in such a way


that his performance changes from a time before being in that situation to a time after being in it” (1970,p. 5). Gagne believed that external learning conditions should be planned before learning can occur. In hiswords, “[d]esigning instruction for initial learning is thus seen to be a matter of setting up a total set of circumstances within which learning can be embedded” (Gagne, 1970, p. 322). Gagne’s principle of Instructional Design forms the basic underlying principle of this study. One external condition studied using Bloom’s Taxonomy is the prompt type used. Prompts can take many different forms. As defined by Berthold, Eysink, and Renkl, “prompts are requests that require the learners to process the to-be-learned contents in a specific way” (2009, p. 347). One way to classify prompts is as Convergent or Structured Divergent prompts. Andrews’ (1980) implied that Convergent prompts require students to think too broadly or too narrowly, which restricts students from effectively solving the problem. Prompts which are classified as Convergent include general invitation, analytic convergent, single, multiple consistent, shotgun, quiz show, and funnel questions (Andrews, 1980). For example, in this study we used the following convergent funnel prompt for the control group: “What are your thoughts about how the legal system or education system works as discussed in chapter one? What jumped out at you that you did not already know? What are some things you already knew? Do you still have questions about the legal system?”Divergent prompts help students focus on the question at hand and require learners to work together to provide an answer. Structured Divergent prompt types include the following: playground, brainstorm, and focal questions. First, playground questions are prompts which focus on “a promising sub-aspect of the material,” such as a specific aspect of literature, history, or concept being studied (Andrews, 1980, p. 157). As Andrews discussed, the playground question takes an aspect of the content and asks students to discuss the topic within the bounds set. This keeps students talking about the same topic, yet allows for different aspects to be discussed. In this study, for example, the following playground prompt was used:“What are your thoughts about limiting or eliminating gangs in schools?” This prompt allows discussion on how to limit gangs, possibly through stricter regulations, extra-curricular activities, or omit stronger support systems within the school. It allows students to also discuss the positive and negative aspects of limiting gangs in schools. Many aspects can be discussed. However, the prompt restricts students from discussing anything outside of limiting gang activity, such as the manifestation of gangs, signs of gangs in schools, and impact of gangs in a school system.Second, Andrews described brainstorm questions as those that generate ideas and solutions by encouraging students to collaborate. As an example, in this study the following brainstorm question was used: “In what ways do you believe that school personnel could be charged with slander or libel?” This prompt requires students to collaborate ways or instances that school personnel could be charged with libel. With such a prompt, students don’t just regurgitate information, but they read each other’s posts and come up with original answers to build upon previous discussion. Third, Andrews (1980) defined focal questions as questions which involve a complex controversy with more than one possible solution. These prompts force students to choose an argument and prepare a supportive rational. In this study, the following focal prompt was used to address the topic of safety:“What security and safety measures should be implemented or eliminated in schools? Explain your argument.” By requiring students to explain their opinions of what safety measures should be put into place, students have to choose and justify their argument. Often times, students need to read others’ arguments first before they come up with their argument. Previous studies, such as Andrews (1980), Bradley, Thom, Hayes, and Hay (2008), and Wruck (2010) created a foundational understanding of prompts and discussed the impact of Structured Divergent prompts on critical thinking. These studies determined that certain prompts can affect students’ thinking, and identified which prompt types are most effective. Without using the above studies, the researchers of this study would not have known which prompts to examine. Once the effective prompts were identified, the researchers measured knowledge construction in students’ discussion posts using the Interaction Analysis Model (IAM), which differs from previous studies.


II. LITERATURE REVIEWA. Impact of Structured Divergent PromptsAndrews (1980) studied eleven types of prompts within the two larger categories of Convergent and Structured Divergent designs. He found that different prompt types can impact and/or limit the extent of student responses within a discussion. According to Andrews (1980) divergent prompts (brainstorm, focal, and playground) helped instructors encourage a more robust discussion because students have the freedom to fully discuss the topic without getting off track. Andrews (1980) investigated the effect of the prompt types in a face-to-face environment.One of his research questions examined certain characteristics of prompts and their association with various levels of participation. His results showed that Divergent prompts are more productive than Convergent prompts. The study also found that higher-level prompts, based on Bloom’s Taxonomy, received more responses from students. Prompts that exhibited structure and boundaries were more productive when generating discussion than prompts without structure and boundaries. Interestingly, there was no significant difference between unfocused prompts and low-level divergent prompts, which address the first three levels of Bloom’s Taxonomy: knowledge, comprehension, and application. It was found that unfocused prompts were less effective than Structured Divergent prompts. However, it was noted that a larger pool of unfocused prompts is necessary to replicate and validate the results.Overall, Andrews (1980) found Structured Divergent prompts to be three times more productive in discussion than other types of prompts. It was seen that when teachers used Structured Divergent prompts consistently, the results were “fruitful” (Andrews, 1980, p. 154). Andrews reported that although the Structured Divergent prompt types were effective, they differed in various qualities. For example, focal questions were most likely to encourage student-to-student interaction while brainstorm questions relied on the teacher to focus productivity. Focal questions encouraged competition, whereas playground questions encouraged collaboration. The study showed that all three Structured Divergent prompt types were classified at the higher end of Bloom’s Taxonomy, but the three types differed in how they required students to process data. Playground questions required students to interpret data; focal questions were deductive; and brainstorm questions were both inductive and deductive. Andrews’ (1980) study showed that prompts are an external condition that affected the internal response of the learner. In addition, the study narrowed down a number of prompt types and found that Structured Divergent prompts were the most effective. It is also important to note that although the study determined which prompt types were designed to reflect levels of critical thinking, Andrews did not code student responses to determine if higher levels of critical thinking were actually demonstrated.In 2008, Bradley et al. used Bloom’s Taxonomy to examine whether different types of prompts influenced learning during online discussions. Prompt types used were direct link, course link, brainstorm, limited focal, open focal, and application. Bradley et al. investigated which prompt type generated the highest word count, generated the most complete answers, and resulted in higher-order thinking in answers and responses. Bradley et al. (2008) used Gilbert and Dabbagh’s (2005) coding scheme to examine higher-order thinking in student responses based on the three highest levels of Bloom’s Taxonomy. The researchers also examined word count and answer completion in the transcripts of eight online discussions provided by 114 undergraduate students in three sections of a hybrid child development course. The eight discussions, each consisting of three prompts, were considered hot-topics in child development. The responses were divided into two groups: answers, which consisted of 1380 analyzed posts, and responses, which consisted of 849 analyzed posts. Statistical analysis of word count showed that limited focal prompt generated the most words. Similar to the conclusions from Andrews’ (1980) study, the authors believed this was because controversy introduced by the prompts allowed for opinions and other alternatives to be discussed. Application prompts encouraged the least amount of words.


Gilbert and Dabbagh (2005) also rated student responses using a four-point Likert scale from “noresponse” to “full response.” A statistical analysis showed that limited and open focal prompts encouraged students to complete the answers while application and course link prompts did not. This was again thought to have occurred because the limited and open focal prompts encouraged opinions. However, writing a lengthy comment that addressed each part of the prompt did not mean students gave higher-level answers and responses.In the third portion of the study, Gilbert and Dabbagh’s coding system was employed to determine whether the answers and responses were rated as higher-order in Bloom’s Taxonomy. The following codes were given to students’ posts: knowledge (1), comprehension (2), application (3), analysis (4), synthesis (5), and evaluation (6). When students answered incorrectly, a zero was given. It is important to note that when students demonstrated two levels of critical thinking in their answers, an average score was given. For example, “a student with two responses rated a one and three received the average rating of two” (Bradley et al., 2008, p. 893). This average does not show the highest level a student eventually reached, and it is not noted how many responses were an average and how many responses reached only one level. The results showed that most of the participants’ answers reflected lower-order thinking skills; however, at a statistically significant rate, course link, brainstorm, and direct link prompts encouraged higher levels of critical thinking than open focal and application, which produced the lowest. Although course link prompts did not score high in the other research questions, the prompt type produced higher-order thinking responses because it encouraged students to discuss prior knowledge and other resources.The brainstorm prompts and direct link prompts ranked high. Bradley et al. stated that the brainstorm question type “seemed to facilitate students justifying their solution by bringing in prior knowledge or examples from their own life…” (p. 898). Bradley et al. (2008) did not discuss which specific levels were reached. Instead, the authors reported the results in terms of two groups: lower levels of thinking (levels 1-3 of Bloom’s Taxonomy) or higher levels of thinking (levels 4-6 of Bloom’s Taxonomy). At times, Bradley et al. noted that an average level was demonstrated. However, the reader cannot determine if any posts ever reached a particular higher level, since no specifics were given.Wruck (2010) also studied the effect of prompt types on critical thinking using Bloom’s Taxonomy.Unlike the study done by Bradley et al. (2008), this study used archived discussions from graduate students and, overall, was more elaborate in describing its findings. The study examined which levels of Bloom’s Taxonomy were exhibited when students responded to several Computer-Mediated Communication (CMC) prompt types and the pattern among responses in relationship to Bloom’s Taxonomy for each prompt type. Wruck (2010) stated that “[t]he review of literature commonly illustrate[d] five prompts, including (a) read and respond, (b) scenario, (c) case study, (d) controversy/debate, and (e) search and critique (Christopher et al., 2004; Hmelo-Silver, 2004; Hughes & Dayken, 2002; McDade, 1995; Moore & Marra, 2005)” (p. 1). Wruck (2010) examined courses in a Doctor of Business Administration (DBA) program that used at least four of the five prompt types she was investigating. Two courses from the DBA core and two from the concentration areas were selected using the Excel sampling feature.After the discussions were chosen, each post within the discussion was assigned a level of Bloom’s Taxonomy: knowledge (1), comprehension (2), application (3), analysis (4), synthesis (5), and evaluation (6). The codes were used to determine if a pattern existed between the type of prompt and the level of Bloom’s Taxonomy. A total of 491 learner responses were analyzed. The researcher counted the number of posts in each category for each strategy to obtain the average cognitive level.Wruck (2010) found that 83% of the learner responses consisted of the second, third, and fourth levels of Bloom’s Taxonomy (application, analysis, and synthesis). Read and respond prompts averaged a level three (application) but also had reached level four (analysis) once. Scenarios demonstrated a level four (analysis) as an average and reached a level six (evaluation). Case study averaged a level five (synthesis) and also reached a level six (evaluation). The average level of controversy/debate prompts was a level


three (application). Search and critique prompts demonstrated a level three (application) as well. The overall results of Wruck’s (2010) study showed a link between cognition and instructional design, as some prompt types yielded higher levels of Bloom’s Taxonomy than others. This demonstrates that the external condition of prompt type affects learner achievement.

These studies from Andrews (1980), Bradley, Thom, Hayes, and Hay (2008), and Wruck (2010) show Bloom’s Taxonomy can be used to evaluate discourse and that prompt designs affect student learning. However, Bloom’s taxonomy was not used for this study. Booker (2007) stated that Bloom’s taxonomy has been misused in education because it was created by thirty-four educators, psychologists, and school examiners to classify test questions, not to assess student responses. The previous articles discussed used Bloom’s taxonomy to classify both the prompts and the student responses. Therefore, to more accurately determine whether prompt designs affect student responses, the Interaction Analysis Model (IAM), whichis a tool designed to assess student knowledge construction within discourse, was used for this study. In addition, IAM was used for this study because previous studies have not examined prompt design using this method.

B. The Interaction Analysis ModelGunawardena and colleagues (1997) used France Henri’s (Henri, 1992) research to create a tool specifically designed to analyze knowledge construction in CMC. Another model was needed because Henri’s model, based on instructor-centered learning, did not examine the collaborative learning process.In addition, Henri’s model did not holistically investigate discussion and made it difficult to distinguish between the metacognitive and cognitive dimensions (Gunawardena et al., 1997). After investigating a number of analysis tools, the Interaction Analysis Model (IAM) was designed, based on constructivism, to categorize segments of online discussion posts into five levels of knowledge construction. Each level had multiple sub-category descriptions solely to aid the coders in determining the level of knowledge construction (Figure 1). However, the IAM analysis tool did not give individual scores that distinguished among subcategories.

Gunawardena et al. (1997) used a coding sheet to identify the level (Figure 1: next page) demonstrated within each message. Submitted comments were given a number associated with a level of knowledge construction. Posts which yielded multiple codes were recorded at the highest level (Gunawardena et al., 1997; Dunlap et al., 2007). Gunawardena et al. found that the number associated with each level could be used to quantifiably track knowledge construction. Gunawardena et al. (1997) noticed two themes when analyzing dialogue using IAM. First, knowledge construction proceeded from level one to level five in some discussion threads. Second, multiple levels of knowledge construction could be demonstrated within one post. The authors found most posts demonstrated only the first level of knowledge construction. However, some students started at lower levels and moved towards the third level, but rarely exceeded the third level.

C. SummaryWhile previous studies have addressed the effect of divergent prompts on critical thinking in both online and face-to-face environments, the effect of divergent prompts on knowledge construction using IAM has not been studied. Therefore, the question remains, how can higher levels of knowledge construction be consistently achieved in online learning? Hopkins et al. (2008) claimed that it is probable that some taskdesigns can promote knowledge construction. A better understanding of which external conditions create successful online discussion is needed. Studies have already examined prompt types using Bloom’s Taxonomy (Bradley et al., 2008; Wruck, 2010), but student responses using IAM have not been studied in correlation to specific prompt types


Figure 1

1. Sharing/Comparing Informationa. Statement of observation/opinionb. Statement of agreementc. Supportive examples/commentsd. Asking/answering questionse. Define/describe/identify problem

2. Dissonance or inconsistency among ideasa. Disagreeingb. Asking/answering in concerns to disagreementc. Restating position

3. Negotiation of meaning/co-construction of knowledgea. Negotiation or clarification of the meaning of termsb. Negotiation of the relative weight to be assigned to types of argumentc. Identify areas of agreementd. Proposal and negotiation of new statements showing compromise and co-constructione. Metaphors or analogies

4. Testing/modifying synthesis or co-constructiona. Testing synthesis against shared responsesb. Testing against schemac. Testing against experienced. Testing against datae. Testing against literature

5. Agreement statement(s)/applications of newly constructed meaninga. Summarization of agreement(s)b. Application of new knowledgec. Metacognitive statements of participants illustrating understanding

Figure 1. Knowledge Construction Hierarchy (Gunawardena et al. 1997)

III. STUDYA. QuestionsQuestion 1: How does the playground prompt affect the levels of knowledge construction based upon IAM scores compared to students who respond to prompts that do not use any of the three Structured Divergent prompts after controlling for pre-test IAM scores?Question 2: How does the brainstorm prompt affect the levels of knowledge construction based upon IAM scores compared to students who respond to prompts that do not use any of the three Structured Divergent prompts after controlling for pre-test IAM scores?


Question 3: How does the focal prompt affect the levels of knowledge construction based upon IAM scores compared to students who respond to prompts that do not use any of the three Structured Divergent prompts after controlling for pre-test IAM scores?

B. ParticipantsThe participants in this study consisted of 58 students (N = 58) enrolled in a mandatory online graduate level school law course taught by three instructors in the spring of 2013. The southern faith-based university at which the study was conducted has a student population of approximately 6,000 students with 1,000 students enrolled in graduate education programs. As part of the course orientation during the first week of the semester, students were asked to participate in the study. Students declined or gave permission for their work to be collected and analyzed. Participants were identified before the treatment began. All students participating in the study signed an informed consent form approved by the university’s Internal Review Board.

C. Method and ProcedureStudents were randomly assigned to four discussion groups within the course before the study began.Each of the four groups was assigned the type of prompt they would receive. The four groups consisted of15, 16, 15, and 12 students respectively. The groups stayed the same after students identified whether they were willing to participate in the study, and all students were exposed to the same content and expectations. All four groups discussed the same topic, however, prompts were worded differently to reflect the prompt type assigned to each group. In addition, the participants were subjected to different instructors. Three instructors taught four sections, with one instructor teaching two of the four sections. Non-participants’ discussion posts were not included in our results. Before the start of the course, Convergent and Structured Divergent prompts were designed with the input of the course’s instructors. A trained instructional designer created prompts for both the control group and the treatment group. The control group received Convergent prompts (multiple consistent, funnel, general invitation, analytic convergent, quiz show, shotgun, lower-level divergent, and single questions), and the three treatment groups received the Structure Divergent prompts (playground prompt, focal prompt, or brainstorm prompt). For the control group, the instructional designer used prompts that have been implemented in previous sections of the course since they already reflected a variety of Convergent prompts. For example, the researchers used a convergent funnel prompt in the control group for a chapter focusing on the legal system. This prompt had many questions that students had to answer. Therefore, students’ thoughts were not focused on one idea, but many. For the treatment group, the designer used Andrew’s (1980) article, the course textbook, and the section instructor’s input to change existing prompts into Structured Divergent prompts. One of the playground questions for the chapter on gangs asked students to discuss their thoughts about limiting or eliminating gang activity in schools. This prompt allowed discussion on how to limit gang activity in schools, possibly through stricter regulations, extra-curricular activities, or omit stronger support systems within the school. As described earlier, the prompt restricts students from discussing anything outside the scope of gang activity. One of the brainstorm prompts for the chapterdiscussing slander asked participants to come up with ways school personnel could be charged with slander or libel. This required students to develop a list together, not just regurgitate information from text. One of the focal prompts of the chapter focusing on safety asked participants to identify the best security and safety measures that should be implemented or eliminated in schools and required students to defend their answer. This encouraged students to not solely rely upon information they read, but to use personal experiences to back their argument. A pre-test/post-test equivalent group design was used for this study. During the first three weeks of the course, all groups received Convergent prompts during the discussion. Then, the data was collected at week 3 so that students were accustomed to Convergent prompts; therefore, the level of knowledge construction was not influenced by the unfamiliarity of the prompt type. To provide a baseline that would


serve as a covariant, all groups were exposed to Convergent prompts through the third week and week 3 responses were scored using the IAM analysis tool. Each response received a rating of 0-5 on the IAM analysis tool. A zero was given when students did not respond or responded but did not measure on theIAM scale. For each student, the ratings from all three posts were combined to give the student a score of 0-15. Three posts (one initial response and two replies) were required each week. Therefore, only threeresponses from each student were used. During weeks 4 through 12, the three treatment groups received their assigned Structure Divergent prompts. The control group continued to receive Convergent prompts from week 4 to week 12. The data was collected during week 12 to allow students to have an adequate amount of time to be subject to the treatment before an assessment was done. Since the researchers were not concerned about a gradual change, an assessment was not done in the middle of the treatment period.Responses to week 12 discussions were scored as the post-test using the same method described for the pre-test.

D. InstrumentThe Interaction Analysis Model (IAM) tool was designed to detect and understand knowledge construction during collaborative discussions (Saritas, 2006). The IAM responses were classified in the five levels of knowledge construction (Figure 1) in a social constructivist environment (Schellens et al., 2007; Buraphadeja & Dawson, 2008; Wang et al., 2009; Gunawardena et al., 1997): “(1) sharing and comparing information, (2) identifying areas of disagreement, (3) negotiating meaning and co-construction of knowledge, (4) evaluation and modification of new schemas that result from co-construction, and (5) reaching and stating agreement and application of co-constructed knowledge” (DeWever et al., 2009, p. 181). Krippendorff's alpha inter-rater reliability for the IAM has been determined to range between 0.40 and 0.80 (DeWever et al., 2009).

E. Data Collection, Analysis, Storage, and ProtectionData was collected electronically from the course’s learning management system. The instructors compiled students’ responses—both students’ identities and the prompts assigned to each group were masked. Only the first three responses posted by each student were included in the statistical analysis. The researchers used the IAM to assign codes from zero to five to indicate the levels of knowledge construction demonstrated in multiple student discussions (Gunawardena et al., 1997; Hew & Chueng,2011). They used the knowledge construction hierarchy shown in figure one to categorize each of the responses. Following Gunawardena’s procedures, the subcategories were only used in identifying the level of knowledge construction and for analysis did not receive scores that distinguished among them within a single level. When the posts consisted of more than one code, the highest code was used for analysis. When the student did not post or if the post did not demonstrate any of the levels, a zero was assigned. Two teams of researchers at two universities coded participants’ posts. Each team consisted of a lead coder and assistant coder. The lead coders and assistant coders individually rated the posts. The lead coder and assistant coder compared their codes and any conflicted codes were agreed upon. The two lead researchers then compared the codes and a consensus score was assigned to each post. Krippendorff's alpha inter-rater reliability between the two sets of consensus codes was 0.52, which is comparable to the results obtained by DeWever et al. (2009). Intraclass correlation (3, k) between the two sets of codes was also determined to be 0.69. The coding teams were unaware of which prompts were Structured Divergent prompts and Convergent prompts. The Statistical Package for the Social Sciences (SPSS) was used to conduct statistical analysis.Kolmogorov-Smirnov statistics were examined for each group to test the assumption of normal distribution. A one-way analysis of covariance (ANCOVA) of IAM scores, where the week 3 posts were used as the pre-test covariant, was used to test for overall difference between the groups. To specifically address the research questions, simple non-orthogonal contrasts were used as follow-up to ANCOVA, individually comparing each treatment group to the control group.


IV. RESULTSA. Descriptive StatisticsAll students received Convergent prompts for the first three weeks of the course. Week 3 posts were used as a pre-test and analyzed with the IAM rubric. Responses to the Convergent prompts by all groups could then be used as a covariant for analysis. During weeks 4 through 12, groups were given their assigned prompt types: Convergent, playground, brainstorm, or focal. Student responses for week 12 were then analyzed with the IAM rubric as a post-test. Only the first three posts from each student were used for analysis. Summaries of the responses by group are shown in Table 1.

1. Playground group: Pre-testWhen receiving the Convergent prompt, the 15 students in the playground group responded 40 times out of the required 45. One student gave an unrequired fourth post. Five zeros were assigned because no post was given. Thirty posts reached a level one. Eight posts reached a level two. One post reached a level three. One post achieved a level four, and no posts received a level five. The average score on the pre-test was 1.2 (Table 1).

2. Playground group: Post-testFifteen students received the playground prompt and gave 36 out of the required 45 posts. In addition, there were five posts beyond the required three posts per student that were not included in statistical analysis. Nine zeros were given because the participants did not post a required response. Twenty-seven posts reflected level one, three reflected level two, two reflected level three, four reflected level four, and zero reflected level five. The average score on the post-test was 1.2 (Table 1).

3. Brainstorm group: Pre-testSixteen students in the brainstorm group responded to Convergent prompts 29 out of 48 required times.Three of the posts were an unrequired fourth or fifth post. A total of nineteen zeros were given when students did not post a required response. Twenty-three posts reached level one, two posts level two, and four posts level three. No posts achieved levels four or five. The average score on the pre-test was 0.8 (Table 1).

4. Brainstorm group: Post-testSixteen students participated in the brainstorm group with 37 out of 48 required posts. Three additional posts were not included in the statistical analysis as they extended past the three per student-required posts. Eleven zeros were given because students did not post a required response. One post did not score on the IAM rubric and also received a zero. Twenty-seven posts reflected level one, two reflected level two, three reflected level three, four reflected level four, and zero reflected level five. The average score on the post-test was 1.2 (Table 1).

5. Focal group: Pre-testFifteen students in the focal group posted 36 out of a required 45 times on the convergent prompt pre-test.Three of the posts were an unrequired fourth or fifth post and therefore were not included in statistical analysis. Nine zeros were assigned because a student did not post a required response. Twenty-four responses reached level one, seven reached level two, four reached level three, one reached level four, and zero reached level five. The average on the pre-test was 1.2 (Table 1).

6. Focal group: Post-testIn response to focal prompts, 15 students posted 41 out of a required 45 times. An additional four posts were not included in the statistical analysis as they extended past the three required posts. Four zeros were given because students did not post a required response. Twenty-five posts reflected level one, four reflected level two, one reflected level three, eleven reflected level four, and zero reflected level five. The average on the post-test was 1.8 (Table 1).


Table 1

Number of posts at each IAM level.

Note: Pre-test posts were from week 3 when all students received convergent prompts. Post-test posts were from week 12 when students received the type of prompt indicated by their groups. a Students were expected to post 3 times. Some students, however, chose to post more than the expected 3. These were not included in analysis.b One post did not score on the rubric.

Playground Brainstorm Focal Convergent

# of Students 15 16 15 12

Required Posts 45 48 45 36

Pre Post Pre Post Pre Post Pre Post

Total Posts 41 41 32 40 39 45 23 14

No Posts 5 9 19 11 9 4 14 22

Extra Postsa 1 5 3 3 3 4 1 0

Level 0 5 9 19 12b 9 4 14 22

Level 1 30 27 23 27 24 25 13 11

Level 2 8 3 2 2 7 4 8 0

Level 3 1 2 4 3 4 1 1 2

Level 4 1 4 0 4 1 11 0 1

Level 5 0 0 0 0 0 0 0 0

Average Score 1.2 1.2 0.8 1.2 1.2 1.8 0.9 0.6


7. Convergent group: Pre-testThe twelve students of the convergent group posted 22 out of the required 36 times. There was one unrequired fourth post. Fourteen zeros were assigned because students did not post a response. Thirteen posts received a one, eight posts received a two, and one post received a three. No posts reflected levels four or five. The average on the pre-test was 0.9 (Table 1).

8. Convergent group: Post-testTwelve students in the Convergent group posted 14 out of the required 36 times. There were no extra posts. Twenty-two zeros were given because students did not post a response. Eleven posts reflected level one, none reflected level two, two reflected level three, one reflected level four, and zero reflected level five. The average on the post-test was 0.6 (Table 1).

B. Statistical AnalysisSPSS was used to conduct ANCOVA of IAM scores to assess whether various Structured Divergent prompts (playground, brainstorm, and focal) affected graduate students’ level of knowledge construction as compared to the control group when controlling for pre-test IAM scores. Before running ANCOVA, Kolmogorov-Smirnov statistics were examined for each group to test the assumption of normal distribution. The assumption was met for the playground group (p = .051) and for the brainstorm group (p = .189). Although this assumption was not met for the focal group (p = .043) and the control group (p = .002), ANCOVA is known to be robust against such mild violations (Leech, Barrett, & Morgan 2011).The assumptions of homogeneity of variances F (3) = .755, p = .525, and homogeneity of regression slopes F (3, 41) = .708, p = .553, were also examined and met. Results of the ANCOVA revealed that pre-test IAM scores served as a significant covariate in the analysis F (1, 44) = 9.11, p = .004 (Table 2).Furthermore, all treatment prompts together had a probability of .015, which indicated that there was a significant difference in IAM scores between the groups after controlling for pre-test IAM scores (Table 3). It is important to note that the three scores demonstrated by each student were combined for a maximum score of 15. Therefore, the mean reflects a total of fifteen, instead of the five levels used to code each individual post. Because a pre-test was compared to a post-test, any student who did not complete the pre-test or post-test was not used in the analysis. Therefore, the participant numbers dropped to 14, 12, 14, and 9 as seen in Table 3 (next page).

Table 2

Analysis of Covariance for IAM Scores as a Function of all Structured Divergent Prompts, after Controlling for IAM Pre-test Scores

Source df MS F p 2

Pre-test IAM 1 26.90 9.11 .004** .172

All Treatment Prompts 3 11.44 3.88 .015* .209

Error 44 2.95* Significant at p<.05** Significant at p<.005


Finally, to address each research question, a simple contrast analysis was conducted as a follow up to the omnibus ANCOVA comparing each IAM prompt group to the control group. The contrast results indicated a statistically significant difference between the adjusted group mean for the control group and those of the brainstorm group (p = .03, d = 1.01). Similarly, a statistically significant difference was found between the adjusted control group mean and the adjusted mean for the focal group (p = .003, d = 1.42).However, no significant difference was observed between the adjusted control group mean and the adjusted mean for the playground group (p =.18, d = .60). The differences between the control group and the focal and brainstorm groups represented a large effect according to Cohen’s guidelines (Leech, et al., 2011).

Table 3

Post-Test Cumulative IAM Score Means: Unadjusted and Adjusted for the Pre-test IAM Score

Unadjusted Adjusted

Source N Ma SD Ma SD

Playground 14 3.93 1.59 3.77 .46

Brainstorm 12 4.33 2.15 4.47 .50

Focal 14 5.36 1.78 5.16 .46

Control 9 2.33 2.00 2.72 .59a The cumulative score is a sum of the three IAM scores for each student.

V. DISCUSSIONPrevious studies have demonstrated the effect of different prompt types on student responses. Andrews (1980) showed that Structured Divergent prompts in face-to-face discussions caused students to give a more productive answer. Bradley et al. (2008) and Wruck (2010) both used Bloom’s Taxonomy to look at the effect of various prompts in online discussions. Bradley et al. (2008) found that limited and direct link prompts generated longer responses, while limited and open focal responses generated the most complete answers. Course link, brainstorm, and direct link prompts encouraged higher levels of critical thinking than open focal and application. Wruck (2010) found that scenario, case study, controversy/debate, and search and critique prompt types averaged a level three (application) in Bloom’s Taxonomy, while read and respond only averaged a two (comprehension). Although two of these studies looked at the level of thinking skills as a function of prompt type, none addressed knowledge construction. We, therefore, attempted to look at the effect of the Structured Divergent prompts on students’ knowledge construction in an online discussion. Students in this study were assigned to four groups. During the first three weeks of the course all groups were given Convergent prompts as a pre-test. During the third week, three posts from each student were given an IAM score. During weeks 4 through 12, each group was given a different prompt type (playground, brainstorm, focal, or Convergent). Three posts from each student were scored on the IAM rubric during the twelfth week as the post-test. In both the pre- and post-test, the scores for the three prompts from each student were added to give a score out of 15. These cumulative scores were then used as a mean for each group.


A. Question 1How does the playground prompt affect the levels of knowledge construction based upon IAM scores compared to students who respond to prompts that do not use any of the three Structured Divergent prompts after controlling for pre-test IAM scores?Playground prompts focus on one aspect of a topic that is more likely to produce discussion (Andrews, 1980). When controlled for the pre-test, the playground group, with an adjusted mean score of 3.77, did not show a significant improvement over the control group with an adjusted mean score of 2.72 (p = .18) (Table 3). Using the IAM scale of 0-5, the playground prompt averaged 1.2 while the control group averaged 0.6 (Table 1). In the playground group, level one was demonstrated the most in the post-test with twenty-seven posts. Level four was the highest level demonstrated (Table 1).From pre-test to post-test, the number of non-posts increased from five to nine. This was the only one of the Structured Divergent prompts to have the number of non-responses increase. This prompt had only one extra post in pre-test and five extra posts in the post-test, which was the highest number of extra posts of all prompts. This was the only Structured Divergent prompt not to have a statistically significant increase in the mean score. It could be that playground prompts appeal more to some students than others. Because it focuses on one specific area of the overarching subject, students are more likely to participate in the discussion when they are interested in that subject-area. On the other hand, if they are not interested in the topic,their participation may drop off. It can also be speculated that playground prompts, although they provide limits within which to answer, may still have been too broad for students to focus on a concise thought, thus producing insignificant results.

B. Question 2How does the brainstorm prompt affect the levels of knowledge construction based upon IAM scores compared to students who respond to prompts that do not use any of the three Structured Divergent prompts after controlling for pre-test IAM scores?Brainstorm prompts encourage students to work together to generate ideas and discover differentconnections (Andrews, 1980). The number of non-posts dropped from 19 to 11. The adjusted mean for brainstorm prompts (4.47) was statistically significantly (p = .03) higher than the adjusted mean (2.72) for the control group, which received Convergent prompts (Table 3). With the IAM scale of 0-5, the brainstorm prompt averaged 1.2 while the control group averaged 0.6 (Table 1). Out of 36 posts on the post-test, level one was demonstrated the most at 27 times, and level four was the highest level reached. It was also the only prompt for which a post received a zero because it did not reflect a level. Even post treatment, the majority of students achieved only lower levels of knowledge construction. However, because the question required students to come up with solutions or possibilities, they negotiated meaning and synthesized with each other. Therefore, as compared to the control group and the pre-test from the brainstorm prompt group, more students’ prompts reached a level of three or four. The fact that the brainstorm prompt had significantly higher levels of knowledge construction versus the control is consistent with Bradley et al. (2008), who did not study knowledge construction, but found that brainstorm prompts also caused an increase in the levels of the related field of critical thinking.

C. Question 3How does the focal prompt affect the levels of knowledge construction based upon IAM scores compared to students who respond to prompts that do not use any of the three Structured Divergent prompts after controlling for pre-test IAM scores?Focal prompts introduce a complex controversy, which may elicit more than one possible solution (Andrews, 1980). When corrected for the pre-test, responses to focal prompts received a statistically significantly (p = .003) higher adjusted mean score (5.16) over the control (2.72) (Table 3). With the IAM scale of 0-5, the focal prompt averaged 1.8 while the control group averaged 0.6. In the post-test, most of


students’ responses (25) were classified as level one. Level four was the highest level achieved. This group reached level four eleven times, far more than the next closest group with five posts that achievedlevel four. Similar to the Brainstorm prompt, most student posts reflected lower levels of knowledge construction.However, because the question required students to come up with an answer and argue their defense, the participants synthesized together and constructed knowledge with each other. This may have tested their thinking, schema, and prior experience. As a result, eleven posts reflected a level four as compared to only one pre-test post, and only one post from the post-test control group’. The focal prompt group saw very little change in the number of extra posts from pre-test to post-test. Ofthe four groups, the focal prompt group participated the most during the pre-test and the post-test.However, even with an already high participation rate, the focal group saw a small increase in participation. The increase in IAM scores and participation is consistent with the conclusions by Andrews (1980) and Bradley et al. (2008) that introducing controversy allows for opinions and alternatives to be discussed.

D. Convergent PromptsThe following Convergent prompts were used in the study: multiple consistent, funnel, general invitation, analytic convergent, quiz show, shotgun, lower-level divergent, and single questions. These questions were chosen because they best fit the original prompt design given in previous course offerings. Each prompt ranged from being used once to up to four times. Twelve students were included in the Convergent prompt group, which served as a control. While allgroups received Convergent prompts during the first three weeks, when the other three groups received the treatment prompts, the Convergent group continued to receive Convergent prompts. The Convergent group was the only group to have the total number of posts go down from the pre-test (23) to post-test (14). Because students did not post the required responses, level zero was demonstrated the most. When students did post, the posts rarely scored higher than level one. In addition, this group’s posts were more concise than the others. Unlike the other prompts, students in the Convergent group did not post more than required by the course in the post-test. The Convergent group had a statistically significant lower cumulative mean than the focal and brainstorm groups. This group received the lowest average (0.6) on the IAM scale of 0-5 (Table 1). This indicates that Convergent prompts lead students to use lower levels of knowledge construction as compared to theStructured Divergent prompts. This lower participation and lower IAM scores again were similar to previous studies (Andrews, 1980; Bradley et al., 2008; Wruck, 2011) that show that Convergent prompts are less productive, generate fewer student posts, and required lower levels of critical thinking.

E. LimitationsIt should be taken into account that although students were randomly assigned to the groups not all of the students assigned agreed to participate in the study. Only 58 out 76 students enrolled in the course participated, giving a smaller n for statistical analysis, particularly in the control group. The small sample size in this study clearly diminished the statistical power of our analysis. However, the specific effect of the prompt types on the students who chose not to participate is unknown. Had they participated, the results may have been different. In addition, although non-participants’ posts were not analyzed they posted in the discussions with the participants of the study, and the non-participant posts could have influenced participants’ posts. Students were recruited for the study at the beginning of the course. It is possible that the knowledge that they were part of a study may have influenced their responses. There are factors that may have affected the quality of the responses. Three instructors taught the four sections, with one instructor responding to two groups and one instructor each responding to the other two groups. Therefore, the manner in which professors interacted with students in discussions may have affected participants’ responses. Other factors such as the age, gender, and degree program of the participants may have affected the students’ participation. This data was not collected.


VI. CONCLUSIONIn this study, we found that using specific Structured Divergent prompts increased knowledge construction. The two prompts shown to increase knowledge construction as measured by IAM were the brainstorm and focal prompts. This supports portions of the findings by Andrews (1980) and Bradley et al. (2008). Andrews showed Structured Divergent prompts produced posts with higher levels of Blooms’ Taxonomy compared to Convergent prompts. We found in this study higher levels of knowledge construction were present only when using brainstorm and focal prompts, but not the playground prompt.In Bradley et al. (2008) course link, brainstorm, and direct link prompts encouraged higher levels of critical thinking using Bloom’s Taxonomy than open focal and application prompts, which produced the lowest levels. Although we found brainstorm prompts produced higher levels of knowledge construction when analyzed through IAM, our focal prompts were just as successful. Therefore, if instructors wish to stimulate knowledge construction, they should avoid the less productive prompts, which would be playground and Convergent prompts. Instead, instructors and designers should create prompts that naturally encourage students to collaborate in creating solutions and ideas or require students to chose an argument and defend their opinion.

VII. ACKNOWLEDGEMENTSWe would like to thank Dr. Gordon Sutherlin of the Cannon-Clary College of Education, Harding University, for his invaluable help and advice.


VIII. REFERENCESAndrews, J.D. (1980). The verbal structure of teacher questions: Its impact on class discussion.

Professional and Organizational Development Quarterly, 2(3-4), 129-163.

Berthold, K., Eysink, T. H. S., & Renkl, A. (2009). Assisting self-explanation prompts are more effective than open prompts when learning with multiple representations. Instructional Sciences, 37, 345-363.

Booker, M. J. (2007). A roof without walls: Benjamin Bloom's Taxonomy and the misdirection of American education. Academic Questions, 20(4), 347-355.

Bradley, M. E., Thom, L. R., Hayes, J., & Hay, C. (2008). Ask and you will receive: How question type influences quantity and quality of online discussions. British Journal of Educational Technology,39(5), 888-900.

Buraphadeja, V., & Dawson, K. (2008). Content analysis in computer-mediated communication: Analyzing models for assessing critical thinking through the lens of social constructivism. American Journal of Distance Education, 22(3), 130-145.

Crane, L. (2012). Trust me, I’m an expert: Identity construction and knowledge sharing. Journal of Knoweldge Management, 16(3), 448-460.

DeWever, B., Van Keer, H., Schellens, T., & Valcke, M. (2009). Structuring asynchronous discussion groups: The impact of role assignment and self-assessment on students' levels of knowledge construction through social negotiation. Journal of Computer Assisted Learning, 25(2), 177-188.

Dunlap, J. C., Sobel, D., & Sands, D. I. (2007). Supporting students' cognitive processing in online courses: Designing for deep and meaningful student-to-content interactions. TechTrends: Linking Research and Practice to Improve Learning, 51(4), 20-31.

Gagne, R. M. (1970). The conditions of learning. New York: Rinehart and Winston.

Gilbert, P. K., & Dabbagh, N. (2005). How to structure online discussions for meaningful discourse: A case study. British Journal of Educational Technology, 36(1), 5-18.

Gunawardena, C. N., Lowe, C. A., & Anderson, T. (1997). Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), 397-431.

Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative Learning Through Computer Conferencing (pp. 117-136). Berlin: Springer-Verlag.

Hew K. F., & Chueng, W. S. (2011). Higher-level knowledge construction in asynchronous online discussions: An analysis of group size, duration of online discussion, and student facilitation techniques. Instructional Science, 39(3), 303-319.

Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235-266.

Hopkins, J., Gibson, W., Ros i. Sole, C., Savvides, N., & Starkey, H. (2008). Interaction and critical inquiry in asynchronous computer-mediated conferencing: A research agenda. Open Learning,23(1), 29-42.

Hughes, M., & Daykin, N. (2002). Towards constructivism: Investigating students' perceptions and learning as a result of using an online environment. Innovations in Education & Teaching International, 39(3), 217-224.


Leech, N. L., Barrett, K. C., & Morgan, G. A. (2011). IBM SPSS for intermediate statistics: Use and interpretation (4th ed.). New York, New York: Routledge.

McDade, S. A. (1995). Case study pedagogy to advance critical thinking. Teaching of Psychology, 22(1),9-10.

Moore, J. L., & Marra, R. M. (2005). A comparative analysis of online discussion participation protocols. Journal of Research on Technology in Education, 38(2), 191-212.

Saritas, M. T. (2006). Computer-mediated communication in higher education: An exploration of knowledge construction. (Doctoral dissertation). Retrieved from Dissertation and Theses @ Capella University. (Publication No. AAT 3243576).

Schellens, T. T., Van Keer, H. H., Valcke, M. M., & DeWever, B. B. (2007). Learning in asynchronous discussion groups: A multilevel approach to study the influence of student, group and task characteristics. Behaviour & Information Technology, 26(1), 55-71.

Wang, Q., Woo, H., & Zhao, J. (2009). Investigating critical thinking and knowledge construction in an interactive learning environment. Interactive Learning Environments, 17(1), 95-104.

Wruck, L. (2010). Computer-mediated communication: Instructional design strategies that support the attainment of Bloom's higher order cognitive skills in asynchronous discussion questions. (Doctoral dissertation). Retrieved from Dissertations and Theses @ Capella University. (Publication No. AAT 3409304)


Differences in Classroom Versus Online Exam Performance Due to Asynchronous Discussion

Robert L. JorczakDanielle N. Dupuis University of Minnesota

ABSTRACTThis study compares student exam performance of undergraduate students in classroom and online course sections with nearly identical instructional designs. The sections differed only in lecturing and peer discussion activities, which are typical differences of classroom and online instruction. Classroom discussion activities included synchronous speech, while online discussions used asynchronous text. Composite mean exam scores show a large effect size difference that is statistically significant. Results suggest asynchronous peer-to-peer discussion is more effective than traditional classroom lecture-discussion for undergraduate students.

I. INTRODUCTIONOnline instruction, also called web-based instruction, is increasingly used in higher education and K-12 institutions (Allen & Seaman, 2011). The continuing rapid increase in online instruction raises several important questions. Is this trend beneficial for formal learning? How well do students learn in online instructional environments compared to classrooms? What differences between the environments affect instruction and student learning performance, and why do these differences matter? The goal of this study was to determine if students performed differently on exams given in nearly identical courses delivered online and in a classroom. Further, if differences existed, this analysis sought to examine potential causes of the differences in student performance. While the courses were delivered via different instructional environments, much of the instructional design of the online and classroom classes was identical.To understand potential differences between instructional environments that might affect student learning, some concepts must be clearly defined. First, a communication medium is a technological means to store and/or transmit information. Instructional methods are procedures and techniques intended to promote learning of specified outcomes. Instructional methods comprise three functions: presentation of information, specification of learning activities, and assessment. Learning activities go beyond merely presenting information by having students express their knowledge in ways intended to support achievement of learning objectives. Instructional delivery environments, such as classroom and online environments include various media, tools, and functions that support instructional methods.Using a specific instructional environment does not require the use of specific instructional methods. Online and classroom environments are sufficiently flexible to support a wide range of similar methods. While the two instructional environments can be equivalent in learning effectiveness—since they use similar instructional methods—the environments often differ in preferred methods. Each instructional delivery environment has affordances that guide selection of media and methods. Some methods are easier to implement and/or execute within each environment, and these methods are the ones that tend to be used. Specific designs and design elements are encouraged by aspects of the environment because of


the efficiency of implementation and use of those designs. Different instructional methods, therefore, tend to be associated with classroom and online environments. This report focuses on what aspects of preferred instructional methods affect learning, as measured by classroom exam performance.

II. THEORETICAL FRAMEWORKInformation internalization and externalization by students are important processes for learning (Cress & Kimmerie, 2008). Internalization occurs when students encounter new information and process thatinformation into knowledge. Students also learn by expressing their knowledge as information. The process of transforming knowledge into external information results in reinforcement of existing knowledge and the creation of new knowledge by increasing associations among internal representations.Information processing models of learning identify elaborative rehearsal as a key control process by which newly-experienced information is merged with existing knowledge in working memory resulting inlong-term memory encoding (Raaijmakers & Shiffrin, 2004). Elaborative rehearsal also plays a role during externalization as knowledge is elaborated during the process of transforming knowledge into externally represented information. Instructional methods therefore can be productively characterized in terms of presentation of information and specification of learning activities. During information presentation, information is selected and organized to promote internalization of targeted knowledge. Learning activities have students express information in ways thought to promote targeted learning. The concept of “active learning” is theassertion that instructors can most influence learning by designing appropriate learning activities which require deeper processing. Thus, differences in learning activities can be expected to be a key differentiator of learning methods. This perspective suggests that the differences of typical learning activities used in each instructional environment are the major factors distinguishing instructional effectiveness of each environment.Clark (1994) has asserted that information delivery media, as mere conveyors of information, cannot affect formal learning. Clark suggested that research studies finding an advantage for a particular delivery medium did so because the studies did not compare equivalent instructional methods. Meta-analyses have tended to support this position (Bernard et al., 2004; Means, Toyama, Murphy, Bakia, & Jones, 2010; Sitzmann, Kraiger, Stewart, & Wisher, 2006). Means et al. found a statistically significant 0.24 effect size advantage for online and hybrid courses compared to classroom courses. But, the authors issued many cautions about their findings, including the fact that their meta-analysis included many studies that did not use the same instructional methods. These analyses suggest that differences in the effectiveness of instructional environments depend on differences in the instructional methods used in each environment, not on media differences. The internalization-externalization model can be extended to group learning by viewing social interaction (e.g., discussion) as a means for students to receive divergent information from other students, and also as a means for students to express knowledge relevant to the learning task (Jorczak, 2011). Both learning processes are supported via group discussion that promotes both divergent thinking and elaborative rehearsal. The information internalization-externalization model provides a basis for the social constructivist concept of mutual knowledge construction by social interaction within learning groups.

Table 1Exam Score Descriptive Statistics by Condition

Instructional Environment n M SD SE(M) Classroom 35 .67 .09 .015Online 69 .74 .11 .013

Note: Means (M) represent proportion of items answered correct across both exams. SD = standard deviation; SE = standard error.


A. Differences in Instructional Methods of Online and Classroom Environments

The major difference between classroom and online environments is physical presence and synchronicity—classroom environments specify that the instructor and students all be present at the same location and time, while online environments separate instructor and students in location and time. Classroom environments include an information presentation method that is not available to online courses: face-to-face real time speech. Classroom environments are designed to support and promote this presentation method. Online environments cannot, by definition, use this presentation method. To compare instructional delivery environments and better understand how aspects of those environments affect learning, researchers must look beyond media and consider differences in instructional methods determined or influenced by the nature of the instructional delivery environment. A major difference in instructional method typically found in classroom and online environments involves the synchronicity of the instructional method (the degree to which students experience instruction simultaneously). While both environments can support synchronous and asynchronous learning methods, each displays a preference for one type. Classroom courses tend to use synchronous presentations and activities. Online courses tend to use asynchronous methods (in which students work whenever they choose and instructors interact with students when they choose). Online courses can present information to students synchronously, for example by web-casting a lecture or video to all students at the same time, but this approach is rarely used as it is difficult to implement and does not reflect the flexibility offered by the environment. Classroom courses also can have students do activities (including small group discussion) asynchronously.

B. Interaction and Online LearningClassroom and online courses also tend to differ in the type of interaction they promote. A meta-analysis by Bernard et al. (2009) compared interactions in online environments by categorizing interaction into three treatment types: student-content (SC), student-teacher (ST), and student-student (SS). The meta-analysis compared studies that used these types of interaction as well as differences in the “strength” of the interactions. Bernard et al. (2009) found that online ST interactions did not affect learning to the same degree as SC or SS interactions. This result supports the idea that online discussion (which is primarily SS) has a greater effect on learning than teacher-led discussions (which are primarily ST). Classrooms are structured to promote ST interactions, including lecture-discussions. The meta-analysis results suggest a reason for possible differences between student performance in online and classroom environments. The interaction treatment type is an interesting variable, but the categories are broad and ignore important aspects of interaction (e.g., whether it is one-way or two way and the synchronicity of the discussion), so further refinement of the categorization of interaction is required.

C. Asynchronous Discussion in Support of LearningPeer-to-peer guided discussion is an important collaborative learning technique, which has been repeatedly shown to be instructionally effective (Andriessen, Baker & Suthers, 2003; Garrison, Anderson, & Archer, 2000). How the method is implemented, which media are used and what instructional methods are used, tend to differ between classroom and online environments. Classroom instructors usually employ synchronous oral lecture-discussion. Online class discussion tends to be done asynchronously via text in “forums” in which students post messages and responses. Researchers have suggested learning advantages for asynchronous discussion (Garrison et al., 2000; Lapadat, 2002). Asynchronous discussion can be read at whatever pace aids reader understanding, and information can be rescanned and reviewed. Students can take as much time as they wish to respond, so discussion occurs at a much slower pace allowing for more reflection and processing than stream-of-consciousness speech (Garrison et al., 2000; Lapadat, 2002). No single student has the “floor” to speak, and no time limits are set by either outside constraints or limited attention spans. Knowledge and opinion that is externalized via text requires more effort and cognitive processing than speech. This extra processing improves externalized information and aids organization and specification of knowledge


(Newman, Webb, & Cochrane, 1997). Information in asynchronous discussion tends to diverge more (Newman et al., 1997), which aids learning (Jorczak & Bart, 2009). All students have access to the discussion and all students can contribute as much as they like.

D. Research QuestionThis study adds to the online versus classroom research literature by comparing student performance in online and classroom sections in which most instructional variables were kept constant, with one exception. In this in situ study, the key difference in instructional method is a difference between the discussion learning activities, though some differences in information presentation are also present. The overall goal of this analysis is to seek evidence about whether differences in instructional methods typical of classroom and online courses (i.e., synchronous lecture-discussion versus asynchronous small group discussion) are associated with differences in student exam performance. It is hypothesized that online students using asynchronous discussion will have better exam scores due to the instructional advantages of text-based asynchronous discussion.

III. METHODA. Participants and ProceduresParticipants were 104 college students (mostly first year students) enrolled in an introductory psychology course at a medium-size Midwestern public university. The course subject was introductory level psychology, and almost all students were required by their program to take the course. Students were enrolled in one of three course sections: a classroom section meeting for one hour three times a week (n =35), and two online sections delivered via a course management system (n = 69). The instructional design for both the classroom and online sections was nearly identical. The classroom and online courses covered identical material following the same sequence as well as the same schedule.The same instructor taught all three sections. Every section used the same materials including the sametextbook, recorded videos, as well as supplemental materials (e.g., articles, handouts). Learning activities, such as written assignments, were the same for all three sections with one exception—student discussion. The instructional method of the online courses differed from the classroom course in information presentation and learning activities. In the classroom section, information was presented to students via lectures by the instructor, prior to in-class discussion. Classroom students also received copies of lecture slides on which they could take notes. Online students, on the other hand, received brief text-based“lectures” covering the same course content. Learning activities also differed between the two formats. Classroom students participated in both instructor-led and small-group discussion that was synchronous, face-to-face, and via speech; while students in the online sections used asynchronous peer discussion. Online students were graded for discussion participation, while classroom students were not. In short, the classroom course in this study used synchronous lecture-discussion and some synchronous small group discussion, but the online courses used asynchronous small group text discussion as a key instructional activity.

B. Research Design and Data AnalysisThis study is quasi-experimental in design. Students were not randomly assigned to sections. The independent variable is the class delivery environment (online or classroom), and the primary dependent variable is the sum of course exam scores. Students in all sections were given two 50-item exams intended to assess knowledge acquisition and concept formation. These exams were timed and used multiple-choice items that were scored for accuracy (i.e., correct/incorrect). Exams were delivered to all sections via the university’s online learning management system. Overall course letter grades were assigned based on the total sum of learning activity points, weekly quiz results, exam scores, research participation points, and discussion participation (online only). Pearson product moment correlations were used to examine the relations between variables; independent-samples t-tests were used to examine


the mean difference in exam scores between course sections. Effect sizes were calculated following Cohen’s d.

IV. RESULTSAcross all exams, the mean proportion correct for classroom students was 0.67 and for the online students 0.74, which results in a mean difference of 0.08 (see Table 1). The online students did statistically significantly better on the exams, t(102) = 3.56, p = .001. The effect size was d = 0.66, which suggests that an average student who participates in online discussion will score 25 percentile points higher on course exams than an average student who participates in classroom discussion. This effect is moderate to large, suggesting that a difference in the classroom and online environments has an important effect on exam performance. In the online sections, the correlation between exam scores and online discussion participation points was 0.36 (p = .002), while the correlation between exam scores and all other (non-discussion) learning activities was 0.37 (p = .002). This result suggests a moderate association between online discussion participation and exam performance, as well as a moderate association between other learning activities and exam performance. For the total sample, the correlation between exam scores and total course points earned in learning activities (excluding online discussion participation points for the online sections) was 0.30 (p = .01) suggesting that grades do not strictly follow assessed learning. Students with better test scores tended to get better grades, but the association is surprisingly weak. This result suggests that exam results were deemphasized as a determinant of grade in this course design.Students were not assigned to classes randomly, which damages any causal inferences drawn between the independent (discussion method) and dependent (exam scores) variables. The researchers were not able to access student information outside the course that would allow comparisons of the classroom and online students on variables of interest. An attempt was made to measure group equivalence by comparing group performance on the first course quiz. Quizzes were short multiple-choice tests based on chapter readings. Scores on the first quiz, before the course instructional design had time to take effect, were taken as an indicator of group knowledge and test taking skill prior to any effect of the course. No statistically significant difference was observed between the groups on the first quiz score (t(105) = 0.56, p = .572).

V. DISCUSSION A Cohen’s d effect size of 0.66 is a moderate to large effect for an instructional intervention (Cohen, 1977). Such a large difference in classroom versus online student performance is not something that is easily ignored or explained away. The observed difference in student exam performance is likely due todifferences in the instructional methods. Because the instructional methods used in both sections are so similar, it is reasonable to conclude that the student performance gap stems from differences that did distinguish the classroom and online sections. While the methods differed in both manner of information presentation (lecture versus written information) and learning activities (synchronous oral versus asynchronous text discussion), prior research and theory point to differences in the learning activities asthe likely primary cause of performance differences. Studies comparing oral versus text-based presentation of information have long suggested no difference in effect on student performance (Corey, 1934). In addition, both online and classroom students used the same textbook as an information source. We therefore argue that differences in information presentation between online and classroom sections is negligible for explaining differences in exam performance.Previous research conducted by Bernard et al., (2009) suggests that student-teacher interactions withinclassroom lecture-discussions may be partially responsible for the observed lower exam performance of classroom students as compared to the student-student interactions of online students in online discussion. The main difference in learning activities between the sections was synchronous oral teacher-led discussion (classroom) versus asynchronous textural peer discussion (online). We suggest that this


difference was primarily responsible for the observed difference in exam performance. The cognitive advantages of slowed processing, review ability, and equal access of asynchronous online test discussionappear to provide tangible learning benefits. Students in the online sections may retain information better because they choose when to attend to course information and assignments, as well as when to stop or rest. This learner control can contribute to better learning because students who voluntarily give attention, have the opportunity to direct their attention elsewhere, and return to the point at which there attention wandered, are less likely to miss important information.This study does not address aspects of effective discussion design but compares the basic functional differences of discussion typically found in online and classroom courses (i.e., synchronous speech versus asynchronous text). Other characteristics of asynchronous discussion design may also affect performance. We stress that the design of discussion in these classes met minimum quality standards that we take to be typical of current online classes and are typical of online courses that use peer discussion. A potentially important difference in discussion activities is that the online students were graded (i.e., earned points) for their participation in discussion (a common online practice). Classroom students were not assigned participation points and therefore may have had less incentive to participate. This distinctionmay be responsible for differences in learning due to discussion rather than whether the discussion is synchronous of not. This variable should be controlled in future studies of differences in discussion activities. If the difference in discussion activity is responsible for the difference in student exam performance, then online courses need to use and refine online asynchronous discussion in support of learning. In addition,classroom courses should move to hybrid designs that incorporate asynchronous discussion activities with the aid of an online forum.

A. Alternative ExplanationsFactors unrelated to differences in instructional methods between the sections could contribute to the observed effect. As mentioned, students were not randomly assigned to classroom and online sections.Therefore, one explanation that cannot be ruled out is that students who choose online courses tend to be more knowledgeable of course subject matter, are better students, have higher learning ability, are more motivated, and/or are better test-takers than their classroom counterparts. Many instructors offer an intuitive opinion that online students tend to be inferior to classroom students. Few studies haveaddressed this issue, and the ones that have show conflicting results regarding test performance and other variables between classroom and online students (Dutton, Dutton, & Perry, 2002; Kirtman, 2009) Such studies show that online and classroom students are more similar than different, but that online students report higher levels of interest, curiosity, and intrinsic motivation (Stevens & Switzer, 2006). Students in the classroom and online sections in this study were found to perform equally on the first quiz, suggesting that students did not differ in their knowledge or ability.Another possible explanation for the difference in test performance is instructor bias. The suggestion is that the instructor either prefers the online environment or has skills better suited to the online format.This explanation is unconvincing because most aspects of course design are not affected by the instructorduring instruction, minimizing the effect of instructor bias in the two courses. The instructor does influence lecture-discussion, and to a lesser degree online discussion, but has minimum effect on all other course activities. Discrepancies in testing may account for some differences in exam performance. Exams were administered in a very similar way to students in both online and classroom sections. The exams were not “open book”, but students in both environments were allowed to create a one-page sheet of notes that they could refer to during the test. Exams were delivered via a learning management system (i.e., by computer) to all students, and were timed to limit opportunities for cheating. For the classroom section, the instructor was present during the administration of both exams which may have discouraged cheating compared to online sections, which were not monitored.


If discrepancies in testing are responsible for the observed difference in scores, then testing techniques for online exams must be re-evaluated and adjusted. Time limitations on tests should perhaps be decreased to reduce time available to seek answers from sources. The idea of an “open book” test that assesses higher order learning in which simple facts and definitions cannot be simply looked up may be more appropriate for online environments. The use of applications to “lock” browsers from accessing web pages other than the exam may prove useful for online testing (though students can simply use another device to access information). Clearly, more research comparing classroom and online testing is needed along with research about alternative online testing.

B. LimitationsThis study most strictly generalizes to higher education, but there is no reason to expect similar results would not be found for other students if the result is due to differences in discussion activities. The major methodological limitation of this study is the lack of random assignment to groups as discussed above. The instructor for these courses is also the first author of this report, but the instructor did not know he would conduct this analysis prior to the completion of the course. As a result, no study conditions or any research design strategies were considered or established prior to the completion of the course. The idea to compare test scores came well after the course conclusion of the course. The fact that all classes had the same instructor reduces one potential confounding variable. Nevertheless, this study could be biased ifthe instructor had a superior ability and/or preference for online instruction. However, it is hard to see how instructor preference could result in such large differences in test scores given that most aspects of the instruction were identical.This study suggests more experimental work—as well as more theoretical work—is needed regarding asynchronous discussion. In future studies, the effect of discussion synchronicity can be more directly compared by implementing different discussion designs within the same delivery environment. If differences in online instructional methods are identified as being responsible for improved student performance, those methods may be able to be used in classroom or hybrid courses so that theseinstructional environments can take advantage of their added benefits.


VI. REFERENCESAllen, I. E., & Seaman, J. (2011). Going the distance: Online education in the United States, 2011.

Wellesley, MA: Babson Survey Research Group.

Andriessen, J., Baker, M. J., & Suthers, D. (2003). Argumentation, computer support, and the educational context of confronting cognitions. In J. Andriessen, J., Baker, M. J., & Suthers, D. (Eds.), Arguing to learn: Confronting cognitions in computer-supported collaborative learning environments (pp. 1-25). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Bernard, R. M., Abrami, P. C, Borokhovski, E., Wade, C. A., Tamin R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79, 1243-1289.

Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., … Huang, B. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74, 379–439.

Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42, 21-29.

Cohen, J. (1977). Statistical power analysis for the behavioral sciences. New York, NY: Academic Press.

Corey, S. M. (1934). Learning from lectures vs. learning from readings. Journal of Educational Psychology, 25, 459-470.

Cress, U. & Kimmerle, J. (2008). A systemtic and cognitive view on collaborative knowledge building with wikis. International Journal of Computer Supported Collaborative Learning, 3, 105-122.

Dutton, J., Dutton, M., & Perry, J. (2002). How do online students differ from lecture students? Journal of Asynchronous Learning Networks, 6, 1-20.

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2, 1-19. Hammond, M. (2005). A review of recent papers on online discussion in teaching and learning in higher education. Journal of Asynchronous Learning Networks, 9, 9 –23.

Jorczak, R. L. (2011). An information-processing perspective on divergence and convergence in collaborative learning. International Journal of Computer Supported Collaborative Learning, 6,207-221.

Jorczak, R. L., & Bart, W. (2009). The effect of task characteristics on conceptual conflict and information processing in online discussion. Computers in Human Behavior, 25, 1165-1171.

Kirtman, L. (2009). Online versus in-class courses: An examination of differences in learning outcomes. Issues in Teacher Education, 18, 103-116.

Lapadat, J. C. (2002). Written interaction: A key component in online learning. Journal of Computer Mediated Communication, 7.

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning. Washington, D.C.: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development.

Newman, D. R., Webb, B., & Cochrane, C. (1997). Evaluating the quality of learning in computer-supported cooperative learning. Journal of the American Society for Information Science, 48,484-495.


Raaijmakers, J. G. W., & Shiffrin, R. M. (2004). Models of memory. In H. Pashler & D. Medin (Eds.) Stevens' Handbook of Experimental Psychology, Third Edition, Volume 2: Memory and Cognitive Processes. New York, NY: Wiley & Sons, Inc.

Sitzmann, T. K., Kraiger, K, Stewart, D., & Wisher, R. (2006). The comparative effectiveness of Web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59, 623–664.

Stevens, T., & Switzer, C. (2006). Online and traditional student differences. Turkish Online Journal of Distance Education, 7, 90–100.


SECTION III: Mobile Learning

The SAMR Model as a Framework for Evaluating mLearning

Danae Romrell, Lisa C. Kidder, Emma Wood


The SAMR Model as a Framework for Evaluating mLearning

Danae RomrellLisa C. Kidder Emma Wood Idaho State University

ABSTRACT

As mobile devices become more prominent in the lives of students, the use of mobile devices has the potential to transform learning. Mobile learning, or mLearning, is defined as learning that is personalized, situated, and connected through the use of a mobile device. As mLearning activities are developed, there is a need for a framework within which mLearning activities can be evaluated. The SAMR Model provides such a framework (Puentedura, 2013). This paper reviews recent literature on mLearning and provides examples of activities that fall within each of the four classifications of the SAMR Model: substitution, augmentation, modification, and redefinition.

I. INTRODUCTION

The SAMR Model as a Framework for Evaluating mLearning With the predominance of mobile devices in our lives, it is natural for educators to ask how they could be used to support learning. In exploring the possibilities and reviewing the research, it becomes clear that there are many factors that influence the implementation of mobile devices within the educational context. Discussions of mobile learning, or mLearning, often focus on selecting an appropriate mobile device for the learning activity in question. However, it is more important for educators and instructional designers to focus on how mobile devices can be used to improve learning. Often, mobile devices are simply used to perform the same tasks that were previously completed without the use of a mobile device. This level of implementation represents the lowest level on the SAMR model, which includes four levels of technology integration (substitution, augmentation, modification, and redefinition) and provides a framework to support educators and instructional designers in creating optimal learning experiences using mobile devices in education. This paper presents a definition of mLearning and recommends the SAMR model as a framework for evaluating mLearning, facilitating the design of mLearning activities, and supporting atransformation of learning.


II. METHODS

What is mLearning? Before research related to mLearning can be discussed, mLearning must be defined. Currently, researchers do not agree on a single definition of mLearning. While there is a mobile device at the heart of mLearning, it is what that device enables teachers and learners to do that truly defines mLearning. A review of the literature on mobile devices in higher education coursework illustrated several unique characteristics of learning with a mobile device that helped us formulate the proposed definition of mLearning. In particular, mobile devices are personal and personalized. They are situated across contexts and time. And they are connected to information, people, and practices. These three characteristics of mobile devices make mLearning unique and different from other types of eLearning. Thus, the proposed definition of mLearning used in this paper is learning that is personalized, situated, and connected through the use of a mobile device (see Figure 1).

Figure 1. mLearning is learning that is personalized, situated, and connected through the use of a mobile device.


Mobile devices. At the center of mLearning is a mobile device. There are many types of mobile devices, including phones, smartphones, tablets, and even small laptop computers. These devices change rapidly, with users purchasing new ones for a variety of reasons, most of them not related to education (Traxler, 2010). In the context of higher education, studies have looked at various tools and applications available on mobile devices. For example, studies have examined the use of podcasts (Cochrane, 2012; Evans, 2008); the use of short messaging systems (SMS), or texting (Brett, 2008; Chuang & Tsao, 2013; Cornelius & Marston, 2009; Grönlund & Islam, 2010); the use of specially designed mobile applications (Dyson, Litchfield, Lawrence, Raban, & Leijdekkers, 2009; Huang, Jang, Machtmes, & Deggs, 2012; Lan, Tsai, Yang, & Hung, 2012; Pfeiffer, Gemballa, Jarodzka, Scheiter, & Gerjets, 2009; Redondo, Fonseca, Sánchez, & Navarro, 2013; Wu, Hwang, Su, & Huang, 2012); the use of the Global Positioning System (GPS) (Liu & Tsai, 2013); the use of social media applications (Wang, Yu, &Wu, 2013); and collecting data using images, video, or audio (Cochrane, 2010, 2012; Dyson et al., 2009; Gromik, 2012).

Traxler (2010) stated that “mobile devices, especially connected devices, enable students to consume—that is, to access and store—all sorts of knowledge almost instantly and almost wherever they are, with little or no effort compared with earlier technologies” (p. 154). In the current information age, the ability to access information is an important skill. However, learning is more than the consumption of information, and the research shows that the potential of mobile devices surpasses enabling the simple information-consumption mode. Moreover, there are three key characteristics that identify mLearning as a distinct form of eLearning with unique problems for educators and instructional designers.

Mobile devices are personal. Several researchers have identified personalization as one of the key characteristics of mLearning—for example, Kukulska-Hulme (2009) and Kearney, Schuck, Burden, andAubusson (2012). A mobile device can be personalized through the addition of unique cases, backgrounds, sounds, and software. As Traxler (2010) observed, “These devices are personal, universal, and closely linked to identity” (p. 152). Looking across a classroom, an instructor can see the personalities of students reflected not only in their choice of mobile device (smartphone, tablet, cellphone) but also in the personalization of the colors, fonts, apps, and accessories associated with their devices.

The familiarity that the learner has with mobile devices impacts how they are used. There is a difference between devices that are borrowed and those that are owned (Kukulska-Hulme, 2009). Owned devices not only reflect the personality and preferences of an individual but also influence their actions, as afforded or hindered by the mobile device. Borrowed devices are less familiar to the owner, which often makes the device harder for the learner to use and makes the learning feel less personal.

Not only can a mobile device itself be personalized, but the learning that occurs on a mobile device can also be personalized. Mobile Web 2.0 tools allow for the personalization of content and interfaces used on the mobile device (Cochrane, 2010). While the personal nature of the device might suggest the usefulness of personalized learning, in some cases the personal nature of the device can be an obstacle in implementing mLearning. Cochrane (2010) found that at times disconnect occurs when students use their personal mobile phones in education. In spite of this potential obstacle, mobile devices allow for both the device and the content to be personalized to the learner. This is the first characteristic that identifies mLearning—it is personalized.


Mobile devices are situated. Due to their size, mobile devices are portable, making it easy to take them out of the classroom (Cheon, Lee, Crooks, & Song, 2012). This portability points to the fact that not only are the devices mobile, so are the individuals using them, which allows for learning to be situated within a real-world setting and provides context sensitivity (Cheon et al., 2012). As students move through their daily routines with a device within an arm’s reach, they have access to just-in-time information that issituated in the context of their lives. This timeliness of information availability creates opportunities for “new ways of dividing up one's time and crossing boundaries” (Kukulska-Hulme, 2009, p. 153).

There is a constant tension between the formal environment of education and the informal context outside traditional education. Mobile devices enable learning to come to an individual regardless of time or location (Cornelius, Marston, & Gemmell, 2011; Pfeiffer et al., 2009). mLearning provides an opportunity to create a bridge between formal and informal learning in order to create authentic situated contexts (Cochrane, 2012; Pfeiffer et al., 2009; Traxler, 2010). This is the second characteristic that identifies mLearning—it is situated.

Mobile devices are connected. Cheon et al. (2012) noted that one of the advantages of mobile devices is their instant connectivity. Mobile devices allow for instant connectivity by providing users with the ability to access the Internet, view a video, place a phone call, or send a text message. This access to information, people, and practice, creates a community of learners, even if only for a short period of time or within a specific context. This is the third characteristic that identifies mLearning—it is connected.

Learning that is personalized, situated, and connected through the use of a mobile device has the potential to transform learning in ways not previously envisioned. However, this definition of mLearning, while helpful, is not sufficient when designing instruction. As in research, the use of a framework provides boundaries and anchors to learning theory. With regard to mLearning, the SAMR model is a framework that can be used to evaluate how significantly technology has transformed learning.

The SAMR model as a framework for mLearning. Transformational learning activities that are truly personalized, situated, and connected through the use of a mobile device will go beyond merely using a mobile device as a substitute for more traditional tools. The SAMR model provides a framework that can be used to classify and evaluate mLearning activities. Ruben R. Puentedura developed the SAMR model in 2006 as part of his work with the Maine Learning Technologies Initiative (Puentedura, 2006). The model was intended to encourage educators to significantly enhance the quality of education provided via technology in the state of Maine. The SAMR Model consists of the following four classifications of technology use for learning activities:

Substitution: The technology provides a substitute for other learning activities without functionalchange.

Augmentation: The technology provides a substitute for other learning activities but withfunctional improvements.

Modification: The technology allows the learning activity to be redesigned.

Redefinition: The technology allows for the creation of tasks that could not have been donewithout the use of the technology.


Learning activities that fall within the substitution and augmentation classifications are said to enhancelearning, while learning activities that fall within the modification and redefinition classifications are said to transform learning (Puentedura, 2013).

Although Puentedura developed the SAMR Model as a way to encourage the use of technology generally, Hockly (2013) suggested using the SAMR Model specifically for mLearning within the context of English language teaching (ELT). This model provides a framework for instructional designers to evaluate mLearning activities in order to determine how well they meet the goal of transforming learning through the use of a mobile device. Building upon the suggestions of Hockly (2013), we reviewed and evaluated research studies involving the implementation of mobile devices in higher education across all disciplines using the SAMR model.

In recent years, a significant amount of research has been done that examines the use of mobile devices in higher education. However, not all of the studies provided in this research fit the definition of learning that is personalized, situated, and connected through the use of a mobile device. From within this body of research, examples of research that address each of the four classifications of the SAMR Model are provided (see Table 1, next page) and reviewed in light of the proposed definition of mLearning.

III. RESULTS

Applying the SAMR model to recent research.

Substitution. Substitution is the simplest way to implement mLearning (Hockly, 2013). mLearning examples that fit into this classification are those where the learning activity could have been done without the use of a mobile device. The following discussion describes three such studies in which mobile devices were used to replace activities that are more traditionally done without the use of a mobile device.

Evans (2008) conducted a study in which podcast lectures were used to replace other forms of review at the end of the course and prior to a comprehensive final examination. In this case, podcasts were used as a substitute for other review methods students might have used, such as reviewing from textbooks or course notes. One significant weakness of this study is that it looked only at student perceptions. It would be beneficial to have a follow-up study that compared learning gains between students who reviewed using podcasts and students who reviewed without the podcasts. In spite of this weakness, the author determined that the students found the podcasts to be a very efficient and effective review tool. One of the main reasons students cited for preferring the podcasts to more traditional forms of review was their portability. One fourth of the participants listened to the podcasts while travelling (p. 495). The author also determined that the students were more engaged with the podcasts than they were with a textbook or in a review lecture (p. 496).

Gromik (2012) conducted a study in which the video camera capabilities of cell phones were used to create videos for an English language class. University students were required to create thirteen 30-second videos on topics assigned by the teacher. The videos were recorded with their cell phones. The videos were then uploaded, and access was provided to all students in the class so that they could view one another’s videos.


The purpose of the assignment was to help students practice composing and speaking the English language. The study found that the creation of the videos helped increase the confidence and speed with which students could converse in English. Students who were interviewed noted that they appreciated the ability to use their cell phones anytime and anywhere for this assignment; however, this study is an example in which cell phones were used as a substitute for other more traditional methods. The same assignment could have been done using video cameras or in-class speeches. Other than an increase in convenience, the assignment offers no functional improvement over assignments that could be given without the use of a mobile device.

Another study in which the use of mobile devices was used as a substitution for another learning activity was conducted by Lan et al. (2012). In this study the authors compared two groups, one that participated in asynchronous online discussions using mobile devices and one that participated in asynchronous online discussions without using mobile devices. They determined that the group that used mobile devices produced more valuable (richer, more relevant, more useful, and more readable) course materials. The mobile users also participated more frequently and were more likely to be active, rather than passive, participants in the discussion boards.

Table 1

SAMR Classification of 10 Recent mLearning Research Studies

S

Substitution

A

Augmentation

M

Modification

R

Redefinition

Evans (2008)

“The Effectiveness of mLearning in the Form of Podcast Revision Lectures in Higher Education”

Gromik (2012)

“Cell Phone Video Recording Feature as a Language Learning Tool: A Case Study

Lan, Tsai, Yang, and Hung (2012)

Chuang and Tsao (2013)

“Enhancing Nursing Students’ Medication Knowledge: The effect of Learning Materials Delivered by Short Message Service”

Pfeiffer et al. (2009)

“Situated Learning in the Mobile Age: Mobile Devices on a Field Trip to the Sea”

Cornelius et al. (2011)

“SMS Text Messaging for Real-Time Simulations in Higher Education”

Wang, Yu, and Wu (2013)

“Empowering Mobile Assisted Social e-Learning: Students’ Expectations and Perceptions”

Liu and Tsai (2013)

“Using Augmented-Reality-Based Mobile Learning Material in EFL English Composition: An Exploratory CaseStudy”

Redondo, Fonseca, Sánchez, and Navarro (2013)

“New Strategies Using Handheld Augmented Reality and Mobile Learning-Teaching Methodologies, in


All three of these studies found that substituting mobile learning for other methods of learning was beneficial. In particular, all three studies noted that the mobility of the devices was beneficial because it allowed the students to engage in the activities at times and in places that they could not with more traditional learning. The fact that students were able to connect to course information from outside of the classroom demonstrates the connected nature of mLearning. The Gromik (2012) and Lan et al. (2012) studies also illustrate the personalized nature of mLearning, as the students created video and produced text within the context of the assignments. Overall, the students generally enjoyed using the mobile devices and thought that they provided a positive alternative to other methods of learning.

Augmentation. Examples of mLearning activities at the augmentation level of the SAMR Model go beyond the substitution level in that they provide some type of functional improvement over what could have been achieved with traditional tools. The following two studies describe situations in which mobile devices were used to augment traditional learning tools.

A study conducted by Chuang and Tsao (2013) looked at the use of SMS text messages to help nursing students memorize information about medications. The study divided the participants into two groups. One group received twice-daily text messages about specific medications in addition to the regular classroom lecture. This use of mobile technology could be classified as augmentation because it added a functional improvement to the previous model of only providing lectures or having the students create their own flashcards. The short messages prompted the students to take a moment to connect to the information to assist them in memorizing vital information about medications. The researchers determined that students who received daily text messages showed significantly higher learning gains at one week, two weeks, and four weeks after the conclusion of the unit.

Pfeiffer et al. (2009) used portable DVD players to augment a situated learning context for a marine biology course. During a snorkeling field trip, students were divided into two groups. The first group used a static printed field guide to help identify species of fish they observed while snorkeling. The second group used a portable DVD player that used video, audio, and static screenshots to display the same information provided in the printed field guide. The addition of the video and audio connected the students to reference materials that more closely resembled the fish in their true context, providing a more situated learning experience. The researchers found that the students who used the dynamic field guide via the portable DVD players showed greater learning gains on a posttest than the students who used the static field guide.

Both of these examples show the connected nature of mLearning activities. The text messages and DVD players were used to connect learners to information. Additionally, the Pfeiffer et al. (2009) study provides an example of a situated activity because the portability of the DVD players allowed the learners to use them on-site. Neither of these examples was personalized, though, because each learner received the same information and viewed the same content.

Modification. Although research has provided examples of positive benefits from both substitution and augmentation of learning using mobile devices, Hockly (2013) asserts that it is in modification and redefinition that the true potential of mLearning is fully realized. It is at the modification and redefinition levels that learning is transformed.


In a case study by Cornelius et al. (2011), mobile devices were used to modify and significantly redesign a flood disaster simulation in an applied geomorphology course. The use of SMS text messages placed the simulation, based on a real-life scenario, in the students’ real lives. SMS text messages were sent to all of the students to update them on the events occurring in a simulated flood disaster. After each message, the students had to decide whether to mobilize the flood alert procedures or not and respond via text message. The next text message received by the student depended on his or her response to the previous text. Based on questionnaires and interviews, the researchers found that the students generally enjoyed the activity and felt it successfully helped them appreciate flood prevention measures. The use of SMS text messages allowed the simulation to be redesigned and allowed for real-time decision making by the students. This design provided truly situated learning. Just as in the simulation, in an authentic situation everyday life would be interrupted by flood alerts. Each student participated individually, supporting the personalized nature of the learning experience. The students suggested improving the realism of the simulation by providing more details in the text messages and providing the students with more than two options. The addition of more details, as suggested by the students, would not only improve the realism of the simulation but would also make the mLearning activity more connected. The real-time decision making significantly increased the realism of the simulation and thereby increased the educational value of the activity.

In another example of modification, Wang, Yu, and Wu (2013) designed a module, eMASE (mobile assisted social e-learning), for a speech and debate course. Within the module, students were required to work in groups. To support group interaction, training was provided on the most commonly used mobile social applications: Facebook, LINE (a social networking site from Japan), WeChat, Google Hangouts, and YouTube. Students were able to practice within each tool, and the training ensured that all the students and the instructor were included on their contact lists. Students reported that they felt the mobile applications improved their learning and were a useful tool. In this study, the use of the apps was optional, as all the tasks could be accomplished through other traditional means of communication, including meeting in person. The addition of the training and prompting students to think about using mobile social apps within the context of their courses, however, illustrates how mobile devices can connect people. The students’ freedom to choose whether to use the tools or not also supports the personalized nature of mLearning. One interesting pattern noted by the authors of this study was that the students who were frequent users of the social networking applications appeared to be more confident and engaged in their learning.

Redefinition. The exploratory case study described by Liu and Tsai (2013) provides an example of redefinition of learning using mLearning because learners were able to participate in learning activities that would not have been possible without a mobile device. The authors developed an augmented-reality cell-phone application to help Chinese students learn English. Using GPS to pinpoint the student’s location, English language descriptions of the things around that student would be displayed over the image seen through the phone’s camera. Based on student essays and reflections on an open-ended questionnaire, the authors found that participants were engaged in the learning scenario and produced meaningful learning and written essays. The authors concluded that the results of the study suggest that augmented-reality mobile learning may increase the effectiveness of language learning. This implementation of mLearning is personalized in that the students chose to use the application, is situated through the use of GPS and location, and is connected to information through the use of the application.


Redondo, Fonseca, Sánchez, and Navarro (2013) have explored augmented-reality in what they call “geo-eLearning” in architectural studies (p. 188). In one case study, students were divided into two groups based on the capabilities of their personal mobile devices. The experimental group used an augmented-reality overlay to view architectural proposals on-site in 3D. The control group used the traditionalmethods of 2D and 3D drawings. The results indicated that the students in the experimental group were more proficient in adjusting their proposals to fit the proposed site. Over several case studies, Redondo et al. (2013) have found that the students using the 3D augmented-reality overlay are consistently outperforming the control group. This study also illustrates learning that is personalized, situated, and connected. It is personalized with the students’ own projects, situated with the overlay view beingavailable on-site, and connected to information using the architectural applications.

While augmented-reality overlays are good examples that fall into the redefinition level of SAMR, Wu, Hwang, Su, and Huang (2012) provided another relevant example in a different setting. They used a “context-aware mobile learning system” to support nursing students in moving from the novice level to the expert level of proficiency in the physical assessment of patients (p. 223). Traditionally, students are provided with printed lists and demonstrations by instructors followed by time to practice with the dummy patients. The mobile system used in this study replaced the printed lists and guided the students through the practice time with real-time feedback and help. The mobile system connected with the dummy patient when the student approached, beginning with the patient’s chart. After a baseline evaluation, the system guided the student through a physical assessment, providing adaptive feedback and support based on the student’s degree of mastery. As the students practice, the feedback and support information is faded until the student can perform at the level defined by the instructor. Using a pretest–posttest design and including the covariate of the pretest, the authors determined that the students using the mobile system exhibited a significant increase in learning achievement. From the learning logs, Wu et al. (2012) reported that the instant, personalized feedback of the mobile system enabled students to practice more than three times the number of operations as students using the traditional format. In looking at the factors of attitude and cognitive load, the mobile system significantly improves student understanding and self-evaluation and significantly lowers cognitive load. This study exemplified the personalized component of the SAMR model through the use of personalized, adaptive feedback. The simulated lab and patients illustrate a truly situated context. And finally, the students were connected not only to the relevant patient information but also to support information based on their correct or incorrect actions.

Connection between SAMR and the mLearning definition. Puentedura (2013) notes that learning activities that lie at the modification and redefinition levels of the SAMR framework can transform learning. It is at these higher levels of the SAMR framework that the full potential of learning via a mobile device is realized (Hockley, 2013). After the ten articles included in this review were classified based on the SAMR framework, each article was reexamined to determine whether the mLearning example was personalized, situated, and/or connected (see Table 2). This analysis revealed that every example at the redefinition level of the SAMR model was personalized, situated, and connected. This was not true of examples at the lower levels of the SAMR framework. If learning activities involving a mobile device are purposefully designed to be personalized, situated, and connected, the resulting mLearning activities have the potential to redefine and transform learning.


Table 2

The classification of each mLearning example as personalized, situated, and/or connected.

Classification Example Personalized Situated Connected

S

Substitution

Evans (2008)

Gromik (2012)

Lan et al. (2012)

A

Augmentation

Chuang and Tsao (2013)

Pfeiffer et al. (2009)

M

Modification

Cornelius et al. (2011)

Wang, Yu, and Wu (2013)

R

Redefinition

Liu and Tsai (2013)

Redondo et al. (2013)

Wu et al. (2012)

Suggestions for instructional designers. As seen in the research examples above, mLearning activities have the potential to transform learning. Well-designed activities will be personal, situated, and connected through a mobile device to modify or redefine how concepts are taught. However, using a mobile device can also introduce a new set of potential problems.

For mLearning activities at the substitution or augmentation level of the SAMR framework, the increase in technological obstacles presented by the use of a mobile device may prove too cumbersome to justify the use of the mobile device. However, for mLearning activities at the modification or redefinition level of the SAMR framework, the increased technological obstacles will most likely be outweighed by the added benefits of mLearning activities. It is still advisable to develop an implementation design that seeks to decrease the barriers that may be created when using mobile devices in higher education.

While all of the studies mentioned in this review found mLearning to be at least as effective as other methods of learning, some mLearning projects are not successfully implemented. Cochrane (2012) noted that many successful projects initially had problems that were corrected in later iterations of the project. Some of the difficulties that arise during the implementation of a new mLearning activity can be avoided if instructional designers consider and analyze three critical areas as part of the instructional design process: (a) technical issues, (b) pedagogical issues, and (c) management issues (Gedik, Hanci-Karademirci, Kursun, & Cagiltay, 2012).


Technical issues. In regard to issues related to mLearning, it is essential to make an appropriate choice of mobile devices and software (Cochrane, 2012). One of the advantages of using mobile technology is that students often already have their own devices. However, allowing students to use their own devices may lead to difficulty ensuring that the design is compatible with all of the different devices (Gedik et al., 2012). Additionally, researchers have found that students often are not as familiar with their mobile device as might be expected (Dyson et al., 2009). Similar to the training provided in the case study by Wang et al. (2013), it may be necessary to help students learn to use certain features of their personal mobile devices. In some situations, students having different devices might support research, such as that done by Redondo et al. (2013) in which they used students’ devices to determine whether students were in the control or the experimental group. Alden (2013) suggested providing students with a list of approved devices. This approach limits the number of different devices that would need to be compatible with themLearning activities while allowing students to have a selection of devices to choose from.

Regardless of whether students use their own devices or if devices are provided for them, the instructional design plan should include provisions for providing appropriate technological support (Cochrane, 2012). Moreover, it is important that technical support is also provided for the faculty involved in the study (Alden, 2013).

Pedagogical issues. Researchers provided several suggestions for addressing pedagogical concerns when designing mLearning. Cochrane (2012) noted that it is essential to have pedagogical integration of the mobile device. If the mLearning activities are not included in the graded assignments and assessments of the course, students are less likely to take full advantage of the learning opportunities they provide. In addition, the instructor of the course should model appropriate use of the mobile device. Brett (2008) noted the importance of providing specific suggestions to students on how to best use the mobile resources. For example, he observed that when sending course content in text messages, over time students often forget that the information is available. Students will often read the message once and then forget about it. Students benefit when the instructor introduces a new mobile technology with “an explanation of its value . . . to ensure full learner awareness of the technology and the learning benefits of engagement” (Brett, 2008, p. 13). Providing regular formative feedback will also help the students see the benefit of the mLearning activities (Cochrane, 2012).

Another pedagogical consideration is that not all educational tools work well on mobile technology, and the pedagogical value of a learning object should be weighed against its ease of use on mobile technology. This requires designers to take into account things such as the screen size, available bandwidth, and the processing speed of the mobile device (Gedik et al., 2012).

Management issues. Creating a detailed plan for how the mobile technology will be managed will help avoid potential difficulties. This plan should include determining how information will be communicated among all of the participating parties (Gedik et al., 2012). The connected nature of mLearning makes it most successful when used as part of a supportive learning community (Cochrane, 2012), which requires the ability of members of the learning community to communicate with each other. Another important management consideration is how to manage the costs associated with using mobile technology. Although mobile devices are extremely prevalent, some students do not have a device or do not want to use their device for schoolwork (this is especially true of phones) (Brett, 2008). Alden (2013) recommended that the use of mobile devices by students not be required. Often it is possible to allow


students to opt out and use a different method of accessing information. For example, Cornelius et al. (2012) provided their students with the option to receive either text messages or e-mails when participating in a flood disaster simulation. This option allowed students who did not have a cell phone or who did not have a text-messaging plan included with their phone to participate in the simulation without incurring any extra cost. However, for mLearning activities at the highest levels of the SAMR model, students who do not have access to mobile devices will miss the greatest benefits of the activity.

IV. DISCUSSION

The personalized, situated, and connected nature of mLearning provides instructional designers with exciting new instructional strategies to consider. The use of the SAMR framework can assist in decision making when evaluating potential instructional designs that use mobile technologies. At the lower levels of substitution and augmentation, the obstacles of technology, pedagogy, and management may not be worth the learning gains. At the levels of modification and redefinition, however, mobile technologies become integral to the design of the activity and may be worth the potential problems. Nonetheless, instructional designers should carefully consider how to address the technical, pedagogical, and management issues that will arise during the implementation of the mLearning activity.

This review of the literature focused on studies in higher education that used a wide variety of mobile devices. As such, these recommendations may not apply to other learning contexts. The SAMR model, while helpful, is still very subjective. Using the dual lens of the proposed mLearning definition with the SAMR model provided a useful overlap that highlighted the implementation designs most likely to transform learning. It is recommended that other research studies be evaluated using this dual lens. With additional researchers evaluating the studies in this light, perhaps one model that combines SAMR with the mLearning definition would evolve. This model could then be used to guide the design and development of future studies.

Mobile devices provide unique opportunities that allow learning to be personalized, situated, and connected. Instructional designers should take advantage of these unique characteristics of mobile devices as they design mLearning activities. This provides the greatest chance of designing activities that fall at the highest levels of the SAMR framework. The mLearning activities that modify or redefine traditional learning activities have the potential for transforming learning through the use of a mobile device.


V. REFERENCES

Alden, J. (2013). Accommodating mobile learning in college programs. Journal of Asynchronous Learning Networks, 17(1), 109–122. Retrieved fromhttp://sloanconsortium.org/jaln/v17n1/accommodating-mobile-learning-college-programs Brett,

P. (2008). MeLAS mobiles enhancing learning and support final report. JISC. Wolverhampton, UK. Retrieved from www.jisc.ac.uk/media/documents/programmes/elearninginnovation/melasfinalreport.pdf

Cheon, J., Lee, S., Crooks, S. M., & Song, J. (2012). An investigation of mobile learning readiness in higher education based on the theory of planned behavior. Computers & Education, 59(3), 1054–1064. doi:10.1016/j.compedu.2012.04.015

Chuang, Y.-H., & Tsao, C.-W. (2013). Enhancing nursing students’ medication knowledge: The effect of learning materials delivered by short message service. Computers & Education, 61, 168–175. doi:10.1016/j.compedu.2012.09.013

Cochrane, T. D. (2010). Exploring mobile learning success factors. Association for Learning Technology Journal, 18(2), 133–148. doi:10.1080/09687769.2010.494718

Cochrane, T. D. (2012). Secrets of mlearning failures: Confronting reality. Research in Learning Technology, 5 (2012 Conference Proceedings - A confrontation with reality), 123–134. Retrieved from http://www.researchinlearningtechnology.net/index.php/rlt/article/view/19186

Cornelius, S., & Marston, P. (2009). Towards an understanding of the virtual context in mobile learning. Research in Learning Technology, 17(3), 161–172. doi:10.1080/09687760903247617

Cornelius, S., Marston, P., & Gemmell, A. (2011). SMS text messaging for real-time simulations in higher education. In J. Traxler & J. Wishart (Eds.), Making mobile learning work: Case studies of practice (pp. 13–17). Bristol: ES. Retrieved from http://escalate.ac.uk/downloads/8250.pdf

Dyson, L. E., Litchfield, A., Lawrence, E., Raban, R., & Leijdekkers, P. (2009). Advancing the m-learning research agenda for active, experiential learning: Four case studies. Australasian Journal ofEducational Technology, 25(2), 250–267. Retrieved from http://www.ascilite.org.au/ajet/ajet25/dyson.html

Evans, C. (2008). The effectiveness of m-learning in the form of podcast revision lectures in higher education. Computers & Education, 50(2), 491–498. doi:10.1016/j.compedu.2007.09.016

Gedik, N., Hanci-Karademirci, A., Kursun, E., & Cagiltay, K. (2012). Key instructional design issues in a cellular phone-based mobile learning project. Computers & Education, 58(4), 1149–1159. doi:10.1016/j.compedu.2011.12.002

Gromik, N. (2012). Cell phone video recording feature as a language learning tool: A case study. Computers & Education, 58(1), 223–230. doi:10.1016/j.compedu.2011.06.013

Grönlund, Å., & Islam, Y. M. (2010). A mobile e-learning environment for developing countries: The Bangladesh Virtual Interactive Classroom. Information Technology for Development, 16(4), 244–259. Retrieved from http://www.tandfonline.com/doi/abs/10.1080/02681101003746490

Hockly, N. (2013). Technology for the language teacher: Mobile learning. ELT Journal, 67(1), 80–84.doi:10.1093/elt/ccs064

Huang, R.-T., Jang, S.-J., Machtmes, K., & Deggs, D. (2012). Investigating the roles of perceived playfulness, resistance to change and self-management of learning in mobile English learning outcome. British Journal of Educational Technology, 43(6), 1004–1015. Retrieved from http://doi.wiley.com/10.1111/j.1467-8535.2011.01239.x


Kearney, M., Schuck, S., Burden, K., & Aubusson, P. (2012). Viewing mobile learning from a pedagogical perspective. Research in Learning Technology, 20(1), 1–17. doi:10.3402/rlt.v20i0/14406

Kukulska-Hulme, A. (2009). Will mobile learning change language learning? ReCALL, 21(02), 157–165. http://dx.doi.org/10.1017/S0958344009000202

Lan, Y.-F., Tsai, P.-W., Yang, S.-H., & Hung, C.-L. (2012). Comparing the social knowledge construction behavioral patterns of problem-based online asynchronous discussion in e/m-learning environments. Computers & Education, 59(4), 1122–1135. Retrievedfrom http://linkinghub.elsevier.com/retrieve/pii/S036013151200125X

Liu, P.-H. E., & Tsai, M.-K. (2013). Using augmented-reality-based mobile learning material in EFL English composition: An exploratory case study. British Journal of Educational Technology, 44(1), E1–E4. doi:10.1111/j.1467-8535.2012.01302.x

Pfeiffer, V. D. I., Gemballa, S., Jarodzka, H., Scheiter, K., & Gerjets, P. (2009). Situated learning in the mobile age: Mobile devices on a field trip to the sea. Research in Learning Technology, 17(3), 187–199. doi:10.1080/09687760903247666

Puentedura, R. R. (2006, November 28). Transformation, technology, and education in the state of Maine[Web log post]. Retrieved from http://www.hippasus.com/rrpweblog/archives/2006_11.html

Puentedura, R. R. (2013, May 29). SAMR: Moving from enhancement to transformation [Web log post].Retrieved from http://www.hippasus.com/rrpweblog/archives/000095.html

Redondo, E., Fonseca, D., Sánchez, A., & Navarro, I. (2013). New strategies using handheld augmented reality and mobile learning-teaching methodologies, in architecture and building engineering degrees. Procedia Computer Science, 25, 52–61. Retrieved fromhttp://www.sciencedirect.com/science/article/pii/S187705091301212X

Traxler, J. (2010). Students and mobile devices. Research in Learning Technology, 18(2), 149–160.doi:10.1080/09687769.2010.492847

Wang, J., Yu, W., & Wu, E. (2013). Empowering mobile assisted social e-learning: Students’ expectations and perceptions. World Journal of Education, 3(2). doi:10.5430/wje.v3n2p59

Wu, P.-H., Hwang, G.-J., Su, L.-H., & Huang, Y.-M. (2012). A context-aware mobile learning system for supporting cognitive apprenticeships in nursing skills training. Educational Technology &Society, 15(1), 223–236. Retrieved from http://eric.ed.gov/?id=EJ979517


Acknowledgements

We would like to acknowledge Dr. Dotty Sammons of Idaho State University for her suggestions and input on this project.

About the Authors

Danae Romrell has a background in mathematics and mathematics education. After four years of teaching high school, she has been teaching college-level mathematics for eight years. She is currently a member of the mathematics faculty at Brigham Young University - Idaho as well as a PhD candidate in the Instructional Design program at Idaho State University. Her research interests include multimedia design, online education, mathematics education, and gaming.

Lisa Kidder has 20 years of experience in teaching and learning with technology. She currently works in the Instructional Technology Resource Center at Idaho State University. With a background in French, chemistry, and educational technology in both K–12 and higher education, she brings a wealth of experiences to conversations on best teaching practices using technology. Her research interests are in online delivery, faculty development, and visual design.

Emma Wood has a background in special education, specifically deaf education. She has taught in Teacher Education at Idaho State University for the past five years. Emma is also the Instructional Technology Coordinator for the College of Education. Currently a PhD Candidate in Instructional Design, her research interests include underprepared students, Universal Design for Learning, and Self-Directed Learning skills.


SECTION IV: Qualitative Perspectives

Teaching Presence: Co-Creating a Multi-National Online Learning Communityin an Asynchronous Classroom

Leanne Dzubinski


Teaching Presence: Co-Creating a Multi-National Online Learning Community in an Asynchronous

Classroom

Leanne M. Dzubinski, Ph.DCook School of Intercultural Studies Biola University

ABSTRACTThe pace of globalization coupled with the growing institutional pursuit of online education means online classes are increasingly composed of a multi-national, multi-ethnic, and multi-cultural student body. Teaching presence, which is the ability to structure the class, create the social environment, give instruction, and assess student work, is the basis for creating a community of inquiry in an online class where successful learning can occur. However, little is known about effective teaching presence in a multinational classroom. The purpose of this study was to examine my own efforts to co-create a social context in an online classroom environment that was welcoming and supportive to a diverse student population enrolled in a graduate research methods course. Findings show that building student confidence, affirming student voice, and the strategic use of groups can help create a climate of safety. Effective instructor techniques include setting the tone of the class through clear expectations, having ongoing public and private interaction with students, giving effective feedback, and recognizing and valuing cultural differences.

I. INTRODUCTIONIn my first semester at a small, private West-Coast university, I was assigned to teach an online introductory research course in a graduate program. The 16 participants enrolled in the course came from five states, six countries, and represented multiple ethnicities. A number of students were multicultural in origin and in experiences, having been born in one country, raised in another, while working as adults in yet another. Similarly, a number of students married outside their original culture, and their ages rangedfrom 20s to 60s. In sum, they were a diverse group of students. As the course instructor, I was responsible for creating an effective online learning environment for all of them. I too am something of a multicultural person—though I was born and raised in the United States, I have lived and worked in Europe for almost 20 years. Therefore, the purpose of this study was to examine my own efforts to co-create a social context in an online classroom that was welcoming and supportive to a diverse student population enrolled in a graduate research methods course.


II. THEORETICAL FRAMEWORK & LITERATUREWhen facilitating an online class, I co-create a learning community with my students, both by the instructions I give and the by the expectations I explain and model through appropriate interaction and feedback (Cannon, 1990; Chick & Hassel, 2009; Tollman, 2003). As a feminist educator, I desire to create a safe space composed of a respectful and inclusive learning environment (Chick & Hassel, 2009). Each student is encouraged to have a voice, value is placed on personal experiences, and measures are implemented to ensure privileged voices do not dominate the conversation (Cannon, 1990). The online learning environment appears to support feminist pedagogy’s value of diversity and inclusion through its ability to reach students who might otherwise not have access to education (Chick & Hassel, 2009). It also appears to support feminist pedagogy’s value of voice (Johnson-Bailey & Lee, 2005) when it is structured so that all students are expected to post regularly in the discussion threads.Further, the world of online teaching and learning has its own concept of space that must be created for learning to occur. Anderson, Rourke, Garrison, and Archer (2001) first framed the idea of “teaching presence” to describe three functions of the online instructor: designing and organizing the course,facilitating the social environment, and serving as subject-matter expert. They considered teaching presence to be one of three factors that create a community of inquiry online, the other two being cognitive and social presence in the classroom (Anderson, et al., 2001). In 2008, Akyol and Garrison conducted a follow-up study and found that the three elements of teaching presence were well defined and had not shifted since the earlier study. However, in 2010, Shea, Vickers & Hayes reviewed the studies to date on teaching presence in the online classroom and suggested a slightly adjusted framework. They found that course design and organization, facilitated discourse, direct instruction, and assessment better reflected the concept of teaching presence in an online class (Shea, Vickers, & Hayes, 2010). They adjusted facilitation of the social environment to facilitation of discourse because some studies found students could not distinguish between facilitating the environment and direct instruction (Shea, et al., 2010). Therefore, Shea and associates expanded the facilitation of social environment to include some of the instruction categories that revolved around approaches to content and called it facilitating discourse (Shea, et al., 2010). They also added a category of assessment, because teachers assess both participation and assignments, thereby making their presence known in the online classroom (Shea, et al., 2010). The role of co-creating a social environment where learning can take place appears to be as vital to the success of an online class as it is in the traditional classroom. Because online interaction is entirely text-based without affordances such as visual, body-language, and tone of voice cues, the way social interaction is created becomes critical (Anderson, et al., 2001; Archer & Garrison, 2010). A community of inquiry framework, which serves as the theoretical basis for a successful online classroom, relies heavily on collaborative relationships in the classroom (Akyol & Garrison, 2008). The connection between feminist pedagogy’s inclusive, respectful environment and online learning’s emphasis on co-creating the social environment in which learning can occur is clear. However, the role that culture and ethnicity play in creating the social environment is not well understood, since little research has been done on multinational classes (Morse, 2003). Online classes may be designed by instructors who know little about the cultures and ethnicities of their students (Rogers, Graham, & Mayes, 2007). Western approaches and assumptions may not prove useful in teaching across cultures (Rogers, et al., 2007; Tollman, 2003). Western institutions and instructors might reflect “educational and cultural imperialism” (Sadykova & Dautermann, 2009, p. 90), which is contrary to notions of a safe space. Yet given the current pace of globalization (Merriam, 2010) and the continuous movement towards international online education (Sadykova & Dautermann, 2009), understanding how to create social context for learning in a multinational environment is of great significance to educators and institutions. Therefore, an online class consisting of students of multiple nationalities, ethnicities, races, professions, ages, and genders, is an ideal space to explore the intersection of teaching presence in social interaction and feminist concepts of respect and inclusion in the classroom as they relate to creating a virtual learning community.


III. RESEARCH DESIGNThe majority of the research done to date on teaching presence in online courses has been quantitative (Shea, et al., 2010). In fact, Shea, Vickers, and Hayes, in their 2010 review of the literature, found only one study conducted solely using interviews. The majority of studies have been conducted by examining online discussion boards quantitatively, and analyzing instructor and student postings for content. Because the focus of this study was to evaluate the intersection of my deliberate attempts to create a social context, and the students’ evaluation of their own ability to co-create and participate in that social context, qualitative research methods were chosen. Qualitative methods respect the constructivist understanding that knowledge is being co-created between myself and the students in our interactions (Bierema, 2002; Merriam, 2002) both in the classroom and in the research process. Qualitative Methods also recognize the researcher is the primary instrument in the study (Merriam, 2009). Interviews with students allowed me to understand their perspective on how the class worked and what meaning they made of their experiences (Merriam, 2009). This offered a more nuanced understanding of the course dynamics than can be found by analyzing public communications on a discussion board. Further, as recommended by Shea et al. (2010), this approach takes into account the entire course interactions, not just the postings on the main discussion threads. Finally, qualitative methods are suited to feminist inquiry because they take seriously the participants’ own meaning-making, treating it as central data for the study(Merriam, 2009).

A. Research QuestionsThe research questions for this study were:

What constitutes a social environment of safety in online learning community composed ofmultinational, multiethnic, and multigenerational students?What techniques used by the instructor are effective in creating a social environment of safety inonline learning community composed of multinational, multiethnic and multigenerationalstudents? What is ineffective?What is the role of culture in the online learning environment?

B. Data Collection and AnalysisThe first source of data was my researcher’s log. Although a journal is typically thought of as part of the audit trail supporting a study’s validity (Merriam, 2002), in this case I used my journal as a primary data source. It resembles field notes kept as part of an ethnographic study (LeCompte & Preissle, 1993). I kepta running log all semester of the deliberate moves I made which were intended to create a supportive social context for learning in the online class. I also noted points of concern or conflict that arose, as well as jotting down questions I had about the progress of the course. By comparing my journal with student’s perceptions of the class dynamics, I hoped to gain an understanding of the efficacy of various efforts to create a social context for learning with a diverse group of learners.The second source of data was a series of interviews conducted over a period of four weeks after the course was finished and final grades submitted. I sent an email to all 16 students in the class, inviting them to participate in an interview with me to talk about their social experience in the class. The purpose of the interviews was to explore their perceptions of the climate and interactions in the classroom, seeking to understand the meaning they made of the class experience (Merriam, 2009). Eleven students expressed interest in talking with me. During the first two weeks in January, I conducted 30 to 40 minute interviews with six class members. One student, a second-language English speaker, asked if she could send me written responses since she wanted to participate but was concerned about the verbal interaction. She sent me two pages of comments in response to the interview protocol. I recorded and transcribed all of the interviews. Once the first six were transcribed, I loaded them, along with the written commentary from the seventh student and my researcher’s log, into my data analysis software. I began the process of initial coding using constant comparative analysis (Butler-Kisber, 2010), seeking themes and patterns in the


data. The seven student sources plus my journal soon showed some alignment between my intentions and student responses. However, as I continued coding and comparing data segments, I also noticed some variety in student perceptions, which is a common occurrence in qualitative research (Marshall & Rossman, 2006). As a result, I emailed the remaining four students who had expressed willingness to talk with me but who had indicated they were not available until later. Three of them agreed to an interview, and in late January I conducted three more interviews of approximately 30 to 40 minutes, seeking further insight into the social environment of class. With those interviews transcribed and uploaded to the program, I moved on to focused coding, seeking patterns or themes in the data. With the additional interviews some themes in the data became clearer,while new ones surfaced. Finally, using axial coding, I began to group the themes into categories of findings. In the findings all students are referred to by pseudonyms. Also, the majority of the students with non-Western names had chosen a Western name for themselves in class. In creating pseudonyms I have respected their tradition. Table 1 shows the demographic makeup of the participants.

Table 1. Participant Characteristics

Name Born/Raised Live/Work EthnicityKaren Taiwan Philippines, US Taiwanese-AmericanLaura Korea, US Asia Korean-AmericanBetty US Mexico American-MexicanSamantha China US Chinese-AmericanBeth US US African-AmericanJohn US US Japanese-AmericanPeter Asia, Australia India/Singapore Asian-AustralianRobert Canada Korea CanadianSteve US Global WhiteMike Korea US KoreanLeanne US Europe White

IV. FINDINGSA. Safe EnvironmentTo answer the first question, what constitutes a social environment of safety in a multinational online course, findings suggest for some students an online class can offer a safe space to interact, and it may even have some advantages over a traditional face-to-face environment. Building confidence, creating space for each voice, and working with smaller groups were the primary methods of creating a safe classroom space.

1. Building confidenceA number of the students commented on their initial lack of confidence as a hindrance to safety in the classroom. Both Laura and Betty came in feeling anxious about the class. Both have family responsibilities and had been away from formal education for some time. Laura initially felt unsure of her academic abilities. She commented that the class felt safe because “I didn’t ever feel that I was made to feel dumb, or stupid, or how come I didn’t know that.” For her, being safe included a reduction of anxiety and encouragement that she could do the work. For Betty, it was important that the environment not pit the students against one another. She appreciated being “released from a competitive feeling” and instead


learning to support and encourage one another. She thought that “if you feel safe, and if you feel you’re in a warm nurturing type environment, that you can relax more and you’re not so concerned about everything being judged harshly, but as friends working together, maybe.” For her, competition was asource of tension, but cooperation and encouragement created an environment for learning. Karen had a similar response. She had learned in a community college class that “if we are in a comfortable and accepting environment, we can learn better” and she found that true for this course.The students used words like “affirmation” and “encouragement” to describe how they began to know the space was safe. Robert explained that “there was a definite effort to encourage each other and to affirm each other. And I think that was very positive.” Karen added that “sometimes we feel dumb, because it’s not easy to write” so the encouragement to “keep on, press on” was “very helpful.” Betty commented “I liked the affirmation. I think that you’re very affirming, and I think you set the stage for the rest of us.” In my instructor log in week five, I noted that “I continue to blend encouraging messages regarding their progress with pushes to improve.” The student comments show that they noticed this strategy and responded positively to it.

2. VoiceAn important aspect of a feminist classroom is voice, or the idea that each person has something valuable to contribute (Johnson-Bailey & Lee, 2005; Tisdell, 1998). In setting up the class, I established the requirement that each student post their responses to the weekly questions, and then interact with three fellow-students. This approach is common in online classes and primarily serves as a learning strategy, but it also can serve to help each student develop voice since it requires participation. Several students recognized that the online environment supported development of voice. Robert commented that the students were encouraging each other, and that the online environment was conducive to positive interactions, perhaps more so than a traditional classroom. He reflected on a recent face-to-face class environment where “some people didn’t participate quite as much . . . because of the fact that [other] people are more dominant.” In contrast, he thought that the online participation was more balanced, “perhaps that might have been more possible with the online format than might have happened in class. I’m not sure if everybody would have always done that in a class setting.” Peter agreed with that idea: “in an online forum we might have been able to say things we might not have said face to face. . . . It forced us . . . to think of an interaction because we have to post something.” For him the requirement to postcombined with the online environment created interaction that would not have occurred in person. As a bi-cultural person, with both Asian and Western backgrounds, he understood the different cultural dynamics clearly. He explained that in Asia students are not expected to speak up in class, but in Western classes participation is valued. So participation can be a difficult skill to learn. He commented, “I think Asians coming into our school environment for the first time have to take one or two years to learn that. And the online classes actually provide structure to do that.” He saw that the online environment created astructure that supported the development of voice and confidence. Online interaction supported another aspect of voice, as well, for the second-language participants. Mike described himself as “shy” which made it hard for him to participate, but he also commented on his “poor English listening skills.” Because everything was online, he could take time to look up words and think about his responses. He could also ask for more explanation when he did not understand something. So for him the online environment was beneficial to participation. Similarly, Samantha needed extra time to understand. She called it the “two-second processing delay” and explained that the online format gave that extra time she needed to interact. Peter, though not a second-language speaker himself, also commented on the benefit for Asians of having time to think through their responses. He explained that“learning to find your comments, process the composition and comments, and then post them, is like a safe environment . . . it gives a lot more processing time, but you’re learning the flow.” So for some second language speakers, the release from time constraints in the online class helped support their safety and their learning.


3. GroupsA third aspect of safety described by the students had to do with groups. Realizing that 16 students weretoo many for productive discussion, after the first week’s introductions I divided students into two groups of eight for the discussion threads. I maintained the same groupings throughout the course. Students appreciated being divided into smaller groups for the weekly discussions, and they appreciated receiving group feedback, especially when it was of a corrective nature. Samantha explained, “I enjoyed the discussion forum has been resized into a smaller group, so that I would be able to focus and follow up each other’s work closely.” Robert thought the “group size was advantageous; it wasn’t overwhelming that way and we got to know each other a little better.” Karen said, “It’s good that you divide in two” because at first it was too many people, but she “enjoyed them” when it was only eight. Peter agreed, saying, “At the beginning it was feeling unwieldy, to keep track of a large number of discussions, with the large class size. The smaller group was more intimate.” He said it enabled stimulating discussions while allowing him to keep track of them. No one really wanted a larger group. Laura expressed some dismay at losing the input of her other classmates. So I asked if she would rather have stayed all together, and her response was decided: “Oh, no! Because it was overwhelming at times for me, with . . . limited time to read all seven other posts and threads . . . . I think six to eight is a good number.” Interestingly, the second aspect of groups that made students feel safe was group feedback. In my log, I recorded that I struggled regarding how to give feedback so that individual students would not lose face or be shamed publicly. In week three I wrote, “I actually considered sending feedback individually by email so no one would be embarrassed. But then I decided this goes against the ethos of the class. Plus there were so many mistakes in common” that it made sense to correct publicly. The following week I described what happened:

This week I played around again with public correction. It’s a mistake that a lot of students are likely to make, and frankly it’s easier for me to point it out once than 16 separate times. So I worded it as “do you mind being our experiment” and pointed out what went wrong and how to fix it, after saying what went right. For the second mistake, I simply created an announcement and reminded everyone to do something different, and tried to make is slightly humorous.

As the description shows, I tried a two-part combination. First, I did give individual feedback publicly on the discussion threads to everyone. Second, for common, repeated mistakes I simply posted a correction to the entire class at once, rather than pointing out each individual’s errors. Many of the students found this approach to be very helpful. Beth commented, “That was good. Totally do the same thing [in future] and it would help people.” Laura said, “I appreciated when you had the greater posts for the entire class. That was nice because we could learn from both groups.” And Betty was very clear: “If you did give us a reprimand . . . you gave a whole group reprimand. And so even a whole group reprimand makes people feel part of a group. Because we feel like we’re the group.” For her, group correction strengthened themfeeling “like a team” and did not “cause people to compete.” That was a significant part of the safety for her.

B. Instructor TechniquesIn answer to the second question regarding techniques an instructor can use to create a social environment of safety, the findings suggest two primary categories of actions that I as the instructor did which were effective in establishing a social context for learning in an online class. The first category relates to things I did to set up the course and launch it in the early weeks of the semester. The second category relates to ways I interacted with students during the course of the semester. Interestingly, these two functions closely resemble Anderson et al.’s (2001) other two aspects of teaching presence: designing the course and serving as a subject matter expert. I will return to this point after describing the various techniques that the students found helpful.


1. Setting up the courseThe very first set of readings I gave for the class in the first week included a rubric for online discussions and a link to a web page about Netiquette. The rubric laid out expectations that posts would be scholarly, reply to and build upon one another’s ideas, and show respect and sensitivity in all interactions. The netiquette link laid out similar expectations in more detail, describing explicit guidelines for helpful and ethical online behavior. Of course, this approach to discussion threads is quite Western, but making expectations explicit at the start is one recommendation for successful online interaction in international courses (Sadykova & Dautermann, 2009). Five of the students mentioned those early readings as useful for setting up the tone of the class. For example, Beth said, “What was great about the whole program was you gave us the netiquette portion.” Robert explained, “The outline . . . from the very beginning is that we’re learning together. There is respect that should be part of our whole approach. . . . So that just set the tone right from the beginning.” Laura explained that the “initial readings kind of set up the social environment… [they] really enhanced the connection.” Betty thought it kept them from commenting incorrectly:

I think it was very good, right before we started posting you had us read an etiquette article, and that was really good because it gave us some guidelines. And again I think that also kept us from coming across too competitively or aggressively.

Mike also expressed appreciation: “You taught us etiquette for the online discussion. That was very helpful.” Clear expectations and guidelines seemed to promote a good atmosphere.

2. IntroductionsThat first week of class I also invited every student to introduce themselves and upload a photo. I started by introducing myself with a short biography and a picture. As each student posted an introduction, I responded that I was looking forward to learning with them this semester. In my log I recorded that “modeling appropriate responses by responding personally to every student’s introduction” was a strategy that I used in setting up the course. In the interviews six students commented on the significance of those introductions and photos to help them get to know their fellow classmates. For example, Mike commented that “in the first class, you let us introduce ourselves with picture. That was very helpful for us to know each other and to remember.” But not everyone did remember. In the interview, Peter said:“I’ve never even seen the photos of any of these people. So I don’t actually know what they look like… Maybe that’s something that could be done in the first week. And I can’t remember whether we did that.”When I reminded him that we did indeed do that, he was surprised; he had completely forgotten. So it didnot help him any. But another student, Laura, who was also having trouble remembering, said that she “took screen shots of what everybody wrote of their personal introductions, and in the early weeks I did refer back to that a few times.” She wanted to remember the others and found a way. Karen noted that she found it helpful that I also introduced myself to the class. “I think first you introduced yourself to us” she commented. “And also the beginning, you [get] us to introduce ourselves, and also the picture, so we get to know each other” she added.

3. Dedicated social spaceThe third strategy I used in setting up the course was to create a space called the water cooler for social interactions. I encouraged students to use this space to post anything not directly related to course content, and I launched the thread with some humor about doctoral students as well as a discussion about managing studies and life. Students used the space for several different kinds of topics. Six students specifically named the water cooler as something that helped them feel socially connected to the class. Robert liked that they could use the water cooler “if we had secondary issues.” Karen said “we can have something that is not related to the assignment . . . and I find that is very helpful. That’s very good.” Laura commented that “it was excellent to enhance the non-academic side.” Betty recalled a specificincident where “one person—he was really suffering because of the loss one of the colleague’s children, and I think it was a very painful time for him. And . . . it kept it from being too academic in a way.” They


liked having a space to post things they found useful and thought others would appreciate. But two students also thought it was insufficient. Peter thought people did not really use it enough. He said, “Even at the water cooler I didn’t see other people saying, oh, tell me a little bit about yourself.” And Mike wanted it to have a more spiritual focus where students could pray for each other. He said this would help them “have more strong relationship.” So the water cooler was useful, but less so for some students.

4. Instructor interactionThe second aspect of the class that every participant mentioned was their interactions with me. They commented on my interactions on the discussion boards, my feedback on their work, and my personal interactions with them by phone and email outside of the official class setting. They noticed how I interacted with them on the discussion boards. Steve commented that “when responses became [superficial], you pushed us all a bit—be specific, be academic. I think those were important.” Betty noticed something similar, explaining that “you did give some personal comments when we started to go off on the wrong track.” She also commented that she looked to me as a model for how to post: “I know for sure that I watched your… comments because they were a model for me.” Beth described my presenceas “your leadership. You were always there. So if there was a problem or concern, someone else had already asked the same question and you had already responded. So…your presence was important to me as a student.” Samantha expressed something similar: “my professor…is actively involved in the class and very quick to answer our questions.” John also appreciated my involvement in the discussion board, saying “As we’re interacting, it’s good to hear from you. Sometimes I wonder are we on the right track, are we doing this right? So I thought that that was good.” The students appreciated my visible presence on the discussion board.

5. FeedbackSpecific individual feedback on course work was found to be important to each student. Karen appreciated that I was “friendly, and give us feedback, and also write emails every week giving us information and greeting us. That is really helpful and kind of friendly, encouraging me to keep on.” Peter liked that “your comments on our work were very detailed. . . In this subject, more feedback is better, and detailed feedback is even better.” Another student emailed me at the end of the class, saying:

I have really enjoyed what is probably going to be my only on-line course, thanks to you. I've appreciated your guidance and input through the course and have learnt a lot from interactions with you. Thanks also for the effort you put in to give us such detailed feed-back, which has been particularly helpful.

Towards the end of the course, I decided to try a new approach to giving feedback. They were beginning to post early drafts of their final paper for feedback from me and their peers. One group spontaneously posted their drafts directly into the threads, rather than uploading attachments. I wrote in my log:

In one group, they posted their drafts in the discussion blocks, and in the other they posted attachments. So in the first group I gave interlinear comments in red directly in the discussion boxes. This made it REALLY easy for everyone to see what critique I was giving. The feedback was very positive. They liked seeing their own feedback this way and they liked seeing each other’s drafts. They found it easy and clear.

In the interviews a number of students commented on how much they liked that particular form of feedback. Beth really liked it; she said, “So your feedback, if you do this again, don't change that. That was phenomenal. When you finally put it in red, it was like, wow!” Betty also really liked it. She talked about “the first week you added comments in red directly to our homework. I, and many others, felt that this was very helpful. Several students commented favorably online about this kind of feedback.” She is correct; the online responses to the interlinear red feedback were overwhelmingly positive. The following week, our online platform was updated and I could no longer give interlinear feedback effectively, and the group was quite disappointed. While I provided feedback within their word documents, students did not feel it had the same effect as when they saw it right in-line in the discussion thread.


Another issue was ineffective or harsh feedback. Beth was frustrated because she felt she did not get good peer feedback. She said, “The feedback to me was so superficial, and I felt like I was always giving but not getting.” She also noticed that some students gave inappropriate feedback, which she explained: “they were very highly critical, so I did see that. As if they were the professor.” Betty experienced this too. She said, “one student used the marking tool on Word, corrected my paper, and I felt like he was my professor.” She also noticed harsh feedback from one student to another. In this case, she added, “You interceded and softened it for the person who was receiving it, I think, but also kind of stopped that kind of aggressive [feedback]. I felt like it was a little aggressive.” Finally, some of them noticed that I waited a little bit before engaging with their posts. Betty said, “Yougave the question, or the comment, and then you waited…a little bit to let people post…and then you gave some general pluses, and some individual positive remarks to kind of sum up.” She liked the summing up because it helped her know what was important, and also let her know that it was okay to move on to the next topic. Mike also liked that I gave them time to interact first, and then added myfeedback. He explained:

You let us give comment [to] each other at least three times a week. Through the feedback we could be interested in each other and take care of each other and encourage each other. . . . And you gave comments to us the end of week. Very good strategy because if you had given us comments the earlier week, we could not have given comments each other.

He was right, because I deliberately waited until late in the discussion period to give feedback, thinking that for some students, once I as the instructor/expert spoke, they would not say more. Mike’s comments supported that thinking.

6. Individual communicationIn the log I also recorded having “personal communication by email and phone, as needed with individual students requesting extra support and input.” There were multiple journal entries regarding personal communication, most of them positive but a few of them more challenging. Every participant mentioned their appreciation of this interaction. For example, Peter explained:

I was able to just even comment on my own personal sense of I’m not a very good proofreader. And you did suggest, you know, maybe one option is to pay for a proper proofreading right at the end of your dissertation. And you explored things that were more for life process, rather than just academic process. So I appreciated those little dialogues that we had, even though they were just one or two lines sometimes.

He talked further about the benefit of personal communication, remembering a discussion I initiated withhim about his academic goals.

I did appreciate at one stage you [asked about my academic goals]. And that generated a few emails and included the program chair. I felt as if you cared for me and my academic world and my personal world more than just, let’s get this subject done. And I did appreciate what I felt was going beyond the call of duty.

I also had some dialogues with Samantha on her study, and she wrote “I enjoyed the professor would be able to participate with each one of the student [sic].” Karen too found it helpful, saying “I appreciate that you give me time and you return my emails promptly, and I think you are open and welcoming to ask a question.” Steve also appreciated the extra personal communications. He said, “you and I have talked on the phone or even through email interaction somewhat” and he liked that better than purely course communication. He also liked the invitation to participate in the study, saying, “I think an exit interview is positive.” Laura was particularly enthusiastic in her comments, saying:

Oh! I appreciated the one to one emails, like you were giving us personal attention and care. Yes, we were part of the group online, but I felt like you were giving me personal care when I sent an email directly to your school address and then you responded back to my school address outside of the threads for everybody. That was very, very nice! I felt it was almost one to one teacher


interaction. That was helpful. I would suggest continue doing that. I know it’s more work for you but I appreciated that.

Personal communication was also important when difficulties arose in the course. One student made a post that could be read as racist. I emailed John, who received the comment, and asked what he thought. Iemailed Steve, who made the post, and pointed out how it could be understood as racist. In the interviews, I asked each student to reflect on the experience. John saw my interactions as a “good balance of the intercultural, the cross-cultural component…and then if I could use this term, critical consciousness.” He did perceive the post as being micro-aggressive and appreciated my intervention, saying, “I thought it was quite thoughtful and sensitive actually. In other words, you were the one that was proactive in asking me how did I feel about that, because that doesn’t always happen.” Steve also was thankful: “As I re-read my actual responses…I can definitely see how this could be misunderstood. So I thought you did a good job there and I was appreciative, and thankful that I got a chance to respond.” Careful use of personal communication helped correct a misunderstanding in the class.

7. Hindrances to Online LearningOf course, not all students liked the online venue. Two in particular were fairly strong in expressing their overall dislike of online learning. For one student, Beth, the entire experience was fairly negative. When I asked about the class, her reply was an emphatic, “I hated it to no end!” She was frustrated by the superficial feedback she received from her peers on her work, was frustrated by the poor English skills and lack of confidence of some students, and was keenly aware of gender and race power issues she saw play out in the environment. Another student, Steve, was seriously frustrated by the lack of personal interaction in the course. He said, “The social aspect for me was a downer.” The main problem was the misunderstanding he had with John, and he found that challenging to work out in the online framework. He was sure that it would have never happened face to face, because personal conversation would have allowed for immediate clarification and correction. That incident, coupled with his strong desire for visual and verbal communication, seriously affected his feelings about the class. Both students recognized that they had made some academic progress, but neither would be likely to take an online course if they had other options available. Although both students were able to name specific aspects of the course they found beneficial, overall they did not enjoy the experience.

8. Desire for face to face contactEvery student in the study wished for face-to-face contact. A few of the students had access to the university’s campus and were able to attend two library training workshops that were offered during the semester. The sessions were taped and uploaded, but those who attended in person felt they gained most from the sessions. Mike said, “That was special time, just one and a half hours, but through the class we had very good relationship [sic] with each other.” He explained, “To have good deep relationships, you need to meet face to face.” A few students were able to see me face to face as well, and they expressed appreciation for that. John realized it helped him, saying:

It was a real benefit to me, to come to your office and interact the way we did. It just helped get me on a good track, on a good path. So in that regard I was at an advantage being on campus.

Betty remembered a personal conversation we had right at the start of the semester: “When you told me that you had also gone back to school when your kids were older and it can be done! Because I looked around at all these young students, and I’m thinking, what have I done!” That conversation in week one reassured her. Karen did not initially realize it was okay to visit my office since the class was online. But Betty mentioned having seen me, and Karen decided to come as well. In the interview she explained:

In the middle I learned that I can ask you. I learned from Betty that…she can make appointment with you and ask you questions, so oh, okay, I can do that. Because I thought it was an online class and I didn’t feel comfortable to ask.


9. Lack of face to face contact hinders learningThose who did not have face-to-face contact with me and the other students lamented that fact. Laura explained, “I…just wish I could have run over to meet people face to face for two-way dialogue, just to ask for clarification or for two way dialogue to happen, so that was a little frustrating at times.” John explained, “The distance of not knowing people personally was a little bit of a challenge.” Robert thought he had better interactions with the two students he had met face to face in a previous course: “I knew a couple of the people already. . .so interacting with them would probably be on a little deeper level.” Peter agreed:

In the online courses the ones who I know from the extension center, I felt much more ease in communicating. Those would be the ones that, quite often, I’ll comment on their papers first. There was a sense that I would know how they will react, even if I say something which is not positively constructive, but may come across as negative, I know the person and I know they’ll take it okay because we have a relationship.

Steve and Beth, who expressed the most dislike of the class, both thought the online format was a hindrance to their learning. Steve would have preferred personal contact, saying, “I like the face to face, I like facial expression and dialogue that takes place, learning within a social environment where you see and hear the auditory aspects and visual aspects.” Beth too commented, “If we could have met first, in the classroom, to see everyone, get understanding, hear their voices, it would have been probably easier for me as well.”

C. The Role of CultureTo answer the third question, what role does culture play in creating an online learning community, the findings show that acknowledging and discussing the different cultural perspectives contributes to the learning climate, but does not completely overcome potential challenges. Most of the participants in the study discussed the role that culture had played in our online interactions. They saw both benefits and drawbacks. John thought the variety added depth, saying, “I thought it was interesting hearing people reference their experience as they thought through the questions and so forth.” Samantha said something similar: “My classmates have various background, their opinions, life philosophies and their depth of knowledge are something they brought into this class.” But she went on to say that she “felt I lacked” when her classmates discussed things outside her personal experience. Betty said that initially it affected how she communicated, explaining, “I think at the beginning, because we were from various cultural backgrounds, I felt like I needed to be cautious in how direct I was.” She went on, “I didn’t know how fragile people would be…but I think that will happen any time you get into a multicultural situation.” She was aware, but also thought it was normal. Laura appreciated the more Western approach, saying:

I came away feeling more equipped but positively, not in the Asian context where if you don’t get it right the first time then why don’t you have it done, and I’m going to give you a bad grade, that kind of a situation. So it was fun to learn together!

This particular class was primarily an Asian and Western group. So a significant amount of reflection involved contrasting those two ways of thinking and approaching the course. In my log I recorded that early in the semester we had a discussion regarding disagreement. “Western/US students want disagreement and push-back, so they can hear a new perspective and learn from it. Asian students are uncomfortable with open disagreement and prefer to express agreement. There has to be a way to disagree in Asia, but what is it and how can it apply to this course?” We worked quite a bit on the concept of critical review of publications and peer writings. Karen felt she made progress in her thinking.

[For] the Asian, it’s sometimes difficult to distinguish—to separate—people and things. And so sometime I hesitated to give my feedback…but later I learned that critical thinking doesn’t mean to say negative thing, that you don’t write well, or something like that. But to train myself to see this is a good one, and that isn’t, but I can express my opinion. And that’s kind of training to help me to see, oh, this is okay, I need to learn to be critical. Critical doesn’t mean negative. That was


very helpful.Peter also talked about this concept at length. As an Asian-Australian he bridged the divide and was able to see from both perspectives. Speaking of another Asian student, he said, “Take this as an observation,” and explained:

He was very cautious in his interactions in the classroom, especially in the beginning. The whole thing of Asians speaking up in the large classroom—they’re a bit more hesitant. I think the online class has really helped him get used to what this interaction is all about. Because we’ve just finished a subject in the extension center, and it’s the most I’ve seen him interact in class, ever.

Peter thought perhaps the online class helped the Asian students increase their participation.Finally a couple of participants thought the combination of online and cultural differences was ahindrance. Steve thought being online might hamper Asian students, commenting, “I find that within an Asian context, when an Asian student struggles, they would have a lot of trouble with not wanting criticism, and facial expression was really important in communicating with each other. So I think that would be a challenge in the online learning environment.” And Beth thought the cultural challenges hindered the online learning.

Culture played a huge role and the passivity of the feedback, of the self-confidence. Gender played a role in it, a huge role. . . . Cultural differences, male dominating female, was huge in this class. And it made me sick to my stomach. Second language, same thing. I think some of us may have thought less of those with that language barrier. Also, that language barrier played a role . . . in feedback.

For Beth, the issues that came with cultural and linguistic and gender differences hindered rather than helped her learning. All of the participants noticed the cultural differences, and most of them were willing to discuss them. But for some of them, the challenges seemed to present significant levels of stress that hindered their learning.

V. DISCUSSION AND RECOMMENDATIONSThe purpose of this study was to examine my own efforts to co-create a social context in an online classroom that is welcoming and supportive to a diverse student population enrolled in a graduate research methods course. I wanted to discover what a safe space looked like in that context, what instructor techniques were effective in building an online social community and what role culture played in the social environment. Early literature suggested teaching presence included setting up the course, facilitating the social environment, and the instructor serving as subject matter expert (Anderson, et al., 2001). Later reviews added a fourth aspect, assessment. The findings from this study support the four categories, and suggest a few specific factors that may be of particular importance in a multicultural online class (Akyol & Garrison, 2008). First, the findings show that social context is in fact inseparable from the broader construct of teaching presence. While coding the interviews, I wrote several memos about how interesting it was that students seemed unable to separate the academic and social aspects of the course. I would ask a question about the social environment, and students would respond with comments about the academic work. Karen stated it quite clearly: “the social is also related to class as well.” For them, the course organization, the conversations, the instruction, and the feedback were inseparable, and together built the concept of a safe space. The kind of instruction I gave and the way I gave feedback were equally as important as the overtly social things like introductions and the water cooler space. In the end, the students told me that all four strands were vital. Without good course design, and content expertise, and supportive feedback, the social environment would fail to materialize (Akyol & Garrison, 2008). Second, the findings suggest that the online environment may be useful for students who are reluctant to


speak up, initially lack confidence, or are working in a second language. The actual structure of requiring everyone to participate regularly seemed beneficial for this group of students. They knew participation was expected and counted toward their grade, and so they consistently engaged in discussions. Receiving positive feedback from me and from their peers enabled them to keep trying, and to slowly develop confidence. The asynchronous nature of the class allowed for time to think and to construct responses, which was beneficial for the less confident as well as those working in a second language. They could take time to process, think, and polish their responses before posting them online. This aligns with the findings of Gunn, McSporran, Macleod, and French (2003) that shy students can benefit from the online format, and that discussion boards with posting requirements can help build student confidence. Third, findings from this group suggest that building in an opportunity for initial face-to-face contact might enhance learning in a multicultural online course. Students who did have an opportunity to meet me or some of their fellow-students invariably thought this helped them with the class content. The two students who struggled most with the course thought face-to-face contact would have significantly improved their experience. If it were feasible to launch a class with a short, intensive time together, learning might be enhanced, as Morse (2003) found to be true with his study of high context, primarily Asian students. Alternatively, phone calls, Skype calls, or even videos posted online might help with this aspect of the course. However, not all students have access to high quality networks, so feasibility would need to be studied carefully before adding certain elements to a class (Rogers, et al., 2007). Finally, the findings suggest that the online class can work well with a multicultural student population, but it may require some extra care and attention on the part of the instructor. Being willing to surface and discuss an area of challenge seemed helpful. Sadykova and Dautermann (2009) recognize this dynamic, saying “cultural and individual differences are not only acknowledged but also become a matter of exploration and pride” (p. 102) Early on we talked about needing to express disagreement in the form of constructive criticism. Repeatedly through the early weeks, I modeled this for the class. I also encouraged students who were simply posting positive responses to the readings to offer some critique as well, and gave them examples of what they could say. This type of formative assessment requires time and detailed involvement by the instructor, and seemed to show positive results. Some of them, like Karen, learned easily, while others struggled more, but all students made some progress in learning to critically evaluate the work they were reading, which is considered an essential skill in a Western doctoral program.Interaction between the instructor and the students also played a significant role in students’ perceptions about this course. Both in terms of feedback online, and personal communication through email, phone, and office visits, the students needed to feel they could interact individually with me. Different students preferred different methods, as well. Steve liked phone calls so he could hear my voice. John and Karentook advantage of personal visits when they were on campus because they preferred oral communication.Laura liked personal emails because her phone and internet service were unreliable. Being accessible to them in the form they preferred added an extra dimension of strength to the class. Like all studies, this one had limitations. The sample was small, and the findings may not necessarily resonate with other multicultural online communities. The fact that I as the professor conducted the study may have limited some student’s responses, even though the class was over and grades had been submitted. Additionally, this group was primarily Asian and Western, so the findings might be different with other cultural groups. One aspect of diversity was absent from this group, as well. The school is a Christian one, so there was no religious diversity in the representation. Further study needs to be done with students from other cultural and religious backgrounds to see how well these findings apply, and what additional factors might contribute to a successful multicultural online learning community. Overall, the results of the class were very positive. The students performed very well on their final summative assessments, and many of them expressed appreciation for what they had learned during the course. The online environment clearly has some challenges that are different from those of a face-to-face class, and adding multiple cultures and ages can increase the challenge. But the variety can also add to the depth and richness of the interactions, as a number of students mentioned in their comments.


VI. CONCLUSIONThe benefits of online learning in terms of accessibility, and the opportunities in terms of globalization, seem clear. There is a growing need for adult educators to be competent cross-culturally in our teaching and research (Johnson-Bailey, Baumgartner, & Bowles, 2010). However, our reach may currently extend beyond our theory, as western educators can now “teach” people from around the globe (Rogers, et al., 2007; Tollman, 2003) without having any clear sense of how multicultural, multiethnic, and multigenerational challenges impact our classrooms. This study offers a small insight into my attempts to be deliberate about my teaching presence, particularly in co-creating a social context where students feel safe, accepted, open, and willing to learn from me and from each other.


VII. REFERENCESAkyol, Z., & Garrison, D. R. (2008). The development of a community of inquiry over time in an online

course: Understanding the progression and integration of social, cognitive and teaching presence. Journal of Asynchronous Learning Networks, 12(3-4), 3-22.

Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2), 1-17.

Archer, W., & Garrison, D. R. (2010). Distance education in the age of the internet. In C. E. Kasworm, A. D. Rose & J. M. Ross-Gordon (Eds.), Handbook of adult and continuing education (pp. 317-326). Los Angeles, CA: Sage Publications, Inc.

Bierema, L. L. (2002). A feminist approach to hrd research. Human Resource Development Review, 1(2), 244-268.

Butler-Kisber, L. (2010). Qualitative inquiry: Thematic, narrative, and arts-informed perspectives. Los Angeles, CA: Sage Publications Inc.

Cannon, L. W. (1990). Fostering positive race, class, and gender dynamics in the classroom. Women's Studies Quarterly, 18(1/2), 126-134.

Chick, N., & Hassel, H. (2009). “Don’t hate me because I’m virtual”: Feminist pedagogy in the online classroom. Feminist Teacher, 19(3), 195-215.

Gunn, C., McSporran, M., Macleod, H., & French, S. (2003). Dominant or different? Gender issues in computer supported learning. Journal of Asynchronous Learning Networks, 7(1), 14-30.

Johnson-Bailey, J., Baumgartner, L. M., & Bowles, T. A. (2010). Social justice in adult and continuing education: Laboring in the fields of reality and hope. In C. E. Kasworm, A. D. Rose & J. M. Ross-Gordon (Eds.), Handbook of adult and continuing education (pp. 339-349). Los Angeles, CA: Sage Publications, Inc.

Johnson-Bailey, J., & Lee, M.-Y. (2005). Women of color in the academy: Where's our authority in the classroom? Feminist Teacher, 15(2), 111-122.

LeCompte, M. D., & Preissle, J. (1993). Ethnography and qualitative design in educational research (2nd ed.). San Diego, CA: Academic Press, Inc.

Marshall, C., & Rossman, G. B. (2006). Designing qualitative research (4th ed.). Thousand Oaks, CA: Sage Publications, Inc.

Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San Francisco, CA: Jossey-Bass.

Merriam, S. B. (2010). Globalization and the role of adult and continuing education. In C. E. Kasworm, A. D. Rose & J. M. Ross-Gordon (Eds.), Handbook of adult and continuing education (pp. 401-409). Los Angeles, CA: Sage Publications, Inc.

Merriam, S. B. (Ed.). (2002). Qualitative research in practice: Examples for discussion and analysis. San Francisco, CA: Jossey-Bass.

Morse, K. (2003). Does one size fit all? Exploring asynchronous learning in a multicultural environment. Journal of Asynchronous Learning Networks, 7(1), 37-55.

Rogers, P. C., Graham, C. R., & Mayes, C. T. (2007). Cultural competence and instructional design: Exploration research into the delivery of online instruction cross-culturally. Educational Technology Research and Development, 55(2), 197-217.


Sadykova, G., & Dautermann, J. (2009). Crossing cultures and borders in international online distance higher education. Journal of Asynchronous Learning Networks, 13(2), 89-114.

Shea, P., Vickers, J., & Hayes, S. (2010). Online instructional effort measured through the lens of teaching presence in the community of inquiry framework: A re-examination of measures and approach. International Review of Research in Open and Distance Learning, 11(3), 127-154.

Tisdell, E. J. (1998). Poststructural feminist pedagogies: The possibilities and limitations of feminist emancipatory adult learning theory and practice. Adult Education Quarterly, 48(3), 139-156.

Tollman, J. (2003). Classroom teaching in Botswana and online teaching from Georgia: Hard knocks and earned successes. Journal of Education for Library and Information Science, 44(1), 39-57.


jaln guidelines - Online Learning Consortium · grading as a learning assessment method in the MOOC...

Documents

Transcript of jaln guidelines - Online Learning Consortium · grading as a learning assessment method in the MOOC...