A Review of Research Methods for Assessing Content of Computer-Mediated Discussion Forums

Marra, Rose. (2006). A Review of Research Methods for Assessing Content of Computer-Mediated Discussion Forums. Journal of Interactive Learning _Research,17(3),243-268.

Reporter ： Chia-Yen FengDate ： 2007/12/08

1

Introduction (1/6) Instructors and students rely on these

asynchronous forums to engage one another in ways that replace face-to-face communication.

The goal of such interactions is to promote critical thinking, meaningful problem solving, and knowledge construction (Kirschner, Strijbos, Kreijns, & Beers, 2004).

In order to assess any meaning that results from these discussions, it is necessary to perform some kind of semantic analysis of them.

This article provides an overview of current methods of quantitative and qualitative research paradigms for analyzing the content of asynchronous computer-mediated discussion forums.

2

Introduction (2/6) In discussing online learning

Interactivity as the most striking characteristic of CMC and the factor with the greatest potential to impact learning.

Web-based learning courses must rely on online discussion forums to create these interactions (Similarly, Garrison, Anderson, and Archer ,2001).

In CMC, participants who were trying to increase communication efficiency and decrease typing requirements also decreased the use of unneeded elaborative statements and repetitions(Condon and Cech , 1996) .

3

Introduction (3/6)

CMC has the ability to promote knowledge construction and meaningful learning.

Asynchronous CMC may promote knowledge construction because online environments emphasize student-centered interaction through collaboration with other students .

It is through these interactions that learners construct their knowledge and can learn more effectively (Romiszowski & Mason, 2004).

4

Introduction (5/6)

The quality of online postings by focusing on four dimensions – social (e.g., "I'm feeling great today“) interactive (statements that refer to other

postings) metacognitive (statements about reasoning) Cognitive (five types of reasoning skills)

elementary clarification in-depth clarification Inference Judgment strategies

(Henri's work ,1992) 5

Introduction (6/6)

Henri's model Defines not only the types of skills and

interactions demonstrated in online postings, but also attempts to qualitatively define the nature and content of online interactions that evidence cognitive development and meaningful learning.

Henri's model has provided researchers with a structure for many ensuing qualitative analysis methods.

6

CMC discussion research methods (1/6)

Quantitative - Descriptive Statistics “computer-generated statistics about logons,

messages sent and read, levels of participation and number and length of entries” (Mason ,1992) .

Many course management software provide access to quantitative data easily and allow instructors to track student logons and their usage of course materials.

Scaffolded discussion forums accompanied with more sophisticated tools for

analyzing the quantitative and qualitative nature of those contributions. 7

CMC discussion research methods(2/6)

Mason (1992) lists the number of messages sent and read, levels of participation, and the number and length of entries for each discussion forum. number of postings per student for overall forum; number of times each posting is read; total depth per thread; postings per student listed by depth of postings

the number of postings at depth level 1 - n where n is the number of the most embedded posting(ex. Figure 1).

The metrics are directed towards threaded discussion forums - that is an online dialog or conversation that takes the form of a series of hierarchically linked messages.

8

9

CMC discussion research methods (3/6)

Qualitative Content Analysis two CMC discussion forum analysis protocols

Interaction Analysis Model (IAM) (Gunawardena, Lowe, & Anderson, 1997) .

Critical thinking model(Newman, Webb and Cochrane's ,1996; Newman, Webb, Johnson, & Cochrane, 1997).

Interaction Analysis Model (IAM) social presence, teacher presence, and cognitive

presence （ Garrison et al. , 2000) . attempt to further understand and describe the

processes of negotiating meaning and knowledge co-construction in a collaborative online discussion environment (Gunawardena et al., 1997).

10

CMC discussion research methods (4/6) Phase 1,2

provide further detail on points that have already been made in prior postings

Phase 3 negotiation of meaning or knowledge co-

construction, where students build on prior postings in order to construct the new idea

11

CMC discussion research methods (5/6)

Critical thinking model (CT) 1) identification, 2) definition, 3) exploration, 4)

evaluate

12

CMC discussion research methods (6/6)

“Critical thinking ratio" (CT) model The Newman et al. (1996) based on the frequencies of plus (+) or minus (-) codes

for each letter category. CT = (x+ - x-) / (x+ + x-)

User-Classified Forum Content - Constrained CMC Forums designed to encourage certain types of thinking by the

participants. The most commonly used constraint system supports argumentation, however, a limitless combination of rhetorical structures are possible.

The forum participants used message types that corresponded to a formal argument in order to develop argumentation skills that were hypothesized to enable solving ill-structured problems (Oh, 2004).

13

14

CMC discussion research methods (7/)

constrained discussion Message types included

hypothesizing cause ("What is the cause?“)solution generation ("What should I do about it?“)

research evidence ("Research shows..." or, "A scholar says...“)

experiential evidence ("My experience is..." or, "I believe...“)

elaboration ("Your explanation is unclear; can you be more specific?“)

rebuttal ("I don't agree because...")15

CMC discussion research methods (8/)

Discussion Analysis Tool (DAT)( Jeong, 2003) help researchers examine and measure

student interactions and sequences of interactions that may lead to critical thinking outcomes in threaded discussions

the probabilities generated by DAT can be converted into graphical depictions that provide what may be a useful visual representation of student interactions.

16

17

Methods (1/4) Research Questions and Results

18

Methods (2/4)

Reliability an indication of consistency of measures across time. To ensure reliability, researchers should re-check

calculations or when using automated tools, check for the accuracy of the input data.

Validity The validity of these metrics is less well defined.

Validity refers to whether a measure, in fact, measures what it purports to.

These metrics are not open to much interpretation and as long as researchers do not attempt to attribute more meaning to them than they have, then the metrics are valid (face value , the total number postings).

19

Methods (3/4) IAM

The IAM's validity hinges on whether the results produced when the protocol is reliably applied are in fact an indication of the degree to which knowledge construction occurred in the forum.

CT Although the CT ratios are easy to calculate, one

must question how to meaningfully interpret them.

Constrained forums The reliability and validity issues associated with

the user categorized constrained forums center around the fact that the users themselves do the coding or labeling of the messages. Reliability is largely a question of consistency of results.

20

Methods (4/4) Constrained forums

The validity of the user categorizations in constrained discussion forums depends on how accurately users label their messages.

Another validity issue for these constrained forums concerns the content validity or meaning of the suite of message types.

DAT For DAT because it is algorithmic, the reliability

can be assumed to be nearly perfect - assuming that the underlying programming is sound and that the same data is provided as input for the tool.

The validity of DAT is directly tied to the validity of the coding scheme that provides the data on which DAT operates.

21

Conclusion (1/2) Each method is appropriate for addressing

different types of research questions. Further, each method offers its own unique strengths and weaknesses.

Sequential analysis methods(DAT) builds on content analysis methods and provides

quantitative data on discussion interaction sequences.

the software program DAT are completely reliable and the method has the potential to answer more complex questions regarding discussion boards, the method implicitly inherits all the reliability and validity issues of the content analysis protocol that it uses to do the initial coding.

22

Conclusion (2/2)

Researchers need to choose an analysis methods based upon what questions they wish to answer ability to carry out the chosen methodology in the

fashion intended (e.g., does my research team have the qualitative methods expertise to conduct a content analysis?).

Although none of the methods are perfect in terms of their applicability to all situations or in terms of their validity and reliability, all the methods reviewed - when applied appropriately - can provide researchers with effective tools for describing online discussions.

23