Computers in Human Behavior 28 (2012) 405–413

Contents lists available at SciVerse ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier .com/locate /comphumbeh

Developing methods for understanding social behavior in a 3D virtuallearning environment

Matthew Schmidt a,⇑, James M. Laffey b,1, Carla T. Schmidt c,2, Xianhui Wang a,3, Janine Stichter d,4

a School of Information Science and Learning Technologies, University of Missouri, 118 London Hall, Columbia, MO 65211, United Statesb School of Information Science and Learning Technologies, University of Missouri, 303 Townsend Hall, Columbia, MO 65211, United Statesc Juniper Gardens Children’s Project, 444 Minnesota Avenue, Suite 300, Kansas City, KS 66101, United Statesd Department of Special Education, University of Missouri, 303 Townsend Hall, Columbia, MO 65211, United States

a r t i c l e i n f o

Article history:Available online 17 November 2011

Keywords:Three-dimensional virtual learningenvironments (3D VLE)Multi-user virtual learning environments(MUVEs)Autism spectrum disordersAll-views analysisSocial competency curriculum

0747-5632/$ - see front matter � 2011 Elsevier Ltd. Adoi:10.1016/j.chb.2011.10.011

⇑ Corresponding author. Tel.: +1 573 882 2374.E-mail addresses: schmidtma@missouri.edu (M. Sc

(J.M. Laffey), xw7t4@mail.missouri.edu (X. Wa(J. Stichter).

1 Tel.: +1 573 882 5399.2 Tel.: +1 913 321 3143.3 Tel.: +1 573 882 2374.4 Tel.: +1 573 884 9157.

a b s t r a c t

This paper presents a case study of developing and implementing methods to capture, code and compre-hend reciprocal social interactions in a three-dimensional virtual learning environment (3D VLE). Theenvironment, iSocial, is being developed to help youth with autism spectrum disorders (ASD) developsocial competencies. The approach to identifying, classifying and coding behavior in the 3D VLE usesan adaptation of reciprocal interaction coding methods traditionally used in single-subject research withindividuals with ASD. These adaptations consider the unique characteristics of the 3D VLE technology andthe nature and context of learning in this type of environment. A description of the coding methodsemployed is provided. Selected results are presented to illustrate how this methodology can offerdetailed descriptions of learning and social interaction behavior in context. Such results demonstratethe potential of this approach for building new knowledge about how learning takes place and progressesin a 3D VLE and for making data-driven design decisions for improving the learning experience in theonline social context.

� 2011 Elsevier Ltd. All rights reserved.

1. Introduction

The purpose of this article is to describe our case of developing amethodology to study users’ reciprocal interaction behavior in athree-dimensional virtual learning environment (3D VLE). Since2008, we have been designing, developing and testing a 3D VLEcalled iSocial for helping youth with autism spectrum disorders(ASD) acquire social competence. We chose to use 3D VLEtechnology because it has shown promise as a viable instructionaltechnology both in general education (Dalgarno & Lee, 2010; Liv-ingstone, Kemp, & Edgar, 2008; McLellan, 2004) and special educa-tion (Cheng & Ye, 2010; Cobb et al., 2002; Wallace, Parsons,Westbury, White, & Bailey, 2010). In general education, systemssuch as Second Life and Active Worlds have made multi-user vir-tual environments (MUVEs) accessible to educators who wish toemploy them. Applications built using MUVE technology such asQuest Atlantis (Barab, Sadler, Heiselt, Hickey, & Zuiker, 2007; Barab

ll rights reserved.

hmidt), laffeyj@missouri.edung), stichterj@missouri.edu

et al., 2011), River City (Dede, Clarke, Ketelhut, Nelson, & Bowman,2005; Ketelhut, Nelson, Clarke, & Dede, 2010) and EcoMUVE(Metcalf, Dede, Grotzer, & Kamarainen, 2010) are showing the po-tential of 3D VLEs designed as teaching and learning tools. While3D VLE systems have a number of potentially effective learningbenefits such as impacting motivation and engagement, these ben-efits and how to best leverage them are not yet well understood(Dalgarno & Lee, 2010). Great interest exists in using this mediumfor realizing potential learning benefits, but much of the literaturein the field is based on anecdotal data and personal interpretations.Hence, many of the assumptions about potential learning benefitsin 3D VLEs are not empirically supported.

In the field of autism spectrum disorders, 3D VLE systems arealso gaining traction. ASD is a pervasive developmental disorderwhich, according to the American Psychiatric Association’sDiagnostic and Statistical Manual of Mental Disorders, Fourth Edition(1994), is manifested by impairments in social interaction, com-munication impairments and restricted, repetitive and stereotypedpatterns of behavior, interests and activities. Of particular note arethe difficulties individuals with autism spectrum disorder (ASD)experience with social interactions which result in individualswith ASD having a higher likelihood to exhibit problematic socialbehavior and become socially withdrawn (Weiss & Harris, 2001).Failure to address social deficits can can ultimately affect quality-of-life for these individuals (Myles & Simpson, 2002). Hence, iden-

406 M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413

tification of social competence deficits and targeted interventionsdesigned to improve these deficits are critical for increased successand independence (Office of Special Education Programs, 2003). 3DVLEs are a promising technology and provide a number of potentialbenefits for social competence instruction such as predictable taskrepetition, the potential to adapt to the individual and control ofinput stimuli (Parsons & Mitchell, 2002). In addition, 3D VLEs offerthe potential for role-playing in a realistic and compelling environ-ment that may be intrinsically motivating, safe and completelycontrolled (Sansosti & Powell-Smith, 2008).

In a 2002 study (Leonard, Mitchell, & Parsons, 2002), seven par-ticipants with ASD were given the task to find a place to sit in a vir-tual café and a virtual bus. Participants (four male and threefemale, 14–16 years old with full-scale IQ scores between 65 and110, mean verbal IQ of 81.9 and mean performance IQ of 87.1)were able to learn appropriate responses by using the VLE, and re-sponded to feedback from the system and from individual coachingby the researchers. Findings indicate that social competence learn-ing transferred between VLE and video media, but not between thecafé and bus contexts. The authors maintain these findings supportVLE social competence training as a safe, calm alternative to in situsocial competence training. Parsons, Leonard, and Mitchell (2006)performed a case study on two individuals with ASD (both male,one 14 and the other 17 years old, with relative VIQ scores of 70and 91 and FSIQ scores of 73 and 100) to investigate whether par-ticipants relate VE experiences to the real world and whether theyenjoy using the VE and to provide rich examples of participant/facilitator interaction in the VE. Virtual café and bus scenes wereagain used and the task was for participants to find a seat. Partic-ipants reported that they felt the VE had helped them and that theythought it was useful. They appeared to be motivated by the VEand stated that they enjoyed using it. When the participants mademistakes, they would laugh and joke about the mistakes, indicatingthat the social blunders they made may not have been interpretedas being as serious as in real life, nor as anxiety inducing.

While a handful of other studies exist supporting the use of so-cial competence instruction in 3D VLEs for individuals with ASD(e.g., Cheng & Ye, 2010; Mitchell, Parsons, & Leonard, 2007; Par-sons, Mitchell, & Leonard, 2004; Parsons, Mitchell, & Leonard,2005), knowledge in the field is limited and further research isneeded (Cobb et al., 2002; Parsons et al., 2005). Little is knownabout how skills learned in the virtual environment might transferto the real world. Further, the majority of studies dealing with so-cial competence instruction in 3D VLEs focus on single user envi-ronments. These studies do not explore how multiple usersmight experience VE systems simultaneously. However, multi-userVEs might provide a number of benefits over single-user VEs. Forinstance, peer groups could discuss social competence collabora-tively with a facilitator (Cobb et al., 2002) or social norms couldbe negotiated and developed between users (Parsons et al., 2005).

1.1. iSocial: a multi-user 3D VLE for social competence instruction

iSocial is a multi-user 3D VLE built using the Open Wonderlandvirtual worlds toolkit (http://openwonderland.org). The goal of iSo-cial is to develop a multi-user 3D VLE for helping youth with (ASD)acquire social competence. To this end, we have selected a success-ful face-to-face curriculum on social competence for adolescentswith ASD that is typically administered in a clinic-like setting(Stichter et al., 2010) and have transformed the units and lessonsto make them appropriate for delivery in a multi-user 3D VLE.The transformed curriculum includes units on recognizing facialexpressions, sharing ideas, turn taking, recognizing emotions andproblem solving. Instruction takes place in social competencegroups of four to six youth (aged 11–14), facilitated by an online

guide who is a specialist in the curriculum and in working withyouth with ASD.

In iSocial, youth come together online in the 3D VLE where theyare represented by and act through avatars. They communicateand act using audio speech, text chat, gesture and movement inthe virtual worlds. Various virtual worlds such as castles, pirateships and restaurants provide a context for learning and practiceactivities. In these worlds, students are able to perform lessonactivities which promote the discussion of social competence col-laboratively with a facilitator and allow for the negotiation of so-cial norms between users. The project has undergone extensiveusability and usage testing and is currently being field tested inpublic schools. Testing iSocial with youth with ASD sensitized usto the challenges of facilitating and managing behavior so as toprovide a coherent curriculum and social experience and to thecomplexity of studying and understanding behavior in a 3D VLE.We became aware that we needed a way to identify, codify andexamine behavior in our multi-user 3D VLE that would allow usto better understand how students behave socially in the system,how their behavior in the virtual worlds is mediated by the systemand how different lesson activities impact their social behavior.

In the following sections, we describe our case of developing amethodology to study users’ reciprocal interaction behavior iniSocial. While the methods and tools we are developing are some-what customized to youth with ASD developing social competence,we believe they are of interest to developers, researchers and prac-titioners who want to move beyond measures of perceptions, behav-iors and outcomes outside the virtual experience and be able toanalyze the implications of design decisions on student behaviorwithin the 3D VLE in order to make connections between behaviorand learning outcomes. To this end, we present a methodologywhich looks specifically at users’ reciprocal interactions in the 3DVLE, is sensitive to the tools which mediate these interactions andalso to the learning and activity contexts in which these interactionsare couched.

2. Conceptual framework

Individuals with ASD typically desire to be social but do nothave the requisite social competencies to do so successfully, withtheir inability to identify and act on nonverbal social cues and so-cial prompts tending to result in displays of socially unacceptablebehavior (Myles & Simpson, 2002). The iSocial project and under-lying social competency curriculum are designed to help remediatesuch social defecits and foster social interaction. The curricular ap-proach to teaching and learning depends on social interaction andconversation among students and with the online guide. For thisreason, it was necessary for us to develop a system of analysiscapable of measuring and representing social interaction. We iden-tified extant methods for analyzing conversations and their corre-sponding coding schemes such as computer-mediated negotiatedinteraction (Smith, 2003) and conversation analysis (Sacks,Schegloff, & Jefferson, 1974). In addition, we investigated priorwork that seemed similar to ours such as the coding methods usedin the virtual peers project (Arie, Tartaro, & Cassell, 2008) and dis-course analysis methods used in the Quest Atlantis project (Barabet al., 2007). These initial investigations helped us to narrow ourfocus to the framework of reciprocal social interaction (cf. Patter-son & Reid, 1970).

Based on Bandura’s social cognitive theory, reciprocal interac-tions impact human development through a two-way relationshipbetween the person, the environment and their behavior (Merrell& Gimpel, 1998). Successful reciprocal interactions occur when alearner makes and receives approximately the same amount ofpeer initiations and responses (McConnell, Sisson, Cort, & Strain,1991) and are contingent on the context and planned opportunities

M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413 407

to provide them (Roeyers, 1996). As a result, these reciprocal inter-actions assist in the acquisition of new skills and overall socialcompetence (McConnell et al., 1991). In studies that attempt to eli-cit social interaction among individuals with ASD, researchers of-ten look at initiations, responses and continuations as a primaryoutcome measure (Morrison, Kamps, Garcia, & Parker, 2001;Thiemann & Goldstein, 2001).

Most studies looking at reciprocal interaction with individualswith ASD utilize some form of single-subject design, a methodolog-ical design type in which the participant or participants act as thecontrol, as opposed to using a control group. The goal of single-subject research is to demonstrate a functional relationshipbetween an intervention and a target behavior. Single-subjectresearch demonstrates experimental control in the same vein asrandomized control group designs (Horner et al., 2005) and is recog-nized by the US Department of Education as an effective means toevaluate research questions appropriate to the methodology in ap-plied and clinical fields (Kratochwill et al., 2010). While we haveelected to use a reciprocal interaction framework and to borrowfrom single-subject design research methodology, it is importantto clarify that our approach differs from a traditional single-subjectdesign in that we are not looking for an experimental effect. Instead,our methodology considers what reciprocal interaction behaviorsparticipants demonstrate when using iSocial and how those behav-iors might relate to the unique characteristics of the 3D VLE experi-ence and technology features. As a result, we needed not only tomeasure reciprocal interaction behaviors in the iSocial 3D VLE, butalso to identify any associations specific to the medium and to high-light technological activities that might impact the behaviors.Therefore, we adapted the traditional approach of looking at conver-sational initiations, responses and continuations to consider thetechnology characteristics which might impact participants’ inter-actions, as well as the virtual world contexts in which the reciprocalinteractions occur. Our methods section provides an overview of ourprocess for building this methodology.

3. Methods

A field-test of one unit of the iSocial curriculum was undertakenin the Fall of 2008 at a large autism research and treatment centerin the Midwest which yielded our data set. The curricular unit fo-cused on turn-taking in basic conversation, in which students learnthe cues and processes of conversational turn-taking. Lesson activ-ities consisted of the youth having opportunities (1) to learn aboutturn-taking in the 3D VLE through lessons provided by the onlineguide, (2) to try out and practice the skills in both structured andmore naturalistic contexts and (3) to interact in social ways withpeers and the online guide. The unit consisted of four lessonswhich were built in a virtual lighthouse. In each lesson, areas wereprovided in the lighthouse for students to perform lesson activities,including such activities as introduction, modeling, structuredpractice activities and more naturalistic practice activities. In thephysical world, participants sat at a computer with a facilitatorand used microphone-equipped headsets to communicate withmembers in the virtual world. In the virtual world, participantsinteracted with each other and the online guide as avatars withthe ability to speak, gesture, move and act on the environment.

3.1. Participants

Four youth undertook the instruction with assistance by an on-line guide and facilitators who physically sat with them. These fouryouth met a set of pre-defined qualifications for enrollment in thesocial competency curriculum: they (1) were between 11 and14 years old, (2) had a medical diagnosis of autism determined

by the Autism Diagnostic Interview Revised (ADI-R) (Rutter, LeCouteur, & Lord, 2003) and/or the Autism Diagnostic ObservationSchedule (ADOS) (Lord, Rutter, DiLavore, & Risi, 2002) were ver-bal/capable of speech and (4) had an intelligence quotient withinone standard deviation of the mean for the typical range (e.g., ascore of 85–115). Participants were enrolled in a social competencycurriculum using the same curriculum as iSocial. They and theirparents were asked if they were interested in taking part in iSocialtesting, and permission forms that were approved by the campus’institutional review board were completed by each participant andhis parents before taking part in the testing.

3.2. Data collection

Video recordings of participants using the 3D VLE system werecollected both virtually and in the physical world during each ofthe four sessions. iShowU (http://www.shinywhitebox.com/)screen recording software was used to capture participants’ inter-action in the virtual world. Video cameras were set up in the par-ticipants’ physical spaces to capture their interaction with thecomputer and with their physical facilitators. Videos were createdfor each participant in each of the four lessons in the turn-takingunit.

3.3. Coding process

ELAN Linguistic Annotator (http://www.lat-mpi.eu/tools/elan/),an open source software tool that allows the creation, editing andsearching of annotations in video and audio data, was used for cod-ing. ELAN was configured for simultaneous viewing and coding ofmultiple synchronized videos and transcripts (Fig. 1). This madepossible a comprehensive and simultaneous view of all users bothat the computer and within the virtual world, a method we term‘‘all-views analysis’’ (Goggins, Schmidt, Guajardo, & Moore, 2011).

3.4. Inter-observer agreement

Measures of inter-observer agreement (IOA) were taken for re-ciprocal interaction coding. Agreement was assessed using thepoint-by-point agreement method as per Kazdin (1982) and wasobtained for 25% of the video data, as per Cooper and colleagues(2007). IOA for all coding levels was between 80% and 90% for allsessions. Agreement was determined by both observers agreeingthat a behavior had occurred, whereas if one marked the behavioras occurring and the other did not or marked it as a differentbehavior, it was marked as a disagreement. If agreement was lessthan 85%, coders would review the coding scheme and clarify theirunderstanding of codes and then re-code the data independently.Given the significant calibration effort on the part of the codingteam to reach agreement scores that were greater than 85%, weare confident in our agreement metric; however, we also recognizethat more robust methods could provide even more confidence,such as Cohen’s Kappa, which would account for coincidentalagreement. Indeed, we will use the Kappa statistic for assessingagreement in future studies.

3.5. Coding scheme description and history

The reciprocal interaction coding scheme was developed itera-tively by the iSocial team. A review of reciprocal interaction codingschemes (for example, McEvoy et al., 1988; McKenney et al., 2005)served as the basis for development of a provisional reciprocalinteraction coding scheme. We realized that these coding schemeswere not able to represent unique elements of the 3D VLE whichimpact interaction, such as technological tools and the activitycontexts within the virtual space. To better characterize the unique

Fig. 1. A sample ELAN project which includes synchronized video, codes assigned by researchers and an audio transcript.

408 M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413

nature of reciprocal interaction within the 3D VLE, the coding teamrevised the provisional coding scheme to capture reciprocal inter-action on three levels: (1) the context for reciprocal interaction(context level), (2) the reciprocal interaction itself (the interactionmodel level) and (3) the means of interaction (the interactionmode level). These levels are described in Table 1.

3.5.1. Interaction model levelThe interaction model considers initiation, response and contin-

uation behaviors and represents them as socially appropriate orinappropriate. Definitions for the interaction model codes arefound in Table 2. These codes allow us to examine not onlywhether interaction occurred, but also whether that interactionwas socially appropriate.

Table 2Interaction model of the reciprocal interaction coding scheme.

Code Description

Initiation One participant emits a verbal, textual or gestural

3.5.2. Interaction mode levelThe convention of capturing reciprocal interaction behaviors

using initiation, response and continuation was developed primar-ily for face-to-face interactions and did not provide a means to ac-count for the range of interaction possibilities provided by 3D VLEtechnology. For example, avatar interaction can take the form of

Table 1Level definitions for the reciprocal interaction coding scheme.

Coding level Description

Interactionmodel level

Describes reciprocal interactions as a series of initiations,responses and continuations. Allows researchers to see ata glance, for example, if a verbalization (mode ofinteraction) made while discussing a presentation(context) is a conversational initiation, response orcontinuation, and whether the interaction is appropriateor inappropriate to the social context

Interactionmode level

Provides a detailed view of the means (e.g., verbalization,text message, avatar gesture) of reciprocal interaction.Allows researcher to see at a glance, for example, that for agiven duration of time learners’ mode of interactionconsisted primarily of mouse clicks, with someverbalizations and text messages

Context level Describes where in the virtual world interactions areoccurring, the activity in which learners are engaged andwhat technological supports (if any) form the specificcontext for those interactions. Allows researcher to see at aglance, for example, that at a specific time in a lesson,learners were engaged in playing a browser game in thefoyer area of the lighthouse virtual environment

gestures, such as nodding yes or no, but these gestures are initiatedwith a mouse click and not simply by kinesthetic body movement.Participants are also able to move about the virtual space and taketurns making selections or interacting with objects. Consequently,we needed a set of codes to provide a more detailed characteriza-tion of the technological means of interaction within the 3D VLE,i.e., verbalization (via headset), text message (via keyboard entry),avatar gesture (via mouse click), etc. Interaction mode codes anddefinitions are provided in Table 3.

3.5.3. ContextThe context level of coding considers three different contexts: a)

where the participants are interacting in the virtual world (the pri-mary context); what curricular task they are engaged in (the lessoncontext); and what technological tools are enabling the activitythey are undertaking within the curricular task (the affordance

behavior to indicate communication. Coded as aninitiation only if it is not preceded by another interactionfrom the peer(s) or instructor(s) to whom the initiationwas directed within 3 s

Response Acknowledgement of interaction that is executed withinthree seconds by the peer(s), instructor or physicalfacilitator from whom the response was solicited by theinitiator and is on-topic

Continuation Designates the beginning of sustained interaction that ison-topic and happens within three seconds of thepreceding response

Non-response A response or continuation to an interaction is expectedbut the participant fails to acknowledge the interactionin any way within three seconds

Interruption A verbal articulation that occurs when someone else isspeaking or has the floor to speak, regardless ofrelevance to the current conversation/lesson

Overlap An inadvertent or unintentional interruption in whichone person is speaking and another begins speaking in a

way that is not inappropriateInappropriate

initiationAn initiation, response or continuation which: is off-topic; indicates that the participant is not payingattention; uses an inappropriate tone; is rude orinsensitive; is socially unacceptable; or does not followdirections

Inappropriateresponse

Inappropriatecontinuation

Table 3Codes and code descriptions from the interaction mode level of coding.

Code Description

Verbal Used to identify participant verbalizationsGesture Used to identify participant use of available avatar-based

gesturesAction Used to identify mouse clicks on elements within the virtual

world, such as gamesMovement Used to identify avatar movement within the virtual spaceText Used to identify participant use of text chatOther Interaction mode behaviors that do not correspond with

interaction model behaviors. For example, movement that is nota response or unarticulated verbal expression, such as oneparticipant who often was humming during the sessions

M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413 409

context). These three levels are necessary since (1) participants’interactions in the virtual world are completely mediated by tech-nological tools, (2) activities vary across lessons, from highly struc-tured to less structured practice activities, and take place indifferent areas of the virtual world, and (3) participants move theiravatars through the virtual world as lessons progress. The threelevels of context codes represent the virtual world as a set of stagesfor reciprocal interaction, characterized as a complex ecology ofuser behavior and interaction, curricular activities, virtual scenesand technological tools. This ecological perspective of learning iniSocial presents a view of context as being mediated, dynamicand elastic.

4. Application

Using the all-views technique described previously, video datawere prepared for coding by importing them into the ELAN linguis-tic annotator and creating coding projects for each session andeach group. Using the coding manual discussed in the previous sec-tion, coding of the data was split into three phases. Data were

Fig. 2. Sample graph depicting results of reciprocal int

coded at the interaction mode level during the first phase, the con-text level during the second phase and the interaction model dur-ing the third phase. This three phase coding process was amenableto the ELAN software’s workflow and was adopted because coderswere able to code with higher agreement when focusing on onlyone phase of coding at a time. The coded data were imported toMicrosoft Excel and graphed. These graphs provide an aggregateview of reciprocal interactions in context with reciprocal interac-tion rates plotted on the Y-axis, corresponding contexts on the X-axis and percentages of interaction mode codes displayed asstacked 100% columns within each lesson context. These graphsrepresent participants’ coded reciprocal interaction-in-context(Fig. 2). Graphs were created using this method for each partici-pant and for each session.

Because these graphs contained such a large magnitude of data,they were both difficult to read and interpret and were thereforeused as masters for creating derivative graphs to highlight specificportions of the data. Derivative graphs were created by collapsingthe three levels of context into a single context (see Fig. 3). Due tospace constraints, contexts are represented in the graphs as abbre-viations. Definitions for these abbreviations are provided in Table 4.

5. Findings

In applying this methodology to data from our pilot study forthe iSocial research project we found that we were able to repre-sent participant behavior at a very detailed level and analyze thesebehavioral data at a variety of levels of granularity. This helped usto understand how well the youth are meeting and progressing onexpectations for certain forms of social behavior, as well as pin-point the contexts and activities that best promote the types ofbehavior that the social skills curriculum teaches, both at an indi-vidual level and in comparison to one another.

For example, Figs. 4 and 5 show the full set and range of appro-priate behavior for participants FN and UB during lesson two of the

eraction coding for one participant in one session.

Fig. 3. Annotated visualization of interaction model, interaction mode and contextlevel codes charted across a single session.

Fig. 4. Complete aggregate of reciprocal interaction model across contexts forparticipant FN in Lesson 2.

Fig. 5. Complete aggregate of reciprocal interaction model across contexts forparticipant UB in Lesson 2.

410 M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413

turn-taking unit. Based on the objective of the examiner, severalforms of representation could be employed. These two figuresuse the percentage of observed behavior to total opportunitiesfor that behavior to represent interaction model behavior andmode behavior, but another representation could use frequencynumbers. One way to gain insights about the youth behavior andperformance in the curriculum is to look at how different contextsdrew different behaviors. For example examining Fig. 4 shows thatsome activities drew singular forms of behavior such as the practiceactivities (PA) and finishing activities (FA) eliciting only verbalbehavior while other activities such as starting activities (SA) elic-ited multiple forms of behavior (talk, movement and gestures).Also, comparisons of youth provide insight into how responseand behavior in the system may vary across participants. Forexample, examining initiations shows for FN that initiating inter-action rose and fell from activity to activity but often reached40% of the total opportunity for behaviors in a session, whereasfor UB the level of initiation remained mostly lower than thisacross the session.

Across all participants and all lessons, we found that the domi-nant interaction model behavior in iSocial was response. The meanpercentage of response across participants and across lessons was55.65%. Initiation was the least dominant interaction model behav-ior, with a mean percentage across participants and lessons of20.99%. For some participants (F.N., U.B.), initiations were on aver-age higher, and for others (Q.C., U.Z.), continuations were on aver-age higher. Across participants and lessons, the most dominantform of inappropriate behavior was interruption, with a mean of9.33% of all interaction model codes. Breakdowns of average per-centages of interaction model behaviors are provided in Table 5.

Table 4Definitions for lesson context abbreviations.

SA IS MS

Starting activities Introduction of the skill Modeling of the skill

The dominant mode of interaction was verbalization. The meanpercentage of verbalization across participants and across lessonswas 51.48%, followed by movement at 7.62%, gesture 5.95% and ac-tion at 3.64%. Text was the least dominant interaction mode, with amean percentage across participants and lessons of 0.66%. Break-downs of average percentages of interaction mode codes are pro-vided in Table 6.

To summarize, the dominant appropriate reciprocal interactionbehavior was response, the dominant inappropriate reciprocalinteraction behavior was interruption and the dominant mode of

VP PA FA

Verbal practice Practice activities Finishing activities

Table 5Percentile means of initiations, responses, continuations and interruptions among participants across all four lessons of the turn-taking unit.

Participant Initiation (%) Response (%) Continuation (%) Interruption (%)

F.N. 26.19 51.03 21.83 4.58U.B. 29.09 45.25 17.38 21.08Q.C. 8.82 72.01 19.17 2.76U.Z. 19.85 54.31 25.84 8.92

Table 6Mean percentages of interaction mode codes for all four lessons of the turn-taking unit.

Participant Verbalization (%) Gesture (%) Action (%) Movement (%) Text (%)

F.N. 49.41 3.40 5.97 3.91 0.99U.B. 48.13 5.23 6.20 6.73 0.46Q.C. 47.52 5.35 1.77 14.62 0.00U.Z. 60.86 9.81 0.62 5.22 1.17

M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413 411

interaction was verbalization. Referring the reader again to Figs. 4and 5, we are able to see these trends represented visually in thegraphs. For both participant FN and UB, response is clearly repre-sented as the dominant reciprocal interaction across nearly all con-texts. Similarly, verbalization dominates all other interaction modebehaviors across all contexts.

6. Discussion and conclusions

Our key objective for establishing the methods presented in thisreport has been the thorough description and representation of re-ciprocal social interaction behaviors in iSocial. This approachseems to be a good fit for researching small group interaction incollaborative 3D VLEs. Not only does this approach take into ac-count prior methods of coding and analyzing reciprocal interactionin the real world, it also accounts for characteristics and affor-dances unique to the 3D VLE. Our report describes an approachto collecting and representing discrete behaviors of individualswithin activity and learning contexts. These representations offerperspectives on studying single subjects in learning technologycontexts that fit with the formative nature of testing an evolvingsystem and address the social nature of the learning context. Thiscan be helpful in analyzing implementation processes in the 3DVLE and iterating theory and design (Wang & Hannafin, 2005),which can in turn help to systematically improve practices and fea-tures to best support the teaching and learning processes, (Reeves,2006). Researchers who desire to establish a gestalt representationof the ecology of small group learning and interaction in 3D VLEsare likely to benefit from similar methods.

Representations such as Figs. 4 and 5 and Table 6 illustrate thepotential of our methodology to examine reciprocal interactionbehaviors in context and how conclusions gleaned from such rep-resentations can be used to improve design. Information fromthese types of representations could be used to answer such ques-tions as whether performance matches expectations for competentbehavior, or whether skills develop as youth progress through les-sons, which could in turn provide guidance for decisions aboutwhether activities or activity spaces could be improved to enhanceinteraction or performance. For example, Fig. 4 shows participantFN’s rates of appropriate response to total opportunities averagedaround 45% across lesson activities, and rates of appropriate con-tinuations were quite variable, averaging around 25%. Under idealcircumstances we would hope that the youth would improve per-formance as they moved from early activities to later activities.However, the data do not indicate a trajectory of improved perfor-mance, which could reflect some differences across the activities or

needs for improvement within the design of the activities. Theserates do, however, show us that youth can participate by respond-ing and continuing interactions in these contexts, although onemight hope for higher rates of performance. While the data as rep-resented in Figs. 4 and 5 do not provide a sufficient answer fordecisions of what to improve or whether the system is satisfacto-rily meeting learning objectives, they do indicate that improve-ments can be made. In the case of these lesson activities, furtherdesign is called for so that data from usage of the next versionscan be compared to these figures for a test of progress.

A limitation of our approach, however, is that the representa-tions presented in this report do not represent the sequence of par-ticipant interactions through a lesson. However, the sequence ofbehaviors can be very informative. For instance, an interactionwith several continuations back and forth between the youthmight indicate a desirable turn taking incident. In turn we couldlook at the context to see what activities or design features wereassociated with that form of interaction. Other sequences of inter-est might be an initiation without any response, a series of initia-tions or a series of initiation and response patterns that do notresult in continuations. Recognizing the value of such data repre-sentations, we have begun to investigate more rigorous meansfor accounting for activities in context and how the behavior ofmultiple actors might elicit reciprocal behaviors.

As the iSocial project moves forward we note several ways thatwe and other researchers can advance the basic methods we havedescribed in this report. First, we can envision extensions of thecoding system or new systems to address lessons with differentobjectives. For example a lesson that includes problem solvingmay want to include codes for when the youth propose or providea solution (Hou, Chang, & Sung, 2008) or a lesson that focuses onknowledge construction may want to include codes for sharingand negotiation (Gunawardena, Lowe, & Anderson, 1997). Second,we are investigating and developing ways of having the computerprovide some forms of learning analytics that can either standalone as behavioral representations or be used by the manual co-der in making decisions about codes (Hasler, 2010). Third, wecould identify high value or high interest sequences of behavior,such as a response followed by three appropriate continuationsor an initiation without any response, and then search the dataset for all occurrences. Fourth, we are looking into more advancedways to represent user behavior, such as using transitional statediagrams to investigate interaction patterns (Jeong, 2005) and so-cial network analysis (de Laat, Lally, & Lipponen, 2007). As a finalnext step we can look for associations between levels of behaviorsand other measures. Do youth with certain characteristics partici-pate in ways that are different from other youth? Is a high level of

412 M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413

reciprocal interaction associated with other measures of positiveoutcomes? Do environments with more structure or less structurelead to different levels of reciprocal interaction? Can we designspecial features into the 3D environment to invite or constrain cer-tain forms of behavior, and do they work?

As researchers, designers, developers and implementers of 3DVLE there is much to learn and discover about 3D VLE and its po-tential as an educational resource. We believe that a focus andadvancement for how we study the behaviors and processesundertaken in these systems is critical. The methods described inthis report attempt to address the complexity of learning whenmediated through a 3D VLE and using avatars as representativeof participants and when the learning behavior is dynamic and so-cial with an online guide and with other youth. The tools for cap-turing the data, the protocols and tools for coding the data andthe representations we use to comprehend the data have provenhelpful in our investigation as well as pointing the way to improve-ments in methods and advancement in our understanding of learn-ing in 3D VLE.

Acknowledgements

This research was made possible by the generous contributionsof the Thompson Center for Autism and Neurodevelopmental Dis-orders and a grant from our university research board.

References

American Psychiatric Association (1994). Diagnostic and statistical manual of mentaldisorders. Washington, DC: American Psychiatric Association.

Arie, M., Tartaro, A., & Cassell, J. (2008). Conversational turn-taking in children withautism: Deconstructing reciprocity into specific turn-taking behaviors. London:International Meeting for Autism Research.

Barab, S. A., Sadler, T., Heiselt, C., Hickey, D., & Zuiker, S. (2007). Relating narrative,inquiry, and inscriptions: A framework for socio-scientific inquiry. Journal ofScience Education and Technology, 16(1), 59–82.

Barab, S., Gresalfi, M., Ingram-Noble, A., Jameson, E., Hickey, D., Akram, S., et al.(2011). Transformational play and virtual worlds: Worked examples from thequest Atlantis project. International Journal of Learning and Media, 1(2).doi:10.1162/ijlm.2009.0023.

Cheng, Y., & Ye, J. (2010). Exploring the social competence of students with autismspectrum conditions in a collaborative virtual learning environment – The pilotstudy. Computers & Education, 54(4), 1068–1077.

Cobb, S., Beardon, L., Eastgate, R., Glover, T., Kerr, S., Neale, H., et al. (2002). Appliedvirtual environments to support learning of social interaction skills in userswith Asperger’s Syndrome. Digital Creativity, 13(1), 11–22.

Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis. PrenticeHall: Pearson/Merrill.

Dalgarno, B., & Lee, M. J. W. (2010). What are the learning affordances of 3-D virtualenvironments? British Journal of Educational Technology, 41(1), 10–32.doi:10.1111/j.1467-8535.2009.01038.x.

de Laat, M., Lally, V., & Lipponen, L. (2007). Investigating patterns of interaction innetworked learning and computer-supported collaborative learning: A role forsocial network analysis. International Journal of Computer – SupportedCollaborative Learning, 2, 87–103.

Dede, C., Clarke, J., Ketelhut, D. J., Nelson, B., & Bowman, C. (2005). Students’motivation and learning of science in a multi-user virtual environment. In Paperpresented at the annual meeting of the American Educational Research Association,Montréal, Québec.

Goggins, S., Schmidt, M., Guajardo, J., & Moore, J. (2011). 3D virtual worlds:Assessing the experience and informing design. International Journal of Socialand Organizational Dynamics in IT, 1(1), 30–48.

Gunawardena, C., Lowe, C., & Anderson, T. (1997). Analysis of global online debateand the development of an interaction analysis model for examining socialconstruction of knowledge in computer conferencing. Journal of EducationalComputing Research, 17(4), 397–431.

Hasler, B. S. (2010). Report on research and evaluation. In T. Buecheler, N. Labhart, &B. Rowland (Eds.), The ShanghAI Lectures 2009: How to conduct a globaltechnology-supported academic lecture (pp. 61–81). Zurich: ArtificialIntelligence Laboratory, University of Zurich.

Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The useof single-subject research to identify evidence-based practice in specialeducation. Exceptional Children, 71(2), 165–179.

Hou, H.-T., Chang, K.-E., & Sung, Y.-T. (2008). Analysis of problem-solving-basedonline asynchronous discussion pattern. Educational Technology & Society, 11(1),17–28.

Jeong, A. (2005). A guide to analyzing message-response sequences and groupinteraction patterns in computer-mediated communication. Distance Education,26(3), 367–383.

Kazdin, A. E. (1982). Single-case research designs. New York: Oxford University Press.Ketelhut, D. J., Nelson, B. C., Clarke, J., & Dede, C. (2010). A multi-user virtual

environment for building and assessing higher order inquiry skills in science.British Journal of Educational Technology, 41(1), 56–68.

Kratochwill, T., Hitchcock, J., Horner, R., Levin, J., Odom, S., Rindskopf, D., & Shadish,W. (2010). Single-case designs technical documentation. What WorksClearinghouse, 34.

Leonard, A., Mitchell, P., & Parsons, S. (2002). Finding a place to sit: A preliminaryinvestigation into the effectiveness of virtual environments for social skillstraining for people with autistic spectrum disorders. In Proceedings of the 4thinternational conference on disability, virtual reality and associated technologies,Veszprém, Hungary, pp. 249–257.

Livingstone, D., Kemp, J., & Edgar, E. (2008). From multi-user virtual environment to3D virtual learning environment. Association for Learning Technology Journal,16(3), 139–150.

Lord, C., Rutter, M., DiLavore, P. C., & Risi, S. (2002). Autism diagnostic observationschedule. Los Angeles, CA: Western Psychological Services.

McConnell, S. R., Sisson, L. A., Cort, C. A., & Strain, P. S. (1991). Effects of social skillstraining and contingency management on reciprocal interaction of preschoolchildren with behavioral handicaps. The Journal of Special Education, 24,473–495.

McEvoy, M., Nordquist, V. M., Twardosz, S., Heckaman, K. A., Wehby, J. H., & Denny,R. K. (1988). Promoting autistic childrens peer interaction in an integrated earlychildhood setting using affection activities. Journal of Applied BehavioralAnalysis, 21, 193–200.

McKenney, E. L. W., Asmus, J. M., Conroy, M. A., Sellers, J. A., Sloman, G. M., & Nakao,T. (2005). Project GATORSS: A comprehensive model for assessing and treatingsocial skills in children with autism spectrum disorders. In Association forbehavior analysis conference, Chicago, IL.

McLellan, H. (2004). Virtual realities. Handbook of research on educationalcommunications and technology (pp. 461–498). Mahwah, NJ: LawrenceErlbaum Associates.

Merrell, K. W., & Gimpel, G. A. (1998). Social skills of children and adolescents:Conceptualization, assessment, and treatment. London: Lawrence ErlbaumAssociates.

Metcalf, S., Dede, C., Grotzer, T., & Kamarainen, A. (2010). EcoMUVE: Design ofvirtual environments to address science learning goals. In Presented at theAmerican Educational Research Association (AERA) conference, Denver, CO.

Mitchell, P., Parsons, S., & Leonard, A. (2007). Using virtual environments forteaching social understanding to 6 adolescents with autistic spectrumdisorders. Journal of Autism and Developmental Disorders, 37(3),589–600.

Morrison, L., Kamps, D., Garcia, J., & Parker, D. (2001). Peer mediation andmonitoring strategies to improve initiations and social skills for students withautism. Journal of Positive Behavior Interventions, 3(4), 237.

Myles, B., & Simpson, R. (2002). Asperger Syndrome: An overview of characteristics.Focus on Autism and Other Developmental Disabilities, 17, 132–137.

Office of Special Education Programs (2003). Twenty-fifth annual report to congresson the implementation of the individuals with disabilities education act.Washington, DC: US Department of Education.

Parsons, S., & Mitchell, P. (2002). The potential of virtual reality in social skillstraining for people with autistic spectrum disorders. Journal of IntellectualDisability Research, 46(5), 430–443.

Parsons, S., Leonard, A., & Mitchell, P. (2006). Virtual environments for social skillstraining: Comments from two adolescents with autistic spectrum disorder.Computers & Education, 47(2), 186–206.

Parsons, S., Mitchell, P., & Leonard, A. (2004). The use and understanding of virtualenvironments by adolescents with autistic spectrum disorders. Journal of Autismand Developmental Disorders, 34(4), 449–466.

Parsons, S., Mitchell, P., & Leonard, A. (2005). Do adolescents with autistic spectrumdisorders adhere to social conventions in virtual environments? Autism, 9(1),95.

Patterson, G. R., & Reid, J. B. (1970). Reciprocity and coercion: Two facets of socialsystems. Behavior modification in clinical psychology (pp. 133–177). New York:Appleton.

Reeves, T. C. (2006). Design research from a technology perspective. EducationalDesign Research, 1(3), 52–66.

Roeyers, H. (1996). The influence of nonhandicapped peers on the socialinteractions of children with a pervasive developmental disorder. Journal ofAutism and Developmental Disorders, 26(3), 303–320.

Rutter, M., Le Couteur, A., & Lord, C. (2003). ADI-R: Autism diagnostic interview –revised: Manual. Los Angeles: Western Psychological Services.

Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics forthe organization of turn-taking for conversation. Language, 50(4),696–735.

Sansosti, F. J., & Powell-Smith, K. A. (2008). Using computer-presented social storiesand video models to increase the social communication skills of children withhigh-functioning autism spectrum disorders. Journal of Positive BehaviorInterventions, 10(3), 162–178.

Smith, B. (2003). Computer-mediated negotiated interaction: An expanded model.Modern Language Journal, 87(1), 38–57.

Stichter, J., Herzog, M. J., Visovsky, K., Schmidt, C., Randolph, J., Schultz, T., et al.(2010). Social competence intervention for youth with Asperger Syndrome and

M. Schmidt et al. / Computers in Human Behavior 28 (2012) 405–413 413

high-functioning autism: An initial investigation. Journal of Autism andDevelopmental Disorders, 40(9), 1067–1079.

Thiemann, K. S., & Goldstein, H. (2001). Social stories, written text cues, and videofeedback: Effects on social communication of children with autism. Journal ofApplied Behavior Analysis, 34(4), 425.

Wallace, S., Parsons, S., Westbury, A., White, K., & Bailey, A. (2010). Sense ofpresence and atypical social judgments in immersive virtual reality: Responses

of adolescents with Autistic Spectrum Disorders. Autism: International Journal ofResearch and Practice, 14(3), 199–213.

Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhancedlearning environments. Educational Technology Research and Development, 53(4),5–23.

Weiss, M. J., & Harris, S. L. (2001). Teaching social skills to people with autism.Behavior Modification, 25(5), 785–802.