Contents lists available at ScienceDirect

Entertainment Computing

journal homepage: www.elsevier.com/locate/entcom

A personality-based emotional model for embodied conversational agents:Effects on perceived social presence and game experience of usersPejman Sajjadia,b,⁎, Laura Hoffmanna, Philipp Cimianoa, Stefan Koppaa CITEC, Bielefeld University, 33619 Bielefeld, Germanyb PennSylvania State University, 16803, State College, USA

A R T I C L E I N F O

Keywords:Embodied conversational agentVirtual realityGame experienceSocial presencePersonalityBehavioral model

2019 MSC:00-1099-00

A B S T R A C T

This paper reports on an experiment that investigates the effect of interacting with a personality-driven em-bodied conversational agent (ECA) on the perceived social presence and game experience of people in a VRsocial simulator. Furthermore, the dynamics between the different metrics of game experience and social pre-sence of people are explored to determine which game experience metrics are the strongest predictors of per-ceived social presence in this context. A personality-based emotional model is used for personifying the em-ployed ECA, which governs the manifestation of its non-verbal behaviors. Three experimental conditionsmanipulating the existence and intensity of non-verbal behaviors exhibited by the ECA were used to investigatethe effect of this proposed approach. The results of the experiment with 41 participants indicate that people whowere exposed to an extrovert ECA experienced significantly higher levels of behavioral involvement as part oftheir social presence compared to the other conditions. These results suggest that incorporating personality bymeans of non-verbal behavior in the emotional model of an ECA influences users perceived feeling of socialpresence. Furthermore, our results reveal that there is a bidirectional relationship between game experiencemetrics and perceived social presence of people as each predict the other.

1. Introduction

Over the past years we have witnessed an increasing interest in theuse of interactive and immersive applications such as games [1] andsimulations for serious purposes (e.g. social skill training [2], physicalrehabilitation [3], logical problem solving [4]). In contexts where socialinteractions are considered as an important essence of learning, socialentities generally referred to as embodied conversational agents (ECA)[5] capable of imitating the behavior of real people are crafted andutilized by these systems. ECAs facilitate social interactions to takeplace with users through verbal, non- and para-verbal behavioral cues(e.g. [6]). As learning in these systems occur through socialization withan ECA, perceiving the social presence (a feeling of sensing anotherentity being present) of the interlocutor (i.e. the ECA) is of key im-portance. Research has shown that there is a correlation between theperceived social presence and learning where higher levels of perceivedsocial presence in online education lead to higher levels of perceivedlearning and satisfaction with the instructor [7]. Social presence can bestudied in light of constructs such as immediacy (psychological distance)and intimacy (interpersonal closeness) [8]. These experiences are highly

dependent on the degree and quality of verbal, non-verbal, and para-verbal behaviors demonstrated by the ECA, as well as the game/simu-lation experiences of users (e.g. immersion, flow, positive and negativeaffects). These ECA behaviors provide unobtrusive, and yet extremelyexpressive feedback to the users about the effect of their actions [9].Such feedback could evoke a sense of empathy towards the ECA, elicit asense of behavioral dependency, and drive the learners to adjust theiractions. In educational contexts, with an objective of behavior adjust-ment or improvement, this is highly desired and at the core of thelearning strategy.

Extensive argumentation can be found in the literature highlightingthe importance of good game experience, particularly in light of metricssuch as immersion and flow for effective learning (e.g. [10–12]). Fur-thermore, it has been suggested that the immersiveness of an experi-ence actually provides the boundaries within which presence (thefeeling of being there) can occur [13]. As such, in serious games andsocial simulations where learners have to actively engage in social in-teraction with ECAs, it is highly desirable to evoke both: good gameexperience and high levels of perceived social presence to facilitateeffective learning. In light of this, we have opted to investigate whether

https://doi.org/10.1016/j.entcom.2019.100313Received 10 January 2019; Received in revised form 11 March 2019; Accepted 25 July 2019

⁎ Corresponding author.E-mail addresses: sfs5919@psu.edu (P. Sajjadi), lahoffmann@techfak.uni-bielefeld.de (L. Hoffmann), cimiano@techfak.uni-bielefeld.de (P. Cimiano),

skoppg@techfak.uni-bielefeld.de (S. Kopp).

Entertainment Computing 32 (2019) 100313

Available online 26 July 20191875-9521/ © 2019 Elsevier B.V. All rights reserved.

T

the perceived social presence and game experience of people can bepositively affected in a social simulation environment perceivedthrough virtual reality (VR), by employing an ECA that expresses itsemotional states through non-verbal behavioral cues governed by itspersonality. Similar research works have investigated the effect ofpersonified ECAs with different characteristics on the perceived socialpresence of people. However, investigating the effect of manipulatingthe behavioral model of an ECA based on personality, and its con-secutive influence on social presence and game experience has not beensufficiently addressed. Furthermore, although the literature suggeststhat immersion is an influential metric that informs the elicitation ofpresence, it is not clear whether the same principle applies for socialpresence. As such, it is important to understand what game experiencemetrics are the strongest contributing factors to the feeling of perceivedsocial presence. Such insight can inform and improve the design offuture virtual social experiences.

On these grounds, this paper introduces a multi-modal prototypethat utilizes a personality-driven ECA equipped with verbal and non-verbal behaviors. Moreover, it presents the results of a pilot study in-vestigating its effects on the perceived social presence and game ex-perience of users, as well as the dynamics between them. In a pilotstudy with 41 participants, the existence and intensity of the non-verbalbehaviors of the used ECA was manipulated in three conditions to as-sess its effect on the reported game experience and perceived socialpresence of users: (1) no apparent non-verbal behaviors exhibited bythe ECA (2) non-verbal behaviors governed by an extrovert-basedemotional model, in which the exhibitions by the ECA are more as-sertive and pronounced (3) non-verbal behaviors governed by an in-trovert-based emotional model, in which the exhibitions are moresubmissive and minimal. Our results suggest that the projection of non-verbal behaviors as a result of incorporating personality as part of theemotional model of an ECA, influence the elicitation of the feeling ofsocial presence from the users. More precisely, the results suggest thatmore assertive and pronounced non-verbal behaviors produced by anECA are more noticeable by users and seem to have higher impacts ontheir feeling of social presence, compared to more submissive andminimal non-verbal behaviors. Furthermore, our results show thatamong the measured game experience metrics, the experienced levelsof flow, immersion, and tension are the strongest predictors for theperceived feeling of social presence with flow as the strongest predictor.These results suggest that the aesthetics and richness of the experience,in addition to minimal experienced levels of tension (although sig-nificantly influential), are surpassed by a feeling of losing track of timeand being occupied and engaged with activities involving interactionwith a human-like entity in relation to experiencing social presence.

The rest of this paper is organized as follows: Section 2 discussesrelated works on the effect of computer-mediated artifacts (mostly witha focus on ECAs) on perceived social presence, as well as the role ofgame experience in effective learning. Section 3 discusses the under-lying model of the used ECA, and Section 4 elaborates on the used VRsimulation environment. Section 5 discusses the experiment with theprototype, and Section 6 concludes the paper while addressing limita-tions and future work.

2. Background & related work

ECAs have been a topic of interest in the human-agent interactionresearch community for decades. Extensive works can be found in theliterature on the use of ECAs in social interaction scenarios for seriouspurposes. For instance, in the work of Hoque et al. social agents areused as interlocutors to train individuals for job interviews [14], and inthe work of Linssen et al. they are used as crime suspects for trainingpolice officers on how to perform effective interrogation [2]. Dependingon the objective and context of use, ECAs are personified based onvarious human-like characteristics, such as personality, mood, emotion,moodiness (e.g. temperamental vs. lethargic), and motivation (e.g.

[15–18]), and manifested as an interlocutor on different levels of rea-lism. Numerous research projects have focused on designing behavioralmodels that exert personified ECAs (e.g. [17,15]). The term personifi-cation is generally used to refer to the process of giving human-likequalities to an agent, and can be studied from the facets: physical, ex-pressional, logical, and emotional [17].

One of the frequently researched aspects of human-agent/robotinteraction is the level of perceived social presence by humans wheninteracting with these entities. This special form of presence (“the feelingof being located in a perceptible external world around the self”[19], page39), the so called “social presence” has been defined differently byvarious researchers (e.g. [20,21]). One of the most frequently citeddefinitions for this is by Biocca and Harms as “having some level ofaccess or insight into someone else’s affective, cognitive, and inten-tional states” [22]. In this paper, we subscribe to this definition for thenotion of social presence. Furthermore, extensive argumentation can befound in the literature highlighting the importance of virtual reality ineliciting social presence (e.g. [23]). Being exposed to an inherentlyimmersive experience in which one experiences the non- and para-verbal behaviors of a virtual counterpart from a natural viewpoint e.g.,though VR glasses, is one reason to sustain this claim.

Different factors (e.g. human controlled avatars vs. autonomousagents, degree of personification, and realism of non-verbal and para-verbal signals) may contribute to the perceived social presence of aperson when interacting with an ECA. For instance, in the thresholdmodel of social influence proposed by Blascovich et al. [23], it is arguedthat if users are aware of the fact that they are interacting with a humancontrolled avatar, the behavioral realism of the avatar is of less im-portance to cause social influence on the user. This claim is based on theassumption that this awareness already leads to a high level of socialpresence. Conversely, an autonomous agent is low on the agency axis,and thus more behavioral realism in the appearance is needed to evokethe same social influence than an avatar with low behavioral realism.Notwithstanding, von der Pütten et al. [24] showed that such aware-ness of interacting with an autonomous agent versus a human-con-trolled avatar does not affect the perceived social presence of users.When participants were led to believe that they were interacting witheither an autonomous agent or a human controlled avatar in a socialsituation (although both conditions were using the same agent and noavatar), no significant differences between the self-reported measuresof perceived social presence between the two conditions were observed.

With respect to the effect of personification level exhibited by anagent, Nowak and Biocca [25] have demonstrated that higher levels ofanthropomorphism does not by default lead to a higher sense of per-ceived social presence. It was reported that participants who were ex-posed to a less anthropomorphic artifact had experienced a higher levelof social presence than those who were exposed to either no artifact, ora highly anthropomorphic one. This finding was interpreted as highlevels of anthropomorphism increasing the expectation of the people,and reducing their social presence when the expectations are not met.Von der Pütten and colleagues, [24] however, offer an alternativeperspective. An experiment on the effect of behavioral realism fidelityexhibited by an ECA on the social presence of the participants showedto be significant. When participants were exposed to two versions of anECA, one with rich non-verbal behavior signals and one with less richersignals, the self-reported social presence was significantly higher in therich condition. On similar grounds, Chuah et al. [26] state that in-creasing the physicality level of the ECA increases the perception ofsocial presence by users. Furthermore, Daher et al. [27] reported thatobserving two intelligent virtual agents (IVAs) interact with each other(priming) has a significant positive effect on the perceived social pre-sence of a user in the consecutive interaction with one of the IVAs.Therefore, it can be used as a mechanism for enhancing the social in-teraction experience with IVAs (ECAs).

Furthermore, Lee et al. [28] have shown that the para-verbalcharacteristics of a voice-mediated interaction between users and a

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

2

computer can have an effect on their perceived social presence. In anexperiment which did not involve an ECA, it was observed that usersexperienced a higher level of social presence when they heard a com-puter voice synthesized with a personality (extrovert vs. introvert,differentiated based on speed, volume, and pitch) matching their own.Additionally, a significant cross-over interaction between the person-ality-voice and personality-text (based on length, strong or friendlytone, and assertion) for social presence was reported. That is, an ex-trovert voice narrating an extrovert text caused greater social presence,and vice versa for the introvert versions. Contradictory to this finding,in the context of human-robot interaction, Lee and colleagues [29]report that when the personality of the user is not similar to the one ofthe robot, they perceived the robot to be more intelligent and sociallypresent.

The work of Gajadhar et al. [30], although focusing on interactionsamong humans, shows that rich social interactions occur when the in-volved parties are co-located in the same environment where verbaland non-verbal behaviors can be continuously monitored. This ob-servation relates to social processes such as immediacy, mirroring, andemotional contagion that characterize the feeling of social presence,which according to De Kort [31] affect game experience. Gajadhar et al.have shown that being co-located with a human opponent has a sig-nificant effect on the game experience in comparison to playing againsta computer (with no embodied agents involved), or a mediated co-player. They conclude that being co-located with a human co-player(immediacy) significantly improves positive affect, decreases tension,and increases a feeling of competence. These findings are in line withresults reported by Mandryk and Inkpen, and Ravaja et al. [32,33] in-dicating that immediacy plays a determining role in experiencing en-joyment. Furthermore, De Kort et al. [31] have stated that “higher levelsof social presence may be attained between remote players who are con-tinuously and mutually engaged in a collaborative game, than between co-located players who are each concentrated on attaining their individual goalswithout the need to interact or share” ([31, ], page 7).

Moreover, extensive argumentation can be found in the literaturehighlighting the importance of good game experience for effectivelearning. The degree to which factors such as involvement (flow) andimmersion are experienced in an environment designed for educationalpurposes can be predictors for its success [10]. Additionally, Accordingto Witmer and Singer [34] both immersion and involvement are pre-requisites for experiencing presence in a virtual environment. Theyhave defined immersion as “a psychological state characterized by per-ceiving oneself to be enveloped by, included in, and interacting with an en-vironment that provides a continuous stream of stimuli and experiences”([34], page 227); and involvement as “a psychological state experiencedas a consequence of focusing one’s energy and attention on a coherent set ofstimuli or meaningfully related activities and events” ([34], page 227).Involvement is a prerequisite for experiencing the flow state introducedby Csikszentmihalyi [35] as a state of absolute absorption to a task to apoint of losing self-consciousness where the activity itself becomes re-warding in its own, and this enables an individual to function at his/herfullest capacity. Moreno and Mayer [36] argue that a massive amountof information imposed by an activity (e.g. game) can overload theworking memory capacity of the player, leading to weak or incorrectlearning. Conversely, if the experience is under-loading, it can lead toboredom and disengagement, and consequently deterioration of per-formance. Similar arguments are made for the necessity to maintain abalance between challenge and competence based on the flow theory ofCsikszentmihalyi. On similar grounds Dede [12] has stated that apartfrom its rich entertainment value, a high level of immersion can en-hance learning.

In summary, the state of the art highlights the importance ofstudying the effect of an artifact (particularly ECAs) on the elicitedfeeling of social presence from users. Furthermore, it demonstrates howmanipulating different features of an ECA can influence this feeling.Moreover, there are indications to the existence of a relationship

between game experience, particularly immersion (feeling of co-loca-tion and immediacy), involvement (mutual engagement in activities)and presence. Also it is known that these experiences are important forthe realization of effective learning. What remains open is the questionto what extent social presence and game experience can be enhanced byutilizing an ECA that expresses its emotional states through non-verbalbehavioral cues, governed by its personality. Moreover, it is not clearlyknown which game experience metrics are the strongest predictors forthe extent to which one experiences a special form of presence mostrelevant in social contexts: social presence. In light of this, the researchpresented in the paper aims at investigating the effect that in-corporating personality as part of the emotional model of an ECA wouldhave on the perceived social presence and game experience of users.Furthermore, we will investigate which metrics of game experience arethe strongest predictors for the feeling of perceived social presence ofusers in a social interaction simulation that employs a personality-driven ECA equipped with non-verbal behaviors.

As such, we hypothesize: H1: A highly emotionally-personified agentwith apparent non-verbal behavior, governed by a personality-driven beha-vioral model would elicit (a) a higher level of social presence, and (b) abetter game experience, compared to a less emotionally-personified agentwith no apparent non-verbal behavior and H2: Immersion and involvementare prerequisites for social presence and thus predict the perceived sense ofsocial presence. To test these hypotheses we built a prototype consistingof an ECA with a personality-driven model that governs the expressionof her emotional state using a variety of non-verbal cues (see Sections 3and 4 for details), and evaluated its effect in a pilot study (see Section5).

3. The embodied conversational agent: Linda

An embodied conversational agent named Linda was designed in thecontext of this research. Linda is personified on the three facets: phy-sical, expressional, and emotional. Her corporeal representation whichembodies her facial and bodily features in form of a realistic female(can be also changed to a male character) in mid thirties provides herphysical personification. Furthermore, a series of facial expressions,gestures, and postures in form of synthesized animations constitute herexpressional personification. The emotional personification facet how-ever is more complex. This facet is considered as the “mind” of the ECAthat guides its emotional responses [17], and is therefore a compositionof multiple factors. Inspired by the emotional model proposed byBecker et al. [16,15], Linda was emotionally personified based on thefollowing human-like characteristics: personality, emotion and moodvalences, moodiness, and boredom. Personality is defined as “char-acteristics of a virtual human that distinguishes him from the others”, whileemotion is defined as “a state of mind that is only momentary”, and moodas “a prolonged state of mind, resulting from a cumulative effect of emo-tions” [17].

Our model calculates a concrete emotional state to be exhibited byLinda, by taking into account the interrelations between the afore-mentioned characteristics during each interaction instance with theuser. Briefly, the presence of a stimuli (i.e. what the user says to Linda)will have either a fortifying or an alleviating affect (depending on thenegative or positive nature of the stimuli) on the interrelations betweenemotion and mood valences, moodiness, and boredom characteristics ofLinda. Moreover, if no stimuli is produced by the user, the boredomcharacteristic of Linda will gradually increase after a specific period oftime, which in turn affects the interrelations between the aforemen-tioned characteristics. These newly derived characteristics are thentranslated to a point of reference (e.g. point P in Fig. 1) in a three in-dependent bipolar dimensional space represented by pleasure (dis-pleasure), arousal (sleepiness), and dominance (submissiveness) (PAD)introduced by Russel and Mehrabian [37] (for a more elaborate ex-planation regarding the underlying model please refer to [15,16]).

According to the creators of PAD, the three dimensions of pleasure,

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

3

arousal, and dominance are sufficient to adequately define and re-present any emotional state. In light of this, Russel and Mehrabian havesuggested 151 concrete emotional states represented as triples in thePAD space. Similar to Becker, other research works (e.g. [18]) haveeither directly incorporated a subset of these emotional states as part oftheir model, or have slightly adapted them to their context before doingso. Based on the interaction scenario of the social simulation used inthis research, we have incorporated 18 emotional states (Es) into ourbehavioral model, inspired by the triples listed by Becker et al., andBoulic et al. [15,18]. These states with their corresponding PAD valuescan be seen in Fig. 1.

In order to guide Linda on which emotional state to exhibit, anemotion awareness likelihood function [15,16] is used to calculate theproximity of the translated reference point P to either of the emotionalstates Es defined in the PAD space. This proximity is calculated on twolevels. Each Es is assigned two zones called activation (ϕ) and saturation(Δ), and a base intensity level (i) as its attributes. The activation zone hasa higher radius than the saturation zone. The emotion awareness like-lihood for each Es would increase, the closer the point of reference getsto it. An example for these zones can be seen for the Es “Sad” in Fig. 1.

At every interaction instance (i.e. presence of a stimuli or lackthereof), and for every Es point, if the distance between the point P andthat Es is less than the activation zone of that Es (i.e. ϕEs), the calcu-lation of its awareness likelihood ωEs (Eq. (1)) will commence. How-ever, it is possible for the reference point P to fall into the activationzone of more than one Es. In such situations, the Es with the highestlikelihood will be selected as the most appropriate candidate. More-over, if the distance between the reference point and each of the Espoints is less than the saturation zone of that Es (i.e. ΔEs), the awarenesslikelihood for those Es(s) will be calculated. If the reference point fallsinto the saturation zone of more than one Es, the Es with the highestlikelihood will be selected as the most appropriate candidate. Using thismechanism, Linda’s emotional model decides which emotional stateshould be exhibited by her. It is important to note that although thepoint of reference P will move within the PAD through the course ofinteraction with Linda, there is a possibility that P is not within thesaturation or activation zone of any Es. In this situation, Linda willmaintain her previous emotional state.

= Distance i1 ·EsEs

Es EsEs (1)

> …Es Es Es{ }Es Es 1 18

Furthermore, Linda is assigned a personality profile limited to theextrovert-introvert dimension of the Five Factor Model (FFM) [38].Depending on whether Linda is extrovert or introvert, a personalityprofile is synthesized and expressed in the PAD space (PPAD in Fig. 1)based on the characteristics of this personality dimension. As such thePPAD point can be looked at as a point where the ECA is most com-fortable since it corresponds to its personality. The PPAD point servestwo purposes: First, it manipulates the activation and saturation zoneradius, as well as the base intensity of certain Es(s). This is performedbased on the relationship between personality and the likelihood ofexhibiting (or not) certain emotional states. This mechanism will in-crease (or decrease) the probability of certain Es(s) to be exhibited bythe ECA. Second, the PPAD acts as a center of gravity when moving Pinto the PAD space after each interaction. If through the course of in-teraction P gets farther from (i.e. away from ECA’s comfort zone) PPAD(e.g. P in Fig. 1), its movement is manipulated in such a way that ittravels less distance. Conversely, if the P is getting closer to (i.e. to-wards the ECA’s comfort zone) the PPAD, it will travel more distance.This mechanism governs the emotional state exhibition by the ECAbased on its personality.

4. The simulation

To produce an immersive experience for people when interactingwith the ECA, a social simulation in virtual reality was created. Thisexperience takes place in a virtual office environment built usingUnity3D, where the characters are sitting across a table from eachother. The user assumes the role of a company manager, and is requiredto review the performance of one of his/her employees (i.e. our ECALinda). The simulation is perceived through a virtual reality headset(using Oculus Rift1), and the user has a 360 degree view of the room.

Fig. 1. 18 Emotional States Represented as (P, A, D).

1 https://www.oculus.com/rift.

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

4

This mechanism promotes a feeling of co-location with the ECA, andtherefore immediacy. As such, the user experiences the non-verbal be-haviors of the ECA from a point of view natural to them. This designchoice was made based on the assumption that it will positively affectthe game experience of users, and consequently their level of perceivedsocial presence as proposed by the research presented in [30]. Ad-ditionally, a Leap Motion2 device is attached on top of the VR headsetthat facilitates the virtual representation of user’s hands if s/he bringsthem within its field of vision. This mechanism provides partial virtualbody ownership and was added under the assumption that it will mo-tivate the users to employ body language (in terms of hand gestures)when interacting with the ECA. The primary mode of interaction withthe ECA however is by voice (using Microsoft speech API) in a turn-taking system. At each turn, multiple conversation options are availablefor both the user and ECA. The user would need to saying out loud thecontent of the option they want to utter to the ECA. As a consequence,that option will be highlighted and then selected (Fig. 2). Based on theoption chosen by the user, the ECA will select the appropriate responseand communicate it to the user by means of speech synthesis, aug-mented with lip-syncing, eye gazing, facial expressions, postures, andgestures. The state of the art on open-ended, but meaningful commu-nication between a human and an intelligent system is not advancedenough to allow for the implementation of such mechanism. Therefore,a pre-scripted conversation graph was used in this social simulation, inwhich the conversation options for both characters are synthesizedbeforehand.

The conversation that takes place between the user and the ECA isfocused on reviewing the performance of the ECA (i.e. the virtual em-poloyee) by the user, and the responses of the ECA to these revisions. Itis worth mentioning that this topic was chosen since it directly relatesto the educational aspect of this social simulation. However, the eva-luation of our simulator’s success in teaching users how to effectivelyevaluate the performance of an employee is out of the scope of thisarticle and is not further discussed. Throughout the course of interac-tion, each conversational choice selected by the user will introduce aneffect (i.e. the stimuli) on the human-like characteristics of the ECA(Section 3). These effects would be of either positive or negative natureand feature different levels of intensity. However, for a preliminaryevaluation of our proposed model the conversation options available tothe user were restricted to only the ones that would introduce a nega-tive effect (with the same level of intensity) on the emotional state ofthe ECA. This choice was made to ensure that the experience of users inrelation to interaction with the ECA remain consistent regardless ofwhich option they choose.

As such, the dynamic between Linda’s characteristics will constantlygrow towards the negative end of the spectrum, and therefore theirtranslation into a point of reference in the PAD space would eventuallyresult in a negative value for the Pleasure dimension. In light of this, thescenario of the simulation can be looked at as the user constantly givingharsh comments to Linda, thus violating the generally accepted rulesfor effectively reviewing an employee’s performance (e.g. be specificand complete, evaluate performance but not personality, stick to factsnot assumptions).

5. Experiment

To be able to adequately evaluate the effect of our proposed per-sonality-driven ECA on perceived social presence and game experienceof users, three experimental conditions were devised: neutral, extrovertand introvert. Each condition uses a different variation of the person-ality-driven behavioral model for governing the exhibition of emotionalstates by Linda, as well as their manifestation into non-verbal beha-viours.

In the first condition (i.e. neutral), Linda does not project heremotional state by any form of gestures, postures, or facial expressions.Her eye-gazing, however, is set to look at the user 60% of the time(averaged based on the values assigned to the other conditions).

In the second condition (i.e. extrovert), she is assigned a controlledextrovert personality as her PPAD at (80, 50, 100). These values are setbased on the characteristics of extroverted people given by McCraeet al. [39], in which they are deemed to have a tendency towards po-sitive emotion (P 80), are seeking excitement (A 50), and are assertive(D 100). Based on the assertiveness of the ECA in this condition, theECA will only exhibit emotional states with positive dominance values.However, since the scenario only allows for Linda to experience nega-tive emotions, the emotional states that can be exhibited by her arelimited to Annoyed, Anxious, and Angry. Furthermore, based on thecharacteristics of an extrovert person, it should require more effort tomake Linda experience negative emotions. Given that her personality isnot within the area of negative emotions with positive dominance va-lues, transitions from one negative emotion to another based on con-stant negative stimuli received from the user will be slower (as herpoint P is getting farther from her personality). Lastly, as a feature ofan extrovert person, Linda’s eye-gazing behavior is set to look directlyat the user 90% of the time.

In the third condition (i.e. introvert), the ECA is assigned a con-trolled introvert personality as her PPAD at (−80, 30,−100), based onthe characteristics of introvert people [39], which are characterized bya tendency towards negative emotion (P −80), low excitement seeking(A 30), and low assertiveness (D −100). Based on the assertiveness inthis condition, the ECA will only exhibit emotional states with negativedominance values. Given the negative nature of the scenario, and Lindabeing submissive in this condition, she can exhibit the emotions Sad,Overwhelmed, and Afraid. Considering Linda’s personality in this con-dition, it should not require a lot of effort to make her experience ne-gative emotions. Given that her personality is also within the area ofnegative emotions with negative dominance values, transitions fromone negative emotion to another based on constant negative stimulireceived from the user will be faster (as her point P is getting closer toher personality). As a feature of an introvert person, Linda’s eye-gazingbehavior is set to look directly at the user 30% of the time. Further-more, for an introvert personality the activation (set to 90) and sa-turation zones (set to 70), as well as the base intensity (set to 95) ofnegative emotions will be larger, increasing the chances of Linda ex-hibiting these emotional states.

The initial values for emotion and mood valences are set to zeroacross all conditions. Furthermore, the default values for all emotionalstates are 60 for activation zone and 40 for saturation zone, with a baseintensity value of 75. Although in both extrovert and introvert condi-tions the emotional states exhibited by Linda are expressed by means offacial expressions, gestures, and postures, their manifestations intohuman-like features are different. More precisely, the aforementioned

Fig. 2. Voice interaction with Linda and virtual embodiment of user’s hand.

2 https://www.leapmotion.com.

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

5

behavioral cues in the extrovert condition are more pronounced andassertive, and in the introvert condition they are more subtle andsubmissive. For instance, Fig. 3 demonstrates the expression of twoemotional states Angry (−70, 80, 100) and Afraid (−70, 80,−100) thatare identical to each other except for their dominance value. Further-more, in the extrovert condition, the ECA can exhibit the Angry emotionmanifested by mild frown eyebrows, direct eye contact, shoulders up,and leaning to the side, and in the introvert condition the ECA canexhibit the Afraid emotion manifested by slightly raised eyebrows,avoiding eye contact, dropped shoulders, and hands on legs.

5.1. Methodology

The perceived social presence and the game experience of users, aswell as the dynamics between these constructs were evaluated in a la-boratory experiment with N=41 participants. We used a between-subjects design in which participants were exposed to either the neu-tral, introvert or extrovert condition. In addition, the effect of partici-pants’ sex (gender) on the reported experiences was investigated.

5.1.1. MeasuresParticipant’s perception and experiences were collected via ques-

tionnaires. A general questionnaire was administered before interactionwith the prototype that asked for demographic information (age,gender, frequency of gaming, past experiences with creating ECAs, andwith virtual reality) from participants, in addition to measuring theirempathy level. The Toronto empathy questionnaire [40] was used tomeasure the empathy level of participants by summing their scores on16 statements (with lowest overall possible score 16, and highest 80).

After interaction with the prototype we administered a secondquestionnaire that asked for participants’ perceived social presence,game experience, and empathy towards the agent. Several instrumentswere used, and their scores combined to measure participants’ levels ofperceived social presence: five items by Bailenson et al. [41], the socialpresence module of GEQ [42] (in terms of empathy, and behavioralinvolvement), and the attentional allocation scale of Biocca et al. [43].Using a combination of these instruments, we were confident that wecould adequately measure social presence. Furthermore, a combinedvalue of social presence was calculated by averaging the scores on theaforementioned four social presence metrics. This combined valuepresents a single score for users’ levels of perceived social presence.

The game experiences of participants were assessed using the coremodule of GEQ [42] (based on immersion, flow, tension/annoyance,and negative and positive affects). Furthermore, participants’ percep-tion of and attitude towards the ECA were measured. Given the nega-tive nature of the interaction scenario, it was of interest to assess howwell the participants could estimate the ECA’s affective states. There-fore, we measured their perception of Linda’s negative feelings usingthe PANAS instrument of Watson et al. [44]. Additionally, we measuredtheir level of empathy towards Linda using the instrument of von der

Pütten et al. [45]. Furthermore, using the likeability scale of Bailensonet al. [41], it was measured to what extent the participants liked Linda.Lastly, using six items, the perception of Linda’s features were assessedin terms of the degree to which the users noticed changes in her facialexpressions, gestures and postures, lip-syncing, blinking, gazing, andfollowing the head movements of the user. All items were rated on aLikert-type scale from 1 (“strongly disagree”) to 5 (“strongly agree”).For data analyses, the items were collapsed into average scores per sub-scale.

5.1.2. ProcedureBefore interacting with the prototype, the participants were given a

brief presentation which introduced them to the scenario and me-chanics of the interaction. After the presentation, the first questionnairewas administered (i.e. five demographic questions and the Torontoempathy questionnaire). Afterwards, the participants were given ampleopportunity to tryout the experience by exchanging some genericgreeting statements in a sample interaction. The tryout was adminis-tered to familiarize the participants with the mechanics of the proto-type, while providing them with the opportunity to ask any questionsthey might have. Afterwards, the participants were randomly assignedto either neutral, introvert, or extrovert conditions, and interacted withthe prototype. The duration of interaction with the prototype was onaverage 15min. During this phase, the participants were left alone inthe room so they would feel comfortable when interacting with Lindaverbally and possibly non-verbally (e.g. by using hand gestures). Oncethe interaction was over, the participants had to fill in the secondquestionnaire for the evaluation of their perceptions and experiences(see Section 5.1.1). Furthermore, an informal interview was conductedby the experimenter to gather further insights on individual im-pressions. Finally, participants were fully debriefed at the end of thesession.

5.1.3. Participants41 participants (20 male, 19 females, 1 who preferred not to an-

swer, and 1 indicating “other”) took part in the preliminary evaluationof this prototype. Out of them, 13 were assigned to the neutral condi-tion and 14 to each the extrovert and introvert conditions, respectively.The mean age of the participants was 25.73 (SD=4.46), ranging from20 to 40 years old. The participants were recruited using a universityexperiment participation platform and flyers, and each was compen-sated €5 for 20–30min (includes all the steps in the procedure) of theirparticipation time. To explore the effect of gender on how the prototypeis perceived, we compared male and female participants using an in-dependent t-test, which revealed that females were in general moreempathetic (t=−2.36, p < .05), experienced less tension/annoyance(t=2.32, p < .05), less negative affect (t=3.5, p < .001), and weremore empathetic towards Linda (t=−2.310, p < .05). However,gender did not affect our main findings reported in this paper.

5.2. Analysis

To investigate the formulated hypotheses, at first a series ofANOVAs were calculated to unveil the differences between the condi-tions. We controlled for the influence of participants’ empathy state as acovariate in the analysis, but since it did not affect the results, it is notfurther discussed. In the case of ANOVA indicating a significant dif-ference, the LSD post hoc test was used to unveil which conditions aresignificantly different from each other.

To explore the relationship between social presence and game ex-periences, a series of regression analyses were performed. To in-vestigate whether game experience can predict social presence, a step-wise multiple regression was calculated with each of the measuredmetrics of social presence as the dependent variable, and each of themeasured game experience factors as an independent variable.Conversely, to investigate whether social presence can predict game

Fig. 3. Angry emotion on the left, and sad emotion on the right.

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

6

experience, a number of linear regressions were calculated. In eachcalculated linear regression, one of the measured game experiencemetrics was the dependent variable and the social presence metric theindependent variable. All analyses were performed using IBM SPSSStatistics 22.

5.3. Results & discussion

The overall scores (all participants across all conditions) for themeasured metrics are represented in the overall column of Table 1. As isevident from the these scores, interaction with an ECA inside virtualreality, and using voice as a natural communicative mean, prompts arather rich experience. This is corroborated by the above average scoresfor the mean overall social presence (3.36) of Bailenson et al., the meanoverall empathy (3.34) and behavioral involvement (3.67) as part ofthe social presence module of GEQ, the mean overall attention alloca-tion (3.76) of Biocca et al., and the mean overall combined score forsocial presence (3.53). These scores indicate a positive evaluation of theprototype in successful elicitation of a feeling of perceived social pre-sence as a result of interaction with a personality-driven ECA. Fur-thermore, the results indicate that participants had a positive experi-ence interacting with the prototype in light of the game experiencemetrics. This is corroborated by their above average mean scores onimmersion (3.62), flow (3.72), and positive affect (3.66), and low meanscores on tension/annoyance (1.94) and negative affect (1.98) of GEQ.In summary, the mean scores indicate a promising positive evaluationof the prototype (note that the lowest possible score is 1 and the highest5).

Furthermore, we observed that mean score for the perception ofECA’s negative feeling is slightly below average (2.73). Given the ne-gative nature of the interaction scenario, this was expected and can beinterpreted as the ECA being successful in expressing its negativeemotional states in a comprehensible fashion. Moreover, we observethat the expression of these negative feelings has resulted in a highmean score for empathy towards the ECA (3.16). Interestingly, how-ever, feeling empathetic towards the ECA did not result in participantsliking the ECA, as the mean score for this factor is only slightly belowaverage (2.89). We attribute this result to the negative nature of theexperience, where the ECA had to respond negatively to the actions ofthe user. However, empathy towards the ECA and its likableness wereshown to be positively and significantly (p < .05) correlated to oneanother.

To test H1a, an ANOVA test with the social presence measures asdependent variables revealed a significant difference between thegroups for the factor behavioral involvement (F(2,41)= 3.348,p < .05, part.eta2= .150).

A post hoc test (LSD) indicated that the behavioral involvementscore in the extrovert condition was significantly higher than the neu-tral (p < .05, Mean difference= .366), and the introvert (p < .05,Mean difference= .333) conditions, but the introvert condition was notsignificantly different from the neutral (see Table 1 for means). Thebehavioral involvement factor as part of the social presence module ofGEQ was measured using items that asked participants to what extentthey felt their actions were dependent on the actions of the ECA andvice versa, to what extent they paid close attention to the ECA and viceversa, and to what extent the ECA affected what they did and vice versa.The results indicate that an emotionally-personified extrovert ECA withmore pronounced and assertive non-verbal behaviors elicits a higherfeeling of behavioral involvement in the interaction with the ECA,compared to both a less emotionally-personified ECA with no apparentnon-verbal behavior, and an emotionally-personified introvert ECAwith more subtle and submissive non-verbal behaviors. As such, ourresults suggest that the manifestation of a personality-driven emotionalmodel in which the non-verbal behavior of the ECA is more elaborateand profound (i.e. more assertive and pronounced gestures, and higherrate of making direct eye contact with the user) evokes a higher feelingof behavioral involvement.

The results therefore only partially support our hypothesis H1a forthe extrovert version evoking a higher sense of perceived social pre-sence than the neutral condition, whereas no anticipated difference forthe introvert condition was observable. To investigate why the introvertcondition did not differ from the neutral, we re-considered the controlquestions that addressed in how far participants recognized changes inthe ECA’s behavior. ANOVA revealed a significant effect (F(2,41)= 14.398, p < .05, part.eta2= .431) for changes in the ECA’sposture and gestures. Compared to the neutral condition, in both ex-trovert (P= .000, Mean difference = 1.96) and introvert (P= .016,Mean difference= .964) conditions, participants noticed that Lindachanged her posture and gesture a few times. This was expected since inthe neutral condition, there was no apparent non-verbal behavior fromLinda. However, there was also a significant difference between theextrovert and introvert conditions for this feature (p < .01, Mean dif-ference =1) indicating that people noticed changes in the non-verbalbehavior of Linda in form of gestures and postures significantly more inthe extrovert version. The results highlight the importance of how apersonality-driven emotional model should be manifested to ensue thehighest effect, since a significant difference between the extrovert andintrovert conditions was surprisingly observable. Furthermore, similarto what was reported by von der Pütten et al. [24], our results do notsupport the claim of Nowak and Biocca [25] that high levels of an-thropomorphism raises expectations in users (i.e. expecting the ECAbehavior to be identical to a real human) and this will negatively affect

Table 1Comparison of means and standard deviations of the overall population & three condition groups.

Factors Overall Baseline Extrovert Introvert

Mean SD Mean SD Mean SD Mean SD

Social Presence (Bailenson et al.) 3.36 .617 3.29 .700 3.57 .548 3.22 .591Social presence - empathy (GEQ) 3.34 .578 3.46 .689 3.27 .448 3.31 .606

Social presence - behavioral involvement (GEQ) * 3.67 .434 3.53 .361 3.90 .411 3.57 .451Social presence - attention allocation (Biocca et al.) 3.76 .571 3.83 .476 3.85 .629 3.61 .600

Social presence (combined) 3.53 .390 3.53 .450 3.65 .320 3.43 .392

Game experience - immersion (GEQ) 3.62 .653 3.62 .863 3.66 .543 3.57 .568Game experience - flow (GEQ) 3.72 .655 3.66 .718 3.78 .380 3.72 .832

Game experience - tension/annoyance (GEQ) 1.94 .805 2.10 .926 1.83 .748 1.90 .778Game experience - negative affect (GEQ) 1.98 .595 2.05 .501 2 .596 1.89 .698Game experience - positive affect (GEQ) 3.66 .622 3.61 .772 3.64 .544 3.72 .579

Perception of agent’s negative feeling (PANAS) 2.73 .483 2.56 .448 2.79 .331 2.82 .617Empathy towards the agent 3.16 .642 3.06 .706 3.11 .651 3.31 .590Likableness of the agent 2.89 .770 2.96 .828 2.75 .914 2.96 .570

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

7

their perceived social presence when the expectations are not met. Infact, we have witnessed that being exposed to a highly personifiedversion of Linda equipped with pronounced non-verbal behaviors, re-sult in a higher sense of perceived social presence compared to a versionwith minimal non-verbal behaviors or one with non.

With regard to H1b, ANOVA with the game experience measures asdependent variables did not reveal any significant differences betweenthe conditions and, hence H1b is rejected. Considering that the overallgame experience across all conditions was quite positive, we cantherefore conclude that the combination of VR, voice input, speechsynthesis, lip-syncing, and eye gazing evokes an adequate level of gameexperience in users, regardless of the inclusion of personality-drivennon-verbal behavior.

In light of the reported results, we opted to further investigate H2 byexploring the dynamics between the different game experience metricsand the perceived social presence of all participants, using the com-bined measure of perceived social presence. Since our prototype elicitsa rather consistent game experience and perceived social presenceacross all conditions (with the exception of behavioral involvement aspart of social presence and only for the extrovert condition), we decidedto investigate the aforementioned dynamics over all participants andusing a single value that represents social presence.

The result of the step-wise multiple regression analysis indicate thatindeed (some of) the game experience metrics are significant predictorsfor the combined perceived social presence measure of users (F(3,37)= 12.113, p < .000), with an adjusted R2 of.455. The calculatedmodel indicated immersion, flow, and tension/annoyance as the pre-dictors explaining 45.5% of variance in social presence, and excludednegative and positive affect metrics. The model indicates that the par-ticipants’ predicted perceived social presence is equal to 2.284+ .179(immersion)+ .240 (flow)− .148 (tension/annoyance), where im-mersion is measured as how rich and impressive the experience was;how pleasant the aesthetics were; how imaginative and explorative theusers felt; and how interesting the story of the simulation was. Flow ismeasured as how occupied the users were with the activity and con-centrated in the experience; and to what extent they forgot everythingaround them and lost track of time as well as the connection to theoutside world. Tension/annoyance measured to what extent the usersfelt annoyed, frustrated, and irritable. All three game experience me-trics were revealed to be significant predictors of social presence (im-mersion p < .05, flow p < .01, and tension/annoyance p < .05).

Based on these results (Table 2), we observe that flow is thestrongest predictor of social presence among the three game experiencemetrics that were part of the model (B= .240 p < .01). Given themeasures of flow in GEQ, our results suggest that the actual interactionwith the ECA (engagement with the activity) is the strongest predictorfor perceived social presence when interacting with an ECA in VR.Notwithstanding, tension/annoyance and immersion were also sig-nificant predictors. We interpret this as low levels of frustration, irri-tation, and annoyance with regard to the experience, in addition to theaesthetics, and richness of the experience although significantly influ-ential, are surpassed by a feeling of losing track of time, and being

occupied and engaged with an activity that involves interaction with ahuman-like entity. As such, the results suggest that similar to presence,both immersion and flow (involvement) are necessary requirements forexperiencing social presence. In light of these results we accept H2.

In addition, it was also revealed that the participants’ levels ofperceived social presence is a significant predictor for their game ex-perience. The results of these linear regression analyses are shown inTable 2. As such, the results of this experiment suggest that there isindeed a bidirectional relationship between the perceived social pre-sence and game experience in the context of human-ECA interaction inVR. In light of these results, we argue that more emphasis should beplaced on the design of engaging activities to be performed in relationto an ECA in human-agent interaction contexts in order to elicit a highlevel of perceived social presence.

6. Conclusions, limitations, & future work

The results of a pilot study with our prototype indicate an overallpositive perception of it in terms of perceived social presence, gameexperience, and attitude towards the incorporated ECA. Based on theresults of our analysis, H1a was partially accepted. It was observed thatan emotionally-personified ECA with an extrovert-based personalityelicits a higher sense of behavioral involvement in users, compared to aless emotionally-personified agent with no non-verbal behavior, or onewith an introvert-based personality. In light of these results, and limitedto the context of this experiment, we conclude that (1) higher levels ofpersonification based on incorporating personality as part of the be-havioral model of an ECA seem to induce an increase in the behavioralinvolvement of users with an ECA as part of their social presence, (2)the way a personality-driven behavioral model is manifested as an ECAequipped with non-verbal behaviors seem to have a substantial influ-ence on the first conclusion. These conclusions are corroborated by theobservations that an extrovert ECA with more pronounced, assertive,and noticeable non-verbal behaviors evokes a higher sense of beha-vioral involvement in users. Furthermore, based on the results of ouranalysis H1b was rejected as no significant differences between theconditions were observed with respect to the measured game experi-ence metrics.

Furthermore, our results indicate that similar to presence, immer-sion and flow (engagement) are bidirectionally related to social pre-sence. Although the results do not enable us to generalize that flow andimmersion are mandatory requirements for experiencing social pre-sence particularly when interacting with an ECA in VR, we have shownthat they are strong predictors for the perceived feeling of social pre-sence of users in the context of this research. Based on this result H2was accepted. Moreover, we have witnessed that engagement with theactivity performed in the virtual environment and in relation to the ECA(actual interaction with the ECA) is a more significant predictor forperceived social presence, compared to the immersiveness of the ex-perience itself. Furthermore, the role of tension/annoyance as part ofthe game experience of users that was largely neglected by previousstudies when examining the relation between game experience and

Table 2Results of regression analyses.

Game experience predicting social presence

Social presence (combined)Flow B= .240 p < .01

Immersion B= .179 p < .05Tension/annoyance B=−.148 p < .05

Social presence predicting game experience

Flow Immersion Tension/ annoyance Negative affect Positive affectSocial presence (combined) B= .856

p < .01B= .926p < .000

B = -.841p < .01

B=−.629p < .01

B= .939p < .000

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

8

presence, was shown to be significant. Our results suggest that the lesstension the users experience, the more likely they are to perceive thesocial presence of the ECA with respect to our experimental setup.Based on these observations, we conclude that (3) the game experiencemetrics defined as flow, immersion, and tension seem to be significantpredictors for the perceived social presence of people in social human-ECA interaction in VR, with flow as the strongest predictor.

Notwithstanding, it is important to keep in mind that this researchwas carried out using a prototype and only a sample size of 41 parti-cipants. As such, it unavoidably has some limitations and presents op-portunities for improvement. One of the most frequently receivedcomments from the participants during the informal interview after theexperiment sessions was related to the way conversation options werevisualized and presented to them. Given the fact that participants per-ceived the virtual world through VR glasses, each option was presentedto them as an standalone 3D object in the environment visualized onthe top left, center, or top right side of the ECA (depending on thenumber of conversation options available at each turn). As is the case innormal conversation, people often utter more than a few words whenexpressing their opinion and emotions. Therefore, in some cases thecontent of the options spanned over two sentences and required theparticipants to invest the necessary time for studying them before de-ciding which one to choose and utter to the ECA. This forced the par-ticipants to frequently divide their attention between the conversationoptions and the ECA for a relatively long duration. Therefore, theycould not dedicate their undivided attention to observing the ECA’snon-verbal behavioral cues all the time, which might have influencedtheir perceived social presence as well as game experience. As part offuture work, we are working on a mechanism that would display theoptions one at a time, placed right under the ECA. The users can thenflip through the list of options (e.g. using the controllers of the VRheadset), while having the ECA present in their area of attention at alltimes.

Furthermore, although all participants were introduced to thefunctionality of Leap Motion and were told that they had the freedom touse body language in form of hand gestures when interacting with theECA, almost no participant used this feature. This could be due to thereason that people do not tend to use no-verbal language when inter-acting with artificial entities, and/or the lack of participants’ full em-bodiment in VR (lack of a body ownership feeling). Therefore, as part ofthe future work, we are planning to incorporate a full embodiment ofthe users in this environment and perform real-time body tracking usinginverse kinematic techniques. This is of interest to the continuation ofthis research, as it will aid us in monitoring, analyzing, and interpretingthe signals produced by the users in terms of gestures and postureswhen interacting with the ECA. Such addition would open the door forinvestigating rather interesting research questions, such as the evalua-tion of the effect of full embodiment in VR on perceived social presence.

Moreover, with respect to the dynamics between game experiencemetrics and the perceived social presence of participants, our results donot enable us to draw any conclusions on what exactly affects the gameexperiences that predict the perceived social presence of users. In lightof this, as part of the future direction of this research, different aspectsof this virtual experience (e.g. interaction modality, the use of VR,content of the interaction, quality of the environment and the ECA, etc.)will have to be manipulated to assess their effects on different gameexperience metrics, and consequently on the perceived social presenceof users.

Lastly, based on the gained insights on incorporating personality aspart of the emotional model of an ECA and the effect of its manifesta-tion on the perceived social presence and the overall game experiencescores of users, we are confident in having a good enough system tostart evaluating its learning effect on social behavior competency ofusers. Therefore, as part of future work, we will be focusing on trainingindividuals for adequate social behavior competencies in the context ofemployee evaluation using our prototype.

References

[1] C.C. Abt, Serious Games, University Press of America, 1987.[2] J. Linssen, M. Theune, M. Bruijnes, et al., Beyond simulations: serious games for

training interpersonal skills in law enforcement, Social Simulation Conference,2014.

[3] M.K. Holden, Virtual environments for motor rehabilitation, Cyberpsychol. Behav.8 (3) (2005) 187–211.

[4] P. Sajjadi, E. El Sayed, O. De Troyer, On the impact of the dominant intelligences ofplayers on learning outcome and game experience in educational games: theTrueBiters case, GALA 2016, Springer International Publishing, Cham, 2016, pp.221–231, , https://doi.org/10.1007/978-3-319-50182-6_20.

[5] J. Cassell, T. Bickmore, M. Billinghurst, L. Campbell, K. Chang, H. Vilhjálmsson,H. Yan, Embodiment in conversational interfaces, Proceedings of the SIGCHI con-ference on Human factors in computing systems the CHI is the limit – CHI ’99 (5),1999, pp. 520–527, , https://doi.org/10.1145/302979.303150.

[6] P. Sajjadi, L. Hoffmann, P. Cimiano, S. Kopp, On the effect of a personality-drivenECA on perceived social presence & game experience in vr, 2018 10th InternationalConference on Games and Virtual Worlds for Serious Applications (VS-GAMES),IEEE, 2018, pp. 1–8.

[7] J. Richardson, K. Swan, Examining social presence in online courses in relation tostudents’ perceived learning and satisfaction.

[8] F. Biocca, C. Harms, J.K. Burgoon, Toward a more robust theory and measure ofsocial presence: review and suggested criteria, Presence: Teleop. Virt. Environ. 12(5) (2003) 456–480.

[9] W.L. Johnson, J.W. Rickel, J.C. Lester, et al., Animated pedagogical agents: face-to-face interaction in interactive learning environments, Int. J. Artif. intell. Educ. 11(1) (2000) 47–78.

[10] K. Kiili, Digital game-based learning: towards an experiential gaming model,Internet Higher Educ. 8 (1) (2005) 13–24, https://doi.org/10.1016/j.iheduc.2004.12.001.

[11] J. Webster, L.K. Trevino, L. Ryan, The dimensionality and correlates of flow inhuman-computer interactions, Comput. Hum. Behav. 9 (4) (1993) 411–426,https://doi.org/10.1016/0747-5632(93)90032-N.

[12] C. Dede, Immersive Interfaces for engagement and learning, Science 323 (5910)(2009) 66–69, https://doi.org/10.1126/science.1167311.

[13] M. Slater, Place illusion and plausibility can lead to realistic behaviour in immersivevirtual environments, Philos. Trans. Roy. Soc. B Biol. Sci. 364 (1535) (2009)3549–3557.

[14] M.E. Hoque, M. Courgeon, J.C. Martin, M. Bilge, Mach: My automated conversationcoach, UbiComp ’13: Proceedings of the 2013 ACM International Joint Conferenceon Pervasive and Ubiquitous Computing, 2016, pp. 697–706, , https://doi.org/10.1145/2493432.2493502.

[15] C. Becker-Asano, S. Kopp, N. Pfeiffer-Leßmann, I. Wachsmuth, Virtual humansgrowing up: from primary toward secondary emotions, KI - Künstliche Intelligenz 1(2008) 23–27.

[16] C. Becker-Asano, WASABI: affect simulation for agents with believable interactivity319 (2008) 186.

[17] S. Kshirsagar, A multilayer personality model, Proceedings of the 2nd InternationalSymposium on Smart Graphics – SMARTGRAPH ’02 (January 2002), 2002, pp.107–115, , https://doi.org/10.1145/569005.569021.

[18] R. Boulic, J. Ahn, S. Gobron, N. Wang, Q. Silvestre, D. Thalmann, Towards theinstantaneous expression of emotions with avatars, in: ACM/EurographicsSymposium on Computer Animation. https://doi.org/10.1007/978-3-319-43639-5_13.

[19] J.A. Waterworth, E.L. Waterworth, G. Riva, F. Mantovani, Presence: form, contentand consciousness, Immersed in Media, Springer, 2015, pp. 35–58.

[20] C. Youngblut, Experience of presence in virtual environments, Tech. rep., Institutefor Defense Analyses Alexandria VA, 2003.

[21] F. Biocca, C. Harms, Defining and measuring social presence: contribution to thenetworked minds theory and measure, Proc. Presence 2002 (2002) 7–36.

[22] F. Biocca, K. Nowak, Plugging your body into the telecommunication system:mediated embodiment, media interfaces, and social virtual environments,Commun. Technol. Soc. (2001) 407–447.

[23] J. Blascovich, J. Loomis, A.C. Beall, K.R. Swinth, C.L. Hoyt, J.N. Bailenson,Immersive virtual environment technology as a methodological tool for socialpsychology, Psychol. Inq. 13 (2) (2002) 103–124.

[24] A. von der Pütten, N.C. Krämer, J. Gratch, S.-H. Kang, “it doesn’t matter what youare!” explaining social effects of agents and avatars, Comput. Hum. Behav. 26 (6)(2010) 1641–1650.

[25] K.L. Nowak, F. Biocca, The effect of the agency and anthropomorphism on users’sense of telepresence, copresence, and social presence in virtual environments,Presence: Teleop. Virt. Environ. 12 (5) (2003) 481–494.

[26] J.H. Chuah, A. Robb, C. White, A. Wendling, S. Lampotang, R. Kopper, B. Lok,Increasing agent physicality to raise social presence and elicit realistic behavior,Virtual Reality Short Papers and Posters (VRW), 2012 IEEE, IEEE, 2012, pp. 19–22.

[27] S. Daher, K. Kim, M. Lee, R. Schubert, G. Bruder, J. Bailenson, G. Welch, Effects ofsocial priming on social presence with intelligent virtual agents, InternationalConference on Intelligent Virtual Agents, Springer, 2017, pp. 87–100.

[28] K.M. Lee, C. Nass, Designing social presence of social actors in human computerinteraction, Proceedings of the SIGCHI Conference on Human Factors in ComputingSystems, ACM, 2003, pp. 289–296.

[29] K.M. Lee, W. Peng, S.-A. Jin, C. Yan, Can robots manifest personality? An empiricaltest of personality recognition, social responses, and social presence in hu-man–robot interaction, J. Commun. 56 (4) (2006) 754–772.

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

9

[30] B. Gajadhar, Y. De Kort, W. IJsselsteijn, Influence of social setting on player ex-perience of digital games, CHI’08 Extended Abstracts on Human Factors inComputing Systems, ACM, 2008, pp. 3099–3104.

[31] Y.A. De Kort, W.A. IJsselsteijn, B.J. Gajadhar, People, places, and play: a researchframework for digital game experience in a socio-spatial context, DiGRAConference, 2007.

[32] R.L. Mandryk, K.M. Inkpen, Physiological indicators for the evaluation of co-locatedcollaborative play, Proceedings of the 2004 ACM Conference on ComputerSupported Cooperative Work, ACM, 2004, pp. 102–111.

[33] N. Ravaja, T. Saari, M. Turpeinen, J. Laarni, M. Salminen, M. Kivikangas, Spatialpresence and emotions during video game playing: does it matter with whom youplay? Presence: Teleop. Virt. Environ. 15 (4) (2006) 381–392.

[34] B.G. Witmer, M.J. Singer, Measuring presence in virtual environments: a presencequestionnaire, Presence 7 (3) (1998) 225–240.

[35] M. Csikszentmihalyi, I.S. Csikszentmihalyi, Optimal Experience: PsychologicalStudies of Flow in Consciousness, Cambridge University Press, 1992.

[36] R. Moreno, R. Mayer, Interactive multimodal learning environments, Educ. Psychol.Rev. 19 (3) (2007) 309–326, https://doi.org/10.1007/s10648-007-9047-2.

[37] J.A. Russell, A. Mehrabian, Evidence for a three-factor theory of emotions, J. Res.Pers. 11 (3) (1977) 273–294, https://doi.org/10.1016/0092-6566(77)90037-X.

[38] J.M. Digman, Higher order factors of the Big Five, J. Pers. Soc. Psychol. 73 (6)(1997) 1246–1256.

[39] R.R. McCrae, P.T. Costa Jr, T.A. Martin, The neo–pi–3: a more readable revised neopersonality inventory, J. Pers. Assess. 84 (3) (2005) 261–270.

[40] R.N. Spreng, M.C. McKinnon, R.A. Mar, B. Levine, The Toronto empathy ques-tionnaire: scale development and initial validation of a factor-analytic solution tomultiple empathy measures, J. Pers. Assess. 91 (1) (2009) 62–71.

[41] J.N. Bailenson, R.E. Guadagno, A.C. Beall, J. Blascovich, Comparing behavioral andself-report measures of embodied agents’ social presence in immersive virtual en-vironments, 7th Annual International Workshop on Presence, 2004, pp. 216–223.

[42] W.A. IJsselsteijn, Y.A.W. De Kort, K. Poels, The game experience questionnaire.[43] F. Biocca, C. Harms, J. Gregg, The networked minds measure of social presence:

pilot test of the factor structure and concurrent validity, 4th Annual InternationalWorkshop on Presence, 2001, pp. 1–9.

[44] D. Watson, L.A. Clark, A. Tellegen, Development and validation of brief measures ofpositive and negative affect: the panas scales, J. Pers. Soc. Psychol. 54 (6) (1988)1063.

[45] A. von der Pütten, F.P. Schulte, S.C. Eimler, S. Sobieraj, L. Hoffmann, S. Maderwald,M. Brand, N.C. Krämer, Investigations on empathy towards humans and robotsusing fmri, Comput. Hum. Behav. 33 (2014) 201–212.

P. Sajjadi, et al. Entertainment Computing 32 (2019) 100313

10