Social Interaction with Agents

AAMAS 2003 CARTE @ USC / Information Sciences Institute

Social Interaction with AgentsSocial Interaction with Agents

Lewis JohnsonLewis JohnsonDirector, CARTEDirector, CARTE

USC / Information Sciences InstituteUSC / Information Sciences Institute


Jeff Rickel 1963 - 2003


Rationale: Reeves & Nass’s Rationale: Reeves & Nass’s Media EquationMedia Equation People tend to relate People tend to relate

to computers and to computers and other media as they other media as they do to peopledo to people

Confirmed by Confirmed by multiple multiple experimental studiesexperimental studies

Synthetic agents Synthetic agents exploit this tendencyexploit this tendency


ClaimsClaims Synthetic agents raise expectations of Synthetic agents raise expectations of

Ability to understand user’s activitiesAbility to understand user’s activities Social interaction skills, i.e., Social interaction skills, i.e., social intelligencesocial intelligence

The challenge: to meet these The challenge: to meet these expectationsexpectations

Primary focus: pedagogical agentsPrimary focus: pedagogical agents Educational context helps constrain the problemsEducational context helps constrain the problems Implications for other types of human-agent Implications for other types of human-agent

interactioninteraction


Social Intelligence Implies:Social Intelligence Implies: Ability to model people, other agentsAbility to model people, other agents

May include goals, plans, emotions, motivations, May include goals, plans, emotions, motivations, personality, etc.personality, etc.

Ability to engage in social interactionAbility to engage in social interaction Sensitive to the model of the person or agentSensitive to the model of the person or agent Sensitive to the social contextSensitive to the social context In coordination with task activitiesIn coordination with task activities To motivate, influence, develop rapportTo motivate, influence, develop rapport


Adele Pedagogical AgentAdele Pedagogical Agent


Some Failures in Social Some Failures in Social IntelligenceIntelligence Criticizing the same mistake over and overCriticizing the same mistake over and over Interrupting the learner after minor mistakesInterrupting the learner after minor mistakes Giving impression of negative emotional Giving impression of negative emotional

reactions to learner’s actionsreactions to learner’s actions Failing to show respect for the learner’s workFailing to show respect for the learner’s work Failing to offer encouragement when the Failing to offer encouragement when the

learner needs itlearner needs it Failing to offer help when the learner is Failing to offer help when the learner is

confused and frustratedconfused and frustrated


Teasing Apart Social IntelligenceTeasing Apart Social Intelligence


One Aspect of SI: Team One Aspect of SI: Team CoordinationCoordination Stems from an agent-oriented view of Stems from an agent-oriented view of

human-computer interactionhuman-computer interaction Learner, virtual tutor, other agents act Learner, virtual tutor, other agents act

together as a teamtogether as a team Roles and responsibilities dynamically Roles and responsibilities dynamically

allocated among team membersallocated among team members Learning activities, teamwork, and task Learning activities, teamwork, and task

work are flexibly integratedwork are flexibly integrated Examples: Steve, MREExamples: Steve, MRE


SteveSteve

Team demo


Mission Rehearsal ExerciseMission Rehearsal Exercise


Steve/MRE Team/Task ModelSteve/MRE Team/Task Model Team task represented as a Team task represented as a

hierarchical, nonlinear planhierarchical, nonlinear plan Including possible alternative courses of actionIncluding possible alternative courses of action Courses of action evaluated for expected utilityCourses of action evaluated for expected utility

Step responsibilities (and authorities) Step responsibilities (and authorities) assigned to team membersassigned to team members

Possibility of dynamic sharing of responsibilitiesPossibility of dynamic sharing of responsibilities Each team member has own (possibly Each team member has own (possibly

partial) model of team plan & statuspartial) model of team plan & status

Rickel & Johnson, IJCAI ‘01; Traum, Rickel, Gratch, & Marsella, AAMAS ‘03See also: Scerri et al., AAMAS ’03; Davies et al., IUI ‘01


Another Aspect of SI: Face-to-Another Aspect of SI: Face-to-Face CommunicationFace Communication Example: MREExample: MRE Dialog model tracks state of communication Dialog model tracks state of communication

among team membersamong team members Components:Components:

Which team members are in contactWhich team members are in contact Who and what is being attended toWho and what is being attended to Who are participants (or overhearers) in the conversationWho are participants (or overhearers) in the conversation Who has the conversational initiativeWho has the conversational initiative Common knowledgeCommon knowledge

Including social commitments to actions and to factsIncluding social commitments to actions and to facts Communicative acts and grounding actsCommunicative acts and grounding acts Negotiation actsNegotiation acts Eye gaze, nonverbal gestures signal attention, groundingEye gaze, nonverbal gestures signal attention, grounding

Marsella, et al., 2003; Traum & Rickel, AAMAS ’02; Traum et al., AAMAS ‘03

Dialog demo


Another Aspect of SI: Another Aspect of SI: Interaction TacticsInteraction Tactics How to help the learnerHow to help the learner

Respecting learner’s autonomy & sense of controlRespecting learner’s autonomy & sense of control How to influence the learnerHow to influence the learner

Motivating the learner as neededMotivating the learner as needed When When notnot to help the learner to help the learner

Reinforcing autonomy, engagementReinforcing autonomy, engagement Assume appropriate social stance toward the Assume appropriate social stance toward the

learnerlearner Interaction in the context of a social relationshipInteraction in the context of a social relationship

Bottom line:Bottom line: Influence of social relationships on human-agent interactionInfluence of social relationships on human-agent interaction Rhetoric for human-agent interactionRhetoric for human-agent interaction


Example:Example: Carmen’s Bright Carmen’s Bright IDEASIDEAS

Marsella, Johnson, & LaBore: Agents 2000, AI-Ed ‘03


Gina’s Dialog Model Gina’s Dialog Model

SuggestStrategy

PromptNextStep

Summarize

AnswerQuestion

Reassure

Praise

Gina’s main struggle: Get Carmen thru the Gina’s main struggle: Get Carmen thru the I-D-E-A-SI-D-E-A-S steps steps At each step, suggest a joint strategy (e.g. “old 5Ws”)At each step, suggest a joint strategy (e.g. “old 5Ws”) Prompt/motivate Carmen thru that strategyPrompt/motivate Carmen thru that strategy React to Carmen’s emotional and cognitive stateReact to Carmen’s emotional and cognitive state Employ interaction tactics based upon Carmen’s state; some focus on Employ interaction tactics based upon Carmen’s state; some focus on

cognitive state, others on emotional statecognitive state, others on emotional state


Questions About the “Gina Questions About the “Gina Model”Model” How appropriate is it for educational How appropriate is it for educational

applications?applications? It is based on clinicians’ counseling-oriented view of It is based on clinicians’ counseling-oriented view of

trainingtraining It is a dramatization of instructional interactionIt is a dramatization of instructional interaction

Built from a deconstructed scriptBuilt from a deconstructed script Empirical studies of tutorial interaction were Empirical studies of tutorial interaction were

neededneeded To see how this model applies to other educational To see how this model applies to other educational

settingssettings To determine which learner characteristics are most To determine which learner characteristics are most

relevantrelevant To study social interaction processes in such settingsTo study social interaction processes in such settings


Experimental Study Experimental Study Videotaped sessions of computer-Videotaped sessions of computer-

based learning with human tutors, over based learning with human tutors, over multiple sessionsmultiple sessions

Students read tutorial on line and Students read tutorial on line and perform series of exercises with Virtual perform series of exercises with Virtual Factory Teaching SystemFactory Teaching System

Johnson, Pain, Shaw, et al: IUI ’03, AIEd ‘03


Conclusions from StudyConclusions from Study Wide variation in learners’ preferred Wide variation in learners’ preferred

interaction stylesinteraction styles Some prefer collaboration, some prefer working Some prefer collaboration, some prefer working

alonealone Wide variation in confidenceWide variation in confidence

Between subjectsBetween subjects Over timeOver time

Tutor generally able to assess learner Tutor generally able to assess learner confidence, ability, preferred confidence, ability, preferred interaction styleinteraction style


Conclusions from Study (Cntd.)Conclusions from Study (Cntd.) Information used by tutor:Information used by tutor:

Expectations from knowledge of taskExpectations from knowledge of task Eye gaze, mouse locationEye gaze, mouse location Verbal feedback from studentVerbal feedback from student


Techniques for Promoting Techniques for Promoting Learner EngagementLearner Engagement Tutor phrased comments in order to Tutor phrased comments in order to

reinforce learner control and joint reinforce learner control and joint activity. E.g.:activity. E.g.:

““Why don’t you go ahead and read your tutorial Why don’t you go ahead and read your tutorial factory”factory”

““You want to save the factory”You want to save the factory” ““I’d skip this paragraph”I’d skip this paragraph” ““So why don’t we do that?”So why don’t we do that?”

Tutor avoided giving direct Tutor avoided giving direct instructionsinstructions

Except for operating the interfaceExcept for operating the interface


Theoretical Framework: Learner Theoretical Framework: Learner MotivationMotivation Motivational factorsMotivational factors

““Four Cs”:Four Cs”: CuriosityCuriosity ChallengeChallenge ConfidenceConfidence ControlControl

Learner goals and meta-goals:Learner goals and meta-goals: Persistent goalsPersistent goals Attitudes toward goal achievementAttitudes toward goal achievement


Role of Motivational FactorsRole of Motivational Factors CuriosityCuriosity

Employ tactics that promote inquiryEmploy tactics that promote inquiry ChallengeChallenge

Select tasks according to difficultySelect tasks according to difficulty Intervene in response to learner confusion, hard Intervene in response to learner confusion, hard

impassesimpasses ConfidenceConfidence

Regulate amount of feedbackRegulate amount of feedback ControlControl

Employ tactics that promote learner goal-settingEmploy tactics that promote learner goal-setting Learner goalsLearner goals

Employ tactics that promote learner goal identificationEmploy tactics that promote learner goal identification


Theoretical Framework: Theoretical Framework: Politeness (Brown & Levinson)Politeness (Brown & Levinson) Social actors motivated by face wantsSocial actors motivated by face wants

Negative face: freedom of action and freedom from Negative face: freedom of action and freedom from imposition; autonomyimposition; autonomy

Positive face: consistent self-image, and desire that self-Positive face: consistent self-image, and desire that self-image is appreciated and approved of by othersimage is appreciated and approved of by others

Face-threatening acts pervasive in interactionFace-threatening acts pervasive in interaction Warnings, offers, promises, challenges, emotional Warnings, offers, promises, challenges, emotional

displaysdisplays Face threat depends upon power, distance, Face threat depends upon power, distance,

ranking of threats due to social contextranking of threats due to social context Social actors employ politeness tactics to Social actors employ politeness tactics to

mitigate face threatmitigate face threat


Role of Politeness Factors in Role of Politeness Factors in Tutorial InteractionTutorial Interaction Common tutorial actions (advice, hints) are face-Common tutorial actions (advice, hints) are face-

threatening actsthreatening acts Tactic failures impact agent’s positive faceTactic failures impact agent’s positive face Face threat depends upon distanceFace threat depends upon distance

Distance depends on duration of interchange, established Distance depends on duration of interchange, established trust, learner’s negative face wants (preference for autonomy trust, learner’s negative face wants (preference for autonomy vs. collaboration)vs. collaboration)

Choose tactics to promote learner positive face, Choose tactics to promote learner positive face, mitigate negative face threatmitigate negative face threat

By promoting shared goalsBy promoting shared goals By avoiding direct instructionsBy avoiding direct instructions By reinforcing positive (self-)assessment of goal achievementBy reinforcing positive (self-)assessment of goal achievement When dictated by social distance, learner motivational factorsWhen dictated by social distance, learner motivational factors


Example Interaction TacticsExample Interaction Tactics Rhetorical requests to give hintsRhetorical requests to give hints

““Can I give you a hint? Try this…”Can I give you a hint? Try this…” Question reinforces learner negative face; failure to Question reinforces learner negative face; failure to

wait for answer avoids positive face threatwait for answer avoids positive face threat Hints phrased as questionsHints phrased as questions

““Do you want to do Do you want to do xx?”?” Reinforces learner control (positive face), can Reinforces learner control (positive face), can

influence learner goals (positive face)influence learner goals (positive face) Hints as suggestionsHints as suggestions

““You could do You could do xx.”.” Similar face effects as questionsSimilar face effects as questions


Interaction Tactics (Cntd.)Interaction Tactics (Cntd.) Hints as suggestions of joint goalHints as suggestions of joint goal

““Let’s do Let’s do xx.”.” Suggestion mitigates negative face threat; Suggestion mitigates negative face threat;

reference to joint goal influences positive face reference to joint goal influences positive face wants; depends on learner autonomy preferenceswants; depends on learner autonomy preferences

Hints as references to tutorial authorsHints as references to tutorial authors Deflect blame for face threat to authorsDeflect blame for face threat to authors

Imperative hintsImperative hints Used only when blame is deflected (I.e., to Used only when blame is deflected (I.e., to

interface), or possibly when distance is reducedinterface), or possibly when distance is reduced


SI Text GeneratorSI Text Generator Generates text for interaction tacticsGenerates text for interaction tactics

Input: type of intervention, object(s) of intervention, Input: type of intervention, object(s) of intervention, style of reference (e.g., as joint goal, user’s goal, style of reference (e.g., as joint goal, user’s goal, etc.)etc.)

Parameters: social distance: importance of Parameters: social distance: importance of motivational influencemotivational influence

Common tutor wording styles captured and Common tutor wording styles captured and codifiedcodified

Wording style chosen randomly if not specifiedWording style chosen randomly if not specified


Architecture of SI SystemArchitecture of SI System


Tracking Learner Attention & Tracking Learner Attention & ConfusionConfusion Useful:Useful:

To detect proactive interaction opportunitiesTo detect proactive interaction opportunities Avoids learner frustrationAvoids learner frustration

To avoid inappropriate interruptionsTo avoid inappropriate interruptions Avoids negative effects on learner affect, trustAvoids negative effects on learner affect, trust

To assess learner engagementTo assess learner engagement Helps determine objectives for interaction tacticsHelps determine objectives for interaction tactics


Assessing Learner Attention: Assessing Learner Attention: MethodsMethods Track learner’s interactions with Track learner’s interactions with

tutorial and simulation interfacetutorial and simulation interface Track learner gazeTrack learner gaze Fuse using Bayesian techniques to Fuse using Bayesian techniques to

determine focusdetermine focus Instrument tutorial with expected Instrument tutorial with expected

learner goals, time demandslearner goals, time demands Infer overall learner activity (e.g., scan Infer overall learner activity (e.g., scan

vs. problem-solve), overall vs. problem-solve), overall engagement, specific impassesengagement, specific impasses


Student InterfaceStudent Interface


Student Interface with Student Interface with Motivational Self-AssessmentMotivational Self-Assessment


VFTS PlansVFTS Plans


Inferring Learner FocusInferring Learner Focus


Tracking Focus of AttentionTracking Focus of Attention


A Final Comment: Social Actors A Final Comment: Social Actors or Dramatic Actors?or Dramatic Actors? Social actor view:Social actor view:

Interaction with agent is a social interactionInteraction with agent is a social interaction Agent should act in a manner consistent with human social Agent should act in a manner consistent with human social

interactioninteraction Inspiration: theories of social interactionInspiration: theories of social interaction

Dramatic actor view:Dramatic actor view: Interaction is part of an unfolding storyInteraction is part of an unfolding story Agent should act so as to contribute to the story and its messageAgent should act so as to contribute to the story and its message

Make action clear, understandable, and engagingMake action clear, understandable, and engaging Inspiration: theories of drama and narrativeInspiration: theories of drama and narrative

Intersection: Intersection: Social theories of presentation of self – e.g. GoffmanSocial theories of presentation of self – e.g. Goffman Relationship between rhetoric and drama in effective Relationship between rhetoric and drama in effective

communicationcommunication

Social Interaction with Agents

Documents

Transcript of Social Interaction with Agents