Cognitive Design

Nishida Lab

Yoshimasa Ohmoto

Conversational Informatics, June 15th, 2016

What is Cognitive Design?

◎Purpose ◎Designing the interaction with artifacts based on a physio-

cognitive approach.

◎“Remaking products, services and organizations to fit and enhance the human mind”

◎Communication◎The process by which people exchange information or

express their thoughts and feelings [Longman]

◎The word “communication” will be used in a very broad sense to include all of the procedures by which one mind may affect another. [Shannon and Weaver]

◎From the perspective of cognitive design◎Most important points are “exchanging intentions” and “sharing

thoughts and feelings”

Communication model of Shannon and Weaver

◎Send messages from Sender to Receiver

◎This model is very simple but very important

◎This model includes exchange and share of the information (intention in communication).

◎It is different between information in source and that in destination because of the noise.

◎The intersection of the information is sharing part.

[Claude E. Shannon and Warren Weaver. “The mathematical theory of communication” -- University of Illinois Press, 1949]

Sender Receiver

transmitter receiverInfo. source

destination

noise

cannel

Frameworks to interpret social signals

◎A framework coming from linguistics◎This framework approaches social interaction from the viewpoint

of dialogue understanding.◎Vocal prosody and gesture are treated as annotations of linguistic

information. [Cassell, 2000]◎This framework has proven useful for computer graphics and language

production systems.

◎ It has been difficult to apply to dialogue interpretation, perception, and unconscious behaviors.

◎A framework coming from cognitive psychology◎This framework focuses on emotion.◎The key idea is that people perceive others’ emotions through

stereotyped displays of facial expression, tone of voice etc. [Picard, 1997]◎The simplicity and perceptual grounding of this theory has recently given

rise to considerable interest in the scientific and engineering literature.

◎ It has been difficult to measure these displays and even more difficult to use them to build practical applications.

A framework coming from cognitive design

◎A person’s attitude or intention is conveyed through unconscious behavior (social signals).

◎such as changes in the amplitude and frequency of prosodic and gestural activities.

◎The relationship between interaction features are communicated rather than content of the dialogue or emotion.

◎Total information from auditory perception and visual perception is synthetically interpreted to find the relationship.

◎This is “multi-modal” interaction.

◎There are many signals but a few signals can be used.

Influence of social signals

◎Explicit signals (verbal expressions) unilaterallycommunicate final and conclusive information.

◎Social signals bilaterally influence to thoughts and behavior.

◎Most of the social signals are unconscious and reliable.

◎For example,

◎Mimicry (a kind of social signals) triggers a positive response among all of the communication partners and own. [Chartrand, Maddux & Lakin, 2005]

◎People think they understand after nodding behavior. [Brinol & Petty, 2003]

An example of influence

◎You tube video

Car

Driving

Tree

Absorbing CO2

Human

Walking

Pretty dog

Following the person

Cognition

??

Car

Driving

Tree

Absorbing CO2

Human

Walking

Pretty dog

Following the person

Information

Environment

Emotion

The largest object

Shape

Follow the person

Near the car

Car

Driving

Tree

Absorbing CO2

Human

Walking

Pretty dog

Following the person

Small hatchback car

On the way home

Road side trees

Absorbing CO2

This is “me”

Walking across the road

Cool dog

Following but not a pet

Different cognition!

Small hatchback car

On the way home

Road side trees

Absorbing CO2

It is “me”

Walking across the road

Cool dog

Following but not a pet

Information

Environment

Projection

Internal, objective

External, subjective

External, objective

Emotion

Internal, subjective

objective

subjective

external

internal

Information

Emotion

EnvironmentProjection

Speculating Intention

◎Intention◎We cannot objectively find “intention.”◎We speculate depending on situations.

◎People cannot report intentions precisely.

◎People’s memories are easily rewritten. [*]

◎Hypothesis◎Intentions in interaction are

extemporarily shaped based on the underlying and ambiguous wish.

◎The external stimulus◎partner’s behavior, new information etc.

◎The internal pressure◎reflection of own activity, own preferences

etc.

[*] Edelson, M. and Sharot, T. and Dolan, R.J. and Dudai, Y.: Following the Crowd: Brain

Substrates of Long-Term Memory Conformity. Science, vol. 333, num. 6038, pp 108-111, 2011.

Bubbling intention

Analysis of the behaviors of a good facilitator

◎Discussion is critical to build consensus in our social activities. ◎The facilitator can conduct smooth and effective discussions by

mediating between participants in the discussion. [Reagan-Cirincione 1994]

◎This is a kind of speculating intentions.

◎We investigated how experienced facilitators control the discussions effectively and smoothly.◎We conducted experiments in face-to-face discussion for

statistical analysis to find the factors which caused good facilitating behaviors.

◎by context, agreement or disagreement, and nonverbal and paralinguistic data

◎Discriminant analyses were applied to the data.◎The facilitator paid attention to not only the context of the

discussion but also the individual behavior◎The participants conveyed their requests to the facilitator by

using nonverbal behavior.

Problems

◎The facilitator mediates between participants.

◎The facilitator has to support the group's social and cognitive processes, allowing participants to focus their attention on more substantive issues and ultimately reach the most appropriate solution to a problem.

◎It is not known that the good facilitator decides the facilitating behavior based on what kinds of information.

◎Most of previous works did not directly investigate what information the facilitator pays attention to.

Outline of the experiment

◎To obtain the participants' behavioral data in face-to-face discussion

◎Five participants formed a group

◎four undergraduate students ("discussers“) and one facilitator

◎The discussers were acquainted with each other.

◎A total of five groups participated (a facilitator and 20 students)

◎"the plan of overnight trip in this summer.“

Experimental settings

◎sitting on seats in a circle

◎There were no table and they could not take notes

◎Obtained data

◎head orientation◎By a motion capture system

◎Voice◎By throat microphones

◎Video◎By four HD-video cameras

Video analysis: facilitating behavior

◎We could classify facilitating behavior to four types in discussions.

◎Diverging◎the facilitator interposed someone to encourage the

divergent discussion

◎Converging◎the facilitator interposed someone to encourage the

convergent discussion.

◎Conflicting person◎the facilitator asked a discusser who had conflicting opinion

against the last speaker.

◎Objectifying◎the facilitator asked the last speaker for objectifying his/her

opinion.

Results: The discriminant analyses

◎Linear discriminant analyses were applied to classify the facilitating behaviors◎each facilitating behavior type and the others.

◎We could selected which facilitating behavior was appropriate by the discriminant functions◎The concordance rate between the selected behavior

and the actual behavior was 78.6%.

◎the explanatory variables could correctly classify the facilitating behaviors.

No-good facilitator

◎The feature of the facilitation by the no-good facilitator

◎The number of the facilitating behavior was smaller (4.3 times / 10min.) than that of the good facilitator (7.1 times / 10min.).◎No-good could not decide which facilitating behavior had to

be selected.

◎No-good could not interpose at the appropriate time. ◎He could not understand the context of the discussion.

◎No-good often changed the discussion topic◎He could not control the discussion

◎These were some of the reasons why no-good could not conduct the smooth and effective discussion.

Conclusions

◎The statistical analysis to find the factors which caused good facilitating behaviors

◎We could suggest some factors to which a good facilitator used to conduct smooth discussion.◎Whether the discussion was diverging or converging.

◎Which discussers had a similar opinion.

◎the total time of speaking

◎the total time which discussers paid attention to the facilitator

◎The facilitator controlled the discussions by the facilitating behaviors.

◎We achieved 78.4% accuracy in classifying four types of the facilitating behaviors.

Behavior model based on bilateral speculation

◎The facilitation is basically one-way speculation of intention.◎Social signals bilaterally influence to thoughts and

behavior.

24

[Ohmori. 2010]

◎The behavior model has nested structure.◎Lv0: behaving based on my own

intention

◎Lv1: behaving based on partner’s intention

◎Lv2: behaving based on consideration that the partner speculates my own intention

◎Lv3: ・・・・・・

Induction of Intentional Stance in HAI

◎We want to develop an embodied conversational agent that is regarded as a social partner, not just multimodal interface. ◎The interaction behavior is very different.

◎The mental stance of people when they interact with agents is usually different from when they interact with humans. ◎Social agency theory proposes that more social cues lead to more

social interaction

◎But a result of a previous study was the exact opposite.

◎The purpose of this study is to investigate how to induce the intentional stance in human-agent interaction.

The agent has information for me.

Hello, Dylan! How are you doing?

The mental stances

When the guardian robot stands by a gate,

a person who try to pass the gate is caught by the robot

The robot think “I do not permit the passage because this gate is now broken.”

When the sensors of the robot detect a person, the robot catches me.

The actuator and the computer controls the robot body.

Intentional stance

Physical stance

Design stance

We propose a method to induce the intentional stance, implement the method in an

agent and experimentally investigate the effect of inducing the intentional stance.

Methods for Presenting Goal-oriented Behavior

◎We expected that making a person speculate the agent's subjective perspectives is important.

◎We implemented two methods for presenting goal-oriented actions:◎showing a trial-and-error progression towards a goal using

multimodal behavior (trial-and-error agent)◎People construct an imperfect behavior estimation model

◎because it is difficult to precisely interpret multimodal behavior in terms of estimating the behavioral goal.

◎displaying the agent’s behavioral intention using text (text display agent)◎This method encourages the construction of an complete behavior

estimation model.

◎Both of them presented goal-oriented behavior.

◎We investigate how effective they are at inducing the intentional stance.

Outline of architecture

◎The agents decide their behavior through three layers.

◎The goal layer shows the task goal. ◎The first goal is predefined.◎The goal is changed depending on

the others’ behavior.

◎The behavior category layer shows the category of possible behavior. ◎Each category is a sub-goal of a

concrete behavior.

◎The concrete behavior layer shows the concrete behavior produced by an agent. ◎Each category contains some of the

concrete behavior.

Goal layerchasing long time,

equalizing the time of a tagger,Equalizing the number of a tagger

Behavior category layer”chasing”, ”provocation”,

”dissatisfaction”, ”escape” and ”appeal”.

Concrete behavior layer

Etc.

Waving hand near the player

Standing near

Middle distance

Large distance

Sometimes looking around

Stay behind an obstacle

Presentation method

◎Trial-and-error agent

◎The agent produces a concrete behavior with higher expression strength than before.

◎The changes for achieving goal are made in a trial-and-error fashion.

◎Text display agent

◎The agent produces patterns of text corresponding to the behavior category.

◎If only presenting goal-oriented behavior is important, people should take the intentional stance.

Jumping and waving hand

Small distance

Waving hand near the player

Standing near

Middle distance

Large distance

Provocation Chase

Crapping hands

Sometimes looking around

Stay behind an obstacle

Appeal

Lv 1

Lv 3

Lv 2

Success

Fail

Provocation Chase Appeal

“Try to catch me!”

Following near

“Hey, come on!”

Provoking

“Escape or tagged”

Chasing carefully

“I’m stay here!”

Looking around

Hiding in the house

Fail

Outline of an Experiment

◎To investigate the effect of inducing intentional stance, we conducted an experiment using the two agents:◎a ”trial-and-error agent” and a ”text display agent”◎The behavior planning of the task and the expressions were

automatically controlled.

◎We used a virtual reality ”customized tag game” as an interaction task.◎Two human participants and one agent joined the game.◎Participants could freely communicate and perform the game.◎They could add new rules, cooperate with others, ignore the agent, an

so on.

◎We compared two types of the experimental results.◎we analyzed the number of communicative actions towards the

agent throughout the experiment. ◎we asked the participants to answer questionnaires after the

experiment.

Experimental settings

◎The player’s virtual avatar could be controlled by their body motions using a motion capture system. ◎The participants could easily interact using body motions.

◎”The agent can recognize your speech. But the agent does not respond to your speech and actions except when they need to respond for performing the task.”

Results: Analysis of the number of communicative actions towards the agent

◎The purpose is to investigate whether performing goal-oriented behavior influenced the communication behavior. ◎We divided the time series of the experiment evenly into four periods.

◎We compared the number of communicative actions in the 2nd and 4th periods.

◎The number in the 4th period was significantly less than that in the 2nd period only for TD-agent (p = 0.0003). ◎All of the participants decreased the number of com. actions.

◎They took the design stance towards this agent in the end.

◎The number in the 2nd period for TD-agent was more than that for TaE-agent (but there was no significant difference).

The design stance interaction The intentional stance interaction

Results: Questionnaire analysis

◎ The purpose is to investigate how the presentation method influenced participants’ final subjective impressions in the end.

◎ We performed a Mann-Whitney U test on the data in the questionnaire.

• How human-like do you feel that the agent is?

– relatively higher but no significant difference.

• How strongly do you think the agent has a definite goal?

– The TaE-agent had significantly more definite goals than TD-agent (p = 0.039).

• How smart is the agent?– The TaE-agent was significantly smarter than

TD-agent (p = 0.015).

• How well does the agent understand your intentions?

– The TaE-agent understood their intentions better than TD-agent (p = 0.054, marginally significant difference).

– But the average scores were not high.

• How much did you enjoy the game?– The participants enjoyed the game

significantly more with TaE-agent than with TD-agent (p = 0.025)

– The scores for both agent were fairly high.

Discussions

◎To sum up the experimental results: ◎the trial-and-error agent could not quickly induce

participant’s intentional stance

◎but, when induced once, the participant maintained intentional stance long time.

◎We suggest that the process of constructing a behavior model influences the mental stance of the participants.◎An obvious presentation of the inner state is not always

effective.

◎Therefore, for the text display agent, the participants did not maintain the intentional stance.

◎On the other hand, two participants could not understand the trial-and-error behavior.◎This is future work.

Conclusion

◎To establish social partner relationships between humans and agents, we tried to induce intentional stance by presenting goal-oriented behavior.

◎We implemented two methods.

◎We conducted an experiment to investigate the effect of inducing intentional stance.

◎Trial-and-error agent and text display agent

◎We suggest that both of the goal-oriented behaviorand the continuous model estimation are needed to induce and maintain the intentional stance.

Dynamically estimating emphasizing points

◎We often make a decision interactively as a group with our friends and advisers.◎We dynamically and interactively change the factors

that we emphasize (termed “emphasizing points”).

◎E.g. travel planning: Country? Budget? Members?

◎It is difficult even for human◎to directly estimate changing emphasizing points.

◎to estimate internal states of communication partners.

◎I think “intention” is constructed throughinteractions with humans, artifacts and environments.

We tried to estimate the emphasizing points through

decision-making interaction.

Method

◎We dynamically estimating emphasizing points for decision-making.◎using physiological responses and verbal and

nonverbal information.◎Keywords, nodding, SCR, heart rate, skin temperature

◎Estimate based on active demands and passive responses◎Active demands◎Keywords which are related to demands

◎Comparative results of proposals

◎Passive responses◎Keywords which are related to preferences

◎Nonverbal reactions

◎Physiological indices

HAI demo

Evaluation Experiment

◎Objectives

◎To confirm accuracy of our estimation method

◎To confirm that estimation of emphasizing points were useful to understand causes of decision-making

◎Measured data

◎Physiological indices◎SCR

◎Electrocardiograph (LF/HF)

◎Skin temperature of a finger

◎Videos

◎Questioner to confirm the accuracy and degree of satisfaction

Experimental setting

◎Each participant interacted with a WOZ controlled agent to design two types of mobile robots.

◎In a session: one participant and one experimenter (WOZ operator)

◎Total: 7 participants (average 20.6 years old)

To confirm accuracy of the estimation

◎Participants answered top three emphasizing points out of 23 factors.

◎We compared the answers with the results by each method (t-test)

average

S.D.

Ours Gradual

提案ours gradual

To confirm that estimation were useful

◎Participants answered their degree of satisfaction (-3～+3) (Wilcoxon signed-rank test)

◎Which method provided satisfied proposals (-3～+3) (sign-test)

average

S.D.

Ours Gradual

提案ours gradual

average

S.D.

Ours > gradual

（提案＞既存）Ours > gradual

Conclusion

◎We proposed one of the methods to estimate emphasizing points based on human behavior.

◎We confirmed

◎Proposed method could achieve relatively accurate estimation of emphasizing points.

◎By using proposed method, participants were more satisfied in decision-making task.

Extended Methods to DEEP for Group Decision-making

◎We often make a decision interactively as a group◎We want to make an Embodied Conversational Agent to

join the group decision making.

◎The purpose of this study is to extend DEEP to estimating the emphasizing points of a group.

◎We observed and investigated interaction processes in group decision making with friends and advisers.

◎We proposed two extended methods corresponding to the different interaction processes.

◎We then conducted an experiment to evaluate the methods using ECAs.

◎In conclusion, these proposed methods accurately estimate proposals and satisfy participants in the appropriate group.

Background

◎In many cases of group decision-making, people often have conflicting opinions.

◎People consider not only their demands but also the effects on their relationships.

◎We extend the our proposed method to estimating the emphasizing points of groups (“group-DEEP”).

◎Based on typical verbal and nonverbal information and physiological indices◎Since we can only use basic and general representations of

verbal and nonverbal information, to cover the loss, we use physiological indices

Interaction process: Expressing Opinion group

◎First, the members clearly expressed which proposal was better and why.

◎In the case of conflicting opinions, the members never changed their own opinions

◎They only changed when their partner proved them wrong.

◎The emphasizing points of the group only contained those held by all members.

◎We expect that the emphasizing points of the EO group can be estimated by focusing on clearly accepted opinions.

Interaction process: Avoiding Conflict group

◎The members often used ambiguous words (e.g. relatively, so-so, not so bad …) to express their opinion.◎They repeatedly confirmed their partner’s emphasizing points

and tried to establish consensus.

◎They often retracted their opinions during the group conversation if their partner did not emphasize that point.◎In addition, if the partner did not challenge their opinion, they

regarded their opinion as accepted.

◎The AC group’s emphasizing points contained all not refused opinions.

◎We expect that the emphasizing points of the AC group can be estimated by focusing on the responses to a partner’s opinion.

Two methods of group-DEEP

◎Union-base◎The union-based method focuses on the responses to the other members’

opinions.◎We expect that this method will be used by the AC group.◎Estimated emphasizing points contain as many emphasizing points as

possible as identified by members.

◎ Intersection-base◎The intersection-based method focuses on clearly accepted opinions. ◎We expect that this method will be used by the EO group.◎Estimated emphasizing points only contain the emphasizing points shared

by all the members.

Positive opinions

Union-base

Intersection-base

Evaluation experiment

◎Objective

◎to investigate whether we should change the estimation method depending on the interaction style.◎union-based method or intersection-based method

◎avoiding conflicts or expressing opinions

◎Participants

◎16 Japanese college students (all female).◎The participants were divided into eight pairs.

◎Each participant interacted with two agent which was implemented different estimation method◎Union-base and Intersection-base.

◎Task

◎Choosing a present

Experimental setting

◎Each participant interacted with a WOZ controlled agent.

◎The ECA capture keywords, nodding motion and physiological indices (electrocardiogram (LF/HF) and SCR)

displaying an ECA

skin conductance

response (SCR)：electrodes placed on two

fingers of a left hand

electrocardiogram：measured by connecting

electric poles with paste

to the participant’s left

and right sides and to

both ears for grounding

and reference.

Results of accuracy of ECA’s final proposal

◎All the participants chose their best proposal out of 40 prepared proposals at the end of both sessions.

◎We calculated the concordance rates between the proposals chosen by the participant and the proposals estimated by each ECA.

SS df MS F pGroup 0.031 1 0.031 0.14 0.72error 3.2 14 0.23Method 0.28 1 0.28 1.5 0.25error 2.7 14 0.19interaction 1.5 1 1.5 8.0 0.014*Total 7.7 31

SS MS F p

Group (union-base) 0.56 0.56 2.7 0.11

Group (intersection-base) 1.0 1.0 4.8 0.038*

error 0.21

Method (AC group) 0.25 0.25 1.4 0.27

Method (EO group) 1.6 1.6 8.1 0.013*

error 0.19

Results of participant satisfaction with HAI

◎The participants answered rating questions regarding the level of satisfaction with HAI using a seven-point scale.

◎We then calculated averages in each group and each method.

SS df MS F pGroup 0.78 1 0.031 0.14 0.72error 28 14 0.23Method 0.031 1 0.28 1.5 0.25error 20 14 1.5interaction 9.0 1 9.0 6.2 0.026*Total 59 31

SS MS F pGroup (union-base) 2.3 2.3 1.3 0.27Group (intersection-base) 7.6 7.6 4.3 0.047*error 0.21Method (AC group) 5.1 5.1 3.5 0.084+Method (EO group) 4.0 4.0 2.7 0.12error 1.5

Conclusion

◎The purpose is to extend our proposed method to estimating the emphasizing points of a group.

◎We conducted an experiment and confirmed that the interaction process differed between the group that tried to avoid conflict and those that tried to express their opinions.

◎From the results of the experiment, we propose two extended methods: the union-based method and the intersection-based method.

◎We also conducted an experiment to evaluate the methods using ECAs.

◎As a result, we suggest that the methods accurately estimated proposals and satisfied participants in the appropriate group.

Behavior model based on mutual speculation

◎We basically consider one-way speculation of intention.

◎We especially focused on internal pressure.

54

◎The behavior model of bubbling intention needs mutual speculation.

◎We also have to consider external stimuli.

◎The external stimuli are mainly produced communication partner(s).

The effect of convergent interaction using subjective opinions

◎ DEEP encouraged decision-making by awakening the intrinsicemphasizing points.

◎ Extrinsic subjective interpretations, such as friend’s opinion and word-of-mouth advertising, also encourage decision-making◎ because they provide case examples to interpret the factors we have

to consider and emphasize to reach an appropriate decision.

◎ In this study, we investigated the effect of extrinsic subjective interpretations of the adviser in interactive decision-making.

◎We conducted an experiment that compared the results of interactive decision-making with two types of ECAs◎ A facilitative agent: who provided subjective opinions to realize

divergent and convergent processes in decision-making◎ An estimation agent: who only provided proposals that reflected the

emphasizing points of each participant

◎ As a result, we can confirm that the facilitative agent could improve the interaction◎ the participant’s satisfaction of interaction with the ECA, the

naturalness of ECA’s interaction, and the impression of decision-making process.

Background

◎We have to materialize ambiguous demands in decision-making.

◎Subjective information provide case examples to interpret the factors we have to consider and emphasize to reach an appropriate decision.

Extrinsic stimulus

Intrinsic consideration

Well…”green dress”

Red is also good.The sleeveless dress doesn’t

suit you.

I like green dress.

But I already have long-

sleeved dress.

Red is also good.The sleeveless dress

doesn’t suit you.

Green!

Important?

Important?

Important?Important!

Not

focused

Not

important

Important!Important?

NO (first)

Method

◎To “facilitate” decision-making by controlling divergent and convergent process.Switching rules between the divergent and convergent

• There are more than three emphasizing points, with a degree of emphasis of more than one.

• The degree of emphasis does not change during the interaction.

or• The user offers a convergent opinion.

Divergent• The agent provides a small nod once in reaction to the

user’s utterance.• The frequency of providing a new proposal is low.• The agent provides a new proposal after she explains

three emphasizing points.• The furthest proposal from the previous one is selected

as a new proposal. • The degree of emphasis decreases if the emphasizing

point is not explained in the previous proposal.

Convergent• The agent provides two nods in reaction to the user’s

utterance.• The frequency of providing a new proposal is high.• The agent provides a new proposal after she explains one

emphasizing point, which is a recommendation.• The nearest proposal to the previous one is selected as a

new proposal.• The degree of emphasis decreases only when the

emphasizing point is refused in the previous proposal.

YES (last)

Selection method of a next proposal

Current proposal

Agent’s

preferences

Divergent Convergent

Next

proposal

farthest

Nearest

Next

proposal

Agent Control

Estimation of emphasizing points

Switch or not between divergent

and convergent

Next proposal

Agent behavior

ResultsEvaluating estimation

process

Results

Select next proposal

Next proposal

Agent behavior

Results

Select next proposal

Facilitative agent Estimation agent

Divergent

Social signals

Social signals

Estimation of emphasizing points

Evaluation experiment

◎ Task: gift-wrapping (30 factors)◎ The participants did not know what was appropriate gift-wrapping.◎ The participants would take advice from the agent because they tried

to predict what the receiver of the gift would like.

◎ Participants: 20 Japanese college students (all female)◎ The reason why the participants were females was that there was

motivation gap for the gift-wrapping task between males and females.

Example of the experiment

Result of reaction latency analysis

◎To investigate whether participants attentively listening proposals by the ECA, we extracted a reaction latency for each participant.

◎There is a significant difference in the second half of the interaction. (p=0.029)◎There is also a significant difference between the reaction

latency in the estimation agent group in the first half and that in the second half.

When the participantinteracted with the estimation agent, she carefully thought about the proposal in the second half of the interaction.

→ participants did not care the interaction with the agent

Result of analysis of emphasizing point changes

◎To investigate whether the control of divergent and convergent processes influences participants’ emphasizing points, the participants chose emphasizing points that they changed during the interaction.

◎The number in the facilitative agent group was significantly higher than that in the estimation agent group (t=-2.63, p<0.05).

Because participants made their decision only based on intrinsic emphasizing points, they could not recognize changes to the emphasizing points.

→ subjective opinion influence the recognition of search space

Results of questionnaires

◎To investigate the impression of the decision-making process with ECAs, the participants answered three rating questions on the ECA’s behavior using a seven-point scale.

◎We conducted Wilcoxon signed-rank tests on each questionnaire result.

Conclusion

◎We investigated the effect of the subjective information by the agent in interactive decision-making.

◎We conducted an experiment that compared the results of interactive decision-making with two agents◎A facilitative agent: who provided subjective opinions that

were external stimuli for decision-making.◎An estimation agent: who only provided proposals that

reflected the emphasizing points of each participant.

◎As a result, ◎we evaluated a facilitative agent who provided subjective

opinions could encourage interaction in decision-making.◎we found that the facilitative agent led to higher scores for

participant satisfaction regarding agent’s interactions, the naturalness of agent’s interaction, and positive impressions of the decision-making process.

The Effects of Extended Estimation on Affective Attitudes in a Series of Tasks

◎Agent characters that interact with users often appear on several situations. ◎These characters are regarded as multi-modal interfaces, but are not

regarded as social partners.◎-> Our aim is to develop an agent that could be regarded as a

communicative partner like human in continuous interaction.

◎Active affective attitude is needed.

◎In previous studies, we proposed methods to support a decision-making process◎Dynamic Estimation of Emphasizing Points (DEEP)◎“Emphasizing points”: we have to consider and emphasize them to reach

an appropriate decision.

You are tired because you had hard work

yesterday. I provide easy training today.

OK! Today’s training menu

is a little easy.

Purpose

◎This study aimed to propose a method to induce an user’s active attitude in decision-making process in a series of interactions. ◎If the agent cannot induce the attitude, the users do not

demonstrate and share their own preferences, mental attitudes, and inner states with the agent.

◎We used historical estimated emphasizing points within a series of tasks to estimate emphasizing points within a new but similar task. (“extended estimation”)

◎We expected the provision of consistent estimation, using accumulated data on interactions, to induce a positive human attitude toward the agent and the interaction.

TaskTask TaskTask Task Task

Stored historyStructured knowledge

Estimation

Basic method to estimate emphasizing points

"facilitative DEEP" (fDEEP)

Emphasizing factors

Candidates DB

Input

Verbal

Nonverbal

Physiologic

al

Proposal

s

Estimation of emphasizing

points

Proposal generationIntroduction planning

Output

User’s demands

Agent’s

Opinions

Extended estimation through maintenance of emphasizing points within a series of

interactions

Emphasizing factors

Candidates DB

Input

Verbal

Nonverbal

Physiologic

al

Proposal

s

Estimation of emphasizing

points

Emphasizin

g factor DB

Stored

and

structured

Relational

network

Convert

and

compleme

nt"facilitative DEEP with extended estimation" (feeDEEP)

Output

User’s demands

Agent’s

Opinions

Proposal generationIntroduction planning

Outline of the human-agent interaction

Repeating and conversing

Asking emphasizing points for the task

User’s utterances and reactions

Proposals including user’s demands and agent’s

opinion

Emphasizing pointsB

uild

ing a co

nsen

sus b

etween

h

um

an an

d agen

tin

an

interactio

nal series o

f tasks

Common ground

Proposals including common ground factors

User’s utterances and reactions

Emphasizing factors

Experiment

◎The purpose was to investigate how feeDEEP affects the efficiency of the decision-making process and impressions related to agent behavior in a series of tasks

◎The participants were asked to coordinate a new living space.◎The primary task included three tasks.◎furniture selection, electronics selection, and living space planning.

◎The emphasizing points contained 16 factors.◎relaxing, natural, for work, clean, leisure, high spec and so on

The feeDEEP agent provided proposals using maintained emphasizing factors in planning.

Selection Planning Proposals using maintained

factors

furniture

electronics

planningsimple relaxing

natural

leisure

work space

clean

high spec

relaxing

feeDEEP

fDEEP

Converting by

the predefined rules

Relaxingand leisure

Naturaland relaxing

Integration of the emphasizing points through a series of interactions

Experimental settings

◎Participants

◎11 participants (8 males and 3 females) interacted with the feeDEEP agent (feeDEEP group)

◎10 participants (8 males and 2 females) interacted with the fDEEP agent (fDEEP group)

Desktop

PC

Note PC

Polymat

e

Agent

Control PC

USB

camera

USB

camera

Experimente

r

Participant

60-inch Monitor

Microphone

The agent and the task were displayed on the monitor.

Skin Conductance Responses and electrocardiogram were recorded. These are reflected mental

states to the agent proposals in real-time.

The experimenter manually inputted verbal reactions. The inputted words were listed

in advance.

Results of interaction behavior

◎To investigate whether the extended estimation contributed to effective decision-making, we counted the number of proposals from the agent in the 2nd selection task and the planning task.

◎We performed a paired t-test on the data of each group.◎There was a significant difference in the feeDEEP group (p < 0.05).

◎There was no significant difference in the fDEEP group (p = 0.26).

◎We performed an unpaired t-test on the data in the planning. ◎The value for the feeDEEP group was significantly less than that for the

fDEEP group (p = 0.0078).

These results indicate that the feeDEEP agent achieved more

effective decision-making support in the planning task

than the fDEEP agent.

4.9

2.6

5.3

4.5

0

1

2

3

4

5

6

7

8

second selection planning

The feeDEEP-group The fDEEP-group

＊

＊

Results of the questionnaire

◎The purpose of this analysis was to investigate how the extended estimation influenced the participants' subjective impressions related to the agent's behavior. ◎The participants answered six questions using a seven-point scale.

◎We performed a Mann-Whitney U test on the data from the questionnaire.

5.7

4.8

5.3

5.7 5.6

3.5

5.6

3.4

5.34.9 4.9

5.3

1

2

3

4

5

6

7

How satisfied are you

with the final products?

How human-like do you

feel that the agent's

behavior was?

How natural do you feel

that the agent's

interaction was?

How satisfied are you

with the interaction

process?

How would you rate the

agent's level of effort?

How frequently did you

accept the agent's

proposals?

feeDEEP fDEEP

* *+

Discussions

◎An important finding was that the rate of participant acceptance of the feeDEEP agent's proposals was significantly low despite a decrease in the number of interactions.

◎ This was possible because the agent provided a consistent estimation for each participant due to the extended estimation.

◎We suggest that the feeDEEP agent induced an active attitudetoward the decision-making interaction.

◎The attitudes indicate that they regarded the agent as communicative.

◎To create a system that is user-centric, it is necessary for the user to maintain an active attitude toward the decision-making interaction in order to accomplish their goals.

Future work

◎In our previous work, we analyzed physiological indices (SCR and LF/HF values) that were obtained experimentally. ◎However, we could not include an analysis of these indices in

this paper.

◎We are now analyzing these data in detail to investigate the underlying reasons for these experimental results.

12

13

14

15

16

17

18

19

20

1 45

89

13

31

77

22

12

65

309

35

33

97

441

48

55

29

57

36

17

66

17

05

74

97

93

83

78

81

925

96

91

01

31

05

71

10

1

fDEEP group

12

13

14

15

16

17

18

19

20

1 50

99

14

8

19

7

24

6

29

5

34

4

39

3

44

2

49

1

54

0

58

9

63

8

687

73

6

78

5

83

4

88

3

932

98

1

10

30

10

79

11

28

1177

12

26

12

75

13

24

feeDEEP group

Conclusion

◎We investigated the effects of the consistent estimation of a human's preferences on the human's affective impressions.

◎We conducted an experiment to evaluate the effect of the method using two agents;◎a feeDEEP agent, which was proposed in this study

◎a fDEEP agent, which was proposed in our previous work.

◎The results showed that ◎ feeDEEP agent could reduce the number of interactions in the decision-

making process

◎ it also improved some of the affective impressions related to the agent's character.

◎ In addition, we found that the rate of participants' acceptance of the feeDEEP agent's proposals was significantly low. ◎This was possible because the agent provided a consistent estimation of

emphasizing points for each participant.

◎The participants' attitude indicates that they regarded the agent as being communicative.

(Again) Speculating Intention

◎speculating intentions ≠ understanding the parson’s thought

◎Hypothesis

◎Intentions in interaction are extemporarily shaped based on the underlying and ambiguous wish.

◎The external stimulus◎partner’s behavior, new information and so on

◎The internal pressure◎reflection of own activity and so on

Bubbling intention