Automated Negotiation Agents

Sarit Kraus Dept. of Computer Science

Bar-Ilan University

Negotiation

“A discussion in which interested parties exchange information and come to an agreement.” — Davis and Smith, 1977

What is an Agent?PROPERTY MEANING

Situated Sense and act in dynamic/uncertain environments

Flexible Reactive (responds to changes in the environment) Pro-active (acting ahead of time)

Autonomous Exercises control over its own actions

Goal-oriented Purposeful

Persistent Continuously running process

Social Interacts with other agents/people

Learning Adaptive

Mobile Able to transport itself

Personality Character, Emotional state

No Agent is an Island: automated agents negotiate with other automated agents

• Monitoring electricity networks (Jennings)• Distributed design and engineering (Petrie et al.)• Distributed meeting scheduling (Sen & Durfee, Tambe)• Teams of robotic systems acting in hostile environments

(Balch & Arkin, Tambe, Kaminka)• Electronic commerce (Kraus et al.)• Collaborative Internet-agents (Etzioni & Weld, Weiss) • Collaborative interfaces (Grosz & Ortiz, Andre)• Information agent on the Internet (Klusch, Kraus et al.)• Cooperative transportation scheduling (Fischer)• Supporting hospital patient scheduling (Decker & Jin)

Agents negotiate with humans

Training people in negotiations Trade agents for the Web Elves agents– representing

people

Plan of talk: agents negotiate with humans

Automated agent for bilateral negotiations with complete information: the fishing dispute (collaborators: Penina Hoz-Weiss, Jon Wilkenfeld)

Automated agent for multi-party negotiations: the Diplomacy game(collaborators: Daniel Lehmann and Eitan Ephrati)

On going work: learning, incomplete information; mediation(collaborators: Dudi Sarne, Barbara Grosz Lin Raz, Michal Halamish)

Fishing Dispute Negotiators: Canada and Spain Canada’s stock of flatfish decreases over the

years. Spain has fished this same stock of flatfish

for many years, but outside the Canadian exclusive economic zone (EEZ).

Canada would like Spain to restrict its fishing near her EEZ. Spain is dependent on fishing in the area outside the EEZ for employment and trade purposes.

Possible Outcomes An agreement on Total Allowable Catch (TAC). An agreement on limiting the length of the fishing

season. Canada enforces conservation measures with

military forces against Spain. Spain enforces its right to fish throughout the

fishery with military force against Canada. If the negotiation has not ended prior to the

deadline, then it terminates with a status quo outcome.

World State Parameters World state parameters are also negotiable

and affect the utility of players: Canada subsidizes removal of Spain's ships (0, 5,

10, 15, 20 ships). Spain reduces the amount of pollution caused by

the fishing fleet (0%, 15%, 25%, 50%). Canada imposes trade sanctions on Spain. Spain imposes trade sanctions on Canada

Fishing DisputeOutcomes

TAC Limit Season Opt Out Status Quo

World State Parameters

Canada subsidizes Spain reduces Canada imposes Spain imposes

ships Pollution Trade Sanctions Trade Sanctions

Negotiation Process Each of the parties can make requests, threats,

offers, conditional offers and counteroffers, as well as to comment on the negotiation.

The utility of each ending is affected by the period when the negotiation ended.

Canada loses over time since Spain continues to fish while negotiating. Spain gains over time for the same reason.

Spain Thule Canada Ultima

Negotiations in the Fishing Dispute

S

Spain offers to set TAC at 44 thousand tons.

C

Canada offers to set TAC at 18 thousand tons.

SE

Spain asks that Canada compensate Spain for Spain’s restricted fishing practices by replacing the income of twenty ships.

Other Negotiations Games Team Games (SPIRE); negotiations on coordination;

exchange of information; finding solutions is complex Competitive games: when agents can benefit from

reaching an agreement (also in bilateral games). Trade games: Monopoly, Traders of Genoa, Kohle, Kies &

Knete,Treasure game War games: Diplomacy, Risk Crisis games: Hostage Crisis.

Semi-cooperative games: Color Trail, Majority Game

Chess Programs play chess as well as people Programs play chess in a way much different

than people: they mainly search the game tree

. . .

. . .

A

. . .

B

. . .

A

0-1 +1 Evaluation

Final states. . .

Search tree for Tic-Tac-Toe

Fishing dispute vs. ChessFishing dispute vs. Chess

Type of game: crisis game vs. war game.Coordination game vs. zero sum game

Number of players: 2 Moves: simultaneously + negotiations vs.

sequentially– need to reach an agreement.

Number of pieces to move: no pieces vs. one piece at a time

Information: Complete information. Needed capabilities: Negotiation skills

vs. strategic skills.

Playing Techniques

NEGOTIATIONS Game theory techniques:

formalize the game; find an equilibrium; follow the equilibrium strategy.

Market techniques. Appropriate for games of many players that can exchange similar items.

Heuristics: domain specific; “advice” books; human like strategies

Markov Decision Processes. Modeling the opponent Learning from DB Learning from experience

CHESS

Heuristic Search

The Automated Negotiator Agent (fishing dispute)

•The agent plays the role of one of the countries.

•During the negotiation the agent receives messages, analyzes them and responds. It also initiates a discussion on one or more parameters of the agreement.

•It takes actions when needed.

Nash Equilibrium

An action profile is an order set a=(a1,…,aN) of one action for each of the N players in the game.

An action profile a is a Nash Equilibrium (Nash 53) of a strategic game, if each agent j does not have a different action yielding an outcome that it prefers to that generated when chooses aj, given that every other player i chooses ai.

Strategy of Negotiation

Formal strategic negotiation theory:

The agent is based on the a bargaining model. By backward induction the agent builds the strategy to be reached at each time period according to the sequential equilibrium

(Kraus, Strategic Negotiation in Multiagent Environments, MIT Press 2001).

When the agent plays against humans Not Enough

Heuristics

Automated agent: Using equilibrium strategy when playing against humans

Human negotiators do not use equilibrium strategies even though game is not complex and the automated agent finds equilibrium fast.

Not surprising: Kahneman & Tversky showed that humans do not use decision theory.

The agent using the equilibrium did not reach beneficial agreements.

Heuristics

• Negotiation tactics

• Attributes

• Risk Attitude

• Opting out

• Fine tuning

Attributes

• Number of points lower than the equilibrium utility value that the agent will agree to.• The number of fish ton (TAC) the agent will increase/decrease in his offer. •Sending the first message / waiting to receive a message.

• Full offer message or not.

Modeling the risk attitude of theopponent

The agent is always neutral toward risk, but is sensitive to the risk level of its human opponent and will change its view of the human’s utility function accordingly.

Risk attitude influences the agreement an opponent is willing to accept.

The agent begins with the assumption that its opponent is risk neutral. It uses a heuristic method to decide whether to change the estimation of the risk attitude of the opponent.

When the agent decides that its opponent is risk prone, it changes the opponent’s utility function. This leads the agent to a recalculation of his strategy.

1190.81349.6 1301.21289.4

1445.8 1434

0

200400

600800

1000

12001400

1600

P/P C-A S-A

Total

Agreements

Experiments Results

605

580

607.5

580

560570580590600610

HumanCanada

Agent Canada

684.4 610.5

846.7 772.4

0

200

400

600

800

1000

Human Spain

Agent-SPAIN

Fishing Dispute: Conclusions• We developed an agent that can play well against a human player.•The agent was tested on students in their third year of computer science studies. •The results of the experiments implied that the agent plays well and fair.• It raised the sum of the utilities in the simulation it was involved in.• The agent played as Spain significantly better than a human did, and just as good as a human Canada player.

Diplomacy’s Rules Each player represents one of seven European

powers: England, Germany, Russia, Turkey, Austria-Hungary, Italy and France.

Diplomacy’s Map

Diplomacy’s Rules (Cont.) Winner: The power that gains control over the majority of

the board. Beginning: 1901; two seasons a year. A season: consists of a negotiations stage and a move

stage. Moves: All players secretly write the orders for all of

their units simultaneously. Negotiations: Coalitions and agreements among the

players reached in the negotiations stage significantly affect the course of the game. The rules of the game do not bind a player to anything she says. Deciding who to trust as situations arise is part of the game.

Negotiations in Diplomacy

R

If you support my attack on Vienna I will support your attack on Rumania

G

I know that Italy is going to attack Trieste

F

Don’t trust Germany

E

If you will not help me I will attack you

Moves in Diplomacy Only one unit may be in any space at one time. A unit can be ordered to: move, support, hold

(convoy). An army or a fleet may support the move of

another unit of that country or any other country in making a move.

Support can also be given on a defensive basis. Opposing units with equal support do not move.

An advantage of only one support is sufficient to win.

Moves in Diplomacy

The Need for Negotiations in Diplomacy

Moves require close cooperation between various allied powers.

Incomplete information: communications between players are done secretly.

The game is complex: 834 possible moves in each step of the game (without negotiation moves) . Negotiation is used to obtain information about the goals of the other players.

Others negotiate.

Diplomacy vs. ChessDiplomacy vs. Chess

Type of game: war games. Number of players: 7 vs. 2 Moves: simultaneous vs. sequential. Number of pieces to move: all pieces vs. one

piece. Information: uncertainty about messages

exchanged between other players vs. full information

Needed capabilities: negotiation skills vs. strategic skills.

Playing Techniques

NEGOTIATIONS Game theory techniques:

formalize the game; find an equilibrium; follow the equilibrium strategy.Impossible in Diplomacy because of complexity.

Market techniques. Appropriate for games of many players that can exchange similar items.

Heuristics: domain specific; “advice” books; human like strategies

Markov Decision Processes. Learning from DB Learning from experience

CHESS

Heuristic Search

Diplomat: an Automated Diplomacy player

Analysis &

Strategies

Finder

Negotiations

PreviousAgreements

Beliefs on other players

Board Status

AgreementsDetailed plans

and theirestimated value

for possiblecoalitions

Analysis &

Strategies

Finder

Moves

Analysis OthersMoves

Diplomacy Structure

Prime

Minister

Foreign

Office

Military

Headquarters

Ministry

Of Defense

Intelligence

Strategies

Finder

Front1

Front2

Front3

Desk10

Analyzer13

Desk11

Desk12

Analyzer14

Writeorders

15

Writeorders

15

Secretary

Strategies Finder (SF) Front: possible enemies and possible allies, e.g., Russia

and Italy against Austria and Germany.

Diplomat’s strategy for a given front includes: A list of orders associated with their purpose. The expected average profit from carrying out the strategy for each

power who participates in the strategy and the common expected profit for all of the powers.

A Venice (I) moves to Triests in order to attack Triest A Vienna (R) supports A Venice to Trieste in order to attack Trieste ……… Expected outcome: Aver: 10617 Min: 5002 Max: 20862

Russia: 3358 Italy 18117

Strategies Finder (SF) (Cont)

Diplomat identifies possible front based on on-going agreements, beliefs about other agents and their relations.

SF finds some strategies for each front using domain specific heuristics. The value of each strategy is computed by finding strategies for the enemies of the front.

The negotiation is done based on the identified strategies.

Question: What is the best strategy?

Diplomat’s negotiationExchange information;Decide what kind of agreement to try to achieve.Find common enemies.

Negotiating about the generalpurposes of an agreement: spaces on the board to attack, to defend, to leave or to enter.

Deciding on the specificmovements in order to achieve the purposes From previous stage

Signing the final Agreement; Deciding if to keep it.

Diplomat’s behavior is not deterministic

Diplomat has special ``personality'' traits that affect its behavior and may be varied easily from game to game.

Diplomat ``flips coins'' in the following cases: To decide whether to pretend to keep an agreement or to tell

the other partner that it will break the agreement (the probability depends on the personality traits.)

To decide whether to give more details about a suggestion. To decide which opening to use. When SF searches for possible strategies. For example, to

decide which units will participate in the attack or defense of a given location and to guess which of the enemy's units will participate in the battle of that location.

Diplomat’s Evaluation

Diplomat was evaluated and consistently played better than human players.

It did not play enough games to gain statistical results.

It was hard to evaluate what contributed to its success.

Conclusions

It is possible to develop automated negotiators!!

Is it possible to develop standard methods for playing negotiation games (as in Chess)?

On going work incomplete information Modeling the opponents’ preferencesLearning to negotiate

Learning to negotiate: 3-players majority game

You are one of 3 Players:

You need to divide the rights for a goldmineYouYou Player 1Player 1 Player 2Player 2

Simple Game Protocol (cont.)

Each Game Round one player is selected Randomly

And he/she gets to make a division proposal

You15%

Player 220%

Player 165%

YouYou Player 1Player 1 Player 2Player 2

YouYou Player 1Player 1 Player 2Player 2

Simple Game Protocol (cont.)

Based on the proposals the players vote

It takes a majority to make a decision – the proposer and one other playerYouYou Player 1Player 1 Player 2Player 2

You15%

Player 220%

Player 165%

Simple Game Protocol (cont.) Once a majority was reached the game ends –

each player gets his/her share

Otherwise (no agreement) – A new proposer is selected and an additional round is being played

YouYou Player 1Player 1 Player 2Player 2

Simple Game Protocol (cont.) However – it is not certain that a new round will

take place!!! There is a continuation probability – if no

agreement was reached, there is a possibility that the game will suddenly end and all players will get zero

No Agreement

P(New Turn)=0.9 P(End Game)=0.1

Agent Design Collect and Manage a DB of previous games Given a new game– find similar situations in DB Maximize utility given previous behaviour

Color Trail Game

Co-developer: Barbara Grosz Harvard University