Potentialet og principper i Scrum og Agile, som de fleste ikke udnytter v/Mads Troels Hansen
Nesterov’s excessive gap technique and poker Andrew Gilpin CMU Theory Lunch Feb 28, 2007 Joint...
-
date post
21-Dec-2015 -
Category
Documents
-
view
216 -
download
1
Transcript of Nesterov’s excessive gap technique and poker Andrew Gilpin CMU Theory Lunch Feb 28, 2007 Joint...
Nesterov’s excessive gap technique and poker
Andrew GilpinCMU Theory Lunch
Feb 28, 2007
Joint work with:Samid Hoda, Javier Peña, Troels Sørensen, Tuomas Sandholm
Outline
• Two-person zero-sum sequential games
• First-order methods for convex optimization
• Nesterov’s excessive gap technique (EGT)
• EGT for sequential games
• Heuristics for EGT
• Application to Texas Hold’em poker
We want to solve:
If Q1 and Q2 are simplices, this is the Nash equilibrium problem for two-person zero-sum matrix games
If Q1 and Q2 are complexes, this is the Nash equilibrium problem for two-person zero-sum sequential games
What’s a complex?
It’s just like a simplex, but more complex.
Each player’s complex encodes her set ofrealization plans in the game
In particular, player 1’s complex is
where E and e depend on the game…
A B C D E F G H
Recall our problem:
where Q1 and Q2 are complexes
Since Q1 and Q2 have a linear description,this problem can be solved as an LP. However,current LP solution methods do not scale
(Un)scalability of LP solvers
• Rhode Island poker [Shi & Littman 01]– LP has 91 million rows and columns– Applying GameShrink automated abstraction algorithm yields an
LP with only 1.2 million rows and columns, and 50 million non-zeros [G. & Sandholm, 06a]
– Solution requires 25 GB RAM and over a week of CPU time
• Texas Hold’em poker– ~1018 nodes in game tree– Lossy abstractions need to be performed– Limitations of current solver technology primary limitation
to achieving expert-level strategies [G. & Sandholm 06b, 07a]
• Instead of standard LP solvers, what about a first-order method?
Convex optimization
Suppose we want to solve
where f is convex.
For general f, convergence requires O(1/ε2) iterations(e.g., for subgradient methods)
For smooth, strongly convex f with Lipschitz-continuous gradient, can be done in O(1/ε½) iterations
Note that this formulation capturesALL convex optimization problems(can model feasible space using anindicator function)
Analysis based on black-box oracleaccess model. Can we do better bylooking inside the box?
Strong convexity
A function is strongly convex if there exists such that
for all and all
is the strong convexity parameter of d
Recall our problem:
where Q1 and Q2 are complexes
Equivalently:
where
and
, ,
Unfortunately, Φ and f are non-smooth
Fortunately, they have a special structure
Let d1,d2 be smooth and strongly convex on Q1,Q2
These are called prox-functions
Now let μ > 0 and consider:
These are well-defined smooth functions
Excessive gap condition
From weak duality, we have that f(y) ≤ Φ(x)
The excessive gap condition requires that
fμ(y) ≤ Φμ(x) (EGC)
The algorithm maintains (EGC), and gradually decreases μ
As μ decreases, the smoothed functions approach thenon-smooth functions, and thus iterates satisfying (EGC)converge to optimal solutions
Nesterov’s main theorem
Theorem [Nesterov 05]There exists an algorithm such that after at most N iterations, the iterates have duality gap at most
Furthermore, each iteration only requires solving three problems of the form
and performing three matrix-vector product operations on A.
Nice prox functions
A prox function d for Q is nice if it is:1. Strongly convex continuous everywhere in Q,
and differentiable in the relative interior of Q
2. The min of d over Q is 0
3. The following maps are easily computable:
Nice simplex prox function 1: Entropy
Nice simplex prox function 2: Euclidean
sargmax can be computed in O(n log n) time
From the simplex to the complex
Theorem [Hoda, G., Peña 06]
A nice prox function can be constructed for
the complex via a recursive application of
any nice prox function for the simplex
Prox function example
Let be any nice simplex prox function.The prox function for this matrix is:
Solving
(similar to b(i-vii))
Heuristics [G., Hoda, Peña, Sandholm 07]
• Heuristic 1: Aggressive μ reduction– The μ given in the previous algorithm is a
conservative choice guaranteeing convergence– In practice, we can do much better by aggressively
pushing μ, while checking that the excessive gap condition is satisfied
• Heuristic 2: Balanced μ reduction– To prevent one μ from dominating the other, we also
perform periodic adjustments to keep them within a small factor of one another
Matrix-vector multiplication in poker[G., Hoda, Peña, Sandholm 07]
• The main time and space bottleneck of the algorithm is the matrix-vector product on A
• Instead of storing the entire matrix, we can represent it as a composition of Kronecker products
• We can also effectively take advantage of parallelization in the matrix-vector product to achieve near-linear speedup
Memory usage comparison
Instance CPLEX IPM CPLEX Simplex EGT
10k 0.082 GB >0.051 GB 0.012 GB
160k 2.25 GB >0.664 GB 0.035 GB
RI 25.2 GB >3.45 GB 0.15 GB
Texas >458 GB >458 GB 2.49 GB
Poker
• Poker is a recognized challenge problem in AI because (among other reasons)– the other players’ cards are hidden;– bluffing and other deceptive strategies are needed in
a good player;– there is uncertainty about future events.
• Texas Hold’em: most popular variant of poker• Two-player game tree has ~1018 nodes
Potential-aware automated abstraction[G., Sandholm, Sørensen 07]
• Most prior automated abstraction algorithms employ a myopic expected value computation as a similarity metric– This ignores hands like flush draws where although the
probability of winning is small, the payoff could be high
• Our newest algorithm considers higher-dimensional spaces consisting of histograms over abstracted classes of states from later stages of the game
• This enables our bottom-up abstraction algorithm to automatically take into account positive and negative potential
Solving the four-round model
• Computed abstraction with– 20 first-round buckets– 800 second-round buckets– 4800 third-round buckets– 28800 fourth-round buckets
• Algorithm using 30 GB RAM– Simply representing as an LP requires 32 TB– Outputs new, improved solution every 2.5 days
[G., Sandholm, Sørensen 07]
[G., Sandholm, Sørensen 07]
[G., Sandholm, Sørensen 07]
Future research
• Customizing second-order (e.g. interior-point methods) for the equilibrium problem
• Additional heuristics for improving practical performance of EGT algorithm
• Techniques for finding an optimal solution from an ε-solution
Thank you ☺