Neural Network Implementation of Poker AI
description
Transcript of Neural Network Implementation of Poker AI
Neural Network Implementation of Poker AI
Christopher Kramer
Outline of Information The Challenge
Application, problem to be solved, motivation
Why create a poker machine with ANNE?
The Flop The hypothesis
Can a Poker AI run using only an ANNE?
The Turn The approach and methodology
How is this machine created?
The River Experimentation
How should it be tested?
The Showdown Conclusions and future work
Did it work?
The Challenge The approach focused on creating an AI capable of playing poker at a
reasonable level of play using Neural Networks. The difficulty:
Incomplete knowledge
Multiple competing agents
Risk management
Opponent modeling
Deception
Unreliable information
Creating solutions to address these difficulties can be applied to other problems with similar domains.
NOISE!
The Fold (Hypothesis)
The strengths of computers: Quick computation
Large memory
Purely logical
Using a series of machine learning techniques including neural networks, it is predicted that a machine will be able to play at a moderate level while ignoring typical features of poker programs such as statistical analysis and pot odds.
The Turn (Approach)
Problem split into a number of stages
• Hardcoded• Action chosen based off of known values
of starting handsPre-Flop
• Neural Net • 5 card inputPost-Flop
• Neural Net• 6 card inputRiver
• Neural Net• Player hand analyzed and given as set of
input valuesShowdown
The Turn (continued)
• Hardcoded• Action chosen based off of known values
of starting handsPre-Flop
• Information gathered from percentage of games won with starting hand from poker databases
• All hands in top 30th percentile were analyzed for characteristics which were hardcoded into if statements
• All hands below threshold were folded. • Those with characteristics in top 5th percentile had a
higher random chance of being raised
The Turn (continued)
• Neural Net • 5 card inputPost-Flop
• Inputs the unique id of each card in hand and play• Hidden nodes utilizing sigmoid function• Output using sigmoid function between -1 and 1• Output values above .3 indicated an action to raise• Below -.3 fold• Else, Call
The Turn (continued)
The Turn (continued)
• Neural Net• 6 card inputRiver
• Inputs the unique id of each card in hand and play• Hidden nodes utilizing sigmoid function• Output using sigmoid function between -1 and 1• Output values above .3 indicated an action to raise• Below -.25 fold• Else, Call
The Turn (continued)
• Neural Net• Player hand analyzed and given as set of
input valuesShowdown
• Cards analyzed for poker hands such as Flush, Fold, etc. and high card of each returned.
• Input nodes for each poker hand. Value is the value of the highest card in hand.
• Hidden nodes utilizing sigmoid function• Output using sigmoid function between -1 and 1• Output values above .5 indicated an action to raise• Below -.2 fold• Else, Call
The Turn (opponent modeling)
Opponent modeling generalized to fit into previous Neural networks Opponents action modeling taken into account by Neural nets due to
opponents actions changing the weights of neural network to account for their actions.
2 Factors Player’s calls
For each raise by opponent, have a higher required confidence to raise again
Players treatment of poker hand values
As player actions change, the confidence values of the Neural nets should change to reflect the modeling of the player
The River (experimentation)
Experiment was developed to overcome short term failure due to bad luck 1000 dollars given to each player. $10 bet .
Engines initial weights were determined experimentally for each ANNE Weights were assigned randomly and run for 10000 generations against a set of random poker hands
and card tables. Winning gave an expected value of 1, loss -1, and draw 0.
Network was then checked against 1000 random validation hands. A degree of accuracy was required for each ANNE
Required accuracy lowered as amount of information known decreases.
Once Neural Network was fit level of confidence its results were accepted as the Engines default weights.
Weights continue training during the course of game to profile player. Tested against a number of human opponents based on skill level to measure success.
The River (experiment results)
Beginners AI won 82% of games against 50 opponents
Moderate (play at least 1 game a month) AI won 20% of games against 20 opponents
Advanced (play at least 1 game a week) AI won 1 game out of 10 against 2 opponents.
Random (never folded) AI won 63.2% of games out of 1000.
100% won against Random opponent who could also fold
The Showdown (Conclusion)
The complete reliance on generalized Neural Networks for a system with extreme noise is not a viable option for higher levels of play.
While it did not shown significant advantage over moderate and advanced opponents, it did well against beginning opponents who knew how to play poker but did not do so often.
So much noise…. While a degree of noise was expected, generalized poker domain’s noise was is in excess of
10-40% (approx.)
This led to extreme cases of overlearning regardless of an extremely low learning rate.
While this was partially handled using confidence, the amount of folds late in the game due to incorrect early predictions led to the machine still failing to moderate opponents.
Was still shown to do well against a random opponent and beginners.
The Showdown (future research)
There are numerous areas for improvement Current systems adjusts slower than humans Possible GA implementation instead of Neural Network to avoid direct
noise issues. Statistical analysis and pot odds would help as inputs for neural
network.
Questions?