Using Performance Metrics to Forecast Success in the National Hockey League
description
Transcript of Using Performance Metrics to Forecast Success in the National Hockey League
Using Performance Metrics to Forecast Success in the National
Hockey League
Josh Weissbock
Outline
• Introduction to Hockey• Introduction to Performance
Metrics• Predicting the outcome of a Single
Game• Exploration of Performance Metrics
J. Weissbock (2013)
Aim
• Performance Metrics (or “Advanced Stats”) have been shown on the internet to correlate much higher to wins and points in the standing, for the National Hockey League, than traditional statistics posted by the NHL.
• Can we use these advanced stats predict success in the NHL?
J. Weissbock (2013)
Introduction
• Lack of academic attention to hockey.• Hard to analyze due to the lack of events (goals).• We attempt to use Machine Learning to predict a single
game in the National Hockey League:– Using Traditional Statistics;– Using Performance Metrics; and– “Tuning Performance Metrics”
J. Weissbock (2013)
Sports in Machine Learning
• Chen et al. (1994) used Neural Networks to predict greyhound races.
• 2006 Soccer World Cup prediction accuracy 75%.• NFL accuracy with neural networks: 78.6%.• NCAA Football games prediction accuracy: 76%.• NBA basketball accuracy at 76%.
J. Weissbock (2013)
Background
• Hockey is a sport played on a rectangular sheet of ice 60-61m x 25-30m.
• 2x teams of 5x players and 1x goal keeper (goalie).• Team the scores the most goals in 60 minutes wins.• NHL top league in the world.
– Other major leagues: KHL, SHL, ELH.• Top 16 teams at the end of the year compete in an
elimination tournament for the Stanley Cup.– 4x rounds of best-of-seven series.
J. Weissbock (2013)
Background
• Traditional Statistics are “Real Time Scoring System” Statistics, based on goals (low events), usually are simple goal based stats and subject to rink bias.– i.e. Goals, Assists, +/-, Giveaway, Takeaway, etc.– Demonstrated bias amongst RTSS statistics in NHL
arenas.• Advanced Statistics based on more events (all shots,
misses, blocks, and goals), shown to be highly correlated to wins and points.
J. Weissbock (2013)
Advanced vs Traditional StatisticsStatistic Home/Points
r-squaredRoad/Points r-squared
Fenwick Close 0.623 0.218Goals Against 0.358 -0.119Goal Differential 0.297 0.196Goals 0.233 0.084Wins 0.186 0.199Points 0.177 0.177Blocked Shots 0.157 -0.117Hits 0.116 -0.021Giveaways -0.001 -0.002Takeaways -0.001 -0.003
Source: http://blogs.thescore.com/nhl/2013/02/25/breaking-news-puck-possession-is-important-and-nobody-told-the-cbc/ J. Weissbock (2013)
Advanced vs Traditional StatisticsTeam Stat Relationships
vs Wins R^2 vs Points R^25-5 F/A 0.605 0.655GA/G 0.472 0.510PP+PK 0.372 0.390G/G 0.352 0.360Sv% 0.227 0.263PP% 0.221 0.231SA/G 0.198 0.191S/G 0.170 0.203S% 0.160 0.145PK% 0.152 0.160FO% 0.097 0.109
Source: http://www.nucksmisconduct.com/2013/2/13/3987546/exploring-marginal-save-percentage-and-if-the-canucks-should-trade-aJ. Weissbock (2013)
Advanced Statistics
• Fenwick Close: statistic of posession, summation of shots, missed shots, blocks, and goals. Correlates to zone time.– “Close” refers to only when the score is within 1 in
the 1st/2nd period, or when the score is tied in 3rd/OT to eliminate “Score Effects”.
• PDO: Statistic of “luck” (or random chance), summation of Shooting % + Save %. Regresses to 100% +/- 2% over a 82-game season.
• 5/5 Goals For/Against: The ratio of goals scored for and against during even strength play.
J. Weissbock (2013)
Experiment 1
• Predicting a single game in NHL using both advanced and traditional statistics. – To see how high of an accuracy we can obtain– To see if advanced or traditional statistics better help
predicting at the micro-scale.
J. Weissbock (2013)
Data
• 517 games of 2012-2013 NHL Season (72% of season)– Python script to collect data daily.
• 14 Features/Team collected including:– Location, Goals For & Against, Season Goals For &
Against, PP%, PK%, Sh%, Sv%, 5v5 Goals For/Against, Win Streak, Conference Standing, Fenwick Close, PDO.
• Data collected before and after game:– Goals scored for & against, shots for & against.– To assist calculating statistics for future games.
• Sources: NHL.com, BehindTheNet.com, TSN.ca/NHL.
J. Weissbock (2013)
Example of data
J. Weissbock (2013)
Experiment
• Data represented as differentials between both teams.• Two entries for each game, one for each team.• Labelled as either “win” or “loss”. • Weka’s implementations of SMO (Support Vector
Machines), Neural Networks, J48 (Decision Tree), and NaiveBayes.
• Binary classification using 10-fold cross-validation.• Compared datasets of only traditional and advanced
statistics, as well as both.
J. Weissbock (2013)
Experiment
Traditional Advanced Mixed
Baseline 50% 50% 50%
SMO 58.61% 54.55% 58.61%
NB 57.25% 54.93% 56.77%
J48 55.42% 50.29% 55.51%
NN 57.06% 52.42% 59.38%
J. Weissbock (2013)
Experiment
• Best results from Neural Networks:– With additional tuning, accuracy of 59.38%;– Not statistically different than SMO.
• Splitting the data into testing/training (66%/33%) – accuracy of 57.83%:– Looked at pairs labelled Win/Win or Loss/Loss by
algorithm, keeping the label with the highest confidence and inverting the other, accuracy of 59%.
• Ensemble learning w/ stacking and voting returned similar accuracy.
J. Weissbock (2013)
Experiment
• Using Consistency Subset Evaluation, the top three features were:– Location– Goals against– Goal differential
J. Weissbock (2013)
Experiment• Second half of our experimental evaluation, we consider
shortening PDO to the last n games to see how “lucky” a team has been recently
PDO1 PDO3 PDO5 PDO10 PDO25 PDOAll
Baseline 50% 50% 50% 50% 50% 50%
SMO 58.61% 58.61% 58.61% 58.61% 58.61% 58.61%
NB 56.38% 56.96% 56.38% 56.58% 56.58% 56.77%
J48 54.93% 55.71% 55.42% 55.90% 55.61% 55.51%
NN 57.64% 56.67% 58.03% 58.03% 57.74% 58.41%
J. Weissbock (2013)
Discussion
• Altering PDO does not appear to have a significant affect on accuracy.
• Can predict ~60% of games correctly.• Despite possession shown to be more useful in long
term predictions, the traditional statistics are better for predicting a single game.
• In a single game the most valued features are: goals against, goal differential and location.
J. Weissbock (2013)
Future Work
• Collecting additional features:– Rest days, days of travel, time-zone shifts, altitude
shifts, change in weather at arena, gambling odds, injures, score-adjusted Fenwick, possession over the last n games.
• Collecting a full season of data (1230 games regularly).• Training on past seasons of data.• Compare prediction of single game in other league with
same features• Predicting the playoffs with best-of-seven series.
J. Weissbock (2013)
Conclusion
• ~60% Accuracy to predict a single game.• Traditional statistics more effective in predicting a single
game than advanced statistics.• Predicting a single game is difficult due to large
variance in the standings.• Theoretical limit in prediction for machine learning for a
single game in the NHL appears to be 62%.– Changes based on the parity of the league and
number of events.
J. Weissbock (2013)
Questions?Joshua Weissbock
[email protected] my hockey analysis on twitter: @joshweissbock
J. Weissbock (2013)