Honors Research Colloquium Presentation Slides

Producing Valid College Football Rankings in

Reasonable TimeMark Edwards, Department of Mechanical Engineering

Jonathan A. Shumaker, Department of Chemical Engineering C. Richard Cassady, PhD, Department of Industrial Engineering

5th Annual FEP Honors Research Symposium 4/20/13

2

NCAA Football

NCAA Football Bowl Subdivision: 124 colleges/universitieso each team plays approximately 12 games against other FBS

teamso only 9% of possible matchups occur

determining the FBS champion: one game, 2 teamso 4-team playoff beginning in 2014o more than just a game: $172M on the table ($450M in 2014)

effectively ranking the teams is critical, but difficulto 124! possible rankingso what to consider in the rankingso maintaining fairness in the rankingso conflicting results

Edwards, M. and J. Shumaker 5th Annual FEP Honors Research Symposium

3

Current Ranking System

Bowl Championship Series rankingso two opinion polls: 1/3 eacho six computer rankings: 1/3

BCS flawso opinion polls: bias, who is actually voting, weird ballotso computer rankings: secrecy, data errors, design bias

annual controversy

Edwards, M. and J. Shumaker5th Annual FEP Honors Research

Symposium

5th Annual FEP Honors Research Symposium

4

Our Platform

4-team playoff increases need for a better ranking systemo selection committee will need guidanceo harder to distinguish between #4 and #5 than #2 and

#3 no such thing as an unbiased approach

o what counts should be clearly stated a computer-based system should be used

o humans cannot simultaneously process results of 650-800 games

o computer can apply biases consistently across all teams

Edwards, M. and J. Shumaker


5

Our Approach: CMS+

define the ranking problem mathematicallyo quadratic assignment problemo inputs: degree of victory, relative distanceo output: who gets ranked where

degree of victory: how we compare pairs of teamso head-to-head? margin of victory? date? location?o transitive results? conference champions? schedule

strength? relative distance: how far apart are positions in

the rankingo bell curve



6

Solving the QAP

large QAP are impossible to solve to guaranteed optimality

CMS+ uses a two-stage heuristico stage 1: genetic algorithm

• uses “survival of the fittest” concepts to convert good rankings into better rankings

• simulate 100,000 generations of rankingso stage 2: local search

• seek pair-wise switches in the best GA solution that improve solution quality

initial GA population is randomo replicate the heuristic 20 times



7

CMS+ Issues

What to include in degree of victory

o Objective 1: to investigate what factors should be included (and how to include them) in degree of victory (in partnership with T. Dodson and A. McElhenney)

Long time required to solve the QAP

o Objective 2: to investigate the impact of reducing the number of GA generations and heuristic replications on solution quality


8

Objective 1: Degree of Victory (DOV)

identify possible factors to include define ways to quantify these factors collect data from past seasons (1998-2011) generate alternative rankings using various

combinations of the identified factors

Edwards, M. and J. Shumaker5th Annual FEP Honors Research

Symposium


9

DOV: Factors Considered

winners and losers of regular season games four optional factors

o game location• winning away from home generates a bonus

o common opponents• which team won more games against common opponents

o conference champions• if both teams are in the same conference, did one win the

conferenceo media perception

• which team is ranked higher in the Associated Press opinion poll



10

DOV: Quantifying the Factors

Team A receives 15 DOV points over Team B ifo game location is considered and Team A defeated Team

B at Team B’s home field• 7.5 points for a neutral site victory

o common opponents are considered and Team A has more victories than Team B against their common opponents

o conference champions are considered, Team A and Team B are in the same conference, and Team A is the conference champion

o media perception is considered and Team A is ranked above Team B in the Associated Press opinion poll

Team A receives d DOV points over Team B if Team A defeated Team Bo d = 100 – 15*(# of optional factors considered)



11

DOV: Data Collection

Collected data from past seasons (1998-2011)1

Validated data with two parallel effortso T. Dodson, A. McElhenneyo B. Wiles, industrial engineering senior

4 optional factors (off/on) results in 16 DOV sets for each yearo generated in Microsoft Excel using a Visual Basic for

Applications (VBA) macro


1Data collected from: http://homepages.cae.wisc.edu/~dwilson/rfsc/history/howell/http://espn.go.com/college-football/conferenceshttp://www.collegepollarchive.com/football/ap/seasons.cfm?seasonid=1998

http://homepages.cae.wisc.edu/~dwilson/rfsc/history/howell/

http://homepages.cae.wisc.edu/~dwilson/rfsc/history/howell/

http://espn.go.com/college-football/conferences

http://espn.go.com/college-football/conferences

http://www.collegepollarchive.com/football/ap/seasons.cfm?seasonid=1998

http://www.collegepollarchive.com/football/ap/seasons.cfm?seasonid=1998


12

DOV: Rankings

16 DOV sets for 2010 used as test cases CMS+ executed for each set

o Shown below with comparison to BCS Rankings from 2010


BCS Rankings DOV 05(AP Rank)

DOV 16(Everything on)

Fitness 246025 196313

1 Auburn Oregon Oregon

2 Oregon Texas Christian Auburn

3 Texas Christian Boise State^ Texas Christian

4 Stanford Auburn Boise State^

5 Wisconsin Ohio State Michigan State

6 Ohio State Nevada^ Wisconsin

7 Oklahoma Wisconsin Virginia Tech^

8 Arkansas Stanford Nevada^

9 Michigan State Michigan State Stanford*

10 Boise State Utah^ Ohio State


13

Objective 2: Run Time (RT)

define test cases for the QAP using past seasons experiment with various combinations of the

number of GA generations and the number of heuristic replications

assess the impact of reduced run time on solution qualityo in partnership with T. Dodson and A. McElhenney



14

RT: Experimental Plan


experiments decrease in generations and replicationso Based on the original parameters

number of replications did not have an affect on run time

fitness and rankings never changed number of generations is the only factor that

alters run time


15

RT: Results


In the 4,500 to 2,500 range of total generations, the lowest time value can be achieved. Beyond this in either direction, the time starts to increase.


16

RT: Impact on Solution Quality

for any given DOV set and a corresponding rankingo fitness can be computed as described by CMS+o larger fitness implies a better solution/ranking

T. Dodson and A. McElhenney concluded that fitness is normally distributed

they provide estimates ofo the average fitness and the standard deviation of fitnesso the number of standard deviations by which the CMS+

solution exceeds the averageo the number of the 124! rankings that have a better

fitness value



17

RT: Impact on Solution Quality

Fitness remained constant for all generation amountso Reduction of generations without having to worry about

loss of quality Changing replication amount had no affect on

fitnesso Also had no affect on time

Time was able to be reduced to as low as about 25 secondso At 3,500 generations

• More or less generations meant longer time• Optimal range is around 3,500 generations



18

Conclusions

objectives allowed for CMS+ to be improvedo DOV research created varying factors o Run Time research found ideal specifications within the

program CMS+ system can be improved with future

researcho More parameters for point allocation

• Only used four parameters while there are many more parameters that could be used

o Adjust other factors within the CMS+ program to see how they affect run time and quality• Only used two parameters, total generations and

replication amount• Could change population size


Honors Research Colloquium Presentation Slides

Engineering

Transcript of Honors Research Colloquium Presentation Slides