Introduction - catsr.vse.gmu.edu€¦ · Web viewThe word "event" refers to what has triggered...

ASSESSMENT OF SOCCER REFEREE PROFICIENCY IN TIME-SENSITIVE

DECISION-MAKING

George Mason UniversitySenior Design Project: B. Sc. Systems Engineering

Final Report

Group Leader Instructor /Faculty SponsorNathan [email protected]

Dr. Lance [email protected]

Group MembersAndrew CannHina PopalSaud Almashhadi

On behalf of the Assessment of Soccer Referee Proficiency in Time-Sensitive Decision-Making George Mason University Senior Design Group (2011-2012) we would like to extend a special thanks to our project sponsor Pat Delany (Metro-DC Virginia State

Referee Program) and our faculty sponsor Dr. Lance Sherry.

Sponsor Testimony:

“The analysis done by the students has been incredibly eye-opening. They have changed the way our management at MDCVSRP think about referee development and

where to use our budget.” Pat Delaney, MDCVSRP

Table of Contents

Introduction.................................................................................................................................................. 1

Organization of American Referees....................................................................................................3

Referee Call Making Process..................................................................................................................5

Evaluation of Referee Quality................................................................................................................7

Problem Statement.....................................................................................................................................8

Need Statement............................................................................................................................................8

Design Alternatives.................................................................................................................................... 8

Baseline Fitness Test............................................................................................................................ 9

Estimated Cost....................................................................................................................................9

Game Flow Evaluation...................................................................................................................... 10

Estimated Cost.................................................................................................................................10

Combined Evaluation........................................................................................................................ 11

No Assessment......................................................................................................................................11

Evaluation of Alternatives....................................................................................................................12

Part I: Discrete Event Soccer Game Simulator............................................................................13

Part 2: Monte Carlo Analysis...............................................................................................................22

Results........................................................................................................................................................... 24

Discrete Event Soccer Game Simulator Results.....................................................................24

Monte Carlo Analysis Results.........................................................................................................25

Utility/Cost Analysis And Recommendation...............................................................................26

Additional Findings................................................................................................................................. 27

Management............................................................................................................................................... 30

Works Cited................................................................................................................................................ 38

Appendix.......................................................................................................................................................... i

ANOVA Analysis....................................................................................................................................... i

Arsenal.................................................................................................................................................... i

Manchester........................................................................................................................................... ii

Stoke...................................................................................................................................................... iiiSYST 495 Final Report

Wigan..................................................................................................................................................... iv

Simulation Output: Regression Analysis......................................................................................v

Data Points........................................................................................................................................... v

General Regression Analysis: Accuracy versus Fitness, GFU.........................................v

Monte Carlo Trials...............................................................................................................................vii

Survey Administered to MDCVSRP Senior Referees..........................................................viii

Electronic Appendix.......................................................................................................................... xiv

SYST 495 Final Report

Table of Figures

Figure 1: Professional Sports Generated Revenue Between 2009 and 2010..................1

Figure 2: Referee Positioning (MR - Main Referee, AR - Assistant Referee).....................2

Figure 3: Grade Progression for U.S. Soccer Referees................................................................3

Figure 4: Distribution of Male Referees by Grade – 2010.........................................................4

Figure 5: System Component Interaction........................................................................................4

Figure 6: Referee Call Making Process..............................................................................................5

Figure 7: Stochastic Discrete Event Soccer Game Simulator................................................12

Figure 8: Movement Polygons............................................................................................................14

Figure 9: Cycle of Events.......................................................................................................................15

Figure 10: Possession Change Between Two Teams................................................................16

Figure 11: Guardian Chalkboard Data............................................................................................17

Figure 12: Referee 2-D Movement Area.........................................................................................19

Figure 13: Call Event Probabilities Based on Field Location................................................20

Figure 14: Call Accuracy Function (Distance <20 yards).......................................................20

Figure 15: Call Accuracy Function (Distance >20 yards).......................................................21

Figure 16: Normal Distribution for Referee Attributes...........................................................23

Figure 17: Analysis of 25 Profiles: Call Accuracy (Fitness, GFU)........................................24

Figure 18: Cost vs. Utility Analysis for Alternatives.................................................................27

Figure 19: Impact of Team Combinations on Referee Call Accuracy................................28

Figure 20: Distance from Calls for United vs. United and Stoke vs. Stoke......................29

Figure 21: Total Work Breakdown Structure for SYST 490/495.......................................30

Figure 22: Work Breakdown Structure for SYST 490..............................................................30

Figure 23: Work Breakdown Structure for SYST 495..............................................................31

Figure 24: PERT Chart............................................................................................................................35

Figure 25: Earned Value Chart...........................................................................................................37

Figure 26: Arsenal - Pass Completion Percentage by Time/Score.........................................i

Figure 27: Manchester United - Pass Completion Percentage by Time / Score..............ii

Figure 28: Stoke City - Pass Completion Percentage by Time / Score...............................iiiSYST 495 Final Report

Figure 29: Wigan - Pass Completion Percentage by Time / Score.......................................iv

Table of Tables

Table 1: Assessment Methods...............................................................................................................7

Table 2: Design Alternatives..................................................................................................................9

Table 3: Admin Material Cost for Baseline Fitness Test............................................................9

Table 4: Assessor Cost for Baseline Fitness Test.......................................................................10

Table 5: Total Estimated Cost for Baseline Fitness Test.........................................................10

Table 6: Equipment Cost for Game-Flow Evaluation...............................................................10

Table 7: Total Estimated Cost for Game Flow Evaluation......................................................11

Table 8: Solomon et. Al Simulation vs. New Simulation..........................................................13

Table 9: Effect of Game Situation on Pass Completion............................................................17

Table 10: Referee Profiles.................................................................................................................... 22

Table 11: Design Alternatives Attribute Cutoffs.........................................................................23

Table 12: Call Accuracy Regression Analysis...............................................................................25

Table 13: Utilities for Grade 8 Evaluation Alternatives..........................................................26

Table 14: Task Breakdown.................................................................................................................. 31

Table 15: Task Budgeting..................................................................................................................... 35

Table 16: Earned Value..........................................................................................................................36

Table 17: Regression Analysis Data Points......................................................................................v

Table 18: Monte Carlo Trials...............................................................................................................vii

Table of Equations

Equation 1: Number of Refresh Rates.............................................................................................13

Equation 2: New Polygon Movement Algorithm........................................................................16


Assessment of Soccer Referee Proficiency in Time-Sensitive Decision-Making 4/23/2012

INTRODUCTIONSoccer is generally recognized as the most popular sport in the world.

Between 2009 and 2010, European soccer alone generated roughly $21.6 billion in

revenue, with the English Premier League accounting for $3.2 billion (Figure 1) [1].

This is significantly higher then other popular American sports such as football and

baseball. International soccer competitions such as the FIFA World Cup and UEFA

Champions League also draw the highest average attendance for international club

competitions.

Figure 1: Professional Sports Generated Revenue Between 2009 and 2010

Much of soccer’s recent success and growth in popularity can be attributed to

improvement in viewer experiences. With rapid increases in camera technology,

fans can now watch games from angles, at a high resolution, and view replays of key

events.

Two teams, each fielding eleven players, compete against one another in a

soccer match. The duration of the match is commonly two, forty-five minute

periods. The teams play on a rectangular field that is 115 by 74 yards in dimension.

The administration and integrity of the game is overseen by one main

referee, who operates on a left-hand diagonal route across the center of the field,

and two assistant referees, who operate on the left and right hand sides of the field

(Figure 2). In order to uphold the integrity of the game referees must consistently

make accurate calls on the field and ensure that these calls do not interrupt the

overall flow of the game. Most importantly, referees are responsible for instilling in



fans a belief that the game being officiated is fair and impartial in a manner in which

both teams have an equal opportunity to succeed.

Figure 2: Referee Positioning (MR - Main Referee, AR - Assistant Referee)

Although technical upgrades have been implemented to enhance viewer

experience, the governing bodies of soccer have been mostly unwilling to implement

referee support technology, such as replays, for fear that it will interfere with game

flow [2]. Thus, as the quality of soccer broadcasting has improved, the tools

available to the referee have remained the same. This imbalance of technology has

lead to an asymmetry in information where fans often have better information for

judging the accuracy of a call than the referees on the field. This allows fans to easily

identify injustices in the administration of the game, and has caused backlashes

against the sport when incorrect calls alter the outcome of the match [2]. Therefore,

poor referee performance can be considered one of the greatest threats currently

facing the sport of soccer.



ORGANIZATION OF AMERICAN REFEREES

Within the United States, soccer referees undergo a structured training and

evaluation process. The process is broken into eight levels of seniority (grades) in

which grades 8-7 represent entry level referees, 6-5 contain state referees, 4-3

comprise national referees, and 2-1 are reserved for FIFA international referees

[3,4] (Figure 3). Grade 8 referees are typically referred to as “junior” referees

whereas referees in grade 7-1 are referred to as “senior” referees.

Figure 3: Grade Progression for U.S. Soccer Referees

Progression of referees beyond grade 8 is voluntary and requires classes, written

examinations, fitness tests, and game performance evaluations. A referee’s grade

determines the level of game he is recommended to officiate [4].

The United States Soccer Federation (USSF) oversees all referees in grades 4-

1 where those in grades 8-5 are overseen by state level referee organizations [4].

The state level organization within the Common Wealth of Virginia, the Metro DC

Virginia State Referee Program (MDCVSRP), serves as the sponsor for this project.

Within the MDCVSRP, 96.8% of referees reside within grade 8 while the remaining

3.2% of referees are distributed throughout grade 7-1 (Figure 4) [5]. The USSF

provides funding to the MDCVSRP in exchange for the MDCVSRP training and

promoting top-level, high-quality referees to the national level. The interactions of

the MDCVSRP and with its stakeholders can be seen in Figure 5.



Figure 4: Distribution of Male Referees by Grade – 2010

Figure 5: System Component Interaction

The success of efforts to improve on-field performance hinges on an ability to

evaluate referee quality. Evaluating referee quality is key to progressing referees to

more senior grades and properly assigning referees to games [4,6].



REFEREE CALL MAKING PROCESS

A referee’s quality is defined as the percent of correct calls during games. A

referee’s call accuracy is dependent upon on how effectively he is able to carry out a

standard decision making process whenever a call event is triggered (Figure 6).

Figure 6: Referee Call Making Process

The ability of a referee to carry out this process is dependent upon his ability

to perform a series of functions whenever a call event occurs. Certain referee

attributes determine a referee’s ability to perform these functions. The first step in

the process involves the referee perceiving an event. Whenever a call event occurs, a

referee must visually recognize that a decision needs to be made through a Sensory

Function. Once an event is detected, the referee must begin to process information

based on the event he witnessed. In this instance the referee must make an accurate

decision regarding the nature of the call (infraction, no infraction) using a mental

model of what occurred in the event and knowledge of the laws of soccer. This

process is combined into a function known as Cognition for Making Calls and

determines a referee’s call accuracy.

The ability of a referee to make correct calls through the Cognition for Making

Calls function is dependent on a referee’s distance from the call, which is

determined through the interaction of two functions. The first function, Cognition



for Positioning, defines a referee’s ability to choose an optimal position to make calls.

This function is dependent on a referee’s Sensory Function and mental model of

game flow. Referees may either reactively position themselves based on game flow,

or if they have a high level of game flow understanding, they may proactively

position themselves to make calls. The second function, Propulsion, is a physical

function determining a referee’s ability to move to the position identified during the

Cognition for Positioning function in a time effective manner. Once a referee has

processed all of the information necessary to make a decision, the referee must then

take physical action and executes their command based on the information

processed.

The ability of a referee to carry out the Cognition for Making Calls, Cognition

for Positioning, and Propulsion functions is assumed to depend on three attributes.

Game flow understanding (GFU) is the ability of a referee to perform Cognition for

Positioning. Game flow understanding describes a referee’s ability to interact with

the flow of the game and to either reactively position oneself based on ball

movement or to proactively positions oneself based on probable ball movement and

call events. Fitness is the ability of a referee to carry out Propulsion, which involves a

referee’s athletic ability to transition themselves from a starting location to a

desired end location in a timely manner. Call decision-making (CDM) is the ability of

a referee to carry out Cognition for Making Calls. This involves the referee’s ability to

construct a mental model of the occurred event and draw upon his knowledge of the

rules of the game to make an accurate decision with regards to the nature of the

event.



EVALUATION OF REFEREE QUALITY

Evaluating referee quality focuses on assessing referees in terms of their

game flow understanding, fitness, and call decision-making attributes.

Currently, the MDCVSRP implements three different assessment methods to

assess these attributes (Table 1). Game flow understanding is currently evaluated

indirectly through annual on field assessments conducted by official assessors1 for

referees grades 7 -1 [4]. Fitness is evaluated through various fitness tests including

a series of sprints and long distance runs. This test is comparable to the

“presidential” fitness test administered to public high school students. The fitness

test is administered annually to referees grade 7-1 [4]. Call decision-making is

evaluated through written examinations administered to all referees and annual on

field assessments for referee’s grades 7-1 [4]. Performance metrics for each of these

assessment methods vary and increase in difficulty as the grade of the referee being

tested progresses.

Table 1: Assessment Methods

Referee Attributes Assessment MethodFitness Fitness Test (Senior Referees)

Call Decision Making (CDM) Written exam on rules (All referees)

Game Flow Understanding (GFU)Indirectly using on field assessment

(Senior Referees)

This current assessment methodology has significant gaps in assessing

referees based on attributes. In particular, referees in grade 8, which account for the

majority (96%) of referees within the Commonwealth of Virginia, do not receive any

evaluations for game flow understanding or fitness [6].

PROBLEM STATEMENT

1 “Assessors are experienced coach-mentors, whose referee experience enables themto observe how the referees handle the challenges presented to them by the match” [7]. Assessors must have experience at the senior referee level before converting to Assessors.



Due to the gaps in assessment methodology, 96% of Metro DC Virginia State

Referee Program referees (Grade 8) currently do not undergo assessments for game

flow understanding and fitness attributes as predictors of call accuracy.

NEED STATEMENT

An evaluation system is needed to predict the quality (call accuracy) of grade

8 referees overseen by the MDCVSRP based on their fitness and/or game flow

understanding attributes.

DESIGN ALTERNATIVES

Four evaluation system concepts have been identified to assess the quality of

grade 8 referees (Table 2). The specifics of design and implementation of these

concepts are considered outside the scope of this project. The cost of each

alternative is defined as the investment necessary to purchase required physical

resources and carry out a one-time quality evaluation of all grade 8 referees. Three

alternatives would involve a Baseline Fitness Test and a Game Flow Evaluation,

either single or in combination. A fourth alternative would involve no testing (status

quo).

Table 2: Design Alternatives

# Alternative Description Tests Total Cost (5,139



Referees)

1 Fitness Test

A baseline fitness test equivalent to those

administered to grade 7-1.

Fitness $26.990

2Game Flow Evaluation

Video performance assessments conducted

by official assessors.

Game-Flow Understanding

$337,995

3Combined Evaluation

Combination of first two evaluations

Fitness and Game-Flow

Understanding$341,870

4 No AssessmentNot conducting any referee evaluations

(status quo).None $0.00

Baseline Fitness Test

This alternative involves a baseline fitness test administered to all grade 8

referees within MDCVSRP. The results of the baseline fitness test would be used to

assign each referee a fitness attribute rating as a means of assessing overall quality.

This would be the same fitness test currently administered to referee grades 7-1.

Estimated Cost

The estimated cost for this assessment method involves the rate of pay for

the fitness assessor, who would be responsible for administering the fitness tests

and noting the referees’ performances. The administrative resources necessary to

complete this test were also factored in to the total estimated cost, which summed

out at $26,990.

Table 3: Admin Material Cost for Baseline Fitness Test

Admin Material

Cost for 1 Set Needed Sets Total Cost

Clipboards (24) $40.00 5 $200.00

Pens (60) $7.00 17 $200.00

Paper (2500) $19.00 4 $76.00

Total $66.00 26 $395.00



Table 4: Assessor Cost for Baseline Fitness Test

Equipment Cost for 1 Assessor/RefFitness Assessor $5.00

Table 5: Total Estimated Cost for Baseline Fitness Test

EquipmentMaterial

CostCost

Assessor/Ref

Cost /Year (5319

Referees)Admin Materials & Fitness

Assessor$395.00 $5.00 $26,990.00

Game Flow Evaluation

A video recording would be made of each referee’s in-game performance.

These videos would then be transmitted to official assessors who would review the

footage and assign each referee a game flow understanding rating using expert

opinion. An assigned game flow understanding rating would be taken as a means of

assessing overall referee quality. This test is a video based version of the same

evaluation currently administered to referee grade 7-1.

Estimated Cost

This assessment method utilizes an official assessor as well as technical

equipment and operators to document the referee’s performance and assess it.

Technical equipment includes a video camcorder, a tripod to secure it and a

cameraman to operate the camera. An evaluation of all grade 8 referees would

result in an estimated cost of $337,995.

Table 6: Equipment Cost for Game-Flow Evaluation

Equipment Cost (per one)Camera $849.00

Camera Man $30.00Tripod $60.00



Assessor Fee $25.00

Table 7: Total Estimated Cost for Game Flow Evaluation

Equipment Equipment Cost Cost/RefCost / Year

(5319 Referees)

Camera, Camera Man, Tripod, and Assessor Fee

$45,450.00 $55.00 $337,995.00

Combined Evaluation

This method utilizes both the baseline fitness test and the game flow

evaluation to assign each grade 8 referee fitness and game flow understanding

ratings as a means of assessing overall quality. Although this methodology utilizes

the same equipment and resources of the previous two alternatives, due to differing

resource allocation the total estimated cost is not the direct sum of the previous two

estimated costs. Evaluating all grade 8 referees in this fashion would require an

estimated cost of $341,870.

No Assessment

Under this alternative, no assessment is conducted to assess the game flow

understanding or fitness attributes of referees. This alternative exists as a point of

reference against which to compare the cost and benefit of the three preceding

alternatives and represents the status quo for assessments at the grade 8 level

requiring no implementation cost.



EVALUATION OF ALTERNATIVES

The utility of each alternative is defined as the expected call accuracy of the

top 100 referees identified using each alternative within the junior referee pool of

MDCVSRP (roughly 5000 referees). In order to determine the utility of each

alternative, a two part analysis was conducted to select the most beneficial system

for grade 8 referees.

The first part of the analysis utilized a stochastic discrete event simulator

modeling a referee’s ability to position and make calls based on fitness and game

flow understanding attribute levels (Figure 7). Through performance evaluation of

25 referee profiles defined as combinations of fitness and game flow understanding

attributes (scaled 0 – 100), the simulator was used to generate a regression

equation quantifiably describing the impact of fitness and game flow understanding

on a referee’s call accuracy.

Figure 7: Stochastic Discrete Event Soccer Game Simulator



The second part of the analysis consists of a Monte Carlo analysis in which

5000 referees were randomly generated (representing grade 8 referees in

MDCVSRP) with independent fitness and game flow understanding attribute levels.

Utilizing the regression from part I, call accuracy was calculated for each of theses

referees. The utility of the No Assessment alternative was defined as the mean

average call accuracy of referees within this pool over 30 scenarios. Each remaining

evaluation program is used to identify the top 100 referees for each of the 30

scenarios. The mean average call accuracy of these 100 referees is used to represent

the utility for each alternative.

PART I: DISCRETE EVENT SOCCER GAME SIMULATOR

The original concept for this simulator was derived from a previous George

Mason University student project [8]. This concept consisted of probability guided

ball movement over a soccer field grid where a modeled referee would position and

respond to randomly generated call events [8]. From this initial concept, the

simulator used in this project was redesigned and coded independently of past

work. For a direct comparison between the simulator used in this project verses the

pervious simulator, see Table 8.

Table 8: Solomon et. Al Simulation vs. New Simulation

Simulation Element Solomon, et. Al. New Simulation

Probability Maps1 map for all teams, all time

and all score19 maps dependent on team, time, and score

Ball Position Function 1 event 4 state cycle scaled to time

Referee Position Function1-D, chase ball on left

diagonal2-D, based on GFU, scaled to

timeFitness 3 levels 5 levels

Game Flow Understanding None5 levels based on probability

maps

Call Grids NoneDetermined from survey

administered to 16 senior state referees

Call Event Trigger Simple ProbabilityCall grids and position in

cycleDistance vs. Call Accuracy Estimated Figure of Merit Determined from survey



Functionadministered to 16 senior

state referees and generated regression

Number of Teams in Game Home vs. Home Home vs. Away (4 options)Number of Teams Simulated 1 4

Team Strategy Changes Never Time / ScoreReferee/Ball Movement

Scaled to TimeNo Yes

Ball Movement

The stochastic soccer game simulator divides a soccer field into a fine set of

8,510 square cells where each cell represents a 1 x 1 yard area. Each of these cells is

allocated to 1 of 60 movement polygons (Figure 8) and 1 of 24 call grids.

Figure 8: Movement Polygons

Throughout a 90 minute simulated game, the ball moves from cell to cell

adhering strictly to a play cycle of four events. This cycle begins with a pass

reception (0.5s) and transitions into local dribbling (4.5s) in which the ball moves

within its current polygon. This is followed by either a shot on goal (0.5s) or a pass

(0.5s) (Figure 9). If a pass, the ball will move to its reception location over a period

of time depending on the distance traveled (See Equation 1). This play cycle repeats

until the simulation terminates.



Figure 9: Cycle of Events

Number of RefreshRates=2√(Start x−Finish x)2+(Start y−Finish y )2

Speed∈ yardsseconds

Equation 1: Number of Refresh Rates

As the ball moves throughout the play cycle, it refreshes its position every 0.5

seconds of simulated game time. At any instant, the ball is possessed by one of two

teams, each executing its own unique strategy. For each team, a set of probability

maps represents that team’s strategy and style of play. For each of the 60 polygons,

these maps specify the probability that the ball moves to any other polygon or is

shot at the goal. A further dimension of the map indicates probabilities that a pass or

shot is successful. Changes in possession occur due to failed passes or shot events

(Figure 10).



Figure 10: Possession Change Between Two Teams

Based on data collected from 80 English Premier League games, probability

map sets were formulated for 4 teams: Wigan, Manchester United, Arsenal, and

Stoke. These teams were chosen to give a broad representation of different play

styles and enable the simulator to replicate a vast number of game flow situations.

Data was collected from the Guardian Chalkboard website which tracks English

Premier League games and records all pass and shot events in the form of vectors

(Figure 11). In order to collect this data a java based data collection tool was created

and with it over 35,000 shot and pass events (80 games) were collected manually.



Figure 11: Guardian Chalkboard Data

Using pass completion as a metric representing team strategy, an ANOVA

analysis was conducted (Table 9) to determine if teams changed their strategy

based on score differential (ahead, behind, tie) or elapsed game time (divided into 6

discrete 15 minute time periods). The results of this analysis were used to

determine how many probability maps were needed to encapsulate each team’s

strategy and when maps should be changed, based on situation, to reflect strategy

alterations.

Table 9: Effect of Game Situation on Pass Completion

Situation Arsenal United Stoke WiganTime p = 0.777 p = 0.142 p = 0.001 p = 0.001Score p = 0.231 p = 0.001 p = 0.000 p = 0.000

Time*Score p = 0.338 p = 0.000 p = 0.000 p = 0.116

It was concluded that Arsenal utilizes a single probability map for all game

situations. Stoke, Manchester United, and Wigan utilize six probability maps each

representing situations where the team is ahead, behind, or tied in the first and

second half respectively (19 total maps formulated).



Upon concluding the dribbling event in the play cycle, the probability that the

ball is passed (versus shot at the goal) depends on game situation determined using

the active probability map of the team with possession. If a shot occurs, the active

map indicates the destination polygon of the pass and chance of success (Equation

2).

Polygon (n+1 )=Polygon (n )× Probability MapEquation 2: New Polygon Movement Algorithm

If a shot occurs, the active map indicates the probability that the shot will

result in a goal. Executing passes and shots in this fashion allows the simulator to

accurately represent the flow of a soccer game in which a referee must interact.

To ensure the time of ball movement accurately represents that of a soccer

game, whenever the ball is being dribbled or passes a single destination cell is set.

The ball moves to that destination in a straight-line trajectory, which it follows for a

duration of simulated time (Equation 1).

Referee Movement

In the simulation, a single referee is modeled running within a standard

diagonal system of control 2 – dimensional area (Figure 12). The speed of the

referee is calibrated to represent the fitness level of the referee profile being tested.

Every 0.5 seconds, the referee sets his desired position using one of two

movement scripts. In script I, the referee sets his destination to the closest cell

within 11 – 13 yards of the ball’s current location. This script represents a referee

positioning himself in a reactive manner. In script II, the referee sets his destination

to the closest cell with 11 – 13 yards of the next most probably pass destination as

determined using the active probability map of the team with possession. This script

represents a referee positioning himself in a proactive manner



Figure 12: Referee 2-D Movement Area

Upon setting his destination using script I or II, a referee will begin moving

towards his destination using the same straight line movement algorithm described

previously for ball movement (Equation 1).

At the beginning of each play cycle, the probability that the referee utilizes

script II is determined by the referee’s game flow understanding level (higher game

flow understanding yields higher probability). Furthermore, this same game flow

understanding probability is used to determine the likelihood that if a call were to

occur in the current cycle, the referee will recognize the buildup to the call and

switch to script I until the call transpires.

Call Events

At the beginning if each play cycle, the ball location is used to reference a set

of probabilities indicating probability that the referee will need to make a call in that

cycle (Figure 13). These probabilities were developed using an expert survey

administered to 16 senior referee within the MDCVSRP and tailored to ensure that

roughly 65 call events occur per game (See Appendix for survey questions and

results). Data from the survey were also used to determine the probability of the call

event occurring at the receiving (0.21), dribbling (0.44), passing (0.21), or pass en

route (0.15) events of the play cycle.



Figure 13: Call Event Probabilities Based on Field Location

When a call event occurs, the probability of a correct referee decision is

determined based on the referee’s distance to the ball. This is assuming that the

calls occur at the location of the ball. In order to develop an understanding of the

effect distance has on a referee’s call accuracy, 16 senior MDCVSRP referees were

surveyed asking them to rate call accuracy at a series of 12 distances. Using the

results of this survey, a regression was performed relating the probability of making

a correct call to a referee’s distance from the call (Figures 14, 15).

Figure 14: Call Accuracy Function (Distance <20 yards)



Figure 15: Call Accuracy Function (Distance >20 yards)

The results of this regression was a piecewise equation with call accuracy

peaking at around 11-13 yards and decreasing at a rapid pace past 20 yards getting

to as low as ~28% at 60 yards. It should be noted however that the average

standard deviation for the 12 distances polled on the survey was 21.3%, indicating

disagreement among participants.

Over the course of the simulated game, the call accuracy of a referee is

defined as the number of correct calls divided by the total number of calls made.

Simulation Methodology

To determine the impact of fitness and game flow understanding on call

accuracy, each of the 25 distinct referee profiles representing different

combinations of fitness and game flow understanding (scale from 0 – 100) was

simulated through 2000 games representing 200 games for each combination of the

Arsenal, Manchester United, Stoke, and Wigan play styles. Referee speeds

corresponding to profile fitness ranged linearly from 2.023 yards/second at fitness

= 0 to 3.911 yards/second at fitness = 100. Probabilities corresponding to profile

game flow understanding ranged linearly from 0.25 at GFU = 0 to 0.90 at GFU = 100

(Table 10). The average call accuracy for each profile over the simulated games was



used to formulate a multivariate regression for call accuracy as a function of fitness

and game flow understanding level.

Table 10: Referee Profiles

PART 2: MONTE CARLO ANALYSIS

For each Monte Carlo scenario, 5000 referees are randomly generated under

the assumption that each referee’s fitness and game flow understanding levels are

uncorrelated and represent independent draws from normal distributions (mean

50, standard deviation 15) (Figure 16). In Figure 16, the variable X on the x-axis

represents one of the two independent referee traits.



Figure 16: Normal Distribution for Referee Attributes

Using the regression from the part I analysis, average call accuracy for each

referee profile was determined. Using normal cumulative density functions to

ensure the selection of roughly 100 top referees, fitness and/or game flow

understanding cutoffs were defined for the first three alternatives based on

attributes (Table 11).

Table 11: Design Alternatives Attribute Cutoffs

AlternativeAttribute Assessed

CutoffAvg. # of

Referees ChosenFitness Test Fitness Fitness > 81 97Game Flow Evaluation

Game Flow Understanding

GFU > 81 97

Combined Evaluation

Fitness, Game Flow Understanding

Fitness > 66 &Game Flow

Understanding > 66102

No Assessment N/A N/A 100

The mean average call accuracy of selected referees over 30 scenarios was

used to define the utility of these alternatives. The utility of the No Assessment

alternative was defined simply as the mean average call accuracy of referees within

each pool. The analytical method assumes that each alternative has an idealized

ability to evaluate the attributes assessed.



RESULTS

Discrete Event Soccer Game Simulator Results

Analysis of each of the 25 referee profiles over 2,000 simulated games

yielded results for average call accuracy as a function of fitness and game flow

understanding (Figure 17).

Figure 17: Analysis of 25 Profiles: Call Accuracy (Fitness, GFU)

Fitness and game flow understanding levels are scaled where a rating of 0 is

the worst possible and 100 the best possible. In Figure 17, the z-axis represents the

call accuracy where as the x-axis and the y-axis represent the fitness and game flow

understanding levels. Across the referee profiles, call accuracy ranged from 71.22%

to 75.67%. The highest call accuracy resulted during the highest levels of fitness and

game flow understanding levels. By increasing fitness to its maximum, call accuracy

peaked at roughly 74.5% where as increasing a referee’s game flow understanding

level to its maximum resulted in a peak accuracy of roughly 72.5%. This implies that

a referee’s fitness level has a greater impact on their ability to make a correct call

then their game flow understanding attribute. These findings are fairly intuitive

when analyzed. A referee can have a high game flow understanding level and be able



to proactively position themselves based on probable ball movement but no matter

how well the referee can understand the flow of the game if they are physically inept

from moving from one point to another their high level of game flow understanding

is decently minimized. Over the 25 profiles, the average 95% confidence interval

half-width for mean call accuracy was 2.866e-3. This indicates an acceptable level of

confidence in the data points.

A multivariate regression for call accuracy was computed with an R-squared

value of 99.51% representing a strong fit (Equation 3).

Call Accuracy ( Fitness ,GFU )=¿0.713491+(0.000923486 × Fitness )+ (1.28791×10−5× GFU )−(6.4846 ×105 × Fitness2)+ (1.12504 ×GFU 2)+ (1.26193× 10−6 × Fitness3 )−(6.75305× 10−9 × Fitness4)

Equation 3: Call Accuracy Regression Function

The regression analysis indicates that accuracy varies nonlinearly with

fitness and game flow understanding. Adding polynomial terms for fitness and game

flow understanding until p-values for leading terms jumped above acceptable levels

(p > 0.05) resulted in a fitness degree of 4 and game flow understanding degree 2

(Table 12). The generated regression does not include an interaction term between

fitness and game flow understanding, since adding an interaction term resulted in a

p-value of 0.813.

Table 12: Call Accuracy Regression Analysis

Term T P - valueConstant 1042.55 0.000Fitness 7.84 0.000Fitness2 -10.97 0.000Fitness3 13.35 0.000Fitness 4 -14.36 0.000

GFU 0.55 0.590GFU2 4.99 0.000

Monte Carlo Analysis Results



Utilizing the Call Accuracy equation generated from the regression above, the

expected call accuracy was calculated for the 5000 randomly generated referees in

each Monte Carlo scenario. Using the established attribute cut offs the mean call

accuracy was calculated for each referee pool selected using each assessment

alternative. The Monte Carlo analysis implied that the most effective evaluation

method was the Fitness test, which had an average call accuracy of 74.9%, followed

by the Combined Evaluation (74.2%), Game Flow Evaluation (72.7%), and No

Assessment (72.1%) (Table 13).

Table 13: Utilities for Grade 8 Evaluation Alternatives

Alternative CutoffAvg. Call Accuracy

95% Half-Width Call Accuracy

Fitness Test Fitness > 81 0.74926 0.00012Game Flow Evaluation

Game Flow Understanding > 81

0.72693 0.0028

Combined Evaluation

Fitness > 66 & Game Flow

Understanding > 660.74174 0.00021

No Assessment N/A 0.72099 0.00004

The extremely small 95% confidence interval half-widths indicate a high

level of confidence in the results.

UTILITY/COST ANALYSIS AND RECOMMENDATION

Based on a cost vs. utility analysis conducted on alternatives (Figure 18) it

can be concluded that the Fitness Test dominates both the Combined Evaluation and

Game Flow Evaluation due to its higher utility and lower cost. Therefore, the choice

of alternatives lies between conducing a Fitness Test at grade 8 (74.9% Accuracy,

$26,990 Cost) and conducting no assessments at this level (72.1% Accuracy, $0

Cost). As the average accuracy of the top 100 referees selected using the Fitness

Test exceeds the overall referee accuracy by only 2.8 percentage points, the

improvement in selection due to implementing the Fitness Test over the status quo

can be considered statistically but not practically significant. Thus, the benefit of



implementing Fitness Tests for all grade 8 referees is outweighed by its cost. It is

therefore the recommendation of this project that the status quo be maintained and

no referee evaluations be conducted for fitness and/or game flow understanding at

the grade 8 level.

Figure 18: Cost vs. Utility Analysis for Alternatives

ADDITIONAL FINDINGS

To determine the effect of game flow on a referee’s call accuracy, an analysis

was conducted on the extent to which referee call accuracy was affected by the

playing styles of the teams competing in a game (See Figure 25). Based on the range

of performance from best performing referee profile to worst profile (indicated by

error bars), it can be concluded that team playing styles can have a significant

impact on referee performance.



Figure 19: Impact of Team Combinations on Referee Call Accuracy

Given this finding, further analysis was conducted to determine why certain

team combinations result in decreased referee performance. 500 simulated games

were run using referee profile 33 (fitness =50, game flow understanding = 50) for

Stoke vs. Stoke and United vs. United play styles. Over these games, the simulated

referee made roughly 30,000 calls for each team combination. For all call events, the

distance from the call was recorded and analyzed.

It was concluded that differences exist in the distributions of call distance as

a result of team play styles. United vs. United games resulted in density

concentrating heavily around 11-13 yards and decreasing consistently with further

increases in distance (See Figure 26). However, Stoke vs. Stoke games resulted in a

bimodal density concentrating around 11-13 yards and again at around 44-47 yards

(See Figure 26). This second peak along with the increased density between peaks

accounts for the decreased referee performance in Stoke vs. Stoke games due to the

referee more frequently being out of position to make calls. This analysis shows that

the same referee when placed in two different games can have a decreased

performance and be out of position far more often in one game due exclusively to

different team combinations and their effect on game flow.



Figure 20: Distance from Calls for United vs. United and Stoke vs. Stoke

The results of this finding are nontrivial and point to a key characteristic that

is currently lacking in referee criticism and evaluation. When assessing the in-game

call performance of a referee, the difficulty of the match being officiated (in terms of

game flow) must be taken into account due to its large and unavoidable effect on call

performance. Furthermore, when comparing the performance of different referees,

the games in which referees are evaluated must be synchronized to ensure that

team combination does not act as a confounding variable in the analysis.



MANAGEMENT

The following figure is the work breakdown structure for the entire project.

Its split up into two sections, the first is for SYST 490 and the second part is the

work that was completed during SYST 495.

Figure 21: Total Work Breakdown Structure for SYST 490/495

The following figure shows the work that was completed for Systems 490. It

focused mainly on gathering data for the discrete soccer game simulator.

Figure 22: Work Breakdown Structure for SYST 490



The following figure shows the work that was completed for Systems 495. It

focused on creating the simulation, formulating conclusions, and presenting the

project in conferences.

Figure 23: Work Breakdown Structure for SYST 495

The following table shows the entire list of tasks that were set out for this

project. It shows the dates for each task, the amount of hours that have been

allocated to the task, and who is assigned to complete the task. The project was

budgeted to take a total of 1462 hours. At $30.00 an hour the total cost of this

project is estimated to be $43,860.

Table 14: Task Breakdown

Outline Number Task Name Duratio

n Start Finish Hours Assignee

1

Soccer Referee

Evaluation System

248 days

Mon 8/29/11

Thu 5/3/12 1462

1.1 Research 219 days

Mon 8/29/11

Tue 4/3/12

1.1.1 Preliminary Research

48 days

Mon 8/29/11

Sat 10/15/1

160 All

1.1.2 Ongoing 171 Sun Tue 75 All



Research days 10/16/11 4/3/12

1.2 Referee/Game Data

138 days

Fri 9/2/11

Tue 1/17/12

1.2.1Simulation input data Collection

138 days

Fri 9/2/11

Tue 1/17/12

1.2.1.1 Ball speed data

29 days

Mon 12/19/1

1

Tue 1/17/12 30 Saud

1.2.1.2Ball

movement data

88 days

Fri 9/2/11

Mon 11/28/1

1

1.2.1.2.1 Develop data collector

50 days

Fri 9/2/11

Fri 10/21/1

120 Nathan

1.2.1.2.2Team

selection for strategy

7 days Mon 10/3/11

Mon 10/10/1

12 Saud

1.2.1.2.3 Data collection 30 days

Sat 10/22/1

1

Mon 11/21/1

175 All

1.2.1.2.4Data analysis for strategy

trends7 days

Mon 11/21/1

1

Mon 11/28/1

120 Andrew

1.2.1.3Call Ability &

Frequency Function

11 days

Fri 11/18/1

1

Mon 11/28/1

1

1.2.1.3.1Situational

Characteristics Survey

9 daysFri

11/18/11

Sat 11/26/1

1

1.2.1.3.1.1 Create survey 2 days

Fri 11/18/1

1

Sun 11/20/1

110 Hina

1.2.1.3.1.2

Send survey to referees 4 days

Mon 11/21/1

1

Fri 11/25/1

115 Hina

1.2.1.3.1.3

Analyze survey results 1 day

Fri 11/25/1

1

Sat 11/26/1

110 Hina

1.2.1.3.2 Create call ability function 1 day

Sun 11/27/1

1

Mon 11/28/1

12 Hina

1.2.2 Simulation 30 Mon Wed



input data analysis days 11/28/1

112/28/1

1

1.2.2.1Creation of

call accuracy functions

30 days

Mon 11/28/1

1

Wed 12/28/1

140 Hina

1.2.2.2

Creation of ball

movement matrices

30 days

Mon 11/28/1

1

Wed 12/28/1

125 Nathan

1.2.2.3Creation of

call occurance matrices

30 days

Mon 11/28/1

1

Wed 12/28/1

120 Nathan

1.3Referee

Evaluation Simulator

37 days

Mon 12/19/1

1

Tue 1/24/12

1.3.1 Development 36 days

Mon 12/19/1

1

Mon 1/23/12

1.3.1.1 Define "sim" referees 7 days Mon

1/9/12Mon

1/16/12 20 Andrew

1.3.1.2 Develop simulator

30 days

Mon 12/19/1

1

Tue 1/17/12 250 Nathan/Hina/

Andrew

1.3.1.3 Test simulator 7 days Mon 1/16/12

Mon 1/23/12 20 Nathan

1.3.2 Evaluation 8 days Tue 1/17/12

Tue 1/24/12

1.3.2.1Run "Sims"

through simulator

7 days Tue 1/17/12

Tue 1/24/12 10 Nathan

1.3.2.2 Record "Sim" performance 7 days Tue

1/17/12Tue

1/24/12 3 Nathan

1.4 Formulation of Conclusions

14 days

Wed 1/25/12

Wed 2/8/12

1.4.1 Analyze "sim" performance

14 days

Wed 1/25/12

Wed 2/8/12 30 All

1.4.2

Determine ratings for

survey metric combinations

14 days

Wed 1/25/12

Wed 2/8/12 10 All

1.5 Literature Review

49 days

Mon 2/13/12

Mon 4/2/12 300 All



1.6 Communication of Results

215 days

Fri 9/30/11

Thu 5/3/12

1.6.1 Preparation of Deliverables

185 days

Fri 9/30/11

Mon 4/2/12

1.6.1.1 Preliminary Project Plan 2 days Fri

9/30/11Sun

10/2/11 25 All

1.6.1.2 Final Report Proposal

10 days

Fri 11/25/1

1

Mon 12/5/11 40 All

1.6.1.3 Proposal Final Report Slides

10 days

Fri 11/25/1

1

Mon 12/5/11 20 Hina

1.6.1.4 Conference Paper Draft

15 days

Sun 11/20/1

1

Mon 12/5/11

40 All

1.6.1.5 Poster Draft 15 days

Sun 11/20/1

1

Mon 12/5/11

40 All

1.6.1.6 SIEDS Abstracts Due (University of

Virginia)

5 days Wed 2/8/12

Mon 2/13/12

20 All

1.6.1.7 Final Conference

Paper

48 days

Tue 2/14/12

Mon 4/2/12

30 All

1.6.2 Presentations 213 days

Mon 10/3/11

Thu 5/3/12

1.6.2.1 Project Briefing # 1

0 days Mon 10/3/11

Mon 10/3/11

30 All

1.6.2.2 Project Briefing # 2

0 days Mon 10/24/1

1

Mon 10/24/1

1

40 All

1.6.2.3 Dry Run Final Presentation

0 days Wed 11/9/11

Wed 11/9/11

50 All

1.6.2.4 Faculty Presentation

0 days Fri 12/2/11

Fri 12/2/11

30 All

1.6.2.5 SIEDS Conference (University

and Virginia)

0 days Fri 4/27/12

Fri 4/27/12

20 All

1.6.2.6 Westpoint Capstone

Conference

0 days Thu 5/3/12

Thu 5/3/12

30 All



The following chart is the PERT chart for the project. It shows that the

Referee Evaluation Simulator had no slack time. In order to make sure the simulator

was completed properly within the allotted time, a test as you go coding method

was used to reduce the time of the debugging process. Formulating conclusions

must be completed by the identified date. If needed, additional hours will be added

into the budget to make sure that these tasks are completed.

Figure 24: PERT Chart

The following charts show the project cost. The earned value and planned

value are very close showing that the project was completed on time. The actual cost

is lower than anticipated. The reason for this is was that work was completed more

efficiently than initially anticipated. In addition, the Cost Performance Index (0.955)

and the Schedule Performance Index (0.967) are both in a range indicating the

project was carried out successfully.

Table 15: Task Budgeting

Task Predicted Velocity Cost

Research 135 hours $4,050

Referee/Game Data 244 hours $7,320

Referee Evaluation Simulator 303 hours $9,090

Formulation of Conclusions 40 hours $1,200



Communication of Results 415 hours $12,450

Project Management 330 hours $9,900

Total 1467 hours $44,010

Table 16: Earned Value

Week Planned Value Actual CostEarned Value

1 11.42 17.95 8.572 22.84 43.05 19.983 34.26 71.65 31.414 45.68 102.05 42.845 82.1 178.45 109.276 125.52 207.75 122.77 136.94 250.85 134.148 179.79 298.55 1379 201.54 334.55 137

10 253.29 368.05 205.7511 295.04 416.05 224.512 326.79 447.05 253.2513 451.79 488.05 33214 576.04 552.55 39615 600.29 554.55 39616 624.54 556.55 39617 718.79 589.55 41618 791.79 622.55 478.519 864.79 655.55 54120 957.79 701.05 613.521 997.79 733.05 736.2522 1020.79 762.05 780.523 1043.79 788.05 831.7524 1066.79 827.55 88925 1116.94 861.05 928.7926 1167.09 881.05 988.5827 1217.24 904.05 1030.3728 1267.39 926.55 1047.8729 1317.54 956.55 1047.8730 1367.69 981.05 1055.1631 1417.84 1016.05 1062.4532 1427.84 1029.05 1069.74



33 1437.84 1092.85 142034 1447.8435 1457.8436 1467.84

1 4 7 10 13 16 19 22 25 28 31 340

300

600

900

1200

1500

Earned Value Chart

Planned ValueActual CostEarned Value

Week (starting 8/29/2011)

Co

st (

ho

urs

)

Figure 25: Earned Value Chart

Cost Performance Index=¿

Earned Value∗Project Completion PercentageActual Cost

=0.955

Schedule Performance Index=¿

Earned Value∗Project Completion PercentagePlanned Value

=0.96



WORKS CITED[1] C. Gordine-Wright, Z. Reilly, (2011, June 9) European football market grows

to €16.3 billion. [online]. Available: http://www.deloitte.com/view/en_NL/nl/7fdea05260570310VgnVCM2000001b56f00aRCRD.htm

[2] J. Wilson. (2010, June) Soccer could use instant replay, but not at expense of sport’s flow. [online]. Available: http://sportsillustrated.cnn.com/2010/soccer/world-cup-2010/writers/jonathan_wilson/06/28/soccer.technology/index.html

[3] (2003, April) United States Soccer Federation Referee Grades. [online]. Available: http://www.pawestsoccer.org/Assets/ documents/Announcement forgradechanges.pdf

[4] Definitions of Referee Grades [online]. Available: http://www.vadcsoccerref.com/docs/DEFINITION%20OF%20REFEREE%20GRADES.pdf

[5] Pat Delaney (2011, January 3) Annual Assessor, Assignor, Instructor and Administrator Meeting. [Presentation].

[6] Pat Delaney (2011, November 10) MDCVSRP Sponsor Meeting [Verbal]

[7] Assessment Program Handbook [online]. Available:http://www.ussoccer.com/Referees/Referee-Development/~/media/729EB9FF07A34EC5AAD7504A6E78ECCB.ashx

[8] A. Solomon, A. Paik, T. Phan, A. Alhauli, (2011) A Decision Support System for the Professional Soccer Referee in Time-Sensitive Operations. [online]. Available: http://catsr.ite.gmu.edu/SYST490/DSTSO_IEEE_SIEDS.pdf


Appendix 4/23/2012

APPENDIX

ANOVA Analysis

Arsenal

One Two Three Four Five Six74.00%

76.00%

78.00%

80.00%

82.00%

84.00%

86.00%

Arsenal - Pass Completion Percentage by Time / Score

AheadTieBehind

15 Minute Time Period

Pas

s Co

mle

tion

Per

cen

tage

Figure 26: Arsenal - Pass Completion Percentage by Time/Score

General Linear Model: Pass Success versus Time Period, Score Factor Type Levels ValuesTime Period fixed 6 1, 2, 3, 4, 5, 6Score fixed 3 Ahead, Behind, Tie

Analysis of Variance for Pass Success, using Adjusted SS for Tests

Source DF Seq SS Adj SS Adj MS F PTime Period 5 0.8009 0.3850 0.0770 0.50 0.777Score 2 0.6035 0.4526 0.2263 1.47 0.231Time Period*Score 10 1.7369 1.7369 0.1737 1.13 0.338Error 9393 1448.8720 1448.8720 0.1543Total 9410 1452.0134S = 0.392747 R-Sq = 0.22% R-Sq(adj) = 0.04%

SYST 495 Final Report i

Appendix 4/23/2012

Manchester


10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Manchester United - Pass Completion Percentage by Time / Score

AheadTieBehind


Pas

s Co

mle

tion

Per

cen

tage

Figure 27: Manchester United - Pass Completion Percentage by Time / Score

General Linear Model: Pass Success versus Time Period, Score

Factor Type Levels ValuesTime Period fixed 6 1, 2, 3, 4, 5, 6Score fixed 3 Ahead, Behind, Tie


Source DF Seq SS Adj SS Adj MS F PTime Period 5 1.5800 1.2753 0.2551 1.65 0.142Score 2 0.5949 2.2746 1.1373 7.37 0.001Time Period*Score 10 10.5742 10.5742 1.0574 6.85 0.000Error 7915 1221.5527 1221.5527 0.1543Total 7932 1234.3018

S = 0.392854 R-Sq = 1.03% R-Sq(adj) = 0.82%

SYST 495 Final Report ii

Appendix 4/23/2012

Stoke


10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Stoke City - Pass Completion Percentage by Time / Score

AheadTieBehind


Pas

s Co

mle

tion

Per

cen

tage

Figure 28: Stoke City - Pass Completion Percentage by Time / Score





S = 0.474260 R-Sq = 2.84% R-Sq(adj) = 2.63%

SYST 495 Final Report iii

Appendix 4/23/2012

Wigan


10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Wigan - Pass Completion Percentage by Time / Score

AheadTieBehind


Pas

s Co

mle

tion

Per

cen

tage

Figure 29: Wigan - Pass Completion Percentage by Time / Score





S = 0.447638 R-Sq = 0.94% R-Sq(adj) = 0.76%

SYST 495 Final Report iv

Appendix 4/23/2012

Simulation Output: Regression Analysis

Data Points

Table 17: Regression Analysis Data Points

Profile Fitness GFUAccurac

y11 0 0 0.713912 0 25 0.712513 0 50 0.717914 0 75 0.721315 0 100 0.726221 25 0 0.715322 25 25 0.714223 25 50 0.715524 25 75 0.719125 25 100 0.725831 50 0 0.712232 50 25 0.713033 50 50 0.719034 50 75 0.720135 50 100 0.725441 75 0 0.736342 75 25 0.738243 75 50 0.740844 75 75 0.743045 75 100 0.749551 100 0 0.743552 100 25 0.744953 100 50 0.748454 100 75 0.750855 100 100 0.7567

General Regression Analysis: Accuracy versus Fitness, GFURegression Equation

Accuracy = 0.713491 + 0.000923486 Fitness + 1.28791e-005 GFU - 6.4846e-005 Fitness*Fitness + 1.12504e-006 GFU*GFU + 1.26193e-006 Fitness*Fitness*Fitness - 6.75305e-009 Fitness*Fitness*Fitness*Fitness

Coefficients

SYST 495 Final Report v

Appendix 4/23/2012

Term Coef SE Coef T PConstant 0.713491 0.0006844 1042.55 0.000Fitness 0.000923 0.0001177 7.84 0.000Fitness*Fitness -0.000065 0.0000059 -10.97 0.000Fitness*Fitness*Fitness 0.000001 0.0000001 13.35 0.000Fitness*Fitness*Fitness*Fitness -0.000000 0.0000000 -14.36 0.000GFU 0.000013 0.0000235 0.55 0.590GFU*GFU 0.000001 0.0000002 4.99 0.000

Summary of Model

S = 0.00117864 R-Sq = 99.51% R-Sq(adj) = 99.35%PRESS = 0.0000473882 R-Sq(pred) = 99.07%

Analysis of Variance

Source DF Seq SS Adj SS Adj MS FRegression 6 0.0050711 0.0050711 0.0008452 608.397 Fitness 1 0.0035778 0.0000855 0.0000855 61.526 Fitness*Fitness 1 0.0005429 0.0001672 0.0001672 120.354 Fitness*Fitness*Fitness 1 0.0001382 0.0002477 0.0002477 178.319 Fitness*Fitness*Fitness*Fitness 1 0.0002863 0.0002863 0.0002863 206.086 GFU 1 0.0004913 0.0000004 0.0000004 0.300 GFU*GFU 1 0.0000346 0.0000346 0.0000346 24.913Error 18 0.0000250 0.0000250 0.0000014Total 24 0.0050961

Source PRegression 0.000000 Fitness 0.000000 Fitness*Fitness 0.000000 Fitness*Fitness*Fitness 0.000000 Fitness*Fitness*Fitness*Fitness 0.000000 GFU 0.590477 GFU*GFU 0.000095ErrorTotal

Fits and Diagnostics for Unusual Observations

Obs Accuracy Fit SE Fit Residual St Resid 6 0.715322 0.713129 0.0006844 0.0021936 2.28597 R 13 0.719050 0.716541 0.0005977 0.0025088 2.46965 R

R denotes an observation with a large standardized residual.

SYST 495 Final Report vi

Appendix 4/23/2012

Monte Carlo TrialsTable 18: Monte Carlo Trials

SYST 495 Final Report vii

Fitness Virtual Both Nothing0.74969572

3 0.726793193 0.741860637 0.721207750.74945932

7 0.726480148 0.742312934 0.721023880.74898779

2 0.727937977 0.7412423020.72104507

60.74931834

9 0.726535364 0.7412058160.72088827

50.74880053

6 0.726326643 0.74146514 0.721029220.74962531

4 0.728024425 0.7432878510.72097714

30.74945335

3 0.727459887 0.7412372820.72088934

80.74955406

3 0.726086242 0.7419398430.72095182

90.74907090

2 0.727370923 0.7412265320.72110788

7

0.7484863 0.726851215 0.741076590.72084940

70.74920919

4 0.725415465 0.7419955040.72105511

80.74892375

2 0.726986663 0.7424869490.72090563

50.74963521

3 0.726578691 0.741948181 0.721066410.74942863

7 0.727739189 0.741744458 0.720897280.74937041

1 0.726607546 0.7411405640.72105886

40.74936226

5 0.726649863 0.7413251370.72081892

50.74959925

1 0.726643132 0.7412892530.72104055

40.74978618

6 0.728142026 0.7422515420.72104107

30.74915812

7 0.727288764 0.7419184380.72094958

30.74936796

6 0.725400257 0.742010705 0.72096444

0.74896588 0.727781771 0.7416446660.72095585

10.74926543

4 0.726033107 0.7418641010.72094249

50.74912048

5 0.728528591 0.7422377080.72118001

30.74917125

2 0.726735782 0.74130510.72095286

10.74841404

9 0.726793865 0.740461490.72120146

70.74903986

2 0.726566538 0.7417923050.72081306

30.74970903 0.72112837

Appendix 4/23/2012

Survey Administered to MDCVSRP Senior Referees Soccer Referee Simulator

Data Collection Survey

Nathan Jones, Hina Popal, Andrew Cann, Saud Almashhadi [email protected]

Personal Information:

Name: _______________

Email:_______________

Grade: _______________

Years of Experience: _______________

Call Occurrence Questions:

** Calls refer to both direct and indirect free kicks

How many calls (estimated) would you expect to make in a typical game: _______________

SYST 495 Final Report viii

mailto:[email protected]

Appendix 4/23/2012

Given the Field image below:

Please estimate the percent of total calls that occur in each cell: (Sum of percent designated to cells should be 100)

1:________ 13:_________

2:________ 14:_________

3:________ 15:_________

4:________ 16:_________

5:________ 17:_________

6:________ 18:_________

7:________ 19:_________

8:________ 20:_________

SYST 495 Final Report ix

Appendix 4/23/2012

9:________ 21:_________

10:_______ 22:_________

11:_______ 23:_________

12:_______ 24:_________

Please estimate what percentage of calls occur when: (Total sum of percentages should equal 100)

A player is dribbling the ball:_________

A player is in the process of passing the ball:_________

A passed ball is in route from one player to another: _________

A player is in the process of receiving a pass: _________

None of the above (Please Specify): _________

Referee Call Accuracy Questions:

The following questions are asking about the call making ability of a referee with regards to their distance from the call event. The word "event" refers to what has triggered the need for a call. Please answer the following questions to the best of your ability.

At a distance of 5 yards from the event, what percent chance does a referee have of making a correct call? __________


SYST 495 Final Report x

Appendix 4/23/2012











SYST 495 Final Report xi

Appendix 4/23/2012

The following questions are asking about the call making ability of a referee regarding blind spots with respect to where they are located from the call event. The word "event" refers to what has triggered the need for a call. Please answer the following questions to the best of your ability.

If a referee is 40 yards behind the event, what is the chance he misses a call due to obstruction of vision? _________








If a referee is even with the event, what is the chance he misses a call due to obstruction of vision? _________

SYST 495 Final Report xii

Appendix 4/23/2012

If a referee is 5 yards ahead of the event, what is the chance he misses a call due to obstruction of vision? _________








SYST 495 Final Report xiii

Appendix 4/23/2012

Electronic AppendixThe following elements can be found in the electronic appendix attached to the back of this report:

1. English Premier League Strategy Analysis – this includes the complete set of collected data, the analysis done for team strategy and the polygon output for each team.

2. Monte-Carlo Analysis – This includes the output of the excel based Monte-Carlo simulation system.

3. Simulation Output Analysis – This includes the complete data set for all 25-referee profiles. The output analysis for each referee profile, team-by-team output analysis and complete referee profile analysis. In all this portion of the appendix includes 36 pages of simulation output analysis.

4. Source code for software – This includes the source code for all developed software utilized throughout this project. This entails the source code for the data collection tool, the simulator, and the VBA strategy analysis scripts, which were used to analyze the English Premier League collected data for team strategy.

5. Impact of Teams Analysis – This includes the output from the call distance simulation trials for United vs. United and Stoke vs. Stoke (Mentioned in the additional findings section).

6. Software – This includes the compiled source code (functioning software) for the Simulator and the Data Collection tool.

7. Assessment Criteria National Assessment Program – This is a pdf copy of the National Assessment Program’s Assessment Criteria. This pdf was taken from the www.ussoccer.com. This document is the official assessment sheet utilized by assessors when assessing referees during their on-field assessments. This document provides insight into the qualitative and quantitative elements of the on-field assessment process.

8. Referee Administration Handbook – This is a pdf copy of the Referee Administration Handbook taken from www.ussoccer.com. This document is a complete guide for all referees and includes information such as: Referee Grade Definitions, Certification Criteria (e.g., Fitness Tests, Exams, Assessments, etc.), and logistical policies and procedures for referee conduct.

9. Referee Survey output Analysis – This is an excel file providing all information submitted as part of referee surveys. It also includes quantitative computations used to help generate probabilities and regressions for the discrete soccer game simulation.

10. Literature Review – This section of the electronic appendix includes a volume of academic articles relating to the scope of the work done for this project. It provides a compiled pdf with multiple articles for further reading to help expand upon the nature of this work for future iterations.

SYST 495 Final Report xiv

Introduction - catsr.vse.gmu.edu€¦ · Web viewThe word "event" refers to what has triggered...

Documents

Transcript of Introduction - catsr.vse.gmu.edu€¦ · Web viewThe word "event" refers to what has triggered...