edge.rit.eduedge.rit.edu/edge/C15004/public/docs/Final Report... · Web viewNeeds Statement....

RIT, Deptartment of Computer Engineering

Pocket Billiards Trainer

Senior Projects II: Project Final Report

Gregory Dias, Ryan Gault, Collin Reeser, Beau Sattora

5/12/2015

The Pocket Billiards Trainer is a visually interactive system that assists in improving mechanical skill and decision-making in pocket billiards games.

Pocket Billiards Trainer P a g e | 1

Contents1. Overview.................................................................................................................................................................................... 4

I. Needs Statement.................................................................................................................................................................4

II. Objective Statement........................................................................................................................................................4

III. Description.........................................................................................................................................................................4

2. Requirements Specification.............................................................................................................................................. 5

I. Overview................................................................................................................................................................................ 5

II. Customer Needs/Marketing Requirements..........................................................................................................5

III. Engineering Specifications..........................................................................................................................................6

3. Concept Selection...................................................................................................................................................................7

I. Overview................................................................................................................................................................................ 7

II. Display................................................................................................................................................................................... 8

III. Table-Top Detection......................................................................................................................................................9

IV. Computational Platform...............................................................................................................................................9

V. Player Input Sensor.......................................................................................................................................................10

VI. Wireless Protocol......................................................................................................................................................... 11

VII. Image Processing........................................................................................................................................................ 11

VIII. Visual Output...............................................................................................................................................................11

IX. Mounting Solution........................................................................................................................................................12

4. Design....................................................................................................................................................................................... 12

I. Overview............................................................................................................................................................................. 12

II. Camera................................................................................................................................................................................ 13

III. Image Processing..........................................................................................................................................................16

a. Overview..................................................................................................................................................................16

b. Table Feature Detection....................................................................................................................................16

c. Ball Detection.........................................................................................................................................................16

d. Ball Color Identification.....................................................................................................................................16

e. Process Flow...........................................................................................................................................................16

IV. External Sensor.............................................................................................................................................................20

V. Simulation/Feedback System...................................................................................................................................21

a. Feedback Mechanism......................................................................................................................................... 21


b. Simulation System................................................................................................................................................22

VI. Table Display System..................................................................................................................................................24

VII. Engineering Standards.............................................................................................................................................26

VIII. Multidisciplinary Aspects......................................................................................................................................26

IX. Background..................................................................................................................................................................... 26

5. Constraints and Considerations....................................................................................................................................27

I. Extensibility....................................................................................................................................................................... 27

II. Manufacturability.......................................................................................................................................................... 27

III. Reliability.........................................................................................................................................................................28

6. Cost Estimates...................................................................................................................................................................... 28

7. Testing Strategy................................................................................................................................................................... 28

I. Overview............................................................................................................................................................................. 28

II. Camera................................................................................................................................................................................ 29


a. Table Feature Detection....................................................................................................................................29

b. Ball Detection.........................................................................................................................................................29

c. Ball Color Identification.....................................................................................................................................30


a. Shot Detection........................................................................................................................................................30

b. Information Relay................................................................................................................................................ 30

c. Power Consumption............................................................................................................................................30

V. Simulation/Feedback System...................................................................................................................................30

a. Enter the testing mode of the simulation software on a laptop, to simplify interfacing with the software...........................................................................................................................................................................31


VII. Results............................................................................................................................................................................. 32

8. Risks.......................................................................................................................................................................................... 32

I. Overview............................................................................................................................................................................. 32

II. Camera................................................................................................................................................................................ 33


a. Overview..................................................................................................................................................................34

b. Table Feature Detection....................................................................................................................................35

c. Ball Detection.........................................................................................................................................................35


d. Ball Color Identification.....................................................................................................................................35

e. Risk Mitigation Design.......................................................................................................................................36

f. Uncertainties.......................................................................................................................................................... 36


a. Overview..................................................................................................................................................................36

b. Shot Detection........................................................................................................................................................37

c. Relay Information / Power Consumption.................................................................................................37

d. Risk Mitigation Design.......................................................................................................................................37

e. Uncertainties.......................................................................................................................................................... 37

V. Simulation System.........................................................................................................................................................37

a. Overview..................................................................................................................................................................37

b. Concepts Considered..........................................................................................................................................38

c. Concepts Chosen...................................................................................................................................................38

d. Risk Mitigation Design.......................................................................................................................................39

e. Possible Simulation Omissions.......................................................................................................................39

f. Implementation Overview................................................................................................................................41

g. Uncertainties.......................................................................................................................................................... 42


9. Milestone Chart....................................................................................................................................................................42

10. Perspective.......................................................................................................................................................................... 42

I. Camera................................................................................................................................................................................. 42

II. Image Processing...........................................................................................................................................................43

III. Simulation........................................................................................................................................................................44


V. Table Display System...................................................................................................................................................44

11. Acknowledgements......................................................................................................................................................... 45

A. Appendices........................................................................................................................................................................ 45

I. References.......................................................................................................................................................................... 45

II. Diagrams............................................................................................................................................................................46

III. Test Cases........................................................................................................................................................................ 47

IV. Milestones........................................................................................................................................................................67


1. Overview

I. Needs StatementPocket billiards is a popular family of cue sports, accessible to beginners but with a long road to mastery. Unguided practice will initially improve the skill of novice players, but the improvement gained solely from practicing this way will likely plateau at the level of “advanced beginner.” As advanced beginners, players have a working knowledge of the shallower aspects of the game, but lack the depth of knowledge and the mechanical skill characteristic of more competent players. Through access to resources capable of communicating the more elusive intricacies of the game, amateur players are able to propel themselves beyond the limits of basic intuition. Acquiring professional instruction is a prohibitive expense to most, while books and instructional videos lack the interactivity critical to improving quickly. A system is needed which bridges this gap, by providing instruction in developing refined mechanical skills and guiding fledgling players to a deeper understanding of the game in a direct, interactive, and cost-effective way.

II. Objective StatementThis project aims to design and implement a system capable of effectively improving the mechanical skill and intuitive understanding of the intricacies of pocket billiards for amateur players. The system will integrate with existing billiard table installations, and will be non-intrusive when the functionality of the system is not desired. Through this integration, the system will be able to interact directly with players as they practice and learn more about the game. The system will be capable of analyzing the positions of billiard balls on the table, performing shot simulations, and displaying visual prompts directly on the table. Through this method of interaction, the system will be able to instruct about and tightly couple theoretical principles with their practical implications, making efficient use of practice time in an engaging way.

III. DescriptionThis project will see the development of a non-intrusive system that integrates with existing billiard table installations. The system will extract information from the play area using overhead camera(s), and will use a projection device for displaying information onto the table. Computer vision techniques will be utilized in finding key game features such as the bounds of the table and the location and identity of the balls in the play area. Accurate software simulations of potential shots will be translated into variable degrees of visual guidance, such as aim lines, ghost ball visualizations, and textual instruction. Post-shot analysis will provide feedback to players about shots taken, maximizing the opportunities available for providing guidance toward improvement. In addition to in-game analysis and instruction, the system will also provide a series of drills that focus more on building player understanding of individual


components of the game, such as shot strength. A visualization of the potential system in use follows.

2. Requirements Specification

I. OverviewRequirements are an integral component of any project, as they define what the project is and any constraints that may be imposed on the design process. The project requirements are initially defined by the customer needs, which state what should be accomplished on a high level. The customer needs then dictate the engineering requirements, which specify the technical details of how the project components will be implemented.

For the Pocket Billiards Trainer system, the customer needs primarily focus on fulfilling the goal of assisting players in improving their skills while limiting the potential for the system to interfere with gameplay. The engineering specifications then expand upon how each customer need can be technically satisfied.

II. Customer Needs/Marketing Requirements1. The system shall not physically interfere with gameplay.2. The system shall not be affected by normal gameplay.3. The system shall be able to display accurately content on the table surface.4. The system shall be able to display content in a well-lit environment.5. The system shall not physically modify or alter the table.6. The system shall not permanently modify game tools (balls, rack, cue, etc.).7. The system shall work with standard pocket billiards equipment (balls, rack, cue,

etc.).


8. The system shall work on a standard size billiards table.9. The system shall be able to display a "best shot" for the current layout of the table.10. The system shall be able to select alternate shots instead of "best shot," should they

exist.11. The system shall display a set of visual markers (ghost ball, dotted line, etc.) to guide

the player's shot.12. The system shall be able to provide feedback on shot strength for player per shot.13. The system shall feel responsive in casual gameplay.14. The system shall be ceiling mountable.

III. Engineering SpecificationsMarketing Requirements Engineering Requirements Justification

1, 2, 3, 5, 6, 14

A. The system shall have a minimum overhead display distance of 4 ft. from the table surface.

Limitations of projector and camera hardware.

9, 10B. The system shall be able to

locate all balls on the playing surface.

For calculating the potential shots.

3, 4, 11C. The projection from the

display component shall be comfortably visible.

So the user can use the information to improve their gameplay.

7, 9, 10D. The system shall be able to

know each ball’s identity.

For calculating the best shot relative to the game being played.

4, 12E. The system shall have an

external/untethered sensor to detect shot strength.

To gather information about the player’s shot.

3, 7, 8

F. The system shall have a means to auto calibrate between system components and the pocket billiards equipment.

To simplify system setup and increase the accuracy of ball detection.

9, 10, 13

G. The system shall be powerful enough to perform all calculations in less than 10 seconds per turn.

To maintain a responsive system.

7, 8, 9, 10, 13

H. The system shall be able to simulate accurately interactions between game components to determine the outcome of potential shots.

To provide the foundation on top of which core system functionality is provided.

1, 14I. The system shall be

mounted above the table.

Limits the system from becoming intrusive to gameplay.


3. Concept Selection

I. OverviewThe proposed project is structurally similar to some existing systems, such that an overhead camera and projector combine image processing with projection directly onto a table. Of those systems, both Cassapa (named after the Portuguese word for a pool table pocket) and Pool Live Aid simply locate the position of the cue stick and draw lines on the table indicating the path travelled by each ball involved in the shot. These systems are real-time, and quickly and dynamically readjust the projection based on movement of the cue stick. Cassapa and Pool Live Aid do not provide any structured guidance on how to take a shot, what shots are better than others, or any feedback on the shot taken. Instead, they assume that the player will use a follow shot, and only project what will happen given the current orientation of the cue stick. That is, Cassapa and Pool Live Aid, by their nature, cannot provide path predictions based on the effects of a draw shot. These systems both essentially exist for demonstration purposes only, with no real benefit beyond entertainment value, as players using these systems can interact with them only as a toy.

ARPool (“Augmented Reality Pool”) is a system that appears to incorporate two distinct modes. The first mode is exactly analogous to that of Cassapa and Pool Live Aid. The second mode much more closely mirrors the intended goal of this project, performing pre-shot analysis of the table to determine a viable shot, and projecting guidance lines to assist the player in making the shot. By inspection of recent demonstration videos and images, however, it is clear that the algorithm by which the system determines a “best shot” is severely flawed, as often an objectively more difficult shot is highlighted in lieu of an objectively easier shot. In addition, once the cue stick is identified by the camera, the system reverts to the first mode. The ARPool system does appear to account for english on the cue ball, but it cannot account for backspin and assumes that the cue ball will be rolling upon first contact with an object ball, as with Cassapa and Pool Live Aid. The website for ARPool claims to be able to provide an “instant replay” of the shot taken, but no evidence supporting that claim could be found. The ARPool system is therefore only a shallow improvement over the other two systems.

The Pocket Billiards Trainer aims above all else to be directly involved in helping to improve player intuition and skill. The gimmick of real-time trajectory projection does nothing to help the player, especially if that is the only feature of the system. The ARPool “best shot” algorithm is clearly flawed, at least as last seen in recent demonstration videos, a feature upon which the Pocket Billiards Trainer will greatly improve. Further, the player’s ability to cycle through all available shots, and cycle through varying degrees of guidance, while also receiving textual hints for the mechanics of more complex shots, already outshines the existing systems with respect to their actual educational value. Finally, the feedback system for shots made will provide specific insight into the ways in


which the player can improve their mechanical skill. Overall, there does not appear to be an existing system that fully overlaps in functionality with the Pocket Billiards Trainer, at least in terms of execution.

Many concepts must be considered for the Pocket Billiards Trainer in order to minimize the complexity of the system while maintaining complete and efficient functionality. The following table shows a summary of all systems necessary, concepts considered for each, the concept chosen for each, and a brief description of its justification.

System Concept(s) Considered

Concept Chosen

Justification

Display Projector Table-Embedded

DisplayProjector

Most dynamic, yet non-intrusive system for displaying content on a pre-existing surface.

Table-Top Detection

Camera Pressure-

sensitive table surface

Camera

Easiest solution for obtaining all information for determining location and identity of balls as well as the bounds of the table.

Computational Platform

Embedded Linux Computer

Laptop/Desktop Computer

Microcontroller

Embedded Linux Computer

Small yet powerful platform with common integrated circuit communication protocols.

Player Input Sensor

TI SensorTag Accelerometer Gyroscope

TI SensorTagIncludes accelerometer and a combined programmable microcontroller and Bluetooth (BLE) in a single package.

Wireless Protocol

ZigBee Bluetooth (BLE) Wi-Fi

Bluetooth (BLE)Low power, high range protocol that is simple to use.

Image Processing

OpenCV MATLAB

OpenCVComputer vision standard libraries for image processing, available in many programming languages.

Visual Output OpenCV OpenGL SDL

OpenCVSimple shapes, lines and text can be drawn to an image. Allows re-use of OpenCV libraries.

Mounting Solution

Ceiling Tripod Swing-arm Tableside

Ceiling

Mounting the system to the ceiling is the least intrusive solution.

II. DisplayThe display component is responsible for displaying visual information on the billiards table to assist the player. The following components were evaluated for the display:

Projectoro Mid-to-high cost


o Large display areao Non-intrusive

Table-Embedded Displayo High Costo Requires multiple displayso Alters the playing surface

The chosen display solution is a projector because of its lower cost and adaptability to a wider range of projection surfaces. In addition, the projector will not alter the billiards table as per customer requirement 5. The projector will also easily facilitate the customer requirement of an overhead system.

III. Table-Top DetectionThe table-top detection component is responsible for obtaining the play area information for processing. The following components were evaluated for table-top detection:

Camerao Readily availableo Easy to interface witho Low cost

Pressure-sensitive table surfaceo Requires a custom solutiono High cost

The chosen table-top detection solution is a camera because it will be able to capture the table’s state fully at any given point within time with high precision without the use of several different sensors. It also has a simple interface to work with and is compact in size making it easy to mount and position.

IV. Computational PlatformThe computational platform component is responsible for handling all image processing, game calculations, and communication of information between other components. The following components were evaluated for the computational platform

Embedded Linux Computero Low-to-mid costo High computational capabilityo Compact size

Laptop/Desktop Computero Very High costo High computational capabilityo Bulky

Microcontroller


o Low costo Compact sizeo High computational capability for simple softwareo Less interfaces than a full computer

The chosen computational platform is an embedded Linux computer because of its fully featured operating system, compact size, and list of interfaces that come on most development boards. The power of most embedded Linux computers can run complex software with multiple systems and threads effectively.

V. Player Input SensorThe player input sensor component is responsible for reading the force of a player’s shot. The following components were evaluated for the player input sensor:

TI SensorTago Mid costo Includes accelerometero Includes configurable MCUo Includes Bluetooth (BLE)o Requires USB dongle

TI Keyfobo Mid-high costo Includes accelerometero Includes programmable MCUo Includes Bluetooth (BLE)o Includes USB dongleo Includes debugger

Accelerometero Low costo Single sensoro Measure g-force impacto Requires custom circuitry to process datao Multi-axis motion information

Gyroscopeo Low costo Single sensoro Measure angular velocityo Requires custom circuitry to process data

The chosen player input sensor is the TI SensorTag because it includes a 3-axis accelerometer, combined Bluetooth (BLE) and configurable MCU, and is easily synced with a USB dongle for wireless communication to the embedded Linux computer. This offers a lower cost solution while still not requiring a custom circuit board to be built.


The TI SensorTag comes as a small package, which may be useful as a sensor mounted to a person’s wrist or cue stick.

VI. Wireless ProtocolThe wireless protocol is responsible for the method in which data are communicated between the player input sensor and the embedded Linux computer. The following components were evaluated for the wireless protocol:

ZigBeeo Low-powero Less data throughputo Would require connecting the module to other components

Bluetooth (BLE)o Low-powero More data throughputo Is a feature of an existing circuit board

Wi-Fio Not a low-power solution

The chosen wireless protocol solution is Bluetooth (BLE) because of its ease of use and its availability on the TI SensorTag. Bluetooth (BLE) also provides more data throughput in case there is a need to sample a high volume of data to calculate the shot strength.

VII. Image ProcessingThe image processing component is responsible for processing the table-top play area image and locating relevant features for game calculations. The following components were evaluated for image processing:

OpenCVo Cross-platformo Built-in image processing functionalityo Can run on embedded devices

MATLABo Built-in image processing functionalityo Requires a PC

The chosen image processing solution is OpenCV because it contains extensive image processing libraries that will simplify the task of locating features in the images captured from the camera. OpenCV was also chosen because it is the only viable option for the embedded Linux computer.


VIII. Visual OutputThe visual output component is responsible for drawing images which need to be displayed by the display component. The following components were evaluated for visual output:

OpenCVo Cross-platformo Able to draw simple shapes, lines, and text

OpenGLo Cross-platformo Able to draw complex shapes and imageso Complex to use

SDL (Simple DirectMedia Layer)o Bitmap-based

The chosen visual output solution is OpenCV because it allows re-use of already available packages from the image processing component.

IX. Mounting SolutionThe mounting solution is necessary in that the system needs to obtain and display information from above without being intrusive to the play area. The following components were evaluated for the mounting solution:

Ceilingo Non-interfering with gameplay and accommodates needs for the camera and

projector locations Tripod

o Large, clunky, and likely to interfere with gameplay Swing-arm

o Requires a stable pole or mounting platform on wall close to the play area Tableside

o Likely to interfere with gameplay

The chosen mounting solution is a ceiling-mounted system because it will not interfere with gameplay and provides a mounting location for both the camera and projector such that the game information can be easily obtained and information may be displayed directly on the table’s surface.

4. Design

I. OverviewThe Pocket Billiards Trainer requires a collaborative design in which many subsystems work together. The basis of this design incorporates the retrieval of an image of the play


area of a billiards table, processing of the image in order to locate key features in the play area, simulation of the table’s current state in order to determine an ideal shot, and the relay of this information in a meaningful way to a player through use of an overhead projection system. A system level diagram is shown in Figure 1 to visualize the interaction of the primary components of the system.

Figure 1: System Overview

II. CameraThe camera selected for this project uses a 2 million pixel (2MP) image sensor connected to a microcontroller which implements a basic serial communication protocol. The camera can be instructed to reset, capture an image, and initiate an image transfer via the serial interface. The serial data lines are sensitive to a 3.3 V TTL signal, which interfaces with the Embedded Linux computer’s UART interface. For longer transmission distances, a low voltage TTL to RS-232 converter IC will be used to guarantee transmission and receipt of information. A block diagram of the serial conversion can be seen in Figure 2 and a schematic of the MAX232 IC in a basic configuration can be seen in Figure 3.

Figure 2: Serial Conversion Diagram


Figure 3: MAX232 IC Schematic

The camera can be powered using 3.3 V to 5.0 V. In order to maintain a simple means of connecting system pieces together the data and power lines for the camera will be run together in a single cable. A simple PCB will be designed to provide power to the camera from the BeagleBone Black’s onboard power rails.

The software design for the camera is a controller that will accept a request from another process to take a picture of the table’s state. The controller will send the required commands to initiate the transfer from the camera and write the image to a file. The camera encodes the picture in JPEG format so no conversion needs to be performed in order to inspect the camera’s output from a debug level. The serial device on the BeagleBone Black is controlled using Linux device handles. The device is initialized and configured to operate at 115200 baud with 8 bits of data and no parity. The command sequence to initiate a read and receiving data form the camera is seen below in Figure 4. The image received over the serial interface is saved to the file system and then loaded by the Image processing system to detect the game state.


Figure 4: Command Sequence


III. Image Processing

a. OverviewThe pocket billiards trainer project requires an image processing component capable of analyzing the play area of the pocket billiards table. This component shall identify the play area of the pocket billiards table, which includes the pockets and rails of the table. This component shall also identify the location of all balls on the table and the color of each ball. The game information shall be obtained by an overhead camera and analyzed on an embedded Linux computer using computer vision and image processing techniques through the use of OpenCV.

b. Table Feature DetectionDetection of the table’s features (rails, pockets) is necessary in order to define the boundaries of the play area for accurate game calculations. This information is used by the simulation component to locate goal regions (pockets) and barriers (rails) for calculation of ball physics geometry on the play surface.

c. Ball DetectionDetection of the size and location of all balls on the table is necessary in order for the simulation component to determine accurately which shots can be made and to calculate meaningful shot predictions for the player.

d. Ball Color IdentificationDetection of each ball’s color is necessary in order for the simulation component to understand how to perform under different modes of play and to calculate shots for both striped and solid balls.

e. Process FlowThe image processing component shall identify the required features from the image captured from the camera by following the process flow diagram shown in Figure 5. This flow chart shows the software components necessary to take an image from the camera and output positional and feature information to the simulation component.


Figure 5: Image Processing Component Process Flow

The processing sequence consists of the following steps.

The image is read from the camera for processing. Canny edge detection is performed to obtain an edge image. Hough transforms are applied to the edge image to obtain a Hough line

image and a Hough circle image. Rectangles are located in the Hough line image. Pocket circles are located in the Hough circle image. Ball circles are located in the Hough circle image. Rails are identified in the Hough line image. Pockets are identified in the Hough circle image. Balls are identified using comparison of histograms. Location data for all objects is sent for game calculations.


1. Edge DetectionCanny edge detection is used in obtaining an edge image from the input image from the camera. This algorithm operates using the following four-step process [4]:

1. Blur the image with Gaussian smoothing to remove noise from the image.2. Apply a gradient operator to obtain the gradients’ intensity and direction.

3. Determine if a pixel is a candidate for an edge by use of non-maximum suppression.

4. Find the beginning and end of edges using hysteresis thresholding.

An example Canny edge image of a pool table is shown below in Figure 6. The Canny edge image will be acquired by use of the Canny() function in OpenCV.

Figure 6: Canny Edge Image

2. Hough TransformThe Hough transform is used in locating straight lines and circles in the edge image.

i. Hough LinesLocating lines in the image is performed by using the Hough transform to solve for all lines in a polar coordinate system where

ρ (θ )=x0 cosθ+ y0 sinθ . Candidate lines are voted for by assessing the

number of matching lines through a set of points. An example image of the Hough transform used to detect lines in a pool table image is shown below in Figure 7.

The rails of the table will be acquired by use of the HoughLines() function in OpenCV.


Figure 7: Hough Line Image

ii. Hough CirclesLocating circles in the image is performed by using the Hough transform to solve for all circles such that x=a+Rcosθ andy=b+Rsinθ, solving for a, b and R. Similar to the line detection, candidate circles of a specified radius are voted for by assessing the number of matching circle edge pixels a distance R away from the center pixel. An example image of the Hough transform used to detect pool balls in an image is shown below in Figure 8 [1]. The location and radii of the balls will be acquired by use of the HoughCircles() function in OpenCV.

Figure 8: Hough Circle Image [1]

3. Histogram ComparisonA comparison of histograms is used to identify the color of each ball in the image. This method will use a table of reference histograms to define the ideal distribution for each ball based on the calibration phase. Once a ball has been


located from the Hough transform, the color data for that ball’s region will be analyzed and compared to the histogram of each ball in the table. The comparison which yields the highest correlation will be the candidate for that ball’s identity. An example of the histogram data from balls in Hue Saturation Brightness (HSB) space is shown in Figure 9 [1] below. The histogram comparison will be performed using the histCompare() function in OpenCV.

Figure 9: HSB Histogram Data for #2 and #4 Balls [1]

4. Data TransferOnce all data have been acquired from the image processing component, such as the bounds of the play area, ball coordinates, and identity of each ball, this information will be passed to the simulation component in the form of a data structure.

IV. External SensorThe external sensor will acquire raw data from players’ shots. The external sensor will then transmit the data to the embedded Linux computer to be processed to quantify the force used to hit the cue ball. A Texas Instruments (TI) Bluetooth LE SensorTag was chosen to accomplish the data acquisition. The SensorTag can easily be synced with a USB dongle connected to the embedded Linux computer. The SensorTag has a 3-axis accelerometer that can be configured to different data rates. This one small package accomplishes the proposed data acquisition. The SensorTag is small enough that it will be mounted on a cue stick or a wrist strap that will be worn by the player to collect the necessary data. The system interaction with the external sensor can be seen in Figure 10.


Figure 10: External Sensor Interaction

V. Simulation/Feedback System

The simulation and feedback system is the software backbone of the Pocket Billiards Trainer. It is responsible for taking as input both data from the external sensors and the results of the image processing component, such that it can first provide feedback on the shot most recently taken, followed by performing the analysis necessary in preparation for the upcoming shot. The flow for general use of the Pocket Billiards Trainer system will be to attempt a shot suggested by the system, receive feedback from the system based on the results of that shot, and then repeat by taking the next shot based on the follow-up shot suggestion. All of the data necessary for this analysis will be supplied by the image processing system and the external sensor system. The image processing system will provide information about the current state of the play area, including the relative bounds of the play area and the position and identity of each ball within the play area. The external sensor will provide force data about the most recent shot taken. The simulation system itself will remember what the selected recommended shot was, to make sense of the force data and current state of the play area.

a. Feedback MechanismFeedback generated about the shot performed will be based on the results of the analysis that most closely matches the current state of the play area. What constitutes a match will be determined by the method of analysis. For each failed match for a method of analysis, a less optimistic method will be used, until finally the feedback scheme is unable to provide relevant feedback due to lack of determinable similarity between expected and actual results.

The first analysis approach will determine if the current positions of the balls on the table are all within a reasonable tolerance of the expected positions of the balls given perfect recreation of the simulation. This is the ideal result, and will yield the most favorable feedback, but is also unlikely to occur frequently.

The second approach will attempt to determine if the position of the cue ball is the only major discrepancy from the expected result. When this is the case the system will assume that the object ball was pocketed using the strategy proposed by the simulation, implying that the method by which the discrepancy was achieved was primarily through using more or less force than suggested. That way, the cue ball


should be “found” along the simple travel path of the original simulation, within some very small tolerance. Feedback for a match on this method of analysis would amount to alerting the player of failure to accurately judge the amount of force necessary for the shot.

If the object ball was not pocketed by the shot, then the discrepancy from the expected result is more severe than simply using the wrong amount of force. The third approach will determine if the object ball is still on the table, and further attempt to determine how far off of the desired angle the cue ball struck the object ball. This will be achieved by attempting to find the object ball along one of several potential simple travel paths (within a small tolerance), each with a small angle difference from the ideal strike angle. If the object ball is found in this way, the system can provide feedback in the form of the likely difference between the ideal strike angle and the actual strike angle between the cue ball and the object ball. Note that this method does not take into account the positions of the other balls, and in particular the cue ball, as the object ball was not pocketed.

The fourth approach will determine whether the object ball has even moved from its initial position, and will perform a similar operation as with the third approach to attempt to determine the difference between the ideal and actual angle at which the cue ball was struck. Feedback will be in the form of general advice, such as reminding the player to take several practice strokes to line up the shot, and to ensure a reliable bridge.

The fifth “approach” is the fallback, wherein the play area is not in a recognizable state, and the feedback system is unable to provide tailored feedback. This will most probably be the result of two situations: either because of a complete failure to follow the guidance provided by the system in mechanically performing the shot, or because in all of the approaches detailed above, if any of the tracked balls make contact with any other balls on the table, they will no longer be able to be “followed.” This “match” will lead to providing general advice as feedback, as with the fourth approach. Since it is likely that amateur players will completely fail to produce the expected result of the simulation, this will probably be the most common match, necessitating many and varied helpful suggestions available for display.

b. Simulation SystemThe simulation system performs two very related but distinct functions. The first is to apply simple heuristics using basic geometry to determine what shots are available, and to use some simple ranking scheme to determine which shots are easier than others. The second is to use sophisticated equations that exactly define the physical relationships between interacting bodies, with the primary goal to determine the final resting place of all of the billiard balls after the shot has been made, especially the cue ball.


To determine the available shots, basic geometry can be used. For each object ball, and for each pocket, “draw” a line from the pocket to the object ball, and then draw a line from the ghost ball position against the object ball to the cue ball. As long as there are no obstructions along these paths, the path is an available shot. For any object ball that does not have a direct path from cue ball to object ball to pocket, a single cushion bank will be allowed for consideration as a potential shot. This will require attempting to determine a possible shot both from needing to bank the object ball and from needing to bank the cue ball, so finding available shots that require banks will be more involved.

The ranking scheme for the available shots will involve scoring each basic-geometry shot based on the cut of the shot angles, the length of the shots, the angle of entry into the pocket for the object ball, and whether or not a bank is necessary to perform the shot. The more severely each of these elements affects the shot, the worse the ranking is in terms of difficulty. Ultimately, each of the available shots will be sorted by perceived difficulty, such that when the player is presented with the opportunity to cycle through guidance for each available shot, they will be cycled through by increasing difficulty.

For each in the final list of available shots, the shot itself will be accurately simulated, so as to determine the final resting place of the cue ball. Determining the final resting place of the cue ball is necessary to attempt to categorize the available shots even further, this time on whether, by making the shot, follow-up shots are available to the player. This will be done by performing a series of heuristics again, as in the initial step, albeit to a lesser degree.

The actual simulation process will be backed by well-understood equations that describe the interactions between the different bodies at play in a game of billiards. The key component of the design of the simulation system implementation, and moreover how the system is utilized, is to what degree the flexibility of the simulation is used to control the cue ball positioning. The more the potential of the system is capitalized upon, the more time the simulations will take to perform, and the more demanding the potential shots may come to be.

For each available shot, different amounts of force, top-spin, back-spin and english can be applied in an attempt to better position the cue ball. In addition, if the simulation detects that a scratch occurs, different shot mechanics can be used to avoid that. That is, for each available shot, run the shot through a series of simulation possibilities, applying different shot mechanics for each, in an attempt to find an ideal placement for the cue ball. Only a few different sets of shot mechanics will be attempted for each available shot, to reduce computation time, and only more than one if the dynamics of the shot suggest that alternate shot mechanics could have a significant impact on the placement of the cue ball.


The design of the simulation system will allow for supreme customization in ultimately how much work is done at each stage, and in total between actual shots. It is unknown how much real time the simulations will take, but the primary contributing factor to computation time is the number of available shots simulated, and the number of alternate shot mechanics attempted with simulations for each available shot. The algorithm implementation for the simulation system will closely follow a functional style, to maximize control over the amount of work performed.

Additionally, the simulation system will provide a “developer mode”, which allows for essentially direct access to the internals of the simulation system, to greatly assist in testing of the system against real-world results.

VI. Table Display SystemA pool table’s size is generally 9 x 4.5 feet with a playable area of roughly 8.3 x 4.16 feet. The projection system must be able to cover the play area fully in order to properly display feedback and possible shots to the player. A projector with a projection size of approximately 10 feet measured diagonally will cover the table sufficiently. The pool table has a width:depth ratio of 2:1. In order for the projection to properly fit the table, it is important that the projector has a similar output aspect ratio compared to the table. Projectors come in a 16:9 aspect ratio, which can be represented by 1.78:1, and will be sufficient to cover at least the playable area of the table. The estimated coverage can be seen in Figure 11, indicated by the rectangle with red outline.

Figure 11: External Sensor Interaction

In order to limit the height requirements of the system, a short throw projector will be used. The throw ratio of a projector is calculated using the projector’s distance from the screen, in this case the table’s surface, and the preferred width of the projected image. The projection diagram in Figure 12 helps to illustrate this concept.

Pool TableTable Surface


Figure 12: Projection Diagram [5]

In short, throw ratio is Throw Ratio=D /W . Based on our previous need for a 10 foot screen and an estimated max height of 5 feet, our throw ratio would need to be 0.5.

The projector needs to be able to project a clear picture in a commonly lit room. Any projector with a minimum brightness of 2000 lumens should be able to be seen in most normal lighting situations. This conclusion is based upon the testing of a projector with a similar luminosity and then projecting onto differently colored materials at the expected projection distance.

The Sanyo PRM-20A projector provides a 1280x800 native resolution display at 2000 lumens along with a throw ratio of 0.5. The device uses a VGA input to receive a video signal, which can be obtained by converting the BeagleBone Black’s MicroHDMI to VGA through a set of converters.

OpenCV contains a basic graphics library that will be used as the visual engine for the Table Display System. The library is capable of drawing shapes, lines, and text to the Linux frame buffer with minimal effort. This will serve as the main software component in a larger custom design for the visual feedback system.


VII. Engineering StandardsThis system will include a few protocols and methods that are standardized for use in these types of applications. They can be seen in the following table.

Standard Used by system PurposeRS-232 Camera System Serves as the electrical

standard of communication between the camera and the BeagleBone Black.

USB 2.0 External Sensors Used by the Bluetooth dongle to connect to the BeagleBone Black.

Bluetooth Low Energy External Sensors Used as the wireless communication protocol between the TI SensorTag and the BeagleBone Black.

C++11 Camera System, External Sensors, Image Processing, Simulation/Feedback System, Table Display System.

Serves as the language that will be used to program all of the subsystems

High-Definition Multimedia Interface (HDMI)

Table Display System Protocol used for video output from the BeagleBone Black.

Video Graphics Array (VGA)

Table Display System Video format used by the projector as input.

VIII. Multidisciplinary AspectsThis project brings together multiple disciplines of engineering including computer, electrical, software, and mechanical engineering. Aside from the multidisciplinary engineering aspect, this project relies heavily on physics, mathematics, and computer science for its successful operation.

IX. BackgroundSome specific courses that prove useful in this project include:

Computer Vision Interface & Digital Electronics

In addition to the courses listed, the respective experiences on coop for each member will prove useful towards the successful design of the system and all of its components.


5. Constraints and Considerations

I. Extensibility

The software design of the system leaves room for the inclusion of new features should they ever be requested. The modular design of all software components allows for the easy adjustment or full replacement to add functionality. In addition, most of the software libraries are open source which will keep the costs of development down while allowing for the software to be constantly improved.

The image processing library OpenCV is a very popular open source computer vision library which has wide accreditation in the Maker community. This software is great to use since it is widely supported and under constant development.

The software component for communicating with the Camera will be a custom implementation of a proprietary serial based communication protocol designed by Linksprite. While the implementation of the software and how it utilizes the proprietary protocol is custom, it can be released under an open source license such as GPL and LGPL since it is homegrown software.

The physics back-end of the simulation system will be a custom implementation, allowing for a refined set of functionality from larger more complex physics simulation libraries.

The TI SensorTag uses proprietary Texas Instruments code which would require licensing for commercial use, and is ultimately closed source. This would limit the Bluetooth development in a commercial environment and as such it would be better to find an open source version for both hardware and software.

II. Manufacturability

Considerations for manufacturing are limited for this design since most of the selected components are prebuilt. Minor design considerations for items such as the custom implementation of the serial and power lines to the camera may lead to the manufacturing of a custom printed circuit board for the project. Plastic housings for the unhoused components may need some design, and subsequently require a prototype to be made to verify that it will protect the components from the surrounding environment. These custom designed enclosures could possibly be 3D printed using various online manufacturers. The estimated cost for one possible enclosure for a BeagleBone Black is $34.77.


III. Reliability

This system design is comprised of a number of components each with a large role in the successful operation of the system. Reliability of each component is crucial, especially those that are physical in nature, such as the camera, external sensors, projector and most importantly the Embedded Linux Computer. In order to maintain the operation of these devices it is important to house them in a protective case. The projector and external sensor module is manufactured in a plastic housing, however the BeagleBone Black and camera modules are exposed circuit boards. To increase the longevity of the product, a housing can be purchased for the BeagleBone Black and a housing can be designed for the Camera module such that it doesn’t interfere with its operation.

6. Cost EstimatesBelow is an estimate of all components necessary for the Pocket Billiards Trainer system, summarized in Table 1 with costs and quantities.

Table 1: Cost Estimates

Part Price Qty. Cost to us Lead timeLogitech c930e USB 2.0 webcam $ 129.99 1 $ - 7 DaysTI SensorTag $ 25.00 1 $ 25.00 3 DaysShort Throw Projector $ 124.50 1 $ 124.50 7 DaysMicro HDMI (Type-D) to Active VGA (Female) Adapter

$ 14.99 1 $ 14.99 3 Days

BeagleBone Black $ 55.00 1 $ - 1 DayPowered USB 2.0 hub $ 15.99 1 $ 15.99 2 DaysUSB Bluetooth 4.0 Dongle $ 12.99 1 $ 12.99 4 DaysTotal $ 378.46 $ 193.47

7. Testing Strategy

I. Overview

The testing strategy is split into a long-form textual explanation, and a listing of matrix tests to cover unit testing and integration testing. Specific unit and integration tests are listed in Section III of the Appendices.


II. Camera

The testing strategy for the camera hardware system requires implementing the communication software that will be used to poll and request an image from the camera. Once this software is implemented, the first plan is to continually request images from the camera to benchmark how quickly we can capture images from the device. This information will be useful in the event we need to run the camera at a higher speed than simply capturing before and after shots of the table’s surface. It also acts as a stress test for the communication software and physical data lines. A byproduct of this test is a collection of images to inspect for quality. The camera’s image quality will be gauged based on the clarity and fidelity of the images, as well as the absence of any form of distortion or image warping from the lens. Meeting these requirements is important for when the camera is used to capture test images for the Image processing software.

Testing the camera’s communication interface is done via the successful download of an image from the camera over its serial interface. Testing of the physical interface and its associated hardware is important to guarantee communication with the device while the system is running. Testing the serial communications at the TTL level, receiving and transmitting, and when it is converted to RS-232 is important to verify the data is correct at both stages. Investigation into the maximum distance 3.3 V serial communication will work properly will be performed and compared against RS-232 serial. It may be possible to reduce the need for custom circuity and PCBs to convert the TTL levels up to RS-232 levels. This testing will be continuous as new factors are introduced, such as running power and data in the same cable.

III. Image ProcessingThe overall testing strategy for the image processing component is similar for each of its individual risks. The goal is to develop a base algorithm using known computer vision and image processing techniques, apply the algorithm to a large set of images for analysis under various conditions, and then integrate the working code with the remaining components for use during live gameplay.

a. Table Feature Detection Develop feature detection algorithm(s) in Matlab. Test algorithm(s) on a large set of test images. Port working Matlab code to C/C++ using OpenCV. Test C/C++ port on test images and compare results. Integrate with related components. Test live on a table during gameplay.

b. Ball Detection Develop ball detection algorithm(s) in Matlab. Test algorithm(s) on a large set of test images.


Port working Matlab code to C/C++ using OpenCV. Test C/C++ port on test images and compare results. Integrate with related components. Test live on a table during gameplay.

c. Ball Color Identification Develop color identification algorithm(s) in Matlab. Test algorithm(s) on a large set of test images. Port working Matlab code to C/C++ using OpenCV. Test C/C++ port on test images and compare results. Integrate with related components. Test live on a table during gameplay.

IV. External SensorThe testing strategy for the external sensor can be broken into three different components: shot detection, information relay, and power consumption. Shot detection is the most important component as this will drive development for the entire external sensor component. The following components will be tested based on the success of the shot detection.

a. Shot Detection Set up a pre-owned development board with a 3-axis accelerometer to begin

looking at data generated from movement. Discover profile of shot movement repeatability.

b. Information Relay Connect Bluetooth SensorTag to a PC to verify communication between dongle

and SensorTag. Test configuring software for acceptable accelerometer data still using PC setup

to verify results. Verify that information is still being received on the embedded Linux computer

through the Bluetooth LE dongle.

c. Power Consumption Measure power consumption of the SensorTag to determine battery life.

V. Simulation/Feedback SystemTesting of the simulation system will require comparing real-world results with the simulation of the same event. To simplify testing of the simulation software, while simultaneously boosting confidence in the results, verification will require a robust secondary mode of operation.

The normal mode of operation for the complete system is to analyze the table, determine viable shots, and simulate them accurately to determine the mechanics of the shot for guidance projection and ball placement prediction. In this mode of operation,


besides shot feedback, it is irrelevant to the system what the actual result of the shot is in terms of shot simulation. If the simulation system were to be tested under this mode of operation, it would be very difficult to verify results, as the real-world testing procedure would require a human to reproduce the expected shot mechanics of the simulation very accurately, and then verify whether the real-world result matched the simulation. This is borderline impossible to repeat reliably for any discrete test, and borderline impossible to faithfully recreate the preconditions of the shot in the first place (such as shot strength and the exact angle of the shot).

With this in mind, the simulation software will need to provide a secondary mode of operation, with significantly more interactivity to allow for manually setting certain variables that are normally controlled by the simulator. If within this testing mode the simulation software is capable of receiving explicit values for variables such as the force of a shot and the angle of the cue stick, then the testing process becomes a fairly streamlined procedure.

The cue ball used for testing will be marked, such that when it is oriented on the play area, the intended center point for the shot will be surrounded by markings that indicate the amount of sidespin and top- or back-spin placed on the cue ball with the shot.

The testing procedure will then be as follows, implying the functionality of the testing mode of the simulation software:

a. Enter the testing mode of the simulation software on a laptop, to simplify interfacing with the software.

b. Orient the cue ball and one or more potential object balls in the play area, on top of pre-placed markers to accurately repeat the same ball positions.

c. Instruct the simulation software to capture an image of the play area.d. Place guiding tape along the axis of the shot to be taken. The software system

will consider some chosen side of the table to be zero degrees of rotation, such that the tape is some angle of rotation from the chosen origin. The placement of the guiding tape is arbitrary, as the testing procedure is simply to determine if the simulation is able to recreate the shot taken, given input parameters that match the shot. The guiding tape will point directly at the center point of the marked-up cue ball.

e. Have a human take the shot, either with the intention to pocket the object ball, or else to simply bank the cue ball or the object ball off of the side cushions. An external sensor will be used to record some quantifiable value related to the force of the shot, to have a reliable force value for input into the simulation software. The simulation software will have been previously calibrated to communicate to the user in terms of the relative values produced by the external sensor.

f. The angle of the shot about the cue ball, the drift from the center point of the cue ball in terms of the measured markings, and the force of the shot are sufficient to


relay the components of the shot to the simulation software. Execute the simulation with these parameters, and determine if the simulated result closely recreates the real-world result of the shot. Given that the small variability in the recorded values for the components of the shot can have a noticeable impact on the result of the simulation, if it appears that the simulation is otherwise correct, it may be necessary to tweak the input values to the simulation to try to produce a closer simulation. That is, while this method of testing is significantly more repeatable and reliable than the reverse strategy, it is still necessary to allow for a small margin of error in the input to the simulation.

g. Repeat steps b-g for different angles, forces, and drifts from the cue ball center point, and for different ball positions, to verify simulation accuracy.

VI. Table Display SystemThe table display system requires verification of the ability to draw useful shapes and text and to be able to display them on a billiards table surface with a projector. Testing of the table display system shall include but is not limited to the following tasks:

Test drawing simple shapes, lines, and text on a PC with OpenCV. Test displaying with a projector on a billiards table surface in various lighting

conditions. Re-test using BeagleBone Black as the computing platform

VII. ResultsAn overview of the system’s test results and some brief comments for each test can be found in the appended “TestPlan.xlsx” document.

8. Risks

I. OverviewThe Billiards Trainer system contains many high-risk components for it to operate as required. A summary of these risks is shown in the table below and each is addressed in the following sections.

Risk Description

CameraThe camera must be able to provide enough information to the system at a reasonable rate so that the system can operate as required.

Image Processing

The image processing must be able to detect the objects of interest and related characteristics on the pool table under different lighting conditions.

External SensorThe external sensor must be able to acquire the data used to quantify the shot strength.

Simulation The simulation must return accurate results based on many variables.Table Display

SystemThe table display system must be able to provide usable feedback to the players.


II. CameraThe camera is the keystone piece for this project as it is the major data input to the system. In order to gather information about the state of the table during play a sensor must be in place that can gather all the information about the table in a reasonable amount of time so that the player does not need to wait for the system between shots. The solution is a device that can take a snapshot of the game and then use that snapshot to calculate a set of feedback for a player. In this case, a camera can take a full snapshot of the table’s state and store it for future processing, doing the work of several sensors with one device. Should the device not operate properly, or be unable to provide enough information to the system, the system will not operate properly if at all.

Below is a brief summary of all risks relating to the operation of the camera.

Risk Cause Mitigation Plan

Data Loss Over Serial Lines

Low voltage TTL signals are susceptible to noise and get weak over a long travel distance

- Convert the TTL signals to RS-232 using a TTL to RS-232 conversion IC

Camera Field of View

Camera may not be able to see entire table from mounting position

- Test FOV by downloading image from camera

- Mount camera in a different location- Get a second camera and have 2 cameras to

cover the one half of the table each

Camera Detail

Camera may not have a high enough resolution or fidelity to accurately sense ball’s identities and positions

- Investigate if the camera’s quality can be augmented by either software filtering or by adjusting the conditions around the camera to enhance image quality

- Be prepared to use a different camera if quality requirements can’t be met with current hardware

In order to reduce the chance of data loss while receiving or transmitting to the camera, the TTL signals coming to and from both the camera module and the embedded Linux computer will have the voltage level brought up to RS-232 voltage spec. Using RS-232 voltage levels will allow the signals to travel a farther distance without interference from external noise. It also will allow for a longer transmission length. The part is small and simple to use, and needs only 5 V to operate properly. These kinds of low voltages can be delivered easily to more remote modules that are far from a base station. If power will be sent to the devices in the same cable the RS-232 standard will help prevent the serial communication from being contaminated by the power lines in the same cable.

In order to gauge the field of view of the camera, a picture will need to be downloaded from the camera. Using the provided communication explanation from the camera’s


datasheet software can be written to request an image from the camera module. By writing a serial communications program to send the required command sets to the camera and then reading the information as it arrives on the serial interface to a file, an image encoded as a JPEG will be available to open and inspect from the file system of the host computer.

Once the image is acquired from the camera, it can be examined to ensure that the entire table surface is in the picture. In addition, the image quality can be verified with the same picture. Assuming the picture taken had a set of balls set on the table it can be used to verify that the camera’s fidelity and level of detail is high enough for this application.

Should the camera’s quality be acceptable, but the field of view is not covering enough of the table, it may be necessary to purchase a secondary camera module and place each camera centered above one half of the table. This would guarantee coverage but would require image stitching in software to create a full image of the table.

III. Image Processing

a. OverviewThe image processing component of this project has many risks associated with it. The functionality of this component holds significant importance in that it provides vital information to many of the other components of the system. In particular, the game simulation component relies heavily on the information obtained from the image processing component. These risks are outlined below with their associated engineering and marketing requirements.

Risk Marketing Requirements Engineering RequirementsThe image processing component shall detect the table’s pockets and rails.

1, 2, 5, 7, 8, 13 A, G

The image processing component shall detect each ball on the table.

1, 2, 6, 7, 13 A, B, G

The image processing component shall identify each ball’s color.

1, 2, 6, 7, 13 A, D, G

b. Table Feature DetectionDetection of the table’s features (rails, pockets) is necessary in order to define the boundaries of the play area for accurate game calculations. This information is used by the game calculation component to locate goal regions (pockets) and barriers (rails) for calculation of ball physics on the play surface.

i. Concepts Considered/Chosen


1. Hough Transformii. Rationale

The Hough transform was chosen for identifying the table’s features. This method was chosen because the Hough transform is able to locate all straight lines and circles in an image. This process may need to be separated into two separate transforms, one for straight lines and another for circles. This will allow for simple detection of the table’s borders and pockets. No other techniques were considered because the Hough transform is extremely robust, easy to compute, and has an implementation in OpenCV.Examples of the Hough transform being used to detect lines and circles in images can be seen in Figures A1 [2] and A2 [3] of Appendix A.II, respectively.

c. Ball DetectionDetection of the size and location of all balls on the table is necessary in order for the game calculation component to calculate meaningful shot predictions for the player.

i. Concepts Considered/Chosen1. Hough Transform

ii. RationaleThe Hough transform was chosen for locating the balls on the table. This method was chosen because the Hough transform is able to locate all circular objects in an image. The Hough transform can also be limited to search a specified parameter space. This could be useful in minimizing computation time by limiting the parameter space of the Hough transform to circles that only fit the radius of pool balls from the calibration phase. No other techniques were considered because the Hough transform is extremely robust, easy to compute, and has an implementation in OpenCV.An example of the Hough transform being used to detect circles in an image can be seen in Figure A2 [3] of Appendix A.II.

d. Ball Color IdentificationDetection of each ball’s color is necessary in order for the game calculation component to track different modes of play and to calculate shots for both striped and solid balls.

i. Concepts Considered/Chosen1. Histogram Comparison2. Color Thresholding

ii. RationaleComparison of histograms was chosen because a histogram of the ball from the calibration phase can be used as a unique feature to describe each ball. This makes determining the ball’s identity a much simpler task by comparing the histogram of the current ball in question to a lookup table of expected histograms, where the minimum absolute error between histograms indicates


the closest match. There are many histogram comparison techniques which can be performed, and histogram comparison has multiple implementations in OpenCV.

The concept of color thresholding the image to find each ball would be a tedious approach and could require a large amount of processing time with no guarantee of finding the intended ball.

An example of a sample histogram from two pool balls can be seen in Figure A3 [1] of the Appendix.

e. Risk Mitigation DesignThe risk associated with the image processing component will attempt to be mitigated by following a simple process flow to solve each of the addressed problems. In the case that there is a failure, this should make it simple to determine where the component is failing. This design also separates this process into many discrete components which can be implemented as individual software components for unit testing. The process flow for this component is shown in Figure 5.

f. UncertaintiesThere is a minimal amount of uncertainty in the table feature detection area; however it is uncertain how the system will handle interference from outside objects entering the frame such as a human or pool cue.

The detection of balls on the table has some uncertainty when balls are clustered together or when lighting situations cause the ball to lose edge information and blend in with the background.

Identification of each ball based on its color holds the highest amount of uncertainty in this component due to the variation of ball orientation, the color variation due to lighting, and the color variation due to striped/solid balls.

IV. External Sensor

a. OverviewThe external sensor is responsible for quantifying the force that players used to hit the cue ball and relaying the information to the embedded Linux computer. The embedded Linux computer will process this information to give players feedback on their shot. The intention of this feedback is to give players a better sense of the required force to place the cue ball in a desirable location for their next shot.

b. Shot DetectionThe data from the accelerometer will have to be differentiated from the actual strike of the cue ball and all other movements of the cue stick. Ideally the strike of the cue ball will generate a distinct pattern that can be recognized and processed to determine the force used to strike the cue ball. Other possible considerations would


be detection of the cue stick in the camera frame and/or requiring the user to press a button on the SensorTag to indicate that they are taking a shot.

c. Relay Information / Power ConsumptionThe data from the sensor needs to be relayed to the embedded Linux computer. Since the sensor is not tethered to the embedded Linux computer it will need its own power source. The communication module must use minimal power so that the battery cell will last for the duration of at least one game. While staying under power constraints, the module must also provide enough bandwidth to detect the shot profile. Bluetooth LE was chosen over ZigBee and WiFi for its low power consumption, faster data rates, and ease of use.

d. Risk Mitigation DesignThe design chosen to alleviate many of the risks was to use a Texas Instruments (TI) Bluetooth LE development kit. The development kit comes with a pre-assembled board with two buttons and a 3-axis accelerometer. This choice eliminates the challenge of assembling a circuit board with all the necessary components. The development kit provides an interface option in the case that it is required for shot detection. The kit also comes with example code that should provide initial communication capabilities.

e. UncertaintiesThere are some uncertainties with this component. One is that a 3-axis accelerometer will not be able to gather enough data on its own to be able to obtain the needed information for calculating the shot strength. Another uncertainty is the ability to detect a cue strike profile from the accelerometer data that can easily be differentiated from other movements of the cue stick.

V. Simulation System

a. OverviewThe simulation system is responsible for calculating the interactions between the physical bodies of the game so that shot guidance can be provided to the player. The interactions of the physical bodies at play in the game can become complex, as velocity, momentum, angular velocity, angular momentum, friction, acceleration, and more all come into play, and each play a different role, varying in impact, depending on the bodies involved in the current interaction under scrutiny. This system, as well as being the foundation on top of which core project functionality is implemented, is a high-risk component that will require delicate implementation to ensure proper function. The difference between a very accurate simulation of potential real-world actions and even a slightly less accurate simulation will result in unusable or inapplicable results.

Risk Marketing Requirements Engineering Requirements


The simulation system shall be capable of accurate simulations of real-world behaviors.

7, 8, 9, 10, 13 H

b. Concepts ConsideredThere are many existing pocket billiards shot simulation programs, mostly in the form of computer games. For most of these, significant shortcuts were taken in implementing the physics simulator for shots, especially considering that from start to end, the user is playing a computer game, and would not have a frame of reference to tell the difference between the interactions on a physical table and what happens in the computer game. A few programs, such as “FooBilliard” and “Virtual Pool” both provide significantly more depth in the provided simulations. “Virtual Pool” in particular appears to boast an entirely complete physics simulation of the possible interactions in pocket billiards, going well beyond the functional expectations of the Pocket Billiards Trainer. The Pocket Billiards Trainer will need to attempt to strike a balance between the simplifying lack of depth of the pure computer game programs, and the extremely complex depth of simulation that “Virtual Pool” and the like provide. Ultimately, the Pocket Billiards Trainer will be attempting to simulate real-world interactions, followed immediately by putting those simulations to the test, so some of the popular shortcuts made by “toy” simulators cannot be taken. However, the full breadth of power exhibited by “Virtual Pool” and others is entirely unnecessary for the Pocket Billiards Trainer application.

c. Concepts ChosenThere are really only two possible plans of attack in simulating possible shots. Either an existing simulation system is utilized, or a simulator is implemented from scratch. There are many existing physics libraries available for either 2D or 3D physics. It seems that with the peculiarities of pocket billiards physics, such as that a billiard ball can have complex spin, and that a cue can strike the cue ball at angles not aligned to a 2D grid, that a 2D physics engine is not powerful enough to accommodate the requirements of the Pocket Billiards Trainer. At the same time, 3D physics engines would be altogether too powerful and complex, as the system is expected to run on a relatively cheap, single-board computer. With performance being an immediate concern, and with the requirement for only a narrow range of interactions that need to be simulated, it seems the best course of action is to implement the whole of the shot analysis system in-house. Additionally, fine control over the simulation system will be necessary when actually trying to match and calibrate the simulation results against real-world results.

d. Risk Mitigation DesignThe shot analysis component of the Pocket Billiards Trainer has a highly variable potential for reasonable success, and is perhaps the most likely to become a liability,


depending entirely on the degree to which the simulator is faithful to real-world interactions, and the complexity of interactions it is expected to be able to handle. That said, the absolute core functionality of the Pocket Billiards Trainer is dependent on a functional shot analysis system, so it is fitting that it is potentially the most volatile component. Whatever goal-complexity is decided for the shot-analysis system, there is significant literature providing insight into the physics behind those interactions. It becomes a matter of what is expected of the system, what is actually necessary, and whether clever simulation omissions can greatly reduce the complexity of the system and the difficulty of the implementation without actually causing the end-product to suffer from the viewpoint of the target audience. The results of the collisions between billiard balls, the cue and the cue ball, and a ball and the cushion are all obviously necessary to the simulation. Topspin and backspin, the first interactions to come to mind beyond simple geometry, are both knowingly utilized even in beginner play, and are therefore completely necessary for the simulation. Friction between the ball and the table cloth is necessary for determining the final resting positions of the balls. Limiting the simulation simply to these elements already provides an impressively comprehensive simulation, as much of the potential for further simulation is limited to more expert play. What is considered expert level techniques and what is considered amateur level techniques that are simply more difficult for amateurs (proper use of english being the most immediate example), is a subjective conversation. Much of the complexity of the final shot analysis system rests on what is determined to be both useful and simple enough to include in the simulation, without omitting anything critically important, and without including anything categorically unnecessary for the application of the Pocket Billiards Trainer.

e. Possible Simulation OmissionsThere are several kinds of interactions in pocket billiards that can potentially be omitted from the simulation. English, or sidespin, is a companion to both top- and backspin in pocket billiards play. Whereas topspin and backspin directly translate to fairly intuitive movements in the cue ball, english is an advanced technique that is situationally useful and requires skill to use properly. English on the cue ball has two immediate consequences: it affects the angle at which the cue ball rebounds from the cushion, and it affects the “throw” of the target ball. The former is more useful, whereas the latter is often an unintended consequence of english. A complicating factor in simulating english is the “swerve effect,” wherein a tiny amount of unintentional curvature in the initial path of the cue ball is introduced by the combination of both application of english and the slightly elevated angle of the cue due to the table itself. This unintentional curvature of the initial cue ball path is in contrast to the massé shot, where the curvature of the initial path is intended. Swerve is effectively imperceptible, especially in short-distance shots, and can potentially be omitted even if english were to be included in the simulation. Another complicating factor in simulating english is “squirt,” or deflection. Due to the angle of impact on the cue ball when english is applied, the cue ball will travel in


a direction slightly at an angle to the aimed direction, opposite the side where english was applied. As with swerve, squirt is an unintended complication of applying english, and is another effect that may need to be properly simulated for the overall simulation of english to be correct. Throw is when the english of the cue ball is able to induce a spin on the target ball, causing the target ball to be deflected from the intended impact line by some throw angle. That is, throw is essentially the target ball equivalent of squirt. The equations that govern the behavior of throw are particularly complex, and it seems likely that investigation into real-world examples would reveal that its actual impact is minimal. English is significantly more readily applied and observed in amateur play than any of the following possible simulation omissions, and so a proper simulator for amateur play likely requires a proper simulation of english, but it does add a significant extra layer of complexity to the simulation that could be removed for the sake of simplicity. A possible compromise would be to omit simulating any combination of swerve, squirt, and throw if further analysis deems their impact to be negligible, in an effort to reduce the impact the simulation of english has on the complete implementation.

A jump shot is a shot where the cue ball is struck such that it leaves the surface of the table, with the intent that it ``jump over'' some non-target ball to hit a target ball on the other side, perhaps after first rolling some distance, or perhaps by striking the target ball from the air. While spectacular, jump shots require extraordinary skill to perform reliably to accomplish some goal. The intended audience of the Pocket Billiards Trainer is beginner-to-intermediate pocket billiards players, and with the jump shot as advanced as it is, it would be outside of their skill level to suggest that a jump shot would be the best way for a particular target ball to be pocketed. Additionally, the implementation of the simulation would be greatly complicated by the introduction of the necessary scaffolding to handle simulating the jump shot. Without the jump shot, Z-axis forces can be almost completely ignored, for the most part considered only with cue-to-cue-ball contact. With the jump shot, projectile simulations would need to be added, and the effects of a possible impact between a target ball and an airborne cue ball would need to be considered. Being as this advanced shot would introduce a great deal of complexity to the simulation, is useful only under rare circumstances, and has effectively zero utility to the target audience of the Pocket Billiards Trainer, the jump shot and necessary simulation scaffolding will be absent from the final simulation.

The massé shot is a technique through which a significant force is introduced from

above (typically the cue stick is angled 60° or more above the horizontal), rather than from the horizontal, causing the cue ball to initially follow a noticeably curved path. Normally, the cue ball can be made to follow only a curved path after it strikes

some obstacle, such as another ball or the cushion. The massé shot is a very advanced technique that is used either to strike target balls that are blocked by some obstacle on the linear path between the cue ball and the target ball, or in some extreme situations to simply strike the target ball at a different angle than would


otherwise be possible by causing the cue ball to follow an arc. Due to the advanced

nature of the shot, and especially that proper control of a massé shot goes beyond

even the jump shot in difficulty, the massé shot will be omitted from the final simulation to simplify the implementation.

f. Implementation OverviewThere are many resources available, in varying degrees of detail, describing the physics of the interactions between bodies in pocket billiards. The computer vision component of the Pocket Billiards Project will provide accurate position information for all of the balls, pockets, and rails of the current table state. This system will attempt to address both of the two equally important goals when taking a shot in pocket billiards. First, a target ball should be pocketed. Second, the cue ball position should be controlled such that it comes to a rest in a favorable position for a follow-up shot by the same player. It is the first goal on which a beginning and intermediate player spends most of their effort, and the second goal that allows expert players to sink every ball on the table without ever leading the cue ball to a disadvantageous position.

The implementation of the shot analysis system will be a two-part endeavor. The first subcomponent will be a heuristic system that attempts to perform basic geometry to determine if any potential target balls can be reasonably sunk by a player at the skill levels targeted by the Pocket Billiards Trainer. This by itself will likely perform an adequate job at helping with the first of the two important goals in pocket billiards. The second subcomponent is the physics simulator system, which will be used almost entirely to determine simply where the cue ball be after the shot, and further whether any steps can be taken to influence that position. Any omissions from the simulation limit the number of ways that the final cue ball position can be influenced. English is an excellent example of something that, if omitted, may cause a very real drop in the number of possible shots. However, english is also a complex technique that may be reasonably outside the bounds of beginner and intermediate play, giving rise to the argument about whether its inclusion is warranted. The actual calculations for the simulation would be handled by the equations discussed at length in various sources. Some are quite complex with many input variables, and so fully understanding their impact on the system will take considerable effort in itself. The full simulation will essentially be a time-stepped piecewise function, where at each distinct event, such as collisions, the next major part of the simulation is used to guide the balls.

g. UncertaintiesThe shot analysis system can fairly be considered complete while omitting a great many possible extra simulation features, given the target audience of the Pocket Billiards Trainer being beginner-to-intermediate players. The available literature on proper simulation of pocket billiards interactions is dense, with no clear line differentiating between the theoretical and the practical. It is not immediately clear


whether, for some applications, approximations would suffice instead of extremely complex equations. For example, one resource provides an equation for the angle of throw after a collision between the cue ball and the target ball that uses no fewer than nine distinct variables. More to the point, assuming that reliable testing data can be obtained, until the shot analysis system is in the process of being implemented and experimental results can be obtained, it is unclear what is necessary and what is unnecessary in building a reasonable simulation of pocket billiards interactions. While it is unlikely that pure simulation calculation time will become an issue in terms of responsiveness of the system, if that does become a concern, it would be very useful to know what could be cut with minimal impact on the result of the simulation.

VI. Table Display SystemThe table display system holds risk in that it is a coordinated effort between a software component and the physical projector. The software component associated with this system will use OpenCV to draw visual cues which will be displayed for the user via the projector. The primary risk associated with this component is whether or not the libraries used will provide useful enough functionality to give the user valuable feedback through the display and whether or not the display will be visible in normal lighting conditions.

9. Milestone ChartPlease refer to the milestones listed in Section IV of the Appendices.

10. Perspective

I. CameraThe camera system had a major change in hardware from the proposed method of implementation. Originally a serial communication based camera was going to be used for its compactness and supposed ease-of-use, however due to poor documentation and generally unreliable hardware, the camera was swapped out for a different solution. The selection of the new camera was based on BeagleBone Black having a USB interface. After researching more about how USB webcams operate within Linux, it was discovered that Linux has a common interface for communicating with most webcams known as USB Video Class (UVC). This protocol meant a large number of cameras would work within Linux without proprietary drivers, including the camera that was ultimately selected for use: the Logitech C930e HD webcam. This camera met the same criteria as the previous camera while operating over USB, which through Linux proved to be a reliable communication method. Writing the software for the controlling and using the camera became simplified through OpenCV’s VideoCapture class, which already communicated via USB to the camera using the low level Linux drivers. The


level of abstraction between the camera and controller software became much higher, however for the purposes of this project all functionality needed was easily implemented at the level provided through the VideoCapture class. It was later discovered that the UVC drivers provided by the selected operating platform, which at the time was Ubuntu 14.04, were buggy and unreliable. Extended research suggested that it was a problem with Linux’s kernel, so a new operating system was selected. The new platform was Arch Linux, and while various portions of the platform needed to be reconfigured to match those used in Ubuntu, it proved to be much more reliable for multiple system components. With this change the camera system worked as originally planned.

II. Image ProcessingThe image processing components of the system were generally successful. The rails were able to be correctly located and identified using the Hough line transform and an algorithm which segmented the lines into each side of the table for sorting. The balls were able to be reliably located using the Hough circle transform. The balls were able to be identified by use of the HSV color histograms for each ball, which proved successful in many cases. Though the color identification was not perfect and perhaps the least reliable component of the image processing system, it still performed well given the time constraint.

With each of the primary image processing components implemented, the issue of the misalignment and skewing of the projector output needed to be resolved. As a fix to this issue, a calibration routine was developed to align the projector output with the table, as neither were perfectly level. The image taken from the camera had to be de-skewed once the edges and corners were located to make the table space rectangular. An alignment box was then projected onto the table, an image was taken of the projected alignment box, and those edges and corners were then located. With the relation between the edges and corners from each image, the final projected output could then be skewed to be correctly aligned with the table’s edges. The calibration and alignment process was a difficult problem to resolve, but was necessary in order to have an accurate display. The problems associated with this solution included a slight stretching of the output image, which resulted in slightly incorrect ball locations. Also, there was an issue of the projected image trying to go through the bottom rail of the table due to the location of the projector. A large down-side of having to perform a warping operation on the output before every image was displayed was that it was very memory intensive and took a non-trivial amount of time on the Beaglebone Black.

Overall, the image processing component of the system was largely successful. In being able to locate and identify the key features of the play area, integration of this information with the other system components was very simple and helped to make for a successful system overall.


III. SimulationThe backend shot analysis and simulation system was only implemented up to the shot analysis stage, and the simulation component was left largely unimplemented. This was not a limitation of feasibility, but rather that with the limited time remaining before the project demonstration, low-hanging fruit was the focus of the remaining available development time. In fact, nearly all of the necessary physics equations and relationships were broken down into core components and implemented into coded functions, and it was only the structure of the simulator itself that wasn't implemented. The combination of more important features demanding more priority, and the lack of progress on meaningful shot strength data which was a core component of the simulation software, left it to be abandoned in favor of other features.

IV. External SensorThe external sensor was completed with limited functionality. There was a lot of time spent trying to implement the Bluetooth LE communication in C++ instead of implementing the desired functionality of the external sensor. This difficulty existed because there were no good resources for this type of implementation. Most examples used an interactive tool provided by Bluez, which is what this project ended up using as well. The communication was already accomplished with a Python program early on. In hindsight the Python program should have been used to complete all the desired functionality before trying to implement the communication in C++.

V. Table Display SystemImplementing the display system was a two part task; the first part was understanding how to configure the BeagleBone Black so that the projector that was selected could be used to display content to the tabletop, and the second part was using that foundation to build a display painter using functions from OpenCV. To start, a Linux distribution needed to be selected, and ultimately the distribution chosen was Arch Linux. In early development a version of Ubuntu was selected since it matched the environment that had been created for development. However continued use of Ubuntu proved to be unstable and was causing flickering of the display when the software was doing any form of intensive processing, or even when trying to take a picture of the tabletop. Switching to Arch solved these problems, however the laborious steps of configuring the platform so it worked with all of the current development software had to be repeated. The implementation of the software library used to paint to the tabletop was implemented using OpenCV’s basic shape drawing functionality. Using a few clever methods of drawing text to the table, a menu system was created complete with sub-menus and intelligent placement based on the tabletop’s state. In general the Table Display System was implemented without any major issue. The only problem that arose was the needed reconfiguration of the operating platform, however that decision was not solely based on problems within the display system alone.


11. AcknowledgementsWe would like to thank:

Dr. Melton for his contribution of valuable feedback to our design and for lending us an additional BeagleBone Black.

Dr. Becker-Gomez for being kind enough to lend us a BeagleBone Black.

Rick Tolleson for ordering project parts and addressing laboratory needs.

Ray Sattora for his assistance with building the camera-projector apparatus.

Casey Jabbour for her assistance with the project poster.

A. Appendices

I. References[1] Baekdahl, J. & Have, S. (2011). Detection and Identification of Pool Balls using Computer Vision. Aalborg University, Denmark.

[2] http://docs.opencv.org/doc/tutorials/imgproc/imgtrans/hough_lines/hough_lines.html

[3] http://docs.opencv.org/doc/tutorials/imgproc/imgtrans/hough_circle/hough_circle.html

[4] http://en.wikipedia.org/wiki/Canny_edge_detector

[5] http://upload.wikimedia.org/wikipedia/commons/1/1e/Projector_Screen_Geometry_Throw_Ratio.png

II. Diagrams

Figure A1: Hough Lines Example Image [2]

http://upload.wikimedia.org/wikipedia/commons/1/1e/Projector_Screen_Geometry_Throw_Ratio.png

http://en.wikipedia.org/wiki/Canny_edge_detector

http://docs.opencv.org/doc/tutorials/imgproc/imgtrans/hough_circle/hough_circle.html

http://docs.opencv.org/doc/tutorials/imgproc/imgtrans/hough_lines/hough_lines.html


Figure A2: Hough Circles Example Image [3]

Figure A3: Sample Histogram of White & Black Balls in HSB Color Space [1]

III. Test CasesTest Component: Image ProcessingTest Case Name: Matlab Base Image Processing – Canny Edge DetectionTest ID: PBT-IP-001Description: This test verifies that a Canny edge image can be obtained for the set of test

images using Matlab.Test InformationTest Setup: Run the Matlab program for obtaining Canny edge images on the set of test

images.Test Expected Result Pass Fail Comments

1 All rails are identifiable in the edge images.

2 All balls are identifiable in the edge images.

Overall Results:


Test Component: Image ProcessingTest Case Name: OpenCV Base Image Processing – Read ImageTest ID: PBT-IP-002Description: This test verifies that an image can be read from the camera into an OpenCV

program.Test InformationTest Setup: Run the OpenCV program for reading an image from the camera.Test Expected Result Pass Fail Comments

1 Image is successfully loaded into the program.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Base Image Processing – Canny Edge DetectionTest ID: PBT-IP-003Description: This test verifies that a Canny edge image can be obtained for the set of test

images using OpenCV.Test InformationTest Setup: Run the OpenCV program for obtaining Canny edge images on the set of test


1 All rails are identifiable in the edge images.

2 All balls are identifiable in the edge images.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Base Image Processing – TimingTest ID: PBT-IP-004Description: This test verifies that the Base Image Processing component is completed within

TBD seconds. This includes reading an image and performing Canny edge detection.

Test InformationTest Setup: Run the OpenCV program which does the base image processing on the set of

test images and observe the timing analysis.Test Expected Result Pass Fail Comments

1 TBD seconds to process.Overall Results:

Test Component: Image ProcessingTest Case Name: Matlab Table Feature Detection – Hough LinesTest ID: PBT-IP-005Description: This test verifies that a Hough line image can be obtained for the set of test


images using Matlab.Test InformationTest Setup: Run the Matlab program for obtaining Hough lines on the set of test images.Test Expected Result Pass Fail Comments

1 Hough lines are identifiable in the images.

Overall Results:

Test Component: Image ProcessingTest Case Name: Matlab Table Feature Detection – Rail IdentificationTest ID: PBT-IP-006Description: This test verifies that the rails of the table are detected for the set of test images

using Matlab.Test InformationTest Setup: Run the Matlab program for determining table rails on the set of test images.Test Expected Result Pass Fail Comments

1 Rails are identified in the images.

Overall Results:


Test Component: Image ProcessingTest Case Name: Matlab Table Feature Detection – Corner PointsTest ID: PBT-IP-007Description: This test verifies that the intersection points for all rail corners are located for the

set of test images using Matlab.Test InformationTest Setup: Run the Matlab program for obtaining Hough lines on the set of test images.Test Expected Result Pass Fail Comments

1 Corner points are identified in the images.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Table Feature Detection – Hough LinesTest ID: PBT-IP-008Description: This test verifies that a Hough line image can be obtained for the set of test

images using OpenCV.Test InformationTest Setup: Run the OpenCV program for obtaining Hough lines on the set of test images.Test Expected Result Pass Fail Comments

1 Hough lines are identifiable in the images.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Table Feature Detection – Rail IdentificationTest ID: PBT-IP-009Description: This test verifies that the rails of the table are detected for the set of test images

using OpenCV.Test InformationTest Setup: Run the OpenCV program for determining table rails on the set of test images.Test Expected Result Pass Fail Comments

1 Rails are identified in the images.

Overall Results:


Test Component: Image ProcessingTest Case Name: OpenCV Table Feature Detection – Corner PointsTest ID: PBT-IP-010Description: This test verifies that the intersection points for all rail corners are located for the

set of test images using OpenCV.Test InformationTest Setup: Run the OpenCV program for obtaining Hough lines on the set of test images.Test Expected Result Pass Fail Comments

1 Corner points are identified in the images.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Table Feature Detection – TimingTest ID: PBT-IP-011Description: This test verifies that the Table Feature Detection component is completed within

TBD seconds. This includes obtaining the Hough line image, determining the location of the table’s rails, and determining the location of intersecting points from the rail lines.

Test InformationTest Setup: Run the OpenCV program which does the table feature detection on the set of



Test Component: Image ProcessingTest Case Name: Matlab Ball Detection – Hough CirclesTest ID: PBT-IP-012Description: This test verifies that a Hough circle image can be obtained for the set of test

images using Matlab.Test InformationTest Setup: Run the Matlab program for obtaining Hough circles on the set of test images.Test Expected Result Pass Fail Comments

1 Hough circles are identifiable in the images.

Overall Results:


Test Component: Image ProcessingTest Case Name: Matlab Ball Detection – Ball LocationTest ID: PBT-IP-013Description: This test verifies that the balls and their location are detected for the set of test

images using Matlab.Test InformationTest Setup: Run the Matlab program for determining ball locations on the set of test images.Test Expected Result Pass Fail Comments

1 Ball center points are identified in the images.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Ball Detection – Hough CirclesTest ID: PBT-IP-014Description: This test verifies that a Hough circle image can be obtained for the set of test

images using OpenCV.Test InformationTest Setup: Run the OpenCV program for obtaining Hough circles on the set of test images.Test Expected Result Pass Fail Comments

1 Hough circles are identifiable in the images.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Ball Detection – Ball LocationTest ID: PBT-IP-015Description: This test verifies that the balls and their location are detected for the set of test

images using OpenCV.Test InformationTest Setup: Run the OpenCV program for determining ball locations on the set of test images.Test Expected Result Pass Fail Comments

1 Ball center points are identified in the images.

Overall Results:


Test Component: Image ProcessingTest Case Name: OpenCV Ball Detection – TimingTest ID: PBT-IP-016Description: This test verifies that the Ball Detection component is completed within TBD

seconds. This includes obtaining the Hough circle image and determining the center point location of each ball.

Test InformationTest Setup: Run the OpenCV program which does the ball detection on the set of test images.Test Expected Result Pass Fail Comments


Test Component: Image ProcessingTest Case Name: Matlab Ball Color Calibration – Hough CirclesTest ID: PBT-IP-017Description: This test verifies that a Hough circle image can be obtained for the calibration

image(s) using Matlab.Test InformationTest Setup: Run the Matlab program for obtaining Hough circles on the calibration image(s).Test Expected Result Pass Fail Comments

1 Hough circles are identifiable in the image(s).

Overall Results:

Test Component: Image ProcessingTest Case Name: Matlab Ball Color Calibration – Ball LocationTest ID: PBT-IP-018Description: This test verifies that the balls and their location are detected for the calibration

image(s) using Matlab.Test InformationTest Setup: Run the Matlab program for determining ball locations on the calibration

image(s).Test Expected Result Pass Fail Comments

1 Ball center points are identified in the image(s).

Overall Results:


Test Component: Image ProcessingTest Case Name: Matlab Ball Color Calibration – Ball Color HistogramsTest ID: PBT-IP-019Description: This test verifies a color histogram is obtained for each ball in the calibration

image(s) using Matlab.Test InformationTest Setup: Run the Matlab program for obtaining ball color histograms on the calibration


1 Ball color histograms are uniquely identifiable.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Ball Color Calibration – Hough CirclesTest ID: PBT-IP-020Description: This test verifies that a Hough circle image can be obtained for the calibration

image(s) using OpenCV.Test InformationTest Setup: Run the OpenCV program for obtaining Hough circles on the calibration image(s).Test Expected Result Pass Fail Comments

1 Hough circles are identifiable in the image(s).

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Ball Color Calibration – Ball LocationTest ID: PBT-IP-021Description: This test verifies that the balls and their location are detected for the calibration

image(s) using OpenCV.Test InformationTest Setup: Run the OpenCV program for determining ball locations on the calibration


1 Ball center points are identified in the image(s).

Overall Results:


Test Component: Image ProcessingTest Case Name: OpenCV Ball Color Calibration – Ball Color HistogramsTest ID: PBT-IP-022Description: This test verifies a color histogram is obtained for each ball in the calibration

image(s) using OpenCV.Test InformationTest Setup: Run the OpenCV program for obtaining ball color histograms on the calibration


1 Ball color histograms are uniquely identifiable.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Ball Color Calibration – TimingTest ID: PBT-IP-023Description: This test verifies that the Ball Color Calibration component is completed within a

reasonable amount of time. This includes obtaining the Hough circle image, determining the center point location of each ball, and obtaining a histogram of the color for each detected ball.

Test InformationTest Setup: Run the OpenCV program which does the ball color calibration on the calibration

image(s) and observe the timing analysis..Test Expected Result Pass Fail Comments


Test Component: Image ProcessingTest Case Name: Matlab Ball Color Identification – Ball Color HistogramsTest ID: PBT-IP-024Description: This test verifies a color histogram is obtained for each ball in the set of test

images using Matlab.Test InformationTest Setup: Run the Matlab program for obtaining ball color histograms on the set of test


1 PBT-IP-012 Passed2 PBT-IP-013 Passed3 Ball color histograms are

uniquely identifiable for each image.

Overall Results:


Test Component: Image ProcessingTest Case Name: Matlab Ball Color Identification – Histogram ComparisonTest ID: PBT-IP-025Description: This test verifies a ball is identified by comparing its histogram to those

determined from the calibration phase using Matlab.Test InformationTest Setup: Run the Matlab program for comparing ball color histograms for each image.Test Expected Result Pass Fail Comments

1 PBT-IP-024 Passed2 Balls are correctly identified

for each image.Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Ball Color Identification – Ball Color HistogramsTest ID: PBT-IP-026Description: This test verifies a color histogram is obtained for each ball in the set of test

images using OpenCV.Test InformationTest Setup: Run the OpenCV program for obtaining ball color histograms on the set of test


1 PBT-IP-014 Passed2 PBT-IP-015 Passed3 Ball color histograms are

uniquely identifiable for each image.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Ball Color Identification – Histogram ComparisonTest ID: PBT-IP-027Description: This test verifies a ball is identified by comparing its histogram to those

determined from the calibration phase using OpenCV.Test InformationTest Setup: Run the OpenCV program for comparing ball color histograms for each image.Test Expected Result Pass Fail Comments

1 PBT-IP-026 Passed2 Balls are correctly identified

for each image.Overall Results:


Test Component: Image ProcessingTest Case Name: OpenCV Ball Color Identification – TimingTest ID: PBT-IP-028Description: This test verifies that the Ball Color Identification component is completed within

TBD seconds. This includes obtaining the color histogram for each detected ball and identifying each ball by comparing its histogram to that obtained from the calibration phase.

Test InformationTest Setup: Run the OpenCV program which does the ball color identification on the set of



Test Component: Image ProcessingTest Case Name: Matlab Complete Image Processing ComponentTest ID: PBT-IP-029Description: This test verifies that the entire Image Processing component functions as a

complete unit, resulting in a data structure containing the location of rail corners, ball center point coordinates, and the identity of each ball in Matlab.

Test InformationTest Setup: Run the Matlab program for the complete image processing component on the

set of test images.Test Expected Result Pass Fail Comments

1 Correct rail corner coordinates obtained.

2 Correct ball center coordinates obtained.

3 Correct ball color identification.

Overall Results:


Test Component: Image ProcessingTest Case Name: OpenCV Complete Image Processing ComponentTest ID: PBT-IP-030Description: This test verifies that the entire Image Processing component functions as a

complete unit, resulting in a data structure containing the location of rail corners, ball center point coordinates, and the identity of each ball in OpenCV.

Test InformationTest Setup: Run the OpenCV program for the complete image processing component on the

set of test images.Test Expected Result Pass Fail Comments

1 Correct rail corner coordinates obtained.

2 Correct ball center coordinates obtained.

3 Correct ball color identification.

Overall Results:

Test Component: Image ProcessingTest Case Name: OpenCV Complete Image Processing Component - TimingTest ID: PBT-IP-031Description: This test verifies that the entire Image Processing component is completed within

TBD seconds. This includes reading an image form the camera, performing Canny edge detection, performing Hough line and Hough circle transforms, detecting table rails, obtaining table rail corner points, detecting balls, obtaining ball center points, and determining the identity of each ball.


set of test images and observe the timing analysis.Test Expected Result Pass Fail Comments

1 PBT-IP-030 Passed2 TBD seconds to process.

Overall Results:


Test Component: SimulationTest Case Name: Real-world Shots vs. SimulationsTest ID: PBT-SIM-001Description: This test verifies that a shot made closely relates to the simulated result. Due to

the inexact nature of human ability, this test tests the simulation software by attempting to determine if the simulation is capable of recreating a shot made by a human.

Test InformationTest Setup: Enter the “developer mode” of the system, so that parameters of the simulation

can be directly controlled. Potential shots can be set up arbitrarily. The developer mode will accept at least parameters for the angle of the shot on the cue ball from a known origin, the drift from the center-point of the cue ball that the cue struck, and the force of the shot, to accurately relay to the simulation the dynamics of the shot made.

Test Expected Result Pass Fail Comments1 With an object ball placed just outside of

a side pocket and a cue ball placed in line with the object ball and the pocket, a follow-shot pockets the object ball and the cue ball.

2 With an object ball placed just outside of a side pocket and a cue ball placed in line with the object ball and the pocket, a draw-shot pockets the object ball, and the cue ball stops in the “ghost ball” position.

3 With the object ball placed just outside of a corner pocket, and the cue ball at a moderate cut angle, a follow-shot will see the cue ball deflect from the OB by the “30 degree rule” and then follow the geometric line path thereafter until coming to rest, depending on force.

4 With the object ball placed just outside of a corner pocket, and the cue ball at a moderate cut angle, a “good action” draw shot will see the cue ball deflect from the OB by the “90 degree rule”, and then follow first the parabolic, then the geometric line path thereafter until coming to rest, depending on force.

5 With the object ball placed outside of a corner pocket, the cue ball placed in line with the object ball, and a third ball resting in-between the OB and the CB on the aim line, a draw shot will pocket the object ball, leave the third ball at the OB


ghost ball position, and the CB at the third ball ghost ball position.

6 With the OB placed outside of a side pocket, and the CB in position to make a moderate angle kick shot, a center-ball kick shot will pocket the OB and the cue ball will follow the “30 degree rule” and follow the geometric path thereafter.

7 With the OB and CB placed to make a bank shot into a side pocket, a center-ball bank shot will pocket the OB and then the CB will follow the “30 degree rule” and follow the geometric path.

Overall Results:

Test Component: External SensorTest Case Name: Shot DetectionTest ID: PBT-ES-001Description: This test verifies that shot strength is quantifiable.Test InformationTest Setup: Attach external sensor to the cue stick to gather snapshot of different shots. Run

the snapshots through the external sensor algorithm to verify results.Test Expected Result Pass Fail Comments

1 The algorithm outputs shot strength of “1” when the cue ball was hit with TBD force.



4 The algorithm does not provide output when the cue stick is moved, but not used to hit the cue ball.

Overall Results:


Test Component: External SensorTest Case Name: Relay InformationTest ID: PBT-ES-002Description: This test verifies that the external sensor can communicate with the embedded

Linux computer.Test InformationTest Setup: Have external sensor synced to Bluetooth dongle connected to the embedded

Linux computer.Test Expected Result Pass Fail Comments

1 When the system is looking for external sensor data, the buffers should fill up with data.

Overall Results:

Test Component: External SensorTest Case Name: Power ConsumptionTest ID: PBT-ES-003Description: This test verifies that the external sensor will function on battery power for an

extended period of time.Test InformationTest Setup: Measure the current used when acquiring and transmitting data and when in

standby mode.Test Expected Result Pass Fail Comments

1 The external sensor will use no more than TBDµA when streaming data.

2 The external sensor will use no more than TBDµA when in standby mode.

Overall Results:


Test Component: Camera SystemTest Case Name: Camera Commands VerificationTest ID: PBT-CS-001Description: This test verifies that the camera hardware module communicates properly using

the described command set in the product documentation for the camera module.

Test InformationTest Setup: Connect camera to Development PC and start the test version of the camera

software. The software will test each command individually and will print back the ACK message, if it gets one.

Test Expected Result Pass Fail Comments1 Run test software and verify

the ACK messages per command.

Overall Results:

Test Component: Camera SystemTest Case Name: Camera Communication TestingTest ID: PBT-CS-002Description: This test verifies that communications between the camera module and

development PC or BeagleBone Black are working properly.Test InformationTest Setup: Connect camera to Development PC and start the test version of the camera

software. Attach an oscilloscope to the RX and TX lines of the camera.Test Expected Result Pass Fail Comments

1 Run test software. Verify Terminal output

2 Run test software. Check that the waveforms from the RX pin range from 0 - 3.3 V and have square edges. Measure the smallest period, it should be about 8.7 uS.

3 Run test software. Check that the waveforms from the TX pin range form 0 - 3.3 V and have square edges. Measure the smallest period, it should be about 8.7 uS.

4 Disconnect the TX line. Attach an oscilloscope probe to the disconnected line. Verify that the line stays high (3.3 V)

Overall Results:


Test Component: Camera SystemTest Case Name: Camera Software TestTest ID: PBT-CS-003Description: This test verifies that the camera software module properly interfaces with the

camera hardware module.Test InformationTest Setup: Connect camera to Development PC and start the camera software. The software

will attempt to download an image from the camera.Test Expected Result Pass Fail Comments

1 Run camera software. Examine the filesystem for a new image file with the extension .jpg

Overall Results:

Test Component: Camera SystemTest Case Name: Camera Coverage VerificationTest ID: PBT-CS-004Description: This test verifies that the camera is capable of viewing the entire table top.Test InformationTest Setup: Mount camera above table starting at 5 ft. Connect camera to Development PC

and start the camera software. The software will attempt to download an image from the camera.

Test Expected Result Pass Fail Comments1 Run camera software.

Examine the filesystem for a new image file with the extension .jpg. Open the file and verify the entire table top is in view.

2 If 1 failed, raise camera .5 feet and repeat test.

Overall Results:


Test Component: Camera SystemTest Case Name: HV Serial TestTest ID: PBT-CS-005Description: This test verifies that the MAX232 serial converters are properly converting the

3.3 V TTL to RS-232.Test InformationTest Setup: Connect camera to Development PC through a set of MAX232 ICs and start the

test version of the camera software. Attach an oscilloscope to the RX and TX lines of the MAX232 chips.

Test Expected Result Pass Fail Comments1 Run test software. Verify

Terminal output2 Run test software. Check

that the waveforms from the RX pin are -13 to 13 V and have square edges. Measure the smallest period, it should be about 8.7 uS.

3 Run test software. Check that the waveforms from the RX pin are -13 to 13 V and have square edges. Measure the smallest period, it should be about 8.7 uS.

4 Disconnect the TX line. Attach an oscilloscope probe to the disconnected line. Verify that the line stays at-13 V

Overall Results:

Test Component: Display SystemTest Case Name: Projection System TestTest ID: PBT-DS-001Description: This test verifies that the Embedded Linux computer’s video output operates

properly with the ProjectorTest InformationTest Setup: Connect BeableBone Black’s micro-HDMI to micro-HDMI to DVI convert. Connect

the DVI to VGA converter. Use VGA cable and connect projector to the BBB’s output.

Test Expected Result Pass Fail Comments1 Power on projector. Image

should come up of the BBB’s Xorg front end.

Overall Results:


Test Component: Display SystemTest Case Name: OpenCV Painter TestTest ID: PBT-DS-002Description: This test verifies that OpenCV is capable of “painting” the Linux Frame Buffer,

effectively modifying the projector’s output.Test InformationTest Setup: Connect BeableBone Black’s micro-HDMI to micro-HDMI to DVI convert. Connect

the DVI to VGA converter. Use VGA cable and connect projector to the BBB’s output. Run the OpenCV test program.

Test Expected Result Pass Fail Comments1 Power on projector. Using a

terminal, run the OpenCV Test program. The screen will go black. Each second a new shape should appear on the screen until the program is halted.

Overall Results:

Test Component: Display SystemTest Case Name: Projection Coverage TestTest ID: PBT-DS-003Description: This test verifies that the projector is capable for covering the Pool Table’s playing

surface.Test InformationTest Setup: Mount projector above playing surface about 5 feet, OR project on a wall of

similar size from 5 feet away.Test Expected Result Pass Fail Comments

1 Power on projector. Verify adequate coverage of the playing surface from a mount point of 5 ft.

Overall Results:


Test Component: Projection Alignment – IntegrationTest Case Name: Projection AlignmentTest ID: PBT-PA-001Description: This test verifies that the table projections are aligned with the internal

representation of table state and intended guidance output.Test InformationTest Setup: For each test, place the object ball at least at each of the six distinct locations (a

small distance from the pocket in each location), in order to determine that the projection is aligned with the whole table.

Test Expected Result Pass Fail Comments1 A ghost ball projected adjacent to an

object ball near each of the six pockets is flush with the tangent of the object ball (a real ball occupying the space of the ghost projection would just touch the object ball).

2 A ghost ball projected behind a ball placed near every pocket in turn gives a straight line through the ghost- and object-ball center-points into the middle of each pocket.

3 An aim-line projected through the center-points of the cue ball and the object ball ends at the edge of each ball, and is coincident with the real line between those points.

4 An aim-line projected through the center-point of the object ball and the target location ends at the edge of the ball and the target location, and is coincident with the real line between those two points.

Overall Results:


Test Component: Image ProcessingTest Case Name: Image Processing & Simulation IntegrationTest ID: PBT-IP-032Description: This test verifies that the Image Processing component is able to send valid

information to the simulation component. This information should be in the form of a data structure including the coordinates of the play area’s corners, the coordinates of each ball, and the identification of each ball.


set of test images and pass the information to the simulation component.Test Expected Result Pass Fail Comments

1 PBT-IP-030 Passed2 Valid information received

by simulation component.Overall Results:


IV. MilestonesThe Gantt chart can be found in the appended “PBTGantt.pdf” document. Listed below is the milestone chart with all objectives and their completion dates.

Task Scheduled End

Personnel Actual End Notes

Pull Image from Camera

1/26/2015 GJD 1/25/2015 Completed

Resolve MAX232 Serial Conversion

1/26/2015 GJD 1/25/2015 Unnecessary

Hough Transform Lines – MATLAB

1/26/2015 BAS 2/8/2015 Completed

Hough Transform Circles - MATLAB

1/26/2015 BAS 2/8/2015 Completed

Ball Color Identification - MATLAB

1/26/2015 BAS 2/8/2015 Completed

Interface SensorTag with PC

1/26/2015 RRG 3/8/2015 Completed

Design SensorTag Housing

2/9/2015 BAS 2/9/2015 Completed

Test Projector Output from BBB

2/17/2015 GJD 2/27/2015 Completed

Determine System Mounting Solution

3/2/2015 GJD, BAS 3/7/2015 Completed

Website Template 3/5/2015 GJD 3/5/2015 CompletedUpdate Gantt Chart with New Milestones

3/8/2015 GJD, BAS, CJR, RRG

3/8/2015 Completed

System API Defined 3/12/2015 GJD, BAS, CJR, RRG

3/15/2015 Completed

Camera Library Complete

3/15/2015 GJD 3/17/2015 Completed

Display Output Solution


ImagineRIT Proposal 3/20/2015 GJD, BAS, CJR, RRG

3/20/2015 Completed

Rail-Identification Algorithm Implemented

3/22/2015 BAS, CJR 3/25/2015 Completed


Task Scheduled End


Camera-Projector Alignment Algorithm Implemented

3/22/2015 BAS, CJR 4/19/2015 Completed

Scoring of Shot Difficulty

3/23/2015 CJR 3/23/2015 Completed

Simulation of Game Rules

3/23/2015 CJR 3/23/2015 Completed

SensorTag Button Software

3/29/2015 RRG 4/29/2015 Completed

Quantify Shot Strength

3/29/2015 RRG Not Completed

Hough Transform Lines - OpenCV

4/6/2015 BAS 4/19/2015 Completed

Hough Transform Circles – OpenCV

4/6/2015 BAS 4/12/2015 Completed

Ball Color Identification - OpenCV

4/6/2015 BAS 4/29/2015 Completed

Develop Feedback System

4/6/2015 RRG, CJR Not Completed

Interface SensorTag with BBB

4/6/2015 RRG 4/29/2015 Completed

Core Simulation 4/6/2015 CJR Not CompletedUpdated Proposal Document

4/12/2015 GJD, RRG, CJR, BAS

5/12/2015 Completed

Integrate Camera & Image Processing Software


Display Feedback to User

4/20/2015 RRG, CJR Not Completed

SensorTag Power Analysis

4/20/2015 RRG Not Completed

Camera/Projector Calibration Software

4/20/2015 BAS, CJR, GJD

4/19/2015 Completed

Integrate Image Processing & Simulation Software

4/20/2015 CJR, BAS 4/29/2015 Completed

Integrate Simulation Software & Projector Output

4/20/2015 GJD, CJR 4/29/2015 Completed


Task Scheduled End


Table Display System Library and Backend

4/20/2015 GJD, (CJR) 4/24/2015 Completed

Poster Draft 4/21/2015 GJD, RRG, CJR, BAS

4/21/2015 Completed

Project Poster 4/23/2015 GJD, RRG, CJR, BAS

4/23/2015 Completed

Website Draft 4/30/2015 GJD, RRG, CJR, BAS

4/16/2015 Completed

ImagineRIT 5/1/2015 GJD, RRG, CJR, BAS

5/2/2015 Completed

Final Report 5/5/2015 GJD, RRG, CJR, BAS

5/12/2015 Completed

Website Complete 5/7/2015 GJD, RRG, CJR, BAS

5/11/2015 Completed

edge.rit.eduedge.rit.edu/edge/C15004/public/docs/Final Report... · Web viewNeeds Statement....

Documents

Transcript of edge.rit.eduedge.rit.edu/edge/C15004/public/docs/Final Report... · Web viewNeeds Statement....