Adam Hedji Mantas Pulinas Philip San III. Viktorious
description
Transcript of Adam Hedji Mantas Pulinas Philip San III. Viktorious
![Page 1: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/1.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Adam HedjiMantas PulinasPhilip SanIII. Viktorious
SSIP 2009, Debrecen, Hungary
![Page 2: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/2.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
![Page 3: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/3.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Project NatalX-Box 360Microsoft Corporation
Full Body Motion Detection
New project that promises to use two
cameras for full-body motion detection
![Page 4: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/4.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
CHECKERS!
![Page 5: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/5.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Overview of Code
Board Plane Geometry Detection
Board Plane Geometry Detection
![Page 6: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/6.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Overview of Code
Relaying of Data with Checkers A.I.
Relaying of Data with Checkers A.I.
FeedbackFeedback
Identification of Square Status by
Colour Classification
Identification of Square Status by
Colour Classification
Segmentation of Individual Game
Squares
Segmentation of Individual Game
Squares
Board Plane Geometry Detection
Board Plane Geometry Detection
This is blue!This is blue!
![Page 7: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/7.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection
• Find the corners
• 2 approaches were used:• Edge
detection and region growing
• Hough transforms
![Page 8: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/8.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Edge Detection and Region Growing
![Page 9: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/9.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Edge Detection and Region Growing• Edge detection with Sobel
operator• Dilate the image to fill the
gaps in the border
![Page 10: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/10.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Edge Detection and Region Growing• Edge detection with Sobel
operator• Dilate the image to fill the
gaps in the border• Thin the image to get the
true border
![Page 11: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/11.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Edge Detection and Region Growing• Edge detection with Sobel
operator• Dilate the image to fill the
gaps in the border• Thin the image to get the
true border• Dilate the image several
times to remove useless edges
![Page 12: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/12.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Edge Detection and Region Growing• Edge detection with Sobel operator• Dilate the image to fill the gaps in
the border• Thin the image to get the true
border• Dilate the image several times to
remove useless edges• Region growing• Determine the corner tiles• Track corners in real-time
![Page 13: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/13.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Projective Correction• Edge detection with Sobel operator• Dilate the image to fill the gaps in
the border• Thin the image to get the true
border• Dilate the image several times to
remove useless edges• Region growing• Determine the corner tiles• Track corners in real-time
![Page 14: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/14.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Projective Correction
Calculate homography using corner coordinates
![Page 15: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/15.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Geometry Detection – Hough Transform
• Hough transform– We had time so we developed a better
solution– Based on Hough transformations– Better real-time line detection
![Page 16: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/16.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional ControllersDevelopment of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
![Page 17: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/17.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour ClassificationHow can the computer classify the colours?
(171,154,158)(171,154,158)
(32,61,105)(32,61,105)
![Page 18: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/18.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour ClassificationUse image processing algorithms to make the RGB values only 0 or 255
(255,255,255)(255,255,255)
(0,0,255)(0,0,255)
![Page 19: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/19.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification
(255,255,255)
(0,0,0)
(255,255,0)
(0,0,255)
•Select individual tile
•Analyse the predominant colour inside to classify the square state (white, black, blue, yellow)
•Sample of pixels used as opposed to whole square
![Page 20: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/20.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification
•How to determine if each R, G and B values are 0 or 255?
-Need to choose threshold value
e.g.A given pixel of value (15, 19, 250)A threshold of 126Output is (0,0,255)
![Page 21: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/21.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification - But how to choose the threshold?• Webcams can be of relatively poor quality and
provide poor contrast.– For example, blue pieces used were relatively
hard to distinguish from the black tiles.
![Page 22: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/22.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional ControllersDevelopment of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
![Page 23: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/23.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
But how to choose the threshold?
• Can instead normalise the image and use a threshold of 127, given by (256/2)-1.– Select a white tile and take
an average of the colour values.
– Do the same for a black tile.– Use these averages to
normalise the image
p1=(241,209,210)
p2=(232,204,214)
p3=(240,211,205)
BlackWhite
BlackInputBGROutput
),,(
![Page 24: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/24.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification - Normalisation
![Page 25: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/25.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification - Normalisation
![Page 26: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/26.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification - Normalisation
![Page 27: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/27.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification - Normalisation
• Then reduce the image to absolute values of 0 and 255
• Use a threshold of 126 (half of full intensity value)
![Page 28: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/28.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification - Normalisation
![Page 29: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/29.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Colour Classification - Normalisation
![Page 30: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/30.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
![Page 31: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/31.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
![Page 32: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/32.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Summary
• Main results– Successfully used 2 approaches to chess
board detection• Edge detection and region growing• Hough transform
– Removing perspective distortion– Identification of individual tiles and pieces,
including classification– Connection to engine interface with feedback
system
![Page 33: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/33.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Improvements
• Aim towards real-time 60fps processing• Use a more efficient programming language
such as C++• Use of GPU using CUDA or Open CL
programming language• More complex algorithms
– Motion detection of hand• Use of overlay of 3D structures onto camera
image.• Virtual humans...
![Page 34: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/34.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
Unused Virtual Human
![Page 35: Adam Hedji Mantas Pulinas Philip San III. Viktorious](https://reader036.fdocuments.us/reader036/viewer/2022062500/56815467550346895dc281d0/html5/thumbnails/35.jpg)
Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers
The Team
Adam HedjiUniversity of Zagreb, [email protected]
Mantas PaulinasVilnius Gediminas Technical University, [email protected]
Viktor BlaskovicsUniversity of Szeged, [email protected]
Philip SanUniversity College London, [email protected]