JugEm : Software to help the world juggle
description
Transcript of JugEm : Software to help the world juggle
![Page 1: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/1.jpg)
JugEm: Software to help the world juggle
Level 4 Project 2013 PresentationDavid McBrierty
![Page 2: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/2.jpg)
Juggling
The act of continuously tossing into the air and catching (a number of objects) so as to keep at least one in the air while handling the others.
![Page 3: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/3.jpg)
The idea…
• Build a piece of software that can be used to teach people how to juggle.
![Page 4: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/4.jpg)
Existing Software
• Lots of existing tutorial help out there already….
![Page 5: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/5.jpg)
Existing Software
![Page 6: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/6.jpg)
More Existing Software
![Page 7: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/7.jpg)
Theres something missing….
• Existing tutorials are not interactive!• Learn by doing!
![Page 8: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/8.jpg)
What is JugEm?
• JugEm is an interactive application to help teach people how to juggle.
![Page 9: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/9.jpg)
Initial ideas….
• Use mobile devices to get sensor data• Make users juggle actual devices• Strap devices to users hands
• Wii Controllers• Existing project has done this!
![Page 10: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/10.jpg)
Kinect
• Kinect Sensor provides an interesting interface
![Page 11: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/11.jpg)
How good is a good juggle?
• Lots of lovely physics behind what happens• Claude Shannon’s Uniform Juggle Theorem
![Page 12: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/12.jpg)
Third Party Libraries
• Kinect for Windows SDK• Gives access to raw Kinect Data
• EmguCV Image Processing• .NET Wrapper around OpenCV library
• Microsoft Speech Platform 11• Allows Kinect Sensor to use voice commands
• Built around Microsoft XNA Framework• Framework for making Xbox and Windows software
![Page 13: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/13.jpg)
System RequirementsMUST HAVE SHOULD HAVE
COULD HAVE WOULD LIKE TO HAVE
Detect user throwsDetect user catchesDetect peaksTrack users hand positionsProvide report on techniqueDetect 1,2 and 3 Ball patterns
Calculate Shannon ScoreSuggestions on better techniqueDetect dropsDetect 4 Ball patterns
Tasks to improve timingTasks to improve throwing
Define patterns using Siteswap
![Page 14: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/14.jpg)
The good stuff….
• The system goes through 5 stages:
![Page 15: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/15.jpg)
Kinect Data Input
• Color Image represented as byte[]
• byte[] can easily be converted to an image.
![Page 16: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/16.jpg)
Kinect Data Input
• Depth Data provided as DepthImagePixel[]
• Each pixel from Color Image has DepthImagePixel for its distance from sensor
• Depth data has to be processed before it is useful!
![Page 17: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/17.jpg)
The good stuff….
• The system goes through 5 stages:
![Page 18: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/18.jpg)
Depth Image Filtering
• Raw depth data is very noisy, has to be filtered• 76800 pixels filtered (30 times a second)• Image is converted to gray scale (0-255)• All non player pixels are removed• EmguCV Binary threshold applied to results
• Anything behind players wrists is removed!
• Kinect thinks the balls are part of the player
![Page 19: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/19.jpg)
![Page 20: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/20.jpg)
![Page 21: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/21.jpg)
![Page 22: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/22.jpg)
![Page 23: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/23.jpg)
![Page 24: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/24.jpg)
The good stuff….
• The system goes through 5 stages:
![Page 25: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/25.jpg)
Ball Detection
• EmguCV runs Canny Edge Detection on image• Finds edges with specific radius• Need to detect circles and capped ellipses
![Page 26: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/26.jpg)
Screenshot
![Page 27: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/27.jpg)
Ball Color Detection
• Uses Kinect Color Image• RGB Color Detection proved to be inaccurate
• Hue, Saturation, Value Color Detection instead!
• Extra work, but necessary
![Page 28: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/28.jpg)
Frames
• A Frame is used to store all the data collected from processing
• All the Juggling Balls seen with color, direction and position• Positions of Users hands• Distance from each ball to each hand• Time Frame was seen
• Each new Frame is compared to the previous one to find out directions and detect Events.
![Page 29: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/29.jpg)
The good stuff….
• The system goes through 5 stages:
![Page 30: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/30.jpg)
Detecting Events
• Two areas around users hands• Hand• Throw
![Page 31: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/31.jpg)
Detecting Throws and Catches
• THROW: Ball seen inside Throw Zone and then outside
• CATCH: Ball seen outside Throw Zone and then inside
![Page 32: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/32.jpg)
The good stuff….
• The system goes through 5 stages:
![Page 33: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/33.jpg)
• System collects the Events generated and processes them once the user is finished
• Each type of Juggle has a pattern
• System must be able to cope with missed events
Processing Events
![Page 34: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/34.jpg)
Missed Events
• Inferring missed Events can be tricky…
• Pattern = P, RT, RC, P, LT, LC.
![Page 35: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/35.jpg)
Missed Events
• Inferring missed Events can be tricky…
• Pattern = P, RT, RC, P, LT, LC.
![Page 36: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/36.jpg)
• Inferring missed Events can be tricky…
• Pattern = P, RT, RC, P, LT, LC.
Missed Events
![Page 37: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/37.jpg)
• Inferring missed Events can be tricky…
• Pattern = P, RT, RC, P, LT, LC.
Missed Events
![Page 38: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/38.jpg)
• Inferring missed Events can be tricky…• The times of some inferred Events are slightly
out• Just sometimes a problem…
• Overall effect on time is small.
Missed Events
![Page 39: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/39.jpg)
Juggle Report
• After the Events have been analysed and processed
• System has all the information it needs for Shannon Score
• Display report to the user
![Page 40: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/40.jpg)
Personal Reflection
• Learned how to juggle!• Got to work in a new language• Working with Kinect• Working with all the frameworks• Image processing• Easily the biggest bit of work completed to
date….
![Page 41: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/41.jpg)
Personal Reflection
![Page 42: JugEm : Software to help the world juggle](https://reader036.fdocuments.us/reader036/viewer/2022062410/56816092550346895dcfb7d1/html5/thumbnails/42.jpg)
fin