Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti...
Transcript of Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti...
![Page 1: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/1.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Hand TrackingM Tech Seminar
Pratik Kalshetti
IIT Bombay
November 22, 2016
![Page 2: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/2.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Outline1 Introduction
Immersive Virtual EnvironmentChallenges in Hand TrackingAim of Seminar
2 ContributionElements of a Hand Tracking System
Sensor TypeHand ModelReal-time MeasurementsCandidate Evaluation
Pipeline of a Hand Tracking System
3 ImplementationSensorHand ModelForward Kinematics for AnimationExperimental SetupResults
4 Conclusion
![Page 3: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/3.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Immersive Virtual Environment
• Computer-generated scene
• First person point of view
• Manipulating virtualobjects using human hand
• Hand needs to be trackedfor communicating withthe virtual world Figure: Hologram
Image Source: [4]
![Page 4: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/4.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Challenges in Hand Tracking
High DoF
Figure: Human Hand.
Image Source: [2]
Sensor Limitation
Figure: Fingers behindare occluded.
Robustness
Figure: Robustnessdecreases withdistance.
![Page 5: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/5.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Aim of Seminar
• Understand various hand tracking strategies in recent era
• Provide a holistic view of the major stages in a handtracking system
• Implement a hand tracking system to test the state of theart technology
![Page 6: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/6.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Elements of a Hand Tracking System
Figure: Elements of Hand Tracking System.
![Page 7: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/7.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Sensor Type
Marker-based Tracking
Figure: Tracking using Color Glove
Image Source: [8]
Vision-based Tracking
Figure: Tracking using DepthSensor
Image Source: [6]
![Page 8: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/8.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Hand Model
Complex Mesh Model
Figure: Linear blend skinnedmodel. The numbers near eachjoint indicate the allowed DoF.
Image Source: [1]
Simple Mesh Model
Figure: From left to right: Thecylinder model used for tracking,the skeleton, the BVH skeletonexported to Maya to drive therendering, the rendered handmodel
Image Source: [7]
![Page 9: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/9.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Real-time Measurements
Accurate measurements help in recovery from tracking loss
Figure: Hand Segmentation using skin silhouettes.
Image Source: [5]
![Page 10: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/10.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Candidate Evaluation
• Appearance-based Method - Fast; Fails when Occlusion
• Model-based Method - Slow; Works for Partial Occlusion
Figure: Evaluation Strategies
![Page 11: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/11.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Pipeline of a Hand Tracking System
Figure: Overall Pipeline.
![Page 12: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/12.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Sensor Type
Leap Motion Sensor
(a) Hand Tracking usingLeap Motion Sensor.
(b) Inside Leap MotionSensor.
Figure: Hand Tracking Hardware
Image Source: [3]
![Page 13: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/13.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Hand Model25 DoF Model made of cylinders and quad in OpenGL
Figure: Front View of implemented Hand model.
![Page 14: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/14.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Forward Kinematics for Animation
Forward Kinematics
Q: n - vector of joint anglesX : vector defining endpoint’s position and orientationX = F(Q)
(a) Step 1 (b) Step 2 (c) Step 3 (d) Step 4
Figure: Animation of bending Index Finger using Forward Kinematics
![Page 15: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/15.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Experimental Setup
Software: Leap Motion SDK; OpenGL API
Figure: Experimental Setup.
![Page 16: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/16.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Results
(a) Pose 1. (b) Pose 2.
Figure: Successfully Tracked Poses
![Page 17: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/17.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Results
(a) Occlusion. (b) Fast motion.
Figure: Incorrectly Tracked Poses
![Page 18: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/18.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Conclusion
• Open new doors to the world of human computerinterfaces
• Modular division of stages will help focus on each elementindependently
• Implemented model suggest the persisting problems withcurrent hand tracking systems
![Page 19: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/19.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
References I
Luca Ballan et al. “Motion capture of hands in actionusing discriminative salient points”. In: EuropeanConference on Computer Vision. Springer. 2012,pp. 640–653.
Anatomy Human Body. Diagram Hand Bones AnatomyHand And Wrist Diagram Human Anatomy Diagram.url: http://www.anatomy-diagram.info/diagram-hand-bones-anatomy/diagram-hand-bones-anatomy-
hand-and-wrist-diagram-human-anatomy-diagram/
(visited on 11/17/2016).
Mike Hord. Leap Motion Teardown. url:https://learn.sparkfun.com/tutorials/leap-
motion-teardown (visited on 11/08/2016).
![Page 20: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/20.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
References II
Jason Mick. Microsoft CEO: HoloLens V1 Will Debut forDevs in 2016, See 5 Year Evolution. url:http://www.dailytech.com/Microsoft+CEO+
HoloLens+V1+Will+Debut+for+Devs+in+2016+See+5+
Year+Evolution/article37456.htm (visited on11/17/2016).
Juan Manuel Ramirez-Cortes et al. “Shape-based handrecognition approach using the morphological patternspectrum”. In: Journal of Electronic Imaging 18.1 (2009),pp. 013012–013012.
Toby Sharp et al. “Accurate, robust, and flexible real-timehand tracking”. In: Proceedings of the 33rd Annual ACMConference on Human Factors in Computing Systems.ACM. 2015, pp. 3633–3642.
![Page 21: Hand Tracking - M Tech Seminarpratikm/projectPages/...art technology Hand Tracking Pratik Kalshetti Introduction Immersive Virtual Environment Challenges in Hand Tracking Aim of Seminar](https://reader035.fdocuments.us/reader035/viewer/2022070812/5f0b3cb87e708231d42f85c9/html5/thumbnails/21.jpg)
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
References III
Andrea Tagliasacchi et al. “Robust Articulated-ICP forReal-Time Hand Tracking”. In: Computer GraphicsForum. Vol. 34. 5. Wiley Online Library. 2015,pp. 101–114.
Robert Y Wang and Jovan Popovic. “Real-timehand-tracking with a color glove”. In: ACM transactionson graphics (TOG). Vol. 28. 3. ACM. 2009, p. 63.