Visug: Say Hello to my little friend: a session on Kinect
description
Transcript of Visug: Say Hello to my little friend: a session on Kinect
![Page 1: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/1.jpg)
Say hello to my little friend
Tom Kerkhove
![Page 2: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/2.jpg)
Nice to meet you
Tom KerkhoveKinect for Windows MVPIntegration Professional at Coditwww.kinectingforwindows.com
be.linkedin.com/in/tomkerkhove/
@TomKerkhove
+32 473 70 10 74
![Page 3: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/3.jpg)
How Codit can help?
Integration services• Advice
• Projects
• Implementation
• SOA Governance
• Managed Services
• Integration as a Service
• Codit Integration Cloud
![Page 5: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/5.jpg)
Agenda
• Hardware• Features Overview• Diving into the SDK• Kinect for Xbox vs Kinect for Windows• Scenarios• Kinecting the cloud
![Page 6: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/6.jpg)
Hardware
![Page 7: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/7.jpg)
Sensor Components
Power Light
RGB Camera
IR Emitters
Depth Sensor
Microphone Array
Image: iFixit, http://www.ifixit.com
![Page 8: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/8.jpg)
Sensor Components
Image: iFixit, http://www.ifixit.com
Power Light
RGB Camera
IR Emitters
Depth Sensor
Microphone Array
Depth Sensor
IR Emitters
![Page 9: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/9.jpg)
Additional information
• Field-of-View• Depth & infrared
• Horizontal – 70°• Vertical – 60°
• Color camera• Horizontal – 85°• Vertical – 54°
• Tilt motor removed
• Running multiple applications using one sensor is now supported• Using multiple sensors on one machine is no longer supported
![Page 10: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/10.jpg)
Feature Overview
![Page 11: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/11.jpg)
Camera
• Full HD 1920 x 1080 camera• 30 or 15 FPS based on lighting• Different color formats• RGBA• GBRA• YUV2
• Mirrored image
![Page 12: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/12.jpg)
Infrared
• Light independent infrared• Ambient light removed• 16-bit IR intensity• No heat detection• 30 FPS at 512 x 424
![Page 13: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/13.jpg)
Depth
• Improved depth fidelity (3x)• Distances provided in mm/pixel• No longer ‘short-range’ mode• Range of 0.5m up to 8 m• 30 FPS at 512 x 424
![Page 14: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/14.jpg)
Body, the new skeletal tracking
• Range 0.5 - 4.5 meters • Fully tracks six bodies simultaneously• Tracks up to 25 joints• Position in 3D space in meters• Rotation available in quaternion
• Hand state tracking for two persons• Unique ID per skeleton• 30 frames per second• Lean
![Page 15: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/15.jpg)
Body Index
• Indicate which pixels have which players• 0 to 5 indicates corresponding body• >5 means no body is track
![Page 16: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/16.jpg)
Interaction
• Ability to interact with your application• Includes control to visualize user• Re-use default XAML controls
Press Panning/Zoom
![Page 17: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/17.jpg)
Face
• Up to 6 bodies simultaniously• 2-Dimensional based
• Face Detection• Detects a bounding box around the face
• Face Alignment• Identified 5 facial landmarks
• Face Orientation• Quaternion of the head in relation with sensor
• Face Expressions• Indication if person is happy, wearing glasses,...
![Page 18: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/18.jpg)
HD Face
• Up to 6 bodies simultaniously (Not recommended)
• 3-Dimensional based
• Creates HD mesh of the face• Face needs to be “build”• Detects hair & skin color• Provides 94 shape units
![Page 19: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/19.jpg)
Fusion
• Creates a 3D mesh of environment & objects• Surface reconstruction done with voxels• Heavy processing (GPU-based)
![Page 20: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/20.jpg)
Speech
• Can automatically focus on engaged user• Automatic ambient noise filter• Steerable cone for audio focus• Manually or automatically
![Page 21: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/21.jpg)
Diving into the SDK
![Page 22: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/22.jpg)
High-level architecture
Kinect Drivers
Kinect Runtime
.NetAPI
.NetApps
Native API
Native Apps
WinRT API
Windows Store Apps
C#, VB, JS, C++/CX
Physical Kinect Sensor
![Page 23: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/23.jpg)
Programming flowSensor
Source
Reader
Frame
Reference
Frame
• Sensor• Represents a physical sensor
• Source• Choose your requested data source• Exposes metadata as well
• Reader• Allow you to access frames by events/polling• Support for readers on a single source• Can be paused
• Frame Reference• Access to actual frame• RelativeTime allows you to correlate frames
• Frame• Acces to the frame data• Contains frame metadata• Only one frame can be processed at a time
![Page 24: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/24.jpg)
Different spaces
• Sources have different spaces
• Conversion between spaces with coordinate mapper
![Page 25: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/25.jpg)
Working with multiple sources
• All data frames arrive at different time• Can be correlated
• Alternative – MultiSourceFrameReader• Acquire all frames at same time• Frames will arrive at lowest FPS
![Page 26: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/26.jpg)
Required Setup
• CPU -> i7 dual core 3.1 GHz (Recommended)
• RAM -> 4 GB (Recommended)
• GPU -> DirectX 11 (Required)
• USB 3.0 -> Inter or Renesas chipsets (Required)
• OS -> Windows 8.0/8.1 (Required)
• Use Kinect Configuration Verifier to validate your setup• Ships with the SDK
![Page 27: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/27.jpg)
Building applications
• Windows Store apps (XAML / HTML)
• Driver automatically installed by Windows Update• RT devices are not supported
• Windows Presentation Foundation• C++ applications• Support for 3th party frameworks
![Page 28: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/28.jpg)
SDK Browser
• Library of samples in C#, C++ & Windows Store• Includes links to other components & tools• Updated over the cloud• Ships with the SDK
![Page 29: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/29.jpg)
![Page 30: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/30.jpg)
Kinect Studio
• Allows you to monitor, record & playback clips• Can now run without connected sensor• Improves testing & debugging• All sources can be recorded
![Page 31: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/31.jpg)
Gestures
Basic GesturesDeveloper detect gestures based on Body joints
Complex gesturesUses a machine learning approach with Gesture Builder
![Page 32: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/32.jpg)
Gesture Builder
• Tag gestures in Kinect Studio clips• Build the tagged clips & generate analysis data• Detect gestures in you app based on data
![Page 33: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/33.jpg)
Human Interface Guidelines
• UX guidelines for creating applications that are user-friendly• Learn how to get all the power out of the sensor• Developers need to read this as well
![Page 34: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/34.jpg)
Kinect for Windows & Xbox One
![Page 35: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/35.jpg)
Comparison for Gen. I
![Page 36: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/36.jpg)
Functionality Penalty in Gen. I
![Page 37: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/37.jpg)
Scenarios
![Page 38: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/38.jpg)
Scenarios
Healthcare Retail Entertainment
Education
![Page 39: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/39.jpg)
![Page 40: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/40.jpg)
![Page 41: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/41.jpg)
Liberty University slide
![Page 42: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/42.jpg)
Liberty University slide
![Page 43: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/43.jpg)
![Page 44: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/44.jpg)
![Page 45: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/45.jpg)
![Page 46: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/46.jpg)
![Page 47: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/47.jpg)
Liberty University slide
![Page 48: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/48.jpg)
Liberty University slide
![Page 49: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/49.jpg)
Kinecting the Cloud
![Page 50: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/50.jpg)
Kinecting the Cloud
ScaleScale based on users
Plug-and-playPlug-and-play applications
Data ProtectionStore and secure data
ComputeHeavy data processing
![Page 51: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/51.jpg)
Kinect On-Demand
![Page 52: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/52.jpg)
Kinect Video On-Demand
Media Services
Notification HubSend Notification
storage blob
Get Video Stream
Assets(Behind the scenes)
Push Notification
Video Playback
Notify clients
Upload, encode & package video
Save Frames
![Page 53: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/53.jpg)
Using Kinect as data farmer
![Page 54: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/54.jpg)
Using Kinect as a data farmer
• Conference attendees surveys at booth• Generates non-Kinect data
• Using Kinect to communicate with other devices locally/over the cloud• Save images of the attendees or integrate with RFID• Perform face analytics
![Page 55: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/55.jpg)
Using Kinect as a data farmer
![Page 56: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/56.jpg)
Using Kinect as a data farmer
• Data available on central hub• Forward data for further processing or analytic
![Page 57: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/57.jpg)
Using Kinect as a data farmer
• See global results of survey• Improved customer service
![Page 58: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/58.jpg)
Burglar Detection
![Page 59: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/59.jpg)
Burglar Detection
• Detects people in certain time window• Records video footage as proof• Notify owner of breach• Enable / Disable monitoring remotely
![Page 60: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/60.jpg)
Summary
• Kinect provides a rich set of features• Easy-to-use SDK• Excellent tooling to help during development• Applicable in a variety of scenarios• The cloud can come in handy in some scenarios
![Page 61: Visug: Say Hello to my little friend: a session on Kinect](https://reader034.fdocuments.us/reader034/viewer/2022052621/558ba5b6d8b42a146d8b465f/html5/thumbnails/61.jpg)
Thanks for partnering with Visug!