Programming with RealSense using .NET
41
-
Upload
matteo-valoriani -
Category
Technology
-
view
113 -
download
4
Transcript of Programming with RealSense using .NET
Matteo ValorianiPhD at Politecnico of MilanoCEO of Fifth ElementSpeaker and Consultant
Microsoft MVP
Intel Software Innovator
email: [email protected]: @MatteoValorianilinkedin: https://it.linkedin.com/in/matteovaloriani
Nice to Meet You
2
RealSense Introduction
Capture Hands Data
Detect HandsGestures
Understands Hardwsare Limits
RealSense Cameras
Agenda
The Intel® RealSense™ 3D Camera
4* Other names and brands may be claimed as the property of others
5Intel Confidential - for use under NDA only
Dell* Inspiron 23 7000
* Other Names and Brands may be claimed as the property of others
Available
Enthusiast-grade performance- Intel 4th-gen Core i7/i5 processors- NV GTX850M/860M
Silent cooling system- Efficient heat removal with minimum fan noise
Specially-designed Game Keys- Specially marked WASD keys- Chiclet Keyboard with Red backlighting
Majestic Sound- SonicMaster Premium- ROG Audio Wizard- Headphone amp to
enhance gaming headset experience
Gaming-centric design- Matte black finish with
red diamond cut- Full HD IPS display
with 178-degree wide-view
- Aluminum Design with illuminated ROG logo
Intel® RealSense3D Camera- Enhance experience with natural
interaction.- Immersive Collaboration/ Creation- Capture faces/ objects in full 3D
Asus* G771
Intel Confidential - for use under NDA only
* Other Names and Brands may be claimed as the property of others
What Kinds of Applications can you develop?
7
Capture and Share
Immersive Collaboration/Creation
Interact Naturally
Gamingand Play
Learning andEdutainment
dem
o
8
When we roll out the public beta SDK later in 2014, we’re exploring a developer kit to help you get startedKit would include:
• The Intel® RealSense™ SDK for Windows Beta
• Peripheral depth camera
• Full VGA depth resolution
• 1080p RGB camera
• 20cm – 120cm range
• USB 3.0 interface
• Requires 4th generation Intel® Core™ processor (or later) & Windows* 8.1
The Developer Kit
9
Competitive technologies focus on a living-room experience or a sub-set of Intel RealSense technology features
Designed for close-range interactions
10
Intel® RealSense™ Hands-On Lab - Milan
120 cm
Intel®RealSense™ 3D camera
56°(v) x 72° (v)
20 cm
Leap, RealSense, Kinect
11
Intel® RealSense™ Hands-On Lab - Milan
2,5 cm
60 cm 2 m 4 m
12
What OS and hardware works with RealSense technologies?
1Specific algorithms may have different range and accuracy.**Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.*Other names and brands may be claimed as the property of others.
Intel® RealSense™ SDKfor Windows*, Version 2014 supports Windows 8.x (64-bit), Windows 10
Download the SDK intel.com/realsense/sdk
Intel® RealSense™ Developer Kit 3D cameraGet started developing quickly :
• Full VGA depth resolution• 1080p RGB camera• 0.2 – 1.2 meter range1
• USB 3.0 interface• Requires Intel® Core™ processor
Reserve Yours Now
13
System Requirements
Required Hardware • A dedicated USB 3 port in the client system (for peripheral dev kit camera), do not use a hub
• 4th generation (or later) Intel® Core™ i5 or i7 processor with Intel® Iris™ Graphics and Intel® HD Graphics
• 8GB free hard disk space• Intel® RealSense™ 3D Camera (front-facing, F200 – integrated or peripheral)
Required OS • Microsoft Windows* 8.1 64-bit Desktop Mode, 10• Microsoft Windows 8.1 New UI (Metro) Mode (coming soon)**• Support for Android* coming soon**
Supported Programming Languages
• C++• C# (Microsoft .Net* 4.0 Framework is required)• Java* (JDK 1.7.0_11 or later)• JavaScript*
Supported IDE • Microsoft* Visual Studio 2010-2013 with service pack 1 or newer• Unity* PRO 4. 1. 0 or later for Unity game development
Supported Development Tools
• Microsoft* .NET 4.0 Framework for C# development• Processing* 2.1.2 or higher for Processing framework development• Java* JDK 1.7.0_11 or higher for Java development
Supported Browsers for JavaScript Development
• Microsoft Internet Explorer* 10.0.13• Google Chrome* 33.0.1750.146• Mozilla Firefox* 27.0.1
Understands 4 basic types of input
14
Categories of Input
Capabilities Features
Hands • Hand and Finger Tracking
• Gesture Recognition
• 22-point Hand and Finger Tracking• 9 static and dynamic mid-air gestures
Face • Face Detection and Tracking
• Multiple Face Detection and tracking• 78-point Landmark Detection (facial features)• Emotion Recognition (7 emotions, coming post-Beta)• Pulse Estimation• Face Recognition (Coming post-beta)
Speech • Speech Recognition • Command and Control• Dictation• Text to Speech
Environment • Segmentation• 3D Scanning• Augmented Reality
• Background Removal• 3D Object / Face / Room Scanning (Coming post-beta)• 2D/3D Object Tracking• Scene Perception (coming post-beta)
Coordinate Systems
15
World coordinates Image coordinates
Mass Center: Image and world coordinates of the calculated hand image mass center.
Extremity Points: Special tracking points such as the closest, left-most, top-most, right-most, bottom-most and center points which form the boundaries of the mask of the hand silhouette.
Body Side: Whether it is a left or right hand.
Palm Orientation: Estimation of where the hand is facing.
Tracked Joints: Positions and rotations of the user’s hand in world and image coordinates.
Hands Data
16
Normalized Joints: Posture of user’s hand without changing dimensions of the hand, i.e., the distances between each joint (bone-length) are always the same.
Finger Data: Degree of foldedness and the radius of a particular fingertip.
Segmentation Image: A mask of the hand silhouette allowing to separate a hand from the background image.
Hand Openness: (0-100 value) indicating if all fingers are completely folded to all fingers fully spread.
Hands Data
17
/* Initialize a PXCMSenseManager instance */
manager = PXCMSenseManager.CreateInstance();
if (manager == null){
Debug.Writeline("SenseManager Initialization Failed");
}
Creating a Session for Hand Tracking
18
PXCMSenseManager: organizes a pipeline by starting, stopping, and pausing the operations of its various modalities.
1. Initialize a PXCMSenseManager instance
2. Enable hand tracking using EnableHand.
3. Retrieve an instance of hand module using QueryHand on the instance of PXCMSenseManager.
4. Initialize the pipeline using Init
Initializing the Pipeline
19
/* Enable the hand tracking module*/ sts = manager.EnableHand();if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR) Debug.WriteLine("PXCSenseManager.EnableHand: " + sts);
/* Retrieve an instance of hand to configure */handAnalyzer = manager.QueryHand();if (handAnalyzer == null)
Debug.WriteLine("PXCSenseManager.QueryHand"); /* Initialize the execution pipeline */sts = manager.Init();if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR){ Debug.WriteLine("PXCMSenseManager.Init Failed");}
handData = handAnalyzer.CreateOutput();
Initializing the Pipeline
20
EnableHand(): Enable the hand tracking module.
QueryHand(): Retrieve an instance of hand to configure
Init(): Initialize the execution pipeline
config = handAnalyzer.CreateActiveConfiguration();
config.EnableAllGestures();
//config.EnableGesture("v_sign");
config.EnableAllAlerts();
config.ApplyChanges();
Enable Gestures
21
Enable All gestures or specific gestures to recognize
private CancellationTokenSource streamCancellationTokenSource;
private CancellationToken streamCancellationToken;
Task streamTask = new Task(new Action(AcquireFrames),
streamCancellationToken);
streamTask.Start();
Acquire Data using Task
22
private void AcquireFrames() {
while (manager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR) {
this.outputData.Update();
PXCMHandData.IHand handData;
PXCMHandData.GestureData gestureData;
//AcquireGestures
// AcquireHands
manager.ReleaseFrame();
}
}
AcquireFrames
23
Acquire new data end refresh the elaborated output
Release the frame or you can not elaborate the next !!!
Gesture Tracking
24
//Gestures
for (int i = 0; i < outputData.QueryFiredGesturesNumber(); i++){
if (outputData.QueryFiredGestureData(i, out gestureData) == pxcmStatus.PXCM_STATUS_NO_ERROR){
outputData.QueryHandDataById(gestureData.handId, out handData);
var bodySide = handData.QueryBodySide();
//Do somethings with data
sb.Clear();
sb.AppendLine("Timestamp: " + gestureData.timeStamp);
sb.AppendLine("Gestrue identified: " + gestureData.name);
sb.ppendLine("Gestrue state: " + gestureData.state);
sb.AppendLine("Gestrue handId: " + gestureData.handId);
sb.AppendLine("Gestrue bodyside: " + bodySide.GetType());
sb.AppendLine();
}
}
Analyze Gestures
25
Hand Tracking Modes
26
Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>[] hands = new Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>[4];
PXCMHandData.IHand[] totalHands = new PXCMHandData.IHand[4];
int numberOfHands = outputData.QueryNumberOfHands();
if (numberOfHands > 0) {
for (int i = 0; i < numberOfHands; i++) {
Dictionary<PXCMHandData.JointType, PXCMHandData.JointData> handDictionary =
new Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>();
PXCMHandData.JointData temp;
if (outputData.QueryHandData(PXCMHandData.AccessOrderType.ACCESS_ORDER_BY_ID, i, out handData) ==
pxcmStatus.PXCM_STATUS_NO_ERROR) {
// iterate through Joints
for (int j = 0; j < PXCMHandData.NUMBER_OF_JOINTS; j++) {
handData.QueryTrackedJoint((PXCMHandData.JointType)j, out temp);
handDictionary.Add((PXCMHandData.JointType)j, temp);
}
hands[i] = handDictionary;
totalHands[i] = handData;
// Do somethings with data
}}
Analyze Hands Joints
27
dem
o
28
Capture Volumes
29
Intel® RealSense™ Hands-On Lab - Milan
The user is performing a hand gesture outside of the capture volume. The camera will not see this gesture
Understands Hardware Limits
30
Vertical Rages
31
Intel® RealSense™ Hands-On Lab - Milan
60cm
58 c
m
120cm
56°
20cm
17 c
m
70cm 35cm73 c
m
Effective Range
Gestures Range
Effective Range
3D Facial Traking Range
2D Facial Traking Range
115c
m
Vertical Misalignment
32
Intel® RealSense™ Hands-On Lab - Milan
120c
m
56°
115c
m
Horizontal Rages
33
Intel® RealSense™ Hands-On Lab - Milan
60cm
87cm
120cm
72°
20cm
24cm
Effective Range
Gestures Range
Effective Range
3D Facial Traking Range
2D Facial Traking Range
170c
m
70cm 35cm108
cm
50cm
RealSense Cameras
FRONT-FACING
Intel RealSense 3D Camera F200Availability starting Q4’14
REAR-FACING
Intel RealSense 3D Camera R200In-Market Target 2H’15
Intel® RealSense™ Cameras
Primary: 8 MP, 3264 x 2448 pixels, autofocus
Secondary: 2 MP
Intel RealSense Snapshot, 2 x 720p cameras
RealSense SnapShot
36
Display: 8.4" OLED infinity (2560 x 1600)
Dimensions: 215.9 x 124.2 x 6.1 mm
Weight: 305 g
OS: Android OS, v4.4.2 (KitKat),
CPU: Intel Atom Z3580,Quad-core 2.3 GHz
Memory: 16 GB (+microSD), 2 GB RAM
Dell Venue 8 7000
37
Target Platform:Tablets, 2-in-1s, and moreThe camera typically faces away from the user. It was designed to sense the environment around it in 3 dimensions.
Specifications- Longer range** (up to 3-4 meters indoors, longer range outdoors)- Depth/IR: 640x480 resolution at 60fps- RGB: 1080p at 30fps- USB 3.0 required- Developer Kit Dimensions: 130mm x 20mm x 7mm- Supports Microsoft Windows* 8.1 (64-bit mode), Android* support coming soon.
Targeted Usages- 3D Capture: faces, people, and environments- Depth-enhanced Augmented Reality- Depth-enhanced photography and video- Measurement- Face detection and tracking
RealSense R200
Free Intel® RealSense™ SDK for Windows Beta in 2nd Half of 2014
Intel® RealSense™ Developer Kit being explored
Huge opportunity to reach customers with integrated 3D camera
Works with languages/frameworks/engines you already use
High-level APIs for NUI beginners. Low-level APIs for NUI experts
Getting Started
39 All products, software, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.
software.intel.com/RealSense
Q&A
Grazie a tutti per la partecipazione
Riceverete il link per il download a slide e demo via email nei prossimi giorni
Per contattarmi
Grazie
