IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...
-
Upload
nguyendien -
Category
Documents
-
view
223 -
download
1
Transcript of IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...
![Page 1: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/1.jpg)
S i l v i a R o d r í g u e z - J i m é n e z
C a r l o s I I I U n i ve r s i t y o f M a d r i d
V i s u a l P e r c e p t i o n S ys t e m w i t h i n H AN D L E E U p r o j e c t
u s i n g R O S
IROS 2012 Handling ROS tutorial
Vilamoura, October 7th 2012 http://mrl.isr.uc.pt/events/iros2012tutorial/
![Page 2: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/2.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 2
Robots need visual 3D perception
Picture from http://thecorpora.com
![Page 3: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/3.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 3
Outline
Depth measurement techniques: Kinect
ROS/Kinect integration
System for robotic visual perception
Interfacing with an object SQL database
ROS vision nodes within HANDLE EU project
![Page 4: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/4.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 4
Outline
Depth measurement techniques: Kinect
ROS/Kinect integration
System for robotic visual perception
Interfacing with an object SQL database
ROS vision nodes within HANDLE EU project
![Page 5: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/5.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 5
Depth measurement techniques
Picture from the Artist Audrey Penven
Depth Measurement
Techniques
Microwaves Light
Waves Ultrasonic
Waves
Triangulation Time-
of-Flight
Structured
light
Laser
Scan
Kinect
PrimeSense Asus Xtion
![Page 6: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/6.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 6
Principles of Kinect
640x480 px @ 30 fps
Operation range:
0.8m~3.5m
Depth resolution [1]:
IR Laser projector
λ=830nm
RGB
camera
IR camera
Picture from iFixit
Theoretical random error
Depth resolution
Distance to plane [cm]
Depth image RGB image
![Page 7: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/7.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 7
How Kinect works?
Libraries: OpenNI, libfreenect y Kinect SDK
Picture from PrimeSense
Projected Light Pattern [2] 3D Reconstruction
![Page 8: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/8.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 8
Outline
Depth measurement techniques: Kinect
ROS/Kinect integration
System for robotic visual perception
Interfacing with an object SQL database
ROS vision nodes within HANDLE EU project
![Page 9: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/9.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 9
OpenNI in ROS
Picture from [3]
http://www.ros.org/wiki/
![Page 10: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/10.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 10
openni_kinect
ROS/Kinect integration
openni_tracker openni_camera
openni_launch image_pipeline
vision_opencv
perception_pcl
3D
2D
![Page 11: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/11.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 11
ROS image and OpenCV image
OpenNI:
RGB format
OpenCV:
BGR format
http://www.ros.org/wiki/cv_bridge/Tutorials/UsingCvBridgeToCon
vertBetweenROSImagesAndOpenCVImages
![Page 12: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/12.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 12
ROS/Kinect integration
Start the Kinect driver:
roslaunch openni_launch openni.launch
Reconfigure camera parameters:
rosrun dynamic_reconfigure
reconfigure_gui
Visualize the images:
rosrun rviz rviz
If you have problems with starting rviz, remove files in ~/.rviz/.
Add a PointCloud2 View: XYZRGB point cloud
For topic camera/depth_registered/points
![Page 13: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/13.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 13
ROS/Kinect integration
![Page 14: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/14.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 15
Outline
Depth measurement techniques: Kinect
ROS/Kinect integration
System for robotic visual perception
Interfacing with an object SQL database
ROS vision nodes within HANDLE EU project
![Page 15: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/15.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 16
Calibrating a Kinect to your robot
TF
Hands-on #2
Session
Checkerboard detector
![Page 16: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/16.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 17
ROS Examples
[4]
[5]
[6]
[7]
[8]
[9]
![Page 17: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/17.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 18
System for robotic visual perception
Segmentation, recognition, pose estimation, object
tracking…
ROS stacks:
http://www.ros.org/wiki/iap: tracking and segmenting visual
features
http://www.ros.org/wiki/perception_blort: object recognition and
pose estimation based on detection and tracking nodes.
http://www.ros.org/wiki/tabletop_object_perception: object
segmentation, object recognition and collision environment
PCL and OpenCV.
![Page 18: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/18.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 19
Table detection using RANSAC
Segmentation
Picture from PCL
![Page 19: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/19.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 20
Object recognition and pose estimation
Picture from PCL
![Page 20: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/20.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 21
Outline
Depth measurement techniques: Kinect
ROS/Kinect integration
System for robotic visual perception
Interfacing with an object SQL database
ROS vision nodes within HANDLE EU project
![Page 21: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/21.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 22
Interfacing with a SQL database
Installed:
A PostgreSQL server.
A database: restored from the provided backup file.
Iros_st2_database package: functions for interfacing with a
specific SQL database.
Iros_st2_database_msgs package: ROS API.
Updated model_root:
![Page 22: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/22.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 23
SQL database: schema
Starting point: household_objects_database
www.ros.org/wiki/household%20objects
![Page 23: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/23.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 24
SQL database: usage
Launch the local server:
roslaunch iros_st2_database iros_st2_database_server.launch
Provided ROS services:
iros_st2_hdb/get_list_by_acquisition_method: list of models
available in the database by given an acquisition method
iros_st2_hdb/delete_model: deletes a model
iros_st2_hdb/download_model: gets a 3D mesh
iros_st2_hdb/insert_model: stores a model in the database
iros_st2_hdb/update_model: updates a stored model
![Page 24: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/24.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 25
SQL database: ROS Services
Get list by acquisition method:
rosservice call iros_st2_hdb/get_list_by_acquisition_method
<name>
![Page 25: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/25.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 26
SQL database: ROS Services
Download a mesh from the database:
rosservice call iros_st2_hdb/download_model
<scaled_model_id>
Insert a model in the database, copying the files to a location
specified by the model_root.
rosservice call iros_st2_hdb/insert_model <model_name>
<acquisition_method> <maker> <description> <barcode>
<geometry_filename> <color_image_filename>
E.g.: rosservice call iros_st2_hdb/insert_model camera iros_st2 “” “” “”
/iros_st2_database/database/examples/camera.model/mesh.ply
/iros_st2_database/database/examples/camera.model/view0000/raw/color.
png
![Page 26: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/26.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 27
SQL database: ROS Services
Delete a model from database:
rosservice call iros_st2_hdb/delete_model <scaled_model_id>
Update a model from database:
rosservice call iros_st2_hdb/update_model <original_model_id>
<model_name> <acquisition_method> <maker> <description>
<barcode> <geometry_filename> <color_image_filename>
E.g.: rosservice call iros_st2_hdb/update_model 9623 camera iros_st2
unknown “” “” “” “”
![Page 27: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/27.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 28
Outline
Depth measurement techniques: Kinect
ROS/Kinect integration
System for robotic visual perception
Interfacing with an object SQL database
ROS vision nodes within HANDLE EU project
![Page 28: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/28.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 29
HANDLE: Kinect/robot calibration
Kinect
Camera
Shadow
Arm
Shadow
Hand
ATI
Force/Torque
Sensors
![Page 29: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/29.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 30
HANDLE: reconstruction using a single view
Color and Depth Image
Table-top Object Detector
Voxel filling by extrusion
Consistency Check
3D Object Model
Color-based Model Refinement
Poisson Reconstruction
3D Scene Points
![Page 30: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/30.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 31
HANDLE: reconstruction using a single view
![Page 31: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/31.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 32
HANDLE: Object recognition and pose
Viewpoint Feature Histogram (VFH) descriptor used for matching and recognition of point clouds [10]
Recognition and 6D pose estimation for objects lying on a table:
Database
ICP
![Page 32: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/32.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 33
![Page 33: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/33.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 34
![Page 34: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/34.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 35
![Page 35: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/35.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 36
HANDLE: Object recognition and pose
![Page 36: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/36.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 37
References
[1] K. Khoshelham and S. O. Elberink. Accuracy and Resolution of Kinect
Depth Data for Indoor Mapping Applications. Sensors, 12(2):1437--1454, 2012.
[2] www.ros.org/wiki/kinect_calibration/technical
[3] http://openni.org/Documentation/ProgrammerGuide.html
[4] www.ros.org/wiki/camera_pose_toolkits/Tutorials/Add_Kinect_to_PR2
[5]
www.ros.org/wiki/openni/Contests/ROS%203D/Automatic%20Calibration%20of
%20Extrinsic%20Parameters
[6]
www.ros.org/wiki/turtlebot_kinect_arm_calibration/Tutorials/CalibratingKinectTo
TurtleBotArm
[7] www.ros.org/wiki/pr2_calibration/Tutorials/NullChainKinectURDFCalibration
[8] www.ros.org/wiki/pr2_calibration/Tutorials/KinectURDFCalibration
![Page 37: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/37.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 38
References
[9] www.ros.org/wiki/camera_pose_toolkits/Tutorials/Calibrate_camera_pose_to
_map
[10] R. Rusu, G. Bradski, R. Thibaux and J. Hsu. Fast 3d recognition
and pose using the viewpoint feature histogram. In Proc. of IROS 2010, pages
2155-2162, 2010.
![Page 38: IROS 2012 Handling ROS tutorial Vilamoura, October 7th 2012 http ...](https://reader031.fdocuments.us/reader031/viewer/2022022000/58a2d8e11a28abaa338b78d7/html5/thumbnails/38.jpg)
Tutorial ST2 IROS 2012 | Hands-on #1 | Silvia Rodríguez-Jiménez 39
Thank you!
Questions/comments?