Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)
description
Transcript of Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)
![Page 1: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/1.jpg)
Bryan WillimonClemson University 2013
Committee:Stan Birchfield (committee chair)
Ian Walker (co-advisor)Adam Hoover
Damon Woodard
Sensing Highly Non-Rigid Objects with RGBD Sensors for Robotic Systems
![Page 2: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/2.jpg)
Robotic systems are divided into two groups• Industrial
• Recycling• Mail• Food• Laundry
• Domestic• Outdoor
• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning
• Indoor• Floor cleaning• Pet assistance• Picking up objects• Laundry
![Page 3: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/3.jpg)
Why Domestic Robots?• robotic systems are able to accomplish
chores around the house• Domestic robotic systems are getting more
attention in the news
• Domestic• Outdoor
• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning
• Indoor• Floor cleaning• Pet assistance• Picking up objects• Laundry
![Page 4: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/4.jpg)
• Domestic• Outdoor
• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning
• Indoor• Floor cleaning• Pet assistance• Picking up objects• L
Laundry is an Important Problem
Laundry
![Page 5: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/5.jpg)
Laundry is an Important Problem• Industries and research institutes are making
attempts to solve the process
PR2 at UC Berkeley
NEDO Laundry Handling System
Laundry
![Page 6: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/6.jpg)
Difficulties in the Laundry Problem• What are current problems that make a laundry
system difficult to automate?
• Highly deformable objects• Infinitely large number of configurations• Lots of possible grasp points
• The laundry problem is still in a research stage
![Page 7: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/7.jpg)
Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.
Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.
Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.
Research PathFocus of thesis
![Page 8: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/8.jpg)
Kita et al. use a humanoid system to recognize the state of clothes using a cloth model with 22 out of 27 attempts correctly classified
Previous Work on Classification for Robotics
Cusumano-Towner et al. were aimed at classifying the category of seven articles with a success rate of 20 out of 30 trials
Our system does not use predefined cloth models or dual manipulators
![Page 9: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/9.jpg)
IsolationGraph-based SegmentationStereo MatchingDetermining Grasp Point
Classification Hanging Position
Binary Silhouettes Visual-based shape and
appearance informationLying Position
Low level features Characteristics Selection Masks
Classification Framework
![Page 10: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/10.jpg)
Classification in a …
Hanging Position Lying Position
![Page 11: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/11.jpg)
Experimental Results With 6 categories, 5 items per category, and 20 images per item,
the dataset collected by the extraction / isolation procedure consists of 600 images
This dataset was labeled in a supervised learning manner so that the corresponding category of each image was known
Two experiments were conducted:
1) Extraction and isolation process
2) Interaction vs. Non-interaction
![Page 12: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/12.jpg)
Extraction and isolation process:
The image taken by one of the downward-facing stereo cameras
The result of graph-based segmentation
The object found along with its grasp point (red dot)
The image taken by the side-facing camera
The binary silhouettes of the front and side views of the isolated object.
Experimental Results
![Page 13: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/13.jpg)
Interaction vs. Non-interaction:
The process of interacting with each article of clothing provided the system with multiple views using various grasp locations, allowing the system to collect 20 total images of each object.
Experimental Results
Two articles were compared by examining the 400 match scores between their pairs of images (20 images per article).
![Page 14: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/14.jpg)
Classification in a …
Hanging Position Lying Position
![Page 15: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/15.jpg)
Classification in a Lying Position
![Page 16: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/16.jpg)
Experimental Results The proposed L-C-S-H approach is applied to a laundry
scenario.
Two different scenarios involved using: 3 categories {shirts, dresses, socks} 7 categories {shirts, dresses, socks, cloths, pants, shorts,
jackets}
Each scenario is run through three experiments:
1) Baseline System (L-H)
2) L-C-H approach
3) L-C-S-H approach
![Page 17: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/17.jpg)
Experimental Results Scenario 1 → 3 categories {shirts, dresses, socks}
Local Global Both0.00
10.0020.0030.0040.0050.0060.0070.0080.0090.00
100.00
BaselineL-C-HL-C-S-H
![Page 18: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/18.jpg)
Experimental Results Scenario 2 → 7 categories {shirts, dresses, socks, cloths, pants,
shorts, jackets}
Local Global Both0.005.00
10.0015.0020.0025.0030.0035.0040.00
BaselineL-C-HL-C-S-H
![Page 19: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/19.jpg)
Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.
Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.
Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.
Research PathFocus of thesis
![Page 20: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/20.jpg)
Cusano-Towner et al. were aimed at flattening a piece of crumpled clothing by implementing a disambiguation phase and a reconfiguration phase.
Previous Work on Unfolding for Robotics
Our system does not use a dual manipulator or a predefined model of the clothing
![Page 21: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/21.jpg)
First PhaseRemove any major wrinkles
and / or folds Pulling the cloth at individual
corners every d degrees
Second PhaseDefine cloth modelCalculate various components
needed for the cloth model
Model to Unfold Laundry into a Flat Canonical Position
![Page 22: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/22.jpg)
Experimental Results The proposed approach was applied to a 3D simulated cloth to
determine the results of the first and second phase.
Two experiments were conducted:
1) Experimental Test of Algorithm
2) Test to Fully Flatten the Cloth
![Page 23: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/23.jpg)
Experimental Test of Algorithm :
This experiment tested the first phase of the proposed algorithm and monitored the process from eight iterations of pulling the cloth.
The models continually change configurations in a manner that flattens and unfolds larger areas of the cloth as the iterations increase.
Eventually, the cloth is mostly flattened out to a more recognizable shape in the final iteration.
Experimental Results
![Page 24: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/24.jpg)
Test to Fully Flatten the Cloth :
This experiment tested the proposed algorithm in determining if this approach would completely flatten a piece of clothing.
The test used the first and second phase of the algorithm to grasp the cloth at various locations and moved the cloth at various orientations until the cloth obtained a flattened percentage greater than 95%.
Experimental Results
![Page 25: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/25.jpg)
Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.
Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.
Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.
Research PathFocus of thesis
![Page 26: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/26.jpg)
Previous Work on Pose Estimation for Robotics
• Elbrechter et al. (IROS 2011) use a soft-body-physics model with visual tracking to manipulate a piece of paper.
• Bersch et al. (IROS 2011) describe a method to bring a T-shirt into a desired configuration by alternately grasping the item with two hands, using a fold detection algorithm.
Both approaches require predefined fiducial markers.
![Page 27: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/27.jpg)
The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:
Energy Minimization Approach
Correspondence term
Depth term
Boundary term
Smoothness term
![Page 28: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/28.jpg)
The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:
Energy Minimization Approach
![Page 29: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/29.jpg)
Energy Minimization Approach
Smoothness term
Correspondence term
Depth term
Boundary term
![Page 30: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/30.jpg)
Smoothness term
Energy Minimization Approach
![Page 31: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/31.jpg)
Energy Minimization ApproachSmoothness term
![Page 32: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/32.jpg)
Correspondence term
Energy Minimization Approach
![Page 33: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/33.jpg)
Depth term
Energy Minimization Approach
Front View Top View
![Page 34: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/34.jpg)
Boundary term
Energy Minimization Approach
![Page 35: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/35.jpg)
Energy Minimization ApproachMinimize energy equation
![Page 36: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/36.jpg)
Segmentation and InitializationForeground / background segmentation
![Page 37: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/37.jpg)
Segmentation and InitializationMesh Model Generator
![Page 38: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/38.jpg)
Segmentation and InitializationReinitialization
![Page 39: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/39.jpg)
Experimental Results
• We captured RGBD video sequences of shirts, pants, shorts, and posters to test our proposed method’s ability to handle different non-rigid objects in a variety of scenarios.
Four experiments were conducted:
1) Estimating pose of clothing
2) Estimating pose of posters
3) Reinitializing mesh after in-plane rotation
4) Quantitative Results
![Page 40: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/40.jpg)
Experimental Results
Shirt partially occluded
Shirt moving in-plane and out-of-plane
Shirt changing scale
Shirt translating from side to side
Shirt moving in-plane
• Estimating pose of clothing•7 shirts•1 pair of pants•2 pairs of shorts
![Page 41: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/41.jpg)
Experimental Results
• Estimating pose of posters
Poster with little texture moving out-of-plane
Poster with a lot of texture moving out-of-plane
![Page 42: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/42.jpg)
Experimental Results
• Reinitializing mesh after in-plane rotation
![Page 43: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/43.jpg)
Experimental Results
• Quantitative Results
![Page 44: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/44.jpg)
Conclusion Clothing Classification
Extraction / Isolation A novel approach in which a pile of laundry is sifted by an
autonomous robot system in order to separate each item. Hanging position
Using interaction to provide multiple views of an object and capture more visual data
The results show that, on average, classification rates using robot interaction are 59% higher than those that do not use interaction.
Lying Position Multi-layer approach involving a mixture of global and local
features Characteristics and selection masks achieve, on average, an
improvement of 27.47% for three categories 17.90% for four categories 10.35% for seven categories
![Page 45: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/45.jpg)
Conclusion Unfolding Clothing
An approach to interactive perception in which a piece of laundry is flattened out into a canonical position by pulling at various locations of the cloth.
The algorithm is shown to flatten a simulated cloth by 95.57% of its total area
Pose Estimation A new and novel algorithm that estimates the 3D configuration of a
deformable object through an RGBD video sequence An energy model is used to create a non-linear energy function and the
information is computed using a semi-implicit scheme Energy terms
Smoothness Feature point correspondence Depth Boundary
![Page 46: Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)](https://reader035.fdocuments.us/reader035/viewer/2022062813/5681669e550346895dda8ad1/html5/thumbnails/46.jpg)
References
Copies of: Publications Code Datasets Videos
are located at
www.bryanwillimon.com