http://www.lsr.ei.tum.de

Segmentation and Classificationof 3D Range Data

Klaas Klasing

Institute of Automatic Control EngineeringTechnische Universität München

kk@tum.de

9-Oct-2009, Universidad de Zaragoza

2

Agenda

1. Introduction

2. Surface-based Segmentation

3. Explaining Occlusions

4. Results

5. Classification

6. Summary & Outlook

• Mobile autonomous robotics is gradually moving from 2D navigation towards 3D mapping, interpretation and manipulation– Household robotics:

– Intelligent behavior in domestic setting requiresunderstanding of 3D environments

3

Motivation

1. Introduction

CoTeSys@TUM UnivTokio/Toyota CMU/Intel IAS@TUM

• Actuated 2D laser range finders: • Real 3D Lidar:

• Civil engineering:

4

Active 3D Sensing Hardware

1. Introduction

SegBot@Stanford

Velodyne

Leica

ActivMedia UniFreiburg UniHannover

IAS@TUM DualArm@TUM

• Time-of-flight cameras:

• Desktop scanning:

1. Introduction 5

Range Images vs. Point Clouds

• Range images popular in the vision communityz = f(x,y)powerful algorithms for segmentationproblems with projection, depth resolutionreal 3D scenes must be fused from several 2.5D images

• 3D point cloudreal 3D coordinates (no z-dependency)continuously growing or windowedcan contain several registered scanssuitable for sweeping 2D range sensors» Point cloud

1. Introduction 6

Ultimate Goal for Domestic Environments

• Quickly and robustly detect objects in 3D environments– Point clouds may be sparse, noisy, and may contain occlusions– Scanning and interpretation should take only seconds

» Simulator » Point cloud

7

Problem Statement

• Consider points ,represented by a data matrix

, where .

• We‘d like to partition the set of points into segments ,such that

are disjoint regions of smooth curvature change.

2. Surface-based Segmentation

2. Surface-based Segmentation 8

Segmentation of 3D Range Data

• A surprisingly large number of works considers only a few, densely sampled and unoccluded objects in range images:

[HJ87], [BJ88], [HJJ96], [B00], [MLM01], [DF02], [BG06]

• Segmentation approaches include– Edge-based [B00], [SD01]– Curvature-based [BJ88], [HJJ96], [PBJ98]– Scanline grouping [JB94], [KMK03]– Geometric primitives [MLM01], [BG06], [SWK07]– Smooth regions [RHV06], [RMB08]

9

Normal Vectors

• The two most important features for surface-based segmentation are normal vectors and curvature values

• For each point a normal vector is estimatedfrom a set of neighboring points .

• Define a neighborhood matrix and an augmented neighborhood matrix .

• A plethora of different approaches for estimating normal vectorsexists [HJ87], [HDD92], [YL99], [GKS00], [HM01], [WTH01], [OF05] ...

For a comparison see Klasing et. al, Comparison of Surface Normal Estimation Methods for Range Sensing Applications, ICRA 2009.

2. Surface-based Segmentation

10

Approaches for Normal Vector Estimation

2. Surface-based Segmentation

Optimization-based Averaging

(a) plane fit over neighborhood(b) angle maximization

(c) Average over vectorsof neighbor triangles

(SVD/PCA)

• The neighborhood of each point is defined by a neighborhoodgraph with vertices for every pointand edges .

• Two types of graph are used for normal vector estimation:

11

Nearest Neighbors

2. Surface-based Segmentation

kNN graph• fixed # of neighbors• overlapping simplices

Delaunay tessellation• variable # of neighbors• no overlapping simplices

12

‘Best‘ Method

2. Surface-based Segmentation

• PlanePCA [HDD92]

Minimize the variance of the plane fitting error:

where contains the mean of the points in each row.

• Equivalent to a PCA of . • Principal components = orthonormal basis explaining variance.• Variance ~ eigenvalues of covariance matrix• Standard deviation ~ singular values of data matrix

• Other aspects not addressed here

13

Curvature Estimation

2. Surface-based Segmentation

• Principal curvatures (Euler)• Minimal and maximum

curvature in orthogonal planek1 and k2

• Gaussian curvature: k1·k2

• Mean curvature: (k1·k2)/2• Curvature estimated from

paraboloid or cuboid fit [MSR07]

• estimates too noisy oversegmentation [HJJ96], [PBJ98]

• Instead, use surface variation:[PGK02], [GHS08]

wikipedia.org

14

Visualization with Curvature

2. Surface-based Segmentation

Points grayscale-colored by curvature value

15

Smooth Region Segmentation

2. Surface-based Segmentation

• Region growing with smoothness constraintproposed in [RHV06]

• Decent results for segmentation of industrial scenes,but• segmentation results data dependent• sensitive to individual outliers:

• curvature criterion not necessary!

[RHV06]

16

Efficient Segmentation

2. Surface-based Segmentation

1. Define two parameters:: The maximum angle that two neighboring normals may

span to belong to the same segment.: The maximum allowed radius of a point neighborhood.

2. For each point calculate the search radius , find the neighbors within this radius and select the ones whose normal vector spans an angle less than with this point’s normal vector.

3. If any of the selected neighbors has been assigned to a segment, assign all selected points to that segment. Otherwise create a new segment and assign all selected points to it. Merge segments if necessary.

17

Occlusions

3. Explaining Occlusions

• Occlusions cause oversegmentation that can not be remedied by the segmentation algorithm.

• A good segmentation toolchain should be able to explain (basic) occlusions and fix them.

18

Existing Approaches & Plausibility

• Not many publications on this topic.• Existing methods can be divided into

– model-free approaches (hole filling) [DF02], [WO03], [BPB06]

– model-based approaches (object recognition) [B00], [BI00], [TB07]

• Important: Reason about plausibility of occlusion [DF02]

3. Explaining Occlusions

19

Merging Oversegmented Surfaces

• Identify candidate surfaces by 4 distance measures

• Identify pairs of boundary points on each segment:

3. Explaining Occlusions

20

Checking for Occlusion

• Check connecting lines for occlusion– Create an omnidirectional range image (ORI) for all points

observed from one view point:

– For each boundary linecheck the correspondingline in the image for occlusion

3. Explaining Occlusions

21

Results - Simulation

4. Results

22

Experiment

4. Results

23

Results - Experiment

4. Results

24

Discussion

4. Results

• The bulk of the processing time is spent on normal vector estimation– Good normal estimates crucial for

high segmentation quality– Runtime complexity linear in – Possible solution: parallelize estimation

• Current limitation:– View points required to explain occlusions

Extend this to view point trajectories

25

Classification

5. Classification

• Goal: use the segmented surfaces as an abstract representation for detecting objects

• Segment features:– Centroid, normal vector, surface variation, axis-aligned

bounding box, convex hull etc.

26

Possible Methods/Frameworks

5. Classification

• Train primitive classifiers for partsSVMs, Decision Trees, NeuralNets etc. trained from segment features

• Find structure in neighboring classified segments– convert neighborhood relationship to graph– complete/sub-graph matching– part-based approaches [RDR95], [GHS08]

• Train with IKEA CAD models Recognition in real data sets?

BE

RN

HA

RD

27

Conclusion

6. Summary & Outlook

• Efficient segmentation toolchain• Applicable to

• sparse and noisy point clouds• variable density data, registered views, partial occlusions

• Segmentation time on the order of the scan time• Segments provide informative features for classification

• Central question: most efficient/robust framework for classification?

This work is supported in part within the DFG excellence initiative research cluster Cognition for Technical Systems –CoTeSys, see also www.cotesys.org.

28

References[BG06] A. Bab-Hadiashar and N. Gheissari. Range image segmentation using surface selection criterion. IEEE Transactions on Image

Processing, 15(7):2006–2018, 2006.[B00] R. Benlamri. Range image segmentation of scenes with occluded curved objects. Pattern Recognition Letters, 21(12):1051–1060,

2000.[BJ88] P. J. Besl and R. C. Jain. Segmentation through variable-order surface fitting. IEEE Transaction on Pattern Analysis and Machine

Intelligence, 10(2):167–192, 1988. [BI00] M. Boshra and M. A. Ismail. Recognition of occluded polyhedra from range images. Pattern Recognition, 33(8):1351–1367, 2000.[BPB06] J. Branch, F. Prieto, and P. Boulanger. Automatic hole-filling of triangular meshes using local radial basis function. In Proceedings of

the Third International Symposium on 3D Data Processing, Visualization, and Transmission, pages 727–734, Washington, DC, USA, 2006. IEEE Computer Society.

[DF02] F. Dell’Acqua and R. Fisher. Reconstruction of planar surfaces behind occlusions in range images. IEEE Transaction on Pattern Analysis and Machine Intelligence, 24(4):569–575, 2002.

[GHS08] S. Gächter, A. Harati, and R. Siegwart. Structure verification toward object classification using a range camera. In Proc. of the 10th International Conference on Intelligent Autonomous Systems (IAS), 2008.

[GKS00] M. Gopi, S. Krishnan, and C. T. Silva. Surface reconstruction based on lower dimensional localized delaunay triangulation. Computer Graphics Forum, 19(3):467–478, 2000.

[HJ87] R. Hoffman and A. K. Jain. Segmentation and classification of range images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(5):608–620, 1987.

[HJJ96] A. Hoover, G. Jean-Baptiste, X. Jiang, P. J. Flynn, H. Bunke, D. B. Goldgof, K. Bowyer, D. W. Eggert, A. Fitzgibbon, and R. B. Fisher. An experimental comparison of range image segmentation algorithms. IEEE Transaction on Pattern Analysis and Machine Intelligence, 18(7):673–689, 1996.

[HDD92] H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, and W. Stuetzle. Surface reconstruction from unorganized points. Computer Graphics, 26(2):71–78, 1992.

[HM01] J. Huang and C.-H. Menq. Automatic data segmentation for geometric feature extraction from unorganized 3-d coordinate points. IEEE Transactions on Robotics and Automation, 17:268–279, 2001.

[KB09a] K. Klasing, D. Althoff, D. Wollherr, and M. Buss. Comparison of surface normal estimation methods for range sensing applications. In Proceedings of the IEEE International Conference on Robotics and Automation, 2009.

[KB09b] K. Klasing, D. Wollherr, and M. Buss. Realtime segmentation of range data using continuous nearest neighbors. In Proceedings of the IEEE International Conference on Robotics and Automation, 2009.

29

References[MSR07] E. Magid, O. Soldea, and E. Rivlin. A comparison of gaussian and mean curvature estimation methods on triangular meshes of range

image data. Computer Vision and Image Understanding, 107(3):139–159, 2007.[MLM01] D. Marshall, G. Lukacs, and R. Martin. Robust segmentation of primitives from range data in the presence of geometric degeneracy.

IEEE Transaction on Pattern Analysis and Machine Intelligence, 23(3):304–314, 2001.[OF05] D. OuYang and H.-Y. Feng. On the normal vector estimation for point cloud data from smooth surfaces. Computer-Aided Design,

37(10):1071–1079, 2005.[PGK02] M. Pauly, M. Gross, and L. P. Kobbelt. Efficient simplification of pointsampled surfaces. In Proceedings of the Conference on

Visualization, pages 163–170, Washington, DC, USA, 2002.[PBJ98] M. W. Powell, K. W. Bowyer, X. Jiang, and H. Bunke. Comparing curved-surface range image segmenters. In Proceedings of the

Sixth International Conference on Computer Vision, page 286, Washington,DC, USA, 1998.[RHV06] T. Rabbani, F. van den Heuvel, and G. Vosselmann. Segmentation of point clouds using smoothness constraint. International

Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 36(5):248–253, September 2006.[RMB08] R.B. Rusu, Z.C. Marton, N. Blodow, M. Dolha, and M. Beetz. Towards 3D Point cloud based object maps for household

environments. Robotics and Autonomous Systems, 56:927-941, 2008.[JB94] X. Jiang and H. Bunke, Fast segmentation of range images into planar regions by scan line grouping. Machine Vision and

Applications, 7:115-122, Springer-Verlag New York, Inc., 1994.[KMK03] I. Khalifa, M. Moussa, and M. Kamel. Range image segmentation using local approximation of scan lines with application to CAD

model acquisition. Machine Vision and Applications, 13:263-274, Springer-Verlag New York, Inc., 2003.[RDR95] E. Rivlin, S. J. Dickinson, and A. Rosenfeld. Recognition by functional parts. Computer Vision and Image Understanding, 62(2):164–

176, 1995.[TB07] R. Triebel and W. Burgard. Recovering the shape of objects in 3d point clouds with partial occlusions. In Proceedings of the 6th

International Conference on Field and Service Robotics, 2007.[WTH01] C. Wang, H. Tanahashi, H. Hirayu, Y. Niwa, and K. Yamamoto. Comparison of local plane fitting methods for range data. In

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 663–669, 2001.[WO03] J. Wang and M. M. Oliveira. A hole-filling strategy for reconstruction of smooth surfaces in range images. In Proceedings of the 16th

Brazilian Symposium on Computer Graphics and Image Processing, page 11, Los Alamitos, CA, USA, 2003.[YL99] M. Yang and E. Lee. Segmentation of measured point data using a parametric quadric surface approximation. Computer-Aided

Design, 31(7):449–457, 1999.[SD01] A. Sappa, and M. Devy. Fast Range Image Segmentation by an Edge Detection Strategy. In Proceedings of the International

Conference on 3D Digital Imaging and Modeling, IEEE Computer Society, 2001.