Classification of Indoor Point Clouds using Multiviews€¦ · Vladeta Stojanovic, Matthias Trapp,...

Classification of Indoor Point Clouds using Multiviews

Vladeta Stojanovic, Matthias Trapp, Rico Richter and Jürgen DöllnerComputer Graphics Systems Group, Hasso Plattner Institute, University of Potsdam, Germany

Web’3D 2019, 19-21 July Los Angeles, USA

Introduction

● Point clouds present a “snapshot” of the physical

environment.

● This makes them useful for base-data for Digital

Twin representations or as-is Building Information

Models (BIMs).

● However, captured point clouds do not contain any

semantics by default.

● In the realm of Real Estate 4.0, the classification of

base-data such as point clouds is an important task.

2Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.

http://www.hpi3d.de

PhD Research Topics

• Focus on automated semantic

enrichment of point cloud data and

generation of basis data for as-is BIM

and Digital Twin representations

• Implementation of these technologies

as a web-based application using a

services-oriented methodology

• Integration with current FM and

Industry 4.0 practices and standards


http://www.hpi3d.de

Motivation and Research Contribution

● 3D point clouds can represent the physical features of indoor

environments, such as typical office environments.

● Point clouds may feature either RGB or intensity color-mapped

representations, but semantics are not explicitly contained in the

data.

● Manually segmenting 3D point clouds and assigning semantics

tends to be a time consuming, error prone, and costly process.

● Our approach classifies 3D point clouds using a retrained CNN

based on image analysis for entropy-selected 3D views that

allows for object segmentation and adding of semantics.


http://www.hpi3d.de

Approach Overview

● We have implemented the presented cluster, viewpoint entropy selection, and point

cloud visualization methods using Web3D technologies.

● A retrained CNN used to classify entropy-selected multiview images of point cluster.

● These multiview images are selected using an entropy function.

● We classify cluster and classify the point cloud automatically using a service-oriented

implementation (client/server model).


http://www.hpi3d.de

Approach Overview - Process Flowchart


http://www.hpi3d.de

Clustering Methods● Prior to entropy computation and multiview

classification, we must cluster the given point

cloud.

● The point cloud first has any homogeneous

structural features (e.g., walls, floor and

ceilings), removed using a region growing

algorithm. Normals are also computed for each

point.

● We use either the k-Means or DBSCAN

clustering methods to further segment the

furniture clusters.


http://www.hpi3d.de

Multiview Selection and Image Generation Algorithm

● Randomly sample a number of points around a

bounding sphere of each cluster.

● From each random vertex on the bounding

sphere, a ray is projected inwards towards the

point cloud cluster.

● For each point normal in cluster, we check if the

normal vector is parallel within a threshold to

the bounding sphere vertex vector.

● If true, the given bounding sphere vertex is used

as a multiview camera position and direction.


http://www.hpi3d.de

Multiview Selection and Image Generation

9

Bounding sphere vertices are selected as multiview camera positions and directions, based on the parallelism entropy measure between the inverted bounding vertex normal and the point cluster normal vectors.

Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.

http://www.hpi3d.de

Classification Implementation

● Classification of multiview images is accomplished using two retrained 2D CNNs (Inception V3

using Tensorflow for Python).

● The first CNN is re-trained using three categories featuring photographs of common office

furniture (chairs, sofas and tables), while the second is re-trained using only photos of chairs

and tables.

● A total of 9759 RGB images for three different categories was used. Each image was resized to

300 x 300 pixels while maintaining the aspect ratio of the image.

● The CNNs were re-trained using 4000 learning steps, with a learning rate of 0.01, with a

predicted classification accuracy of 92.9% and 94.5%.


http://www.hpi3d.de

Service Oriented System Design

● The clustering, entropy viewpoint selection and classification methods are all

implemented as software components within a service-oriented system design.

● We make use of Node.js to create a echo-style server to handle the processing of

multiview images and classification server-side.

● We perform the clustering, 3D point cloud and result visualization client-side (using

JavaScript, with Three.js as the main visualization framework component).


http://www.hpi3d.de

Service-Oriented System Design


http://www.hpi3d.de

Testing Procedure

● 64 different test sets. 32 tests using the Stanford dataset [Armeni et al.

2017] with 16 different scenes, and 32 tests based on 16 synthetic scenes

that we constructed.

● Additional synthetic scenes of varying complexity from furniture point

clusters captured using a mobile phone scanning device, and using the

RGB-D Object Dataset from University of Washington [Lai et al. 2011].

● The Stanford dataset was chosen as it features a high number of

cluttered items in the desired classification categories (e.g., chairs, tables

and sofas).

● For each of the two datasets, we tested each of the 16 scenes using

either k-Means or DBSCAN clustering.


http://www.hpi3d.de

Experimental Results

● We present experimental results using the entropy-based multiview selection and

classification approach.

● We tested our approach using both k-Means and DBSCAN clustering methods.

● Test data included both real-life and synthetically generated datasets.

● Red clusters indicate chair object, blue clusters indicate sofa object, and green

clusters indicate table object classifications.


http://www.hpi3d.de


15

(a) Original (b) k-Means Clustering (c) DBSCAN Clustering

(d) Original (e) k-Means Clustering (f) DBSCAN Clustering


http://www.hpi3d.de


16

(a) Original (b) k-Means Clustering (c) DBSCAN Clustering

(d) Original (e) k-Means Clustering (f) DBSCAN Clustering


http://www.hpi3d.de

Experimental Results - Previous Approach

17

(a) [Stojanovic et al. 2018] Octree-based multiview classification, (b) [Stojanovic et al. 2019] k-Means Clustering and (c) [Stojanovic et al. 2019] DBSCAN Clustering.

(a) (b) (c)


http://www.hpi3d.de

Discussion - Approach

● The entropy-based viewpoint selection method generates multiview images featuring perspective projection

of 3D point clusters.

● Main drawback: using only highly parallel normal vectors as an entropy measure is that for flat surfaces only

directly top-down views will mostly be included in the entropy measure.

● We generate an average of 11 images for a given point cluster using the default parameters.

● This allows us to classify a scene in a few minutes on average using the implement client-server

web-application model.

● The CNN was re-trained using actual photographs of furniture items (chairs, tables, and sofas), thus it is more

general.


http://www.hpi3d.de

Discussion - Results

● The highest probability of correct classification is achieved if the DBSCAN clustering method is used for

clustering.

● We observed that the DBSCAN algorithm generally provides better clustering results, if the point set has

uniform density.

● The DBSCAN algorithm is also faster at computing clusters than the k-Means algorithm.

● The k-Means algorithm is better suited for clustering scenes with lots of clutter and lack of spatial divisions.

● The probability of correct classification largely varies on the types of scenes and the visual style of the 3D

point clusters.

● Method provides a better results than the previous Octree-based approach [Stojanovic et al. 2018].


http://www.hpi3d.de

Conclusions and Outlook

● The experimental results show that our approach is mostly accurate and able to quickly classify more than

half of the objects in the tests scenes correctly.

● We have implemented our approach as a service-based software component, which can simplify its

integration in complex and larger service-oriented enterprise systems.

● For future work, we plan to evaluate spatial inference methods to better evaluate the classification results

(e.g., bounding volume comparisons), increase the classification categories (e.g. different types of chairs,

sofas and tables), and to evaluate the OPTICS clustering algorithm.


http://www.hpi3d.de

Key ReferencesVladeta Stojanovic, Matthias Trapp, Rico Richter, and Jürgen Döllner. 2018. A Service-Oriented Approach for Classifying 3D Points Clouds by Example of Office Furniture

Classification. In Proceedings of the 23rd International ACM Conference on 3D Web Technology. ACM, 2.

Vladeta Stojanovic, Matthias Trapp, Rico Richter, Benjamin Hagedorn, and Jürgen Döllner. 2018. Towards The Generation of Digital Twins for Facility Management based on 3D

Point Clouds. In Proceeding of the 34th Annual ARCOM Conference. 270–279.

Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3D Semantic Parsing of Large-Scale Indoor Spaces. In Proceedings of

the IEEE Conference on Computer Vision and Pattern Recognition. 1534–1543.

Helin Dutagaci, Chun Pan Cheung, and Afzal Godil. 2010. A Benchmark for Best View Selection of 3D Objects. In Proceedings of the ACM Workshop on 3D Object Retrieval. ACM,

45–50.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al.1996. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Kdd, Vol. 96.

226–231.

Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional Neural Networks for 3D Shape Recognition. In Proceedings of theIEEE

international conference on computer vision. 945–953.

Pere-Pau Vázquez, Miquel Feixas, Mateu Sbert, and Wolfgang Heidrich. 2001. View-point Selection using Viewpoint Entropy. In VMV, Vol. 1. 273–280.

Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox. 2011. A Large-Scale Hierarchical Multi-view RGB-D Object Dataset. In Robotics and Automation (ICRA), 2011 IEEEInternational

Conference on. IEEE, 1817–1824.


http://www.hpi3d.de

Classification of Indoor Point Clouds using Multiviews€¦ · Vladeta Stojanovic, Matthias Trapp,...

Documents

Transcript of Classification of Indoor Point Clouds using Multiviews€¦ · Vladeta Stojanovic, Matthias Trapp,...