Investigation of indoor and outdoor performance of two portable
mobile mapping systems
Erica Nocerinoa*, Fabio Mennaa, Fabio Remondinoa, Isabella Toschia,
Pablo Rodríguez-Gonzálvezb,c
a 3D Optical Metrology unit, Bruno Kessler Foundation (FBK), 38123 Trento, Italy
Email: <nocerino><fmenna><remondino><toschi>@fbk.eu, Web: http://3dom.fbk.eu
b TIDOP Research Group, Higher Polytechnic School of Ávila, University of Salamanca, Hornos
Caleros, 50, 05003, Ávila, Spain. Email: [email protected]
c Dept. of Mining Technology, Topography and Structures, University of León, Avda. Astorga, s/n,
24401, Ponferrada, León, Spain. Email: [email protected]
ABSTRACT
The paper investigates the performances of two portable mobile mapping systems (MMSs), the handheld GeoSLAM
ZEB-REVO and Leica Pegasus:Backpack, in two typical user-case scenarios: an indoor two-floors building and an
outdoor open city square. The indoor experiment is characterized by smooth and homogenous surfaces and reference
measurements are acquired with a time-of-flight (ToF) phase-shift laser scanner. The noise of the two MMSs is
estimated through the fitting of geometric primitives on simple constructive elements, such as horizontal and vertical
planes and cylindrical columns. Length measurement errors on different distances measured on the acquired point clouds
are also reported. The outdoor tests are compared against a MMSs mounted on a car and a robust statistical analysis,
entailing the estimation of both standard Gaussian and non-parametric estimators, is presented to assess the accuracy
potential of both portable systems.
Keywords: Mobile mapping system, backpack, handheld, accuracy, robust statistical analysis, length measurement error,
noise
1. INTRODUCTION
The ability of acquiring and recording precise, dense and geo-referenced 3D information is a constant request for a vast
variety of applications, ranging from civil engineering and construction to cultural heritage, from environment to
industry, etc. The most effective way to satisfy this need is represented by mobile mapping systems (MMSs), instruments
able to acquire 3D information on-the-move, using a moving platform. A MMS is a ‘compound’ system consisting of
three main components1: mapping sensors for the acquisition of 3D/2D data (point coordinates and/or images), a
positioning and navigation unit for spatial referencing, and a time referencing unit operating as central system for data
synchronization and integration.
Since their early development in the late 1980s, MMSs have been progressively improved in order to provide
increasingly more precise and denser data, acquired in shorter time. Besides the progresses in optical sensors, one of the
key advances in MMS is related to spatial referencing technology. While the very early applications were restricted to
environments where the sensor positions were computed using ground control2, thanks to advantages in satellite and
inertial technology, today spatial referencing is commonly possible in previously unknown and undiscovered places.
Also, the miniaturization and cost reduction of components have played a fundamental role in the spread of MMS,
allowing for more and more flexible, portable and low-cost systems.
* [email protected]; phone +39 0461 314507; http://3dom.fbk.eu
a)
b)
c)
d)
Figure 1. The devices under investigation: the GeoSLAM ZEB-REVO (a and b) and Leica Pegasus:Backpack (c and d).
The majority of the MMSs rely on light detection and ranging (LiDAR) sensors as 3D mapping unit, but are usually
equipped also with cameras for retrieving color information of the scene. Traditionally, MMSs are fitted into vehicles,
like vans, cars, planes, boats, etc., which represent the optimal choice for recording vast areas, but hardily permit the
acquisition of narrow passages and are ineffective for indoor scenarios. A more efficient alternative for 3D mapping in
complex and indoor scenarios, characterized by rough terrain, small obstacles, stairs, entails robotic platforms, which are
however still limited to the research domain3,4,5. More widespread are MMSs designed and adapted to be easily carried or
worn by persons walking through and, consequently, mapping the environment of interest. Nowadays, popular solutions
available on the market and among research laboratories are:
1. man-portable MMS backpacks: these systems usually feature a conventional positioning and navigation unit
integrating GNSS (global navigation satellite system) and IMU (inertial measurement unit) sensors (Akhka R26;
Leica Backpack:Pagasus7; ROBIN8); however, solutions with only IMU or even without positioning and
navigation components exist9 (HERON10). Some systems are also equipped with cameras providing panoramic
spherical video or still imagery (UltraCam Panther11);
2. handheld MMSs: the main difference from backpack solutions is that the user holds the scanning device in
hand. These systems are usually equipped with IMU based navigation unit (no GNSS), 2D laser profilometer
(iMS2D12, ZEB113, ZEB-REVO14), and may also feature a very wide angle camera (PX-8015, ZEB-CAM16);
3. trolley MMSs: in this configuration, mainly designed for indoor mapping applications the different sensors
(lidar, IMU, cameras, etc.) are fitted in a cart (iMS 3D17, Timms18).
Previous works presented accuracy evaluation of vehicle-based MMSs: (i) using an established urban test field focusing
on planimetric and elevation errors19, (ii) adopting reference values from an existing 3D city model20, (iii) in terms of
target representation21, (iv) through robust statistical assessment with respect to photogrammetry and terrestrial laser
scanner (TLS) data23,24. Backpack MMSs were also tested: (i) in outdoor scenarios, using as reference data 3D points
collected with unmanned aerial vehicle (UAV) equipped with a laser scanner24, reporting planimetric and elevation
errors with respect to a TLS25; (ii) in indoor environment, employing a high-precision laser-based positioning and
tracking system26.
1.1 Paper overview
The aim of the article is to investigate the performance of two portable MMS: the GeoSLAM ZEB-REVO (Figure 1a and
1b) and the Leica Pegasus:Backpack (Figure 1c and 1d). Both systems, described in section 2, are evaluated indoor
(section 3) and outdoor (section 4). In the first test, two floors of a building are surveyed with the two portable MMS and
independent check measurements are acquired using a ToF (time-of-flight) phase-shift TLS (Leica HDS700027). The
indoor scene, characterized by smooth and homogenous surfaces, as well as constructive elements like columns, is also
used to derive meaningful information about noise on horizontal and vertical planes, along with fitting of geometric
primitives for the two systems. The outdoor experiment is carried out in an 80 m by 70 m historical city square, where
the two portable MMS are compared against a classical van-based MMS (RIEGL VMX-45028) whose accuracy potential
is discussed by Toschi et al22. A robust statistical analysis is performed to evaluate the portable MMSs geometric
performance in an outdoor environment.
2. THE TWO PORTABLE MMS
2.1 The GeoSLAM ZEB-REVO device
The GeoSLAM ZEB-REVO is a lightweight portable MMS, commercialized by the GeoSLAM company. The
instrument is the evolution of the ZEB1 device, commercial version of the ZEBedee handheld 3D range sensor
developed by the Autonomous Systems Laboratory, CSIRO ICT Centre in Brisbane (Australia)29. The main difference
between the two devices is that the ZEB1 is equipped with a spring that allows the scan head to nod, or oscillate both in a
‘front-back’ and ‘side-by-side’ direction; in the updated version of the system, the ZEB-REVO, the spring is no longer
present and it has been replaced by an automatic rotating head.
The instrument features a 2D infrared laser scanner profilometer (UTM-30LX) coupled with an IMU sensor, without
GNSS receiver. The 2D laser profilometer is a compact laser scanning range finder (LRSF), which consume less power,
is more compact and light-weight than a classic 3D laser scanner system30. The UTM-30LX emits pulsed light beams in
the near infrared, with a wave length of 905 nm; the travelling time of the pulse from the sensor to the object and back
provides the range measurement, according to the well-known time-of-flight measurement principle. The laser source
scans a 270° semicircular field, so that the coordinates of the recorded points are calculated using the measured distance
and laser pulse step angle31. The industrial-grade microelectromechanical (MEMS) IMU is mounted beneath the scanner
and consists of triaxial gyros and accelerometers, providing measurements of angular velocities and linear accelerations
that, combined with the laser data, allow to estimate the sensor trajectory. The IMU also contains a three-axis
magnetometer to reduce environmental magnetic interference32.
The scanning head, consisting of the laser and IMU sensors, is connected to a backpack containing the battery and a
data-logger unit, where the acquired data are stored in real-time.
The 2D laser profiles are aligned through a 3D SLAM (simultaneous localization and mapping) approach that estimates
the six degree of freedom (6DoF) of the sensor head motion, and generates the 3D point cloud of the scene (i.e. the map).
Details of the implemented SLAM approach are provided in29,33.
Recently, the ZEB-REVO has been equipped with a GoPro camera. The authors haven’t had the opportunity to test the
new version of the device yet; however, according to the specifications from the manufacturer16, the camera is intended
to provide imagery co-registered with the scanning data, but not to enrich the 3D laser point cloud with RGB
information. The authors developed an in-house method to map color information to the 3D data acquired with the
GeoSALM devices based on two GoPro cameras attached to the scan head (Figure 2a). The images from the cameras are
processed through a photogrammetric workflow and the obtained RGB information is mapped onto the ZEB point cloud
(Figure 2b and 2c).
a)
b)
c)
Figure 2. An in-house method is developed to map color information based on two GoPro cameras fixed on the GeoSLAM ZEB1
device (a). The original (b) and colored (c) 3D point clouds.
The proper acquisition approach with both GeoSLAM instruments requires to leave them stationary on a horizontal,
planar surface for the so-called initialization procedure, which allows to define a local coordinate reference system with
the vertical axis perpendicular to this planar surface. Then the user preforms the acquisition walking through the scene
and, at the end, must go back to the initial position to close the acquisition loop.
The collected data are post-processed through proprietary software applications, either running on local machine or via
cloud processing. The data presented in this contribution were processed via the desktop software application (version
v1), where the processing was totally automatic and no user intervention or alignment refinement was possible.
Table 1. Technical specifications of the tested portable MMSs. (*) An optional GoPro camera is now available.
GeoSLAM ZEB-REVO Leica Pegasus:Backpack
Max
range
indoor 30 m 50 m
outdoor 15-20 m
# Laser scanners 1 2
Measuring principle ToF ToF
Scanner data acquisition rate 43,200 points/sec 300,000 points/sec per scanner
Scanner
Resolution
horizontal 0.625° 0.1°-0.4°
vertical 1.8° 2.0°
Scanner angular FOV 270° x 360° 360° x 30° per scanner
Channels 1 16 per scanner
Laser wavelength 905 nm 903 nm
Scanner line speed 100 Hz 5-20 Hz
Laser head rotation speed 0.5 Hz -
Scanner weight 1.0 kg 0.83 kg per scanner
Scanner dimensions 86 x 113 x 287 mm 72 x 72 x 72 mm per scanner
# Cameras 0 (*) 5
CCD / pixel size - 2046 x 2046 / 5.5 um x 5.5 um per camera
Focal length - 6.0 mm
Cameras total angular FOV - 360° x 200°
Cameras max frame rate - 8 Hz
IMU type MEMS FOG
GNSS - Triple band, single and dual antenna support
Relative accuracy 2-3 cm 3-5 cm
Absolute
position
accuracy
indoor 3-30 cm
(10 mins scanning, 1 loop)
5-50 cm
(10 mins scanning, minimum 3 loop closures
or double passes conditions)
outdoor 5 cm
Tot system weight 4.1 kg 11.9 kg
Backpack dimensions 220 mm x 180 mm x 470 mm 310 mm x 270 mm x 730 mm
Operating time 4 hours 3 hours, up to 6 hours with optional batteries
2.2 The Leica Pegasus:Backpack
The Leica Pegasus:Backpack (summer 2016 version) combines two laser profilometers (Velodyne PUCKTM VLP-16)
synchronized with five cameras, a triple band GNSS receiver (NovAtel ProPak6™), and a fiber optic gyroscope (FOG)
IMU.
Each VLP-16 features 16 laser/detector pairs, mounted in a rotating housing providing 360° field of view34. The device is
also equipped with five high-dynamic-range cameras, placed to acquire 360°x200° field of view. The cameras provide
images co-registered with the laser data and RGB color information projected to the point cloud.
The full inertial navigation system (INS) is composed of GNSS and IMU sensors; the absolute and outdoor positioning is
delivered by GNSS, while indoor positioning and in GNSS-denied environments is based on the IMU and a SLAM
algorithm. The integrated IMU is a FOG type, which is usually more costly and of slightly higher performance than
MEMS systems for standalone INS performance35.
The acquired data, i.e. laser profiles and spherical images, are automatically processed via the proprietary software
application; however, the user may refine the alignment by either importing ground control points (GCPs) or identifying
3D tie points on the point clouds. A plug-in working in ArcMAP, component of the geospatial processing toolkit ArcGIS
by Esri, is available to visualize and analyze the data.
To assure a fair investigation of the two MMSs, the Leica Pegasus:Backpack data presented in this study are processed
following an automatic approach with minimal user intervention. Evaluating the system calibration and adjustment
procedures of the collected raw data is out of the scope of the tests here described.
3. INDOOR PERFORMANCE EVALUATION
The method adopted to test the indoor performances of the two portable MMSs is based on two measures for quality
assessment, i.e. the root mean square (RMS) of the residuals from fitting of geometry primitives (section 3.1) and length
comparison (section 3.2).
The scanning operations in the building (Figure 3) are carried out with the two MMSs preforming the initialization
procedure outside the building and following a close-loop acquisition path.
Figure 3. The two-floor building, test area for the indoor performance evaluation. Left: a picture of the interior. Right: cross section of
the ZEB-REVO point cloud.
3.1 Reference data and alignment of point clouds
To evaluate the indoor accuracy potential, a single point cloud, consisting of a great portion of the second floor and part
of the first floor, is acquired with a phase-shift ToF terrestrial laser scanner (TLS), the Leica HDS7000, whose technical
specifications are reported in Table 2. Only one point cloud is used as refence to avoid any additional uncertainty that
might be caused, for example, by uncontrolled residual errors in the registration process of several point clouds.
Figure 4. Up: top view of second floor walls; down: perspective view of a section of the two floors. In white the HDS7000 point
cloud, in cyan the ZEB-REVO, in yellow the Pegasus.
The point clouds acquired with the three different devices are aligned in the same coordinate reference system, defined
as follows: the z axis coincides with the gravity vector, i.e. the vertical direction provided by leveling the HSD7000 with
the digital bubble; the x axis runs along the width of the floors; the y lays along the length. Some views of the aligned
point clouds are shown in Figure 4.
Table 2. Technical specifications of the Leica HDS7000 TLS.
Measuring principle phase-shift ToF
Max range 187 m (ambiguity interval)
Data acquisition rate Up to 1,016,727 points/sec
Scanner resolution (horizontal/vertical) 0.3°-0.004°
Scanner angular FOV 360° x 320°
Laser wavelength 1500 nm
Range noise (on a black 14% target, i.e. worst condition) 0.5 / 2.7 mm rms @ 10 / 50 m
Linearity error ≤ 1 mm
Angular accuracy (horizontal/vertical) 127 urad
3.2 Noise estimation
The device’s noise is evaluated through the fitting of geometric primitives (planes and cylinders) to the acquired point
clouds: vertical planes are fitted on the walls, horizontal planes on the floors and ceilings, cylinders on the columns.
Table 3 reports the RMS results. As expected, the HDS7000 greatly outperforms the two MMSs, which anyway show
RMS values significantly within the levels declared by the vendors (see the ‘relative accuracy’ row in Table 1).
Table 3. Noise estimation: RMS of fitting procedures with outliers removal.
HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack
Vertical planes 0.1 cm 1.1 cm 1.9 cm
Horizontal planes 0.2 cm 0.9 cm 1.2 cm
Cylindrical columns 0.1 cm 0.9 cm 1.8 cm
3.3 Length measurements
The positional or coordinate standard error (SXYZ), theoretical length measurement error (TLME) and relative theoretical
length measurement accuracy (RTLMA) are reported in Table 4.
The SXYZ for the two MMSs are derived from the datasheet, i.e. the ‘absolute position accuracy’ in Table 1, whilst for the
HDS7000 is computed taking into account the angular accuracy, linearity error and range noise for two distances (10 m
and 50 m) in the worst condition (laser reflected from a 14% black surface, Table 2).
The TMLE and RTLMA are, respectively, calculated according to (1)36 and to (2), for two distance values D equal to 3
m and 50 m, considering the best and worst positional standard errors.
XYZs 23TLME (1)
TLME:1RTLMA
DROUND (2)
Table 4. Positional or coordinate standard error (SXYZ), theoretical length measurement error (TLME) and relative theoretical length
measurement accuracy (RTLMA). The value marked with () are taken from the instrument datasheet (Table 1 - ‘absolute position
accuracy indoor’).
HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack Min Max Min Max Min Max
XYZs 0.1 cm @ 10 m 0.5 cm @ 50 m 3 cm 30 cm 5 cm () 50 cm ()
TLME 0.6 cm 2.3 cm 12.7 cm 127.3 cm 21.2 cm 212.1 cm
RTLMA D = 3 m 1:500 1:150 1:25 1:2 1:15 1:1
D = 50 m 1:9000 1:2200 1:500 1:50 1:250 1:25
The theoretical length measurement accuracy of the HDS7000 is from one to two orders of magnitude better than the two
portable MMS, and increases, as expected, with the distance.
Figure 5: Distances measured for the indoor performance evaluation.
Figure 5 depicts the distance measurements extracted from the point clouds of the two-floor building. The length, L, two
width values, B1 and B2, and one height, H1, are measured on the second floor; the height H2 is measured from the first
floor to the ceiling of the second.
Table 5 reports the analysis performed on the measured distances. Each value is the mean of eight distance
measurements between two planes fitted on the opposite walls (L, B1 and B2), and between floor and ceiling (H1 and
H2). The relative length measurement error (RLME) is computed according to (3) as the relative difference between the
measured distance Dm for the GeoSLAM ZEB-REVO and Leica Pegasus:Backpack, and the distance from the HSD7000,
assumed as reference length Dr. The relative length measurement accuracy (RLMA) is defined as the rounded absolute
reciprocal value of the RLME times 100 (4).
100RLME
r
rm
D
DD (3)
rm
r
DD
DROUNDROUND :1
RLME
100:1RLMA (4)
The RLMA results always better than the computed theoretical accuracies (Table 4); in particular, the GeoSLAM ZEB-
REVO provides distance measurements that are closer to the HDS7000 than the Leica Pegasus:Backpack, which also
features higher standard deviation values (σ).
Table 5. Indoor performance evaluation: measured distances with standard deviations (σ), relative length measurement errors
(RMLE) and accuracies (RMLA).
HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack
Dr [m] σ [cm] Dm [m] σ [cm] RLME RLMA Dm [m] σ [cm] RLME RLMA
L 45.112 0.4 cm 45.135 0.7 0.051% 1:2000 45.133 0.9 0.047% 1:2000
B1 14.888 0.5 cm 14.906 0.6 0.115% 1:900 15.227 8.3 2.274% 1:50
B2 14.836 0.3 cm 14.851 2.0 0.103% 1:1000 14.925 0.8 0.600% 1:150
H1 3.021 0.2 cm 3.021 0.3 -0.001% 1:7000 3.094 7.3 2.416% 1:50
H2 7.666 0.5 cm 7.659 0.6 -0.087% 1:1000 7.814 1.7 1.941% 1:50
4. OUTDOOR PERFORMANCE EVALUATION
The outdoor tests are performed in the cathedral square in Trento (Figure 6a). The site represents a challenging scenario
for SLAM based systems due to moving objects, people, cars, trucks, buses, and the façades geometry. The city is
surrounded by high mountains that might constitute an unfavorable environment also for GNSS based systems, although
the square itself is quite big (80 m x 70 m). It is worth noting the evident blunders in the GeoSLAM ZEB-REVO point
cloud (Figure 6c, highlighted in yellow): the fountain and tree are duplicated, clearly due to huge error in the alignment
of the laser profiles.
The point clouds acquired with the two portable MMSs (Figure 6c and 6d) are compared through a robust statistical
analysis with data derived from the RIEGL VMX-450 MMS mounted on a van.
a) b)
c) d)
Figure 6. The cathedral square in Trento (Italy), test area for the outdoor performance evaluation (a). RIEGL VMX-450 MMS colored
point cloud of the square (b); GeoSLAM ZEB-REVO point cloud (c); Leica Pegasus:Backpack colored point cloud (d).
4.1 Reference data
The RIEGL VMX-450 MMS platform (Table 6) integrates two synchronously operated VQ-450 laser scanners, a
portable control unit (VMX-450-CU) and IMU/GNSS navigation hardware. The system measures up to 1.1 million
points and 4 profiles per second, with an online waveform processing that allows to penetrate obstructions, such as
fences and vegetation. The platform is also equipped with the modular VMX-450-CS6 camera system, where up to six
industrial digital color cameras can be integrated. The cameras complement the acquisition of geometric data from the
laser scanners with time-stamped images. A detailed performance investigation of the RIEGL VMX-450 MMS is
presented in Toschi et al22.
Table 6. Technical specifications of the RIEGL VMX-450 system.
# Laser scanners 2
Measuring principle phase-shift ToF
Max range 800 m
Data acquisition rate 550,000 points/sec per scanner
Scanner angular FOV up to 360°
Laser wavelength near infrared
# Cameras up to 6
CCD / pixel size 2452 x 2056 / 3.45 um x 3.45 um per camera
Focal length 5 mm
Cameras angular FOV 80° x 65° per camera
Relative position accuracy 1 cm
Absolute position accuracy 2-5 cm
Tot system weight > 100 kg
4.2 Alignment and signed distance computation
Figure 7. Color-coded map of the signed distances computed between the GeoSLAM ZEB-REVO (left) and Leica Pegasus:Backpack
(right) point clouds and the RIEGL VMX-450 data. The differences are in m.
To evaluate the accuracy potential of the two portable MMSs against the reference data, the three point clouds are first
cleaned from noisy elements, such as pedestrians, vegetation, cars, etc. Due to the blunder in the ZEB-REVO point cloud
(Figure 6c), the fountain is also removed; consequently, only the building façades overlooking the square are used in the
performance evaluation.
The blunder-free point clouds from GeoSLAM ZEB-REVO and Leica Pegasus:Backpack are aligned in a local
coordinate reference systems to the cleaned RIEGL VMX-450 data by means of the iterative closest point (ICP)
registration method implemented in the open source software application CloudCompare v2.8
(http://www.cloudcompare.org/). The signed distances between the portable MMSs and RIEGL VMX-450 point clouds
are then computed, using the CloudCompare M3C2 plugin, which implements the Multiscale Model to Model Cloud
Comparison method37. It allows a direct comparison of 3D points, without the need of a preliminary meshing or gridding
phase. If the data do not contain normal vectors, they are estimated on the basis of the local surface roughness. Then, for
each 3D point, the local distance between the two clouds is computed. The color-coded maps of the signed distances for
both the portable MMMs are shown in Figure 7.
4.3 Robust statistical analysis
ZE
B-R
EV
O
VS
RIE
GL
VM
X-4
50
Peg
asu
s:B
ack
pack
VS
RIE
GL
VM
X-4
50
Figure 8. Histograms of the signed differences with the superimposed curve for the normal distribution (left). Q-Q plots of the
distribution of the signed differences (right).
Several studies38,39,40 have demonstrated that in the accuracy assessment of data provided by laser scanner systems, as
well as photogrammetry, the hypothesis that errors follow a Gaussian distribution is hardly verified. This might be due to
the presence of residual system errors, but also unwanted objects not correctly filtered out from the data. In the following
analyses, the hypotheses (i) that many outliers will exist in comparing data provided by different instruments and (ii) that
the normality assumption of distribution of the differences is not valid, are first verified. Then, suitable accuracy
measures are computed and reported in Table 7.
Two visual diagnostic tests are reported to test the normality assumption (Figure 8), i.e. the histogram of the signed
differences with the superimposed curve for the normal distribution, and the quantile-quantile (Q-Q) plot of the
distribution of the signed differences38. The Q-Q plot depicts the quantiles of the empirical distribution plotted against
the theoretical quantiles of the normal distribution. If the actual distribution is normal, the Q-Q plot should provide a
straight line. Big deviation from the straight line indicates that the distribution of the errors is not normal.
Both graphical evaluations show that the differences do not follow a Gaussian distribution: the Q-Q plot relative to the
GeoSLAM ZEB-REVO comparison shows a shape significantly far away from the theoretical normal hypothesis, while
the histogram for the Leica Pegasus:Backpack a two-peaked distribution with a quite high dispersion.
The previous conclusion is also supported by the computed kurtosis (5) and skewness (6) values, which represent,
respectively, a measure of whether the data are peaked or flat with respect to a normal distribution, and an indication of
departure from symmetry in a distribution (asymmetry around the mean value):
3
1 Kurtosis
41
4
n
xn
ii
(5)
31
3
1 Skewness
n
xn
ii
(6)
Where n, µ and σ are the sample size (i.e. number of data points x), mean and standard deviation. When the sample
distribution follows the normal hypothesis, both values should be equal to zero. The distance distribution for the
GeoSLAM ZEB-REVO reveals a highly peaked shape, while the distribution for the Leica Pegasus:Backpack results
slightly more asymmetric.
In both cases, the distribution of the differences discloses kurtosis, skewness, hence a significant amount of outliers;
consequently the classical µ and σ parameters are not adequate to provide the accuracy measures for the two portable
MMSs. When this is the case, other non-parametric estimators are to be adopted, such as the median m, normalized
median absolute deviation – NMAD (7) and the square root of the biweight midvariance – BWMV (8):
MAD4826.1 NMAD (7)
2
1
22
1
422
511
1BWMV
n
iiii
n
iiii
UUa
Umxan (8)
1,0
1,1
i
ii
Uif
Uifa (9)
MAD9
mxU i (10)
being the median absolute deviation – MAD (9), i.e. the median (m) of the absolute deviations from the data’s median
(mx):
xi mxm MAD (11)
The computed accuracy measures are summarized in Table 7. The value of the median is closer to the mean for the
GeoSLAM ZEB-REVO, while is smaller for the Leica Pegasus:Backpack. The values are inside the expected a-priori
error of the point clouds analysis (Table 1); they are likely to represent the residual of the registration process. In both the
cases, the values of the median are smaller than the absolute position accuracy quoted in the technical sheets (Table 1).
Concerning the values of the standard deviations, they are higher than the NMAD and BWMV, and lower than the
specifications from the vendors. In accordance to the visual analysis of the histograms (Figure 9), the values for the
Leica Pegasus:Backpack indicate a wider dispersion than the GeoSLAM ZEB-REVO.
Since the data sample does not follow a Gaussian distribution, the standard deviation values are affected, so it does not
represent correctly the error dispersion. It appears clearly in the case of GeoSLAM ZEB-REVO when is compared with
the robust measures of the dispersion (NMAD and square root of BWMV), being undervalued (more than two times).
Please note also, the asymmetry of the error distribution, being not possible provide a plus-minus range, but an absolute
interpercentile range. In the case of Table 7, it is computed according to the real error distribution. They (dispersion
measurement and error distribution function) are related, showing for the Leica Pegasus:Backpack a wider dispersion
than the GeoSLAM ZEB-REVO, in according to the visual analysis of the histograms (Figure 9). Moreover, in the
reported interpercentile range at 95 % of confidence level, both portable MMS differs from their relative declared
position precision. As a remark, the 50 % of the observed points (interpercentile range at 50% of confidence level) are in
a 3.0 cm size interval for the GeoSLAM ZEB-REVO, and 13.5 cm for Leica Pegasus:Backpack, while the 95% of data
points (interpercentile range at 95% of confidence level) are within 30 cm and 51 cm for the GeoSLAM ZEB-REVO and
Leica Pegasus:Backpack (respectively).
Table 7. Statistical analysis of the signed distance computation.
ZEB-REVO
VS
RIEGL VMX-450
Pegasus:Backpack
VS
RIEGL VMX-450
Sample size n 4,061,074 1,0351,184
Gaussian
assessment
Kurtosis 19.331 1.157
Skewness 0.079 0.093
Sample mean µ 2.0 cm 1.9 cm
Standard deviation σ 7.1 cm 13.3 cm
Robust
assessment
Median 2.0 cm 1.0 cm
NMAD 2.2 cm 9.6 cm
Sqrt(BWMV) 2.9 cm 12.5 cm
Interpercentile range 50% 3.0 cm 13.5 cm
Percentile 0.025 -9.7 cm -21.8 cm
Percentile 0.975 14.3 cm 29.1 cm
Interpercentile range 95% 23.9 cm 50.9 cm
5. CONCLUSIONS
The presented investigation aimed at evaluating the performance of two portable MMSs, the handheld GeoSLAM ZEB-
REVO and Leica Pegasus:Backpack, in indoor and outdoor scenarios. The tests were designed to specifically address
relevant issues related to mapping environments, such as building interiors or complex city parts, which represent typical
applications for such systems. Consequently, the analyses were performed in order to estimate the magnitude of errors in
measuring distances and acquiring 3D data. To this end, the data provided by the two MMSs were compared against
those acquired by two different reference devices, a TLS for the indoor and a van-based MMS for the outdoor.
While indoor the performance assessment was carried out reporting noise evaluation and length measurement errors, a
robust statistical analysis was performed on data acquired outdoor. In both scenarios, the two portable MMSs performed
within the accuracy specifications provided by the vendors, with the GeoSLAM ZEB-REVO generally outperforming
the declared values despite the evident gross error in the point cloud acquired outdoor. Worth to note is that errors in
Leica Pegasus:Backpack data might be further mitigated by including control points and constraints within the post-
processing adjustment.
The results and statistical analyses presented in this paper were achieved using instruments tested during the summer
2016. Other performances could be expected using new releases of the instruments.
ACKNOWLEDGMENTS
Authors are thankful to Dr. Nadia Guardini (ME.S.A. srl), Mr. Marco Formentini and Mr Simone Oppici (Leica
Geosystem Italy) for giving the possibility to use and test the two instruments investigated in this article.
REFERENCES
[1] Puente, I., González-Jorge, H., Martínez-Sánchez, J. and Arias, P., "Review of mobile mapping and surveying
technologies," Measurement 46(7), pp. 2127-2145 (2013).
[2] Schwarz, K.P. and El-Sheimy, N., "Mobile mapping systems–state of the art and future trends, "International
Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 35(Part B), p.10 (2004).
[3] Surmann, H., Nchter, A., Lingemann, K. and Hertzberg, J., "Kurt3D-an autonomous mobile robot for modelling
the world in 3D," ERCIM NEWS, 55, pp.24-25 (2003).
[4] Nüchter, A., Elseberg, J. and Borrmann, D., "Irma3D—An Intelligent Robot for Mapping Applications," IFAC
Proceedings Volumes, 46(29), pp.119-124 (2013).
[5] Ziparo, V.A., Zaratti, M., Grisetti, G., Bonanni, T.M., Serafin, J., Di Cicco, M., Proesmans, M., Van Gool, L.,
Vysotska, O., Bogoslavskyi, I. and Stachniss, C., "Exploration and mapping of catacombs with mobile robots,"
In Safety, Security, and Rescue Robotics (SSRR), 2013 IEEE International Symposium on, pp. 1-2 (2013).
[6] Centre of Excellence in Laser Scanning Research, http://laserscanning.fi/laser-scanner-in-a-backpack/, (April,
2017).
[7] Leica, http://leica-geosystems.com/products/mobile-sensor-platforms/capture-platforms/leica-pegasus-
backpack, (April, 2017).
[8] ROBIN, Three-in-one LiDAR mapping system, 3D Laser Mapping, < http://www.3dlasermapping.com/robin/>
(April, 2017).
[9] Nüchter, A., Borrmann, D., Koch, P., Kühn, M. and May, S., "A man-portable, IMU-free mobile mapping
system," ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences II-3 W, 5,
pp.17-23 (2015).
[10] Gexcel, http://www.gexcel.it/it/soluzioni/heron, (April, 2017).
[11] Vexcel Imaging, http://www.vexcel-imaging.com/products/ultracam-panther/, (April, 2017).
[12] VIAMETRIS, http://www.viametris.com/wcms/file/323-VIAMETRIS_-_iMS2D_v1b_Leaflet_EN_-_Light.pdf,
(April, 2017).
[13] Geo-SLAM, http://geoslam.com/hardware-products/zeb1/, (April, 2017).
[14] Geo-SLAM, http://geoslam.com/hardware-products/zeb-revo/, (April, 2017).
[15] Paracosm, http://labs.paracosm.io/hubfs/Paracosm-PX-80-Specs.pdf, (April, 2017).
[16] GeoSLAM, http://geoslam.com/wp-content/uploads/2016/12/ZEB-CAM-Brochure-v1.pdf, (April, 2017).
[17] VIAMETRIS, viametris.info/iMS3D/EN/, (April, 2017).
[18] Applanix, https://www.applanix.com/products/timms-indoor-mapping.htm, (April, 2017).
[19] Kaartinen, H., Hyyppä, J., Kukko, A., Jaakkola, A. and Hyyppä, H., "Benchmarking the performance of mobile
laser scanning systems using a permanent test field," Sensors, 12(9), pp.12814-12835 (2012).
[20] Haala, N., Peter, M., Kremer, J. and Hunter, G., "Mobile LiDAR mapping for 3D point cloud collection in
urban areas—A performance test," The International Archives of Photogrammetry, Remote Sensing and Spatial
Information Sciences, 37, pp.1119-1127 (2008).
[21] Lin, Y., Hyyppä, J., Kaartinen, H. and Kukko, A., "Performance analysis of mobile laser scanning systems in
target representation," Remote Sensing, 5(7), pp.3140-3155, (2013).
[22] Toschi, I., Rodríguez-Gonzálvez, P., Remondino, F., Minto, S., Orlandini, S. and Fuller, A., "Accuracy
evaluation of a mobile mapping system with advanced statistical methods," The International Archives of
Photogrammetry, Remote Sensing and Spatial Information Sciences, 40(5), p.245 (2015).
[23] Rodríguez-Gonzálvez, P., Jiménez Fernández-Palacios, B., Muñoz-Nieto, Á.L., Arias-Sanchez, P. and
Gonzalez-Aguilera, D., "Mobile LiDAR System: New Possibilities for the Documentation and Dissemination of
Large Cultural Heritage Sites," Remote Sensing, 9(3), p.189 (2017).
[24] Rönnholm, P., Liang, X., Kukko, A., Jaakkola, A. and Hyyppä, J., "Quality analysis and correction of mobile
backpack laser scanning data," ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information
Sciences, pp.41-47 (2016).
[25] Glennie, C., Brooks, B., Ericksen, T., Hauser, D., Hudnut, K., Foster, J. and Avery, J., "Compact multipurpose
mobile laser scanning system—Initial tests and results," Remote Sensing, 5(2), pp.521-538 (2013).
[26] Lauterbach, H.A., Borrmann, D., Heß, R., Eck, D., Schilling, K. and Nüchter, A., "Evaluation of a backpack-
mounted 3D mobile scanning system," Remote Sensing, 7(10), pp.13753-13781 (2015).
[27] Leica Geosystems, http://hds.leica-geosystems.com/downloads123/hds/general/HDS7000/manuals/
HDS7000_UserManual_en.pdf, (April 2017).
[28] RIEGL, http://www.riegl.com/uploads/tx_pxpriegldownloads/DataSheet_VMX-450_2015-03-19.pdf, (April
2017).
[29] Bosse, M., Zlot, R. and Flick, P., "ZEBedee: Design of a spring-mounted 3-d range sensor with application to
mobile mapping," IEEE Transactions on Robotics, 28(5), pp.1104-1119 (2012).
[30] Mader D, Westfeld P and Maas HG, "An integrated flexible self-calibration approach for 2D laser scanning
range finders applied to the Hokuyo UTM-30LX-EW," The International Archives of Photogrammetry, Remote
Sensing and Spatial Information Sciences, 385 (2014).
[31] Hokuyo, http://www.hokuyo-aut.jp/02sensor/07scanner/download/pdf/UTM-30LX_spec_en.pdf, (April 2017)
[32] Zlot, R. and Bosse, M., "Three-dimensional mobile mapping of caves," Journal of Cave and Karst Studies,
76(3), p.191 (2014a).
[33] Zlot, R. and Bosse, M., "Efficient Large-scale Three-dimensional Mobile Mapping for Underground Mines,"
Journal of Field Robotics 31(5), pp.758-779 (2014b).
[34] Velodyne Lidar, http://velodynelidar.com/docs/manuals/63-9243%20Rev%20B%20User%20Manual%20and
%20Programming%20Guide,VLP-16.pdf (April 2017)
[35] Goodall, C., Carmichael, S. and Scannell, B., "The battle between MEMS and FOGs for precision guidance."
Analog Devices Technical Article MS-2432 (2013).
[36] Luhmann, T., Robson, S., Kyle, S. and Boehm, J., [Close-range photogrammetry and 3D imaging]. Walter de
Gruyter (2014).
[37] Lague, D., Brodu, N. and Leroux, J., "Accurate 3D comparison of complex topography with terrestrial laser
scanner: Application to the Rangitikei canyon (NZ)," ISPRS Journal of Photogrammetry and Remote Sensing,
82, pp.10-26 (2013).
[38] Höhle, J. and Höhle, M., "Accuracy assessment of digital elevation models by means of robust statistical
method," ISPRS Journal of Photogrammetry and Remote Sensing, 64(4), pp.398-406 (2009).
[39] Hasan, A., Pilesjö, P. and Persson, A., "The use of LIDAR as a data source for digital elevation models–a study
of the relationship between the accuracy of digital elevation models and topographical attributes in northern
peatlands," Hydrology and Earth System Sciences Discussions, 8(3), pp.5497-5522 (2011).
[40] Rodríguez-Gonzálvez, P., Garcia-Gago, J., Gomez-Lahoz, J. and González-Aguilera, D., "Confronting passive
and active sensors with non-Gaussian statistics," Sensors, 14(8), pp.13759-13777 (2014).
Top Related