4. 2X_Free-Form Surface Inspection Techniques State of the Art Review

7/26/2019 4. 2X_Free-Form Surface Inspection Techniques State of the Art Review

http://slidepdf.com/reader/full/4-2xfree-form-surface-inspection-techniques-state-of-the-art-review 1/23

Free-form surface inspection techniques state of the art review

Yadong Li, Peihua Gu*

Department of Mechanical and Manufacturing Engineering, University of Calgary, Calgary, Alberta, Canada T2N 1N4

Received 15 August 2003; received in revised form 20 February 2004; accepted 28 February 2004

Abstract

Precision inspection has been widely used in manufacturing to measure the dimensional accuracy of parts and products to meet the quality

requirements. For regular geometric features, coordinate-measuring machines (CMM) can be used effectively to assess the accuracy and

tolerances. For parts with free-form surfaces, the inspection becomes complex. Therefore, numerous researches have been carried out to

tackle both fundamental and application issues concerning free-form surface inspection. In addition to academic research, some commercial

packages have also been developed.

This paper provides a comprehensive literature review of methodologies, techniques and various processes of inspections of parts with

free-form surfaces. The specific topics cover: measurement data acquiring methods including contact and non-contact measurement

approaches; inspection planning; geometric description methods of design models or measurement data; and, the free-form surface

localization and comparison techniques, which are emphasized in this paper and mainly include the establishment of corresponding

relationship, 3D transformation solving, measurement data to design model comparison or surface to surface distance calculations. Other

issues, such as the influence factors to the localization/registration process, definition and inspection of free-form surface tolerance, and

discussions on the functions of some commercial inspection packages available on market, are also discussed.q 2004 Elsevier Ltd. All rights reserved.

Keywords: Free-form surface; Inspection; Comparison and localization

1. Introduction

Free-form surfaces are widely used in many fields

ranging from design and manufacturing of die/mold,

patterns and models, and products in plastics, automotive

and aerospace industries to biomedical, entertainment and

geographical data processing applications. In the last several

decades, significant research and development efforts havebeen made for the design and manufacturing of products/

objects consisting partially or solely of free-form surfaces.

Almost all CAD systems have the capability of designing

and modeling of free-form surfaces. Modern CNC machines

can be used to manufacture free-form surfaces. To ensure

manufacturing quality, free-form surfaces should be

inspected after they are produced.

It is a fundamental issue in product manufacturing to

determine if a manufactured object meets the design

requirements from which this object was made. For products

with regular features, such as plane and cylinder,

the inspection techniques and equipment are mature in

industrial applications. For products with free-form surfaces,

such as marine propellers, complex sculptured surfaces are

produced with extremely high fidelity to the original design

[74]. In general, increasing demands have been observed on

the quality manufacturing of parts with complex surfaces,

which ultimately require precision measurement and inspec-

tion [4,91]. Thus, the dimensional inspection of sculpturedsurfaces is essential as many products with sculptured

surfaces are designed and manufactured with a requirement

for high precision [59]. In addition, for surfaces recon-

structed in reverse engineering and imaging processing, the

reconstructed surfaces also require comparison with the

reference models. The inspection of parts with free-form

surfaces is becoming increasingly critical due to increasing

requirements of higher precision and efficiency, and to the

complexity of the geometry.

In the last several decades, numerous efforts have been

reported in open literature on inspection processes, methods

and data processing techniques. As the first step of the

inspection process, geometric measurements are required.

There are two types of measurement techniques and

0010-4485/$ - see front matter q 2004 Elsevier Ltd. All rights reserved.

doi:10.1016/j.cad.2004.02.009

Computer-Aided Design 36 (2004) 1395–1417www.elsevier.com/locate/cad

* Corresponding author. Tel.: þ 1-403-220-5770; fax: þ1-403-282-8406.

E-mail address: [email protected] (P. Gu).

http://www.elsevier.com/locate/cad

http://www.elsevier.com/locate/cad



facilities: contact measurements, normally by using a

coordinate measuring machines (CMM), and non-contact

measurements, such as laser and optical scanning. There

were several literature surveys about non-contact

measurement inspection techniques for vision-based

inspection of objects including 3D mechanical parts.

These surveys were presented by Chin and Harlow [78],

Chin [79] and Newman [85].

In this paper, a comprehensive literature review of

inspection and comparison techniques for parts with free-

form surfaces is provided, which covers both contact and

non-contact measurements. For vision-based inspection,

Newman provided a review covering the literature before

1995 [85]. This paper, therefore, will concentrate on more

recent developments. Among the inspection processes, there

is another kind of visual inspection approach that mainlydeals with surface quality inspection including the inspec-

tion and detection of defects on surfaces, such as cracks,

corrosion, lightning strikes on a part or component body, or

a damaged part on a component assembly. The remote

visual inspection of an aircraft surface or skin is an example

introduced in the literature [67] and [62]. Such visual

inspection also covers defect detection for metallic work-

piece surfaces [58] or wooden product surfaces, etc. This

review, however, will not cover this type of surface

inspection technique. Instead, we will concentrate on only

the inspection techniques related to geometric information

from both the measurement object and the design model.

1.1. Background

A free-form surface is a type of surface, where the shape

is not constrained by classical analytical forms and is

defined by a set of control points (as with Bezier, b-spline,

and NURBS surfaces) [61]. The characterization of free-

form surfaces is given by P. J. Besl—a free-form surface has

a well-defined surface normal that is continuous almost

everywhere except at vertices, edges and cusps [69]. Despite

its wide applications in many fields, Campbell stated [76]

that definitions of free-form surfaces and objects are often

intuitive rather than formal.

Inspection is the process of determining if a product

(also referred to as a part, object or item) deviates from a

given set of specifications [85,71]. To carry out an

inspection of surfaces, the surface of an inspected part

can be measured by various methods such as laser/optical

scanner and CMM. The acquired data is then compared

with the design model to determine whether the surface

is out of tolerance. For a regular shaped feature such as

cylinder, its dimensions include length and diameter thatcan be measured directly by instruments. The measure-

ment values are then compared with the design

specifications, which, in this case, are the diameter and

length specifications. For a free-form surface, such

comparison does not exist. The comparison of two

free-form surfaces is done by putting these two surfaces

together in the reference position and orientation so that

the point to point comparison can be carried out.

However, the design part is in the design coordinate

system (DCS) and the real part (manufactured part) is in

the measurement coordinate system (MCS). The first step

in conducting such inspection and comparison is to bring

these two surfaces together.Fig. 1 shows a CMM machine in (a) and the

inspection of a simple shaped part by CMM in (b). For

inspection of parts with free-form surfaces, the

relationship between DCS and MCS is complex. Fig. 2

shows the inspection of a part with complex surfaces by

Fig. 1. MCS, DCS and a part under inspection (picture modified from Ref. [7]). (a) Measurement Coordinate System (MCS); (b) Part and Design Coordinate

System (DCS).

Y. Li, P. Gu / Computer-Aided Design 36 (2004) 1395–1417 1396



a contact probe. In this figure, Pi is a design model point

to be measured, denoted by r i in the DCS. Due to

manufacturing errors, the design coordinate frame does

not coincide with the measurement frame. Thus, another

point Pp

i is measured as shown in the figure, which is

denoted by r pi in the MCS. The difference between these

two positions is Dr [91].

To compare the measurement surface with the design

model, it is essential to arrange these two surfaces in a

common coordinate system. This process is called locali-

zation. Localization refers to the determination of positions

and orientations of the DCS of a part with respect to the

MCS [92,91]. Localization is also referred to as registration

of design surface with measurement surfaces in some

literatures. In practise, locating a part is to find a rigid

body 3D coordinate transformation between the DCS and

the MCS.

1.2. Organization of the paper

The remaining sections of this paper are organized as

follows: Section 2 discusses measurement data acquisition

methods. Section 3 reviews the surface description methods

used in existing inspection and comparison approaches.

Section 4 describes the localization techniques betweenmeasurement data and design model or between two point

sets. Specific topics include: Section 4.1 discusses the

methods for establishing the corresponding relationships

and solution techniques for 3D transformations based on

contact measurement data; Section 4.2 provides a similar

discussion but is based on non-contact measurement data;

Section 4.3 focuses on the distance calculation methods

between two surfaces; Section 4.4 deals with localization

accuracy, efficiency and robustness; Section 4.5 briefly

discusses the ICP method; Section 4.6 focuses on the

techniques of solving 3D transformation problems.Section 5

is devoted to inspection planning techniques. Section 6

discusses the techniques for inspection of free-form surfacewith design datum. In Section 7, there are descriptions of

some commercial inspection packages. Finally, conclusions

are given in Section 8.

2. Measurement data acquisition methods

Traditional inspection of free-form surfaces has involved

highly skilled technical personnel who inspect the surface

using a number of mechanical gauges and templates [74].

This kind of process is inefficient in terms of speed and

accuracy. However, the development and use of modern

digitizing devices, such as CMM, laser/optical scanning

measurement systems and associated measuring methods,

make it possible to measure free-form surfaces quickly and

accurately.

There are generally two types of measurement dataacquiring methods: contact measurement and non-contact

measurement. Contact measurement is a measuring method

that acquires surface geometric information by physically

touching the parts using tactile sensors such as gauges and

probes. One example is CMM. Without physically contact-

ing the part, non-contact measurement is used to acquire

surface information by using some sensing devices, such as

laser/optical scanners, X-rays or CT scans. In non-contact

measurement of parts with free-form surfaces, laser and

optical scanners are commonly used.

A review of surface digitizing techniques and classifi-

cations was presented in Ref. [55]. In that article, important

features of some existing digitizing devices and associatedmeasurement strategies used in surface data acquisition

were discussed. Fig. 3 shows the classification of existing

surface digitizing methods shown in this literature.

CMM has been the main tool for part validation in

manufacturing. It has high accuracy, repeatability and

reliability. The measured 3D data is used with various

algorithms to determine positions, orientations and dimen-

sions of objects. Consequently, tolerance can be pro-

grammed, computed and verified [70]. As a contact

measuring device, CMM collects detailed dimensional

data by moving a sensing device called a probe along

work-piece surfaces (refer to Fig. 4). CMM usually acquires

data using a touch trigger probe that contacts individualpoints on a work-piece. It can be used to accurately measure

objects with widely varying size and geometric configur-

ation, and provide the relationship between the features of a

work-piece [8,60]. By applying motion control of the CMM

for non-stop surface scanning, the measuring speed can be

increased. In addition to the much higher accuracy than a

vision-based system, CMM has the advantages that it is not

affected by viewpoint or lighting conditions [60]. Thus, it

does not require clean surfaces or special illumination,

whereas a vision system always does. With the use of orient-

able probes, CMM can inspect surfaces that a light beam

cannot reach [3].

The disadvantages or limitations of CMMs are that thepart needs to be stationary and carefullyplacedand they have

Fig. 2. Measurement Modeling [91].




a slower measuring speed than laser and optical scanners.

This single-point measurement technique can generally

collect data at a maximum rate of about 50 , 60 points per

minute, depending on possible requirements of mechanical

fixture and the needs of programming when a different part is

inspected [1,8,16,71,70]. There is also a limitation to the size

of the part that can be measured. If an object has a very

small size (few hundreds of square millimeters), the tip of

Fig. 3. Classification of various digitizing devices [55].

Fig. 4. CMM measurement of a part [7].




the tactile probe will have problems accessing part features.

Very small objects often include features with light marks or

etchings, on which even the smallest feasible mechanical tip

could not reasonably work [7]. Besides, touch probes of

CMM are also crash-prone [60].

The above-mentioned shortcomings that accompany

CMM machines can be overcome by non-contact measuring

methods such as laser scanning and optical range image

scanning. A non-contact sensor virtually allows the

simultaneous collection of all points inside its field of

view. The scanning speed of the sensor is directly

proportional to its field of view. The minimum measuredentity is not the point but the element. A non-contact

measuring sensor is a device that allows the relating of

pixels of the image or picture taken by the sensor with the

corresponding 3D points of the framed part area by basically

using a CCD camera combined with a suitable lighting

device [7]. Fig. 5 shows one of the non-contact measure-

ment methods. There are three types of standard optical

technologies: point, stripe and area types. These are shown

in Fig. 6.

In summary, non-contact measuring methods such as

laser scanning and optical range image scanning have a

significantly higher speed of measurement (about

20000 points/s, applicable to on-line inspection [70]). Thiskind of system is mandatory for the inspection of parts built

of soft material. The accuracy of recent commercial optical

systems can be as good as about 0.00100 (0.025 mm) [43,70,

44,46], which is sufficient for some applications, such as

inspection of products made from rapid prototyping process.

However, this level of precision is lower compared with the

accuracy of CMM [70,16]. Even though this represents the

most common cases, our surveying results show that some

producersclaim higher accuracy fortheir laser scanners [45].

Based on their characteristics, both contact and non-

contact measuring methods have been applied widely to

research and manufacturing fields, and will continue to be

used for a long period of time. In some applications,

solutions of CMM with a non-contact sensor have been usedor proposed for the inspection of parts with free-form

surfaces [7] and [60]. The literature [60] discussed the

integration of vision and touch sensors in a CMM controller

used for inspection tasks. This vision-probe system was

used in a cooperative interaction integration mode. Global

information generated by the vision systems was used to

guide the movement of the touch probe. Vision provided

information about the positions of part features of interest,

and then the probe was guided to the features to make actual

measurements. Similar research on integrating multiple

sensors and vision probes with CMM in order to achieve

high measuring quality and speed can also be found in some

other literature such as [83] and [88]. In Ref. [88], a systembased on the combination of laser and CMM was used to

inspect parts made of soft material. In this research, a laser

Fig. 5. Non-contact measurement illustration [7].

Fig. 6. Three types of standard optical technology [7].




sensor replaced the probe so as to eliminate the possible

deflection of the component being measured when using

contact probes.

In acquiring free-form surface information by visual

devices, it may be necessary in many cases to acquire

several images and register them together to construct a

complete surface description [56] and [77]. This case has

been well studied in surface acquiring and reconstruction

and will not be covered in this review. This review will

mainly concentrate on the free-form surface inspection

techniques.

3. Surface description methods for localization

and comparison

The description of the 3D shapes or surfaces for

localization is a very basic task. All subsequent operations

are based on it. So in this section, we will give a brief review

about the geometric modeling methods used by existing

inspection techniques, in light of both contact and non-

contact measurement data.

Earlier works in the localization described the objects by

using certain primitives such as point, line and plane, and/or

polyhedral approximations. Faugeras et al. [17] used surface

representation for recognizing and localization of 3D

objects. The surface description was in terms of points,

curves and surface patches. In detail, points were corners on

the surface or centers of symmetry, curves were eitherinternal boundaries or symmetry axes, surface patches were

either planes or quadrics. Detailed algorithms were given

for constructing the representation from range data. In

Ref. [22], the objects were modeled as polyhedral or

polygons. The authors assumed that the objects could be

described by sets of planar faces. Only the individual plane

equations and a polygon embedded in each face is required.

The model faces do not have to be connected and the model

does not have to be complete. To approximate the objects by

polyhedral, assumption was made to the sensed data that the

measured positions were within a known error volume and

the measured surface orientations were within a known error

zone. Gunnarson and Prinz also worked on the localizationof objects described as polyhedral [24].

Later research works applied higher order forms of

surface representation. Many approaches applied parametric

description of the surfaces, such as parametric splines,

Coons, Bezier, B-spline and NURBS surface models.

Patrikalakis et al. [63] used NURBS surface model to

represent the free-form surface patch in localization. In this

research, the positional tolerance was represented in terms

of ball-offset tolerance regions, and the tolerance region

bounding surfaces were represented by NURBS patches.

There were 10 parametric surface patches: two normal offset

surfaces, four canal surfaces and four spherical patches.

This kind of regions was for evaluating if measurementsurface was within the tolerance. In Ref. [54],

the measurement surface was described by a general

parametric surface, a NURBS patch or a number of discrete

points, while the design surface was represented in NURBS.

Jinkerson [74] discussed the localization of marine

propellers that were modeled by NURBS description.

Sahoo et al. [49] discussed the localization of 3D objects

having sculptured surface. No assumption was made to the

surfaces of the part. The surfaces could be modeled as

planner, quadric, parametric polynomial, or any implicit/

parametric form. In Ref. [59], a surface was represented in

parametric form. In Refs. [91,92], the free-form surfaces

being inspected were represented by bi-cubic parametric

Spline models based on the discrete measurement point

data. It was pointed out that the algorithms were suitable for

any smooth parametric models. In Ref. [52], the measured

free-form surfaces of sheet metal parts were approximatedby C 2 continuity parametric surfaces with the specified

tolerance based on least square principle. Initially, the

control points and parameters of the surface were estimated

by a bilinear Coons surface. The parameters were further

adjusted by using bi-cubic B-spline surface. The above

surface estimation constructed an iterative process. In the

CAD-based free-form shape measurement and path plan-

ning research introduced in Ref. [2], the design model was

defined as NURBS surface patches. For localization, the

authors used the offset surface of the original CAD model

with the offset distance equals to the radius of the CMMprobe. The offset surface was obtained through three steps:

sampling points from the original CAD model; offsettingthese points along the surface normal direction; fitting

NURBS surface in least square. The fitting was iteratively

achieved within a user-defined tolerance. Approaches

discussed in Refs. [65,70–73] introduced visual inspection

of common mechanical parts. The design model in these

researches was in the format of IGES or STL, all the

surfaces on the design model were described as NURBS

patches. The measured 3D point clouds were segmented by

computing the distance between every 3D point and all of

the surfaces in the design model and by comparing some

local geometric properties between each 3D point in the

point cloud and its closest point on the design surface. The

local geometric properties used in above researchesincluded normal of the surface, Gaussian and mean

curvatures. The local geometric properties at an arbitrary

point P of 3D point cloud were obtained from the estimated

first and second partial derivatives at P: This was

accomplished by using a parametric second order poly-

nomial, which was obtained by using an N X N neighbor-

hood around P: Bispo et al. [16] investigated the inspection

of free-form surfaces by using dense range image, B-spline

model was used to describe the design model. In Ref. [48]

about optical measurement of free-form surface, a number

of point sets were captured, each of them was modeled as

B-spline surface patch. Then the overall surface was

reconstructed based on these patches, with the first patchas the reference in the reconstruction. Huang et al. [29]




presented a method for evaluating the profile tolerance of

free-form curves and surfaces expressed in parametric form.

Input of the research could be from CMM or laser scanning.

The surface model was created by using B-spline.

In most of above approaches, the traditional surface

description methods such as B-spline/Bezier /NURBS have

been widely used in modeling of the measurement data.

However, such kind of descriptions belongs to tensor

product surface model, they may suitable for the tactile

sensing measurement data that are normally ordered, with

lower density of the volume and the surface patch can be

easily sorted to rectangular topology. While there are

limitations to the tensor product surface models in

describing measurement data from visual sensing

processes, which is normally obtained as point clouds

with dense data volume, and always with arbitrary topology.To describe this kind of measurement data by using tensor

product surface always results in manual interaction due to

the topology issue, and/or accuracy degrading in establish-

ing the surface model sometimes. For example, the

parameterization process for creating the tensor surface

model would cause shape deformations to the measurement

point mesh.

With the developments of non-contact measurement

methods, some recent research used the surface description

methods other than traditional approaches such as B-spline,

NURBS or others. In the approach of data acquisition andprocessing for inspection of industrial parts discussed in

Ref. [47], the data of geometric position and additionalinformation such as the so-called covariance matrices was

stored in a multi-dimensional image array attached to the

points. Triangular meshes were used in the further

processing steps. The mesh generation algorithm discarded

those triangles that the length of any edge exceeds a distance

threshold. The experimental visual measurement system

introduced in Ref. [9] used optical sensors to acquire part

geometric information. A surface description based on

planar faces was chosen as the common domain for design

model and sensed image representation. For this, the

triangulated surfaces of the CAD model were exported

from the CAD system with triangles grouped into faces. The

measured data from different views was firstly integratedinto a single triangulated representation, and it was then

segmented into planar regions by a region-growing

algorithm. For each face, some features were calculated

such as plane equation, normal vector, 3D boundary

polygon and, etc. which were used for creating correspon-

dence and localization process.

The non-contact measurement other than laser and

optical scanners was also carried out. In Ref. [25], the

reconstruction accuracy of industrial parts based on

measurement data from a medical CT scanner was studied.

The object surface was reconstructed by using a modified

marching cubes algorithm, and the reconstructed surface

was then approximated by triangular patches, which wereused for the comparison with design model.

In comparison, the parametric representations such as

B-spline, NURBS and etc consumes less space than the

mesh representation. Once the representation model of the

surface is established, the former representation results in

faster processing in calculation of the surface information

such as differential properties, etc. and in subsequent

operations such as searching corresponding points in the

localization and comparison processes. It is normally

more convenient and accurate to study the surface

properties based on the parametric models other than the

3D mesh representation. Besides, the parametric model

establishing process normally involves higher order

expressions, for example cubic, of the surface. This also

helps reduce the influence of noise contained in the

measurement data. However, the 3D mesh representation

can directly describe the complex shaped objectsespecially those with arbitrary topology and large quantity

of points very easily, which is an obvious obstacle for the

parametric representations.

Besides above mentioned surface description methods,

some researchers carried out free-form surface inspectionbased on implicit representation of the surfaces. In Ref. [11],

inspections in both 2D and 3D situations were studied. The

objects in 2D images were described by their silhouettes and

then represented by 2D implicit polynomial curves. 3D data

were represented by implicit polynomial surfaces. Another

approach for evaluating free-form curves and surfaces using

implicit polynomial equations was presented in Ref. [90]. In

this research, the measurement process of the free-formsurface was driven by design model. This required the

initial implicit polynomials (IP) model of the free-form

surfaces. Thepointssampled fromdesign model werefittedby

degree n IP:

F nð x; y; zÞ ¼X

0#iþ jþk #n

aijk xi y

j z

k ¼ 0 ð1Þ

The fitting process was based on so called 3L fitting

algorithm developed in Brown University, which involved

explicit, least-square computations. The fitted IP were then

used for localization and calculation of the differences. The

quality of the implicit representation of free-form surface is

crucial to the localization and comparison in inspection,while the description of the implicit model requires accurate

estimation of the polynomial coefficients based on themeasurement points. Fitting errors may become larger when

establish high order models, and the coefficients may not be

pose transformation invariant. Besides, in describing com-

plex shaped 3D objects in implicit form, a number of patches

may be necessary instead of a single representation. These

are the limitations of using implicit forms.

4. Localization techniques

Traditionally, localization is achieved by presenting thepart at a desired position and orientation, using special tools,




fixtures or other part presentation/orientation devices totally

dedicated for specific products. This kind of process is

usually costly, and time and effort are required to design and

manufacture new fixtures [24,49,4,13]. In recent practise,

localization has been carried out by mathematically aligning

the DCS to the measuring coordinate system by using some

initially measured data. This process allowed the use of low

precision but general purpose fixtures in flexible and small

batch manufacturing [24]. It has been formulated as the

minimization of the sum of the squared distances between

the measurement points and the design model with respect

to the transformation parameters [49,74,92,91].

In practical applications, localization can be regarded as

a two-step process: find the point– point corresponding

relationship between measurement and design surfaces;

and, solve the 3D rigid transformation between these twosurfaces to bring them into a common coordinate system. In

the following paragraphs, the relevant techniques are

discussed in detail. As the inspections were based on either

contact or non-contact measurements, the review on the

localization technique is carried out from these two groups

accordingly for convenience. It should be noted that this is

not to category or distinguish the localization techniques.

Actually the method used in one group may be suitable to

another.

4.1. Localization techniques in approaches handling

contact measurement data

Traditionally, datums are measured to establish a

reference frame for the part [66]. This is known as 3– 2– 1

approach. Three points are measured from the first datum to

establish a plane. Two points are measured from the second

datum to establish a second plane perpendicular to the first.

Finally, one point is measured from the last datum

perpendicular to the first two [81,66]. Fig. 7 shows a

traditional 3– 2– 1 approach for locating DCS. This

approach has shortcomings in that the parts are required to

have plane surfaces, and the result is very sensitive to

measurement and dimensional errors. Besides, because of

the manufacturing errors on the datums, the localizationquality is sensitive to the selection of those points, there is

no accurate result can be uniquely decided by choosing

these number of points.

Many approaches and methods have been developed for

localization with higher accuracy, efficiency and robustness.

Grimson and Lozano-Perez used local measurements of

positions and surface normal to identify and locate parts

modeled by polyhedral [22]. By using local constrains on

distances between faces, angles between face normal, and

angles (relative to surface normal) of vectors between

sensed points, this approach examined all hypotheses about

pairings between sensed data and object surfaces. Incon-

sistent pairings were discarded efficiently. Gunnarson and

Prinz [24] assumed the design surface was in the ideal

position and required an initial estimate of the actual

localization of the surface. The corresponding point pair

between measurement surface and design model wasdecided by searching for the closest point. Based on the

correspondence, transformation was decided and applied to

the design model. New correspondence was created based

on the new position and localization was solved in an

iterative process, which terminated with the satisfaction of

certain preset conditions. Based on the correspondence

between measurement and design surfaces, the transform-

ation was solved by the minimization of the squared

distance between these two surfaces.

From Refs. [54,63] it is assumed that the correspondence

of edges and vertices between the design surface and the

measurement surface was known as a priori. Localization in

this research was an iterative process. In each step,localization departed from the position reached during

the previous step to provide an even closer position of the

measurement surface with respect to the design surface.

The transformation was solved separately by solving

rotation matrix C and translations based on the minimiz-

ation of the sum of squared distances between two surfaces.

Constraints were applied to C that ½C½CT ¼ ½CT ½C ¼ I3;

where I3 is the identity matrix of dimension 3, and the six

parameters (three rotational angles f ; u ;w and three

translation values t x; t y; t z) were decided by the minimization

of following objective function OF,

OFðf ; u ;w ; t x; t y; t zÞ ¼X I 21

i¼0

X J 21

j¼0

d 2r ij ¼ min ð2Þ

where d 2r ij is the minimum distance between an arbitrary

point of measurement surface (at existing position) and the

corresponding member on the design model represented by

a NURBS surface. This distance was calculated by using a

modified Newton algorithm.

Similar to Refs. [54,63], Jinkerson et al. [74] used the

minimization process to find the elements of transformation.

However, in this paper, besides the six parameters used in

Refs. [54,63], the authors provided a seventh parameter,

namely the offset distance, in the minimization calculation

for solving the transformation. From the experiments, theauthors declared that the use of seven parameters resulted inFig. 7. Traditional 3 –2 –1 approach for localization ([13]).




smaller RMS value after transformation compared with the

case that only six were used. The paper also extracted

the hydro-dynamically relevant features and dealt with the

impact of manufacturing errors on them. The correspon-

dence between the measurement surface and design model

in Refs. [49,13] was decided by choosing the closest point.

This required that the initial values of the transformation be

set close to final result or the two surfaces be close enough at

the initial stage. As indicated by Sahoo and Menq, when the

part deviation from its nominal position and orientation was

small, the transformation could be recovered very rapidly.

The transformation of localization was determined by the

minimization of the objective function F ;

F ¼ Xm

i¼1X

n

j¼1

lT 21

pij 2 qinl2

ð3Þ

where T is the rigid transformation matrix to be determined,

which contains the six variables, including three angles

f ; u ;w and three translation values t x; t y; t z. pij is the jth

measurement point on the ith surface patch, qij is the

corresponding point on design surface. All sixvariables were

obtained by solving the six non-linear equations generated

from minimization of F relative to each of these variables.

The non-linear equations were solved by the combination of

the Steepest-Descent method and the Newton-Raphson

method. As indicated in Ref. [13], solving non-linear

equations took an undesirable amount of computation time

due to the complex operations needed. The problemworsened when a large number of measurement points

were involved.Thus,Menq andhis co-workers also proposed

a modified algorithm. The new algorithm used the pseudo-

inverse method to determine the transformation matrix that

was approximated by a rigid body transformation. It was

claimed that the modified algorithm was 10 times faster.

For each measurement point, Balasubramanian et al. [4]chose the closest point from the design surface as the

corresponding point. The transformation was determined by

the minimization of the objective function, which was the

sum of the squared distance between the measurement and

design surfaces. The solution was achieved through a self-

learning neural network. The neural network was made tolearn the transformation in its homogenous form and the

relative transformation parameters were then determined.According to the authors, the application of neural network

helped to reduce computational cost and enhance the

robustness in localization.

In Refs. [91–93], a simple yet effective pseudo-inverse

approach used in Refs. [24] and [13] had been modified for

localization. A significant improvement in computing

efficiency had been demonstrated. In the research, two

major operations were clearly defined: creating correspon-

dence between the measurement and design surfaces; and

solving the transformation matrix. The correspondence

between the measurement surface and the design surfacewas determined based on the closest point concept. For an

arbitrary measurement point r pi and its closest point (defined

as the corresponding point) r i from the design surface, the

objective function to be minimized for solving the

transformation matrix T was:

F ¼Xmi¼1

f i ¼Xmi¼1

ðT 21

r p

i 2 r iÞT

ðT 21

r p

i 2 r iÞ ð4Þ

For a given set of point r pU representing measured surface

and the corresponding point set r U on the design model, theprocess to find the inverse of transformation matrix ðT21Þ

was straightforward:

T 21

¼ ðr U r pT U Þðr

p

U r pT U Þ

21ð5Þ

The following illustrates the procedures of the localization

process in the research:

Step 1: Using initial r U from the designed surface model

and r pU from the measured surface to calculate the

first transformation matrix T21:

Step 2: Calculate r pU (new): r pU (new) ¼ T21r pU :

Step 3: Calculate the sum of square deviations F from the

two sets of point r p

U (new) and r U .

Step 4: Check if F is minimized within a given tolerance. If

yes, stop the process and output the result;

otherwise, continue with the following steps.

Step 5: Using the transformed points r pU (new) of measured

surface and r U : from the designed surface to find

next approximation of T2

1:Step 6: Go to Step 2.

The above algorithm was extended to localization with

design datum in Ref. [91]. This case is reviewed in Section 6.

Li et al. [95] introduced a machined geometry estimation

and inspection simulation system for 3D parts with complex

shapes. For the inspection function, this system simulatedthe CMM measurement by shooting a number of rays to the

estimated 3D free-form surface; the measurement points

were those intersection points between the rays and the

estimated surface. The localization techniques developed in

Refs. [91,92] were applied.

Pahk [26] proposed a precision inspection method forparts with very thin and sharply curved features. The

localization procedures of this method had rough and finealignments. The rough alignment was based on the

conventional 6 points probing around the clear-cut surfaces

of a reference square block. The fine alignment used

iterative least-square techniques based on the measurement

data of the pressure/suction surfaces. The rough alignment

in this research is actually a 3-2-1 approach.

For localization of measured surfaces to the design

model in the free-form surfaces shape error evaluation

process of sheet metal parts introduced in Ref. [52], it was

assumed that the measurement surface was positioned to

roughly fit to the design model, which was guaranteed bymanual alignment. In this research, only rough localization




of those interested regions was required at the initial state.

The detailed correspondence was based on the point

matching between the measurement surface and design

model, which was based on the evaluation of the curvature

values at those points.

Ainsworth [30] discussed free-form surface inspection

using a CMM and used ICP (Iterative Closest Point) for

determining transformation. To provide a good estimationof transformation for cases that the design model and

measured part were initially grossly misaligned, manual

assistance was applied to find a number of corresponding

points from the measurement surface and the design model

to generate a rough alignment. ICP was then applied to the

subsequent registration.

The implicit polynomial based free-form surfaces

comparison was reported in Ref. [90]. This approachcalculated the errors between the measurement points and

the design model in two steps: fit IP using the points

sampled from design model, and then compute the

perpendicular distance from the implicit polynomial

model to measurement points.

It is found that, for the purpose of localization,

corresponding points between measurement surface and

design model need to be decided. The existing techniques

search the corresponding points mainly based on the idea of

choosing closest point between these two surfaces. This

imposes limitation to the localization process: the locali-

zation can work properly only when it starts from certain

initial condition that the measurement surface and designmodel are close enough in both 3D position and orientation.

To satisfy this requirement, some measures such as manual

assistance, and prior assumption of the transformation or a

number of estimations of preparing transformation is

required. Sometimes design datums are used for the

alignment for this purpose. All these result in lower

efficiency of the localization process, and unique solution

may not be guaranteed.

4.2. Localization techniques in approaches handling

non-contact measurement data

Compared with the contact measurement, non-contactmeasurement normally generates a larger volume of

measurement points. The measurement points may not be

organized and may contain more than one geometric

primitive. To establish the correspondence between the

measurement data and the design model, segmentation is

normally carried out to find possible primitives.

Bispo [16] investigated localization or the matching of

acquired free-form surface image data with the design

model. No salient features (such as planar regions) that

could help guide the registration were assumed. The

matching was based solely on the 3D points with an

estimation of the pose alignment. Studies on the accuracy of

the alignment and relative influence factors in this researchare covered in Section 4.4.

The automated visual inspection system introduced in

Ref. [84] for detecting defects of castings by using range

images was applicable to a class of objects containing planar

and/or quadric surfaces. Localization was based on the

assumption that the expected orientation of the parts was

known. Therefore, the system only needed to solve three

translation equations and one rotational equation.

Methods introduced in Refs. [65,70–73] conducted

visual inspections of common mechanical parts and made

comparisons between measurement surfaces and their CAD

models. The corresponding relationship was established

using a modified ICP process, which selected the corre-

sponding point based on the evaluation of distance between

potential corresponding points and the surface curvature

values (Gaussian curvature and mean curvature) at these

points. Transformation matrix T was found by usingquaternion representation, which was used or discussed in

Refs. [17,5,68]. The geometric models were described in the

NURBS format. Fig. 8 shows the inspection process.

The rotation matrix RðqÞ was expressed as:

RðqÞ ¼

q20 þq

212q

222q

23 2ðq1q22q0q3Þ 2ðq1q3 þq0q2Þ

2ðq1q2 þq0q3Þ q20 þq

222q

212q

23 2ðq2q3 þq0q1Þ

2ðq1q32q0q2Þ 2ðq2q3 þq0q1Þ q20 þq

232q

212q

22

26664

37775

ð6Þ

where q¼ ðq0;q1;q2;q3Þ; q0$0; q20 þq2

1 þq22 þq2

3 ¼1: The

optimal rotation matrix was given by the unit eigenvectorcorresponding to the largest eigen-value of the 4 £ 4 matrix,

whose components were generated by the covariance matrix

between the two point sets.

The inspection method based on implicit polynomial

expression introduced in Ref. [11] assumed that

Fig. 8. Block diagram of the inspection system [65].




the alignment between the measurement object and the

design model was done beforehand. The inspection activity

was then to model the template of the design model by an

implicit polynomial. Edges of the image of the measurement

object were extracted. Each edge point was tested if it was

inside tolerance values.

Pottmann et al. [27] made surface inspections and

comparisons by localizing 3D point clouds from laser

scanning to its CAD model, based on a modified ICP

method. The localization was an iterative process, which

was very similar to the ICP process. Suppose xi is one of the

points from the measured 3D point cloud, yi is the possible

corresponding point on the design model. In each iteration

of the localization process, instead of moving x i towards y i

which is the case of the standard ICP, the modified ICP

method moved xi towards the tangent plane of the designsurface at yi: It was claimed that the modified approach

converged much faster than the standard ICP approach. As

indicated by the author, for low curvature surface regions,

this difference on convergence was more obvious.

Fan and Tsai [48] used two CCD cameras and a laser

diode to acquire a number of free-form surface patches and

reconstructed the complete part surface based on these

image data. The shape error of the matching image of the

free-form surface was defined as the maximum value of the

nearest distance between the different patches. The initial

localization between different patches was carried out based

on human-computer interactions. Then the detailed local-

ization was solved based on the minimization of theobjective function, which was the sum of the squared

distance between the two surface patches to be studied.

In a visual inspection system for stampings with free-

form surfaces introduced in Ref. [75], localization was

actually a 2D operation for solving two translation variables

and one rotational variable. The translation was the planar

motion between the centers of the measured stamping image

and the reference model.

Guehring [47] treated two processes of registration and

localization. Registration of multiple views of measure-

ments was for surface reconstruction. Localization was to

align the reconstructed surface to the design model, for

comparison between those two surfaces. All these regis-trations were based on a modified ICP algorithm. The

corresponding points were defined as the point pair that was

close in both distance and normal directions:

kd 2 sk , t d and nsnd . t a ð7Þ

where d was a point from the design model; s was the

measurement point that potentially corresponds to point

d ; nd and ns were surface normal at these two points; t d

was distance threshold; and t s the cosine of an angular

threshold. Two types of correspondences were discarded:

corresponding points lying on the boundary and correspond-

ing points that were too far away. For solving the

transformation, the rotation matrix was expressed in unitquaternion, and the transformation was estimated based on

the minimization of the covariance-weighted sum of least-

square of the differences between corresponding points.

The visual inspection introduced in Ref. [80] dealt with

the robotic surface inspection of a space platform. In the

space environment, there are larger repeatability problems

due to robotic arm flexibility and object location changes

caused by thermal expansion and structural flexibility. The

reference image and inspected image were registered by

using a Gauss-Newton iterative method. The objective was

to find a suitable transformation of the reference image so

that the residual between the inspection data and trans-

formed reference image was minimized. The Gauss-Newton

algorithm solved this non-linear least-squares problem via

an iterative process, stopping when the least-squares

residual dropped below a pre-defined threshold.

Delingette et al. [25] reported the comparison betweenreconstructed surface mesh with the design mesh. The

correspondence and registration between these two meshes

were based on an ICP approach that iteratively selected the

closest points as the corresponding points and estimated the

best transformation, until a displacement threshold was

reached. To take into account the outliners, an algorithm

was implemented to remove the vertices that were located

too far away.

Similar to those approaches handling contact measure-

ment point data, most of the methods for the localization of

non-contact measurement point data discussed above alsodecided corresponding points by searching closest point.

ICP and/or modified ICP were commonly applied in theseapproaches. The same limitation discussed in previous

Section 4.1 also exists to these approaches in processing

non-contact measurement points.

Different from above approaches that used points directly

for deciding the correspondence, some methods selected

surface primitives or features and used these items to create

the corresponding relationship. Faugeras [17] represented

the surface in primitives and carried out localization in two

tasks: a recognition task to produce a list of corresponding

primitives from measurement range image and design

model; and, a locating task to calculate the transformation

to bring the design model to acquired range image. The first

task was implemented by using the tree search method. Thetransformation was decided by using quaternion. In

selecting the primitives for localization, the authors

recommended that line primitives should not be parallel

and planes should be independent. The minimum number of

primitives required for localization was listed in the

following Table 1. The localization mainly depended on

the existence of planar regions in the object being matched.

Based on the similar localization algorithms of [17], a

vision-based inspection approach presented in Ref. [1]

decided the correspondence by matching the segmented

primitives from the design model and measured image data.

In this research, the objects contained planar, cylindrical or

spherical faces, but only planar faces were used forlocalization. The estimation of transformation was treated




as a least-square minimization problem. If the estimation of

the rotational elements of the transformation matrix was not

a straightforward minimization, quaternion might be used.

Improvements were made in this approach based on the

method introduced in Ref. [17]. However, it has short-

coming that only planer primitives were used for the

localization, this requires a number of planer primitives onthe object studied for the localization.

The correspondence created between the design model

and measurement data in Ref. [9] was based on the

constrained tree search approach. In establishing the

correspondence, the process started with one matching

pair between the design model and the measurement image.

Once each possible pair was identified, the search went to

the next level. The search was a recursive process. In order

to control the search time, constraints were used to bind the

branching in the tree. So for each measurement feature, only

a subset of design model features was selected as possible

matches. During the search, the skipping of features was

allowed; if no correspondence was found for a certainmeasurement feature, this feature was removed from current

matching path and the search continued. Once the

correspondence was established, the rigid body transform-

ation between the measured data and the design model was

estimated.

Different from the approaches by using directly the

surface points in the corresponding and localization process,

the methods based on using attracted surface primitives or

features have the advantage of less user assistance

involvement. This is meaningful for improving efficiency

and automation level. However, the reliability and accuracy

of the primitive or feature extraction are crucial.

4.3. Comparison between the measurement surface

and the design model

The calculation of distance between the measurement

surface and the design model is an essential and critical

operation in the localization process. For most of the

existing approaches, localization is an iterative process and

the calculation of the distance between two surfaces is

required in every iteration. The distance calculation is the

main cost of the localization time. The comparison between

the measurement data and the design model is actually to

calculate the distance between them.

In calculating the distance between the measurementpoint to the design surface, represented by parametric

bi-cubic splines surface patch, Gunnarson and Prinz [24]

searched for the nearest point from the design surface for

each measurement point, and approximated the design

surface at this point with a plane. The distance between the

measurement point and the design model was calculated as

the point-to-plane distance.

To compare the measured points to design model,

Patrikalakis [63] firstly selected the maximum among all

minimum distances from the localized measurement surface

points to the design model. Then, this maximum distance

was used to verify if the measurement surface was within

the pre-defined bounding surfaces of the tolerance region. If

the measurement surface equation was known, the verifica-

tion was reduced to the interference detection between the

localized measurement surface and the bounding surfaces of

the tolerance region. A similar idea was used in Ref. [54].Sahoo and Menq [49,13] discussed two methods for

distance calculation based on the complexity of the part and

the type of surface representation. The first method was

Orthogonal Euclidian Distance, which was suitable for

surfaces represented in either parametric or implicit form.For a general bi-variate surface expressed in terms of

parameters u and v; the squared distance could be calculated

using:

D ¼ ð xmi 2 xÞ2

þ ð ymi 2 yÞ2

þ ð zmi 2 zÞ2

ð8Þ

where ð xmi; ymi; zmiÞ is an arbitrary measurement point,

ð x; y; zÞ is the corresponding point on design model. By

solving the following non-linear equations:

› D

›u¼ ð x2 xmiÞ

› x

›uþ ð y2 ymiÞ

› y

›uþ ð z2 zmiÞ

› z

›u¼ 0 ð9Þ

› D

›v¼ ð x2 xmiÞ

› x

›vþ ð y2 ymiÞ

› y

›vþ ð z2 zmiÞ

› z

›v¼ 0 ð10Þ

The parameters ðu;vÞ can be found to determine the

distance. The second method was Algebraic Distance,

which was suitable for surfaces represented in implicit form.

According to the authors, this method worked well for

surfaces of planar, quadric and lower order parametric

polynomials. For higher order surfaces, this method became

computationally expensive. Therefore, the OrthogonalEuclidian Distance method was recommended for higher

order surfaces [49]. This method was also used in Ref. [4]

for calculating the shortest distance between the measure-

ment points and the design surface. The solution was

achieved by solving non-linear equations.

In Refs. [91–93], for an arbitrary measurement point r pi ;

its closest point (defined as the corresponding point) r i from

the design surface, presented in parametric form, was

defined by:

ðT 21

r p

i 2 r iÞT ›r i

›u ¼ 0 ð11Þ

ðT 21

r p

i 2 r iÞT ›r i

›v ¼ 0 ð12Þ

Table 1

Minimum number of primitives needed to establish the transformation [17]

Translation Rotation

Points 3 3

Lines 2 2

Planes 3 2




where u and v were design surface parameters; T was

transformation matrix; and a modified Powell hybrid

approach was used to solve the non-linear equations.

Pahk and Ahn [26] evaluated the difference between the

measurement points and the design model in such a way

that, the correspondence was decided by the closest point

concept at first. Then, for every measurement point, the

corresponding point on the design surface was calculated

based on an iterative subdivision algorithm. Finally, the

deviations were obtained. For segmentation of 3D measure-

ment data, distance from 3D measurement point to a

NURBS surface was used in Refs. [65,70–73]. The distance

of a point to a NURBS surface was computed as finding a

point on the parametric surface such that the distance

between the 3D measurement point ~r and the point on

the design surface was minimal in the perpendiculardirection to the tangent plane at the point from the design

surface (refer to Fig. 9). The function to be minimized was:

minu0;v0k~r 2 ~sðu; vÞk2; which could be solved by an iterative

process.

Calculation of the difference between the measurement

data and the design model in Ref. [52] was divided into two

categories: local evaluation and global evaluation. The local

evaluation was the comparison between points based on the

value of their Extended Gaussian Curvature L defined as:

L ¼ lðk 01 2 k 1Þðk

02 2 k 2Þl ð13Þ

where k 01 and k 02 are the principle curvatures at the

measurement point; while k 1 and k 2 denote the principlecurvatures of the corresponding design point. A matchingrate function was designed to evaluate the local errors. The

global evaluation was to extract and evaluate the surface

features such as a bend or twist. The relationship between

the aggregate normal vectors of the surface features was

decomposed into a bent angle and a twisted angle. The

differences of bent and twisted angles constituted the global

evaluation results.

In Ref. [48], the nearest distance between two sets of

point clouds represented in B-spline surfaces was studied.

The nearest distance was defined as the intersection between

the surface normal from one of the two patches to another. A

so-called direct method was used, which was a minimi-

zation process based on the Newton-Raphson method.

Huang and Xiong [29] treated the calculation of distance

from the measurement points to the design surface as

solving a non-linear optimization problem. A laconic

algorithm was used to solve it. The calculation was an

iterative optimization process that converged to the solution

linearly.

Comparison between reconstructed mesh obtained from

CT scanning and the design model was made in Ref. [25]

based on several different distance criteria for evaluating

the shape similarity. The distance was between the vertexof the design model and its closest point on the

reconstructed model. The distance criteria were media

distance, maximum signed distance, and maximum and

median distance at the edges and the corner vertices. A set

of parameters relative to the processes from the 3D model

digitization to reconstruction of a 3D mesh were evaluated

for their impact on reconstruction accuracy. It was

concluded that this methodology was well suited for the

inspection of smoothly curved mechanical parts. However,

the resolution of the CT used in the research may create

some limitations for certain applications and a more

accurate scanning device, such as industrial CT scanner,may be necessary for some cases.

In summary, the difference calculation between measure-

ment surface and design model is actually the calculation of

the distance between measurement points to design surface,

which can be implemented by calculating the point-plane

distance such as the methods discussed in Refs. [24,65], etc.

or by directly calculating the point-point distance between

closest point pair/corresponding points between those two

surfaces. Experiment results show that the point– plane

distance calculation is faster than the point–point distance

calculation. For surface expressed in parametric model, the

problem of searching the plane or closest point from design

model can be solved by calculation or deciding the properparameter value. While the required parameter values can

be obtained by solving non-linear equations such as the

approaches discussed in Refs. [13,49,4,93], etc. or by some

other methods such as the subdivision method introduced in

Ref. [26].

For above two different approaches, namely method

based on solving non-linear equations and methods of

subdivision, the first one is normally faster, but may not

guarantee solution for all conditions, for example, for some

boundary points. The second approach has slower calcu-

lation speed, especially when the required subdivision step

is very small. The advantage of this method is that it canalways provide reasonable solution.

Fig. 9. Point to NURBS surface distance [65].




4.4. Measurement point selection, accuracy, efficiency

and robustness in localization

The selection of measurement points is an important

issue for inspection. In CMM measurement of free-form

surfaces, the selection and quantity of measurement points

have an important effect on inspection speed, localization

and the comparative processes. It is desirable to have an

inspection method that requires the fewest measurement

points but provides sufficient information for localization

and comparison between measurement surface and design

model.

A conventional way to define discrete measurement

points on a measured entity is to define a 2D rectangular grid

(parallel to one of the primary planes) within the 3D

model environment. Once the grid has been positioned andthe number of points on the grid specified, the grid points

are then projected onto the model surface, each point

of intersection between the projection lines and the model

will specify a measurement point. Fig. 10 shows such a

process [2]. The disadvantage of this method is that it is not

suitable for complex shapes, such as free-form surfaces with

high curvature.

In selecting the measurement points for localizing

polynomials, Gunnarson and Prinz [24] suggested measur-

ing those facets that have the greatest angular separation.

The points should also be spread over the surface to

facilitate accurate orientation estimation. Sahoo et al. [49]

suggested that one should try to choose as many points onthe planar or less complicated surfaces for faster processing

speed. In the localization process by visual measurement

studied in Ref. [16], the sampled points at multi-scale from

both the design model and measurement point set were used

for localization based on ICP. After the convergence using

coarser grids of data points, better accuracy was claimed by

running an extra iteration of the ICP algorithm with more

data points. Ainsworth [2] decided to choose the measure-

ment point based on adaptive subdivision sampling of the

design model. Several surface sampling criteria were

identified and implemented, including: uniform sampling

in the u and v parameter directions; chord height criterion;

minimum sample density criterion; and, parameterization-

based sampling criterion. These criteria can be used

individually or in combination. Research has also been

carried out regarding measurement point selection and

quantity, and their influences on the accuracy, efficiency and

robustness of the localization process. In Ref. [13], a

statistical analysis was used to determine the number of

measurement points for a surface profile based on the

tolerance specification and manufacturing accuracy. The

authors demonstrated that the number of measurement

points depended not only on the design specifications butalso on various factors in design and manufacturing. Design

specifications and manufacturing information can be

brought together to determine the desired inspection

strategy via statistical analysis. For localization based on

the iterative techniques, accuracy has a direct relationship

with the quantity of measurement points. As observed in

Refs. [13,96,81], the more the measurement points, the

more accurately the localization can be reached, or the more

the registration accuracy can be greatly increased [2]. It was

also indicated in Ref. [4] that the average transformation

error would be reduced as the number of points increased.For dense data, the transformation error would be very small

compared to the average orthogonal error. Based on theexperimental results, Huang et al. [91] concluded that

increasing the number of measurement points could

improve the accuracy in localization.

It should be noted that increasing the point quantity

might not always give positive influence to the localization

process. More points results heaver distance calculation

between measurement point to design model, this is obvious

for many of the localization process that the corresponding

searching and transformation solving is solved in an

iterative process. Besides, noise contained in the points

always cause negative influence to the accuracy of the

correspondence, while for most of the localization

approaches, the transformation information is obtainedbased on the corresponding point sets from both measure-

ment and design surfaces. Adding more points with higher

lever of noise will worsen the localization accuracy instead

of improving. Therefore, judgement measures need to be

considered, such as what kind of points helps improve the

localization accuracy and what kind of points should be

avoided.

The accuracy, efficiency and robustness of the localiz-

ation process are also influenced by many other factors

besides the point quantity. Some of the researchers

provided suggestions and comments based on their

experiments. Gunnarson and Prinz [24] expected, based

on the localization of objects represented in polyhedra, thatlocalization inaccuracy was on the same order asFig. 10. Conventional method for generating measurement points [2].




the combined surface roughness and sensor uncertainty.

Sahoo [49] concluded that the speed of the minimization

algorithm depended on the rate at which distance could be

calculated in the minimization process. In applying the

localization algorithm to actual objects, points on the planar

or less complicated surfaces were preferred because the

distance calculation for these surfaces was faster.

When measuring a surface using CMM, the selection of measurement points is arbitrary. Singularity might occur.

Dimensional errors of the selected measurement points have

a direct influence on the accuracy of the transformation. In

Refs. [49,13], the robustness of the localization algorithm,

as related to the above influence factors, was discussed. One

way to measure the sensitivity of the objective function F to

a small variation in the six transformation variables was

introduced by examining the singular values of a sensitivitymatrix. Moreover, a sensitive measure, based on the

smallest singular value, was proposed and used to estimate

the transformation errors.

For inspection accuracy when using a laser scanning

technique, Marshall [1] concluded from experiments that

the depth information obtained from edges was less accurate

than face data. In Ref. [16], a number of experiments had

also been conducted on the transformation accuracy for

localization/registration in visual inspection. These exper-

iments were run for different values of rotation and

translation, and for different levels of noise. The results

showed that the maximum accuracy that could be achieved

for a given level of noise corresponded to the combinedeffects of the sensor, modeling errors and the number of

matched points. The error in the translation estimation

decreased almost inversely with the number of matched

points. The initial translation estimation had influence on

the final registration accuracy. Based on their and others’

experimental results, they concluded that when the

measurement data set covered a reasonable portion of the

design model, the ICP performance was reasonably

insensitive to the original translation estimation. However,

final accuracy might change considerably and the situation

even worsened when the measurement data covered only a

small portion of the design model. Prieto et al. [73]

indicated that the size of defects, which could be detected,depended in a significant way on the resolution of the

sensor. The authors experimented with a 3D measurement

sensor that had the capability of detecting as small a defect

as 0.05 mm of depth.

4.5. Brief discussion on the iterative closest point (ICP)

method

In almost all of the localization approaches introduced in

Sections 4.1 and Section 4.2, the corresponding relationship

between two surfaces was determined by choosing the

closest point set. As an important and essential step of

localization, the solving of the 3D transformation matrixwas closely related to and depended on the corresponding

relationships. If the selection of the corresponding points is

determined by choosing closest points, as practised in the

above approaches, new corresponding relationships can

again be established after applying the transformation to the

measurement surface. Therefore, the operations of solving

the 3D transformation matrix T and the finding of the

corresponding relationships constitute an iterative locali-

zation process. This iterative process has been called ICP

[68] [15] and has been commonly used by many of the

aforementioned approaches.

ICP tries to find the minimum difference value between

two surfaces. However, the parameter space explored by

ICP can contain many minima [76]. Therefore, measures

need to be taken to avoid converging to a non-optimal

solution. In actual applications, the surfaces being compared

need to be located close enough before applying ICP.Preprocessing or user interaction is required for this

purpose. Some researchers used a number of initial

estimations of registration or localization for finding the

global solution [72].

The ICP method has also been commonly used in surface

reconstruction and surface patches registration processes.

As mentioned in Refs. [76,15], the ICP approach required a

good initial estimation of the transformation that was

reasonably close enough to the true value in order to

converge to a global minimization. Some researchers

provided this initial estimation by using a mechanicaldevice, such as a turn table [56]. There were other methods

that solved this problem: initial guesses of the transform-ation matrix T ; assuming a priori knowledge of an estimated

pose that aligned the different range images and approxi-

mated one of the surfaces being matched using its tangent

planes [94]; or, modification of the ICP process to find a

better initial 3D transformation [23,18,10].

Some researchers also used features either alone or

together with point positions. Chua [12] used principle

curvatures and direction constrains. An approach called

ICPIF has recently been proposed that used shape features in

conjunction with point positions in ICP to register range

images Chua [19]. The weighted invariant features,

including principle curvatures, moment invariants and

spherical harmonics invariants, together with the positionalinformation were combined to construct a distance function

for choosing the closest point in registration.

Another potential shortcoming of the traditional ICP-

based registration is that the transformation estimation is

done by calculating mean square (MS) distance. This was not

robust to outliner points generated either by noise or by the

presence of other parts in the scene [73]. Some methods were

proposed to overcome this problem. Masuda [82] estimated

the transformation between tworange images in a robustway

by fusing the ICP algorithmwith random sampling and least-

median of the square (LMS) estimation. Registration can

then be achieved with a high degree of robustness.

In general, although ICP and its variations are thedominant methods for localization, this approach has an




obvious limitation in that two surfaces in localization must

be initially located close enough in order to determine the

corresponding points. In practical applications, the two

surfaces may have arbitrary positions and orientations in 3D

space, which makes the determination of the corresponding

relationships difficult.

4.6. Discussion on solving 3D transformation

In previous Sections 4.1 and Section 4.2, we mentioned

some techniques for solving the rigid body transformation

for localization process, such as the method of solving non-

linear equations introduced in Refs. [49,80] and the pseudo-

inverse method discussed in Ref. [13]; the minimization

method illustrated in Refs. [54,63,74]; the minimization

approached solved by a self-learning neural network by

Ref. [4]; the modified pseudo-inverse approach developed

in Refs. [92,93]; the method of using quaternion in Refs.

[17,65,73], etc. For above approaches, the method of

solving non-linear equations normally is costly regardingthe computation time, especially when more points are

involved. The pseudo-inverse approach and the self-

learning neural network assisted method improve the

processing speed, while the modified pseudo-inverse further

reduced the processing cost in solving the transformation.

The methods by using quaternion or unit quaternion have

obvious advantages and this will be covered below together

with some other techniques.

Solving 3D rigid body transformation is also a critical

issue in image registration, surface reconstruction and

robotics. There is a common problem of finding the 3D

transformation between two sets of corresponding points.

For example, to register two different image patches orsurface patches together for constructing a complete image

or surface, it is necessary to apply 3D transformation to one

of the image or surface patches to align it to another. In

robotics, measurements in a camera coordinate system need

to be transformed to coordinates in a system attached to a

manipulator. Finding the proper 3D transformation is one of

the major tasks. The solution methodology has a direct

influence on registration accuracy and speed.

Since measurement always contains errors, it is not

possible to find a transformation that maps the measured

coordinates of points in one system exactly to those in

another system. In practice, the transformation is usually

determined by minimizing the sum of squares of residualerrors. There were some iterative methods, such as

empirical, graphical and numerical methods, which could

give an approximate solution leading to a better, but still

imperfect, answer. The iterative process requires an initial

estimation of the transformation.

Horn reviewed analytical methodologies that could

provide more efficient, robust and reliable solutions. He

proposed a closed-form solution for this problem [5]. In this

method, the unit quaternions were used to represent

rotations. The translation offset was the difference between

the centroid of the coordinates in one system and the rotated

centroid of the coordinates in another system. The unit

quaternion representing the rotation was the eigenvector

associated with the most positive eigen-value of a

symmetric 4 £ 4 matrix N : The elements of this matrix

were simple combinations of sums of products of corre-

sponding coordinates of the points:

where S xx ¼P

ni¼1 x0

l;i x0r ;i; S xy ¼

Pni¼1 x0

l;i y0r ;i and so on; and,

ð x0l;i; y

0l;i; z

0l;iÞ and ð x0

r ;i; y0r ;i; z

0r ;iÞ are the coordinates of the

corresponding point pair from those two set of points. To

find the eigen-values, a quartic equation has to be solvedwhose coefficients are sums of products of the elements of

the matrix. The minimum number of the corresponding

point pairs required to determine the transformation is three,

and the points should be non-collinear.

Arun [53] introduced a closed-form solution to least-

square minimization in order to estimate the transformation

between two corresponding point sets. The transformation

was represented by an orthonormal rotation matrix and a

translation vector. The rotation parameters were estimated

by a procedure based on the Singular Value Decomposition

(SVD) of a 3 £ 3 matrix. Horn et al. [6] proposed another

closed-form solution to solve the least-square problem in

estimating the transformation based on the use of anorthonormal matrix. The translation was represented as the

difference between the centroid of the coordinates in one

system and the rotated centroid of the coordinates in another

system; and, the rotation was represented by using

orthonormal matrices. This method depended on eigen-

value–eigenvector decomposition of a 3 £ 3 matrix and so

required the solution of a cubic equation. Upon comparing

the methods introduced in Refs. [5,6], the method based on

unit quaternions was more elegant, according to the authors.

Walker [87] presented an algorithm for estimating the

3D transformation using dual quaternions. The problem

was formulated as a dual number quaternion optimization

problem about a single cost function associated withthe sum of the orientation and position errors. There

N ¼

ðS xx þ S yy þ S zzÞ S yz 2 S zy S zx 2 S xz S xy 2 S yx

S yz 2 S zy ðS xx 2 S yy 2 S zzÞ S xy þ S yx S zx þ S xz

S zx 2 S xz S xy þ S yx ð2S xx þ S yy 2 S zzÞ S yz þ S zy

S xy 2 S yx S zx þ S xz S yz þ S zy ð2S xx 2 S yy þ S zzÞ

26666664

37777775

ð14Þ




are two parts of a dual quaternion q : q ¼ r þ 1s; where

r ¼ sinðu = 2Þn

cosðu = 2Þ; s ¼

ðd = 2Þcosðu = 2Þn þ sinðu = 2Þð p £ nÞ

2ðd = 2Þsinðu = 2Þ:

Based on the experiments, Lorusso [57] made a

comparative analysis on the four closed-form solutions

listed above, from Refs. [87,53,5,6], respectively. The

comparison was based on three issues. The first issue was

the accuracy of the algorithms for solving the transform-

ation using coordinates of the corresponding points with an

increasing amount of noise. The second issue was about the

stability of the algorithms when the original 3D point sets

degenerated into planes, lines and simple points. The third

issue was the relative efficiency in terms of actual

processing time by each algorithm. The comparison result

was summarized in Table 2.A closed-form solution for the least-square estimation of

transformation was also introduced in Ref. [86]. This

method was based on SVD and could be regarded as the

extension or refinement of the SVD approach introduced in

Ref. [53]. This new SVD method could provide a solution of

the transformation even if the input data was corrupted. The

method was applicable to any dimension. In application,

Kanatani [51] applied the SVD method from Ref. [53], the

unit quaternion approach introduced in Ref. [5] and the

orthonormal matrix method presented in Ref. [6] in a refined

manner. The author tried to solve the 3D transformation

based on available correspondence in a concise way based

on the so-called optimal resolution of degenerate rotation. A

covariance matrix of rotation fitting was defined, and the

statistical analysis for the fitting error based on the

covariance matrixes was conducted.

Least-square minimization was used by all the solutions

listed above to estimate the transformation. It has become

the basic and main stream in solving this type of problem.

However, this may not be true in some researches. In

Ref. [64], an optimal rotation estimation algorithm was

proposed. This method was different from other approaches,

which was not based on least squares minimization. This

method first created a theoretical accuracy bound, and then

estimated the rotation by using an optimal solution thatattained that accuracy bound. This optimal solution was

non-linear and it was processed by using quaternions and

renormalization techniques. No translation was handled in

this research. The noise considered by this approach

included anisotropic and inhomogeneous noises. Based on

experiments, the authors claimed that this method was

superior over least square method.

5. Inspection planning techniques

5.1. CMM measurement planning

The major mechanical components of a CMM machine

consist of a CMM column, probe stylus and probe tip.

During inspection, the probe moves at high speed. Collision

can easily damage one of the CMM components. The

human inspector can monitor the inspection and modify the

inspection path manually to avoid collisions. This is,

however, time consuming and prone to error. Therefore,

automated planning for CMM inspection is an importantresearch topic.A hierarchical planning system was proposed in Ref. [59]

for automated dimensional inspection of complex shaped

parts with free-form surfaces using CMM. The system could

detect collisions of the probe tip, the stylus and the column.

The collision-free inspection path was automatically

generated, with the automated selection and modification

of the probe angle, based on a local accessibility analysis of

a given feature. The hierarchical planning and heuristic

matching has also been discussed in Ref. [28]. The authorsconcentrated on the collision-free inspection path when

using the CMM. The goal of the research was to automate

the process for the dimensional inspection of dies and moldswith complex surfaces. Issues of CMM path planning and

collision detection against probe tip, probe stylus and CMM

column trajectories were investigated.

A method for automatic generation of high-level

inspection plans of mechanical parts was proposed in

Ref. [3]. A high-level plan is a collection of the setups of the

part on the CMM, the features to be inspected in each setup,

the probes, and the probe orientations to be used for each

feature. A general method for computing accessibility was

presented, which is one of the most important geometric

constraints for inspection planning.

Spitz [81] presented the development of a fully

automated dimensional inspection planner for CMMs. Theplanner first generated a high-level plan that specified how

to set up the part on the CMM table. This plan was then

expanded to include detailed path plans and ultimately a

program for driving the CMM. The planner was

implemented and included an accessibility analysis module,

Table 2

Comparison of four closed-form solutions for solving 3D transformation [58]

Accuracy Stability Efficiency Comments Reference

SVD Almost same Stable – Good for accuracy and stability [53]

Unit quaternion Almost same Stable – Better than SVD for small data sets [5]

Orthonomal matrix Almost same Less stable Good for small data size – [6]Dual quaternion Almost same Poor Good for large data size Good for large data sets [87]




a high-level planner, a plan validator (through collision

detection), a simulator, and a path planner.

An object-oriented generative inspection planner was

introduced in Ref. [66], which included an inspection

process planner and an inspection path planner. Inspection

related information, such as dimensions, tolerances and

geometric items, was retrieved from STEP model libraries

and used in creating inspection process plans by applyinglinear planning techniques. The inspection process planning

result was used by the inspection path planner to generate

collision-free probe paths that contained two portions: a

global path and a local path. The global path is for the

connecting relationship between different features of the

part under inspection; while the local path is the probe

trajectory for measuring the individual features.

The CMM path planning in Ref. [2] took the CAD modeland the generated sampling points as the input. Design

model entities were used to define the regions of the objects

to be measured and the measurement was performed for

each entity in turn. In some cases, two or more entities were

measured as a single surface. The measurements were

arranged in unidirectional or bi-directional scans depending

on the surface types. The programmed path could also be

edited by users.

5.2. Laser and optical measurement planning

As with the planning of CMM measurements, issues

exist for non-contact inspections such as laser scanning andvision inspection. The visibility problem of vision inspec-

tion is very similar to the accessibility analysis for straightprobes [81]; and similarly, the accessibility in laser scanning

has an analogy with that of CMM-based inspection, or

machine vision-based inspection [50]. In sensor planning of

visual inspection, the objective is the automatic determi-

nation of the appropriate sensor settings when the require-

ments of the task are given [14]. Planning for laser scanning

inspection is the automatic generation of the number and

direction of scans, and of the scan paths.

In many of existing non-contact inspections including

vision and laser scanning, substantial human involvement

using laborious and time-consuming techniques is stillrequired [14,50]. Such processes are not cost effective and

lack flexibility. In recent years, some research has been

proposed to develop automated planning for visual or laser

inspections. Shen et al. [89] introduced a CAD-guided

camera positioning system to aid 3D part inspection by

structured light. The geometric information in the CAD

model and the camera model were used to plan camera

configurations that satisfied certain task constraints. Vision

sensor planning consisted of two main steps: find ‘flat’

patches and determine the viewpoints for each ‘flat’ patch.

The ‘flat’ patch was determined by any surface normal

vectors inside the patch and the patch’s average surface

normal vector. The viewpoints were determined from arecursive approach.

Three planning algorithms were proposed in Ref. [21] for

finding a set of sensing operations for the completely visual

measuring of the exposed surface of an object. The first

algorithm was for use in general cases. The second

algorithm was for the case when a plan was short and

generated quickly; while the third was for a longer plan and

higher data acquiring accuracy. According to the authors,

the above algorithms should be chosen based on application

requirements.

To find an optimal plan was intractable, so Lee et al. [50]

proposed algorithms for the automation of laser scanner

based inspection operations. The algorithms consisted of the

following operations: based on the constraints existing in a

laser scanning operation (such as satisfying the view angle,

the depth of view, checking interference with a part, and

avoiding collision with the probe), the generation of allpossible accessible directions at each sampled point on a

part surface; the calculation of the number of scans and the

most desired direction for each scan; and, the determination

of the scan path satisfying the minimum scan time.

Yang [14] presented an optimized sensor planning

system for active visual inspection of 3D models by

dimensioning a set of edge segments such that the active

vision errors were minimized and the sensor constraintswere satisfied. This research considered quantization errors

and displacement errors, which are considered inevitable in

active vision inspections. The combined density functions

of these errors are used as objective functions in the

optimizations. The constraints of resolution, focus, field-of-view, and visibility were used as the constraint functions.

The genetic algorithm was used for optimization so that the

resulting sensor setting would have the minimum active

vision errors and satisfy all sensor constraints. Given the

CAD model and the entities to be inspected, the active

visual inspection planning system was designed to obtain

the sensor setting that maximized the probabilities of a

required accuracy for each entity.

6. Inspection of free-form surface with datums

In manufacturing, tolerances are assigned to thesurfaces to control the quality of the free-form surfaces.

A tolerance is a specification that defines the acceptable

variation of dimension or geometry of an element. Two

types of tolerances are usually used: dimensional and

angular tolerances; and, geometric tolerances [70]. In

checking tolerance, some research used intensity images,

some used information from CMM. For parts with free-

form surfaces and a relatively few flat surfaces, the

geometric tolerances specified for this type of surface, or

the normally assigned tolerances, are the profile tolerances

of a surface. Such tolerances are not easily inspected by

the traditional techniques of metrology [70,91]. The profile

tolerances assigned can be classified into individual andrelated profile tolerances, based on whether it is designed




with a datum reference [91]. Research has been carried

out on the inspection of free-form surfaces assigned

tolerance with datum.

Ge et al. [20] proposed the tolerance specification

module and the comparative analysis module. The first

module was to specify tolerance information to the 3D CAD

model of a part. The comparative analysis module was to

construct datum reference frames and compare the

measurement data of an actual feature with its nominal

design, based on the tolerance requirements so that the

dimensional quality of the actual feature could be verified.

To construct the datum reference frames, a ‘best fit’ method

along with a statistical analysis were used to find the best

fitted datum feature, which was built based on the

measurement data by a non-linear least-square minimi-

zation approach. The objective function was the sum of squared normal deviations calculated between the measure-

ment points and the fitted features in normal direction. A

number of non-linear equations needed to be solved to

determine the parameters of the datum features. All the

datum reference frames were constructed by using two

elementary datum features: plane and cylinders. In detailed

comparative analysis, individual features and independent

datum features were verified first, and dependent datum

features, which were related features or used as datum

features, were verified second. While those related features

that were not used as datums were verified lastly. Prieto et al[70,71] defined and implemented a methodology to check

dimensional, angular, and geometric tolerances by using aset of 3D point clouds and a CAD model of the part. The 3D

point clouds were registered with the CAD model and then

segmented into different surface patches by using the CAD

model. Finally, the specified tolerances were verified. In

detailed tolerance checking for free-form surface, the 3D

measurement points after segmentation were registered with

the CAD model, and the perpendicular distance between

each 3D point and the NURBS surfaces was calculated. The

distribution of these distances was used to define the zone of

measured tolerance. Limitations exist to above method

since the surface inspection in practical engineering

applications is much more complex. The free-form surface

tolerance may have a relationship with more than onedatum, and these datums also deviate from their nominal or

their design models. The localizations of free-form surface

and the design datums influence to each other. This problem

was addressed in Ref. [91]. Huang et al. [91] extended the

relative algorithms on inspection of individual free-form

surface to cases of sculptured surfaces with design datums.

According to the authors, the inspection strategy for

tolerances with datums was different from that used in

individual tolerance inspection. There were two steps for

this case. The first step was the localization of the design

frame by using datum reference features instead of the free-

form surface itself. The second step was the re-localization

of the free-form surface within allowable ranges inside thedatum tolerance zone to minimize the deviations. In this

second, the localization results from the datum reference

frame were used as the initial values to check if the changes

are out of the tolerance of datums. In localizing the DCS

with datum frames, the research used the least-square

method and represented the homogeneous transformation

matrix T d by

T d ¼ ðr p

U r pT U Þðr

0U r

pT U Þ

21ð15Þ

where r pU represents a set of measurement points, and r U

represents the corresponding point set from design model.

The localization process was carried out from the datum

features one by one. The detailed process is shown in

Fig. 11. The deviation of a manufactured surface was

represented in two components: deterministic error and

random error. It was assumed that the random error was thedominant error; and, the tolerance verification was carried

out based on statistical analysis. The shortcoming of this

approach is that corresponding points from those design

datums have to be decided in order to solve the

transformation information in the first step localization.

This makes the process a little bit complicated.

7. Brief description of sample commercial inspection

packages

There are a number of commercial packages available for

inspection and comparison purposes. Some of these systemsare based on contact measurements such as CMM

measurement; the others are based on non-contact measure-

ments such as optical or laser scanners, or on both contact

and non-contact measurements. Most of these packages can

handle free-form surfaces. However, in many of the

systems, user interaction may be required. For example,

the two surfaces under inspection have to be set closeenough by manual operation. Earlier visual inspection

packages were reviewed in Ref. [85], such as the automobile

body dimensional inspection systems developed by General

Motors and Volkswagen, and the turbine blade inspection

system developed by Westinghouse. Some examples of the

existing commercial packages are discussed in the follow-ing paragraphs.

A geometric inspection system for the comparison of measured point sets with geometric models was introduced

in Ref. [31]. There were two stages in the inspection and

comparison process. In the first stage, an operator manually

moved geometric models to the measurement points and 3D

transformations were also made manually. In the second

stage, fine alignment was implemented to minimize the

deviation function. The system used the STL data format for

describing the geometric model. This was a stand-alone

system.

Ref. [32] reported a 3D optical scanning and inspection

system for measuring the shape and wear-out amount of tires. Ref. [33] showed a product called ‘SPV’, namely




Sitius Part Verification. The measurement points can be

obtained through CMM, laser scanners or optical scanners.

SPV was fully integrated into Unigraphics and CATIA

platforms. The package can be applied for the inspection of

many mechanical products including free-form shaped

parts.

The functions for product verification and inspection wasdescribed in Ref. [34]. The first function was the comparison

of scanned data from the physical part with the nominal

CAD model. The second function was the comparison

between two sets of scanned point clouds, or between a

dense point cloud and the CAD model. This inspection

service has some applications in automotive industry, such

as product part approval prior to first product shipment at

Chrysler, Ford and GM. In Ref. [35], the inspection system

acquired a dense point cloud from products. Then the point

cloud and the CAD surface geometry were registered with

or without the use of datum. Deviations of the point cloud

from the CAD model were identified.

Ref. [36] used a product function called ‘ComputerAided Inspection/Verification’ to scan prototypes or

production components with a laser scanner and to compare

the scanning results directly with the CAD model for

verification. The system generated display plots showing

color gradients corresponding to point deviation from the

nominal CAD surface.

With the requirements and developments of reverse

engineering and its applications, a number of packages havebeen developed, which, in addition to functions for creating

and manipulating reverse engineering models based on

scanned point data, have the functions of inspection and

comparison.

Some of the popular brand names on the market include:

Polyworks, RapidForm, Geomagic, Imageware, Metris and

SpatialAnalyzer [37–42]. These systems have similar

inspection and comparison procedures that input the design

model and measurement data, then align them into a

common coordinate system, and finally make comparisons

and output the inspection results. The methods for the

alignment between design model and measurement data by

these systems normally include a traditional 3-2-1 approach,semi-automatic and automatic processes such as best fit

Fig. 11. Calculation procedure for localization with datum [91].




and feature-based alignment. With the semi-automatic

processes, users need to make the initial alignment by

manually arranging the design model and measurement data

close enough. The systems, then, carry out the remainder of

the registration operations. Features are used for the

automatic alignment including planes, circles, lines, spheres

and/or some other quadratic surfaces. The inspection and

comparison results are outputted in a report and displayed

visually by color gradients.

8. Discussions and conclusions

There existed a number of literature reviews on visual

inspection and automated visual inspection techniques

based on the literature published prior to 1995. None of the published reviews were concerned with the contact

inspection approach such as CMM-based inspection, which

is still widely used in industrial inspection processes. This

paper provided a review of the existing free-form surface

inspection techniques. From a higher level viewpoint, this

paper covered both contact and non-contact measurement

based free-form surface inspection approaches. In this

review contact inspection was mainly focused on CMM

measurement, while non-contact inspection was based on

laser and optical scanning. Characteristics, advantages and

disadvantages of the inspection approaches were discussed.

This paper also reviewed all of the major processes and

techniques involved in free-form surface inspection. Theseincluded: the measurement data acquisition methods by

both contact and non-contact measurements; the measured

data and/or design model geometric modeling techniques;

and, the surface localization and registration methodologies

including the establishment of correspondence and 3D

transformation, and the comparison between the measure-

ment data and the design surface. Various practical issues

such as the measurement point selection, factors influencing

localization process, inspection planning, tolerance inspec-

tion and inspection with design datum were discussed. Some

existing commercial inspection packages were also briefly

reviewed.

Future research on free-form surface inspection willlikely focus on the development of inspection techniques

with higher accuracy, efficiency and robustness, and

reduced cost. The measurement efficiency of contact

measurement devices such as CMM will probably improve,

while still maintaining the high accuracy, by combining

CMM with non-contact sensors. The accuracy of non-

contact measurement approach is expected to be enhanced

by using higher accuracy sensors and optimizing the

measuring parameters. Increased speed and higher accuracy

of inspections should be fully explored. New algorithms are

also expected to be developed for the more robust and

efficient processing of various types of measurement data.

The decreased cost of these solutions is also an impor-tant issue. These are all essential factors for future

improvements of efficient and effective inspections for

higher quality product design and manufacturing.

Acknowledgements

The authors wish to thank the Natural Sciences and

Engineering Research Council of Canada (NSERC) for

providing this research with financial support.

References

[1] Marshall AD, Martin RR. Automatic inspection of mechanical parts

using geometric models and laser range finder data. Image Vis

Comput 1991;9(6):385–405.

[2] Ainsworth I, Ristic M, Brujic D, Visual CAD. Visual CAD based

measurement and path planning for free form shapes. Seventh Int

Conf Central Europe Comput Graphics, Visual Interact Digital

Media’99, Prague 99 A.D; 2000.

[3] Spyridi AJ. Automatic generation of high level inspection plans for

coordinate measuring machines. PhD Dissertation. University of

Southern California; 1994

[4] Balasubramanian S, Gu P. A neural network approach to localization

of parts with complex surfaces for precision inspection. Proc ASME

Comput Engng Conf 1995;1995.

[5] Horn BKP. Closed form solution of absolute orientation using unit

quaternions. J Optical Soc A 1987;4(4):629–42.

[6] Horn BKP, Hilden HM, Negahdaripour S. Closed form solution of

absolute orientation using orthonormal matrices. J Optical Soc A

1988;5(7):1127–35.

[7] http://www.brownandsharpe.com/introduction/ [8] http://www.brownandsharpe.com/mfg/mfg5/index.html

[9] Brenner C, Bohm J, Guhring J. Experimental measurement system for

industrial inspection of 3D parts. Proc of SPIE Vol. 3521, Techn Conf

3521 (3521–27); 1998.

[10] Dorai C, Weng J, Jain AK. Optimal registration of multiple range

views. Technical Report CPS 1994;94-36:11.

[11] Unsalan C. Aytul ERCIL, Automated tolerance inspection by implicit

polynomials. Proc ICIAP’99 Venice, Italy, Septmber; 1999. [pp.

1218–25].

[12] Chua CS, Jarvis R. 3-D free-form surface registration and object

recognition. Int J Comput Vis 1996;17:77–99.

[13] Menq CH, Yau HT, Lai GY. Automated precision measurement of

surface profile in CAD-directed inspection. IEEE Trans Robotics

Automation 1992;8(2):268–78.

[14] Yang CC, Ciarallo FW. Optimized sensor placement for active visualinspection. J Robotic Syst 2001;18(1):1–15.

[15] Zhang D. Harmonic shape image: a 3D free-form surface represen-

tation and its application in surface matching. Doctoral Dissertation,

Robotics Institute, Carnegie Mellon University; 1999

[16] Bispo EM, Fisher RB. Free-form surface matching for surface

inspection. IMA Conf Mathematics Surfaces; 1994. [pp. 119–36].

[17] Faugeras OD, Herbert M. The representation recognition and

localization of 3-D objects. Int J Robotic Res 1986;5(3):27–52.

[18] Blais G, Levine D. Registering multi-view range data to create 3D

computer objects. IEEE Trans PAMI 1995;17:820–4.

[19] Gregory C. ICP registration using invariant features. IEEE Trans

Pattern Anal Machine Intell, January 2002;24(1):90–102.

[20] Ge Q, Chen B, Smith B, Menq CH. Tolerance specification and

comparative analysis for computer-integrated dimensional inspection.

Int J Prod Res 1992;30(9):2173–97.[21] Tarbox GH, Gottschlich SN. Planning for complete sensor coverage in

inspection. Comput Vis Image Understanding 1995;61(1):84–111.


http://www.brownandsharpe.com/introduction/

http://www.brownandsharpe.com/mfg/mfg5/index.html

http://www.brownandsharpe.com/mfg/mfg5/index.html

http://www.brownandsharpe.com/introduction/



[22] Grimson WEL, Lozano-Perez T. Localizing overlapping parts by

searching an interpreting tree. IEEE Trans PAMI 1987;9(4):469–82.

[23] Turk G, Levoy M. Zipped polygon meshes from range images.

Comput Graphics (SIGGRAPH’94 Proceedings); 1994. [pp. 311 –18].[24] Gunnarson KT, Prinz FB. CAD model based localization of parts in

manufacturing. Computer 1987;20(8):66–74.

[25] Delingette H, Seguin O, Perrocheau R, Menegazzi P. Accuracy

evaluation of 3D reconstruction from ct-scan images for inspection of

industrial parts. In: Cepadues, editor. Proc Conf Quality Control

Artificial Vison (QCAV’97), Le Creusot, France, May. 1997.

[26] Pahk HJ, Ahn WJ. Precision inspection system for aircraft parts

having very thin features based on CAD/CAI integration. Int J Adv

Manufact Technol 1996;12:442–9.

[27] Pottmann H, Wallner J, Leopoldseder S. Kinematical methods for the

classification, reconstruction and inspection of surfaces. In: Proceed-

ings SMAI, Pompadour, France; 2001.

[28] Yau H, Menq C. Automated CMM path planning for dimensional

inspection of dies and molds having complex surfaces. Int J Machine

Tools Manufact 1995;35(6):861–76.[29] Huang X, Xiong Y. Profile tolerance evaluation of parametric curves

and surfaces. Chinese J Mech Engng 2002;15(2):121–5.

[30] Ainsworth I, Ristic M, Brujic D. CAD-Based Measurement Path

Planning for Free-form Shapes Using Contact Probes. Int J Adv


[31] http://www.aha.ru/labsapr/geocon.htm

[32] http://www.oes.itri.org.tw/s200/3d.htm^ fitting ^ of ^ free-form…

[33] http://www.sitius.com

[34] http://www.gks3d.com/service.htm

[35] http://www.firstart.com/inspect_procedure.html

[36] http://www.laserdesign.com/cav.htm

[37] http://www.innovmetric.com

[38] http://www.rapidform.com

[39] http://www.geomagic.com/advantage/testimonials/

[40] http://www.eds.com[41] http://www.metris.be/

[42] http://www.kinematics.com

[43] http://www.hymarc.com/index.html

[44] http://shapegrabber.com/

[45] http://www.minolta.com

[46] http://www.metronsys.com

[47] Guehing J. Reliable 3D surface acquisition, registration and validation

using statistical error models. Third Int Conf 3-D Digital Imaging

Modeling (3DIM’01), Quebec City, Canada, May 28–June1; 2001.

[48] Fan KC, Tsai TH. Optimal shape error analysis of the matching image

for a free-form surface. Robotics Comput Integ Manufact 2001;17:

215–22.

[49] Sahoo KC, Menq CH. Localization of 3-D objects having complex

sculptured surfaces using tactile sensing and surface description.

J Engng Indus 1991;113:85–92.[50] Lee KH, Park HP. Automated inspection planning of free-form shape

parts by laser scanning. Robotics Comput-Integ Manufact 2000;16(4).

[51] Kanatani K. Analysis of 3-D rotation fitting. IEEE Trans Pattern Anal

Machine Intell 1994;16(5).

[52] Kase K, Makinouchi A, Nakagawa T, Suzuki H, Kimura F. Shape

error evaluation method of free-form surfaces. Computer-Aided Des

1999;31:495–505.

[53] Arun KS, Huang TS, Blostein SD. Least-squares fitting of two 3-D

point sets. IEEE Trans Pattern Anal Machine Intell 1987;9(5).

[54] Bardis L, Jinkerson RA, Patrikalakis NM. Localization for automated

inspection of curved surfaces. Int J Offshore Polar Engng 1991;1(3):

228–34.

[55] Chen LC. Surface digitization techniques for reverse engineering,

http://www.cc.ntut.edu.tw/coop/atc/aoi/html2/tecedu.html, 2002

[56] Li L, Schemenauer N, Peng X, Zeng Y, Gu P. Design andimplementation of a reverse engineering system for rapid manufac-

turing 2002;18(1):53–67.

[57] Lorusso A, Eggert D, Fisher RB. A comparison of four algorithms for

estimating 3-D rigid transformations. Proc British Machine Vis Conf

BMVC95; 1995. [pp. 237–46].

[58] Bariani M, Cucchiara R, Mello P, Piccardi M. Data mining forautomated visual inspection, in Proc. of PADD’97. First Int Conf

Practical Appl Knowledge Discovery and Data Mining, London, UK;

1997. [(ISBN 0-9525554-7-6), pp. 51–64].

[59] Menq CH, Yau HT. Path planning for automated dimensional

inspection using coordinate measuring machines. Proc IEEE Int

Conf Robotics Automation, April 1991;3:1934–9.

[60] Nashman M, Hong TH, Rippey WG, Herman M. An integrated vision

touch-probe system for dimensional inspection tasks. Proc Int

Conf Intell Syst: a Semiotic Perspective, Gaithersburg, MD; 1996.

[pp. 196– 205].

[61] http://www.olympus.net/personal/mortenson/preview/definitionsf/

freeformsurface.html

[62] Siegel M, Gunatilake P. Remote inspection technologies for aircraft

skin inspection. Proc ET VS-IM, IEEE; 1997. [pp. 69–78].

[63] Patrikalakis NM, Bardis L. Localization of rational B-spline surfaces.Engng Comput 1991;7:237–52.

[64] Ohta N, Kanatant K. Optimal estimation of three-dimensional rotation

and reliability evaluation. IEICE Trans Inf Syst 1998;E82(D(11)).

[65] Boulanger P, Moron V, Redarce T. High-speed and non-contact

validation of rapid prototyping parts. SPIE’s Rapid Product

Development Technologies Boston, Mass. USA, Nov.18–19; 1996.

[pp. 46–60].

[66] Gu P, Chan K. Generative inspection process and probe path planning

for coordinate measuring machines. J Manufact Syst 1996;15(4):

240–255.

[67] Gunatilake P, Siegel M, Jordan A, Podnar G. Image understanding

algorithms for remote visual inspection of aircraft surfaces. Machine

Vis Appl Indus Inspection V 1997;3029:2–13.

[68] Besl PJ, McKay ND. A method for registration of 3-D shapes. IEEE

Trans Pattern Anal Machine Intell 1992;14(2):239–56.[69] Besl PJ. The free-form surface matching problem. Machine Vision for

Three-Dimensional Scenes. H. Freeman (Ed.), Academic Press Inc.,

CA, 1990. [pp. 25–71].

[70] Prieto F, Redarce T, Lepage R, Boulanger P. An automated inspection

system. Int J Adv Manufact Technol 2002;19:917–25.

[71] Prieto F, Lepage R, Boulanger P, Redarce T. Inspection of 3D parts

using high accuracy range data. SPIE Conf Electron Imaging:

Machine Vis Appl Indus Inspect VIII, (EI’ SPIE, San Jose, USA,

San Jose, USA, 23–28 January 2000, vol. 3966.; 2000. [pp. 82–93].

[72] Prieto F, Redarce T, Lepage R, Boulanger P. A non contact CAD-

based inspection system. Int Conf Quality Control Artif Vis

(QCAV’99), Canada: Trois-Rivieres; 1999. [pp. 133–38].

[73] Prieto F, Redarce HT, Lepage R, Boulanger P. Visual system for the

fast and automated inspection of 3D parts. Proc 6ieme

Assises Europeennes de Prototypage Rapide Paris, France; 1998.[pp. 211–27].

[74] Jinkerson RA, Abrams SL, Bardis L, Chryssostomidis C, Clement A,

Patrikaiskis NM, Wolter FE. Inspection and feature extraction of

marine propellers. J Ship Prod 1993;9(2):88–106.

[75] Langenbach R, Ohl A, Scharf P, Smmler J. Automatic visual

inspection system for small stampings with free-form surfaces. Proc

SPIE 2001;4301:9–19.

[76] Campbell RJ, Flynn PJ. A survey of free-form object representation

and recognition techniques. Comput Vis Image Understand 2001;81:

166–210.

[77] Rich Pito, Bajcsy, R., A solution to the next best view problem for

automated CAD model acquisition of free-form objects using range

cameras, in the modelling, simulation, and control technologies for

manufacturing session of the SPIE Symposium on Intelligent Systems

and Advanced Manufacturing, Philadelphia PA, October; 1995[78] Chin RT, Harlow CA. Automated visual inspection: a survey. IEEE

Trans Pattern Anal Machine Intell 1982;6:557–73.


http://www.aha.ru/labsapr/geocon.htm

http://www.sitius.com/

http://www.gks3d.com/service.htm

http://www.firstart.com/inspect_procedure.html

http://www.laserdesign.com/cav.htm

http://www.innovmetric.com/

http://www.rapidform.com/

http://www.geomagic.com/advantage/testimonials/

http://www.eds.com/

http://www.metris.be/

http://www.kinematics.com/

http://www.hymarc.com/index.html

http://shapegrabber.com/

http://www.minolta.com/

http://www.metronsys.com/

http://www.cc.ntut.edu.tw/coop/atc/aoi/html2/tecedu.html


http://www.olympus.net/personal/mortenson/preview/definitionsf/freeformsurface.html





http://www.metronsys.com/

http://www.minolta.com/

http://shapegrabber.com/

http://www.hymarc.com/index.html

http://www.kinematics.com/

http://www.metris.be/

http://www.eds.com/

http://www.geomagic.com/advantage/testimonials/

http://www.rapidform.com/

http://www.innovmetric.com/

http://www.laserdesign.com/cav.htm

http://www.firstart.com/inspect_procedure.html

http://www.gks3d.com/service.htm

http://www.sitius.com/

http://www.aha.ru/labsapr/geocon.htm



[79] Chin RT. Automated visual inspection: 1981 to 1987. Comput Vis,

Graphics Image Process 1988;41:346–81.

[80] Volpe R, Balaram J. Technology for robotic surface inspection in

space. Proc AIAA/NASA Conf Intell Tobots Field, Factory, ServiceSpace 1994;(CIRFFSS’94):21–4.

[81] Spitz SN. Dimensional inspection planning for coordinate measuring

machines, PhD Dissertation, University of Southern California; 1999

[82] Masuda T, Yokoya N. A robust method for registration and

segmentation of multiple range images. Comput Vis Image Under-

stand 1995;61(3):295–307.

[83] Shen T, Huang J, Menq CH. Multiple-sensor integration for rapid and

high-precision coordinate metrology. IEEE/ASME Trans Mechatron

2000;5(2):110–21.

[84] Newman TS, Jain AK. A system for 3D CAD-based inspection using

range images. Pattern Recogn 1995;28(10):1555–74.

[85] Newman TS, Jain AK. A survey of automated visual inspection.

Comput Vis Image Understand 1995;6(2):231–62.

[86] Umeyama S. Least-squares estimation of transformation parameters

between two point patterns. IEEE Trans Pattern Anal Machine Intell1991;13(4):376–80.

[87] Walker MW, Shao L, Volz RA. Estimating 3-D location parameters

using dual number quaternions. CVGIP: Image Understand 1991;

54(3):358–67.

[88] Cheng WL, Menq CH. Integrated laser/CMM system for the

dimensional inspection of objects made of soft material. Int J Adv


[89] Sheng W, Xi N, Song M, Chen Y, Rankin JS. Automated CAD-guided

automobile part dimensional inspection. Proc IEEE Int Conf Robotics

Automation; 2000. [pp.1157–62].

[90] Wolovich W, Albakri H, Yalcin H. The precise measurement of free-

form surfaces. J Manufact Sci Engng 2002;124(1).

[91] Huang X, Gu P. CAD-model based inspection of sculptured surfaces

with datums. Int J Prod Res 1998;36(5):1351–67.

[92] Huang X, Gu P, Zernicke R. Localization and comparison of two free-

form surfaces. Comput-Aided Des 1996;28(12):1017–22.

[93] Huang X, Gu P. CAD-model based inspection of sculptured surfaces.

Proc ASME Comput Engng Conf; 1995. [pp. 795–804].[94] Chen Y, Medioni G. Object modeling by registration of multiple range

images. Image Vis Comput 1992;10:145–55.

[95] Li Y, Gu P. Geometric estimation of 3D machined parts for internet-

based manufacturing system. Proc CIRP Conf Workshop, Hong

Kong, May; 2002.

[96] Li Z, Gou J, Chu Y. Geometric algorithms for workpiece localization.

IEEE Trans Robotics Automation 1998;14(6):864–78.

Peihua Gu is currently Professor and Head of

Mechanical and Manufacturing Engineering

Department at the University of Calgary. He

also holds NSERC Chair in Life Cycle Design

Engineering. His research interests include life

cycle design, rapid product realization andinspection and assembly.

Yadong Li is a PhD candidate at The

University of Calgary. His research interests

include CAD/CAM, Geometric Modeling,

Inspection of Free-form Surface and CNC

Programming/Machining Technology. He

received his BS and MS degrees from Tianjin

University. He had worked as mechanical

engineer on die/mold manufacturing in TON-

TEC Co., research assistant in The Chinese

University of Hong Kong and mechanical engineer in R&D department of

Makino Milling Machine Co., Ltd.


4. 2X_Free-Form Surface Inspection Techniques State of the Art Review

Documents

Transcript of 4. 2X_Free-Form Surface Inspection Techniques State of the Art Review