1/25

Dr.-Ing. Sung Joon Ahn

CurvSurf, Inc.

Industrial 3-D Augmented Reality

2/25

Content

1. CurvSurf, Inc.

2. 2-D AR vs. 3-D AR

3. 3-D Measurement

4. Objects in 3-D Space

5. Data Coverage

6. Least Squares

7. Industrial 3-D AR

8. Local Curvature Analysis

9. Closing.

3/25

CurvSurf, Inc.

• A software company for 3-D data processing

• CEO: Ahn, Sung Joon

• 2013– : CurvSurf, Inc., CEO

• 2005–2013: Sungkyunkwan University, associate professor

• 1990–2004: Fraunhofer IPA, research engineer

• 1985–1990: LG Electronics, research engineer.

• 1985: BS, Seoul National University

• 1987: MS, KAIST

• 2004: Dr.-Ing., University of Stuttgart.

4/25

2-D AR vs. 3-D AR

• 2-D AR

① Image capture & search

② Plane detection &

camera orientation

③ Information overlaid.

• 3-D AR

① 3-D capture (RGB-D)

② Object recognition and

measurement

③ Shape, size, position &

rotation of objects.

© Vuforia

© CurvSurf

5/25

Optical 3-D Measurement Techniques Stereo vision

Random pattern

projection

Stripe projection

6/25

3-D Data Processing

Touch Points

Initial Model

ODF: Orthogonal Distance Fitting

-Inlier Points

-Model Fitting

7/25

Applications of 3-D AR

8/25

Shape, Size, Position & Rotation (1)

• Curve/surface in space

• Conic section: Which a curve is this, ellipse?

𝑎𝑋2 + 𝑏𝑌2 + 𝑐𝑋𝑌 + 𝑑𝑋 + 𝑒𝑌 + 𝑓 = 0.

If so, axis lengths? Center? Pose angle?

• Quadratic surface: ? ? ? ? !

𝑎𝑋2 + 𝑏𝑌2 + 𝑐𝑍2 + 𝑑𝑋𝑌 + 𝑒𝑌𝑍 + 𝑓𝑍𝑋 + 𝑔𝑋 + ℎ𝑌 + 𝑖𝑍 + 𝑗 = 0

• Cubic, quartic, catenary, 3-D circle, helix, sine curve, … … … .

• 2-D Line: 𝑋𝑌

= 𝑢cos𝜅sin𝜅

+ 𝑋

𝑌 , decomposed to

𝑋𝑌

=cos𝜅 −sin𝜅sin𝜅 cos𝜅

𝑥𝑦 + 𝑋

𝑌 Coordinate transformation:

𝐗=𝐑−1𝐱+𝐗

and 𝑦 = 0 or 𝑥𝑦 =

𝑢0

Canonical form:

𝑓 𝑥,𝑦 =0 or 𝑥(𝑢)𝑦(𝑢)

.

9/25

Shape, Size, Position & Rotation (2)

• Parameter decomposition (cf. camera parameters in Photogrammetry)

• Canonical form: Specific to a model curve/surface 𝐗 = 𝐑−1𝐱 + 𝐗o: Common to all curves and surfaces in space

• Intrinsic: Shape parameters (size, radius, vertex angle, etc.)

Extrinsic: Position & rotation parameters (rigid body

motion).

Canonical form of circle

(𝑋 − 𝑋o)2+(𝑌 − 𝑌o)

2−𝑟2 = 0.

+

Don’t

combine!

10/25

Shape, Size, Position & Rotation (3)

• Examples of canonical form (Implementation of ISO 10360-6)

• 2-D line : 𝑦 = 0

• 3-D line : (0, 0, 𝑢)

• Plane : 𝑧 = 0

• 2-D circle : 𝑥2 + 𝑦2 − 𝑟2 = 0

• 3-D circle : (𝑟cos𝑢, 𝑟sin𝑢, 0)

• Sphere : 𝑥2 + 𝑦2 + 𝑧2 − 𝑟2 = 0

• Cylinder : 𝑥2 + 𝑦2 − 𝑟2 = 0

• Cone : 𝑥2 + 𝑦2 − (𝑟 − 𝑧tan(𝜓/2))2= 0

• Torus : 𝑥4 + 𝑦4 + 𝑧4 + 2 𝑥2𝑦2 + 𝑦2𝑧2 + 𝑧2𝑥2 −2 𝑅2 + 𝑟2 𝑥2 + 𝑦2 + 2 𝑅2 − 𝑟2 𝑧2 + 𝑅2 − 𝑟2 2 = 0.

2-D/3-D curve: Ellipse, hyperbola, parabola, sine curve, catenary, helix, ...

Surface:

Ellipsoid, hyperboloid, paraboloid, …

11/25

Shape, Size, Position & Rotation (4)

• Position of cylinder/cone (cf. position of line: 𝐗 )

• A point on axis

• The competent one is the closest point to the mass center 𝐗

of data points, resulting in minimum variances because the

feature extracted is adhering to data points.

12/25

Object

circle/

cylinder

Data Coverage by 3-D Camera

• Data coverage is less than 50% of the object surface

• Conventional least squares fitting algorithms have difficulties

in processing such data set

• E.g. circle/cylinder fitting results in smaller radius than

expected.

Data point

Conventional

least squares fitting

Orthogonal

distance fitting

13/25

Least Squares Curve/Surface Fitting

• Best error measure:

The minimum (shortest/orthogonal/normal) distance

between measurement point and model curve/surface

(Invariant to coordinate transformation)

• Twofold nonlinear minimization problem

• Find the minimum distance points on model curve/surface

• Minimize the sum of the squared minimum distances

through adjusting the model parameters.

14/25

Moment Method for Line/Plane

• Alias: Eigenvalue decomposition, total least squares,

principal component analysis (cf. linear regression)

• Line/plane fitting:

• Minimize the sum of the squared normal distances between

the measurement points and the model line/plane

• Linear problem: Solved in closed form (Pearson 1901).

Invariant to

coordinate

transformation

15/25

Orthogonal Distance Fitting (1)

• Alias: Best fitting, Euclidean fitting, geometric fitting

• Curve/surface fitting:

• Minimize the sum of the squared shortest distances between

the measurement points and the model curve/surface

• Nonlinear problem: Solved through iteration (Boggs 1987)

• S.J. Ahn, doctoral diss., LNCS 3151, Springer-Verlag, 2004.

16/25

Doctoral dissertation

University of Stuttgart

2004

Orthogonal Distance Fitting (2)

17/25

Industrial 3-D AR

• Industrial environment

• Object surfaces are composed of planes, boxes, spheres,

cylinders, cones, and tori.

© CurvSurf

18/25

Local Curvature Analysis (1)

• Initialization of model parameter values

• Type of feature: Plane, sphere, cylinder, cone, or torus,

determined by comparing the signed curvature radii

• Feature parameters: Radius, position, and axis direction,

derived from the principal curvatures and axes.

Plane Sphere (torus) Cylinder (cone) Torus

* k1, k2 : Principal curvatures

19/25

Local Curvature Analysis (2)

• Initial values for nonlinear model fitting

• Radius of sphere/cylinder = |𝑟1| Position of sphere/cylinder = 𝐗f − 𝑟1𝐧 = 𝐗o

Axis direction of cylinder = 𝐯

• Cone and torus:

Beyond this talk’s

scope.

𝐗f : Surface point

𝐧 : Surface normal

𝐮, 𝐯 : Principal axes

𝑟1, 𝑟2 : Curvature radii

= 1 𝑘1 , 1 𝑘2 .

|𝑟1| ≤ |𝑟2|.

20/25

Local Curvature Analysis (3) if planar {

fit a plane;

if there is/are orthogonal plane(s) { fit a box; }

}

else if parabolic {

fit a cone;

if the vertex angle is small { fit a cylinder; }

}

else if elliptic {

fit a torus;

if the mean radius is small { fit a sphere; }

else if the mean radius is large { fit a cylinder; }

}

else fit a torus;

21/25

Middleware FindSurface

© CurvSurf

22/25

• FindSurface with AI

• Input to AI:

Shapes, sizes, positions & rotations of individual

features extracted and measured by ‘FindSurface’

• Output from AI:

List of probable things, e.g. table, chair, bed, ...

Shapes, sizes,

positions & rotations

of individual features

RGB-D

Future Developments (1)

FindSurface

Thing identification

& description AI

23/25

Future Developments (2)

• On-line data processing service

• Google’s business is based on

search & analysis of

1-D text arrays and

2-D images

• CurvSurf’s business will be extended to

search & analysis of

3-D spaces.

24/25

Closing (1)

• The seeing is not the understanding

• Human being sees and understands a scene by using

stereoscopic view and brain

• A computing machine (processing agent) runs

around, touches, and analyzes the 3-D point cloud

• Eventually, the machine will beat human being in 3-D

data processing.

25/25

Closing (2)

• Live demonstration with:

• Google Tango

• Microsoft HoloLens.

Thank you!