1/20

Obtaining Shape from Scanning Electron Microscope Using Hopfield Neural Network

Yuji Iwahori1, Haruki Kawanaka1, Shinji Fukui2 and Kenji Funahashi11 Nagoya Institute of Technology, Japan

2 Aichi University of Education, Japan

2/20

Introduction

Shape from Scanning Electron Microscope (SEM) images is the recent topic in computer vision.The position of a light source and a viewing point are the same under the orthographic projection. The object stand is rotated to some extent through the observation.

Only these conditions can be used to recover the object shape.

2D Image of SEM

Recovering 3D Shape

3/20

Previous Approaches (1)

Photometric Stereo

Estimation using the temporal color space

use multiple images under the different light source directions.

Linear Shape from Shading

Photometric Motionthe position of viewing point (camera) and light source should be widely located

4/20

Previous Approaches (2)

Shape from Occluding Boundariesis limited to a simply convex closed curved surface

Shape from Silhouetteuses multiple images through 360 degree rotation,

is also unavailable to object with local concave shape

Surface Reflectance and Shape from Images Using 90 degree rotation to get the feature points

However the rotation angle is limited to SEM

5/20

New Proposed Approach

Uses optimization with two images observed through the rotation of the object stand

1. The appropriate initial vector is determined using the Radial Basis Function neural network (RBF-NN) from two images during rotation.

2. The optimization is introduced using the Hopfield like neural network (HF-NN).

6/20

Characteristics of SEM Image (1)

Orthographic projection

Rotation angle

Reflectance property

i : incident angle, < 70°

s ≈ 0.5

R(i) is normalized to the range of 0 and 1.

3030

)1(cos

1)( i

ssiR

7/20

Characteristics of SEM Image (2)

z = F(x, y)F : height distribution

p, q : gradient parameters l = (0, 0, 1) : light source direction 　　　　　　　　　 : surface normal

　 　

cos i = n ・ l = nz … (3)

From Eq.(1)(2)(3),

y

zq

x

zp

,

)2(

1

1,,

),,(

22

qp

qp

nnn zyxn

1),( 22 qpssqpR

Cross Section of Reflectance Map(q=0)

8/20

Rotation Axis on Object Stand

Under the orthographic projection the gradient of the rotation axis is the sam

e for both images observed during rotation.

ex.

9/20

Estimation of Rotation Axis

1. Assume A and B move A’ and B’ during the rotation

2. Set A and A’ be the same pixel

3. Then rotation axis is determined so that it becomes perpendicular to the line BB’ and passes through the point A.

10/20

Shape Recovery from Two Images Using Hopfield Neural Network

Hopfield Neural Network (HF-NN) the mutual connection network the connection between the ne

urons are the symmetric HF-NN can be applied to solve t

he optimization problem of the energy function

m1

m2

m3

11/20

Energy Function to be Minimized

C1, C2, C3 : the regularization parameters

D : the target region of the object

E1 : the smoothness constraint

E2 : the error of the observed image brightness I(x,y) and the reflectance map R(p, q)

E3 : the error of the geometric relation for z and (p, q)

D

D

D

dxdyqy

zp

x

zE

dxdyqpRyxIE

dxdyy

q

x

q

y

p

x

pE

ECECECE

22

3

22

2222

1

332211

),(),(

(p,q,z): unknown variables

12/20

Initial Vector for Optimization (1)

Radial Basis Function Neural Network (RBF-NN) is introduced to obtain the approximation of gradient p, qAssume the same pixel (x, y) during the rotation.

The integration of along x direction results in the height distribution.

RBF

NN

I1(x, y)

I2(x, y)

nx

nz

z

x

n

np

13/20

Initial Vector for Optimization (2)

How to make dataset of RBF-NN A sphere is used to make I1 and I2 using R(p,

q), where, R is

　 since a sphere has the whole combination of the surface gradient.How to use learned RBF-NN

The corresponding point of the target object is assumed to be the same during the rotation.

1),( 22 qpssqpR

14/20

Updating Equation using HF-NN

The equation is iteratively used to optimize the energy function, that is, each partial difference becomes 0.

p

qpRqpRyxIC

p

E

y

p

x

pC

p

E

p

E

p

E

p

E

p

E

t

p

y

q

y

z

x

p

x

zC

z

Ez

E

z

E

t

z

),(),(),(2

2

2

22

2

2

2

2

11

321

2

2

2

2

33

3

qy

zC

q

E

q

qpRqpRyxIC

q

E

y

q

x

qC

q

E

q

E

q

E

q

E

q

E

t

q

px

zC

p

E

33

22

2

2

2

2

11

321

33

2

),(),(),(2

2

2

15/20

Iteration for Optimization

The optimization is applied to each of two images repeatedly. The height z’with the rotation angle is given byGradient are also calculated from the height repeatedly during rotation.C1 is gradually reduces

　　　　　　　　　　　　　　　　　　　 E1 : the smoothness constraint

Optimization is terminated the value of energy function converges in comparison with that of one step before.

cos),(sin),( yxzxyxz

332211 ECECECE

16/20

Experiments (synthesis image)

Rotation angle is 10°Image size is 64×64 pixels Rotation axis is along the center of the image.RBF-NN

　 Learning Data : 2000 　 Learning Epoch : 15

MSE 1.8961Maximum Height 10.37Theoretical Height Initial Height Recovered Height

Input Images

17/20

Experiments (SEM image)

Rotation angle taken is 10 °Rotation axis is set from the known feature　 points A and B

MSE 3.8926Theoretical Depth 13.1031

Theoretical Height Initial Height Recovered HeightRelaxation Method

Input Images

18/20

Experiments (SEM image)

Rotation angle taken is 10 °Rotation axis is set from the known feature　 points A and B

Input Images

Initial Height Recovered Height

19/20

Conclusion

A new method is proposed to recover the shape from SEM images.

HF-NN is introduced to solve the optimization problem. The energy function is formulated from two image during rotation.

The initial vector is obtained using RBF-NN.

20/20

Further works

Getting more accurate result using more images

Treatment of the inter-reflection

Thank you