Robust Tensor-based Velocity Estimation of Plant Root Growth M. S. Degree presentation Hai Jiang

1

Robust Tensor-based Velocity Estimation of Plant Root Growth

M. S. Degree presentation

Hai Jiang

Advisor

K. PalaniappanComputer Engineering & Computer Science Department

University of Missouri - Columbia

2

Motivation

• Biological applications demand estimating the plant root growth with high accuracy [Beemster, 1998]. The objective is to generate a growth profile from the root tip along the root. In the computer vision sense, this is motion estimation from image sequences.

• The following plant root was mosaiced from the center frames of 10 segments. The size of each segment image sequence is 640pix*480pix*9 frame, and 1 pixel=0.885um.

3

Motivation

• The application of existing motion estimation methods on this plant root data is not straightforward and the users need to adjust a lot of parameters to fulfill the task.

• This research work was encouraged by the success of the estimation of plant leaf growth by the group of Bernd Jahne in Univ. of Heidelberg, Germany[Schmundt, 1995].

• This research work is valuable for other biological motion estimation applications.

4

Contribution

• Novel work and successful experiences in motion estimation of plant root growth using image processing techniques.

• Higher spatial and temporal accuracy, faster motion extraction than traditional numerical work. Developed an easy-to-use software package to do root growth estimation.

• A novel framework in dealing with non-uniform deformable motions. Knowledge acquired in this work will be useful in extracting aerospace dynamic analysis and flux computation in the sea or rivers.

5

Popular Methods

• Differential methods

• Correlation-based methods

• Frequency-based methods

• Probability-based methods

6

Differential Methods

• Constant illumination assumption

• Optical flow equation

Ix vx + Iy vy + It = 0

• The tensor-based method is an extension of optical flow method

7

Correlation-based Methods

• Based on region similarity, easy to adjust

• Max-convolution or Min-squared error

• Robust functions

• Computation intensity problem

• Local deformation problem

8

Frequency-based Methods

• Correspondence of motion and lines/planes in frequency domain

• Phase-based methods

• Easy to find motions with regular patterns

• Hard to deal with normal motions

9

Probability-based Methods

• Maximization of the possibilities of plans of reconstructing the previous images using pixels in the later images

• Bayesian methods, EM methods, annealing• Ability to deal with occlusions and deformations

10

N l w

Searching domain D

L

W

N’

Robust Matching Method

• Minimization of the cost function

+• Where

matching

Frame at time t1 Frame at time t2

sGr

rsrs vvuu )],(),([ 22 N

ttqpItjiI )]),,,(),,(([ 1wl

1

22

2

),(x

xx

222

2

)(

2),(

x

xx

11

s

s’’ M

M

Robust Matching Method

Confidence Tests• Statistical confidence test.

e.g. threshold on ||AVG(Cost)-MIN(Cost)||/STD_VAR(Cost)

• Forward-backward consistency test

s’

forward

Image at time t Image at time t+ t

backward

ME

12

Results of Robust MatchingResults of Robust Matching

13

The Problem of Robust Matching

• Segmentation not uniform for matching methods:

The anisotropy of the tensor indicates if there is motion or not. e.g. threshold on the coherency.

• Computation intensity: about one hour to analyze one sequence of 9 frames of size 640x480 using robust matching algorithm.

14

Image VolumeImage Volume

• A moving point forms a line in the volume

• A moving edge forms a plane in the volume

• Under constant illumination assumption, the trace of a moving point is a line with constant intensity

y

xt

y

t

15

The Tensor-based MethodThe Tensor-based Method

• In the image volume, we have

• That means the motion estimation can be done by estimating the orientations of the constant lines inside the image volume.

y axis

Image volume

t axis x axis

t

x

y

x

Constant gray level line (The bold line)

y

xTTyx tytx

tan

tan)/,/(),( vvv

16


• The orientation of iso-gray value lines within a 3D spatiotemporal neighborhood U can mathematically be formulated [Jahne, as the direction n being as much perpendicular to all gray value gradients g in U as possible, i. e.

• This can be formulized as and

where

min))()(( 32 xdxIxS

U

Tn

minJxxT

')'()'()'( 3xxIxIxxhJ dT ')'()'(

)'( 3xxx

xxhJ dx

I

x

I

qppq

17


• The 3D structure tensor for x-y-t image volume has the structure

J =

• It can be shown that this expression is minimized if the vector n is given by the eigenvector of the tensor J to the minimum eigenvalue.

ttytxt

ytyyxy

xtxyxx

JJJ

JJJ

JJJ

18

Eigen Analysis of the tensorEigen Analysis of the tensor

321:0)( ranka 321:2)( rankb 321:1)( rankc

11e22 e

33e 22 e22 e

33e33e

11e11e


23131 ))/()(( coherency 2

2121 ))/()(( edge 232132 ))/()(( corner

19


• Velocity computation

For rank 1:

For rank 2:

For rank 0, v=0

For rank 3, no motion can be extracted

3,32,31,3 /),(),( eeevvv TTyx

Tnorm ),( 2,11,12

1,12

2,1

3,1 eeee

ev

20


• Advantages

• Simultaneous segmentation and motion estimation

• Uniform computation framework

• Faster than robust matching method

• The problems:

• Velocity extraction is limited to rank 0, 1, and 2

• Only suitable for low-motion estimation (see following figure)

• Very sensitive to noises

• Aperture problem

t

local neighborhoodimage sequence

Trace of slow motion

Trace of fast motion

21

Robust Tensor MethodRobust Tensor Method

• A new framework has been proposed in the thesis work. It is the combination of the tensor method and the robust matching method.

• The idea is doing the tensor-based method followed by the robust matching method. The robust matching method takes advantage of the segmentation of the tensor method, uses the velocity values computed by the tensor method to initialize the start value of its velocity, and construct a much smaller searching domain.

Searching Domain Local Window P Original site s

H

| L |

Initial v

Final v

22


Original Image Sequence

Preprocessing: Filtering and Illumination Calibration

Compute Tensor Field

Using ‘coherency’ to compute segmentation mask

Compute approximate velocity using tensor method

Optimal Filter Computation

The pure tensor method

Use the velocity results from the tensor method to initialize the robust matching searching domain. Only compute the velocity for the sites

inside the segmentation mask. Statistical confidence test is done here.Compute both the forward matching and the backward matching.

The robust matching method

Confidence Tests(Forward-backward Matching Consistence test)

interpolate the velocity field

23

Robust Tensor Robust Tensor MethodMethod

• Preprocessing• Using Gaussian smooth filter to remove high frequency noises.

• Do illumination calibration for each frame in the image sequencePav = 0;

Read in the image sequence in DD[][][], which should contains tensorsize+2 frames

For t = 1 to tensorsize+2 do

Initialize Av[t] = 0;

For each pixel in the frame DD[t][][] do

Av[t] = Av[t] + DD[t][i][j];

Pav = Pav + Av[t];

Pav = Pav / (tensorsize+2);

For t=1 to (tensorsize+2) do

For each pixel in DD[][][]

DD[t][I][j] = DD[t][i][j] + (Pav – Av[t])/(area of a frame);

24


• Optimal filter and derivative computation in x-y-t space• Optimal filters are used to compute derivatives which is invariant

to the orientation of the coordinates

• 3D optimal sobel filter

(1 0 -1) x = (1 0 –1) x (A)

=

• Derivatives are computed by convolving the sobel filters with the intensity in the local 3D neighborhood

.01673807.05800629.01673807

.05800629.20102252.05800629

.01673807.0580062901673807.

)A(

000

000

000

)A(

25


• Interpolation• Assign a pixel x’ inside the holes with the weighted average over

the local neighborhood N:

where W(x-x’) is a weighting function which post more weights on pixels close to x’, and less weights on distance pixels. Gaussian normal function can be one good choice.

N

N

dW

dIW

Ixxx

xxxx

x)'(

)()'(

)'(

22

1

22

1),0()(

xx

x

T

eNW

26


A

BC

27


E

D

F

28


G

H

vx

vy

29

Velocity Profile MosaicingVelocity Profile Mosaicing

• To mosaic profiles of each root segment, kinetic mosaicing is needed.

• The problem is when imaging later segments, the previous segments are still growing. The whole velocity profile can not be generated by directly concatenate each segment profile.

• The referring coordinates are chosen to be the background, and assume the growth speed is constant to time for one segment. So the solution is first find the motion of camera by finding the registration information of the background images, then directly put the profile in the right place. Vertical lifting is done if the overlapping curves are not jointed. For details please read chapter 7 on my thesis.

30

Designed ExperimentsDesigned Experiments

• To find the optimal frame-displacement range • To find best tensor size • To estimate effects of time on the growth speed• Correctness verification

31

Experiment to find the optimal

frame-displacement range

nk

kt dxxn

tzigzag0

, )|)(|(1

)( v 2/12,

0, }]))(()([

1{)(

dxxvAVGdxxvn kt

nkkt tvariance

Suitable displacement range is found to be 0.4~2.8 pixels/frame, and the

optimal velocity to estimate with lowest error is 2.2 pixels/frame.

32

Experiments to find the optimal tensor size

nk

kt dxxn

tzigzag0

, )|)(|(1

)( v 2/12,

0, }]))(()([

1{)(

dxxvAVGdxxvn kt

nkkt tvariance

Note: The tensor size is PxPxP, P=2r+1. The horizontal axis stands for r.

33

Experiment to extract the profile of tip velocity over time

• Find the tip region by first segmenting the root using tensor method

• Run the robust tensor algorithm on the tip region, extract the average/median velocity of the tip region

• The estimation of the tip region of the next stack can be calculated from the tip velocity currently computed.

34

Correctness VerificationCorrectness Verification

• The robust tensor algorithm has been tested on synthetic root sequences, and high accuracy is achieved.

x

y

x

y

x

y

x

y

V, V(pix/fr)

Method X Direction Y Direction Motion Field Density (%)

Error Rate(%)

Mean(pix/fr)

Std.Var.(pix/fr)

Mean(pix/fr)

Std.Var.(pix/fr)

V =0.0

V =0.0

Pure tensor

0.000000 0.000000 0.000000 0.000000 96.8340 0.000000

Robust tensor

0.000000 0.000000 0.000000 0.000000 99.6133 0.000000

V =0.0V =1.0

Pure tensor

0.367220 0.180056 1.073452 0.110630 48.9141 13.45243

Robust tensor

0.000000 0.000000 1.000000 0.000000 99.0161 0.000000

V =1.0V =0.0

Pure tensor

1.029664 0.071725 0.219483 0.225242 48.6139

5.279663

Robust tensor

1.000000 0.000000 0.000000 0.000000 99.1329

0.000000

V =1.0V =1.0

Pure tensor

0.978694 0.270083 0.950715 0.254160 96.9147 3.519407

Robust tensor

1.000000 0.000000 1.000000 0.000000 98.4884

0.000000

35

Profile Mosaiced for the Whole RootProfile Mosaiced for the Whole Root

Biological scientists can make derivative over the distance from the tip on this curve and find the growth rate of each part of the plant root.

36

Profile Mosaiced for the Whole RootProfile Mosaiced for the Whole Root

37

SummarySummary

• By combining the tensor method and the robust matching method, this thesis work has provided a novel framework to estimate non-uniform small deformable motions.

• A software package to automatically estimate the spatio-variant growth profile of plant roots and the tip velocity profile over time is developed.

• This work was done on SGI Octane with R10k 195M Hz Mips CPU and 512M main memory.

38

• Algorithm competition with other popular motion estimation methods

• Parameter normalization of this algorithm to adapt various image sequences

• Better approaches of tip velocity tracking over time.

• Refine the model of tensor method to reduce the error of tensor method

• Try to implement a fast searching algorithm for the minimum in the robust matching method.

• Theory work of tensor method, about the problems of motion dis-/occlusion, local deformations, illumination changing. Explore the possibility to estimate the velocity for those motions when the tensor is not in rank 0, 1 or 2.

• Intension of combining the tensor method with probability-based motion estimation methods.

Future Research Work

39

1. G.T.S. Beemster and T.I. Baskin, Analysis of Cell Division and Elongation Underlying the Develop-mental Acceleration of Root Growth in Arabidopsis thealiana L., Plant Physiology (1998).

2. D. Schmundt, M. Stitt, B. Jahne and U. Schurr, Quantitative analysis of the local rates of growth of dicot leaves at a high temporal and spatial resolution, The Plant Journal(1998) 16(4), 505-514.

3. S.S. Beauchemin and J. L. Barron, The computation of optical flow, ACM Computing Surveys(1995), Vol. 27, No. 3.

4. H. HauBecker and B. Jahne, A Tensor Approach for Precise Computation of Dense Displacement Vector Fields, Proc. Mustererkennung 1997, Braunschweig, 15-17, Informatik Aktuell, Springer, Berlin, 199-208.

5. H. Knutsson and C-F. Westin. Normalized and Di_erential Convolution: Methods for Interpolation and Filtering of Incomplete and Uncertain Data. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 515{523, New York City, USA, June 1993.

References

40

6. H. Knutsson, C-F. Westin, and C-J. Westelius. Filtering of Uncertain Irregularly Sampled Multidimensional Data. In Twenty-seventh Asilomar Conf. On Signals, Systems & Computers, pages 1301{1309, Paci_c Grove, California, USA, November 1993. IEEE

7. G. Farneback, Spatial Domain Methods for Orientation and Velocity Estimation, LIU-TEK-LIC-1999:13, Dept. of E. E., Linkopings Univ., 1999, ISBN 91-7219-441-3, ISSN 0280-7971.

8. S.S. Beauchemin and J.L. Barron, The Computation of Optical Flow, ACM Comp. Surveys, Vol. 27, No.3, 1995.

References

41

9 J. L. Barron, D. J. Fleet, and S. S. Beauchemin, Performance of optical flow techniques. Technical Report RPL-TR-9107, Queen’s University, Kingston, Ontario, Robotics and Perception Laboratory Technical Report, July 1992.

10. Eero P. Simoncelli, Design of Multi-Dimensional Derivative Filters, First IEEE Int'l Conf. on Image Proc., Austin TX, Nov. 1994.

11. Hany Farid and Eero P. Simoncelli, Optimally Rotation-Equivariant Directional Derivative Kernels, 7th Int'l Conf. Computer Analysis of Images and Patterns, Kiel, Germany, Sept. 10-12, 1997.

References

42

12. H. HauBecker and B. Jahne, A Tensor Approach for Precise Computation of Dense Displacement Vector Fields, Proc. Mustererkennung 1997, Braunschweig, 15-17, September 1997, Informatik Aktuell, Springer, Berlin, 199-208.

13. Ye, M.; Haralick, R.M., Optical flow from a least-trimmed squares based adaptive approach, Pattern Recognition, 2000. Proceedings. 15th International Conference on Volume: 3 , 2000 Page(s): 1052 -1055 vol.3.

14. Ye, M.; Haralick, R.M., Two-stage robust optical flow estimation, Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on , Volume: 2 , 2000, Page(s): 623 -628 vol.2.

References

43

References

Robust Tensor-based Velocity Estimation of Plant Root Growth M. S. Degree presentation Hai Jiang

Documents

Transcript of Robust Tensor-based Velocity Estimation of Plant Root Growth M. S. Degree presentation Hai Jiang