Personalized Bare Hand Modeling Based on Stereo

Vision and Geometry Deformation

Quan Yu State Key Lab of CAD&CG

Zhejiang University

Outline Introduction System Overview Algorithm

Input Features Alignment & Deformation

Result & Feature Work Conclusion

Introduction

Personalized Hand Modeling(NSFC, No.60970078) No markers nor gloves Low-end devices (web cameras)

Challenge Lack of strong features Lack of solution

Introduction(Cont.) Idea

Corse features come from stereo vision Fine features come from a template

Solution Deform a template under constrains of

vision data step by step

System Overview

Fig. 1 System overview. (a)extract convexity defects of contours and build a local coordinate; (b)align the template with defects and refine alignment with ICP algorithm; (c)laplacian deformation under constrains of defects (point level); (d)generate contour points with a single image; (e)laplacian deformation under constrains of contours (line level); (f)extract surface features and construct a point cloud; (g)laplacian deformation under constrains of surface points (surface level).

Input Two pairs of stereo images

front and back faces of a hand A generic template

Denote contours and defects Camera parameters

Web cameras No markers

Features: defect points Convexity defects of contours

Stable Strong

Used to determine: Size Position Alignment with the template

Features: contour points

Generate contour points from a single image Contours of left and right image are

different. Assume the depths of contour points are

constant A non-linear interpolation between defects

( , , ) ( / , / , / )X Y Z X W Y W Z W

1

x X

y YQ

d Z

W

Disparities are unknown.

1 0 0

0 1 0

0 0 0

0 0 1/ 0

x

y

x

c

cQ

f

T

Features: contour points(Cont.)

Approximate contours as Correspondences: arc length

matching

1contours= (front back)

2

mean of contours arc length matching

Features: surface points Image enhancement

Contrast-Limited Adaptive Histogram Equalization Hard to extract robust features of hand

skin. SIFT SURF GLOH ? DAISY ?

Efficient Large-Scale Stereo Matching(ACCV 2010)

Sobel responses on a regular grid

Features: surface points(Cont.)

Find correspondences Estimate normals (MLS) Project 3D points onto the template Split the template at the projection point

Features: surface points(Cont.)

Iterative deformation to eliminate outliers

Reject Threshold

correspondences

deformation iter=1

iter=2

iter=3Details or Outliers?

H=k , 0,1,2,Z

kd

Alignment

Extract defects Local coordinate

Refine: ICP Efficient Variants of the ICP Algorithm[S. R. 2001]

defect templateM M M

/defect templates L L

,arg min( )template defect

R tR P t P

Laplacian Deformation Laplacian Mesh Processing

Ogla Sorkine, 2005

Laplacian Mesh Optimization Andrew Nealen, 2006

2 2( ) 2arg min xj j

x j C

x Lx x c

Result & Feature Work

Result Demo

Feature Work Resampling Geometry Optimization Texture

Conclusion

A novel approach to construct personalized hand model with low-end equipments;

Generate 3D contour points from a single image;

Eliminate outliers with an iterative deformation.

Thank you!Question & Suggestion ?