3D Hand Shape and Pose Estimation from a Single RGB Image...
3
3D Hand Shape and Pose Estimation from a Single RGB Image (Supplementary Material) Liuhao Ge 1* , Zhou Ren 2 , Yuncheng Li 3 , Zehao Xue 3 , Yingying Wang 3 , Jianfei Cai 1 , Junsong Yuan 4 1 Nanyang Technological University 2 Wormpex AI Research 3 Snap Inc. 4 State University of New York at Buffalo [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 1. Qualitative Results We present more qualitative results of 3D hand mesh re- construction and 3D hand pose estimation for our synthet- ic dataset, our real-world dataset, STB dataset [3], RHD dataset [4], and Dexter+Object dataset [2], as shown in Fig. 1. 2. Details of Baseline Methods for 3D Hand Mesh Reconstruction In Section 5.3 of our main paper, we compare our pro- posed method with two baseline methods for 3D hand mesh reconstruction: direct Linear Blend Skinning (LBS) method and MANO-based method. Here, we describe more details of these two baseline methods, as illustrated in Fig. 2. In the direct LBS method, we train the network to regress 3D hand joint locations from the heat-maps and the image features with heat-map loss and 3D pose loss. As illustrat- ed in Fig. 2 (b), the latent feature extracted from the input image is mapped to 3D hand joint locations through a multi- layer perceptron (MLP) network with three fully-connected layers. Then, we apply inverse kinematics (IK) to compute the transformation matrix of each hand joint from the the estimated 3D hand joint locations. The 3D hand mesh is generated by applying LBS with the predefined hand model and skinning weights. In this method, the 3D hand mesh is only determined by the estimated 3D hand joint locations, thus it cannot be adapted to various hand shapes. In addi- tion, the IK often suffers from singularity and multiple solu- tions, which makes the solutions to transformation matrices unreliable. Experimental results in Figure 7 and Table 2 of our main paper have shown the limitations of this direct LBS method. In the MANO-based method, we train the network to regress hand shape and pose parameters of the MANO hand * This work was done when Liuhao Ge was a research intern at Snap Inc. model [1]. As illustrated in Fig. 2 (c), the latent feature ex- tracted from the input image is mapped to hand shape and pose parameters θ, β through an MLP network with three fully-connected layers. Then, the 3D hand mesh is gener- ated from the regressed parameters θ, β using the MANO hand model [1]. Note that the MANO mesh generation module is differentiable and is involved in the network train- ing. The networks are trained with heat-map loss, mesh loss and 3D pose loss, which are the same as our method. Since the MANO hand model is fixed during training and is essen- tially LBS with blend shapes [1], the representation power of this method is limited. Experimental results in Figure 7 and Table 2 of our main paper have shown the limitations of this MANO-based method. 3. Details of the Task Transfer Method In Section 5.4 of our main paper, we implement an al- ternative method (“full model, task transfer”) for 3D hand pose estimation by transferring our full model trained for 3D hand mesh reconstruction to the task of 3D hand pose estimation. Here, we describe more details of our task trans- fer method. As illustrated in Fig. 3, we directly regress the 3D hand joint locations from the latent feature extracted by our full model using an MLP network with three fully- connected layers. We first train the MLP network with 3D pose loss on our synthetic dataset. When experimenting on STB dataset [3] with 3D pose supervision, we fine-tune the MLP network with 3D pose loss. When experiment- ing on STB dataset [3] without 3D pose supervision, we directly use the MLP network pretrained on our synthetic dataset. Experimental results in Figure 8 of our main paper show that our task transfer method is better than the base- line method which is only trained for 3D hand pose estima- tion, even though these two methods have the same pipeline. This indicates that the latent feature extracted by our full model is more discriminative and is easier to regress accu- rate 3D hand pose since our full model is trained with the
Transcript of 3D Hand Shape and Pose Estimation from a Single RGB Image...
3D Hand Shape and Pose Estimation from a Single RGB Image(Supplementary Material)
Liuhao Ge1∗, Zhou Ren2, Yuncheng Li3, Zehao Xue3, Yingying Wang3, Jianfei Cai1, Junsong Yuan4
1Nanyang Technological University 2Wormpex AI Research3Snap Inc. 4State University of New York at Buffalo
[email protected], [email protected], [email protected],[email protected], [email protected], [email protected], [email protected]
1. Qualitative Results
We present more qualitative results of 3D hand mesh re-construction and 3D hand pose estimation for our synthet-ic dataset, our real-world dataset, STB dataset [3], RHDdataset [4], and Dexter+Object dataset [2], as shown inFig. 1.
2. Details of Baseline Methods for 3D HandMesh Reconstruction
In Section 5.3 of our main paper, we compare our pro-posed method with two baseline methods for 3D hand meshreconstruction: direct Linear Blend Skinning (LBS) methodand MANO-based method. Here, we describe more detailsof these two baseline methods, as illustrated in Fig. 2.
In the direct LBS method, we train the network to regress3D hand joint locations from the heat-maps and the imagefeatures with heat-map loss and 3D pose loss. As illustrat-ed in Fig. 2 (b), the latent feature extracted from the inputimage is mapped to 3D hand joint locations through a multi-layer perceptron (MLP) network with three fully-connectedlayers. Then, we apply inverse kinematics (IK) to computethe transformation matrix of each hand joint from the theestimated 3D hand joint locations. The 3D hand mesh isgenerated by applying LBS with the predefined hand modeland skinning weights. In this method, the 3D hand mesh isonly determined by the estimated 3D hand joint locations,thus it cannot be adapted to various hand shapes. In addi-tion, the IK often suffers from singularity and multiple solu-tions, which makes the solutions to transformation matricesunreliable. Experimental results in Figure 7 and Table 2of our main paper have shown the limitations of this directLBS method.
In the MANO-based method, we train the network toregress hand shape and pose parameters of the MANO hand
∗This work was done when Liuhao Ge was a research intern at SnapInc.
model [1]. As illustrated in Fig. 2 (c), the latent feature ex-tracted from the input image is mapped to hand shape andpose parameters θ, β through an MLP network with threefully-connected layers. Then, the 3D hand mesh is gener-ated from the regressed parameters θ, β using the MANOhand model [1]. Note that the MANO mesh generationmodule is differentiable and is involved in the network train-ing. The networks are trained with heat-map loss, mesh lossand 3D pose loss, which are the same as our method. Sincethe MANO hand model is fixed during training and is essen-tially LBS with blend shapes [1], the representation powerof this method is limited. Experimental results in Figure 7and Table 2 of our main paper have shown the limitationsof this MANO-based method.
3. Details of the Task Transfer Method
In Section 5.4 of our main paper, we implement an al-ternative method (“full model, task transfer”) for 3D handpose estimation by transferring our full model trained for3D hand mesh reconstruction to the task of 3D hand poseestimation. Here, we describe more details of our task trans-fer method. As illustrated in Fig. 3, we directly regressthe 3D hand joint locations from the latent feature extractedby our full model using an MLP network with three fully-connected layers. We first train the MLP network with 3Dpose loss on our synthetic dataset. When experimentingon STB dataset [3] with 3D pose supervision, we fine-tunethe MLP network with 3D pose loss. When experiment-ing on STB dataset [3] without 3D pose supervision, wedirectly use the MLP network pretrained on our syntheticdataset. Experimental results in Figure 8 of our main papershow that our task transfer method is better than the base-line method which is only trained for 3D hand pose estima-tion, even though these two methods have the same pipeline.This indicates that the latent feature extracted by our fullmodel is more discriminative and is easier to regress accu-rate 3D hand pose since our full model is trained with the
dense supervision of the 3D hand mesh that contains richerinformation than the 3D hand pose. In addition, althoughthe estimation accuracy of our task transfer method is a lit-tle bit worse than that of our full model, our task transfermethod is faster than our full model, since it does not gener-ate 3D hand mesh. The runtime of our task transfer methodis 15.1ms, while the runtime of our full model which esti-mate 3D hand pose from hand mesh is 19.9ms. Thus, inapplications that only require 3D hand pose estimation butnot 3D hand shape estimation, we can choose to use thistask transfer method, which can maintain a comparable ac-curacy as our full model while runs at faster speed.
References[1] Javier Romero, Dimitrios Tzionas, and Michael J Black. Em-
bodied hands: Modeling and capturing hands and bodies to-gether. ACM Transactions on Graphics (TOG), 36(6):245,2017.
[2] Srinath Sridhar, Franziska Mueller, Michael Zollhofer, DanCasas, Antti Oulasvirta, and Christian Theobalt. Real-timejoint tracking of a hand manipulating an object from rgb-dinput. In ECCV, 2016.
[3] Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu,Xiaobin Xu, and Qingxiong Yang. 3D hand pose trackingand estimation using stereo matching. arXiv preprint arX-iv:1610.07214, 2016.
[4] Christian Zimmermann and Thomas Brox. Learning to esti-mate 3d hand pose from single RGB images. In ICCV, 2017.
Figure 1: Qualitative results for our synthetic dataset (the first row), our real-world dataset (the second row), STB dataset [3](the third row), RHD dataset [4] (the fourth row), and Dexter+Object dataset [2] (the last row).
Figure 2: Pipelines of our proposed method and two baseline methods: direct LBS method and MANO-based method. Thedifferences between the two baseline methods and our proposed method are highlighted in the green dashed line box.
Figure 3: Illustration of our “full model, task transfer” method. We transfer our full model trained for 3D hand meshreconstruction to the task of 3D hand pose estimation. Note that when training for the task of 3D hand pose estimation, thestacked hourglass network and the residual network are keep unchanged with our full model which is fully trained for thetask of 3D hand mesh reconstruction.
![RGB-D object pose estimation in unstructured environmentspeople.ece.umn.edu/~cchoi/pub/Choi15ras.pdfC.Choi,H.I.Christensen/RoboticsandAutonomousSystems75(2016)595–613 597 3Dshapecontext[27],1andrelativeanglesbetweenneighboring](https://static.fdocuments.us/doc/165x107/60445367751083212f70bb84/rgb-d-object-pose-estimation-in-unstructured-cchoipubchoi15raspdf-cchoihichristensenroboticsandautonomoussystems752016595a613.jpg)
