Supplementary Material - Learning Fused Pixel and Feature...
5
Supplementary Material - Learning Fused Pixel and Feature-based View Reconstructions for Light Fields Jinglei Shi * Xiaoran Jiang * Christine Guillemot INRIA Rennes - Bretagne Atlantique, France {firstname.lastname}@inria.fr In this document, additional materials are provided to supplement our main paper. In the first section, we provide the link for demo codes. In the second section, we compare the running time of different learning-based methods. In the third section, network structure is depicted. In the fourth section, more visual comparisons of the reconstructed views are given, which is not possible in the main submission due to the page limit. Note that we also provide video files that show the whole reconstructed light fields both in the case of interpolation and that of extrapolation. 1. Test codes Our demo code is available in the following github link: https://github.com/JingleiSHI/FPFR. The complete source code will be available soon. 2. Runtime We compare the runtime of the three learning-based reference methods[1, 2, 3] against ours. In Table 1, we record the running time for synthesizing a light field of resolution 768 × 768 × 7 × 7 on a GPU of type Nvidia Tesla V-100 with 32GB memory. (The method [3] is run on CPU Intel i7-7600U 2.80GHz 16GB RAM. We did not succeed in running it on GPU, and in their paper the running time comparison was also done with CPU.) The running time includes model loading and computation of color views. Methods DeepBW[1] LLFF[2] EPI[3] FPFR FPFR* Runtime 778s 212s 1332s(CPU) 172s 684s Table 1. Runtime for generating a 768 × 768 × 7 × 7 light field for learning-based methods. 3. Network structure The structure details for layers of PixRNet and FeatRNet are depicted in the Table 2. 4. Additional qualitative results We show in Fig. 1 (for synthetic light fields) and Fig. 2 (for real-world light fields) additional visual comparisons that are not included in the paper due to the page limit. * equal contribution
Transcript of Supplementary Material - Learning Fused Pixel and Feature...
Supplementary Material - Learning Fused Pixel and Feature-based ViewReconstructions for Light Fields
Jinglei Shi∗ Xiaoran Jiang∗ Christine GuillemotINRIA Rennes - Bretagne Atlantique, France
{firstname.lastname}@inria.fr
In this document, additional materials are provided to supplement our main paper. In the first section, we provide the linkfor demo codes. In the second section, we compare the running time of different learning-based methods. In the third section,network structure is depicted. In the fourth section, more visual comparisons of the reconstructed views are given, which isnot possible in the main submission due to the page limit.
Note that we also provide video files that show the whole reconstructed light fields both in the case of interpolation andthat of extrapolation.
1. Test codesOur demo code is available in the following github link:
https://github.com/JingleiSHI/FPFR. The complete source code will be available soon.
2. RuntimeWe compare the runtime of the three learning-based reference methods[1, 2, 3] against ours. In Table 1, we record the
running time for synthesizing a light field of resolution 768× 768× 7× 7 on a GPU of type Nvidia Tesla V-100 with 32GBmemory. (The method [3] is run on CPU Intel i7-7600U 2.80GHz 16GB RAM. We did not succeed in running it on GPU,and in their paper the running time comparison was also done with CPU.) The running time includes model loading andcomputation of color views.
Methods DeepBW[1] LLFF[2] EPI[3] FPFR FPFR*Runtime 778s 212s 1332s(CPU) 172s 684s
Table 1. Runtime for generating a 768× 768× 7× 7 light field for learning-based methods.
3. Network structureThe structure details for layers of PixRNet and FeatRNet are depicted in the Table 2.
4. Additional qualitative resultsWe show in Fig. 1 (for synthetic light fields) and Fig. 2 (for real-world light fields) additional visual comparisons that are
not included in the paper due to the page limit.
∗equal contribution
PixRNet k s in/out inputconv3Px 3 1 16/64 concat({L̃i
t,mit,∀i})
conv2Px 3 1 64/32 conv3Pxconv1Px 3 1 32/16 conv2PxL̂Pixt 3 1 16/3 conv1Px
FeatRNetconv3 1Ft 3 1 1028/256 concat({f̃ i,3
t ,mi,3t ,∀i})
conv3 2Ft 3 1 256/256 conv3 1Ftf̂3t 3 1 256/256 conv3 2Ft
up3 4 2 256/32 f̂3t
conv2 1Ft 3 1 548/128 concat({f̃ i,2t ,mi,2
t ,∀i},up3)conv2 2Ft 3 1 128/128 conv2 1Ft
f̂2t 3 1 128/128 conv2 2Ft
up2 4 2 128/16 f̂2t
conv1 1Ft 3 1 276/64 concat({f̃ i,1t ,mi,1
t ,∀i},up2)conv1 2Ft 3 1 64/64 conv1 1Ft
f̂1t 3 1 64/64 conv1 2Ft
conv0 1Ft 3 1 64/32 f̂1t
conv0 2Ft 3 1 32/32 conv0 1FtL̂Featt 3 1 32/3 conv0 2Ft
Fusionf̆Pix 3 1 16/8 conv1Pxf̆Feat 3 1 32/8 conv0 2Ft
conv3 Fn 3 1 16/8 concat(f̆Pix,f̆Feat)conv2 Fn 3 1 8/8 conv3 Fnconv1 Fn 3 1 8/4 conv2 Fn
M 3 1 4/1 conv1 Fn
Table 2. The proposed network architecture. k, s and in/out represent the kernel size, the stride and the number of input/output channels,whereas ‘up’ and ‘concat’ represent upsampling by deconvolution and concatenation.
buddha 41.08dB 41.86dB 40.66dB 40.05dB 43.06dB
mona 38.90dB 40.92dB 41.20dB 37.54dB 42.86dB
boxes 33.64dB 32.14dB 34.97dB 31.65dB 34.46dB
dino 38.41dB 41.69dB 41.26dB 38.44dB 42.98dB
cotton 47.24dB 48.95dB 47.07dB 47.97dB 48.76dB
Electro devices 34.09dB 36.24dB 36.76dB 31.55dB 38.03dB
Toy bricks 28.90dB 36.58dB 37.98dB 31.46dB 39.35dB
Light field view DeepBW [1] Soft3D [4] LLFF [2] EPI [3] FPFR*
Figure 1. Visual comparison of reconstruction error maps for synthetic light fields.
Cars 31.53dB 27.68dB 29.06dB 28.17dB 32.25dB
Flower2 31.95dB 30.52dB 28.90dB 29.26dB 34.19dB
Rock 34.32dB 32.67dB 32.60dB 32.46dB 36.75dB
Seahorse 32.03dB 30.41dB 28.50dB 26.62dB 34.97dB
Light field view DeepBW [1] Soft3D [4] LLFF [2] EPI [3] FPFR*
Figure 2. Visual comparison of reconstruction error maps for real light fields.
References[1] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. Learning-based view synthesis for light field cameras. ACM
Trans. on Graphics, 35(6):193:1–193:10, 2016. 1, 3, 4
[2] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. on Graphics, 2019. 1, 3, 4
[3] Gaochang Wu, Yebin Liu, Qionghai Dai, and Tianyou Chai. Learning sheared epi structure for light field reconstruction. IEEE Trans.Image Process., 28(7):3261–3273, 2019. 1, 3, 4
[4] Eric Penner and Li Zhang. Soft 3D reconstruction for view synthesis. ACM Trans. on Graphics, 36(6):235:1–235:11, 2017. 3, 4
