Coded Aperture Projection - ACM SIGGRAPH · 2019. 3. 27. · known blur function. In image...
1
We integrate a coded aperture into an off-the-shelf projector to increase its depth of field. With the measured regional defocus on the screen surface, projected images are deconvolved with locally scaled aperture codes. This leads to better results than deconvolving with Gaussian PSFs in cases where regular spherical apertures are used. We achieve interactive frame-rates of up to 16 fps for an XGA resolution. Defocus blur as a convolution Optical defocus is often described as convolution with a filter kernel that corresponds to an image of the aperture being applied by the imaging device. The degree of defocus correlates to the scale of the kernel. Abstract Principle Implementation Results Coded Aperture Projection Max Große and Oliver Bimber Bauhaus-Universität Weimar ©2008 Bauhaus-Universität Weimar Increasing depth of field Depth of field for non planar surfaces can be significantly increased Mathematics of convolution and deconvolution A convolution of an image I with a kernel K of scale s results in a convolved image Iʻ Applying the Fourier transform, the convolution operater becomes a simple multiplication This enables us to invert the equation and calculate the deconvolution. As the Fourier transform of K might have values of very small magnitude, division leads to intensities in spacial domain that exceed the projectors displayable range, resulting in ringing artifacts. Introduction Video projectors apply simple spherical apertures and suffer from a relatively short depth of field. This is problematic when images are projected onto non- planar surfaces, such as in case of spherical or cylindrical projection displays. They might be regionally defocussed if the aperture of the projector is too large. Using smaller apertures for increasing the depth of field, however, decreases the light throughput. Related Work Multi-Projector approach Several approaches have been proposed to increase the depth of field of conventional projectors. Bimber and Emmerling [1], for instance, have described a multi-focal projection concept that applies multiple projectors with differently adjusted focal planes, but overlapping image areas. The individual image contribution of each projector is computed in such a way that –when displayed simultaneously– a single, photometrically and geometrically consistent image appears which is optimally focussed over the entire surface. Spherical aperture defocus compensation Images can be digitally sharpened by convolving them with the inverse of a known blur function. In image processing, this process is known as deconvolution or inverse filtering. Brown et al. [2] as well as Oyamada et al. [3] apply inverse filtering to compensate optical defocus of a single projector. The PSFs produced by projectors using regular spherical apertures is Gaussian. This sets clear limitations in terms of recovering fine image details with inverse filtering. Since Gaussians are low-pass filters, their inverse will cause strong ringing artifacts after deconvolution if the projectorʼs dynamic range is not infinitely high. To reduce these artifacts, Zhang and Nayar [4] formulate the computation of the compensation image as a optimization problem that constraints the solution to the actual dynamic range of the projector while minimizing local optical defocus. Coded Aperture Imaging Coded aperture imaging has been presented recently in the context of computational photography [5, 6], and has been applied previously in astronomy and medical imaging. In contrast to conventional apertures, coded apertures (i.e., apertures that encode a more complex binary or intensity pattern, rather than a simple round opening) in cameras enable post-exposure refocusing, reconstructing scene depth, or recording light fields. Deconvolution / Inverse filtering Deconvolution –i.e., convolving an image with the inverse aperture kernel– will digitally sharpen the image and consequently compensate optical defocus. I ∗ K s =I I · K s = I I= I K s Coded Aperture Projection Ringing artifacts If the magnitudes in frequency domain of the used aperture are very low at some positions, which is the case for ordinary spherical apertures, strong ringing artifacts will be visible. Defocused projection Ringing artifacts that result from overcompensation Hardware Implementation The selected near-optimal aperture code was printed on transparencies (Kodak film didnʼt resist the heat), and was inserted into the objective lens – replacing the projectorʼs exiting spherical aperture. Software implementation For measuring the local defocus of each projector pixel, we apply the momentum preserving principle used in [1]. IPrincipally, each pixel of the original image would have to be deconvolved individually depending on its defocus. To reach interactive frame-rates, larger image portions have to be processed instead. One option is to deconvolve the entire image for all determined aperture scales[2] and compose the compensation image from the different scale results. However, this technique is also not very efficient, since many pixels are deconvolved for unnecessary aperture scales. Partitioning the image with a simple uniform grid structure [3] does not consider the (possibly nonuniform) local distribution of aperture scales, and can therefore still causes a significant amount of overhead. Image partitioning To overcome this, our method partitions the image in a non-uniform way based on the actual distribution of the aperture scales and on the capabilities of the graphics hardware being used. The entire partitioning operation can be carried out off-line, since it is independent from the image content. Optimized aperture codes always perform better for digital defocus compensation than ordinary round openings with the same light throughput in focus out of focus spherical aperture compensated coded aperture compensated * * * -1 ? We introduce coded aperture projection, and show that if coded apertures are applied instead of simple spherical ones, ringing artifacts as a result from deconvolution can be reduced and more image details can be recovered from optical defocus. The coded aperture has less low- magnitude frequencies and preserves high frequencies over all scales. Depixelation Optical defocus diminishes the appearance of the projected pixel grid while deconvolution recovers the image details. This has been referred to as projector depixelation [4], which is another application of projector defocus compensation. Interactive framerates Using our technique, we achieve interactive framerates of 12-16 fps on a NVidia GeForce 8800 Ultra for 1-14 blur different blur levels and an XGA resolution. References [1] BIMBER, O., AND EMMERLING, A. 2006. Multifocal Projection: A Multiprojector Technique for Increasing Focal Depth. IEEE Transactions on Visualization and Computer Graphics (TVCG) 12, 4, 658– 667. [2] BROWN, M. S., SONG, P., AND CHAM, T.-J. 2006. Image Preconditioning for Out-of-Focus Projector Blur. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. II, 1956– 1963. [3] OYAMADA, Y., AND SAITO, H. 2007. Focal Pre-Correction of Projected Image for Deblurring Screen Image. In Proc. of IEEE International Workshop on Projector-Camera Systems (ProCams). [4] ZHANG, L., AND NAYAR, S. K. 2006. Projection Defocus Analysis for Scene Capture and Image Display. ACM Trans. Graph. (Siggraph) 25, 3, 907–915. [5] LEVIN, A., FERGUS, R., DURAND, F., AND FREEMAN, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. (Siggraph) 26, 3, 70. [6] VEERARAGHAVAN, A., RASKAR, R., AGRAWAL, A., MOHAN, A., AND TUMBLIN, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. (Siggraph) 26, 3, 69. http://www.uni-weimar.de/medien/ar [email protected] [email protected] This work was supported by the Deutsche Forschungsgemeinschaft (DFG) under contract number PE 1183/1-1 Steps per frame of deconvolution process on GPU 1. luminance conversion and up-sampling, 2. partitioning and padding, 3. sequencing to array, 4. FFT of patch sequences, 5. divisions in frequency domain for all scales required for each patch, 6. IFFT for all patches and all scales, 7. desequencing to 2D image while selecting pixelsʼ luminance values from correct scale array, 8. blending of overlapping patches in spatial domain, 9. down-sampling to original size and recombination with chrominances. GPU Fourier transformation The Fourier transformation and its inverse require the largest amount of computation time for deconvolution. We apply CUDAʼs GPU implementation of the Fast Fourier Transformation (FFT). We decide to subdivide the image into patches whose edge lengths are powers-of-two. First, we measure the efficiency of the entire deconvolution for each possible patch size directly on the GPU, and compute the average time that is required to process one pixel in each case. In theory, this should be in the order of lg(N) for patches with N pixels. In practice, the overhead of hardware and software specific implementations of the FFT/IFFT can be significant. FFT of (padded) divide each patch sequence by FFTs of aperture kernels IFFT of patch sequences for patch- upsampling downsampli sequencing desequencin applying blending 2 i 2 j 2 i-1 2 j-1 2 i-2 2 j-2 2 i-2 2 j-2 2 j-2 2 i- 2 i-2 2 j-1 2 i-1 2 j-2 2 i-2 2 j-1 2 i-2 2 j-1 2 i-2 2 j-1
Transcript of Coded Aperture Projection - ACM SIGGRAPH · 2019. 3. 27. · known blur function. In image...
![Page 1: Coded Aperture Projection - ACM SIGGRAPH · 2019. 3. 27. · known blur function. In image processing, this process is known as deconvolution or inverse filtering. Brown et al. [2]](https://reader035.fdocuments.us/reader035/viewer/2022071415/611165e32a66fc3e5b4491d2/html5/thumbnails/1.jpg)
We integrate a coded aperture into anoff-the-shelf projectorto increase its depthof field. With themeasured regionaldefocus on thescreen surface,projected imagesare deconvolvedwith locally scaledaperture codes.This leads to betterresults thandeconvolving withGaussian PSFs incases where regularspherical apertures are used. We achieve interactive frame-rates of up to 16 fps for an XGA resolution.
Defocus blur as a convolutionOptical defocus is often described as convolution with a filter kernel that corresponds to an image of the aperture being applied by the imaging device. The degree of defocus correlates to the scale of the kernel.
Abstract Principle Implementation Results
Coded Aperture ProjectionMax Große and Oliver BimberBauhaus-Universität Weimar
©2008 Bauhaus-Universität Weimar
Increasing depth of fieldDepth of field for non planar surfaces can besignificantly increasedMathematics of convolution and deconvolution
A convolution of an image I with a kernel K of scale s results in a convolved image Iʻ
Applying the Fourier transform, the convolution operater becomes a simple multiplication
This enables us to invert the equation and calculatethe deconvolution. As the Fourier transform of K mighthave values of very small magnitude, division leads tointensities in spacial domain that exceed the projectorsdisplayable range, resulting in ringing artifacts.
IntroductionVideo projectors apply simple spherical apertures and suffer from a relatively short depth of field. This is problematic when images are projected onto non-planar surfaces, such as in case of spherical or cylindrical projection displays. They might be regionally defocussed if the aperture of the projector is too large. Using smaller apertures for increasing the depth of field, however, decreases the light throughput.
Related WorkMulti-Projector approachSeveral approaches have been proposed to increase the depth of field of conventional projectors. Bimber and Emmerling [1], for instance, have described a multi-focal projection concept that applies multiple projectors with differently adjusted focal planes, but overlapping image areas. The individual image contribution of each projector is computed in such a way that –when displayed simultaneously– a single, photometrically and geometrically consistent image appears which is optimally focussed over the entire surface.
Spherical aperture defocus compensationImages can be digitally sharpened by convolving them with the inverse of a known blur function. In image processing, this process is known as deconvolution or inverse filtering. Brown et al. [2] as well as Oyamada et al. [3] apply inverse filtering to compensate optical defocus of a single projector.The PSFs produced by projectors using regular spherical apertures is Gaussian. This sets clear limitations in terms of recovering fine image details with inverse filtering. Since Gaussians are low-pass filters, their inverse will cause strong ringing artifacts after deconvolution if the projectorʼs dynamic range is not infinitely high. To reduce these artifacts, Zhang and Nayar [4] formulate the computation of the compensation image as a optimization problem that constraints the solution to the actual dynamic range of the projector while minimizing local optical defocus.
Coded Aperture ImagingCoded aperture imaging has been presented recently in the context of computational photography [5, 6], and has been applied previously in astronomy and medical imaging. In contrast to conventional apertures, coded apertures (i.e., apertures that encode a more complex binary or intensity pattern, rather than a simple round opening) in cameras enable post-exposure refocusing, reconstructing scene depth, or recording light fields.
Deconvolution / Inverse filteringDeconvolution –i.e., convolving an image with the inverse aperture kernel– will digitally sharpen the image and consequently compensate optical defocus.
I ∗ Ks = I�
�I · �Ks = �I�
�I =�I�
�Ks
Coded Aperture Projection
Ringing artifactsIf the magnitudes in frequency domain of the used aperture are very low at some positions, which is the case for ordinary spherical apertures, strong ringing artifacts will be visible.
Defocused projection Ringing artifacts that result from overcompensation
Hardware ImplementationThe selected near-optimal aperture code was printed on transparencies(Kodak film didnʼt resist the heat), and was inserted into theobjective lens – replacing the projectorʼs exiting spherical aperture.
Software implementationFor measuring the local defocus of each projector pixel, we apply the momentum preserving principle used in [1].IPrincipally, each pixel of the original image would have to be deconvolved individually depending on its defocus. To reach interactive frame-rates, larger image portions have to be processed instead.One option is to deconvolve the entire image for all determined aperturescales[2] and compose the compensation image from the different scale results. However, this technique is also not very efficient, since many pixels are deconvolved for unnecessary aperture scales.Partitioning the image with a simple uniform grid structure [3] does not consider the (possibly nonuniform) local distribution of aperture scales, and can therefore still causes a significant amount of overhead.
Image partitioningTo overcome this, our method partitionsthe image in a non-uniform way basedon the actual distribution of the aperturescales and on the capabilities of thegraphics hardware being used. The entirepartitioning operation can be carried outoff-line, since it is independent from theimage content.
Optimized aperture codes always perform better for digital defocus compensation than ordinary round openings with the same light throughput
in focus out of focus spherical aperture
compensated
codedaperture
compensated
*
*
*-1
?
We introduce coded aperture projection, and show that if coded apertures are applied instead of simple spherical ones, ringing artifacts as a result from deconvolution can be reduced and more image details can be recovered from optical defocus.
The coded aperture has less low-magnitude frequencies and preserves high frequencies overall scales.
DepixelationOptical defocus diminishes the appearance of the projected pixel grid while deconvolution recovers theimage details. This has been referred to asprojector depixelation [4], which is another application of projector defocus compensation.
Interactive frameratesUsing our technique, we achieve interactive framerates of 12-16 fps on a NVidia GeForce 8800 Ultra for 1-14 blur different blur levels and an XGA resolution.
References[1] BIMBER, O., AND EMMERLING, A. 2006. Multifocal Projection: A Multiprojector Technique for Increasing Focal Depth. IEEE Transactions on Visualization and Computer Graphics (TVCG) 12, 4, 658–667.
[2] BROWN, M. S., SONG, P., AND CHAM, T.-J. 2006. Image Preconditioning for Out-of-Focus Projector Blur. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. II, 1956–1963.
[3] OYAMADA, Y., AND SAITO, H. 2007. Focal Pre-Correction of Projected Image for Deblurring Screen Image. In Proc. of IEEE International Workshop on Projector-Camera Systems (ProCams).
[4] ZHANG, L., AND NAYAR, S. K. 2006. Projection Defocus Analysis for Scene Capture and Image Display. ACM Trans. Graph. (Siggraph) 25, 3, 907–915.
[5] LEVIN, A., FERGUS, R., DURAND, F., AND FREEMAN, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. (Siggraph) 26, 3, 70.
[6] VEERARAGHAVAN, A., RASKAR, R., AGRAWAL, A., MOHAN, A., AND TUMBLIN, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. (Siggraph) 26, 3, 69.
http://www.uni-weimar.de/medien/[email protected]@medien.uni-weimar.de
This work was supported by the Deutsche Forschungsgemeinschaft (DFG)
under contract number PE 1183/1-1
Steps per frame of deconvolution process on GPU1. luminance conversion and up-sampling, 2. partitioning and padding, 3. sequencing to array, 4. FFT of patch sequences, 5. divisions in frequency domain for allscales requiredfor each patch,6. IFFT for allpatches and allscales,7. desequencing to2D image whileselecting pixelsʼluminance valuesfrom correct scalearray,8. blendingof overlappingpatches in spatialdomain,9. down-samplingto original size andrecombination with chrominances.
GPU Fourier transformationThe Fourier transformation and itsinverse require the largest amountof computation time fordeconvolution. We apply CUDAʼsGPU implementation of the FastFourier Transformation (FFT). We decide to subdivide the imageinto patches whose edge lengthsare powers-of-two. First, wemeasure the efficiency of the entiredeconvolution for each possiblepatch size directly on the GPU, andcompute the average time that isrequired to process one pixel ineach case. In theory, this should be in the order of lg(N) for patches with N pixels. In practice, the overhead of hardware and software specific implementations of the FFT/IFFT can be significant.
FFT of (padded)
divide each patch sequence by FFTs of aperture kernels
IFFT of patch sequences for patch-
up
sam
pli
ng
do
wn
sam
pli
seq
ue
nci
ng
de
seq
ue
nci
n
ap
ply
ing
ble
nd
ing
2i
2j
2i-1
2j-1
2i-2
2j-2
2i-2
2j-2
2j-2
2 i-2i-2
2j-1
2i-1
2j-2
2i-2
2j-1
2i-2
2j-12i-2
2j-1
