The Virtual Photo Set (VPS) · Project results Scientific output-13 journal papers -39...
Transcript of The Virtual Photo Set (VPS) · Project results Scientific output-13 journal papers -39...
The Virtual Photo Set (VPS) IIS11-0081: Data-driven scene characterization
for realistic rendering
2017-11-14
Project overview
Project title: IIS11-0081 Data-driven scene characterisation for realistic rendering
Project leader: Anders Ynnerman and Jonas Unger
Principal investigators: Michael Felsberg, Fredrik Gustafsson, Reiner Lenz, Jonas Unger, Anders Ynnerman
Funding amount: 27,000,000 SEK Funding period: 2012-01-01 - 2016-12-31
The image paradigm shift
The vision of this project is to develop the foundation for the next generation imaging pipelines where reality can be edited and virtual and real objects can be seamlessly mixed.
Application drivers
• Digital design
• Product visualization
• Special effects in movies
• Computer games
• Augmented reality
• ...
Major challengesA High Dynamic Range (HDR) imaging pipeline
• Multi-modal input data • Tracking of input devices • TBs of data per capture • On-line user feedback
Processing and editing RenderingScene Capture
• Geometry and light source extraction, and scene editing
• Data representations and compression
• Interactive processing and user feedback
• Ultra-realistic off-line rendering
• Efficient material models • Compression for real-time
applications • Tone mapping
Project overviewProject team:C-Research, ITN, LiU, (Prof. Anders Ynnerman) Computer Vision Laboratory, ISY, LiU, (Prof. Michael Felsberg) Sensor Fusion Group, ISY, LiU, (Prof. Fredrik Gustafsson) 6 senior researchers, 6 PhD students, and 4 research engineers
Collaborations
• Linköping University • University of Southern California • Warwick University • Bangor University
• IKEA Communications • SpheronVR AG • IrysTech • Swiss International • Volvo PVH
Scientific collaborations Industry collaborations
New projects, funding proposals, and joint papers with new both academic and industrial partners
Project resultsScientific output
- 13 journal papers - 39 contributions at leading conferences in the field - 3 book chapters - 2 PhD theses (supported by this project) + 2 theses spring 2018 - 1 Licentiate thesis - 5 M.Sc. theses
Open data and software- HDR-video sequence data sets for research and educational use - Lighting environments captured using HDR imaging - Depends workflow management system - LumaHDRv high dynamic range (HDR) video codec
URLs: [ www.hdrv.org ], [ www.dependsworkflow.net ], [ www.lumahdrv.org ]
Spin-offs - Materialeyes AB (measurement systems, methods, and
algorithms for appearance capture ) - MassVis AB
VPS solutions driving the state-of-the-art
Image based lighting
Current state-of-the-art: Lighting is captured as a still image at a single position and at a single instant in time
HDR Light Probes
Virtual scene
Virtual
camera
HDR environment
map
L(x, ~!o
) =
Z
⌦Lenv
(~!i
)⇢(x, ~!i
! ~!o
)(�~!i
· ~nx
)V (~!i
)d~!i
- Lighting is described in the panoramic HDR image- Each pixel corresponds to the scene radiance incident over the solid angle subtended by the pixel
Lighting Material Cosine falloff Visibility
Mixing Virtual and Real
L(x, ~!o
) =
Z
⌦Lenv
(~!i
)⇢(x, ~!i
! ~!o
)(�~!i
· ~nx
)V (~!i
)d~!i
Lighting Material Cosine falloff Visibility
Requires high dynamic range (HDR) videoand light re-projection onto scene geometry
Rendering using accurately measured lighting recovered using the VPS approach
Image synthesis using previous state-of-the-art methods
Challenges (and VPS solutions)Capture: lighting conditions, scene geometry, and material information
Light Fields
Capture: lighting conditions, scene geometry, and material information
- Robust capture of lighting conditions in complex environments
- High resolution HDR-video beyond full HD
- Effective dynamic range at least 1:10,000,000
- Accurate radiometric calibration
LiU HDRv: http://www.hdrv.org
Our demands on HDR-video go far beyond commercial solutions
Need for HDR Video
Challenges
Affine Transform
Affine Transform
f(x,y)PSF
Sampling(R,G,B) +
d(x,y)p(x,y)
Affine Transform Sampling(R,G,B) +
ND Filter
ND Filter
Sampling(R,G,B) +ND Filter
F
T1
T2
T3
I2
I3
I1
Zj
Capture: lighting conditions, scene geometry, and material information
- Large scale data: ~1.5GB/s of floating point pixel data
- Minutes of capture leads to TBs of data - Real-time user feedback and HDR image
reconstruction is a requirement - VPS result: Novel image reconstruction
framework for multi-sensor systems suitable for parallel computations and GPU implementation
- Active in COST Action IC1005: Capture, storage, transmission and display of real-world lighting
HDR capture device
Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Anders Ynnerman, Jonas Unger, "A unified framework for multi-sensor HDR video reconstruction", Signal Processing : Image Communications, 29(2): 203-215, 2014.
J. Kronander, S. Gustavson, G. Bonnet, A. Ynnerman, J. Unger: Unified HDR Reconstruction from RAW CFA Data, In proceedings of the IEEE International Conference on Computational Photography (ICCP), 2013
Light Reprojection
• Project HDR light fields onto 3D geometry • Efficient representation for data compression • Encodes high frequency features • User level editing of geometry, materials and light sources
Capture: lighting conditions, scene geometry, and material information
Light Reprojection
• Project HDR light fields onto 3D geometry • Efficient representation for data compression • Encodes high frequency features • User level editing of geometry, materials and light sources
Capture: lighting conditions, scene geometry, and material information
Capture: lighting conditions, scene geometry, and material information
Our demands on the model poses research challenges in tracking and geometry estimation
Scene reconstruction
- Millimeter accuracy in recovered model - Submillimeter accuracy and sub degree
accuracy in camera pose and trajectory estimation
- Fusion of image information and range data - Robustness to non-stationary objects in the
scene
Challenges
Light source extraction
J. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.
J. Unger, J. Kronander, P. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings of EUSIPCO '13, 2013.
Image synthesis
J. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.
J. Unger, J. Kronander, P. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings of EUSIPCO '13, 2013.
Captured Scene
Tables removed
Project HDR data
Panoramic image displaying layout of the scene
Recovered model
J. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.
J. Unger, J. Kronander, P. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings of EUSIPCO '13, 2013.
Room populated with virtual furnitureJ. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.
J. Unger, J. Kronander, P. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings of EUSIPCO '13, 2013.
Norrköping
Computer graphics imageJ. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.
J. Unger, J. Kronander, P. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings of EUSIPCO '13, 2013.
Norrköping
J. Unger, S. Gustavson, J. Kronander, P. Larsson, G. Bonnet, and G. Kaiser. 2011. Next generation image based lighting using HDR video. In ACM SIGGRAPH 2011 Talks
J. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.
Computer graphics image
Norrköping
Computer graphics imageJ. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.
J. Unger, J. Kronander, P. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings of EUSIPCO '13, 2013.
Surface light fields
Store the light field at surfaces in the scene
Learning based compression using Exemplar Orthogonal Bases
Ehsan Miandji, Joel Kronander, Jonas Unger, "Learning Based Compression of Surface Light Fields for Real-time Rendering of Global Illumination Scenes", Proceedings of ACM SIGGRAPH ASIA 2013, 2013.
Real-time rendering
Compression
Compressed rendering
Ehsan Miandji, Joel Kronander, Jonas Unger, “Image reconstruction in reduced union of subspaces”, submitted to Eurographcis 2015
Light field reconstruction
Ehsan Miandji, Joel Kronander, Jonas Unger. Compressive image reconstruction in reduced union of sub-spaces. Computer Graphics Forum special issue Proceedings of Eurographics '15, Vol. 34, No. 2, 2015
Demonstrator integrationDepends workflow
management system• Generality
• Fusion of data from different input modalities• Modularity
• The ability to reuse software components and customize the processing to specific problems
• Data provenance • A highly important aspect is to enable the ability to go back
and easily determine what generated certain results.• Performance
• Large scale input data sizes (often in the order of TBs) and the computational complexity of the processing algorithms present significant challenges.
• Interaction• Advanced visualization and artistic interaction to steer the
computations.
Andrew Gardner, Jonas Unger, "Depends: Workflow Management Software for Visual Effects Production", Digital Production Symposium, DigiPro '14, Vancouver, Canada, August, 2014, DigiPro '14, 2014.
A Swedish House
LibraryC++ API for encoding and decoding of HDR video ApplicationsEncoding of HDR video frames Decoding of HDR video Playback of HDR video DependenciesMatroska media container VP9 (libvpx) from Google
Gabriel Eilertsen, Jonas Unger, and Rafal K. Mantiuk. Luma HDRv: an open source high dynamic range video codec optimized by large-scale testing. Accepted for SIGGRAPH '16 Talks, Annaheim, USA, 2016
Gabriel Eilertsen, Rafal K. Mantiuk, Jonas Unger. A High Dynamic Range Video Codec Optimized by Large Scale Testing, IEEE International Conference on Image Processing '16, Phoenix, USA, September, 2016.
HDR video compression
http://www.lumahdrv.org
Real-time noise aware TMODisplay: adapt input video to display characteristics, viewing environment and perception
HDREdge-stopping
spatial filter (fast detail
extraction diffusion) Local tone curves (noise aware minimum
contrast distortion)
Input HDR-video
Parameters: Peak luminance, Dynamic range, and Ambient light measurements
Noise-awarecontrol over
image details
Parameters: Local/global, Tone compression, Exposure, and Noise visibility control
Parameters: Edge stop
Parameter: Detail scaling, Noise visibility control
Tone-mapped output
Data flowUser parameterDisplay parameterNoise estimate
Noise model
Detail layer,
Base layer,
Input frame, Inverse
display model
Real-time noise aware tone mapping operator
Gabriel Eilertsen, Rafal Mantiuk, and Jonas Unger, Real-time noise-aware tone mapping, Accepted for publication in ACM Transactions on Graphics proceedings of Siggraph Asia '15, Kobe, Japan November, 2015
Glasses free 3D display
Andrew Jones, Jonas Unger, Koki Nagano, Jay Busch, Xueming Yu, Hsuan- Yueh Peng, Oleg Alexander, Mark Bolas, Mark and Paul Debevec. An Automultiscopic Projector Array for Interactive Digital Humans. In ACM SIGGRAPH 2015 Emerging Technologies, August 2015.
Andrew Jones, Jonas Unger, Jay Busch, Xueming Yu, Hsuan-Yueh Peng, Oleg Alexander and Paul Debevec. Creating a life-sized automultiscopic Morgan Spurlock for CNN's ``Inside Man''. In ACM SIGGRAPH 2014 Talks, Vancouver, Canada, August, 2014.
Glasses free 3D display
335.28 CM
121.92 CM
.63 DEGREES
216 PROJECTORS
SCREEN
385 CM
6 DEGREES
CAMERAS
216 projectors
0.63 degrees
Projection screen 250 x 140 cm
340 cm Projectors
Glasses free 3D display