Computer Graphics Image-based rendering...Carranza et al., SIGGRAPH ‘03 Image-Based Visual Hulls...

13
1 Image-based rendering Michael F. Cohen Microsoft Research Computer Graphics Image Output Model Synthetic Camera Real Scene Computer Vision Real Cameras Model Output Combined Model Real Scene Real Cameras Image Output Synthetic Camera

Transcript of Computer Graphics Image-based rendering...Carranza et al., SIGGRAPH ‘03 Image-Based Visual Hulls...

  • 1

    Image-based rendering

    Michael F. Cohen

    Microsoft Research

    Computer Graphics

    Image

    Output

    Model Synthetic

    Camera

    Real Scene

    Computer Vision

    Real Cameras

    Model

    Output

    Combined

    Model Real Scene

    Real Cameras

    Image

    Output

    Synthetic

    Camera

  • 2

    But, vision technology falls

    short

    Model Real Scene

    Real Cameras

    Image

    Output

    Synthetic

    Camera

    … and so does graphics.

    Model Real Scene

    Real Cameras

    Image

    Output

    Synthetic

    Camera

    Image Based Rendering

    Real Scene

    Real Cameras

    -or-

    Expensive Image Synthesis

    Images+Model

    Image

    Output

    Synthetic

    Camera

    Ray

    Constant radiance

    • time is fixed

    5D

    • 3D position

    • 2D direction

  • 3

    All Rays

    Plenoptic Function

    • all possible images

    • too much stuff!

    Line

    Infinite line

    4D

    • 2D direction

    • 2D position

    Ray

    Discretize

    Distance between 2 rays

    • Which is closer together?

    Image

    What is an image?

    All rays through a point

    • Panorama?

  • 4

    Image

    2D

    • position of rays has been fixed

    • direction remains

    Image

    Image plane

    2D

    • position

    Image plane

    2D

    • position

    Image

    Light leaving towards “eye”

    2D

    • just dual of image

    Object

  • 5

    Object

    All light leaving object

    Object

    4D

    • 2D position

    • 2D direction

    Object

    All images

    Lumigraph

    How to

    • organize

    • capture

    • render

  • 6

    Lumigraph - Organization

    2D position

    2D direction

    s q

    Lumigraph - Organization

    2D position

    2D position

    2 plane parameterization

    s u

    Lumigraph - Organization

    2D position

    2D position

    2 plane parameterization u s

    t s,t

    u,v

    v

    s,t

    u,v

    Lumigraph - Organization

    Hold s,t constant

    Let u,v vary

    An image

    s,t u,v

  • 7

    Lumigraph - Organization

    Discretization

    • higher res near object

    • if diffuse

    • captures texture

    • lower res away

    • captures directions

    s,t u,v

    Lumigraph - Capture

    Idea 1

    • Move camera carefully

    over s,t plane

    • Gantry

    • see Lightfield paper

    s,t u,v

    Lumigraph - Capture

    Idea 2

    • Move camera anywhere

    • Rebinning

    • see Lumigraph paper

    s,t u,v

    Lumigraph - Rendering

    For each output pixel

    • determine s,t,u,v

    • either

    • find closest discrete RGB

    • interpolate near values s,t u,v

  • 8

    Lumigraph - Rendering

    For each output pixel

    • determine s,t,u,v

    • either

    • use closest discrete RGB

    • interpolate near values s u

    Lumigraph - Rendering

    Nearest

    • closest s

    • closest u

    • draw it

    Blend 16 nearest

    • quadrilinear interpolation s u

    High-Quality Video View Interpolation

    Using a Layered Representation

    Larry Zitnick Sing Bing Kang

    Matt Uyttendaele

    Simon Winder

    Rick Szeliski

    Interactive Visual Media Group

    Microsoft Research

    Current practice

    Many cameras

    Motion Jitter

    vs.

    free viewpoint video

  • 9

    Current practice

    Many cameras

    Motion Jitter

    vs.

    free viewpoint video

    Video view interpolation

    Fewer cameras

    Smooth Motion

    Automatic

    and

    Real-time

    rendering

    Prior work: IBR (static)

    The Lumigraph

    Gortler et al., SIGGRAPH ‘96

    Concentric Mosaics

    Shum & He, SIGGRAPH ‘99

    Plenoptic Modeling

    McMillan & Bishop, SIGGRAPH ‘95

    Light Field Rendering

    Levoy & Hanrahan, SIGGRAPH ‘96

    Prior work: IBR (dynamic)

    Free-viewpoint Video of Humans

    Carranza et al., SIGGRAPH ‘03

    Image-Based Visual Hulls

    Matusik et al., SIGGRAPH ‘00

    Virtualized RealityTM

    Kanade et al., IEEE Multimedia ‘97

    Dynamic Light Fields

    Goldlucke et al., VMV ‘02

    Stanford Multi-Camera

    Array Project

    3D TV

    Matusik & Pfister,

    SIGGRAPH ‘04

  • 10

    System overview

    OFFLINE

    ONLINE

    Video

    Capture

    Stereo Compression

    Selective

    Decompression Render

    File

    Representation

    Video

    Capture

    concentrators

    hard

    disks controlling

    laptop

    cameras cameras

    Calibration

    Zhengyou Zhang, 2000

    Input videos

  • 11

    Key to view interpolation: Geometry

    Stereo Geometry

    Camera 1 Camera 2

    Image 1 Image 2

    Virtual Camera

    Match Score Match Score Match Score Match Score

    Good

    Bad

    Image correspondence

    Correct

    Image 1 Image 2

    Leg

    Wall

    Incorrect

    Image 1 Image 2

    Local matching

    Low texture

    Number of states =

    number of depth levels

    Image 2 Image 1

    Global regularization

    Create MRF (Markov Random Field):

    A F

    E

    D

    C

    B

    colorA ≈ colorB → zA ≈ zB Each segment is a node

    zA ≈ zP, zQ, zS

    P Q R

    S T

    U

    A

    A

  • 12

    Iteratively solve MRF Depth through time

    Matting

    Interpolated view without matting

    Background Surface

    Foreground

    Surface

    Camera

    Foreground

    Alpha

    Background

    Bayesian Matting Chuang et al. 2001

    Strip

    Width

    Background

    Foreground

    Rendering with matting

    Matting No Matting

  • 13

    Representation

    Main Layer:

    Color

    Depth

    Main

    Boundary

    Boundary Layer:

    Color

    Depth

    Alpha

    Background

    Foreground Strip

    Width “Massive Arabesque”

    videoclip


Sunghyun Cho Seungyong Lee POSTECHrosaec.snu.ac.kr/publish/2009/ID/ChLe-SIGGRAPH-2009.pdf · 2018-04-12 · Cho et al. [2007] used multiple images with space-variant blurs for finding

Sunghyun Cho Seungyong Lee POSTECHrosaec.snu.ac.kr/publish/2009/ID/ChLe-SIGGRAPH-2009.pdf · 2018-04-12 · Cho et al. [2007] used multiple images with space-variant blurs for finding

The human genome Lander et al. 2001 Venter et al. 2001 Patil et al. 2001 Gabriel et al. 2002 Reich et al. 2001 Sabeti et al. 2002 Yu et al. 2002.

The human genome Lander et al. 2001 Venter et al. 2001 Patil et al. 2001 Gabriel et al. 2002 Reich et al. 2001 Sabeti et al. 2002 Yu et al. 2002.

Shape Analysis and Retrieval D2 Shape Distributions Notes courtesy of Funk et al., SIGGRAPH 2004.

Shape Analysis and Retrieval D2 Shape Distributions Notes courtesy of Funk et al., SIGGRAPH 2004.

(12) United States Patent Zhang et al. (45) Date of Patent ... · Lei et al. Murao Fodor et al. Okamura et al. Nguyen et al. Koinuma et al. Zhao et al. Knowles et al. Dobuzinsky et

(12) United States Patent Zhang et al. (45) Date of Patent ... · Lei et al. Murao Fodor et al. Okamura et al. Nguyen et al. Koinuma et al. Zhao et al. Knowles et al. Dobuzinsky et

Image Segmentation Today’s Readings Intelligent Scissors, Mortensen et. al, SIGGRAPH 1995Intelligent Scissors From Sandlot ScienceSandlot Science.

Image Segmentation Today’s Readings Intelligent Scissors, Mortensen et. al, SIGGRAPH 1995Intelligent Scissors From Sandlot ScienceSandlot Science.

Cours de MII 2 - biri/Enseignement/MII2/TDPdf/... · OptiX, NVIDIA 2008 “RenderAnts: Interactive Reyes Rendering on GPUs,” Zhou et al., ACM SIGGRAPH Asia “Hardware-Accelerated

Cours de MII 2 - biri/Enseignement/MII2/TDPdf/... · OptiX, NVIDIA 2008 “RenderAnts: Interactive Reyes Rendering on GPUs,” Zhou et al., ACM SIGGRAPH Asia “Hardware-Accelerated

Unsteady Hydrodynamic Modeling of a Cycloidal PropellerBenedict et al 2014a, Benedict et al 2014b, Zachary et al 2013, Shrestha et al 2012, Shrestha et al 2014, Hrishikeshavan et al

Unsteady Hydrodynamic Modeling of a Cycloidal PropellerBenedict et al 2014a, Benedict et al 2014b, Zachary et al 2013, Shrestha et al 2012, Shrestha et al 2014, Hrishikeshavan et al

Nonphotorealistic rendering Computational Photography, 6.882 Bill Freeman Fredo Durand May 9, 2006 Drawing from: NPR Siggraph 1999 course, Green et al.

Nonphotorealistic rendering Computational Photography, 6.882 Bill Freeman Fredo Durand May 9, 2006 Drawing from: NPR Siggraph 1999 course, Green et al.

et al. et al

et al. et al

(12) United States Patent (io) · Sevcik et al. Ryanetal..... 144/154.5 Govindaraj et al. Kennedy et al. Malone Woods et al. Takahashi et al. Dick et al. Ceroll et al. Caron et al.

(12) United States Patent (io) · Sevcik et al. Ryanetal..... 144/154.5 Govindaraj et al. Kennedy et al. Malone Woods et al. Takahashi et al. Dick et al. Ceroll et al. Caron et al.

EXHIBIT A2 - cases.justia.com · 811996 Konrad 1211996 Koppolu et al. 211997 Duscher et al. 311997 Koppolu et al. 511997 Koppolu et al. 711997 Ashe et al. 511998 Koppolu et al. 911998

EXHIBIT A2 - cases.justia.com · 811996 Konrad 1211996 Koppolu et al. 211997 Duscher et al. 311997 Koppolu et al. 511997 Koppolu et al. 711997 Ashe et al. 511998 Koppolu et al. 911998

EXHIBIT E - cases.justia.com · 811996 Konrad 1211996 Koppolu et al. 211997 Duscher et al. 311997 Koppolu et al. 511997 Koppolu et al. 711997 Ashe et al. 511998 Koppolu et al. 911998

EXHIBIT E - cases.justia.com · 811996 Konrad 1211996 Koppolu et al. 211997 Duscher et al. 311997 Koppolu et al. 511997 Koppolu et al. 711997 Ashe et al. 511998 Koppolu et al. 911998

Global Illumination Jian Huang, CS 594, Fall 2002 This set of slides reference text book and the course note of Dutre et. al on SIGGRAPH 2001.

Global Illumination Jian Huang, CS 594, Fall 2002 This set of slides reference text book and the course note of Dutre et. al on SIGGRAPH 2001.

Quaternary evolution of the Caravelas strandplain – …Dominguez et al. 1998). (Suguio et al. 1985, Martin et al. 1987, 1988, Suguio et al. 1988, Martin et al. 2002). Martin et al.

Quaternary evolution of the Caravelas strandplain – …Dominguez et al. 1998). (Suguio et al. 1985, Martin et al. 1987, 1988, Suguio et al. 1988, Martin et al. 2002). Martin et al.

Shadows 2013 Lecture 1cs5610/lectures/Shadows 2013 Lecture 1.pdf · – Mark Segal, et. al. (SGI), “Fast Shadows and Lighting Effects Using Texture Mapping,” SIGGRAPH 92 46 The

Shadows 2013 Lecture 1cs5610/lectures/Shadows 2013 Lecture 1.pdf · – Mark Segal, et. al. (SGI), “Fast Shadows and Lighting Effects Using Texture Mapping,” SIGGRAPH 92 46 The

More Generative Models - dvl.in.tum.de · Siggraph’9 [Thies et al.]: Neural Textures 3D Geometry Rendering 3D 2D View R, t Neural Texture Neural Textures: Features on 3D Mesh ...

More Generative Models - dvl.in.tum.de · Siggraph’9 [Thies et al.]: Neural Textures 3D Geometry Rendering 3D 2D View R, t Neural Texture Neural Textures: Features on 3D Mesh ...

Rony Goldenthal et al. Presenter : SoHyeon Jeong (SoHyeon.Jeong@gmail.com) ACM SIGGRAPH 2007.

Rony Goldenthal et al. Presenter : SoHyeon Jeong ([email protected]) ACM SIGGRAPH 2007.

Christopher DeCoro Includes content from: Consistent Mesh Parameterizations, Praun et. al, Siggraph 2001 The Space of Human Body Shapes, Allen et. al,

Christopher DeCoro Includes content from: Consistent Mesh Parameterizations, Praun et. al, Siggraph 2001 The Space of Human Body Shapes, Allen et. al,

CMSC 635olano/class/635-03-2/npr.pdfPen-and-Ink Illustration.” Proceedings of SIGGRAPH 97. Pages 402, 403. Real-Time NPR Examples from Markosian et al. “Real-Time Nonphotorealistic

CMSC 635olano/class/635-03-2/npr.pdfPen-and-Ink Illustration.” Proceedings of SIGGRAPH 97. Pages 402, 403. Real-Time NPR Examples from Markosian et al. “Real-Time Nonphotorealistic

To appear in the ACM SIGGRAPH 2010 conference proceedingsvan/papers/2010-TOG-gbwc/paper.pdf · Kajita et al. 2003; da Silva et al. 2008; Yin et al. 2007; Tsai et al. 2010]. Many analytic

To appear in the ACM SIGGRAPH 2010 conference proceedingsvan/papers/2010-TOG-gbwc/paper.pdf · Kajita et al. 2003; da Silva et al. 2008; Yin et al. 2007; Tsai et al. 2010]. Many analytic