CS 395/495-25: Spring 2003CS 395/495-25: Spring 2003

IBMR: Week 6A IBMR: Week 6A

New Directions:New Directions:Applying PApplying P22 & P & P33 Projections Projections

Jack TumblinJack Tumblinjet@cs.northwestern.edujet@cs.northwestern.edu

IBMR-Related SeminarsIBMR-Related Seminars

3D Scanning for Cultural Heritage Applications3D Scanning for Cultural Heritage Applications Holly Rushmeier, IBM TJ Watson Holly Rushmeier, IBM TJ Watson

Friday May 16 3:00pm, Rm 381, CS Dept. Friday May 16 3:00pm, Rm 381, CS Dept.

no title yet ...<BRDF, BRSSDF capture? no title yet ...<BRDF, BRSSDF capture? Optics of Hair? Inverse Rendering?>Optics of Hair? Inverse Rendering?>

Steve Marschner, Cornell University Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.Friday May 23 3:00pm, Rm 381, CS Dept.

RemindersReminders

Take-Home Midterm ExamTake-Home Midterm Exam Due in 48 hours -- Thurs 5/8 3:30pmDue in 48 hours -- Thurs 5/8 3:30pmNo extensions: I’ll hand out solution sheetNo extensions: I’ll hand out solution sheet

Projects A and B ? ‘real soon now’ Projects A and B ? ‘real soon now’ (TA still grading...)(TA still grading...)

----------------------------------------------------------------------------------------------Project CProject C coming Thurs May 8, Due May 15 coming Thurs May 8, Due May 15Project DProject D coming Thurs May 15, Due May 29 coming Thurs May 15, Due May 29Take-Home Final Thurs June 5, Due June 11Take-Home Final Thurs June 5, Due June 11

Cutting Into IBMRCutting Into IBMR

• 5 weeks left:5 weeks left: More Fun, More Fun, More Imaging, Less mathMore Imaging, Less math

• More Recent Research,More Recent Research,Less Book Deciphering… Less Book Deciphering…

• Next:Next:– 2D Warps of all kinds2D Warps of all kinds

• non-planar projections,non-planar projections,• panorama building, panorama building, • ‘‘Light Probes’Light Probes’

– 3D/2D:Camera Matrix(Ch 5)3D/2D:Camera Matrix(Ch 5)– 3D: Epipolar Geom. (Ch 8)3D: Epipolar Geom. (Ch 8)– 4D+: Light Fields & more..4D+: Light Fields & more..

An Image Is…An Image Is…

Image PlaneImage PlaneI(x,y)I(x,y)

Ang

le(

Ang

le(

,, ))

Pos

ition

(x,y

)P

ositi

on(x

,y)

2D Image:2D Image:Collection of rays Collection of rays

through a point through a point

Light + 3D Scene:Light + 3D Scene:Illumination, Illumination,

shape, movement, shape, movement, surface BRDF,… surface BRDF,…

‘‘Center of Center of Projection’Projection’

(P(P33 or P or P22 Origin) Origin)

‘‘Image’ : AngleImage’ : AnglePosition MapPosition Map

• So far: So far: – planar perspective projectionplanar perspective projection– PP22, , PP33 projective transformations projective transformations– how to find these transformations from images.how to find these transformations from images.

• But Planar Perspective Projection But Planar Perspective Projection is just is just ONE KINDONE KIND of mapping, and of mapping, and

• PP2 2 and Pand P3 3 linear projective transformation linear projective transformation HH is just is just ONE KINDONE KIND of image warping of image warping

H = HH = HSSHHAAHHPP

‘‘Image’ : AngleImage’ : AnglePosition MapPosition Map

• All CamerasAll Cameras make mistakes make mistakes (geometry,shading/vignettes, high (geometry,shading/vignettes, high contrast,...) contrast,...) book shows how to correct a few of them.book shows how to correct a few of them.

• Many kinds of camera projections:Many kinds of camera projections:‘Fun-House Mirrors’‘Fun-House Mirrors’– Non-planar perspective projections:Non-planar perspective projections:

• Spherical, Conical, cylindrical, hyperbolic; Spherical, Conical, cylindrical, hyperbolic; even ‘centerless’ projections (Rademacher, SIGG 1998)even ‘centerless’ projections (Rademacher, SIGG 1998)

– Why? Samples of the Light Field...Why? Samples of the Light Field...

• Early forms of IBMR?: warp and ‘morph’Early forms of IBMR?: warp and ‘morph’

Image Warping: General IdeaImage Warping: General Idea

2D2D2D continuous coordinate map, a ‘rubber sheet’2D continuous coordinate map, a ‘rubber sheet’Forward: find output address for each INPUT pt.Forward: find output address for each INPUT pt.

• Demo:Demo: http://www.http://www.angelfireangelfire.com/biz/.com/biz/beamersandblastersbeamersandblasters//PicWarpPicWarp.html.html..

• One way: One way: Patch corner-by-patch corner; bilinear fill...Patch corner-by-patch corner; bilinear fill...

u = u(x,y)=u = u(x,y)=v = v(x,y)=…v = v(x,y)=…

xx

yy

uu

vv1 y y1 y y22

1 1 x x xx22

‘‘Forward Forward Mapping’Mapping’

inputinput

outputoutput

Image Warping: General IdeaImage Warping: General Idea

2D2D2D continuous coordinate map, a ‘rubber sheet’2D continuous coordinate map, a ‘rubber sheet’Inverse: find input address for each OUTPUT pt.Inverse: find input address for each OUTPUT pt.

• One way: One way: Patch-corner by patch-corner; bilinear fill...Patch-corner by patch-corner; bilinear fill...

• x = x(u,v)=x = x(u,v)=y = y(u,v) = …y = y(u,v) = …

uu

vv

xx

yy

1 v v1 v v22

1 1 u u uu22

‘‘Inverse Inverse Mapping’Mapping’

inputinputoutputoutput

Image Warping: General IdeaImage Warping: General Idea

2D2D2D continuous coordinate map: ‘rubber sheet’2D continuous coordinate map: ‘rubber sheet’

PROBLEM:PROBLEM: inverse inverse requiredrequired sometimes (mess) sometimes (mess)

PROBLEM: PROBLEM: pixels aren’t continuous; sampling errorspixels aren’t continuous; sampling errors– aliasing (output pixels skip some input pixels)aliasing (output pixels skip some input pixels)– holes (input pixels skip some output pixels)holes (input pixels skip some output pixels)

xx

yy

uu

vv

Image Warping: IBMR FormImage Warping: IBMR Form

2D2D2D continuous coordinate map: ‘rubber sheet’2D continuous coordinate map: ‘rubber sheet’

PROBLEM:PROBLEM: inverse inverse requiredrequired sometimes (mess) sometimes (mess)

IBMR Answer:IBMR Answer: H matrix is H matrix is (usually) invertible invertible

PROBLEM: PROBLEM: pixels aren’t continuous; sampling errorspixels aren’t continuous; sampling errors

IBMR Answer:IBMR Answer: Use vertices, not pixels: Use vertices, not pixels: let let OpenGL texture mapping keep the image “continuous”OpenGL texture mapping keep the image “continuous”

xx33

xx22

xx11

Further Reading on Warps:Further Reading on Warps:

• Many Misc. SIGGRAPH tutorials, many on-lineMany Misc. SIGGRAPH tutorials, many on-line

• T. Beier and S. Neely. T. Beier and S. Neely. Feature Based Image Feature Based Image MetamorphosisMetamorphosis. . Computer Graphics (Siggraph'92 Computer Graphics (Siggraph'92 Proceedings), Vol. 26 : pages 35--42, July 1992Proceedings), Vol. 26 : pages 35--42, July 1992

• George Wolberg George Wolberg Digital Image WarpingDigital Image Warping (book)©1990, Wiley & Sons(book)©1990, Wiley & Sons

• Jonas Gomes, Lucia Darsa, Bruno Costa, Luiz Velho, Jonas Gomes, Lucia Darsa, Bruno Costa, Luiz Velho, Warping and Morphing of Graphical ObjectsWarping and Morphing of Graphical Objects    ©1999 ©1999 Morgan Kaufmann Publishers Morgan Kaufmann Publishers

Image Warping: IBMR FormImage Warping: IBMR Form

• Projective transformation H in PProjective transformation H in P22 is a planar ‘warp’ is a planar ‘warp’• But many other But many other kindskinds of warps possible! of warps possible!

• P2 change object/camera position for viewing a plane P2 change object/camera position for viewing a plane • P3 changes object/camera postion for viewing a 3D shapeP3 changes object/camera postion for viewing a 3D shape• But shape changes are limited...But shape changes are limited...

Panoramas: Planar ‘Bow-Tie’Panoramas: Planar ‘Bow-Tie’

– For limited-size mosaics only (angle limits)For limited-size mosaics only (angle limits)– Find all Find all HH from correspondence in overlapped regions from correspondence in overlapped regions– Choose a (central) reference image Choose a (central) reference image (book pg. 196)(book pg. 196)

– Reproject, cross-dissolve in reference image planeReproject, cross-dissolve in reference image plane

Planar Panoramas in PPlanar Panoramas in P22 or P or P33

• Choose a ‘reference’ image plane, extend itChoose a ‘reference’ image plane, extend it

• Add images: for each one, Add images: for each one, – find find H H from overlap correspondences (in from overlap correspondences (in PP22))– transform new image to reference planetransform new image to reference plane– re-sample, ‘blend’ re-sample, ‘blend’

(weighted sum) (weighted sum) to one imageto one image

xx33

xx22

xx11

xx33

xx22

xx11

HH11 HH22

HH3…3…

Panorama MakingPanorama Making

Planar:Planar:

• Start with overlapped planar proj. imagesStart with overlapped planar proj. images

• Do 4-point corresp. (or better) for alignmentDo 4-point corresp. (or better) for alignment

• Merge images by cross-dissolve:Merge images by cross-dissolve:(R,G,B)(R,G,B)outout = (R,G,B) = (R,G,B)11·· + (R,G,B) + (R,G,B)22·(1-·(1-))

where:where:

= opacity= opacity11

= barycentric coords= barycentric coords

= linear interp between edges:= linear interp between edges:

11 22

Why use P3 at all?Why use P3 at all?

Make Projective Images from Non-planar objectsMake Projective Images from Non-planar objectsSeparate camera & imaged object coordinatesSeparate camera & imaged object coordinatesSeparate projective x4 from the R3 ‘z’ coord;Separate projective x4 from the R3 ‘z’ coord;

P2 P2 seemsseems to bridge 2D & 3D, but doesn’t: to bridge 2D & 3D, but doesn’t:– -z, +z axis in R3 ‘folds back on itself’ -z, +z axis in R3 ‘folds back on itself’ – -x-x33, +x, +x33 in P2 MUST be symmetric! in P2 MUST be symmetric!

“P2 is a symmetric half-space” pgXX “P2 is a symmetric half-space” pgXX– Only +/-z is “camera’s direction of gaze”Only +/-z is “camera’s direction of gaze”

P3 includes full, complete 3D space; P3 includes full, complete 3D space; – -z, +z can be asymmetric-z, +z can be asymmetric– no ‘camera direction’no ‘camera direction’

BUT ALSO: Non-Planar Panoramas . . .BUT ALSO: Non-Planar Panoramas . . .

Can’tCan’t use planar method beyond 180 use planar method beyond 180°° FOV; FOV;

• Sphere or cylinder can ‘wrap around’ originSphere or cylinder can ‘wrap around’ origin

• How? Spherical coords How? Spherical coords – write 3D sphere eqn in Pwrite 3D sphere eqn in P22 coords: coords:

xx11 = cos( = cos()cos()cos() )

xx33 = -sin( = -sin()cos()cos())

xx22 = cos( = cos()) =longitude, =longitude, = latitude= latitude

– ‘‘Inverse Map’ warp:Inverse Map’ warp:• Output pixel m,n: Output pixel m,n: = 2 = 2n/nmax,n/nmax, = = n/nmax - n/nmax - /2/2

• Find x’ = HFind x’ = H-1-1·[x·[x11, x, x22, x, x33] for each image] for each image• Blend color(s) found at x’ for each imageBlend color(s) found at x’ for each image

Non-Planar PanoramasNon-Planar Panoramas

-x-x33

xx22

xx11

Example: QuickTime VR Example: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

1) Four Planar Images 1) Four Planar Images 1 Cylindrical Panorama: 1 Cylindrical Panorama:

IN:IN:

OUT:OUT:

Example: QuickTime VR Example: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

2) Windowing, Horizontal-only Reprojection:2) Windowing, Horizontal-only Reprojection:

IN:IN:

OUT:OUT:

Spherical Panoramas: Spherical Panoramas:

• Note wildly non-uniform sampling in Note wildly non-uniform sampling in ,,::

Richard Szeliski et al, Microsoft Research, 1998

Practical Panoramas: ‘Box Cross’Practical Panoramas: ‘Box Cross’

• Spherical maps oversample near poles;Spherical maps oversample near poles;

• Cylindrical maps can’t see floor, ceiling spotCylindrical maps can’t see floor, ceiling spot

• Nice solution: ‘Box Cross’Nice solution: ‘Box Cross’– ‘‘unwrap’ a cube around originunwrap’ a cube around origin– 6 square planar images6 square planar images– Easy! Easy!

• for each image,for each image,• for each box side,for each box side,• find reprojection find reprojection HH• find pixels on boxfind pixels on box• rewarp as needed (cyl,sphere, etc.)rewarp as needed (cyl,sphere, etc.)

frontfront rightrightleftleft

toptop

bottombottom

backback

xx33

xx22

xx11

Example: Example: Spherical Spherical PanoramaPanorama

aa

Paul Debevec, from Paul Debevec, from SIGGRAPH2001 short SIGGRAPH2001 short

course ‘Image-Based course ‘Image-Based Lighting”Lighting”

CAREFUL! CAREFUL! EASY TO MISS EASY TO MISS SOMETHING!SOMETHING!

Side Note: Camera DistortionsSide Note: Camera Distortions

Correct many camera/lens errors in PCorrect many camera/lens errors in P22

• Place raw camera input at (x,y,1) planePlace raw camera input at (x,y,1) plane• Write equations for 3D ‘projection surface’Write equations for 3D ‘projection surface’

Example:Example: scaleable, offset sphere for spherical distortion scaleable, offset sphere for spherical distortionxx11 = a+ b*sin( = a+ b*sin()cos()cos(); x); x33 = c+ d*sin( = c+ d*sin()cos()cos(); x); x22 = e+f*cos( = e+f*cos())

• Scan projection surface to find output pixelsScan projection surface to find output pixelsExample:Example: take equal-sized steps in ( take equal-sized steps in (,,); sample input image); sample input image

xx33

xx22

xx11

Panoramic CamerasPanoramic Cameras

Panoramics without ‘stitching’:Panoramics without ‘stitching’:

• ‘‘Fisheye’ Lenses, conics,…Fisheye’ Lenses, conics,…

• Slit-scan: Slit-scan: (WideLux, Noblex, PanoScan…)(WideLux, Noblex, PanoScan…) – cyl. or spherical imagecyl. or spherical image– slow! no action shotsslow! no action shots

• Multiple Planar Cams Multiple Planar Cams – Fast, flexible, expensiveFast, flexible, expensive– can do panor. moviescan do panor. movies

• History: 1843…History: 1843…http://www.panoramicphoto.com/timeline.htmhttp://www.panoramicphoto.com/timeline.htm

Light ProbesLight Probes

• Example images (see Debevec’s site)Example images (see Debevec’s site)

Paul Debevec, SIGGRAPH2001 course “Image Based Lighting”

Try it yourself…Try it yourself…

..Paul Debevec,

SIGGRAPH2001 short course

“Image Based Lighting”

Try it yourself…Try it yourself…

..Paul Debevec,

SIGGRAPH2001 short course

“Image Based Lighting”

Better! We ordered 4 Better! We ordered 4 Chrome Spheres Chrome Spheres

for IBMR students to for IBMR students to borrow...borrow...

Mirror BallMirror BallPanorama ConversionPanorama Conversion

Makes an offset ‘virtual’ sphere camera Makes an offset ‘virtual’ sphere camera located at mirror ball center:located at mirror ball center:

• ? How can we write this in P? How can we write this in P22 and/or P and/or P33??

yymm

zzmm

xxmm

zzcc

yycc

xxcc

Camera Camera Mirror BallMirror Ball

Mirror BallMirror BallPanorama ConversionPanorama Conversion

Not Documented?!? Then figure it out…Not Documented?!? Then figure it out…– Parameterize ball as ‘rings’ of constant Parameterize ball as ‘rings’ of constant – (Ignore invisible backside of sphere)(Ignore invisible backside of sphere)– Warp 1: get ring colors from camera image Warp 1: get ring colors from camera image

yymm

zzmm

xxmm

zzcc

yycc

xxcc

CameraCamera Mirror BallMirror Ball

Mirror BallMirror BallPanorama ConversionPanorama Conversion

– Parameterize ball using ‘rings’ of constant Parameterize ball using ‘rings’ of constant – Warp 1: read color from rings’ camera imageWarp 1: read color from rings’ camera image– Find reflected ray angle Find reflected ray angle ’’ (constant on a ring)(constant on a ring)– (presume the ball diameter ~zero; then (presume the ball diameter ~zero; then ’ = 2’ = 2))

– Use ring at angle Use ring at angle ’’ on spherical projection on spherical projection

yymm

zzmm

zzcc

yycc xxcc

CameraCamera Mirror BallMirror Ball

’’

Light ProbesLight Probes

• ‘‘One-shot’ panoramic cameraOne-shot’ panoramic camera• Clever, fast, simple, cheap, flexibleClever, fast, simple, cheap, flexible• Probe position != Camera position; telephotos…Probe position != Camera position; telephotos…

– allows small probes in tight, risky spacesallows small probes in tight, risky spaces– Little/no image alignment / mosaicingLittle/no image alignment / mosaicing

• Drawbacks:Drawbacks:– Highly non-uniform samplingHighly non-uniform sampling– Camera ALWAYS in the imageCamera ALWAYS in the image

• Daydreams: a better probe? Daydreams: a better probe? – Huge: mirrored weather balloon?Huge: mirrored weather balloon?– Tiny, stochastic: bubbles in a liquid?Tiny, stochastic: bubbles in a liquid?– Dynamic shapes: whirling mirror on arm? Dynamic shapes: whirling mirror on arm? – Other shapes: Nayyar, Carlbom, Shum(MSRChina) etc.Other shapes: Nayyar, Carlbom, Shum(MSRChina) etc.

ENDEND