Creating images the 2-D way Jean-François Lalonde April 20, 2010.
-
Upload
ricardo-rippon -
Category
Documents
-
view
213 -
download
0
Transcript of Creating images the 2-D way Jean-François Lalonde April 20, 2010.
![Page 1: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/1.jpg)
Creating images the 2-D way
Jean-François LalondeApril 20, 2010
![Page 2: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/2.jpg)
Creating images (3-D)
![Page 3: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/3.jpg)
Creating images (2-D + 3-D)
![Page 4: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/4.jpg)
Inserting objects into images
![Page 5: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/5.jpg)
Inserting objects into images(2-D + 3-D)
[Debevec, ’98]
![Page 6: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/6.jpg)
Inserting objects in images
Realistic renderings
Expensive and impractical
Highly detailed geometryHighly detailed materials
Very expensive
[Debevec, ’98]
![Page 7: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/7.jpg)
Alternative: Clip art
Easy, intuitive, cheap
Not realistic
![Page 8: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/8.jpg)
Creating images (2-D)
Photo-realistic Cartoon
Expensive and impractical Cheap and intuitive
Image-based rendering
Clip ArtPhoto Clip Art
??
![Page 9: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/9.jpg)
“Photoshop-ing”
Composite by David Dewey
![Page 10: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/10.jpg)
Inserting objects into images
![Page 11: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/11.jpg)
Challenges
object orientationobject orientation
scene illuminationscene illumination
Insert THIS object: impossible!
![Page 12: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/12.jpg)
The use of dataInsert SOME object: much easier!
![Page 13: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/13.jpg)
QuickTime™ and aMPEG-4 Video decompressor
are needed to see this picture.
The Google modelDatabaseDatabase
Sort the objectsSort the objects
QueryQuery ResultsResults
![Page 14: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/14.jpg)
2-D image vs 3-D scene
![Page 15: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/15.jpg)
Outline
Phase I: Database annotation
Phase II: Object insertion
Name: personSubgroup: person, standingHeight: 1.5m Local context: in shadowIllumination: sunny, bright day, no cloudSegmentation quality: excellent, >40 pointsUpsampling blur: low
![Page 16: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/16.jpg)
Data source: LabelMe
Online (http://labelme.csail.mit.edu), user-contributed
170,000 objects in 40,000 images
Polygons and names
[Russell et al., 2005]
![Page 17: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/17.jpg)
Data organization
Top-level categories (chosen manually, 16 total)
Second-level categories (from annotations or clustering)
![Page 18: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/18.jpg)
Annotating the objectsAnnotating the objects
![Page 19: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/19.jpg)
Camera parameters
Assumeflat ground plane
all objects on ground
camera roll is negligible (consider pitch only)
Camera parameters: height and orientation
![Page 20: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/20.jpg)
Human height Human height distributiondistribution
1.7 +/- 0.085 m1.7 +/- 0.085 m(National Center for Health (National Center for Health
Statistics)Statistics)
Car height Car height distributiondistribution
1.5 +/- 0.19 m1.5 +/- 0.19 m(automatically (automatically
learned)learned)
Camera parameters
![Page 21: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/21.jpg)
Object heightsDatabase image
Pixel heights Real heights
![Page 22: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/22.jpg)
Object Estimated average height (m)
Car 1.51
Man 1.80
Woman 1.67
Parking meter 1.36
Fire hydrant 0.87
Estimated object heights
1.0 m
Car
0.5 m
Man Woman Parkingmeter
Firehydrant
1.5 m
![Page 23: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/23.jpg)
Geometry is not enough
![Page 24: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/24.jpg)
Illumination context
Database image
Environment map rough approximation
Exact environment map is impossibleApproximations [Khan et al., ‘06]
![Page 25: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/25.jpg)
Illumination contextDatabase image P(pixel|class) CIE L*a*b* histograms
Automatic Photo PopupAutomatic Photo PopupHoiem Hoiem et al.et al., SIGGRAPH ‘05, SIGGRAPH ‘05
Automatic Photo PopupAutomatic Photo PopupHoiem Hoiem et al.et al., SIGGRAPH ‘05, SIGGRAPH ‘05
![Page 26: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/26.jpg)
Illumination nearest-neighbors
![Page 27: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/27.jpg)
Other criteria: local context
![Page 28: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/28.jpg)
Other criteria: segmentation
LabelMe contributors not always reliable
Segmentation quality38 points / polygon 4 points / polygon
![Page 29: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/29.jpg)
Other criteria: blur
Resolution: avoid up-samplingx3 up-sampling
![Page 30: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/30.jpg)
Recap
Phase I: Database annotation
Phase II: Object insertion
3-D height Segmentation Blur
Object properties (used for sorting the database)
Label Illumination contextCluster Local contextCamera
![Page 31: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/31.jpg)
Let’s insert an object!Poor user-provided segmentations
Noticeable seams
![Page 32: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/32.jpg)
SeamsInput Destination image
Result
Visible
seam!
[Perez et al., 2003]
![Page 33: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/33.jpg)
[Perez et al., 2003]
Poisson blending: ideaInput Destination
Result
Enforce boundary color
(seamless result)
Enforce same gradient than input
![Page 34: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/34.jpg)
Why gradients? 1-D example
Regular blending
bright
dark
![Page 35: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/35.jpg)
1-D example
Blending derivatives
Original signals Derivatives
Reintegration results
![Page 36: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/36.jpg)
1-D exampleGradient domainIntensity domain
?
![Page 37: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/37.jpg)
2-D: not so easy
+1
+2
-3
4
0 2
5
-2
Non integrable: sum over a loop Non integrable: sum over a loop ≠ 0≠ 0
Actually happens all the time in Actually happens all the time in practicepractice
![Page 38: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/38.jpg)
2-D: some notation
Finite differences
![Page 39: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/39.jpg)
2-D: a (possible) solution
Least-squares solution:
??
![Page 40: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/40.jpg)
Solution: Poisson equation
Popular because:Solution is obtained by solving a linear system of equations
Can be solved (somewhat) efficiently
‘\’ in matlab
FFT
Multi-grid solvers (approximate, but really fast!)
2-D: a (popular) solution
![Page 41: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/41.jpg)
Results & limitations
Image editing
Some limitationsImages need to be very well aligned
Differences in background “bleed through”
Images from [Perez et al., 2003]
![Page 42: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/42.jpg)
Poisson blending: improvements
Drag-and-Drop Pasting
[Jia et al., 2006]
User-selectedboundary
Poisson blending
Refinedboundary
Poisson blending
Images from [Jia et al., 2006]
Color bleeding
![Page 43: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/43.jpg)
Still not right!
Not so sensitive to shadow direction [Cavanagh, 2005]
![Page 44: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/44.jpg)
Shadow transfer
++++ ====
Database image Shadow estimate Refined shadow
Object alone Shadow alone Object with shadow
![Page 45: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/45.jpg)
User interface
QuickTime™ and aMPEG-4 Video decompressor
are needed to see this picture.
![Page 46: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/46.jpg)
Street accident
![Page 47: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/47.jpg)
Bridge
![Page 48: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/48.jpg)
Painting
![Page 49: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/49.jpg)
Alley
![Page 50: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/50.jpg)
Failure cases
Shadow transferShadow transfer
Porous objectsPorous objects
![Page 51: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/51.jpg)
Failure cases
Best matching objectsBest matching objects
![Page 52: Creating images the 2-D way Jean-François Lalonde April 20, 2010.](https://reader035.fdocuments.us/reader035/viewer/2022070306/551676295503461c658b4a19/html5/thumbnails/52.jpg)
Pros & cons
Pros Cons
3-DMore control(camera, geometry,
lighting...)Complex!
2-D + 3-D RealisticComplex!
Need access to scene
2-D(photoshop)
RealisticComplex!
Need to sort through
images 2-D
(automatic)Easy, intuitive “Generic” objects