Image Compositing and Matting. Introduction Matting and compositing are important operations in the...
-
Upload
amber-ryan -
Category
Documents
-
view
227 -
download
6
Transcript of Image Compositing and Matting. Introduction Matting and compositing are important operations in the...
![Page 1: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/1.jpg)
Image Compositing and Matting
![Page 2: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/2.jpg)
Introduction
Matting and compositing are important operations in the production of special effects. These techniques enable directors to embed actors in a world that exists only in imagination, or to revive creatures that have been extinct for millions of years.
During matting, foreground elements are extracted from a film or video sequence.
During compositing, the extracted foreground elements are placed over novel background images.
![Page 3: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/3.jpg)
Traditional approaches to matting
Traditional approaches to matting include blue-screen matting and rotoscoping.
The former requires filming in front of an expensive blue screen under carefully controlled lighting
The latter demands talent and intensive user interaction.
![Page 4: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/4.jpg)
Rotoscoping
Rotoscoping --- the process of tracking contours in a video sequence
![Page 5: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/5.jpg)
Basic problem of blue screen matting
Given an image of a foreground object shot in front of a backing color (blue screen or green screen or other colors)
Obtain a matte of the foreground object so that the foreground object can be blended into a new background image using the matte to produce a new composite image.
![Page 6: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/6.jpg)
Notations Let C = [R, G,B] denote a color with 0 ≤ R, G, B ≤ 1. Let denotes a transparency value with 0 ≤ ≤ 1. Foreground image color: Cf = [Rf ,Gf ,Bf ], f = 1
Backing (screen) color: Ck = [0, 0,Bk], k = 1 (assuming blue screen)
Original foreground object color: Co = [Ro,Go,Bo]
Background image color: Cb = [Rb,Gb,Bb], b = 1
Composite image color: Cc = [Rc,Gc,Bc]
![Page 7: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/7.jpg)
Problem Statement
Given Cf and Cb at corresponding pixels, and Ck a known backing color, and assuming Cf = oCo + (1 − o)Ck
Determine o and Co, which then gives the composite color Cc = oCo + (1 − o)Cb at the corresponding point, for all points that Cf and Cb share in common.
![Page 8: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/8.jpg)
Solution
Since Cf = oCo + (1 − o)Ck , we have:
4 unknowns, 3 equations…
Rf = oRo Gf = oGo
Bf = oBo + (1 − o)Bk .
![Page 9: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/9.jpg)
Case 1: No Blue
There is no blue in Co, i.e., Bo=0, and Bk≠0. Then
This case is very restrictive. It rules out many colors, including grays
because grays have blue.
![Page 10: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/10.jpg)
Case 2: Gray and Skin Color
Assume Ro=aBo (or Go=aBo) and Bk≠0 Then
where
![Page 11: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/11.jpg)
Example 1: Gray
Then
This case applies to science fiction movie in which the spaceships are mostly gray.
![Page 12: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/12.jpg)
Example 2: Skin Color
This case applies to human faces, hands, legs, etc.
![Page 13: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/13.jpg)
Solution for General Cases
To solve the matting problem in general: Need to take the same image with two different
backing colors. This gives four equations for solving the four
unknowns Case 1: Use Two Different Shades of Blue. Case 2: Use Two Different Backing Colors.
![Page 14: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/14.jpg)
Case 1
Use two different shades of blue Bk1 and Bk2 as backing colors.
Then
so
![Page 15: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/15.jpg)
Case 2
Use two different backing colors Ck1 and Ck2.
Or
Over-constrained, 1 unknowns, three equations
![Page 16: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/16.jpg)
Case 2: Solution 1
Adding up 3 equations:
![Page 17: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/17.jpg)
Case 2: Solution 2
Apply least squares method Define:
Then
Least-squares
![Page 18: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/18.jpg)
Least-Squares
Set
![Page 19: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/19.jpg)
Case 1 Example
![Page 20: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/20.jpg)
Case 2 Example
![Page 21: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/21.jpg)
Other Developments in Matting
Matting Without Blue Screen A method proposed by Ruzon and Tomasi
User specify object region and boundary region. Alpha value of object region is set to 1. Alpha value of boundary region is computed by es
timating the contributions of neighboring objects’ colors.
![Page 22: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/22.jpg)
Example
![Page 23: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/23.jpg)
Bayesian Approach
![Page 24: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/24.jpg)
Shadow Matting
Pull a matte of shadow. Acquire photometric and geometric properties
of the target scene by sweeping oriented linear shadows across it.
Then, composite the shadow onto the scene.
![Page 25: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/25.jpg)
Example
![Page 26: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/26.jpg)
Environmental Matting
Left: alpha matte. Middle: environment matte. Right: photo.
![Page 27: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/27.jpg)
References1. Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski. A baye
sian approach to digital matting. In Proc. IEEE CVPR, pages II–264–II–271, 2001.
2. Y.-Y. Chuang, D. B. Goldman, B. Curless, D. H. Salesin, and R. Szeliski. Shadow matting and compositing. ACM Transactions on Graphics, 22(3):494–500, July 2003.
3. M. A. Ruzon and C. Tomasi. Alpha estimation in natural images. In Proc. IEEE CVPR, pages 18–25, 2000.
4. A. R. Smith and J. F. Blinn. Blue screen matting. In Proc. ACM SIGGRAPH, pages 259–268, 1996.
5. D. E. Zongker, D. M. Werner, B. Curless, and D. H. Salesin. Environment matting and compositing. In Proc. SIGGRAPH, pages 205–214, 1999.
![Page 28: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/28.jpg)
Digital Compositing
Digital compositing means “digitally manipulated integration of at least two source images to produce a new image.”
The new image must appear realistic. It must be completely and seamlessly integrat
ed, as if it were actually photographed by a single camera.
![Page 29: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/29.jpg)
Example 1
![Page 30: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/30.jpg)
Example 2
![Page 31: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/31.jpg)
More examples
http://www.beezlebugbit.com/digital/efx/efx_top.htm
![Page 32: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/32.jpg)
Main Topics
Alpha blending: blending foreground and background
Keying: separating foreground and background Luma, chroma, difference keying
Rig removal: removing unwanted elements
![Page 33: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/33.jpg)
Alpha Blending
C = [α F + (1 – α) B] If α = 1, then C = F, foreground is shown, i.e.,
foreground is opaque. If α = 0, then C = B, background is shown, i.
e., foreground is transparent. 0 < α < 1: semi-transparent, e.g., shadow, sm
oke, etc. If α ranges from 0 to 255, then the formula be
comes: C = [α F + (1 – α) B] / 255
![Page 34: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/34.jpg)
Example: No Background
![Page 35: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/35.jpg)
Example: With Background
![Page 36: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/36.jpg)
![Page 37: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/37.jpg)
Note
For shadow, must take fractional value (0 < α < 1). Otherwise, shadow looks unreal.
![Page 38: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/38.jpg)
Boundary area
at boundary area should also be fractional. Otherwise, have dark fringes; unrealistic.
![Page 39: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/39.jpg)
Summary
A good matte has fractional in shadow, and along object boundaries and shadow boundaries.
![Page 40: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/40.jpg)
Keying
Separating foreground from background, creating a matte of foreground.
Also called pulling a matte (of foreground), or keying out (i.e., making transparent) background.
Recall: A good matte has fractional a in shadow, and alon
g object boundaries and shadow boundaries.
![Page 41: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/41.jpg)
Basic methods
Luma keying: based on luminance (i.e., intensity)
Chroma keying: based on color (i.e., blue screen, green screen)
Difference keying: requires a clean plate, i.e., a background image without the foreground element.
![Page 42: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/42.jpg)
Basic Idea
Compute difference between foreground and background (based on luma, chroma, or color)
Very small diff = 0. Very large diff = 1. Intermediate diff inte
rmediate
![Page 43: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/43.jpg)
Luma Keying
Key out the background based on luminance. Useful when background has a uniform lumin
ance that is very different from foreground luminance.
![Page 44: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/44.jpg)
Result
![Page 45: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/45.jpg)
Chroma Keying
Key out the background based on color. Useful when background has a uniform color
that is very different from foreground color. Example: Image shot with blue screen.
![Page 46: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/46.jpg)
Characteristics of blue screen image
![Page 47: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/47.jpg)
Difference Keying
More general than luma and chroma keying. Key out background based on pixel-wise color di
fference between foreground and background footage.
![Page 48: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/48.jpg)
Final Composition
![Page 49: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/49.jpg)
Rig Removals
Rigs are equipment that support the actors or the props.
Sometimes, rigs cannot be removed by keying alone. So, have to apply masking technique to remove rigs. Need clean plate of background footage. If camera moves, then need motion-controlled came
ra: Computer controls camera to move the same way t
wice: Without foreground objects; get clean plate. With foreground objects.
![Page 50: Image Compositing and Matting. Introduction Matting and compositing are important operations in the production of special effects. These techniques enable.](https://reader035.fdocuments.us/reader035/viewer/2022062308/56649d005503460f949d2d4f/html5/thumbnails/50.jpg)
Basic Idea
Apply a mask to mask out the rig. Then, replace pixels in masked area by corre
sponding pixels in clean plate background. If rig moves in footage, then have to animate
the mask accordingly.