Using deep learning to bypass the green screen · Using deep learning to bypass the green screen...
Transcript of Using deep learning to bypass the green screen · Using deep learning to bypass the green screen...
![Page 1: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/1.jpg)
The ADAPT Centre is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.
Using deep learning to bypass the green screen
Marco Forte, François Pitié, Sigmedia
![Page 2: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/2.jpg)
www.adaptcentre.ieGreenscreen keying
Green Screens are used by film and television industry for background replacement.
High quality results still take a lot of artist time, though lower quality is achievable in real time.
| 2
![Page 3: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/3.jpg)
www.adaptcentre.ieNatural Image matting
Alpha matting refers to the problem of extracting the opacity/transparency mask, alpha matte, of an object in an image.
The goal being to compose the object onto a new background.
| 3
Alpha matte
Define unknown regionsImage of object
![Page 4: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/4.jpg)
www.adaptcentre.ieCompositing
To composite an object onto a novel background we need• Object foreground• Alpha matte• background image
I = αF + (1 - α)B
| 4
Object foreground Alpha matte
Background on which to composite(10 pts if you recognise this place)
![Page 5: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/5.jpg)
www.adaptcentre.ieOther applications
| 5
Automatic Portrait Segmentation for Image Stylization
Xiaoyong Shen1 Aaron Hertzmann2 Jiaya Jia1 Sylvain Paris2 Brian Price2 Eli
Shechtman2 Ian Sachs2
1The Chinese Univeristy of Hong Kong 2Adobe Research
![Page 6: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/6.jpg)
www.adaptcentre.ieOther applications
| 6
![Page 7: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/7.jpg)
www.adaptcentre.ieImage matting with CNNs
| 7
CNN
![Page 8: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/8.jpg)
www.adaptcentre.ieTraining procedure - Dataset
| 8
Time consuming to create
manually.
Highest quality needs still object
to be captured in front of
monitor with changing
backgrounds.
Otherwise can manually
annotate existing images with
clean backgrounds in
photoshop.
Greenscreen also possible in
controlled HD or UHD
environment.
We a created dataset of 500
foreground and alpha pairs. Adobe
created one of 450 pairs.
![Page 9: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/9.jpg)
www.adaptcentre.ieProperties of using CNNs for image matting
| 9
● Existing method from Adobe is top
ranking yet uses very large difficult
to optimise network.
● Performs something more akin to
segmentation rather than alpha
matting
● Mathematics of alpha matting
requires some matrix inversion
which is difficult to learn with
standard conv layers structure.
![Page 10: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/10.jpg)
www.adaptcentre.ieTraining procedure - Dataset
| 10
Dataset is small only ~450-1000
images.
Lots of data augmentation
needed.
Composite the foreground onto
1000s of different backgrounds
Random cropping of different
size.
Crop rotation and mirroring.
Slight changes to foreground
contrast and brightness
![Page 11: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/11.jpg)
www.adaptcentre.ieWait actually we need that greenscreen....
| 11
1. We could get thousands of ground truth frames by using a
greenscreen. :)
1. High quality keying is actually non-trivial :/
1. Artists don’t have a very scientific approach, they use a
mismatch of keys with different settings.
1. To get really high quality ground truth we’ll need really
high quality cameras
1. Automatic tools kinda suck, need to make our own
![Page 12: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/12.jpg)
www.adaptcentre.ieGreenscreen setup
| 12
![Page 13: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/13.jpg)
www.adaptcentre.ieAutomatic methods aren’t good enough
| 13
Automatic methods don’t provide good ground truth data.
They may look ok, but there’s alot of detail lost, noise
introduced and color spill not fully removed
![Page 14: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/14.jpg)
www.adaptcentre.ieAlpha matte
| 14
![Page 15: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/15.jpg)
www.adaptcentre.ieImage matting with CNNs
| 15
CNN
Our approach - Joint prediction of alpha
foreground and background.
![Page 16: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/16.jpg)
www.adaptcentre.ieImage matting with CNNs
| 16
Alpha Loss = ∑(𝞪 - 𝞪gt )
Foreground loss = ∑∑(Fg - Fggt )
Background loss = ∑∑(Bg - Bggt )
Loss = ƛL𝞪 + (1-ƛ)(LFg + LBg)
Only define losses on well
defined regions
![Page 17: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/17.jpg)
www.adaptcentre.ieBenefits of modelling foreground and bg
| 17
![Page 18: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/18.jpg)
www.adaptcentre.ieBenefits of modelling foreground and bg
| 18
![Page 19: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/19.jpg)
www.adaptcentre.ieBenefits of modelling foreground and bg
| 19
Direct alpha prediction Joint prediction
![Page 20: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/20.jpg)
www.adaptcentre.ieBenefits of modelling foreground and bg
| 20
Direct alpha prediction Joint prediction
![Page 21: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/21.jpg)
www.adaptcentre.ieSome example results of our network
| 21
![Page 22: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/22.jpg)
www.adaptcentre.ieVideo examples
| 22
![Page 23: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/23.jpg)
www.adaptcentre.ieVideo examples
| 23
![Page 24: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/24.jpg)
www.adaptcentre.ieLessons learned
| 24
1. High quality training data is extremely important
1. Pretrained encoder network essential, helps in all aspects,
not just coarse segmentation but also fine details.
Resnet > Vgg
1. Multitask learning is beneficial
1. Patience when training deep networks, reproducing
another paper took 3 weeks of training time to converge.
1. Deep learning can fail with images that classical algorithms
have no problems with.
![Page 25: Using deep learning to bypass the green screen · Using deep learning to bypass the green screen Marco Forte, François Pitié, Sigmedia. Greenscreen keying Green Screens are used](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec46d10da3ccc65a200f224/html5/thumbnails/25.jpg)
www.adaptcentre.ieBenefits moving forward
| 25
● More loss functions possible
○ Impose constraints on foreground and background when
specifically training for keying
○ General image inpainting loss
○ Impose independence of foreground and background
○ Adversarial triplet loss on Fg, Bg, A
○ Adversarial fg, bg, alpha reconstruction loss
● More practical for artist to work directly with both alpha and
foreground
● Generalises to video better, for example in situations with
stationary background or stationary foreground