Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution,...
Transcript of Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution,...
![Page 1: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/1.jpg)
Addressing GAN limitations: resolution, lack of novelty and control on generations
Camille Couprie
Facebook AI Research
Joint works with O. Sbai, M. Aubry, A, Bordes, M. Elhoseiny, M. Riviere, Y.
LeCun, M. Mathieu, P. Luc, N. Neverova, J. Verbeek.
2019
1
![Page 2: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/2.jpg)
Why do we care about generative models
2
• Scene Understanding can be assessed by checking the ability to
generate plausible new scenes.
• Generative models are interesting if they can be used to go beyond
training data: data of higher resolution, data augmentation to help
train better classifiers, use the learned representations in other
tasks, or make prediction about uncertain events...
![Page 3: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/3.jpg)
• 1/ Design inspiration from adversarial generative networks
• 2/ High resolution, decoupled generation
• 3/ Vector image generation by learning parametric layer
decomposition
• 4/ Future frame prediction
Outline
3
![Page 4: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/4.jpg)
4
Sbai, Elhoseiny, Bordes, LeCun, Couprie, ECCV workshop 17
Design inspiration from generative networks
N o v e l t y
H e d o n i c
Va l u e
1 2 30 4
![Page 5: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/5.jpg)
RANDOM NUMBERS
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORKGenerator
FakeGENERATED
IMAGE
Generative Adversarial Networks
Goodfellow et al, 2014
![Page 6: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/6.jpg)
RANDOM NUMBERS
Real INPUT
Generator
Real
AdVERSARIAL
NETWORK
GENERATED
IMAGE
0 . 3
0 . 7
0 . 1
0 . 8
Generative Adversarial Networks
Goodfellow et al, 2014
![Page 7: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/7.jpg)
Deep convolutional GANsRADFORD ET AL : ICLR 2015
![Page 8: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/8.jpg)
Training with pictures of about 2000 Clothing items
![Page 9: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/9.jpg)
Floral StripedTiled Uniform Dotted Animal Print Graphical
Texture and shape labels
DressSkirt JacketPullover T-Shirt Coat Top
![Page 10: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/10.jpg)
RANDOM NUMBERS
Real INPUT
GeneratorAdVERSARIAL
NETWORK
GENERATED
IMAGE
0 . 3
0 . 7
0 . 1
0 . 8
S h a p e C L A S S
0 / 1
( R E A L / F A K E )
T E X T U R E C L A S S
Class conditioned GAN
![Page 11: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/11.jpg)
• Generator’s
loss
• Discriminator’s
loss
• Auxiliary
classifier
discriminator:
• Additional loss
for the generator:
GAN Optimization objectives
![Page 12: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/12.jpg)
Without conditioning With class conditioning
![Page 13: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/13.jpg)
RANDOM
NUMBERSGenerator
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORK
Generated
Image
Dotted
Floral
graphical
uniform
tiled
striped
Animal print
1 0 0 %
Introduction of a Style Deviation criterion
![Page 14: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/14.jpg)
8 %
6 %
7 %
5 7 %
6 %
1 1 %
9 %
RANDOM
NUMBERSGenerator
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORK
Generated
Image
Dotted
Floral
graphical
uniform
tiled
striped
Animal print
Introduction of a Style Deviation criterion
![Page 15: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/15.jpg)
RANDOM
NUMBERSGenerator
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORKGenerated Image
D o t t e d
F l o r a l
g r a p h i c a l
u n i f o r m
t i l e d
s t r i p e d
A n i m a l p r i n t
With the Style
Deviation criterion
(CAN H)
![Page 16: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/16.jpg)
Multi-class cross entropy loss:
Binary cross entropy loss :
Tested deviation objectives
![Page 17: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/17.jpg)
O v e r a l L L i k a b i l i t y ( % )
6 5 7 0 7 56 08
0
CAN: GAN with Creativity loss, (H) stands for the use of a holistic loss.
CAN
texTUREGAN
Style
CAN(h)
texTURE
CAN(h)
texTUREr e a l i s t i c
A p p e a r a n c e
4 8
5 3 . 5
5 9
6 4 . 5
7
0
Human Evaluation Study
![Page 18: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/18.jpg)
Models with texture deviation are Most Popular
N o v e l t y0 1
L i k e a b i l i t y
6 2
6 6
7 0
7 4
7
8
judged by humans and measured as a distance to similar training images
Can
texture
Can (H)
Shape Can (H)
texture
style
can (h)
GAN
![Page 19: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/19.jpg)
![Page 20: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/20.jpg)
2
0
M. Riviere, C. Couprie, Y. LeCun
Decoupled adversarial image generation
Motivation:
- Take advantage of white background clothing datasets
- Potentially avoid defaults in generated shapes
- Better enforce shape conditioning of generations
![Page 21: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/21.jpg)
1024x1024 generations on the RTW dataset
2
1
Using Morgane’s pytorch “progressive growing of GANs” available online,
Karras et al., ICLR’18
![Page 22: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/22.jpg)
Decoupled architecture
2
2
Shape
Generator
Texture
Generator
Generation
Real image
Shape and
pose classes
Texture, color, shape and pose classes
Discriminator
Real / Fake
Real image
Discriminator
Real / Fake
Texture, color,
shape and
pose classes
×
DgDs
Gs
Gt
z
![Page 23: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/23.jpg)
Random generations
2
3
Progressive growingProgressive growing with decoupled
architecture
![Page 24: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/24.jpg)
Better class conditioning
2
4
Accuracy of classifiers trained on FashionGen
Clothing and FashionGen Shoes on our different
models results (GAN-test metric)
=
+12%
+14%
-12%
Overall average improvement: 4.7%
![Page 25: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/25.jpg)
2
5
Sbai, Couprie, Aubry, arxiv dec18
Vector Image Generation
by learning parametric layer decomposition
Current deep generative models
are great but…
… are limited in resolution, and
control in generations
![Page 26: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/26.jpg)
Kanan et al: Layered
GANs (LR-GANs), ICLR’17
GANIN et al. SPIRAL,
ICML’18
Related work
2
6
![Page 27: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/27.jpg)
Our approach
2
7
Spoiler alert: yes, we can
generate sharper images,
this is just an example.
![Page 28: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/28.jpg)
Iterative generation : !" = $(!, !"'()
Vectorized mask generation *"(+, ,) = $(+, ,, -")
Alpha-blending !"(+, ,) = !"'( +, , . (1-*"(+, ,))+ /"*"(+, ,)
Our iterative pipeline for image reconstruction
2
8
x x
×<latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit>
×<latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit>
+<latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit>
ct<latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit>
ctMt<latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit>
![Page 29: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/29.jpg)
2
9
Our iterative pipeline for image generation
![Page 30: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/30.jpg)
3
0
Training criteria
Adversarial net criterion: Wasserstein loss with Gradient Penalty (WGAN-GP),
Gulrajani et al. NIPS’17
Our generator loss in the GAN setting:
Our generator loss in the image reconstruction setting:
![Page 31: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/31.jpg)
Results using a l1 reconstruction loss
3
1
![Page 32: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/32.jpg)
Results using a l1 reconstruction loss
3
2
Target Our iterative reconstruction
![Page 33: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/33.jpg)
Editing the original image using
extracted masks, by performing
local modifications of
luminosity (top), or color
modification using a blending
of masks (bottom).
Applications
3
3
![Page 34: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/34.jpg)
Image editing using masks:
Using chosen extracted mask(s)
from image reconstruction, we
apply object removal (top)
and color modifications with
object removal (bottom).
Applications
3
4
![Page 35: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/35.jpg)
Image vectorization:
Reconstruction results on
MNIST images. Our model
learns a vectorized mask
representation of digits that
can be generated at any
resolution without
interpolation artifacts.
Applications
3
5
![Page 36: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/36.jpg)
RGB image
Baselines
3
6
RGB imagez
1/ MLP-baseline
MLPResnet
![Page 37: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/37.jpg)
RGB image
Baselines
3
8
RGB imagez
1/ MLP-baseline
2/ ResNet baseline
ResnetResnet
![Page 38: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/38.jpg)
RGB image
Baselines
3
9
RGB imagez
1/ MLP-baseline
2/ ResNet baseline
3/ MLP-xy baseline
MLP
(x,y)
Resnet
![Page 39: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/39.jpg)
Comparative results
4
0
- Using a l1 reconstruction loss
- Parameters for our approach:
20 masks, p of size 10, c of size 3:
260 parameters
- Baselines: size of the latent
code z = 20x13=260
![Page 40: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/40.jpg)
CelebA generations trained on 64x64
images, sampled at 256x256
GAN results
4
1
CIFAR10 generations trained on
32x32 images, sampled at 256x256
![Page 41: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/41.jpg)
GAN
Results on
ImageNet
4
2
trained on
64x64
images,
sampled at
1024x1024
![Page 42: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/42.jpg)
b
Results
4
4
Result
with
perceptual
loss
Perceptual
lossTarget
![Page 43: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/43.jpg)
Conclusion and Future work
4
5
Faster training
Use class conditioning
Texture image generation
![Page 44: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/44.jpg)
Predicting next frames in videos
Michael Mathieu, Camille Couprie, Yann LeCun, ICLR16
4 input images Our 2 predictions
![Page 45: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/45.jpg)
Deep multiscale video prediction beyond Mean square error
• Result with a simple convolutional network trained minimizing an l2 loss
• Our result using
• A multiscale architecture
• an image gradient different loss
• Use adversarial training
![Page 46: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/46.jpg)
4
8
P. Luc, N. Neverova, C. Couprie, J. Verbeek, Y. LeCun ECCV18
Predicting deeper into the future of semantic
segmentations
4 input images Our 2 predictions
• Predictions in the RGB space quickly become blurry despites previous attempts
• Idea: predict in the space of semantic segmentation
![Page 47: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/47.jpg)
Approach – predicting deeper into the future
Single time-step
Autoregressive model
Batch model
St−3 St−2 St−1 St St+1
Lt+1
St−3 St−2 St−1 St St+1 St+2 St+3
Lt+1 Lt+2 Lt+3
St−3 St−2 St−1 St St+1 St+2 St+3
Lt+1 Lt+2 Lt+3
Autoregressive model is either :
• used for inference without
additional training (w.r.t. to single
time step model) AR
• Fine-tuned using BPTT
AR fine-tune
Autoregressive mode is only possible
for X2X, S2S, XS2XS
Same color = shared weights
4
9
![Page 48: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/48.jpg)
Some results
50
Baselines :• Copy the last input frame to the
output
• Estimate flow between the two last inputs, and project the last input forward using the flow
Performance measure (mean IoU) of our approach and baselines on the Cityscapes dataset
Flow baseline
Our AutoRegressive fine-tune result
![Page 49: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/49.jpg)
Instance level segmentation: Mask RCNN
• Extends Faster RCNN [Ren et al.’15] by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition
K. He G. Gkioxari P. Dollar R. Girshick’17
1
0
![Page 50: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/50.jpg)
5
2
P. Luc, C. Couprie, Y. LeCun, J. Verbeek, ECCV18
Predicting future segmentation instances by
forecasting convolutional features
car 1.00
bicycle 0.92bicycle 0.97
person 0.99person 0.97person 0.98 person 0.93person 0.72
person 0.97
bicycle 1.00
person 0.76person 0.97
person 0.99person 0.98person 0.99person 0.94
person 0.91 person 1.00 person 0.56person 0.84
Luc, Neverova et al. ICCV17 F2F predictions
![Page 51: Addressing GAN limitations: resolution, lack of novelty ... › semester2019 › ... · resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f1a999bf5714e48fa43ffdc/html5/thumbnails/51.jpg)
Conclusions
5
3
Some open problems:
- Automatic metrics to evaluate generative models performances
- Non deterministic training losses for future prediction
Torch code online :
- For future video prediction:
- Vector image generation: available soon on Othman Sbai’s github
- of RGBs : on Michael Mathieu’s github
- of semantic segmentations : on Pauline’s Luc github
- of instance segmentation : on Pauline’s Luc github