“Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017...
Transcript of “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017...
![Page 1: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/1.jpg)
“Fast” Neural Style Transfer in CVPR 2017
Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA)
Yongcheng Jing
College of Computer Science and Technology
Zhejiang University
14/8/2017, Hangzhou
1
![Page 2: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/2.jpg)
2
Content
• Introduction of Neural Style Transfer
• “Slow” Neural Style Transfer and “Fast” Neural Style Transfer
• Introduction of three “Fast” Neural Style Transfer papers
![Page 3: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/3.jpg)
3
Introduction
• What is Image Style Transfer?
• Recombine the content of a given photograph and the style of a well-known artwork.
• e.g.
Photograph
Artwork
Stylized Result
Style Transfer
![Page 4: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/4.jpg)
4
Introduction
• Neural Style Transfer
• Use Convolutional Neural Network to finish the task of image style transfer.
• Applications
• Production Tools
• Entertainment
• Visualization & Presentation
• Social Communication
• e.g. “Prisma”, “Ostagram”, “In”
![Page 5: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/5.jpg)
5
Introduction
• Neural Style Transfer
• Use Convolutional Neural Network to finish the task of image style transfer.
• Applications
• Production Tools
• Entertainment
• Visualization & Presentation
• Social Communication
• e.g. “Prisma”, “Ostagram”, “In”
Nine papers published in CVPR 2017 which study Neural Style Transfer
Also lots of related papers published in ICLR 2017, SIGGRAPH 2017, ACM MM 2017
![Page 6: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/6.jpg)
6
Review of Neural Style Transfer
• Taxonomy of Neural Style Transfer Algorithms
• “Slow” Neural Methods Based On Image Optimization (CVPR 2016 主战场)
• “Fast” Neural Methods Based On Model Optimization
• Per-Style-Per-Model “Fast” Neural Methods (CVPR 2016 主战场, CVPR 2017 主战场)
• Multiple-Style-Per-Model “Fast” Neural Methods (CVPR 2017 主战场)
• Arbitrary-Style-Per-Model “Fast” Neural Methods (ICCV 2017 主战场以及预计CVPR 2018 主战场)
![Page 7: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/7.jpg)
7
Review of Neural Style Transfer
• Taxonomy of Neural Style Transfer Algorithms
• “Slow” Neural Methods Based On Image Optimization (CVPR 2016 主战场)
• “Fast” Neural Methods Based On Model Optimization
• Per-Style-Per-Model “Fast” Neural Methods (CVPR 2016 主战场, CVPR 2017主战场)
• Multiple-Style-Per-Model “Fast” Neural Methods (CVPR 2017 主战场)
• Arbitrary-Style-Per-Model “Fast” Neural Methods (ICCV 2017 主战场以及预计CVPR 2018主战场)
• Extensions
• Color style transfer (https://github.com/LouieYang/deep-photo-styletransfer-tf, 150 stars in 2 days)
• Typography style transfer
• Visual attribute transfer
Paper collected at: https://github.com/ycjing/Neural-Style-Transfer-Papers
![Page 8: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/8.jpg)
8
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Pre-trained VGG-19, fully-connected
layers removed
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 9: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/9.jpg)
9
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Content
features
512 x H x W
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 10: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/10.jpg)
10
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Content
features
512 x H x W
Style features
256 x H x WGram
matrix
256 x 256
Gram matrix计算方式为:大小是 [256, H x W] 的feature map矩阵与其转置进行矩阵乘法
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 11: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/11.jpg)
11
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Target gram
matrix
256 x 256
Target
features
512 x H x
W
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 12: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/12.jpg)
12
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Target gram
matrix
256 x 256
Target
features
512 x H x
W
Generated
image
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 13: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/13.jpg)
13
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Target
features
512 x H x
W
Generated
image
Style features
256 x H x W
Gram
matrix
256 x 256
Content
features
512 x H x W
Target gram
matrix
256 x 256
1: Forward pass
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 14: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/14.jpg)
14
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Target gram
matrix
256 x 256
Target
features
512 x H x
W
Generated
image
Style features
256 x H x W
Content
features
512 x H x W
Style
loss
(L2)
Content loss
(L2)
2: Compute loss
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
Gram
matrix
256 x 256
![Page 15: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/15.jpg)
15
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Target gram
matrix
256 x 256
Target
features
512 x H x
W
Generated
image
Style features
256 x H x W
Content
features
512 x H x W
Content loss
(L2)
3: Backward pass
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
Style
loss
(L2)
Gram
matrix
256 x 256
![Page 16: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/16.jpg)
16
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Generated
image
4: Update image
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 17: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/17.jpg)
17
Review of Neural Style Transfer
• Overview of “Slow” algorithm:
Style
Image
Content
Image
Generated
image
5: Repeat many times
VGG figure credit: Kaiming He Other figures credit: Justin Johnson
![Page 18: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/18.jpg)
18
Review of Neural Style Transfer
• Loss function
![Page 19: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/19.jpg)
19
Review of Neural Style Transfer
Input image
“Starry Night”
networkGenerated
imageVGG-19
Style loss + content
loss
• Overview of per-style-per-model “Fast” algorithm:
Train a layer-specific style transfer model
Figures credit: Justin Johnson
![Page 20: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/20.jpg)
20
Three Papers
• ① [Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast
Artistic Style Transfer]
• Per-Style-Per-Model “Fast” algorithm
• Solve the texture scale (or brush size problem) in previous “Fast” algorithm
Slow algorithm resultHigh-resolution content Style “Fast” algorithm result
![Page 21: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/21.jpg)
21
Three Papers
• ① [Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast
Artistic Style Transfer]
• Per-Style-Per-Model “Fast” algorithm
• Solve the texture scale (or brush stroke size problem) in previous “Fast” algorithm
Slow algorithm resultHigh-resolution content Style “Fast” algorithm result
The reason is that 80000 training images are all resized to 256px to speed up the training process. If the test content is also 256px, the texture scale is good.
![Page 22: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/22.jpg)
22
Three Papers
• ② [StyleBank: An Explicit Representation for Neural Image Style Transfer]
• Multiple-Style-Per-Model
• Support incremental learning
• ③ [Diversified Texture Synthesis with Feed-forward Networks]
• Multiple-Style-Per-Model
• Support incremental learning
• Diversity loss
• The authors provide1000-style model for research use.
![Page 23: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/23.jpg)
Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
23
University of California, Santa Barbara and Adobe Research
![Page 24: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/24.jpg)
24
Appeal
• Solve the small texture scale problem (or brush stroke size) in previous Per-Style-Per-
Model algorithm (PSPM).
“Slow” algorithm PSPM algorithm #1 PSPM algorithm #2 This paper 1024px, high-resolution
![Page 25: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/25.jpg)
25
Method
VGG
256
256DS US US
512 1024
1024
512
• Hierarchical Stylization (coarse-to-fine)
• VGG Loss function is the same as before, i.e. content loss and gram-based style loss.
![Page 26: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/26.jpg)
26
Method
VGG
256
256DS US US
512 1024
1024
512
Luminance
channel
RGB
channel
Loss_1
![Page 27: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/27.jpg)
27
Method
VGG
256
256DS US US
512 1024
1024
512
Loss_1
Loss_2
![Page 28: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/28.jpg)
28
Method
VGG
256
256DS US US
512 1024
1024
512
Loss_1
Loss_2
Identity connection to force
it to learn differences
Loss_3
![Page 29: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/29.jpg)
29
Method
VGG
256
256DS US US
512 1024
1024
512
Loss_1
Loss_2
Loss_3
Identity connection to force
it to learn differences
![Page 30: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/30.jpg)
30
Method
• How to train?
• The parameters of former subnets are updated to incorporate the current and latter
stylization losses. → (coarse-to-fine)
![Page 31: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/31.jpg)
31
Method
• How to train?
• The parameters of former subnets are updated to incorporate the current and latter
stylization losses. → (coarse-to-fine)
• i.e., in one iteration, the losses that each subnet optimizes is:
• 1. [style subnet]: 𝜆1𝐿𝑜𝑠𝑠_1 + 𝜆2𝐿𝑜𝑠𝑠_2 + 𝜆3𝐿𝑜𝑠𝑠_3
• 2. [enhance subnet]: 𝜆2𝐿𝑜𝑠𝑠_2 + 𝜆3𝐿𝑜𝑠𝑠_3
• 3. [refine subnet]: 𝜆3𝐿𝑜𝑠𝑠_3
• Latter subnet losses have smaller weights (𝝀𝟏: 𝝀𝟐: 𝝀𝟑=1 : 0.5 : 0.25 in the paper)
![Page 32: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/32.jpg)
32
Experimental Results
“Slow” algorithm PSPM algorithm #1 PSPM algorithm #2 This paper 1024px, high-resolution
![Page 33: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/33.jpg)
33
Conclusion
• Coarse-to-fine network design and training strategy
• Subnetwork is trained to minimize the losses that are computed from the latter
subnetwork outputs.
![Page 34: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/34.jpg)
StyleBank: An Explicit Representation for Neural Image Style Transfer
34
University of Science and Technology of China and Microsoft Research
![Page 35: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/35.jpg)
35
Appeal
• Multiple-Style-Per-Model
• Support incremental learning
• Only need 8 minutes to add a new style into the model.
![Page 36: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/36.jpg)
36
Analysis
• Key points to be considered for Multiple-Style-Per-Model:
• 1. Choice of signal for each style: different style-specific filter bank
• 2. Scability: One network to learn thousands of styles may not be a feasible solution.
• 3. Incremental (Online) learning for new styles: train new filter bank
![Page 37: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/37.jpg)
37
Analysis
• Inspiration for this paper:
• For each content-style pair, actually the target content is always fixed for each style.
• Therefore, in previous Per-Style-Per-Model method, it is redundant to train a
network both for content and style. There may be something shared between
different models.
• Can we use separate networks to extract content representation and style
representation which are independent of each other?
![Page 38: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/38.jpg)
38
Method
• Network architecture
![Page 39: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/39.jpg)
39
Method
• Network architecture
Firstly, train Auto-encoder to learn content representation.
The objective is 𝑶 == 𝑰Content Loss is the same as previous Neural Style algorithm.
![Page 40: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/40.jpg)
40
Method
• Network architecturen is # of styles
Use StyleBank layer to add style elements into the content
![Page 41: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/41.jpg)
41
Method
• Network architecturen is # of styles
Use StyleBank layer to add style elements into the content
• Style loss is also the same as before, gram-based loss
• The training procedure for two branches is inspired by GAN:
• For T+1 iterations,
• train T iterations on branch 𝑳𝑲 (虚线)
• train 1 iteration for branch 𝑳𝑲 (实线)
![Page 42: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/42.jpg)
42
Method
• Network architecturen is # of styles
Use StyleBank layer to add style elements into the content
• Incremental learning:
• Fix auto-encoder and only train a new filter bank in StyleBank Layer
• 8 minutes to train for a new style
![Page 43: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/43.jpg)
43
Method
• Network architecturen is # of styles
Use StyleBank layer to add style elements into the content
![Page 44: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/44.jpg)
44
Experimental Results
![Page 45: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/45.jpg)
45
Conclusion
• GAN-like training strategy
• Fixed one branch and train the other
![Page 46: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/46.jpg)
Diversified Texture Synthesis with Feed-forward Networks
46
University of California, Merced and Adobe Research
![Page 47: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/47.jpg)
47
Appeal
• Multiple-Style-Per-Model
• Support incremental learning
• Diversity loss
![Page 48: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/48.jpg)
48
Appeal
• Multiple-Style-Per-Model
• Support incremental learning
• Diversity loss
No diversity loss:
Overfitting to a particular instance, repetitive patterns
![Page 49: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/49.jpg)
49
Analysis
• Key points to be considered for Multiple-Style-Per-Model:
• 1. Choice of signal for each style: style-specific noise
• 2. Scability: One network to learn thousands of styles may not be a feasible solution.
(it actually works in this paper, the author provides 1000-style model; if larger, I doubt it)
• 3. Incremental (Online) learning for new styles
![Page 50: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/50.jpg)
50
Method
• Network architecture
one-hot
vector
Style-specific noise map
(from uniform distribution)
00
00
![Page 51: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/51.jpg)
51
Method
• Network architecture
one-hot
vector
Style-specific noise map
(from uniform distribution)
00
00
• Content loss is the same. Style loss has
a little modifications.
![Page 52: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/52.jpg)
52
Method
• Diversity loss
• Penalize the difference of different outputs of
the same style in the feature space
• Assume the output stylized results in a batch of images are:
• 𝑃1, 𝑃2, … , 𝑃𝑁 (they are stylized images)
• Let {𝑄1, 𝑄2, … , 𝑄𝑁} be a random reordering of 𝑃1, 𝑃2, … , 𝑃𝑁 and 𝑃𝑖 ≠ 𝑄𝑖
• 𝐿𝑑𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦 = −1
𝑁 𝑖=1𝑁 Φ 𝑃𝑖 −Φ 𝑄𝑖 1, Φ 𝑖𝑠 𝑡ℎ𝑒 𝑓𝑒𝑎𝑡𝑢𝑟𝑒 𝑚𝑎𝑝 𝑜𝑓 𝑐𝑜𝑛4 _ 2 𝑙𝑎𝑦𝑒𝑟 𝑖𝑛 𝑉𝐺𝐺
![Page 53: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/53.jpg)
53
Method
• Diversity loss
• Penalize the difference of different outputs of
the same style in the feature space
• Assume the output stylized results in a batch of images are:
• 𝑃1, 𝑃2, … , 𝑃𝑁 (they are stylized images)
• Let {𝑄1, 𝑄2, … , 𝑄𝑁} be a random reordering of 𝑃1, 𝑃2, … , 𝑃𝑁 and 𝑃𝑖 ≠ 𝑄𝑖
• 𝐿𝑑𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦 = −1
𝑁 𝑖=1𝑁 Φ 𝑃𝑖 −Φ 𝑄𝑖 1, Φ 𝑖𝑠 𝑡ℎ𝑒 𝑓𝑒𝑎𝑡𝑢𝑟𝑒 𝑚𝑎𝑝 𝑜𝑓 𝑐𝑜𝑛4 _ 2 𝑙𝑎𝑦𝑒𝑟 𝑖𝑛 𝑉𝐺𝐺
Style Without diversity loss With diversity loss
![Page 54: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/54.jpg)
54
Method
• Incremental learning: • Similar to curriculum learning.
• Do not forget what is learned and learn
new thing.
Style
I doubt its training time. It is not that good as StyleBank.
![Page 55: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/55.jpg)
55
Experimental Results
![Page 56: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/56.jpg)
56
Conclusion
• Training strategy inspired by curriculum learning
• Diversity loss
![Page 57: “Fast” Neural Style Transfer in CVPR 2017 · “Fast” Neural Style Transfer in CVPR 2017 Seminar in Laboratory of Visual Intelligence and Pattern Analysis (VIPA) Yongcheng Jing](https://reader034.fdocuments.us/reader034/viewer/2022050103/5f41fabd6b87eb19d80695e5/html5/thumbnails/57.jpg)
57
Question?
Q & A