Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate...
Transcript of Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate...
![Page 1: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/1.jpg)
Universal Style Transfer via Feature TransformsAuthors: Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang
Presented by: Ibrahim Ahmed and Trevor Chan
![Page 2: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/2.jpg)
Problem
● Transfer arbitrary visual styles to content images
Content Image Style Image Stylization Result
Image source: https://research.googleblog.com/2016/02/exploring-intersection-of-art-and.html
![Page 3: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/3.jpg)
Related Work
1. Not efficient during inference: A Neural Algorithm of Artistic Style [1].
2. Style Specific Networks: Perceptual Losses for Real -Time Style
Transfer and Super-Resolution [2].
3. Poor generalizing abilities in terms of output quality: Arbitrary
Style Transfer in Real-Time with Instance Normalization[3]
![Page 4: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/4.jpg)
Proposed Method
● Image reconstruction + Feature Transforms
Train autoencoder for image reconstruction, then fix it
![Page 5: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/5.jpg)
Proposed Method
Image source: http://nghiaho.com/?p=1765
![Page 6: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/6.jpg)
Proposed Method
● Encoder - Train VGG-19 on Imagenet Classification task● Decoder - Trained to reconstruct the image● More than one decoder trained for reconstruction
○ 5 trained decoders
Image source: Li et. al
![Page 7: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/7.jpg)
Loss function for Reconstruction Decoder
● Note: no style image is used in process of training
![Page 8: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/8.jpg)
Feature Transforms
![Page 9: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/9.jpg)
Feature Transforms by Whitening/Coloring
● Content features are transformed at intermediate levels by statistics of the style features
● In each layer, need content features to exhibit same characteristics of the style features of the same layer
● WCT achieves this
![Page 10: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/10.jpg)
Data Whitening
● Transform a random vector to be uncorrelated and have unit variance
1. decorrelate the components of original vector
2. scale the different components so they have unit variance
![Page 11: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/11.jpg)
Data Whitening (Step One)
● Eigendecomposition of covariance matrix by Σ = ΦΛΦ-1
● Let Y = ΦTX where Y has uncorrelated components
Σ Φ Λ Φ-1
![Page 12: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/12.jpg)
● Scale the uncorrelated components of Y by W = Λ-1/2 *Y
● W is now the white noise vector
● Components are i.i.d with unit variance
● W’s covariance matrix is identity matrix
Data Whitening (Step Two)
![Page 13: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/13.jpg)
Data Coloring
● Coloring is the inverse of the whitening transform● Transform white noise into random vector with desired covariance
matrix
![Page 14: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/14.jpg)
Applying WCT to Style Transfer
● Disassociate input image style and associate with the input image the style of the style image
● From content image Ic and style image Is, extract their vectorized feature maps fc and fs
● WCT will directly transform the fc to match the covariance matrix of fs
Image source: Li et. al
![Page 15: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/15.jpg)
Whitening and Coloring Transform
Whitening:
Coloring:
Image source: Li et. al
![Page 16: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/16.jpg)
Multi-level Stylization
Single-level stylization Multi-level StylizationImage source: Li et. al
![Page 17: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/17.jpg)
Multi-level Stylization
Image source: Li et. al
![Page 18: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/18.jpg)
Results
● Compared with other style transfer methods
● Other methods were inferior in terms of ○ Handling arbitrary styles○ Efficiency○ Learning-free
Image source: Li et. al
![Page 19: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/19.jpg)
Results
![Page 20: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/20.jpg)
Parameters
● Style weight control
● Image size
Image source: Li et. al
![Page 21: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/21.jpg)
Image Editing
● Edit different areas of a content image with different style images
Image source: Li et. al
![Page 22: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/22.jpg)
Texture Synthesis
● Input noise image as content image, and use texture example as the style image
Image source: Li et. al
![Page 23: Feature Transforms Universal Style Transfer via · Applying WCT to Style Transfer Disassociate input image style and associate with the input image the style of the style image From](https://reader035.fdocuments.us/reader035/viewer/2022071218/604e9d45a75c7f4efa6c62e5/html5/thumbnails/23.jpg)
Takeaways
● Works with any chosen style
● Don’t have to train on style images
● Scale and weight of style transfer can be changed on the fly
● Performs more consistently across a set of widely varying input styles