Parag K. Mital - corpus-based visual synthesis an approach to artistic...
Transcript of Parag K. Mital - corpus-based visual synthesis an approach to artistic...
![Page 1: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/1.jpg)
A C M S y m p o s i u m O n A p p l i e d P e r c e p t i o n | 2 3 A u g u s t 2 0 1 3
Corpus-based Visual Synthesis: An Approach to Artistic StylizationP a r a g K . M i t a l 1 M i c k G r i e r s o n 1 T i m S m i t h 2
Source Image Corpus Images Stylization
1D e p a r t m e n t o f C o m p u t i n g , G o l d s m i t h s , U n i v e r s i t y o f L o n d o n2D e p a r t m e n t o f P s y c h o l o g i c a l S c i e n c e s , B i r k b e c k , U n i v e r s i t y o f L o n d o n
![Page 2: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/2.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 712
Representation✤ Perception must be supported by pre-
attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
![Page 3: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/3.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 713
Representation✤ Perception must be supported by pre-
attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
![Page 4: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/4.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
4
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
![Page 5: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/5.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
5
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories: ✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
![Page 6: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/6.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
6
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
![Page 7: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/7.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
7
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
![Page 8: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/8.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
8
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
![Page 9: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/9.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
9
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
![Page 10: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/10.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
10
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
![Page 11: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/11.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
11
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
![Page 12: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/12.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
12
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
![Page 13: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/13.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
13
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
![Page 14: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/14.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
14
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
![Page 15: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/15.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Related Work
15
✤ Filtering/Clustering
✤ Example-based Images
✤ Texture-transfer/Patch-based
✤ Dictionary methods/Collage approaches
Kyprianidis, J. et al., 2012. State of the “Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE transactions on Visualization and Computer Graphics. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6243138
![Page 16: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/16.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Problem Statement
16
✤ Our approach✤ Build corpus of abstract representations
from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
✤ Create automated artistic stylizations of images/videos using an understanding of the role of abstract representations in art and perception
✤ Allow for a range of styles through a simple set of parameters
✤ Needs to be fast in order to explore different styles quickly / run in real-time
![Page 17: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/17.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
17
![Page 18: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/18.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
18
![Page 19: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/19.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
CBVS Framework
19
✤ Build corpus of abstract representations from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
RC = {R1, R2, ..., RNC}
![Page 20: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/20.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Building the Corpus
20
✤ Need to represent:✤ Sparse/Dense strokes✤ Small/Large strokes
✤ Watershed? ✤ Posterization? ✤ Mean-Shift?
![Page 21: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/21.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Building the Corpus
21
✤ Maximally Stable Color Regions✤ No need for multiple scale detections✤ Implicit ordering of regions✤ Simple set of parameters for
discovering sparse/dense small/large strokes
✤ Fast/Robust across multiple views (used in video tracking)
✤ Similar process to the unconscious representations as theorized before
![Page 22: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/22.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7122
![Page 23: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/23.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7123
![Page 24: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/24.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7124
![Page 25: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/25.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7125
![Page 26: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/26.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7126
![Page 27: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/27.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7127
![Page 28: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/28.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7128
![Page 29: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/29.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7129
Increasing timesteps =
Denser detection More expressive corpus
![Page 30: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/30.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
CBVS Framework
30
✤ Build corpus of abstract representations from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
RC = {R1, R2, ..., RNC}
RT = {R1, R2, ..., RNT }
?
![Page 31: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/31.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Matching
31
✤ Need to match regions in target to similar ones in corpus
✤ Describe Shape and Color
✤ Use Euclidean distance for shape values
✤ Use Perceptual distance for color values (CIEDE2000 formula)
✤ Nearest neighbor matching
![Page 32: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/32.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7132
✤ Need to match regions in target to similar ones in corpus
✤ Describe Shape and Color
✤ Use Euclidean distance for shape values
✤ Use Perceptual distance for color values (CIEDE2000 formula)
✤ Nearest neighbor matching
dRi =⇣µ00, ⌘11, ⌘20, ⌘02, L, a
⇤, b⇤⌘
⌘ij =µij
µ(1+ i+j
2 )00
Matching
![Page 33: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/33.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7133
✤ Need to match regions in target to similar ones in corpus
✤ Describe Shape and Color
✤ Use Euclidean distance for shape values
✤ Use Perceptual distance for color values (CIEDE2000 formula)
✤ Nearest neighbor matchingd(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)
Matching
![Page 34: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/34.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
CBVS Framework
34
✤ Build corpus of abstract representations from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
![Page 35: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/35.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
35
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
d(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)
![Page 36: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/36.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
36
T = centroidRt � centroidRc
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
![Page 37: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/37.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
37
⇥ =1
2⇤ arctan
2 ⇤ µ11
µ00
µ20
µ00� µ02
µ00
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
![Page 38: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/38.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
38
Sx
=width
RT
widthRC
Sy =heightRT
heightRC
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
![Page 39: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/39.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
39
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
![Page 40: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/40.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Interaction✤ Spatial blending
✤ Temporal blending
✤ Motion tracking
✤ Timesteps
✤ Minimum region size
✤ Maximum region size
✤ Blending radius
40
Discrete Parameters
Continuous Parameters
![Page 41: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/41.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Spatial Blending
41
![Page 42: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/42.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Temporal Blending
42
![Page 43: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/43.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Temporal Blending
43
![Page 44: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/44.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7144
Increasing timesteps =
Denser layers, More expressive
Timesteps
![Page 45: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/45.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7145
Decreasing minimum size =
Finer brush strokes
Minimum Size
![Page 46: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/46.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7146
Increasing radius =
More of source texture
Blending Radius
![Page 47: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/47.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ Conclusion
47
![Page 48: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/48.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7148
Target Image Corpus Synthesis
![Page 49: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/49.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7149
![Page 50: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/50.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7150
Target Image Corpus Synthesis
![Page 51: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/51.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7151
![Page 52: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/52.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7152
Target Image Corpus Synthesis
![Page 53: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/53.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7153
![Page 54: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/54.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7154
Target Image Corpus
![Page 55: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/55.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7155
![Page 56: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/56.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7156
![Page 57: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/57.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7157
![Page 58: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/58.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7158
![Page 59: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/59.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7159
![Page 60: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/60.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
60
![Page 61: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/61.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Memory Mosaicing✤ Dynamic target (movie or webcam)
✤ Aggregate corpus over time using target, retaining only most recent N objects
✤ Only allow learning of objects with distance greater than threshold
61
![Page 62: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/62.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7162
Live demo
![Page 63: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/63.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
63
![Page 64: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/64.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Augmented Reality Hallucinations
✤ Memory Mosaicing
✤ Uses Augmented Reality headset
✤ Exhibited during the Victoria & Albert Museum’s Digital Design Weekend co-located during the London Design Festival, 15,000 participants
✤ Short questionnaire for participants rating their experience (21 participants only)
64
![Page 65: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/65.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7165
![Page 66: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/66.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
66
![Page 67: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/67.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
PhotoSynthesizer ✤ Free iOS app allows user to synthesize
target image
✤ No interaction besides selecting target and corpus
✤ Reveals synthesis process as painting regions over time
✤ Reached Top 50 app in Photo & Video in many countries
67
![Page 68: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/68.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
PhotoSynthesizer
68
![Page 69: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/69.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7169
Live demo
![Page 70: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/70.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
70
![Page 71: Parag K. Mital - corpus-based visual synthesis an approach to artistic stylizationpkmital.com/home/wp-content/uploads/2013/08/acm-sap... · 2013. 8. 23. · ACM Symposium On Applied](https://reader033.fdocuments.us/reader033/viewer/2022053118/609d2bf54dd87504107fcf25/html5/thumbnails/71.jpg)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Conclusion✤ Simple shape representation affords
range of stylizations and a range of non/real-time applications
✤ Expressive control in a few parameters
✤ Future?✤ Better method of evaluation✤ Better metrics for shape description✤ Better temporal coherence
71