Parag K. Mital - corpus-based visual synthesis an approach to artistic...

A C M S y m p o s i u m O n A p p l i e d P e r c e p t i o n | 2 3 A u g u s t 2 0 1 3

Corpus-based Visual Synthesis: An Approach to Artistic StylizationP a r a g K . M i t a l 1 M i c k G r i e r s o n 1 T i m S m i t h 2

Source Image Corpus Images Stylization

1D e p a r t m e n t o f C o m p u t i n g , G o l d s m i t h s , U n i v e r s i t y o f L o n d o n2D e p a r t m e n t o f P s y c h o l o g i c a l S c i e n c e s , B i r k b e c k , U n i v e r s i t y o f L o n d o n

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 712

Representation✤ Perception must be supported by pre-

attentive representations

✤ Abstract description of objects/scenes

✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)

Representation✤ Perception must be supported by pre-

attentive representations

Representation

✤ Perception must be supported by pre-attentive representations

Representation

✤ Numerous theories: ✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)

Representation

✤ Artists are well aware of the role of abstract representations in perception

✤ Influence how we look at and where we look in a scene

✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism

Representation

Related Work

✤ Filtering/Clustering

✤ Example-based Images

✤ Texture-transfer/Patch-based

✤ Dictionary methods/Collage approaches

Kyprianidis, J. et al., 2012. State of the “Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE transactions on Visualization and Computer Graphics. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6243138

Problem Statement

✤ Our approach✤ Build corpus of abstract representations

from user chosen images

✤ Match target image’s abstract representations to nearest ones in corpus

✤ Synthesize target image using closest matches

✤ Interact with a simple set of parameters effecting representation detection and synthesis

✤ Create automated artistic stylizations of images/videos using an understanding of the role of abstract representations in art and perception

✤ Allow for a range of styles through a simple set of parameters

✤ Needs to be fast in order to explore different styles quickly / run in real-time

Outline✤ CBVS Framework

✤ Results for Images/Videos

✤ Extensions

✤ Memory Mosaicing

✤ Augmented Reality Hallucinations

✤ PhotoSynthesizer (iOS app)

✤ Conclusion

✤ Extensions

✤ Conclusion

CBVS Framework

✤ Build corpus of abstract representations from user chosen images

RC = {R1, R2, ..., RNC}

Building the Corpus

✤ Need to represent:✤ Sparse/Dense strokes✤ Small/Large strokes

✤ Watershed? ✤ Posterization? ✤ Mean-Shift?

Building the Corpus

✤ Maximally Stable Color Regions✤ No need for multiple scale detections✤ Implicit ordering of regions✤ Simple set of parameters for

discovering sparse/dense small/large strokes

✤ Fast/Robust across multiple views (used in video tracking)

✤ Similar process to the unconscious representations as theorized before

Increasing timesteps =

Denser detection More expressive corpus

CBVS Framework

RC = {R1, R2, ..., RNC}

RT = {R1, R2, ..., RNT }

Matching

✤ Need to match regions in target to similar ones in corpus

✤ Describe Shape and Color

✤ Use Euclidean distance for shape values

✤ Use Perceptual distance for color values (CIEDE2000 formula)

✤ Nearest neighbor matching

dRi =⇣µ00, ⌘11, ⌘20, ⌘02, L, a

⇤, b⇤⌘

⌘ij =µij

µ(1+ i+j

Matching

✤ Nearest neighbor matchingd(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)

Matching

CBVS Framework

Synthesis

✤ From largest to smallest target region

✤ Find nearest neighbor

✤ Translate

✤ Rotate

✤ Scale

✤ Blend

d(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)

Synthesis

T = centroidRt � centroidRc

✤ Translate

✤ Rotate

✤ Scale

✤ Blend

Synthesis

⇥ =1

2⇤ arctan

2 ⇤ µ11

µ00� µ02

✤ Translate

✤ Rotate

✤ Scale

✤ Blend

Synthesis

=width

widthRC

Sy =heightRT

heightRC

✤ Translate

✤ Rotate

✤ Scale

✤ Blend

Synthesis

✤ Translate

✤ Rotate

✤ Scale

✤ Blend

Interaction✤ Spatial blending

✤ Temporal blending

✤ Motion tracking

✤ Timesteps

✤ Minimum region size

✤ Maximum region size

✤ Blending radius

Discrete Parameters

Continuous Parameters

Spatial Blending

Temporal Blending

Increasing timesteps =

Denser layers, More expressive

Timesteps

Decreasing minimum size =

Finer brush strokes

Minimum Size

Increasing radius =

More of source texture

Blending Radius

✤ Extensions

✤ Conclusion

Target Image Corpus Synthesis

Target Image Corpus

✤ Extensions

✤ Conclusion

Memory Mosaicing✤ Dynamic target (movie or webcam)

✤ Aggregate corpus over time using target, retaining only most recent N objects

✤ Only allow learning of objects with distance greater than threshold

Live demo

✤ Extensions

✤ Conclusion

Augmented Reality Hallucinations

✤ Uses Augmented Reality headset

✤ Exhibited during the Victoria & Albert Museum’s Digital Design Weekend co-located during the London Design Festival, 15,000 participants

✤ Short questionnaire for participants rating their experience (21 participants only)

✤ Extensions

✤ Conclusion

PhotoSynthesizer ✤ Free iOS app allows user to synthesize

target image

✤ No interaction besides selecting target and corpus

✤ Reveals synthesis process as painting regions over time

✤ Reached Top 50 app in Photo & Video in many countries

PhotoSynthesizer

Live demo

✤ Extensions

✤ Conclusion

Conclusion✤ Simple shape representation affords

range of stylizations and a range of non/real-time applications

✤ Expressive control in a few parameters

✤ Future?✤ Better method of evaluation✤ Better metrics for shape description✤ Better temporal coherence

Parag K. Mital - corpus-based visual synthesis an approach to artistic...

Documents

Transcript of Parag K. Mital - corpus-based visual synthesis an approach to artistic...

Parag Vs Kotkin: Fight!

INSTITUTIONAL EQUITY RESEARCH Parag Milk Foods (PARAG …backoffice.phillipcapital.in/Backoffice/Researchfiles/PC... · 2016-09-27 · Valuations in dairy industry: The stock of Parag

Parag Singhal* & Priyam Jain** - crsgpp.nujs.educrsgpp.nujs.edu/.../02/Article-11-Parag-Priyam.pdf · Parag Singhal* & Priyam Jain** ABSTRACT In recent years, several Asian countries

Mital Polyplast Pvt Ltd, Mumbai, Colour Copier Machines

...ACM 05 ACM 06 ACM 07 ACM 08 New Document Number ACM 003 ACM 004 ACM 005 ACM 006 ACM 007 ACM 008 3.5. Updated List of Cosmetic Accredited Laboratories The list of accredited testing

Presentation by parag saha

Parag Parikh Tax Saver Fund - Nivesh Sansarniveshsansar.com/.../06/Application-Form-Parag-Parikh-Tax-Saver-Fu… · Parag Parikh Tax Saver Fund (An open-ended equity linked saving

Parag Project on Parag Milk

ITS Presentation by Mr. Mital Damani

101356178 Parag Dairy Word File

Parag Main

Parag Dairy (Word File)

Ning Post 4: Parag Khanna

Parag Grapt-log G

Project on PARAG MILK

Initiating coverage | Dairy products Parag Milk Foods Limited …web.angelbackoffice.com/research/archives/fundamental/company... · Parag Milk Foods (PARAG) is one of the leading

Connectography: Parag Khanna

Parag Report

Parag khanna part 2

PARAG MILK FOODS LIMITED