MTA SzTAKI & Veszprém University (Hungary)Guests at INRIA, Sophia Antipolis, 2000 and 2001

Paintbrush Rendering of Images

Tamás SzirányiTamás Szirányi

Stochastic Paintbrush RenderingStochastic Paintbrush Rendering

Stochastic relaxation method to generate images (based on a

reference image) by simulating a simple painting method,

Sharp contours, well-defined segmentation areas, elaborated fine details,

A priori models or interactions are not needed.

Painted image with raw strokes

Visible contours of strokes

4

Stochastic painting method

Need for optimization

MCMC optimization

MRF and PB for segmentation

Objectives

Result of an impressionist-like process controlled by an edge mapResult of an impressionist-like process controlled by an edge map

(P. Litwinowicz, “Processing Images and Video for An Impressionist Effect”, (P. Litwinowicz, “Processing Images and Video for An Impressionist Effect”, Computer Computer GraphicsGraphics, Proc.SIGGRAPH’1997, , Proc.SIGGRAPH’1997, 1997.)1997.)

Paintings of A. Dürer and M. Munkácsy, representing careful life-style painting:Paintings of A. Dürer and M. Munkácsy, representing careful life-style painting: the painter tries to elaborate the painting without visible effects of the the painter tries to elaborate the painting without visible effects of the

brush-strokesbrush-strokes

Painting following a real-life picturePainting following a real-life picture

PaintingPainting,, following a real-life picture following a real-life picture

13

Painting following an artist`s painting Painting following an artist`s painting

14

Painting following an artist`s paintingPainting following an artist`s painting

15

Painting following an artist`s paintingPainting following an artist`s painting

16

Painting repairing JPEGPainting repairing JPEG

17

Painting following an artist`s paintingPainting following an artist`s painting

18

Original with edge-map

Segmentation

Painted (stroke is 37*9) and its edge

Painted (stroke is 10*3) and its edge

All PB strokes are accepted, then redundants

are removed

Only good matching is accepted

A need for optimal estimation of stroke placement

Markov Chain Monte Carlo (MCMC) methods

Ergodic Markov-chain (X1,X2,…. Xn)Stationary f(X)

Sample series (X1,X2,…. Xn) generated with target density f(X)Paintbrush strokes:

X is the position of a stroke, f(X) is the probability density of distortion error between the stroke and the reference image at position X

Metropolis-Hastings algorithm

IndependentIndependent MH MH

Random WalkRandom Walk MH MH

Probability densities of distortion error of the proposed ( Probability densities of distortion error of the proposed ( YYtt) and accepted ) and accepted

((X X t+1t+1=Y=Ytt) strokes versus the original image when generating the strokes ) strokes versus the original image when generating the strokes

for ‘Barbara’ image. First, coarse (20x5), finally, fine (7x2) strokes are for ‘Barbara’ image. First, coarse (20x5), finally, fine (7x2) strokes are generatedgenerated

Distortion Error of PB Strokes vs. Original Image

0

0,05

0,1

0,15

0,2

0,25

1 4 7 10

13

16

19

22

25

% of Distortion

Pro

b. D

en

sit

y

Distortion Error ofProposed Strokes (Coarse PB)

Distortion Error ofProposed Strokes (FinePB)

Distortion Error ofAccepted Strokes (Coarse PB)

Distortion Error ofAccepted Strokes (Fine PB)

tY

Probability densities of Probability densities of difference btw distortion errorsdifference btw distortion errors of the of the proposed/proposed/acceptedaccepted strokes and the strokes and the distortion on previous area of the strokedistortion on previous area of the stroke ( ( DiffDiff(Y(Ytt) ) ), when generating the strokes for ‘Barbara’ image. First, coarse ), when generating the strokes for ‘Barbara’ image. First, coarse

(20x5), finally, fine (7x2) strokes are generated.(20x5), finally, fine (7x2) strokes are generated.

Difference of Distortion Errors at the Proposed/Accepted Positions

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1 3 5 7 9 11

13

15

17

% of Distortion Difference

Pro

b. D

en

sit

y

Difference of Distortion Errors of AcceptedStrokes (Coarse PB)vs. previous values

Difference of Distortion Errors of AcceptedStrokes (Fine PB) vs.previous values

Difference of Distortion Errors of ProposedStrokes vs. previousvalues

The characteristic variable is the The characteristic variable is the Difference of distortion error instead of Difference of distortion error instead of

the errorthe error

)Diff(Yt )E(Y)E(x t)t(

)Diff(x t )( )E(x)E(x )t()t( 1

accept/reject ruleaccept/reject rule

ttt

tt

tt

Yxx

YxYX

,-1y probabilit with

,y probabilit with )()(

)()1(

1 ,)x|y(q

)y|x(q

)x(f

)y(fminy,x

else ,

0 if 0q

t

t

)Diff(Y

)Diff(Y,~yf

q)t( )Diff(x~xf

31

Narrowing effect of the distribution of the measured conditional probability

proposedtacceptedt )Diff(YP)Diff(YP /

32

else ,

if q

t

t

t

Var)Diff(Y

)Diff(Yyf

0,0~

q

t

t

Var)Diff(xxf

)(

~

ttt

tt

tt

Yxx

YxYX

,

,)()(

)()1(

-1

y probabilitwith

yprobabilitwith

1 ,)|(

)|(

)(

)(min,

xyq

yxq

xf

yfyx

Constraints for the flexible accept/reject rule

Random Stochastic search

35 61 2645

Metropolis Hastings process

26 31 1555

Method Number of non-

redundant PB strokes# thousand

Number of all drawn strokes

# 

Number of all proposed strokes

# acceptedt )(Y

proposedtY

34

Limiting the number of colorsLimiting the number of colors

Coupling the neighboring strokes

MRF-like segmentation

Reference area is greater than the stroke’s area

Halftone effects by strokes (modulation)

Connection btw MRF segmentation and PB

En1:

Difference inside the area of a

brush-stroke

between

the original

and the proposed

stroke

En2: Difference between the color of the stroke and the present neighboring pixels at the boundary of the stroke

En=En1+(1- )En2

0 < < 1.0

38

En 1

En 2

Noisy input Painting ( = 0.6, No.Co. = 30 )

Painting

En < 0 MMD

Segmentation for 4 colors (Trinois 128*128) Running time <-> Misclassification

0

1000

2000

3000

4000

5000

6000

7000

0 5 10 15 20 25 30 35 40

Running time (relative CPU)

# of

mis

clas

sifie

d pi

xels

7x3 PB, Delta E1 < 0, ~MMD

10x4 PB, Delta E1 < 0, ~MMD

13x5 PB, Delta E1 < 0, ~MMD

20x6 PB, Delta E1 < 0, ~MMD

1x1, Delta E1 < 0, ~MMD MRF,Beta=0.1

1x1, Delta E1 < 0, ~MMD MRF,Beta=0.75

1x1, ~MMD MRF, Beta=0.5

7*3 PB 13*5 PB 20*6 PB

E1 < 0, ~MMD

E1 < 0, ~MMD MRF ~MMD MRF

43

Motion-FieldMotion-Field

Gradient and block-matching motion fields

Gradient-based Block-matching

44

Examples 1Examples 1/2/2

transformed video Edge map of transformed video

1. 2.

3.

45

Example Example 2/2a2/2a

frame1: keyframe1 frame2: painted motion area 1 frame3: painted motion area 2

frame4: painted motion area 3 frame5: keyframe2

•1. and 5. are keyframes

46

Example Example 2/2b2/2b

frame1: keyframe1 frame2: painted motion area 1 frame3: painted motion area 2

frame4: painted motion area 3 frame5: keyframe2

47

ComparisonComparison

Original Cinepak

48

ComparisonComparison

With Cinepak-codec

Cinepak PB result

23

49

Thank you for the attention Thank you for the attention

50

AppendixAppendix

51

Segmentation for 2 colors Running time <-> Misclassification

0

20000

40000

60000

80000

100000

120000

0 20 40 60 80 100 120 140 160

Running time (relative CPU)

# o

f m

iscl

ass

ifie

d p

ixe

ls

1x1 PB, ~MMD MRF

5x5 PB, ~MMD

10x10 PB, ~MMD

12x12 PB, ~MMD

14x14 PB, ~MMD

15x15 PB, ~MMD

Segmentation by painting, when distortion error may increase

52

EnergEnergy calculus in optimization of y calculus in optimization of MRF MRF segmentationsegmentation

rsrss

ss V

,E

C,

2

s

2

2

2

Distortion between the original input color and the proposed random

value

rs

rsrsV

if ,

if ,,

Distortion among the neighboring pixels

IndependentIndependent MH MH

Since the probability of acceptance of Yt depends on X(t), the resulting sample is not independent and identically distributed (iid). X(t) is irreducible and aperiodic (thus ergodic) iff g is almost everywhere positive on the support of f. The above algorithm produces a uniformly ergodic chain if there exists a constant M such that

Random WalkRandom Walk MH MH

RW-MH does not enjoy uniform ergodicity properties, but there are necessary and sufficient conditions for geometric ergodicity, e.g. the log-concave case, when