INTELLIGENT STORYBOARD FOR PROTOTYPING ANIMATION … · INTELLIGENT STORYBOARD FOR PROTOTYPING...
Transcript of INTELLIGENT STORYBOARD FOR PROTOTYPING ANIMATION … · INTELLIGENT STORYBOARD FOR PROTOTYPING...
ABSTRACT
Storyboard is a classical animation tool to help the cre-
ators to organize scenes. In this paper, we propose a newmethod for producing animation from storyboard. The 3Dposition and behavior of the characters are estimated from
2D views using the constraints optimization and example-based interpolation method. We also show animation ex-amples produced from the storyboard.
1. INTRODUCTION
Rapid making of CG animation is useful for various fields
such as entertainment and educations. Recent developmentof low-cost PC enables users to make animations for theirown uses. However, making animation is still a difficult taskespecially for novice users. The users have to learn many
technical terms and skills such as bones, skeleton, inversekinematics, and so on.
More intuitive method for describing the scenes is mak-
ing a storyboard. Storyboard is a classical animation tool tohelp the creators to organize scenes [1,2,3,4]. In the worldof animation, Walt Disney and his staff developed a
storyboard system in 1928. Managing the thousands of draw-ings and the progress of a project was nearly impossible, soDisney had his artists pin up their drawings on the studio
walls. This way, progress could be checked, and scenes addedand discarded with ease.
Starboard may be a good user interface for novice users
to organize idea. However, the users still need many stepsto create animation from storyboards. If we can automati-cally generate animation from storyboards, it would be a
useful tool for create animation.
In this paper, we present an intelligent storyboard forprototyping CG animation. We address the two typical prob-lem in generating animation from 2D storyboards:
INTELLIGENT STORYBOARD FORPROTOTYPING ANIMATION
1) Estimation of 3D location of characters from 2D view
2) Generating continuous character motion from the sparsepose specified by the storyboards
Storyboards generally represent a coarse description of
the character motion. We need to interpolate the charactermotion between each storyboards. However, the conventionalkeyframe interpolation is not enough because the pose of
the characters can be very different between the two adja-cent frames. We use the example-based interpolation tosupplement missing character motion.
2. SYSTEM OVERVIEW
Fig.2 shows the concept of the intelligent storyboard. Theinput to the system is storyboards written by the user. The
Fig.2 Concept of the intelligent storyboard
Junichi Hoshino Yumi Hoshino
University of Tsukuba / PRESTO, JST
Fig.1 Example of the storyboards
Estimating 3D location
Example-based interpolation
Storyboards
Animation
Object andMotionDatabase
DigitalBankSystem
problem of making animations from storyboards is that there
is not enough information to generate 3D character motions.We supplement the missing information by using the geo-metric constraint and example-based learning method. 3D
model of the scene and the characters are stored in the cur-rent system.
3. ESTIMATION OF 3D CHARACTERLOCATIONS FROM 2D VIEW
The first step of generating animation is to estimate 3Dcharacter position from 2D view. We estimate the 3D char-acter position from 2D line segments on the storyboards. Fig.
4 shows the relationship of 2D center line on the image planeand a 3D body part. The position and orientation of a bodyparts can be represented as x=[r,t]T where t is the translation
and r is the rotation along the xyz axis.
The error of a 2D center line and a body part can becalculated as the minimum distance between 3D line seg-
ment P=(p1,p
2) and plane M.
h( x, l)= ( Rp1+ t)⋅ n( Rp2+ t)⋅ n (1)
R is a3x3 matrix derived from r. n is a unit normal vectorof plane M.
n= q1′ × q2
′
q1′ × q2
′ (2)
h(x,l) can be linearized around l = l̂i and x = x̂i−1 usingthe Taylor expansion.
h x l h x lh x l
xx x
h x l
ll li i i
i i ii
i i ii( , ) ( ˆ , ˆ )
( ˆ , ˆ )( ˆ )
( ˆ , ˆ )( ˆ )≈ + − + −−
−−
−1
11
1∂∂
∂∂
(3)
where ∂∂h
x ,
∂∂h
l are partial derivatives.
∂∂
∂∂
∂∂
h
x
h
r
h
t=
, , ∂∂h
r
n R p I
n R p I
T
T=
××
[ ]
[ ]1
2 ,
∂∂h
t
n
n
T
T=
(4),(5),(6)
∂∂h
l can be represented as follows.
∂∂
∂∂
∂∂
∂∂
∂∂
h
l
h
p
h
p
h
q
h
q=
1 2 1 2
, , ,' ' (7)
∂∂
h
p
n RT
1 0=
,
∂∂
h
p n RT2
0=
(8),(9)
∂∂
n
q
q qRp t q I nn q I
q qRp t q I nn q I
T T
T T1
1 2
1 2 2
1 2
2 2 2
1
1'
' '
' '
' '
' '
[ ] [ [ ] [ ]]
[ ] [ [ ] [ ]]=
×+ − × + ×
×+ − × + ×
(10)
∂∂
n
q
q qRp t q I nn q I
q qRp t q I nn q I
T T
T T2
1 2
1 1 1
1 2
2 1 1
1
1'
' '
' '
' '
' '
[ ] [ [ ] [ ]]
[ ] [ [ ] [ ]]=
×+ − × + ×
×+ − × + ×
(11)
these coefficients are used for kalman filter.
We minimize the errors between 2D center line and 3Dbody parts. We use Extended Kalman Filter (EKF) for opti-mization. The measurement equation is
z H x vi i i= + (12)
Fig.4 Fitting 2D line segments and 3D body center
Fig.5 Action trajectory in eigen space. The complex changesof each joints can be represented as a simple trajectory ineigenspace.
storyboard
O
3D character
h(x,l) M
PQ
Q'
xk
PQ Q'
x'k
xk
O O
storyboard storyboard
Fig.3 A relationship of the storyboard and 2D-3D line seg-ments
e1
e2
0
where
zh x l
xx h x li
i ii i i= −−− −
∂∂
( ˆ , ˆ )ˆ ( ˆ , ˆ )1
1 1 (13)
Hh x l
xii i= −∂
∂( ˆ , ˆ )1 (14)
vh x l
ll li
i ii i= −−∂
∂( ˆ , ˆ )
( ˆ )1 (15)
The optimization can be done using the Kalman Filter
equations [5,6].
ˆ ˆ [ ˆ ]x x K z H xi i i i i i= + −− −1 1 (16)
K H H H Ri i i i iT
i= +− −
−σ σ1 11( ) (17)
σ σ σi i i i iK H= −− −1 1 (18)
4. EXAMPLE-BASED INTERPOLATIONOF THE CHARACTER MOTION
Storyboards generally represent a coarse description of
the character motion. We need to interpolate the charactermotion between each storyboards. However, the conventionalkeyframe interpolation is not enough because the pose of
the characters can be very different between the two adja-cent frames. We use the example-based interpolation tosupplement missing character motion.
We represent character’s pose parameters in eigen spaceto reduce the computational cost for searching. Fig.5 showsthe example of the walking action in eigenspace. We can see
that the complex change of each joints can be representedas a simple curve in eigenspace.
Let the 3D pose parameters xp of the person p be
xt = [xt1, xt
2,....xtm]T (11)
where i is the number of frames of training gesture and m isthe total number of the parameters (n=30 for the example inFig.2). The average of the pose parameters can be estimatedas
c = 1N
xtΣt = 1
N
(12)
where N is the number of the whole sample action data. Thedifference D
p can be estimated as
D p = xp – c (13)
The covariance matrix of the pose parameters Q is
Q = XXT (14)
where. X = [D1,D2,...D p]. Eigen vector and eigen value canbe obtained from the following equation.
Qui = λ iui (15)
The k-dimensional eigenspace is constructed by the keigenvectors, which correspond to the largest k eigenvalues,
as basis vectors. A set of the pose parameters xi is projected
onto the eigenspace by
gt =[e1, e2,....ek]T(xt – c) (16)
Assuming the pose displacement between two successiveframes to be small, the two projections onto the eigenspaceare close to each other. So the actions are represented as a
continuous trajectory in the eigenspace. These action trajec-tories are used to interpolate character motion.
The personal characteristics of motion can be also repre-
sented in eigenspace. Fig.6 shows the samples from the threepersons doing the same action. (a) shows the difference be-tween the person. By estimating the variance of motion be-
tween different persons, we can synthesize many other mo-tion with different personal characters. (b) shows the aver-age action and reconstructed motion with different personal
characters.
Fig.7 Sample dance sequences in eigenspace
-3
-2
-1
0
1
2
3
-4 -3 -2 -1 0 1
e 2
e1
Fig.6 Analysis and synthesis of personal characters
(a) sample of 3 persondoing the same action
(b) average action and recon-structed action
0
0.5
1
1.5
-1 0 1 2e
2
1e
0
0.5
1
1.5
-1 0 1 2
e2
average action
generated action
5. ANIMATION EXAMPLES
Fig. 7 shows the example of the dance sequences ineigenspace. Sample 3D motion is extracted from video us-
ing our computer vision technique [7]. Fig.8 (a) is the col-lection of storyboards for input. Fig.8 (b) is the generatedanimation sequences.
Fig.9 shows the example of generating walking anima-tion. (a) is the input storyboards and (b) is the generated ani-mation sequences.
6. CONCLUSIONS
In this paper, we propose an intelligent storyboard forprototyping CG animation. The 3D position of the charac-
ters are estimated from 2D views. We also describe the mo-tion synthesis method using the example-based interpola-tion. The future work includes more sophisticated interpre-
tation of line drawing to create characters. Higher level rec-ognition of the scenes and simple language instruction maybe also useful.
7. REFERENCES
[1] John Hart:”The Art of Storyboard: Storyboarding for Film, TV,and Animation”, Focal Press, 1999
[2] Mark Simon:”Storyboards: Motion in Art”, Focal ress, 2000
[3] Amy Spaulding:”The Page as a Stage Set: Storyboard PictureBooks”, Scarecrow, 1995
[4] “A Visit With Disney's the Little Mermaid and FriendsStoryboard”, Mouse Works, 1993
[5] A. Gelb et al.:"Applied Optimal Estimation", Cambridge, MA,MIT Press.
[6] H.W. Sorenson:"Kalman Filtering: Theory and Application",New York, IEEE Press, 1985
[7] J. Hoshino, H. Saito, M. Yamamoto:”Automatic Registrationof Virtual Objects onto Human Image Sequences”, IAPR Int.Conf. on Pattern Recognition, ICPR 2000, pp. 175-177, 2000
(a) input storyboards
(b) Generated animation sequence
Fig. 9 Example of walking animation produced fromstoryboards
Fig.8 Example of generating dance sequences. Colors showsthe correspondance of input storyboards and generated ani-mation sequences.
(a) input storyboards
(b) generated animation sequence