Character Animation and Control using Human Motion Data Jehee Lee Carnegie Mellon University jehee.
-
Upload
mervin-byrd -
Category
Documents
-
view
219 -
download
0
Transcript of Character Animation and Control using Human Motion Data Jehee Lee Carnegie Mellon University jehee.
Character Animation and Controlusing Human Motion Data
Jehee LeeCarnegie Mellon University
http://www.cs.cmu.edu/~jehee
Character Animation
Final Fantasy Movie Characters
from www.finalfantasy.com
Final Fantasy X
NBA Courtside 2002
NFL 2k2 WWF Raw
All game characters from www.gamespot.com
Motion Capture
• Record movements of live performers– Realistic, highly detailed data can be obtained
Motion capture lab at CMU
Animation from Motion Capture
MotionDatabase
Preprocess
On-lineController
Motion EditingToolbox
MotionSensor
Data
ConvincingAnimation
ControllableResponsiveCharacters
High-Level UserInterfaces
The Art ofAnimation
Animation from Motion Capture
MotionDatabase
Preprocess
On-lineController
Motion EditingToolbox
MotionSensor
Data
ConvincingAnimation
ControllableResponsiveCharacters
MappingLive
Performance
High-Level UserInterfaces
The Art ofAnimation
ComputerPuppetry
Interactive 3D Avatar Control
• How to organize data ?– Large collection of motion data
• How to control ?– User interfaces
MotionDatabase
PreprocessOn-line
Controller
MotionSensor
Data
ControllableResponsiveCharacters
High-Level UserInterfaces
Related Work (Motion Control)
Rule-based Control system
[Bruderlin & Calvert 96]
[Perlin & Goldberg 96]
[Chi 2000]
[Cassell et at 2001]
[Hodgins et al 95]
[Wooten and Hodgins 96]
[Laszlo et al 96]
[Faloutsos et al 2001]
Example-based Statistical Models
[Popovic & Witkin 95]
[Bruderlin & Willams 95]
[Unuma et al 95]
[Lamouret & van de Panne 96]
[Rose et al 97]
[Wiley & Hahn 97]
[Gleicher 97, 98, 01]
[Sun & Mataxas 2001]
[Bradley & Stuart 97]
[Pullen & Bregler 2000]
[Tanco & Hilton 2000]
[Brand & Hertzmann 2000]
[Galata et al 2001]
[Lee et al 02]
Related Work (User Interfaces)
Graphical User Interfaces
Performance
(Motion capture devices)
Performance
(Vision-based)
[Bruderlin & Calvert 96]
[Laszlo et al 96]
[Rose et al 97]
[Chi 2000]
[Badler et al 93]
[Semwal et al 98]
[Blumberg 98]
[Molet et al 99]
“Mocap Boxing” (Konami)
[Blumberg & Galyean 95]
[Brand 1999]
[Rosales et al 2001]
[Ben-Arie et al 2001]
Motion Database
• In computer games– Many short, carefully planned, labeled motion clips– Manual processing
Walk Cycle StopStart
Left Turn
Right Turn
Motion Database
• Our approach– Extended, unlabeled sequences of motion– Automatic processing
Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.
Sketch Interface
Motion Data for Rough Terrain
Motion Data for Rough Terrain
Unstructured Input Data
Connecting Transitions
Local Search for Path Following
Comparison to Real Motion
Comparison to Real Motion
User Interfaces
Choice-based Interface
• What is available in database ?– Provided with several options– Select from among available behaviors
Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.
Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.
How to Create Choices ?
Clustering
Find Reachable Clusters
A
B
C
D E
F
G
Most Probable Paths
Cluster Forest
B
C
DE
F
G
BD
E
F
G
Performance Interface
MotionDatabase
SearchEngine
AnimateAvatars
Vision-basedInterface
Silhouette extraction and matching implemented by Jinxiang Chai
Database Search3 sec
Animation from Motion Capture
MotionDatabase
Preprocess
On-lineController
Motion EditingToolbox
MotionSensor
Data
ConvincingAnimation
ControllableResponsiveCharacters
MappingLive
Performance
High-Level UserInterfaces
The Art ofAnimation
ComputerPuppetry
The Art of Animation
• Animators need good tools– Modify, vary, blend, transition, filter, …
MotionDatabase
Motion EditingToolbox
ConvincingAnimation
The Art ofAnimation
Challenges in Motion Editing
• Reusability and flexibility– Motion data is acquired
• For a specific performer• Within a specific environment• In a specific style/mood
• High dimensionality
• Inherent non-linearity of orientation data
Related Work
Physically-based
Signal processing/
Interpolation
Optimization + Interpolation
Stochastic
Modify [Popovic& Witkin 99]
[Unuma et al 95]
[Bruderlin &
Williams 95]
[Sun&Metaxas 01]
[Lee & Shin 01, 02]
[Gleicher 97, 98, 01]
[Lee & Shin 99]
[Perlin 95] [Bradley&Stuart 97]
[Pullen&Bregler 00]
Transition/
Blend[Rose et al 96]
[Lamouret & van de Panne 96]
[Rose et al 97]
[Sun&Metaxas 01]
[Lee & Shin 01, 02]
[Tanco&Hilton 00]
[Brand &
Hertzmann 00]
[Galata et al 01]
Basic Techniques
• Multiresolution Analysis– Signal processing approach– Transition, blend, modify style/mood, and
resequence
• Hierarchical displacement mapping– Constraint-based approach– Interactive editing– adaptation to different characters/environments.
Multiresolution Analysis
• Represent signals at multiple resolutions– give hierarchy of successively smoother signals– facilitate a variety of signal processing tasks
)0(m)3(m )1(m)2(m
Decomposition
• Reduction: upsampling followed by smoothing• Expansion: smoothing followed by downsampling
)(nm
)1( nm
)1( nd
Reduction Expansion
)(nm )1( nm )2( nm )0(m
)1( nd )2( nd )0(d
)(nm )1( nm )2( nm )0(m
)1( nd )2( nd )0(d
Decomposition
Reconstruction
Enhance / Attenuate
Jehee Lee and Sung Yong Shin, General Construction of Time-Domain Filters for Orientation Data, IEEE Transactions on Visualization and Computer Graphics, to appear.
Jehee Lee and Sung Yong Shin, A Coordinate-Invariant Approach to Multiresolution Motion Analysis and Synthesis, Graphical Models (formerly GMIP), 2001.
Enhance / Attenuate
Walk
Limp
Turn
?Turn with a Limp
Walk
Limp
Turn
Turn with a Limp
Analogy
• Low frequency (Content)
Result = Limp + (Turn – Walk)
• High frequency (Style)
Result = Turn + (Limp – Walk)
Walk Turn
LimpTurn with
A limp
Walk
Strut
Run
Stub toes Limp
Stitched
Re-sequence
Reconstruction
)0(m
)0(d
)1(d
)2(d
)0(m
Reconstruction
)(E 0m
)0(m
)0(d
)1(d
)2(d
Reconstruction
)(E 0)0()1( dmm
)0(m
)0(d
)1(d
)2(d
Orientation Representation
• Inherent non-linearity of orientation space
12222 zyxw 3S
Filtering Orientation Data
kikikiki aaa qqqq 0)(F
• How to generalize convolution filters ?
3S
Related Work• Re-normalization
• Azuma and Bishop (94)
• Global linearization• Johnstone and Williams (95)
• Local linearization• Welch and Bishop (97)• Fang et al. (98)• Lee and Shin (2002)
• Multi-linear• Shoemake (85)
• Optimization• Lee and Shin (96)• Hsieh et al. (98)• Buss and Fillmore (2001)
Re-normalization
iq
1iq
2iq
1iq
2iq
5
1,
5
1,
5
1,
5
1,
5
1
Linearization
3S 3R
Exponential and Logarithm3SqT
q
Exponential and Logarithm
1q
qq 1
Exponential and Logarithm33 RSI T
),,,( 0001I
),,,( zyx0
Exponential and Logarithm33 RSI T
log exp
I
Global Linearization
ii pq log I
Angular Displacement
ip
ii pp 1
1ip
1iq
)( iiii pppp 11
iq3S3R
Angular Displacement
ip
ii pp 1
1ip1
1
ii qq
)( iiii pppp 11
ii qqI 1
Angular Displacement
ip
ii pp 1
1ip1
1
ii qq
)( iiii pppp 11
ii
iii
exp1
q
ii qqI 1 11
i log ii qq
Local Linearization
3S 3R
iii pqq
11log
The Drifting problem
iii pqq
11log
Our Approach
3S 3R
iiii ppqq
111log
Filtering Orientation Data
ip
1ip 2ip1ip2ip
i
1i
1i
2i
Filter Properties
• Coordinate invariant
• Time invariant
• Symmetric
ii qqRH,RRH where
niinnn
qqSH,SSH where
3where Sbabqabaq ,,)H()H( ii
Coordinate Invariance
m
)0(
)0(
)1(
)(
mdd
d
n
)0(
)0(
)1(
)(
mdd
d
T
n
mTDecomposition Reconstruction
T
T
Coordinate Invariance
• Independent to the choice of coordinate systems
Basic Techniques
• Multiresolution Analysis– Signal processing approach– Transition, blend, modify style/mood, smoothen,
resequence
• Hierarchical displacement mapping– Constraint-based approach– Interactive editing and adaptation
Motion Editing through Optimization
• Constraints[Witkin & Kass 88] [Cohen 92] [Gleicher 98]
– Features to be retained– New features to be accomplished
• Find a new motion– Satisfy given constraints– Preserve original characteristics
Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99
Motion Representation
• Motion of articulated characters– Bundle of motion signals– Each signal describe positions / orientations / joint angles
Basic Idea
• Inter-frame relationship– Enforce constraints– By inverse kinematics
• Inter-frame relationship– Avoid jerkiness– By curve fitting
Displacement Mapping
Displacement Map
Original Motion
Target Motion
Hierarchical Displacement Mapping
• Representation of displacement maps– An array of spline curves– Over a common knot sequence
• Flexibility in representation– Hard to determine knot density– Adaptive refinement is needed
Adaptive Refinement
• Multi-level or hierarchical B-splines[Lee, Wolberg, and Shin 97] [Forsey and Bartel 95]
– Sum of uniform B-spline functions– Coarse-to-fine hierarchy of knot sequences
Multi-Level B-spline Fitting
0f
1f 2f
10 ff 210 fff
Adaptation to Rough Terrain
Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99
Adaptation to New Characters
Character Morphing
Animation from Motion Capture
MotionDatabase
Preprocess
On-lineController
Motion EditingToolbox
MotionSensor
Data
ConvincingAnimation
ControllableResponsiveCharacters
MappingLive
Performance
High-Level UserInterfaces
The Art ofAnimation
ComputerPuppetry
Hyun Joon Shin, Jehee Lee, Michael Gleicher, and Sung Yong Shin, Computer Puppetry: An Importance-based Approach, ACM Transactions on Graphics, 2001.
The videos were made by Hyun Joon Shin, Tae Hoon Kim, Hye-Won Pyun, Seung-Hyup Shin, Jehee Lee, Sung Yong Shin, and many others at the Korea Broadcasting System.
Summary
• Motion data processing– Multiresolution analysis– Hierarchical displacement mapping
• Interactive control– Motion databases– User interfaces: Choice, sketch, performance
Future Work
• Autonomous virtual humans– Convincing appearance, movements– Reasonable level of intelligence
• Collect real world data– Motions, pictures, videos, voices, facial
expressions, and physical properties
Computer Puppetry
• Immediate mapping from a performer to an animated character
MotionSensor
DataMapping
LivePerformance
ComputerPuppetry
Time Invariance
• Independent to the position in the signal
Time
Statistical Model
Statistical Model
Motion Representation
StatisticalModel
MarkovProcess
User Control
UpdateAvatar Pose
Markov Process
• Raw data– Extended– Unstructured
• Processed data– Connected– Flexible
Cluster Forest
Cluster Forest