Quick! We Need a Dancing

Duck!Mike Gleicher & The Graphics Gang

Department of Computer Sciences

University of Wisconsin, Madison

www.cs.wisc.edu/graphics

A Tale of Motion Use and Re-Use

• Can we use retargeting tools to create animation?

• Spacetime-based retargeting (1998) sounded good, what happened?

• Hey Mike – what have you been doing for the past 3 years?

What have I been doing?

• Welcome to Wisconsin!– (yes the snow is up to the mailbox)

Motion Retargeting in Practice

Why haven’t the methods taken off?

• Retargeting solves only one piece of the character animation puzzle

• The results aren’t as good as we might like

• The methods are a pain to deal with

• Maybe motion re-use isn’t the right problem

The Talk Agenda

• Help!

• Show you the complete story– Our whole motion re-use pipeline– How we are dealing with issues– Where the old stuff fits in

• Show you how far we’re getting– And where we might be going

• Make use of Andy’s nice slides

The Original Plan…

• Create animation by re-use

• Easier to beg, borrow, steal, buy, …

Retargeting CompendiumA retrospective 1998-2001

• Some old stuff– Indulge me – it’s 47 seconds

• Some new examples of the old stuff– Notice what’s different

• Found examples

What did you just see

• Found Animation– Found characters on the web– Found motions on the web– Put them together!

• Animating Characters– Not just skeletons– Simple skinning methods

• All examples from motion capture

What did you see?

• Plenty of problems!

• Many relating to the fact that we have characters not just skeletons

• We’ve been lucky to get some good motion, but not all of it is great

Motion is Tough!And it’s not just motion capture’s problem

Motion CaptureSample every frame

– No structure.– No intent.– lots of data!

Dense samples can represent rich frequency content

Noise, sensor errors

Motion CaptureSample every frame

– No structure.– No intent.– lots of data!

Dense samples can represent rich frequency content

Noise, sensor errors

Hand AnimationData at Keyframes

– Structured? – Remember why?– Maybe lots of data.

Good animators create complex frequency content

Quality takes work

Hand AnimationData at Keyframes

– Structured? – Remember why?– Maybe lots of data.

Good animators create complex frequency content

Quality takes work

Hard for academics to obtain quality hand animation

Mocap Pipeline

• Motion Capture is a misunderstood technology

• Retargeting is in the “Process” box– Found Motion, so no control of Shoot– Distinction between Process and Apply might be

blurred

• Similar story for hand animation

Plan Shoot Process Apply

Processing Pipeline

DefineSkeleton

RawData

CleanMarker Data

Clean Skeletal

Datacleanup

identify clean Solve to Skeleton

Clean Skeleta

lData

Retargeting&Control

Mapping

Animation Data

Application

DefineCharacter

DefineCharacter Controls

Why change a perfectly good motion?

• Animation: bring something to life– means we want to change something about the

performance

• Actor vs. character• Restrictions of realism• Performers aren't perfect• Need for usable data (loops, reference

poses, ...)• Studio is not virtual world• Motion re-use

– sometimes we are stuck with what we have

So You Want a Dancing Duck.

• Our motion pipeline

• Driven by need to use “found” stuff

• An overview of the process includes:– Gather model and motion– Match model and motion poses– Bind the model to the motion– Tweak the results using constraint-based

motion editing techniques

How did we do that?

Get Motion(from library)

AnnotateConstraints

Get Character(from library)

Devise skeletonfor character

ComputeAdapted Motion

Lighting Render

ConvertSkeleton

Get from library(stock CD or Web)

Our Software

Commercial Animation System(Maya or 3D Studio MAX)

Start With a Model.

• Off the web

• Created from scratch

• We just get geometry (a mesh)

(Ant model courtesy Hou Soon Ming)

Grab a Motion.

• Off the web

• Freshly captured

• From company moCap archives

• Everybody has a different skeleton

Size up the Situation.

• Our chosen character size and motion size are clearly different

How to connect?

• Find a skeleton for character– Use topology of original motion

• Find deformation parameters– Skinning– Vertex Weights (since we’re doing SSD)

• This is backwards!– Pros design controls with characters– Map animation data on those controls

Skeleton Matching

• But how does the skeleton connect to the character?

• What are the character’s sizes?

• Where should the joints go?

• What about orientations?

Find Character’s Skeleton

• Not simply a geometric operation– Spine down center or back?– How many joints?– Which d.o.f. where?

• Skeleton is an abstraction

Skeletons for humans

• Animals are very complex

• We understand humans pretty well

CopyrightedImage deleted

(skeleton and muscles from an anatomy text)

CopyrightedImage deleted

(functional diagram fromAn anatomy textbook)

We have favorite abstractions

206 bones, muscles, fat,

organs, clothing, …

206 bones, complex joints

53 bonesKinematic

joints

This is not how humans work

Simple Pin Joint

Complex tendon and bone system

Representation of complex human structure with varying degrees of simplification

This can’t be automated!(without lots of data)

• Artistic decisions in how the character’s geometry responds to movement!

• Need to design the set of controls for moving the geometry– Get the right expressitivity– Get the right response – Get the right ease of use

• We’re limiting ourselves to given format

Building Skeleton From Scratch?

• Appears to be very easy

• The joint coordinate systems may not match – Very Tedious

Rotating Skeleton Into Place?

• Tedious process

• Difficult to get certain rotations correct (wrist, shoulders, etc)

Shoulder Example 1.

• A -90 degree rotation around the Z axis in this case yields the following:

Shoulder Example 2.

• In this case, a 90 degree rotation around the X axis yields the same result as before:

FindSkel: the skeleton finder

Given:

• Geometry (a mesh in “rest pose”)

• Desired skeleton

• Our conventions

Find:

• The right skeleton parameters

• The “binding pose” relative to the “zero pose”

Our Approach.

• Instead of a skeleton, create a point cloud of joints in the mesh

• Easy for the user

• Works with nearly any 3d modeling package out there

This is about usability

• Points are easy to specify• Take whatever the user gives us

– Guess the rest– Symmetry– Conventions– Heuristics – Assumptions

• Pass 1: make a full cloud of points• Pass 2: find skeleton and pose

Rotate the Motion’s Skeleton.

• Rotate each bone in the motion to match model point cloud information

• Relies heavily on knowledge of humanoid kinematics

• Maintains coordinate systems

Our Approach - Prepare Motion.

• Need to pose the motion in a way which allows intuitive adjustments to match model’s bind pose

• Zombie (or Frankenstein) pose

Limb Forward Direction

• Hands, elbows, feet, and knees especially problematic

• Feet have a free guide – the toes point forward

Limb Forward Direction Contd.

• Hands have no forward vector

• Easy to add forward point in cloud, but somewhat unintuitive

Motion Simplification Contd.

• When rotating shoulder into position, it is very difficult to take all rotational degrees of freedom into account

• We only allow for rotation around two axes

Non-Humanoid Motion.

• Unless you know at least some of the complexities behind the creature’s degrees of freedom, all bets are off

Finally We Can Bind!

• Despite all the gotcha’s, our technique still works

• With our matching skeletons, any bone-based skinning technique can be applied

• Skinning depends on the skeleton

Now Are We Done?

• Most certainly not!

• The model is not interacting with its environment as it should

• Its feet are nowhere near the ground

Maybe that paper wasn’t useless…I knew we needed Retargeting!

• Model’s root is still following old root translations

• A simple translation to the floor will yield sliding feet and too much bounce

How Do We Fix This Mess?

• Constraint-based motion editing to the rescue!

• Finally something from the paper!

Constraints and Our Example.

• In our case, adding constraints where the motion’s feet hit the floor would probably be sufficient

• As seen before, our new mesh is clearly not satisfying its foot-hits-floor constraints

Solving for the Constraints.

• A quick solve for the constraints yields our model walking nicely along the floor

What’s wrong with the story?

• Philosophically misguided– Force old controls onto new character– Requires convention in controls– Randomly chose hard to use controls

• Ignores MoCap Wisdom– Begin with the ends in mind– Solve problems earlier, not later

• Dealing with clips

Processing Pipeline

DefineSkeleton

RawData

CleanMarker Data

Clean Skeletal

Datacleanup

identify clean Solve to Skeleton

Clean Skeleta

lData

Retargeting&Control

Mapping

Animation Data

Application

DefineCharacter

DefineCharacter Controls

What’s wrong with the story?

• Results aren’t as good as we’d like– Get better constraints

• Solver is too hard to implement– Find new solution paradigm

• High-level control still hard– Build abstracted representations

• Off-line– See upcoming ToG paper

• Now for the quick tour of the first 3

Better Constraints/Objectives

• Feature of constraint-based approach– Just need to find those right ones– Dates back to Witkin&Kass ’88– We still don’t have the grace functional

• I expected to do more of this– Lots of mileage from using a few

constraints well– Figure out how to use what you have

Want Better Results?Devise Better Constraints!

• Self-intersection

• Higher-level properties

• Style ????

• Form ????

• ????

Collision Constraints

• Get your hands out of your pockets!

Constraints don’t fix everything…

This arm does well

This arm doesn’t go through the body, but…

Wins and Losses of Spacetime

• Fast, practical– Linear complexity?

• Solves real problems• Flexibility in:

– Spatial Constraints– Objective Functions ?– Temporal Constraints ?

• Widely applicable• Nice results

• Hard to implement• Poor integration• Off-line• No guarantees• Spatial constraints not

enforced• Flexibility not exploited• Rely on constraints

Is there an alternative?

• Need to deal with spatial and temporal constraints

• Don’t want the messiness of “whole motion” computation

• Handle spatial and temporal constraints separately!

• Per-Frame IK Plus Filter– Lee & Shin ’99 -- first version– Gleicher ’00 & ’01 -- generalization

Per-Frame IK + Filter (PFIK+F)

• IK per frame to solve spatial constraints– But this messes up temporal

constraints

• Filter changes to enforce temporal constraints– But this messes up spatial

constraints

• Iterate until converges, or …

PFIK+F vs. Spacetime• Fast, practical• Solves real problems• Flexibility in:

– Spatial Constraints– Objective Functions (?)– Temporal Constraints

(?)

• Widely applicable• Nice results

• Yes! (requires fast IK)• Yes!

– (depends on IK)– (depends on IK)– (limited, unexplored)

• Yes!• Um, it’s a matter of

taste, and IK quality

PFIK+F vs. Spacetime

• Use standard pieces!• Use standard pieces!

• Choice in which last• Solve spatial

constraints last

• Need good IK

• Hard to implement• Poor integration• Off-line• No guarantees• Spatial constraints

not enforced• Flexibility not

exploited• Rely on constraints

My PFIK+F solver

• Use pieces I have lying around

• Non-linear optimizing solver for IK

• FIR linear filters for temporal constraints

• Propagation of results for temporal coherence

Downsides of PFIK+F?

• No global decisions– Doesn’t handle “don’t cares” as well– Order dependence– No interframe constraints

• Reliance on quality of IK solver

• Not necessarily faster (or slower)

Higher Level Control

• Need to find abstractions of motion

• Can’t just work with details– Even constraints aren’t enough better

• Path Editing (a factoring approach)– Gleicher ’01

• Motion Graphs– Kovar & Gleicher ‘??

The Idea

Original Motion New Path

What is Path Editing?

Factor Motion intoPath And Detail

Edit Path as Needed(use curve editing tools)

Apply Detail to new Path

Why Edit Paths?(because previous work can’t do it)

• Can’t Capture All Possibilities

• Can’t Synthesize All Styles

• Can’t Simulate Complex Motions

• Current Editing Tools Don’t Apply

• Need tools for sculpting motionINTERACTIVELY!

• Need easier to implement tools!

How to Edit a Path

• Determine what the path is– Easy to manipulate, conceptual

representation

• Factor motion into path and detail

• Edit the path

• Put path and detail back together

• Clean up the details

Motion Use Ideal

• Capture a large corpus of motion

• Everything you need is there– Somewhere – but where– Maybe it needs some adaptation

• Have some specification of needs

• Generate motion from this

Current approaches

• Move Trees– Specially designed motions that connect– Requires planning to create– Requires tricks to fit together– Games use 1000s (yes thousands) of

motions!– Graphs/motions done manually

• Style Machines– ???

Goal

• Generate motion, on demand, using an abstracted representation

Corpus of Motions

Corpus of Motions Preprocessing

AbstractedRepresentation

Requests

First (working) Example

• A few notable failures– Best Linear Unbiased Estimators– Dense tables

• Simple seems to work– State matching– Rigid factoring (timescaling soon)

• Graph structures are useful– Today: random walks– Tomorrow: ???

Tricks up our sleeves

• General Transitions are hard– Can be arbitrarily hard– Can be really easy (if similar motions)– Identify when basic techniques work

• Factored representations– More things to match– More ways to apply things

Thanks!

• UW Graphics Gang

• Mainframe Entertainment, House of Moves Studios, Digital Domain

This research is supported by:

• Microsoft, Intel, Pixar, IBM

• NSF Career Award