Digital Face CloningDigital Face Cloning

David Bennett Christophe Hery Steve Sullivan

George Borshukov J.P. Lewis Lance Williams

Paul Debevec Fred Pighin Li Zhang

David Bennett Christophe Hery Steve Sullivan

George Borshukov J.P. Lewis Lance Williams

Paul Debevec Fred Pighin Li Zhang

BostonSeattle

Los AngelesSan Francisco

DefinitionDefinition

• Process of capturing an actor’s performance and/or likeness in a digital face model

E.g. photogrammetry (input)E.g. photogrammetry (input)

E.g. photogrammetry (output)E.g. photogrammetry (output)

MotivationMotivation

• Replicate a live actor (e.g. celebrity)

– Stunts

– Extreme makeup

– Younger/older/slimmer clone

• Creating realistic faces from first principles is difficult (digitizing is easier than modeling)

Purpose of the coursePurpose of the course

1. Review the field

2. Suggest directions

Digital Clones: HistoryDigital Clones: History

CG Humans in the moviesCG Humans in the movies

• First 3D actor representation in the movies: Peter Fonda in FutureWorld (1976!)

• Not intended to be photorealistic

1990s1990s

1990s1990s

• Terminator 2 (brief, texture projection)

• Jurrasic Park (legs only, does not count)

• Titanic (distant)

• Startup attempt: Virtual Celebrity/Global Icons, purchased virtual rights to Sammy Davis Jr., others

1998: Hollywood is ready 1998: Hollywood is ready

• "I don't see them as competing with real actors,” Richard Masur, head of Screen Actors Guild

• Projected uses: de-aging, celebrity vacuum ads.

• A television series with a virtual cast was considered

Recent and currentRecent and current

Recent and currentRecent and current

• Space Cowboys (2000), Enemy at the Gates (2001): brief and distant, but successful

• Disney Facial Cloning test (afternoon session)

• Matrix sequels (2003): “superpunch” full-screen for 13 seconds (afternoon session)

• SpiderMan 2 (reflectance capture)

• Lemony Snicket: CG baby in several shots (afternoon session)

• Discovery Channel Churchill/Hitler clones (MPC)

ObservationObservation

• There has not been a “breakthrough” introduction of digital clones

• Rather, clones were initially used in the distance, and briefly; they are getting closer, and withstanding longer study.

• Current state of the art: full screen for several seconds (see afternoon Perception session)

Syllabus – morning (theory)Syllabus – morning (theory)

8:30 Introduction and Overview Fred Pighin

9:00 Face Scanning and Dense Motion Capture Technologies

Li Zhang

10:00 Break or Mathematical Background (schedule change)

10:15 Reflectance Modeling and Capture Paul Debevec and J.P. Lewis

11:15 Facial Parameterization and Cross-Mapping J.P. Lewis and Fred Pighin

12:15 Lunch

Syllabus – afternoon (practice)Syllabus – afternoon (practice)

1:30 Case study: Face Cloning at ILM Steve Sullivan and Christophe Hery

2:15 Perception Experiment (schedule change)

2:30 Case Study: Disney Facial Cloning Lance Williams

3:15 Case study: Leaping the Uncanny Valley with Data   (Face Cloning in the Matrix sequels)

George Borshukov

4:00 Break

4:15 Case study: Polar Express David Bennet

5:00 Perception of Facial Realism J.P. Lewis

5:15 Panel on the Future of Digital Face Cloning All

Face Cloning at SiggraphFace Cloning at Siggraph

• Courses

– 10. Realistic Materials in Computer Graphics

– 28. From Mocap to Movie: The Making of "The Polar Express“

– 29. High-Dynamic-Range Imaging and Image-Based Lighting

• Papers (Skin & Faces session on Monday)

– Automatic Determination of Facial Muscle Activations From Sparse Motion Capture Marker Data

– Face Transfer With Multilinear Models

– Postproduction Relighting and Reflectance Transformation with Time-Multiplexed Illumination

• Sketches

– Image-Based Rendering From a Sparse Set of Images

– Implementation of Modeling Hair From Multiple Views

– Model Flowing: Capturing and Tracking of Deformable Geometry

– Performance Geometry Capture for Spatially Varying Relighting

– Joint Motion and Reflectance Capture for Relightable 3D Video