Computer Vision in Graphics Computer Vision in Graphics ProductionProduction

Adrian Hilton

Visual Media Research GroupCentre for Vision, Speech and Signal Processing

University of Surrey

http://www.ee.surrey.ac.uk/CVSSP/VMRG

Overview

Where is computer vision useful in content production?

Where is vision used?

What can and can’t current computer vision do?

Computer Vision in Content Production

Computer vision: analysis & interpretation of real images/video

• Match moving• Matting • Model building• Human motion capture• Studio Production• Facial animation• Image-based illumination

Match Moving

Reconstruction of camera movement to composite CG

Products: 2d3, RealViz … (semi-automatic camera tracking)

… standard tool in film production

Oxford University/2d3

Matting

Separation of foreground and background objects - actor/background separation

- wire/set removal Studio: chroma-key (solved) Post-production: Imagineer, RealViz, …. (open-problem)

Model Building

Capture of real shape for CG models • objects• environments• characters

Shape capture• active sensors (laser/structured-light) - accurate 3D surface measurements - static objects or environments• structure from images - low-accuracy - static objects

Problem: unstructured surface measurements

Model Building (2)

Walking with Dinosaurs - FrameStore

Structured Models

Courtesy Stanford Computer Graphics Lab.

Animation of David

Human Motion Capture

Capture of real motion

Marker based systems• widely used in performance animation whole-body/face• character animation (Golum)• ‘realistic’ motion characteristics• accurate • real-time?

widely used in film production (with post-production)

Human Motion Capture(2)

Markerless human motion capture• advantages: unintrusive; simultaneous appearance capture• model-based visual tracking• low-accuracy• visual ambiguity (uniform apperance, non-rigid shape)

Human Motion Capture(3)

Studio Production

Modelling actors• 3D animated models• animation • re-illumination• loss of visual realism to captured images

Free-viewpoint video• post-production of arbitrary camera views/paths• ‘matrix’ flowmo shots• aim: quality equivalent to captured video

Studio Production (2)

Limitations: - shape detail face/hands/hair - not video quality

Studio Production (3)

Studio Production: Free-view video

Studio Production: Free-view video

Studio Production: Free-view video

Studio Production: Free-view video

Facial AnimationProduction of photo-realistic faces

2D video-based approaches• video rewrite [Bregler’97]• concatenative synthesis [Cossato’98,Ezzat’02]• photo-realistic• limited viewpoint, illumination

3D markers• Performance animation [Phigin’98]• shape only

3D video (shape+appearance)• concatenative synthesis• photo-realistic • control of viewpoint, illumination

3D Video3D Video

3D Video Database3D Video Database

51 people: Expressions + Speech (short sentence)

Initial Face Synthesis from SpeechInitial Face Synthesis from SpeechInput:

Output:

Image-based illumination

Illumination of CG scenes with real illumination• high-dynamic range capture of illumination• environment mapping illumination to CG scene

widely used in production

Debevec SIGGRAPH’00 FiatLux

Computer Vision in Content Production

• Match moving - yes, automatic camera tracking available

• Matting - partial, many unsolved problems

• Model capture – yes, tools for semi-automatic restructuring

• Human motion capture – no, inaccurate

• Studio Production – partial, free viewpoint video

• Facial animation – yes, 3D video

• Image-based illumination – yes, widely used

• Other applications of computer vision….

Future Directions

Video-based rendering• photo-realistic, free-viewpoint rendering of dynamic scenes• relighting required

Video-based animation• animation from captured video (face,whole-body…)• control of motion, viewpoint, illumination

PeopleJon Starck, Joel Mitchelson, Gordon Collins,

Eng-Jon Ong, Ioannis Ypsilos, Rob Dilks, Michael Kalkavouras

CollaboratorsBBC, BT, Sony, Canon, Philips,

Hensons, Framestore, Snell&Wilcox,3D Scanners, AvatarMe

EPSRC – DTI