1

SS5305 – Advanced Motion Capture

2

SS5305 – Advanced Motion Capture • 3D motion capture for entertainment

• Data capturing using a motion capture system

• Pre-processing of 3D motion capture data

• Analysis of processed motion data

• Motion regeneration of entertainment (traditional dancing)

Assessment: Examination: NONE Course Work: 100%The course work comprises of the following:-- 2 Class Tests – (20% each)- Project Assignment – (60%)

3

An Introduction to Advanced Motion Capture

()

()

()

()

DCM

4

Objectives

• History of Motion Capture• Basic Motion Capture System using Digital

Video Cameras• Optical/Video Motion Capture Systems• Introduction to Qualisys• Motion Capture Trends

5

Motion Capture System using Digital Video Cameras

6

Camera Set up

SMN Arosha Senanayake SS4301 - Software Engineering

• Focus, exposure set manually• Shutter speed set to 1/1000• Connect camera to laptop via DV cable and PCMCIA port

7

Calibration of Motion Capture System

SMN Arosha Senanayake SS4301 - Software Engineering

8

Marker Placements and Capturing

• Placed to digitized the points for later analysis• Arm and Leg are computed as angles in the s/w

Shoulder

Wrist

Knee

Toe

Elbow

Hip

Ankle

Heel

9

Stick Motion Regeneration

Hardward for Motion Capture

1. Passive optical

2. Active optical

3. Time modulated active

4. Markerless

5. Non-optical: mechnical, magnetic

Passive Optical Capture

Reflectors are placed on the body.

Advantages:

1. High resolution (sub-pixel)2. Works in ambient light3. No wires or electronics!

Disadvantages:

1. Occlusion – objects may block cameras2. Marker identity – how does the camera tell which marker is which?3. Requires rest of body to be blocked out (another color) 4. Variable lighting is problematic

In common setups 6 to 24 cameras are used to avoid occlusion.

Active Optical Capture

LED lights used as markers.

Advantages:

1. Works in the most lighting conditions – including in dark. 2. Similar resolution as passive

Disadvantages:

1. Power must be supplied to LEDs – wires on the suit2. Occlusion, Marker identity and Error are still the main issues.

Marker swap problem – Caused by lack of marker identity Identity of markers may switch when nearby.

1 1

2 2

1

12

2

Time Modulated Active Markers

Strobe each LED light, one at a time, at the same frame rate as the camera. Camera sees only one marker at a time.

Advantages:

1. Solve marker identity 2. Data recorded is much cleaner and higher resolution

Disadvantages:

1. Harder to implement – requires radio signal to transmit camera frame sync to the LEDs2. LEDs must have additional hardware to determine order of strobe.3. Frame rate is divided by the number of markers. Factual = F camera / N markers

Cost: $50,000 for 8 camera, 1 actor systems

Markerless Systems

Most markerless systems use structured light.A projector or emitter casts out light in a coherent pattern, such as vertical bars.

Advantages:

1. No markers needed on the body2. Can record depth information with one camera

Disadvantages:

1. Most systems record point clouds. The do not record marker positions. 2. Addition steps are needed to fit a skeleton inside the point cloud.3. Occlusion is still a problem4. Frame rate depends more on processing time for data fit than on camera rate.

Motion Capture Workflow

1.

2.

3.

4. 5. 6. 7.

automated by most mocap systems

2007-01-24

Qualisys Quality in Motion

2007-01-24

OQUS camera

2007-01-24

Detection of markers – 2D and 3D

Camera 1 Camera 2

• Oqus 5+ 4 MP @180fps

• Oqus 1 0.3MP @ 250fps

• Oqus 210c 2 MP @ 360fps (Color High Speed Video)

• Oqus 4 3 MP @ 480fps

• Oqus 3+ 1.3 MP @ 500fps 0.3 MP @ 1750fps

• Oqus7 12MP@120fps 3MP@300fps

Oqus models:

•Oqus 5+ series 4.0Mpixel @ 180fps

•Oqus 4 series 3.0Mpixel @ 480fps

•Oqus 3+ series 1.3Mpixel @ 500fps

•Oqus 2 series 2Mpixel @ 360fps

•Oqus 1 series 0.3Mpixel @ 250fpsFocus 5mAperture 2.8Exposure 500usThreshold 200

Integrated WiFi• Run system off a laptop with no cable between the first camera and the laptop.

Suitable for outdoors – Oqus 3+ & Oqus 4• Active filtering technology enables problem free measurements in direct sunlight.

High-speed video mode• The only camera on the market to combine motion capture with high-speed video in the same body! Capture 1.3Mpixel full-frame video @ 500fps• NEW! Stream High Speed Video data to your PC with Oqus 2 series

Oqus210c – Color High-speed Video

• High Speed Video only

• Color image

• Can be calibrated with the rest of the system

• Up to 340 fps in Buffer mode (full frame, full resolution)

• Real-time color video streaming at 100fps (0,5 Megapixel)

• Real-time color video overlay

2007-01-24

Qualisys as a mobile system

Daisy-chain connectivity: Quicker set-up, shorter cable lengths (standard – 15 meters) for larger covered volume

Bundled data/power cables: Only 8 cables and 2 power supplies for 8 cameras

On-board 2D digitizing: No hub needed, run it off a laptop

Camera and cable cases: Standard light-weight Pelican cases, 3 cases for a complete 8 camera system.

WiFi communication: Between the first camera system and PC

Colfibre tripods: Light-weight with enough height (2m).

System connectivity:

1. Direct connection between the first camera and PC No hubs needed

2. Single 120W power supply for 4 Oqus cameras Minimal number of cables from wall outlets

3. Bundled data and power cable Only 8 cables and 2 power supplies for an 8 camera system

4. Daisy-chain connection (from camera to camera)Shorter cable lengths even with larger volumesQuicker setup

Easy to relocate – Pack and go

5. Standard 100Mbit Ethernet Up to 100m between the

cameras (if ever needed)

Major components of Qualisys system:

Oqus - Motion Capture Camera• Tracking movements in 3D with infrared technology

Data processing, analysis and visualization:•Visual3D and Motion Monitor - biomechanical analysis and report generation•Motion Builder, 3DStudioMax, Maya – RT visualization and animation production•Matlab, LabView, Excel - complex mathematic calculations.

QTM – Qualisys Track Manager•Calibration and 3D tracking •Data collection and cleaning•Automatic identification of markers•Synchronization with external equipment (force plates, EMG etc.)•3D Data real-time streaming (to Visual3D, Motion Builder, Matlab, LabView)•Data export to various formats (c3d, tsv, avi etc)

iPhone & iPod touch applications:

In December 2011 Qualisys proudly introduced two new free iPhone applications

The view finder allows you to see individual camera views easing the setup of large volumes.

QTM remote allows users to start andstop measurements remotely as well as add event markers.

Avater (2010),James Cameron

Solution:

Block off the face using individual, head-mounted cameras, which record only the face. Use motion cameras and passive markers for the body.Allows for both large volumes and small details.

Trend –Performance captureis a collection of techniques that combine torecord the totalmotion of an actor.

Markers include:Body capture Green lines, white dotsFacial capture Head-mounted device, /w camera boomsHair capture Blue and red ropes

Trend – Markerless capture:Origins in 3D laser scanning

3D Lego Digitizer http://www.rchoetzlein.com/project/digitizer/

Trend – Markerless capture: Structured Light

Faster: Do all lines at once

Projector with structured light mapped onto the object. Use two cameras to determine object structure.

Structured light can be linear, binary coded, gray coded, or color coded. The encoding allows you to uniquely identify points.

Light may be infrared (Kinetic).

Q: High frequency gives details about height of point. But how do we tell if the point is on left or right side of obj?

A: Low frequency gives overall characteristics of pixels.

No markers.

Structured light creates a point cloud.

Skeleton is fit inside point cloud from root joints to extremities.Torso defines primary orientation,and also constraints placement of next joint layer in hierarchy.

Volume construction

Pointcloud

Fittorse

Fitextremities

Trend – Markerless capture: Direct-to-3D models

http://www.youtube.com/watch?v=dTisU4dibSc&playnext=1&list=PLD31C3C36D294EEDB

Christian Theobalt, Stanford University

http://www.stanford.edu/group/biomotion/Markerless.html

Performance Capture from Sparse Multi-view Video, SIGGRAPH 2010

Trend – Monocular capture

Fabio Remondino, Andreas RoditakisInstitute for Geodesy and Photogrammetry - ETH Zurich, Switzerland3D Reconstruction of Human Skeleton from Single Images or Monocular Video Sequences2003, 25th Pattern Recognition Symposium

One camera, without depth, is under-constrained.

However, the human body has fixed limb lengths and ratios.

Use the body ratios as an additional constraint.

Trend – Low Cost Systems

Cheap hardware: Microsoft Kinect, Web cameras.

Open source software: OpenKinect open kinect drivers libfreenect open kinect drivers

OpenNI skeleton fittingFaceAPI facial tracking

Main challenges: 1) Integration into existing frameworks,2) Usually requires programming experience3) Can be difficult to modify for research