Human Movement Summarization and Depiction from Videos

Hao Jiang

Boston College

Yijuan Lu

Texas State University

How do we describe human movementin videos?

2

Methods to illustration motion

Muybridge’s photos to capture human movement

Methods to illustration motion

4

Manual drawing or depiction from mocap data

S. Bouvier-Zappa, V. Ostromoukhov and P. Poulin, “Motion cues for illustration of skeletal motion capture data”, Symposium on Non-Photorealistic Animation and Rendering 2007.

Our method: depiction from 2D videos

Our method automatically extracts human motion and generates depictions.

video

:wChallenges

Movement segmentation

Finding human body parts and linking them though time

Finding detailed trajectories of “particles” on the human subject

Generating depictions of the movement using arrows, colors, trajectories.

Movement segmentation

8

Trajectories of particles on human subject can beused to quantify the movement at one instant.

Movement segmentation

9

Comparing the trajectory features between successive frames:v u

Body part detection and association

10

• Body part detections are in each single frame; we need to link them through time (tracking)• There are often errors that need to be corrected (error correction)

Body part tracking

11

We formulate the tracking problem as a data association problem

Body part tracking

12

We formulate the tracking problem as a data association problem

Correct tracking errors

13

We use the following network to correct part tracking errors:

Predictions

Correct tracking errors

14

We use the following network to correct part tracking errors:

Predictions

Error correction becomes a shortest path problem.

Finding long term trajectories

15

motionsmoothness

structure

unary

Finding long term trajectories

16

Gradient descent

Generate depiction

17

Trick: polynomialcurve fitting togenerate pleasingResults.

Results

18

Results

19

Results

20

Results

21

More Results

22

Summary

It is indeed possible to automatically generate movement depictions from videos.

Not only the depiction can be treated as an art illustration, it also can serve the purpose of meaningful movement description.

Future work: finding the “optimal” movement depiction.

23

Thank You!

24