ON THE MOVE AUTOMATICALLY MEASURING BODY MOVEMENT

RONALD POPPE

Human behavior can be:

Verbal

Nonverbal

Nonverbal behavior consists of:

Para-verbal (acoustics)

Facial expressions

Gaze

Gestures

Body movement

HUMAN BEHAVIOR

This talk is about:

Body pose

Body movement

And how to measure these automatically

Any questions: just ask straight away!

HUMAN BEHAVIOR

Theory:

Body movement

A short history of measuring body movement

Body movement representation

Practice:

Sensors and devices

Processing and analysis

The future

OUTLINE

THEORY: BODY MOVEMENT

BODY MOVEMENT: QUESTION

1 2 3

4

Q: who said

“No, it wasn’t

me honey”

Body pose and movement tells us about:

Action / Intention

Affect / Mental state

Attitude

Body movement can be conscious or unconscious (automatic)

Individual: frustration, aggression

Interaction: dominance, turn-taking and interruptions

BODY MOVEMENT

We can measure body motion to:

Answer hypotheses about nonverbal behavior itself

Answer hypotheses about phenomena with nonverbal correlates

Make (computational) models of human bodily behavior

Control interfaces and games

Top-down (hypothesis-driven) vs. bottom-up (data-driven)

BODY MOVEMENT

Measurement of body motion:

Qualitative (coding)

Quantitative (motion capture)

Qualitative is typically performed manually:

Have observers count/code specific behaviors

Analyze frequency of behaviors over time

BODY MOVEMENT

BODY MOVEMENT

Coding of nonverbal behaviors:

Interpretation can be given

Irrelevant movements can be ignored

Coding scheme should be known beforehand (requires analysis)

Time-consuming (often multiple coders)

Subjectivity of the coders

Depends on viewpoint/quality of cameras

Qualitative, not quantitative (differences in speed, direction and

form should be additionally coded)

BODY MOVEMENT

Automatic measurement of nonverbal behaviors:

Quick (with minimal effort)

Objective

Quantitative

No interpretation of the behavior

Irrelevant behavior affects the analysis

BODY MOVEMENT

BODY MOVEMENT: HISTORY

Aristotle (330BC), Da Vinci (1500), Borelli (1680)

BODY MOVEMENT: HISTORY

Étienne-Jules Marey (1878) and Eadweard Muybridge (1880)

BODY MOVEMENT: HISTORY

Chronophotographic gun

BODY MOVEMENT: HISTORY

BODY MOVEMENT: HISTORY

BODY MOVEMENT: HISTORY

BODY MOVEMENT: HISTORY

Gunnar Johansson (1973)

BODY MOVEMENT: HISTORY

BODY MOVEMENT: REPRESENTATION

BODY MOVEMENT: REPRESENTATION

The body can be represented as segments and joints

Segments are body parts, and have a length

Joints connect segments

Movement takes place at the joints

Humans have 206 joints

Most of them in the back, hands and feet

BODY MOVEMENT: REPRESENTATION

All joints together form a kinematic tree

Two joints at either end of a segment are connected

The root joint is at the top of the tree

End-effectors do not have only

one attached segment

All connections from root to end-

effort are called a kinematic chain

Joints higher in the tree affect those

below in the chain

BODY MOVEMENT: REPRESENTATION

Joints can move in different directions

Each direction is called a degree of

freedom (DOF)

Joints can have up to 3 DOF

How many DOF in the elbow?

How many in each shoulder?

And in the knee?

BODY MOVEMENT: REPRESENTATION

Joints rotate around axes

For each axis, there is a feasible

range of motion

This range is bounded by rotation

constraints

The DOF, with constraints,

determine the space of movement

BODY MOVEMENT: REPRESENTATION

The order of applying the

rotations is important!

We can specify a body pose

by the rotations of all joints

BODY MOVEMENT: REPRESENTATION

Often, we are interested in the location of the end-effectors (hands,

feet, head)

The same location of a hand can have different sets of joint rotations

Comparing positions based on joint rotations is difficult!

Why not use joint locations directly?

BODY MOVEMENT: REPRESENTATION

BODY MOVEMENT: REPRESENTATION

Joint angles

• Easy to analyze single joint

• Difficult to interpret

• Difficult to calculate

distance between joints

Joint positions

• Difficult to analyze single

joint

• Easy to interpret

• Easy to calculate distance

between joints

Segment

lengths

Joint locations are points in 3D

space

Each point can be written as:

(x, y, z) with distances along axes

from the origin

The origin is the point (0, 0, 0)

Axes need to be defined

E.g. origin in center of a room, axis

in direction of walls

BODY MOVEMENT: REPRESENTATION

A body pose can be

described by the joint

locations of all joints

Joint positions are highly

correlated

Moving in a room causes all

joints to change

BODY MOVEMENT: REPRESENTATION

(2.0, 1.8, 3.9)

(1.7, 0.1, 4.3)

(2.8, 1.8, 0.8)

(2.5, 0.1, 1.2)

Global joint locations:

Locations depend on position in the room

Locations are relative to the origin in the room

Why not use the position in space + the local body pose?

Local joint locations:

Locations relative to “something” in the body local origin

BODY MOVEMENT: REPRESENTATION

Local origin is root joint

Often, the root is the pelvis

Location of a joint = location of pelvis +

relative location of joint

Local location = global location – origin

BODY MOVEMENT: REPRESENTATION

Joint locations no longer depend on position in space

We call this position normalization

We can now quickly compare poses and see if they are the same!

But what happens if you turn around?

BODY MOVEMENT: REPRESENTATION

The pose is exactly the same but…

Rotation changes global and local joint locations

Can we do normalization similar to what we did for position?

BODY MOVEMENT: REPRESENTATION

(0.2, 0.7, 0.0)

(0.0, 0.7, 0.2)

0.2

0.2

We can describe the joint locations relative to the

facing direction of the person

Facing direction is determined by the hips

We align the joints to one of the axes

So the facing direction always is the same after

rotation

We call this orientation normalization

BODY MOVEMENT: REPRESENTATION

First, we determine how much a person is rotated

We calculate the angle between the facing direction and the axis

Then we rotate all joints with this angle around the root joint

BODY MOVEMENT: REPRESENTATION

Recap:

We can represent body poses as 3D joint locations

We can normalize joint locations for global position

We can normalize joint locations for global orientation

We can now calculate differences between poses!

Needed for the quantitative analysis of body pose and motion

BODY MOVEMENT: REPRESENTATION

Poses are aligned on the root joint

Pose difference is the summed distance between all pairs of joints

BODY MOVEMENT: REPRESENTATION

What about calculation of motion?

Motion corresponds to body poses over time

So we can calculate the difference between subsequent frames!

By averaging distances over time, we can calculate average motion

For the whole body or per joint/limb

BODY MOVEMENT: REPRESENTATION

PRACTICE

Theory:

Body movement

A short history of measuring body movement

Body movement representation

Practice:

Sensors and devices

Processing and analysis

The future

OUTLINE

BODY MOTION: SENSORS AND DEVICES

Used extensively in movie and games

BODY MOTION: SENSORS AND DEVICES

With special devices:

Mechanical: measuring angles and distances directly

Marker-based: visible markers on the body

Inertial: magnetic sensors on the body

Without special devices:

Vision-based with depth camera

BODY MOTION: SENSORS AND DEVICES

Mechanical: Animazoo Gypsy 5

Suit with sensors that measure

angle and extension

Direct measurement

Multiple actors

Difficult to set up

Limited number of joints

Limits the freedom of movement

BODY MOTION: SENSORS AND DEVICES

Marker-based: Vicon MX

Retro-reflective markers attached to

a suit or straps

Passive or active

Multiple actors

Freedom in marker configuration

BODY MOTION: SENSORS AND DEVICES

Large recording environement required (many cameras)

Invisible markers due to occlusion

Markers might be swapped

Markers might fall off

BODY MOTION: SENSORS AND DEVICES

Inertial: Xsens MVN

Sensors with gyroscopes in straps

Large performance space

Outdoor recording possible

Multiple actors

Risk of drift due to metal in environment

BODY MOTION: SENSORS AND DEVICES

Depth camera: Microsoft Kinect,

Asus Xtion, PrimeSense

No sensors on the body

Multiple actors

Occlusion is a problem

Accuracy is relatively low

Difficulties with direct sunlight

Limited in performance space

BODY MOTION: SENSORS AND DEVICES

BODY MOTION: PROCESSING AND ANALYSIS

After recording, we have a lot of data

Now what?

BODY MOTION: PROCESSING AND ANALYSIS

The cook book:

1. Transform all data to column format

Each DOF is a column

Each time frame is a row

2. Select start and end frames, select corresponding rows

3. If there is noise, apply a filter

4. Normalize for position (global to local)

5. Normalize for orientation

6. Calculate dependent variable

BODY MOTION: PROCESSING AND ANALYSIS

Filtering of data

Remove outliers

Smoothen tracking inaccuracies

Median filter with small window length often does the trick

BODY MOTION: PROCESSING AND ANALYSIS

Dependent variables:

Amount of movement

Average distance to reference pose

For the whole body or for individual limbs, e.g.:

Total movement in the left leg

Difference in pose between left and right arm

BODY MOTION: PROCESSING AND ANALYSIS

Distance between two subjects

(Average) distance over time

More complex dependent variables

BODY MOTION: PROCESSING AND ANALYSIS

BODY MOTION: THE FUTURE

Bottom-up research:

Collect data

Mine patterns

Typical in information retrieval and pattern recognition

Why not in psychology?

BODY MOTION: THE FUTURE

More detailed/accurate body motion measurements

Facial expressions

BODY MOTION: THE FUTURE

In “the wild”

Deal with “non-cooperative” people

More natural behavior (also in actual environment)

Longer-term behavior

BODY MOTION: THE FUTURE

Recorded footage

Learning behavior from freely available recordings

Automatic re-analysis of material

Quantitative instead of qualitative

BODY MOTION: THE FUTURE

Experiments with virtual characters and robots

Control over behavior (modalities)

Analyze variability of behavior between subjects

BODY MOTION: THE FUTURE

Klette & Tee, “Understanding Human Motion: A Historic Review”,

Computational Imaging and Vision: Human Motion, 2008

Moeslund, Hilton, Krüger & Sigal (Eds.) “Visual Analysis of Humans -

Looking at People”, Springer 2011

Poppe, “Vision-based human motion analysis: An overview, Computer

Vision and Image Understanding”, 2007

Poppe, “A survey on vision-based human action recognition”, Image

and Vision Computing, 2010

Poppe, Van der Zee, Heylen & Taylor, “AMAB: Automated

Measurement and Analysis of Body Motion”, Behavior Research

Methods, forthcoming

BODY MOTION: SOME LITERATURE

Thanks!

Contributors: Dirk Heylen, Paul J. Taylor, Sophie van der Zee

Otherwise, send me an email: r.w.poppe@utwente.nl

QUESTIONS!?