Case Study 2

A hierarchical Bayesian network for event recognition

of human actions and interactionsSangho Park, J.K. Aggarwal

Multimedia Systems, vol. 10, pp. 164-179, 2004

Fuzzy SystemsLifelog management

• Overview

• Post estimation using a hierarchical Bayesian network

• Recognition by DBN

• Relative constraints

• Experiment

• Summary

Outline

2/35

Overview

• Recognition of human interaction– Applications: video surveillance, video-event annotation, virtual

reality, human-computer interaction, and robotics– Difficulty: Ambiguity caused by body articulation, loose clothing,

and mutual occlusion between body parts

• Previous work– A method to segment and track multiple body parts in two-

person interactions– Multilevel processing at pixel level, blob level, and object level

3/35

Motivation

• A methodology– To estimate body-part pose– To recognize different two-person

interactions including pointing, punching, standing hand-in-hand, pushing, and hugging

• System component– Bayesian network: estimate body

poses– Dynamic Bayesian network:

classify a sequence of body poses

4/35

Head Pose Estimation

• Environmental setup: lighting conditions, reflectance of light from the head

• Head pose: head’s 3D rotation angles, visible part

5/35

Head Pose Estimation: Example

• V1: angle of the vector• V2: ratio of the two ellipses• a, B: fixed

• P(V1=C|H2=B) = 0.18

6/35

Arm Pose Estimation

7/35

Arm Pose Estimation: Example

• P(V5=B|H3=C, H4) = 0.34

8/35

Leg Pose Estimation

9/35

Leg Pose Estimation: Example

10/35

Overall Body Pose Estimation (1)

11/35

Overall Body Pose Estimation (2)

12/35

Body-part Pose Recognition by DBN

• DBN hidden states– Q1: set of DBNs for legs {“both legs are together on the

ground”, “both legs are spread on the ground”, “and “one foot is moving in the air while the other is on the ground”}

– Q2: set of DBNs for the torso {“stationary”, “moving forward”, and “at least one arm gets withdrawn”}

– Q3: set of DBNs for arms {“both arms stay down”, “at least one arm stretches out”, and “at least one arm gets withdrawn”}

13/35

Interaction Recognition

• Whole-body pose: q1, q2, q3– {“stand still with arms down”, “move forward with arm(s)

stretched outward”, “move backward with arm(s) raised up”, “stand stationary while kicking with leg(s) raised up”, etc.}

• Two-person interaction– Subject = {torso, arm, leg} – Verb = {raise, lower, stretch, withdraw, stay, move forward,

move backward} – Object = {head, upper body, hand, lower body}

14/35

Relative Constraints: Spatial

• Examples– “standing hand-in-hand”: the torsos of the two persons be side

by side and facing in the same direction– “pointing at the opposite person”: the torsos of face one another

• Relative position and orientation– Gross level: proximity between two persons– Intermediate level: relative orientations of the torso poses

between the two persons– Detailed level: relative configuration of individual body parts

15/35

Relative Constraints: Temporal

• Interval temporal logic: before, meet, overlap, start, during, and finish

• Example– A pushing interaction

• Event A: a person moving forward with arms stretched outward toward the second person

• Event B: m of the second person

16/35

Experimental Results

• Human interaction (9): approaching, departing, pointing, standing hand-in-hand, shaking hands, hugging, punching, kicking, and pushing

• Image– 320*240 pixels– 15 fps– 6 pairs of different people with various clothing– 56(?) sequences (6*9)

17/35

Interaction Examples

18/35

BN’s Belief Changes

19/35

Performance of DBNs

• Leave-one-out-cross-validation– Training: 5 sequences, test: 1 sequence

• Accuracy: 78%– Approaching: 100– Departing: 100– Pointing: 67 similar interaction– Standing hand-in-hand: 83– Shaking hands: 100– Hugging: 50 occlusion– Punching: 67 similar interaction– Kicking: 83– Pushing: 50 similar interaction

20/35

Semantic Interpretation

21/35

Summary

• Contribution– A hierarchical framework for the recognition of two-person

interactions– BN for managing ambiguity in human interaction– A human-friendly vocabulary for high-level event description– Stochastic graphical model

• Future works– Extending the method to crowd behavior recognition– Incorporating various camera-view points– Recognizing more diverse interaction patterns

22/35

Case Study 3

Evolutionary Learning of Dynamic Probabilistic Models with Large Time LagsA. Tucker, X. Liu and A. Ogden-Swift

International Journal of Intelligent Systems,Vol. 16, no. 5, pp.621-646, 2001.

Fuzzy SystemsLifelog management

• Introduction

• Background

• Methodology

• Algorithm

• Evaluation

• Conclusions

Outline

24/35

Introduction

• Multivariate time-series (MTS)– A large number of interdependent variables– Large time lags between causes and effects (ex. Oil refinery

processing)

• Learning dynamic Bayesian networks– Not focused on learning models automatically – Focused on models with small time lags– Challenge task for large datasets with large possible time lags

25/35

Dynamic Bayesian Networks

• Bayesian networks– A set of n nodes {x1,…, xn}, representing the N variables in the

domain

– Each node, xi has a finite set of ri mutually exclusive states, vi1 to viri.

– Each node xi with a set of parents, πi has an associated probability table P(xi|πi).

• Dynamic Bayesian networks consist of BNs at differing time slices– Links over different time lags (non-contemporaneous links) and

within the same time lag (contemporaneous links)

Background

26/35

Learning Bayesian Network Structures

• K2/K3 algorithms– Use a greedy search which begins with an empty structure with no

links– Explores the effect of adding each of the possible links to the

current structure– K2 (a log likelihood metric) / K3 (a description length metric)

• Branch and Bound technique– Perform an efficient exhaustive search by stopping any further

exploration along a search path based on a bound

• Evolutionary methods– Larranaga et al. used a genetic algorithm with a repair operator to

remove cycles– Wong et al. used evolutionary programming with freeze, defrost

and a knowledge guided mutation (KGM)– Sahami used the mutual information

• Missing data management: Structural EM algorithm with Dempster ’s expectation maximization algorithm

Background

27/35

Evolutionary Learning Bayesian Network Structures

• Learning BNs involves scoring candidate network structures– Log likelihood– Description length metric of a network structure– Description length metric of encoding the dataset given that model

– n: number of nodes– ri: possible instantiations of the node– qi: possible instantiations of the parent nodes– Fij = ∑Fijk

– Fijk: frequency of occurrences in the dataset that the node xi takes on the value vik and the parent nodes πi take on the instantiation wij

Background

28/35

Representation

• Assume that a dynamic network contains no contemporaneous links

• n = N (# of variables at a single time slice) + Q (# of variables at previous time slice)

• A list of triples represents a possible networks (a,b,l)– a: the parent variable– b: the child variable– l: the time lag

Methodology

29/35

Useful Heuristics

• No contemporaneous links– Finding a good network structure finding a group of simple

tree structures

• LagMutation: Each mutation is based on a uniform distribution with mean equal to the present lag

• Autoregression links (a,a,1)

Methodology

30/35

Seeded GA for Search

• Seed the entire first population with links found from the single link analysis

– Using an approximate method to find a good list of single links rather than scoring the entire set

– Exploiting this knowledge in the first population by seeding it entirely with a random selection of good links

• EP method is particularly efficient at finding a good selection of links with good correlation

– An individual represents a single triples– Self-adapting parameters (SAP)

Methodology

31/35

Algorithm

32/35

Evaluation: Efficiency (1)

• Adapting static BN search algorithms for DBN search– K2/K3– The genetic algorithm– The evolutionary program

• Knowledge guided mutation (KGM)

33/35

Evaluation: Efficiency (2)

34/35

Evaluation: Structural Comparison

35/35

Conclusion

• Problem: Learning dynamic probabilistic models with large time lags

• Proposed method: EP-Seeded GA

• Future works: Discretisation & parameterisation

• Brainstorming – Mutually not-exclusive states Fuzzy BN– Hybrid of the GA and K2/K3

36/35