Learning realistic human actions from movies

By Abhinandh PalicherlaDivya AkuthotaSamish Chandra Kolli

Introduction

Address recognition of natural human actions in diverse and realistic video settings.

Addresses the limitations (lack of realistic and annotated video datasets)

Visual recognition progressed from classifying toy objects towards recognizing the classes of objects and scenes in natural images .

Existing datasets for human action recognition provide samples for few action classes .

To Address these limitations we implement

• Automatic annotation of human actions • Manual annotation is difficult

• Video classification for action recognition

Automatic annotation of human actions

Alignment of actions in scripts and videosText Retrieval of human actionsVideo datasets for human actions

Alignment of actions in scripts and videos

…117201:20:17,240 --> 01:20:20,437

Why weren't you honest with me?Why'd you keep your marriage a secret?

117301:20:20,640 --> 01:20:23,598

lt wasn't my secret, Richard.Victor wanted it that way.

117401:20:23,800 --> 01:20:26,189

Not even our closest friendsknew about our marriage.…

…RICK

Why weren't you honest with me? Why did you keep your marriage a secret?

Rick sits down with Ilsa.

ILSA

Oh, it wasn't my secret, Richard. Victor wanted it that way. Not even our closest friends knew about our marriage.

…

01:20:17

01:20:23

subtitlesmovie script

• Scripts available for >500 movies (no time synchronization) www.dailyscript.com, www.movie-page.com,

www.weeklyscript.com …

• Subtitles (with time info.) are available for the most of movies• Can transfer time to scripts by text alignment

Script alignment: Evaluation

Example of a “visual false positive”

A black car pulls up, two army officers get out.

• Annotate action samples in text• Do automatic script-to-video alignment• Check the correspondence of actions in scripts and movies

a: quality of subtitle-script matching

Text Retrieval of human actions

“… Will gets out of the Chevrolet. …” “… Erin exits her new truck…”

• Large variation of action expressions in text:

GetOutCar action:

Potential false positives: “…About to sit down, he freezes…”

• => Supervised text classification approach

Video Datasets for Human actions 1

2 m

ovie

s2

0

diff

ere

nt

movie

s

• Learn vision-based classifier from automatic training set• Compare performance to the manual training set

Video Classification for action recognition

SPACE-TIME FEATURES

Good performance for action recognition Compact and provide tolerance to

background clutter, occlusions and scale changes.

INTEREST POINT DETECTION

Harris operator - with a space-time extension.

We use multiple levels of spatio-temporal scales

σ = 2(1+i)/2 , i = 1, …, 6τ = 2j/2 , j = 1, 2

I. Laptev. On space-time interest points. IJCV, 64(2/3):107–123, 2005.

DESCRIPTORS Compute histogram descriptors of volume

around the interest points.(∆x , ∆y , ∆t ) is related to the detection scales by ∆x , ∆y =

2kσ, ∆t = 2kτ

Each volume is divided into (nx, ny, nt) grid of cuboids.

We use k = 9, nx, ny=3, nt=2.

..CONTD

For each cuboid, we calculate HoG and HoF (optic flow) descriptors

Very similar to SIFT descriptors, adapted to the third dimension.

SPATIO-TEMPORAL BOF

Construct a visual vocabulary using k-means, with k = 4000. (Just like what we do in hw3)

Assign each feature to one word. Compute a frequency histogram for the

entire video, Or, a subsequence defined by a spatio-temporal grid.

If divided into grids, concatenate and normalize.

SPATIO-TEMPORAL BOF

Construct a visual vocabulary using k-means, with k = 4000. (Just like what we do in hw3)

Assign each feature to one word. Compute a frequency histogram for the

entire video, Or, a subsequence defined by a spatio-temporal grid.

If divided into grids, concatenate and normalize.

GRIDS

We divide both spatial and temporal dimensions.

Spatial – 1x1, 2x2, 3x3, v1x3, h3x1, o2x2 Temporal – t1, t2, t3, ot2 6 * 4 = 24 possible grid combinations! Descriptor + grid = channel.

NON-LINEAR SVM Classification using a non-linear SVM Multi-channel Gaussian kernel

V = vocab size, A = mean distances between training samples

Best set of channels for a training set is found by a greedy approach.

WHAT CHANNELS TO USE?

Channels may complement each other Greedy approach to pick the best

combination

Combining channels is more advantageousTable: Classification performance of different channels and their combinations

EVALUATION OF SPATIO-TEMPORAL GRIDS

Figure: Number of occurrences for each channel component within the optimized channel combinations for the KTH action dataset and our manually labeled movie dataset

RESULTS WITH THE KTH DATASET

Figure: Sample frames from the KTH actions sequences, all six classes (columns) and scenarios (rows) are presented

• 2391 sequences divided into the training/validation set (8+8 people) and test set (9 people)

• 10 fold cross validation

Table: Confusion matrix for the KTH actions

RESULTS WITH THE KTH DATASET

ROBUSTNESS TO NOISE IN THE TRAINING DATA

Up to p=0.2 the performance decreases insignificantly

At p=0.4 the performance decreases by around 10%

Figure: Performance of our video classification approach in the presence of wrong labels

ACTION RECOGNITION IN REAL-WORD VIDEOS

Table: Average precision (AP) for each action class of our test set. Comparison results for clean (annotated) and automatic training data and also results for a random classifier (chance)

ACTION RECOGNITION IN REAL-WORLD VIDEOS

Figure: Example results for action classification trained on the automatically annotated data. We show the key frames for test movies with the highest confidence values for true/false; pos/neg

• the rapid getting up is typical for “GetOutCar”• the false negatives are very difficult to recognize

• occluded handshake• hardly visible person getting out of the car

CONCLUSIONS

Summary Automatic generation of realistic action samples Transfer of recent bag-of-features experience to

videos Improved performance on KTH benchmark Decent results for actions in real-videos

Future direction Improving the script-video alignment Experimenting with space-time-low-level-features Internet-scale video search

THANK YOU