Gesture Recognition in a Class Room Environment

Michael Wallick

CS766

Virtual Videography

Place cameras in an environment

Automatically edit video off-line

Output should look like a professional editor

Our Implementation

Looking at the classroom domain

Recorded one semester of CS559 (Computer Graphics)

Computer Vision in Virtual Videography Understand what is happening on the

chalkboard Writing on the board

Understand what the professor is doing Location Actions

Chalkboard…

Partition the board into regions

Regions are semantically related groups of writing

Regions can be approximated using computer vision Let’s treat this as a black box … it just

happens

Gesture Recognition

Understand gestures or actions by a performer

Generally used as an input to a computer

Understand what the professor is doing Pointing Writing Reaching

Writing can be confused with Pointing and Reaching

Template Matching for G. R.

Generate templates of known gestures

Match an unknown frame with a template matching algorithm Sum of Squared Difference Cross Correlation Image Difference …

Implement of Gesture Recognition

The user selects several template images Pointing

Reaching

Format the templates

Separate the lecturer

Crop the image

Resize the images 256x256

Build the Recognition Mask

Load each template into the mask

For each “on” pixel, increment the mask at that location

Recognizing Gestures

Separate the lecturer from foreground Crop and resize For every “on” pixel, increment the “Score” by

that value in the mask Compute Confidence as

(float) (Score/Mask_Total) Compute Confidence for all gestures

A Gesture Matches if Confidence is:

Under 50% but much larger than other gestures

Over 50% and not too close to other gestures

Example: Ground State

Example: Pointing

Example: Reaching

Mistakes

Overall the results are good

Sometimes individual frames are not correct

Solution

For each frame, look at surrounding frames

Label frame with gesture of the majority

Where to go from here…

Use the regions to Validate the gestures Determine what is being pointed at

Incorporate the writing information with the gestures

Write paper and webpage!

Conclusions

We want to use gesture recognition for Virtual Videography

Gestures can be used to drive camera model

Find gestures by template matching

For each frame, take the “average” around a region of frames to correct errors

Thank You!

Questions/ Comments?