Gesture Databases

-Thoughts on the subject

MotivationSeamless Human-Computer interactionCurrent research

ARVIKA DWARF ARTHUR Smart Spaces

Cheaper wireless ”spy” cameras

Gesture AnnotationStandard protocol for future expansion Other systems uses XML, e.g. NITE,

PETS, SmartKom Database in a file Queries SQL Database import for efficient sharing

Levels of abstraction

Levels of Abstraction Level 1. Pixel based segmentation

Pixel annotation Contour Bounding box

Level 2. Finger- and hand-tracking 2D/3D – Image plane or 3D space DOF – 3 DOF or 6 DOF.

Level 3. Gesture commands Level 4. Contexts (application states, changing

illumination, etc.)Levels 1-2 describe individual images, while levels 3-4

describes sequences

Levels of Abstraction Short sequences

1. Segmentation: Static gestures on different backgrounds, skin colour, and illumination (environments)

2. Tracking: Short trajectories on a subset of environments

3. Gesture commands: One gesture per sequence with different users in few selected environments

Long sequences 3. Gesture commands: Series of subsequent gestures 4. Contexts: Application contexts

Annotation application

Recording Gesture DatabasesConsequences and Responsibilies

Choosing gesture ”alphabet” Designing interfaces

Approach Human-Human non-verbal communication Scenarios for observation

Environment Attributes Illumination

Indoor light Indoor shadow Indoor/outdoor mixed light Outdoor Light Outdor Shadow Backlight

Environment Attributes Background

Selected coloured cloth Very Easy Floor Easy White wall Easy Clean table Easy Grass Easy Tiled floor Medium Brick wall Medium Tiled ground Medium Messy ground Hard Messy table Hard Dynamic vs. Static

Camera SettingsManual/Auto gainManual/Auto WhitebalanceFOVFocal Length

Static Gesture Commands G1: Pointing gesture G2: Residue (nothing)

Command Gestures:

G3: ”Confirmed”/”Yes” G4: ”Undo”/”Stop”/”No” G5: ”Copy-Paste” G6: ”Grap” G7: ”System menu”/”Object property menu”

Static Gesture Postures

G1: Pointing Gesture

Static Gesture Postures

G2: Residue

Static Gesture Postures

G3: Confirmed, Yes

Static Gesture Postures

G4: Stop, Undo, No

Static Gesture Postures

G5: Copy-Paste object

Static Gesture Postures

G6: Grap object

Static Gesture Postures

G7: Menu (system or object specific)

Recordings

Camera attached to glasses.

Recordings

Indoor light, clean table Outdoor light, clean tableBetter control of light, but hard shadows on table.

Uncontrolled light, skin colour varation, reflection on table.

Recordings

Outdoor light, messy table Backlight, messy tableChaotic edges all the the image, varying background colours

Autogain darkens the important information

Recordings

Outdoor light, floor Changing light, white wallFuzzy pattern in floor and its colours are not so good

Changing the light cause colour variation

Recordings

Outdoor light, tiled wall Changing light and backgroundTiles are repetitive and that information can maybe be used in the segmentation

No control of light, skin colours, nor background information