ViPERVideo Performance Evaluation Toolkit

viper-toolkit.sf.net

Performance Evaluation

• Ideal:– Fully automated– Repeatable– Can be used to compare results without

access to the product– Predictive validity– Useful for the task– General enough to cover any task

Reality• Cannot fully automate for most domains.

– Subjective– Objective

• Results from subjective studies cannot be easily extended, if at all.

• Ground truth is hard to gather, lossy, and evaluation metrics are hard to formulate.

• It is often difficult to determine what is really being measured.

The ViPER Toolkit

• Unified video performance evaluation resource, including:– ViPER-GT – a Java toolkit for marking up

videos with truth data.– ViPER-PE – a command line tool for

comparing truth data to result data.– A set of scripts for running several sets

of results with different options and generating graphs.

The Video Performance

Evaluation Resource

Ground Truth Editor

PerformanceEvaluation Tool

TruthData

Video AnalysisAlgorithm

ResultData

SchemaMapping

Metrics Filters

EvaluationResults

Video AnalysisAlgorithm

ResultDataVideo Analysis

AlgorithmResultData

ViPER (Evaluator View)

PerformanceEvaluation Scripts

TruthData

ResultData

SchemaMapping

MetricsTemplateFilter

Template

EvaluationResults

ResultData

EvaluationResultsEvaluation

Results

SchemaMapping

PerformanceEvaluation Tool

MetricsParamsMetrics

ParamsMetricsParams

ViPER (Developer View)

ResultData

ViPER File Format

• Represents data as set of descriptors, which the user defines in a schema.– Each descriptor has a set of attributes,

which may take on different values over the file.

– Like a temporally qualified relational database for each media file, where each row is an instance of a descriptor.

ViPER File Format: Descriptors

Descriptor Types– FILE (Video Level Information)– CONTENT (descriptors of the scene)

• Static attribute values• Single instance of one type for any frame

– OBJECT* (descriptors of instances, including events)• Attributes are dynamic by default• Multiple instances can exist at a single frame.

Attributes• Attribute Types:

– Strings, numbers, booleans, and enumerations

– Shape types, including bounding boxes and polygons

– Relations (Foreign keys)

ViPERGround Truth Editing

viper-toolkit.sf.net

Ground Truth Editing

The Difficulty of Authoring Ground

Truth• Ground truth is tedious and time

consuming to edit.• Ground truth is lossy.

A Generic Video Annotation Tool

• Lets the user specify the task and the interpretation.

• Provides a

Competition• VideoAnnEx

– IBM AlphaWorks MPEG-7 Editor

• OntoLog (OWL)– Jon Heggland’s RDF Video Ontology Editor

• Informedia– CMU Digital Video Library

• PhotoStuff– Still image annotation for the semantic web

Time Line View

• Provides summary of ground truth.• Direct manipulation across frames.• Feedback for indirect manipulation.

Time Line View

• Provides summary of ground truth.• Direct manipulation.

– Quick editing of activities, events, and other CONTENT descriptors.

– Some ability to modify descriptors with dynamic attributes directly, if not the attribute values.

• Feedback for indirect manipulation.– Easier to notice massive changes.

Enhanced Keyboard Editing

• Support for real-time mark-up of events and activities.– Keys for creating and deleting activities.– Keys for controlling rate of display (jog dials).

• Enhance mark-up of spatial data.– Keys for creating, editing of a single descriptor's

attribute.

• Overall attempt to minimize effort in a GOMS model.– Mouse events unnecessary except for polygon editing.

Enhanced Keyboard Editing:

Real-time Example• User assigns keys for three content types. Each key toggles between off/on states for each content type.

• Forward/back decelerate/accelerate video playback. May skip frames, rewind,etc. – In paused mode, space goes to next

frame.

• USB jog dial might be useful.

Enhanced Keyboard Editing:

Spatial Example• Mode selection:– Control-d cycles

through descriptor types.

– Control-a cycles through attribute types.

– Control-s cycles through available descriptors.

• Editing:– Control-n creates new

descriptor of given descriptor type.

– Control-f creates a new attribute of given type if none exists.

– Arrow keys move, arrow+modifier resizes.

Frame View

Schema Editor

ViPER-GT Internals

ViPER-GT:A Video

Ground Truth Annotation

Tool

Schema Editor

ViPER Metadata API

Pure Java MPEG Decoder

AppLoaderPlug-In Manager

Jena

Core GT API

Plug-InsPlug-Ins

Plug-Ins Native Decoders:VirtualDubQuickTime

JMF

Latest Version in Series

Latest Version in Series

• Schema editor.• Timeline view.• Supports undo/redo.• New video annotation widget.

GTF Inputter (Original V-GT)

ViPERPerformance Evaluation

viper-toolkit.sf.net

PE Methodology•Ground truth and results are represented by a set of

descriptive records–Target: An object or content record delineated temporally in the ground truth along with a set of of attributes (possibly spatial)

–Candidate: An object or content record delineated temporally in the results along with a set of of attributes (possibly spatial)

•Requirements–Matching records which are close enough to satisfy a given

set of constraints on:• Temporal range• Spatial location of object• Values of attributes in date-type specific parameter space

Detection and Localization

Detection: whether a target object or content record is properly identified

Localization: how well the target is detected

•Simplest level: –A target is detected if its temporal range overlaps the

temporal range of a single candidate

•Qualifiers and localization constraints–Temporal overlap must meet a certain tolerance (% or #)–Spatial attributes must overlap within a tolerance a frame

by frame basis–Non-spatial attributes must be within a given tolerance

Temporal Localization

•MetricsOverlap coefficient:

• # Of % of target frames detected

–Dice coefficient• # Or % in common• Similarity measure

–Extent coefficient• Deviation in the endpoint

location of ranges

TARGET

CANDIDATES

Attribute Localization

•Each datatype has its own metric–Svalue: edit distance–Point: euclidean distance–Bboxes, oboxes, circle: overlap and dice coefficients–Bvalue, lvalue: exact match [0,1]–Remainder: absolute difference

•Object correspondence–Optimal subset

•Temporal constraints–Frame by frame tolerance–Virtual candidate

TARGET

CANDIDATES

Reporting Metrics •Detections

–List of correct, missed and false detections–Summary of absolute detection scores as a percentage–Summary of overall precision and recall

•Localization–Optimal subset of matching frames–Frame by frame tolerance–Mean, median, SD and maximum values reported

• Issues:–Many-to-one, many-to-many

Evaluation Using “Gtfc”

•Used to provide basic evaluation mechanisms–Requires configuration

• Equivalence classes• Evaluation specification

–Reports• Attribute and descriptor level recall and

precision

Evaluation Configuration

#BEGIN_EQUIVALENCEDISSOLVE : FADE-IN FADE-OUT TRANSLATE

#END_EQUIVALENCE

#BEGIN_EVALUATION_LISTCONTENT Shot-Change

TYPE: CUT FADE-IN FADE-OUT

OBJECT TextTYPE: FULL OVERLAY SCENE*POSITION*CONTENT*MOTION

#END_EVALUATION_LIST

Set up Equivalencies

Evaluate subsets of GT

Allow Selected Performance Measures

Video Evaluation• Providing metrics to judge correctness of

–Value of Attributes–Range of Frames (temporal)–Detection and Localization of objects (spatial)–Moving objects (spatio temporal)

•Degree of Correctness related to –similarity of or distance between descriptors–cost of transformation between result and ideal data

•Performance Metrics reported as–% of correct/incorrect instances

PE Methodology•Ground truth and results are represented by a set of

descriptive records–Target: An object or content record delineated temporally in the ground truth along with a set of of attributes (possibly spatial)

–Candidate: An object or content record delineated temporally in the results along with a set of of attributes (possibly spatial)

•Requirements–Matching records which are close enough to satisfy a given

set of constraints on:• Temporal range• Spatial location of object• Values of attributes in date-type specific parameter space

Detection and Localization

Detection: whether a target object or content record is properly identified

Localization: how well the target is detected

•Simplest level: –A target is detected if its temporal range overlaps the

temporal range of a single candidate•Qualifiers and localization constraints

–Temporal overlap must meet a certain tolerance (% or #)

–Spatial attributes must overlap within a tolerance a frame by frame basis

–Non-spatial attributes must be within a given tolerance

Temporal Localization

•MetricsOverlap coefficient:

• # Of % of target frames detected

–Dice coefficient• # Or % in common• Similarity measure

–Extent coefficient• Deviation in the endpoint

location of ranges

TARGET

CANDIDATES

Attribute Localization

•Each datatype has its own metric–Svalue: edit distance–Point: euclidean distance–Bboxes, oboxes, circle: overlap and dice

coefficients–Bvalue, lvalue: exact match [0,1]–Remainder: absolute difference

•Object correspondence–Optimal subset

•Temporal constraints–Frame by frame tolerance–Virtual candidate

TARGET

CANDIDATES

Metric and Tolerance

Specification•Specification in the evaluation parameter file

descriptor-type descriptor-name [METRIC TOLERANCE]

attribute1 [METRIC TOLERANCE]

attribute2 [METRIC TOLERANCE]

Match Scenarios

FALSE

MISSED

CORRECT

CORRECT

Error Graphs

Localization Graphs

Enhanced Don't Care Example

• In an activity detection, certain segments are often more important than others:– The moment someone enters or exits the

scene.– The moment a thief grabs a bag.

• These segments might be marked up explicitly as part of an activity descriptor, and treated as important during the evaluation.

Enhanced Don't Care Regions

• For object evaluation, Don't Care currently applies only to entire descriptors. – Needs to apply to dynamic attributes at a per-

frame level, as it does for framewise evaluations.

• Need enhanced rules for computing don't-care regions spatially and temporally. For example:– Region of body not part of torso or head is

unimportant.– Frames before this event are unimportant.

Scripting ViPER

• RunEvaluation– Runs sets of comparisons with different

input parameters.