AT&T Research at TRECVID 2009

Content-based Copy Detection

TRECVID 2009

•TREC Video Retrieval Evaluation•Specials for 2009 •Tasks

▫surveillance event detection▫high-level feature extraction▫search (interactive, manually-assisted,

and/or fully automatic)▫content-based copy detection

Video data

•Sound and Vision▫The Netherlands Institute for Sound and

Vision news magazine, science news, news reports,

documentaries, educational programming, and archival video

•BBC rushes unedited material

All materials in MPEG-1.. yep!)

Datasets • Development

▫ tv7.sv.devel (32.9 GB) (reference) ▫ tv7.sv.test (31.4 GB) (reference) ▫ tv8.sv.test (64.3 GB) (reference) ▫ tv7.bbc.devel (12.2 GB) (non-reference) ▫ tv7.bbc.test (10.9 GB) (non-reference) ▫ tv8.bbc.test (10.8 GB) (non-reference)

• Test ▫ tv7.sv.devel (32.9 GB) (reference) ▫ tv7.sv.test (31.4 GB) (reference) ▫ tv8.sv.test (64.3 GB) (reference) ▫ tv9.sv.test (114.8 GB) (reference) ▫ tv7.bbc.devel (12.2 GB) (non-reference) ▫ tv7.bbc.test (10.9 GB) (non-reference) ▫ tv8.bbc.test (10.8 GB) (non-reference) ▫ tv9.bbc.test (19.0 GB (non-reference)

Content-based copy detection

•copyright control•business intelligence•advertisement tracking•law enforcement investigations

Video transformation • Picture in picture (The original video is inserted in front of

a background video) • Insertions of pattern • Strong reencoding • Change of gamma • Decrease in quality

▫ Blur, change of gamma, frame dropping, contrast, compression, ratio, white noise

• Post production ▫ Crop, shift, contrast, caption (text insertion), flip (vertical

mirroring), insertion of pattern, Picture in Picture (the original video is in the background)

• Change to randomly choose 1 transformation from each of the 3 main categories.

AT&T Research at TRECVID 2009Content-based Copy Detection•Applications

▫discovering copyright infringement of multimedia content

▫monitoring commercial air time▫querying video by example

•Approaches▫digital video watermarking▫content based copy detection (CBCD).

Overview

Content based sampling•Shot boundary detection (SBD)

▫Adopts a “divide and conquer” strategy▫Six independent detectors:

Cut, fade in, fade out, fast dissolve (less than 5 frames), dissolve and motion

▫Each detector is a finite state machine (FSM)

•FSMs depent on two types of visual features:▫Intra-frame (only one frame)▫Inter-frame (current frame+previous frame)

Overview

Transformation detection andnormalization for query keyframe•Letterbox detection•Picture-in-picture detection•Query Keyframe Normalization

Transformation detection andnormalization for query keyframe• Letterbox detection• Picture-in-picture detection

• Canny edge detection operatorhttp://en.wikipedia.org/wiki/Canny_edge_detector

Transformation detection andnormalization for query keyframe•Query Keyframe Normalization

▫Equalize and blur the query keyframe to overcome the effect of change of Gamma and white noise transformations.

Transformation detection andnormalization for query keyframe

•And we have 10 types of query keyframe: original, letterbox removed, PiP scaled, equalized, blurred and flipped versions of these five types

Overview

Reference keyframe transformation

•Only 2 transformations ▫Half-resolution rescaling

For compared with the detected PiP region in the query keyframes

▫Strong re-encoding For dealing with the strong re-encoded query

keyframes.

•And we have 3 types of reference keyframe

Overview

Scale-invariant feature transform SIFT Extraction

Scale-invariant feature transform SIFT Extraction•It’s main feature for locating video copies

▫Locating the keypoints that have local maximum Difference of Gaussian values both in scale and in space. (specified by location, scale and orientation)

▫Computing a descriptor for each keypoint. The descriptor is the gradient orientation histogram, which is a 128 dimension feature vector.

Overview

Locality sensitive hashing (LSH)•The basic idea

▫hash the input items so that similar items are mapped to the same buckets with high probability

a – random vector following a Gaussian distribution with zero mean and unit variance

w – preset bucket sizeb – in range [0,w]

Overview

Indexing and search by LSH

•Sort LSH values independency•Save with SIFT identifications in separate

index file•SIFT identifications: (String)

▫Reference video ID▫Keyframe ID▫SIFT ID

Overview

Keyframe level query refinement

•Two issues:▫the original SIFT matching by Euclidian

distance is not reliable

▫it‘s possible that two SIFT features that are far away mapped to the same LSH value

Keyframe level query refinementRandom Sample Consensus (RANSAC)

Keyframe level query refinementRandom Sample Consensus (RANSAC)

• Randomly select 3 pairs of matching keypoints (having the same LSH)

• Determine the affine model

• Transform all keypoints in the reference keyframe into the query keyframe

• Count the number of keypoints in the reference whose transformed to the coordinates of their matching keypoints in the query keyframe. These keypoints are called inliers

• Repeat steps 1 to 4 for a certain number of times, and output the maximum number of inliers

Keyframe level query refinement

Transformations: PiP, shift, ratio..

Overview

Keyframe level result merge

• If one reference keyframe appears more than once in the 12 lists

• New relevance score set to be maximum score

Pair Query keyframes Reference keyframes

1 Original

Original

2 Flipped

3 Letterbox removed

4 Letterbox removed & flipped

5 Equalized

6 Equalized & flipped

7 Blurred

8 Blurred & flipped

9 OriginalEncoded10 Flipped

11 Picture in Picture (PiP)Half12 PiP & flipped

Overview

Video level result fusion

Get pair (i, j) with the best sum relevance

Overview

Video relevance score normalization

•Normalize the relevance scores into range [0,1]

x – original relevance scorey – normalized one

Overview

CBCD result generation

•Query video ID•Reference video ID•Information of copied reference video

segment•Starting frame of copied segment in the

query video•Decision score

CBCD Evaluation Results• Dataset

▫1407 short query videos▫838 reference videos▫208 non-reference videos

• Extract▫For entire reference video set

268,000 keyframes 57,000,000 SIFT features

▫For entire query video set 18,000 keyframes 2,600,000 SIFT features

CBCD Evaluation Criteria

Parameters for NoFA profile

Parameters for Balanced profile

CBCD Evaluation Results

CBCD Evaluation Results

CBCD Evaluation Results

TransformationsATTLabs.NoFA.1 ATTLabs.Balanced.2

Actual Minimum Actual Minimum

T2 55.4 0.672 1.283 0.732

T3 55.0 0.224 0.59 0.214

T4 0.381 0.381 0.413 0.391

T5 0.239 0.239 0.7 0.214

T6 55.1 0.284 0.974 0.291

T8 0.269 0.269 1.585 0.329

T10 0.515 0.515 2.045 0.52

About• http://trec.nist.gov/• http://www.itl.nist.gov/iaui/894.02/projects/trecvid/• http://

www-nlpir.nist.gov/projects/tvpubs/tv9.papers/att.pdf

Want more information?

Kirill Lazarev

Skype: kirill_lazarevMail: k.s.lazarev@gmail.com

Twitter: http://twitter.com/kslazarev