Efficient Video Brows-ing

Using Multiple Synchronized ViewsHeymo Kou

What is the two main technologies applied for efficient video browsing? (one for audio, one for visual content)

Question

Background Current technology Advanced technology Summary Reference

Table of contents

Growth of digital contents data

Digital video market growth

From your◦ Smart phones◦ Notebooks◦ Webcams◦ Digital camera and camcorders◦ Security and monitoring cameras

With advanced streaming technology◦ Fast Internet access◦ MPEG-4 format

Digital video becomes ubiqui-tous

Search through categories◦ Similar to Internet shopping mall

We search for big categories Then smaller categories …and so on…

User should choose which to browse◦ Should check whether the selected data matches

what user was finding Time consuming!

Manual categorizing and annotation◦ One by one?

Current technology for finding a video data

Too complicated◦ Lack of efficient algorithm

Time consuming◦ Multimedia calculation ∝ exponential

Inaccuracy◦ Video data is increasing exponentially

Cataloging manual has a somewhat limit point◦ Manually cataloging is done by human hand that

mistakes can be happened

Problem with current video search and browsing technologies

MPEG-7 Standards Speech indexing Shot Boundary Detection Time Scale Modification of Audio Signals Storyboards, Moving Storyboards and Ani-

mation Adaptive Accelerating Fast Playback Streaming Synchronized Views

Technologies for advancedimage and video retrieval

Standardized by ISO/IEC◦ International Standard Organization◦ International Electrotechnical Commission

Not a video encoding format XML to store metadata

◦ Attached to timecode in multimedia By this tag

◦ Able to index and search efficiently Yet, improvement is needed

MPEG-7 standard

Search through speech transcripts◦ Finds familiar metaphor of free text search

Automatic speech recognition (ASR)◦ Indexed transcript → semantic information

Main advantage : Representation◦ Speech is built of words

Speech indexing

Frame

Key frame

Shot◦ Group of frames which represents similar frames

Definitions

Start key frame end key frame animation

Context◦ Meaningful information within multimedia data

3 levels of video browsing◦ Browsing a large collection of videos◦ Browsing a ranked list of videos◦ Browsing a single video to find relevant segments

Definitions

Shot Boundary Detection(SBD) algorithm◦ Completely automatic

Key frames are selected and extracted◦ Saved as JPEG files

High Accuracy and Efficiency◦ Still, fault detection problem is unsolved

Shot Boundary Detection

SBD algorithm

Similar to scene selection of dvd

Audio browsing is as important asvideo browsing◦ Except images, most digital contents are audible

Faster audio browsing is necessary Speeding up of audio signal by

◦ By deleting small audio segments◦ Especially, human speech signals are quasi-peri-

odic

Time Scale Modification ofAudio Signals

Improvement of TSMTime-Domain Harmonic Scal-

ing(TDHS) technique

Time-Domain, Pitch Synchro-nous Overlap Add

Time Scale Modification(TSM) algorithm

Waveform Synchronous Over-lap(WSOLA)

Synchronous Overlap-Add SOLA

Storyboard◦ a set of one or more pages, each consists of a two

dimensional array of key-frames, sorted in chrono-logical order.

Animation◦ a quick slide show, where each of the key-frames is

shown for a fixed short period (e.g., 0.6 seconds) Moving Storyboard (MSB)

◦ the animated key frames, fully synchronized with the original audio track. Each key-frame is shown for the entire duration of the associated shot.

Storyboards, Moving Story-boards and Animation

Very fast video playback (without audio) Ordinary fast forward depends only on

speed◦ There is a chance to miss important scene

Accelerates until new scene is met Requires less computation load

Adaptive Accelerating Fast Playback

Image for adaptive fast play-back

Example in surveillance camera

Real-use of adaptive fast play-back

Server preprocesses media◦ Keep same media, but different speed encoded

When user selects other speed◦ 1. pause current media◦ 2. open file with same content with selected

speed◦ 3. seek to the corresponding position◦ 4. play the selected view

Needs no extra computational load◦ However, requires more storage: Tradeoff

Streaming Synchronized Views

Can browse multiple videos at once

Split frames every given time◦ (i.e 10 seconds)

Strong information scent is visible◦ With aggregation of occurrences

Browsing Multiple Videos: MovieDNA

Image of movieDNA

Summary of main proper-ties

ViewVisual

AudioTypical

speedup rateStati

cDy-

namic

Full video (w/o TSM) ○ ○ 1 – 2XVideo Skim ○ ○ 2 – 20XSlide show (w/o TSM) ○ ○ 1 – 2XAdaptive Fast Playback ○ 5 – 30XAnimation ○ 10 – 40XStoryboard, mosaic ○ NA

Streaming synchronized views and movieDNA◦ Less computation, multiple videos at once

Active accelerating fast playback◦ Most useful at analyzing surveillance videos

SBD & TSM◦ Efficient for implementing above technologies

Then, what is current limitation?

Conclusion

Any questions?

Q & A

What is the two main technologies applied for efficient video browsing? (one for audio, one for visual content)

Answer : The two main technologies are Shot Boundary Detection(SBD) for visual content and Time Scale Modification(TSM) for audio signals

Answer

Shot Boundary Detection◦ http://muvis.cs.tut.fi/sbd.html

Key frame◦ http://en.wikipedia.org/wiki/Key_frame

Synchronous Overlap-Add◦ http://www.surina.net/article/time-and-pitch-scaling.html

Digital Video Market Growth◦ http://

articles.businessinsider.com/2011-12-13/research/30508929_1_fios-cable-providers-video-streaming-service/2

Amount of Digital data◦ http://

www.emc.com/collateral/analyst-reports/diverse-exploding-digital-universe.pdf

References (1/2)

Streaming synchronized view◦ http://

www.visus.uni-stuttgart.de/forschung/visualisierung-und-visual-analytics/visuelle-analyse-videostroeme/adaptive-fast-forward-for-video-surveillance.html

MovieDNA◦ http://

homepage.mac.com/juggle5/WORK/publications/HICSS34.pdf

References (2/2)