Audiovisual content exploitation in the networked information society

Roeland Ordelman

Research & Development

Netherlands Institute for Sound and Vision

rordelman@beeldengeluid.nl

Crowdsourcing Rock ‘n Roll

Multimedia Retrie

val

contents

1. AV content exploitation, annotation technology and user needs– NISV context: digitization in Images

of the Future– Annotation technology for enabling

access– Annotation technology and user

needs

2. Example: Crowdsourcing Rock ‘n Roll Multimedia Retrieval

NISV context

• +700.000 hours of radio, television, documentaries, films and music, over 2 million photographs, 20.000 objects like cameras, televisions, radios, costumes and pieces of scenery

• and growing:• digitally born television and radio programs

made by the Dutch public broadcasting companies (video: ~15K/hours/year)

• PROARCHIVE: archiving service• selection of (Dutch) AV content from the web

IMAGES OF THE FUTURELARGE DIGITIZATION PROGRAM

Images of the Future

• Selection, restoration, digitization, encoding and storage of 137,000 hours of video, 20,000 hours of film, 124,000 hours of audio and more than three million photographs.

• Three goals:• Safeguarding heritage for future generations• Creating social- economical value (“unlock the

social and economic potential of the collections”)

• Innovation: new infrastructure for strengthening knowledge economy

INVESTMENTS

BUSINESS MODELS

The cultural heritage sector is challenged

to re-evaluate its business models

Business model

• The total investment of this initiative sums up to 173 million Euro• A strong business model is necessary to support this kind of

investment and prove that such an investment will result in long-term socio-economic returns

• The outcome of a Cost-Benefit analysis was positive: “The total balance of costs and returns of restoring, preserving and digitising audio-visual material (excluding costs of tax payments) will be between: 20+ and 60+ million.’’

• Economic benefits:• Direct effects of the investment are revenues from sales,

access for specific user groups, the repartition of copyright for the use of the material and so on.

• The indirect effects concern the product markets and labour market.

• Social benefits:• conservation of culture, reinforcement of cultural awareness,

reinforcement of democracy through the accessibility of information, increase in multimedia literacy and contribution to the Lisbon goals set by the EU

http://www.prestoprime.org/project/public.en.html

Content exploitation: from content is king ...

... to metadata rules

MANUAL ANNOTATION

costly & limited

DECADE+ RESEARCH EFFORTS

(SEMI) AUTOMATIC ANNOTATION

Research on automatic annotation

• automatic information extraction based on:• visual features• information from audio

• crowdsourcing• deploying collateral data sources:

• subtitles, production scripts, meeting minutes, slides

PROGRESS? YES!

Various (laboratory) showcases

Commercial systems (e.g., blinkx, google)

work in progress

• institutional: reorganisation of traditional archival workflows

• national: development of common services • OAI, Persistent Identifiers, ASR service,

Vocabulary Repositories• commercial: uptake by MNCs (Google and

Microsoft) and SMEs • individual: bring about a shift regarding

defensive attitude of content owners towards• opening up their funded and protected

archives • use of possibly noisy content

descriptions(trust/reliability)

Automatic annotation: NISV as a user

• Participation in international research projects• Video Active, MultiMATCH, VIDI-video, LiWA, P2P-

Fusion, Sterna, EUscreen, PrestoPrime• Collaboration agreement with Dutch research

institutes• Researchers stationed at Sound and Vision• Provide data (TRECVID, VideoCLEF)

• Research environment: exact copy of iMMix production environment for testing new technology• speech recognition• video analysis• fingerprinting• linking of context data (web, program guide,

production data)

DISPARITY BETWEEN TECHNOLOGY AND USER NEEDS

media professionals

journalists

researchers

educators

general public

User perspective

• Rapidly evolving networked information society• Opening up• Focus on community specific

requirements• search needs• presentation/interaction needs

• Draw communities into libraries

COMMUNITY SPECIFIC REQUIREMENTS

From document level search to fragment level search

19

Broadcast professionals

In: Huurnink, Hollink, van Den Heuvel 2009 (submitted)

User survey (broadcast professionals)

Researchers

• Verteld Verleden aims at establishing a shared information space on distributed Dutch Oral History collections:• distributed collections (harvested via OAI)• search & interlink collections via centralized search

• project goals:

1. provide demonstrator portal to show how technology could help researchers

2. acquire information on specific user requirements • search• collaboration• linking• privacy• dedicated work space

http://www.verteldverleden.org

DRAW COMMUNITIES INTO LIBRARIES

Goals

• exploiting community tagging (tagging games, etc)

• exploring the wisdom of crowds by hooking up with user communities (e.g., everyone-as-commentator, unexpected experts)

• capturing relevant information from the internet and aligning this with archived items.

• finding new ways for communities to interact with the data.

Technology perspective

Technology:• provide anchor points for linking up with the

`cloud’ (entity detection, segmentation, cross-collection SID, etc): people, places, events, topics, quotes, etc.

• synchronization of web-content/UGC with AV documents

• users in the loop: UGC for adapting/training analysis tools

• technology aided annotation: Documentalist Support System• provide documentalist/archivist with

relevant context during manual annotation

WEB-ARCHIVINGCOLLECT CONTEXT DATA FROM THE WEB

Web-archiving

• extend Sound and Vision archive with audiovisual content from the internet

• archive internet web content • preserve broadcast related

websites • to use as context information for

audiovisual data in the Sound and Vision archive

AUDIOVISUAL INTERNET CONTENT

BROADCAST RELATED INTERNET CONTENT

iMMixAV ARCHIVE

WEB-ARCHIVE

CONTEXTCONTEXT

Special Use Case: documentalist support

• in the process of generating metadata for an archived AV item, a documentalist searches for relevant information on this item, for example on the internet

• internet search might fail as such information is typically available only for a limited amount of time

• the “internet archive” works as a “contextdatabase” for relevant internet context

TAGGING GAMEwww.waisda.nl

CROWDSOURCING ROCK N’ ROLL MULTIMEDIA RETRIEVAL

Netherlands Institute for Sound and Vision

University of Amsterdam – Visual Search (Cees Snoek)

University of Twente – Speech Recognition (Franciska de Jong)

VideoDock – User Interface (Bauke Freiburg)

Background

• 40th birthday of popular annual Dutch rock festival Pinkpop

• from only summary to almost unabridged recordings, even including raw, unpublished footage as well as interviews

• collection digitized in Images for the Future• goal: build an application for showcasing

history of the festival in an attractive way using state-of-the-art technology

Rationale

• Use state-of-the-art visual analysis to allow browsing collection on the basis of visual concert concepts

• Use speech recognition for browsing interviews

• Exploit popularity of festival to get rock ‘n roll enthusiasts community into the loop:• general feedback on technology• improve and extend automatic labeling• share video fragment

IPR

• Various Dutch broadcasters hold the copyrights of the content.

• Granted dispensation to use content to enable a large scale study of community-aided annotation and verification via an open internet platform• for a limited time period of three months,• video displayed in a secured player• (access to experimental results)

Visual search

• visual concept detection: for each concept a ‘detector’ is trained on the basis of manually labeled training data.

• number of concepts in concerts more or less fixed (in contrast to BN domain), 12 were choosen based on:• frequency • visual detection feasibility• previous mentioning in literature• expected utility for users

• for each concept several hundred examples were labeled

Fragment level concept detection

• video fragments instead of more technically defined shots or keyframes

• fragment algorithm finds the longest fragments with the highest average scores for a specific concert concept

• Only the top-n fragments per concert concept areloaded in the video player

Speech Recognition

• Speech transcripts generated by open-source speech recognition toolkit SHoUT developed in MultimediaN and CATCH projects

• Words in transcripts have time-labels• Transcripts converted to filtered term

frequency list on the basis of tf.idf statistics for generating a time-synchronized term cloud:• jump to relevant interview parts via terms

Player

• timeline-based videoplayer• colored dots represent concepts • clicking dot starts playback• feedback window:

• right/wrong label • comment• share (email/twitter)

• embed integrated video player,• including crowdsourcing mechanism

Encouraging User Feedback

• balance between appealing user experience and maximized user participation

• full-length concert videos (no ‘commercials’)• no interruptions, only graphical overlays• participation threshold kept low:

• no signing up• just click buttons (thumps-up/down)

• all user feedback with IP adresses and user sessions stored in database

DEMO

Preliminary results

• 12,563 visitors of which 9,595 unique in 3 months. • visitors watched on average 3.5 pages, with an

average viewing time of 4,57 minutes. • busiest day was December 3, with 1,566 visitors,

immediately after launch and media attention • Most traffic (65%) originated from 255 referrer

sites. The best referrer sites being: • pinkpop.nl (festival organization)• oor.nl (music magazine)• google

• users provided feedback more than 4000 times.• We are currently investigating how this

feedback can be exploited to improve automated multimedia analysis results

Wrap up

• value of archive is strongly related to access opportunities

• access is to a large extend technology driven• but next to technology development we need

to make a shift:• from a ‘laboratory view’ on users to drawing

users and communities into the loop• NISV is aiming towards this two-way strategy:

• incorporate advanced access technology• discuss access requirements with the

stakeholders