FIAT-IFTA 2013 - Television linked to the web: the case for audiovisual archives
-
Upload
johan-oomen -
Category
Documents
-
view
2.731 -
download
3
Transcript of FIAT-IFTA 2013 - Television linked to the web: the case for audiovisual archives
www.linkedtv.eu
Johan Oomen and Lotte Belice Baltussen@johanoomen and @lottebeliceNetherlands Institute for Sound and Vision
Television linked to the web: the case for audiovisual archives
FIAT/IFTA | Dubai, United Arab Emirates – 27 October 2013
www.linkedtv.eu
2
multimediacontentanalysis
new distributionplatforms
lots of archive content
www.linkedtv.eu
2
multimediacontentanalysis
new distributionplatforms
lots of archive content
where the magic happens!
www.linkedtv.eu
Where the magic happens
3
multimediacontentanalysis
new distributionplatforms
lots of archive content
Audiovisual broadcasts
Audiovisual broadcasts
Audiovisual broadcasts
Professional annotation
Audiovisual broadcasts
Professional annotation
Search engine
Audiovisual broadcasts
Professional annotation
Search engine
Television makers
Audiovisual broadcasts
Professional annotation
Search engine
Television makers
Academics
Audiovisual broadcasts
Professional annotation
Search engine
Television makers
General public
Academics
General public
Academics
Audiovisual broadcasts
Professional annotation
Search engine
Television makers
Machine analysis Data gathering
WikisTwitter
Subtitles
Importer
NewsCrowdsourcing
multimedia content analysis
Data gathering
Data Gathering:Pros and Cons
Data Gathering:Pros and Cons
• Large-scale data for some programs
• Access to opinions about television
• Puts programs in larger national context
• Affordable
• Noisy data (sometimes)
• Not created for the archive
• Unclear how the data can be used (sometimes)
multimedia content analysis
Machine Analysis
Low-level features
Multimediacontent analysis
multimedia content analysis
Machine Analysis
Low-level features
Multimediacontent analysis
today sniper fire disrupted the funeral of an eleven year old ethnic albanian boy he was killed yesterday while while chopping wood his family blames serb police before his death louisiana state police now say six workers were killed after a natural gas well exploded and caught fire about forty five miles east of shreveport four others were injured in yesterday's blast a police spokesman says the derek started to melt in the intense heat the
Audio transcripts Concept detectors
Speaker recognition Face recognition
Machine Analysis: Pros and Cons
Machine Analysis: Pros and Cons
• Extremely detailed, millisecond-level descriptions
• Directly describe video content
• Affordable
• Large scale!
• Noisy
• Limited to what the machine “knows”
• No summarisation of what is important
• Confusing for users
www.linkedtv.eu
10
Media fragments
linkedtv.eu
www.linkedtv.eu
Where the magic happens
11
multimediacontentanalysis
new distributionplatforms
lots of archive content
www.linkedtv.eu
12
Context guestimates
linkedtv.eu
thousands of hrs of new television content produced byDutch public broadcasters every year
Hours of archival content of Sound and Vision used in this new content
www.linkedtv.eu
How do broadcast professionals use archive content?§ Query log analysis.* (50k user
sessions, 264k queries)
13
§ 33% programs§ 17% stories, and fragments§ 50% audiovisual fragments with a start and end time
specified by the users themselves.
* Paper: Search Behavior of Media Professionals at an Audiovisual Archive: ATransaction Log Analysis. By Bouke Huurnink
0 500
1000 1500 2000 2500 3000 3500 4000
0-11-22-33-44-55-66-77-88-99-1010-1111-1212-1313-1414-1515-1616-1717-1818-1919-2020-2121-2222-2323-2424-2525-2626-2727-2828-2929-30>30
Num
ber o
f ord
ers
Order duration in minutes
broadcast orderstory order
fragment order
www.linkedtv.eu
Preference for new material
§ 46 % of the orders are for footage broadcast more than 1 year before the order date
§ 12% of orders are for material that is over 10 years old.
§ => a lot of high quality content is not used.
14
www.linkedtv.eu
15
Context guestimates
linkedtv.eu
thousands of hrs of new television content produced byDutch public broadcasters every year
Hours of archival content of Sound and Vision used in this new content
www.linkedtv.eu
16
Context – the potential
linkedtv.eu
Total av-collection Sound and Vision750k of hours (& growing daily)
www.linkedtv.eu
17
Context – the potential: on-demand personalised services
linkedtv.eu
Total av-collection Sound and Vision750k hours (& growing daily)
www.linkedtv.eu
Where the magic happens
18
multimediacontentanalysis
new distributionplatforms
lots of archive content
www.linkedtv.eu
19
§ 86% of TV viewers surf the Web at the same time (Yahoo!, USA)
§ 88% of USA consumers use mobile as second screen (Business Insider, May 2013)
§ Manual efforts to link TV program to additional content (Shazam, IntoNow)
Web and TV exist in parallel
linkedtv.eu
www.linkedtv.eu
Samsung advert
20 http://www.youtube.com/watch?v=UwmoBmlkaGQ
www.linkedtv.eu
21
Examples – BBC Interactive Video Player
linkedtv.eu
www.linkedtv.eu
22
Examples – Hulu Face Match
linkedtv.eu
www.linkedtv.eu
23
LinkedTV: interweaving Web and TVinto a single experience
The LinkedTV vision
linkedtv.eu
§ Mixing lean back and lean forward§ how to maintain immersion while
removing fourth wall illusion?§ will this work across genres and
audiences?§ Does second screen help or hinder?
www.linkedtv.eu
24 linkedtv.eu
Research Organisations
SME Technology Providers
User Interface and Scenario Partners
Technical coordinator Market Studies
www.linkedtv.eu
25
LinkedTV – Use cases
Interactive News
§ Professional news content produced by RBB for general audience
§ local news show rbb Aktuell§ People, locations and events
can be linked to previous events.
Hyperlinked Documentary: Antiques Interactive
§ Program watched predominantly by 50+ers
§ AVRO:Tussen Kunst en Kitsch (TKK)
§ Objects, locations and experts provide anchors to related materials (targets).
www.linkedtv.eu
26
Use case – Antiques InteractiveBert and Anne
“I like to relax and sit back when I watch Antiques Road Show. It’s
always exciting when something’s worth a lot. In some cases, I like to
look up extra info about an object or art style after the show on the PC,
but if something doesn’t interest me, I’ll just read a magazine.”
“As an antiques shop owner, by work is also my hobby. I’m always looking up
information about art objects on my PC or tablet. I like watching my colleagues
talking about are on Antiques Road Show and I use my tablet to bookmark specific objects or other information that I like. I
also do this for Anne, and send her overviews.”
www.linkedtv.eu
27
How LinkedTV helps Bert and Anne
linkedtv.eu
Video Object and word detection
Presentation engine
Selecton of relatedconcepts
Selection of relatedcontent
Connection to concepts
www.linkedtv.eu
27
How LinkedTV helps Bert and Anne
linkedtv.eu
“...painting of Jan Sluijters....”
Video Object and word detection
Presentation engine
Selecton of relatedconcepts
Selection of relatedcontent
Connection to concepts
www.linkedtv.eu
27
dbpedia.org/resource/Jan_Sluyters
How LinkedTV helps Bert and Anne
linkedtv.eu
“...painting of Jan Sluijters....”
Video Object and word detection
Presentation engine
Selecton of relatedconcepts
Selection of relatedcontent
Connection to concepts
www.linkedtv.eu
27
dbpedia.org/resource/Jan_Sluyters
How LinkedTV helps Bert and Anne
linkedtv.eu
Paintings by Jan Sluijters
“...painting of Jan Sluijters....”
Video Object and word detection
Presentation engine
Selecton of relatedconcepts
Selection of relatedcontent
Connection to concepts
www.linkedtv.eu
27
dbpedia.org/resource/Jan_Sluyters
How LinkedTV helps Bert and Anne
linkedtv.eu
Paintings by Jan Sluijters
“...painting of Jan Sluijters....”
Video Object and word detection
Presentation engine
Selecton of relatedconcepts
Selection of relatedcontent
Connection to concepts
www.linkedtv.eu
27
dbpedia.org/resource/Jan_Sluyters
How LinkedTV helps Bert and Anne
linkedtv.eu
Paintings by Jan Sluijters
“...painting of Jan Sluijters....”
Video Object and word detection
Presentation engine
Selecton of relatedconcepts
Selection of relatedcontent
Connection to concepts
www.linkedtv.eu
28
From one screen...
http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2
www.linkedtv.eu
28
From one screen...
http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2
www.linkedtv.eu
28
From one screen...
http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2
www.linkedtv.eu
§
29
www.linkedtv.eu
30
...to paper prototype...
linkedtv.eu
www.linkedtv.eu
31
www.linkedtv.eu
32
...to second screen demo
linkedtv.eu
www.linkedtv.eu
33
dbpedia.org/resource/Jan_Sluyters
How do we give broadcasters some control?
linkedtv.eu
Paintings by Jan Sluijters
“...painting of Jan Sluijters....”
Hypervideo analysis
Presentation engine
Named Entity Recognition
Enriching NEs with web content
Generation of media fragments
Editor tool
www.linkedtv.eu
34
[Useful tool example] editor tool
, Roeland Ordelman1
§ Broadcasters want to have some control over noisy data.§ Editors have the expertise to determine relevance +§ The user interface can only show a limited amount of links, and there are many
§ Research: sample 805 videos, average 55 NEs per video. 55 NEs = many, many possible links depending on the size of the white list [Li et al., 2013]
NE - person: Jan Sluijters
NE – art syle: Fauvism
NE - person: Emiel Aardewerk
Link white list 1
Link white list 2
Link white list 3
Link white list 6
Link white list 7
Link white list etc etc
Link white list 4
Link white list 5
www.linkedtv.eu
35
Editor tool overview – selecting entities and enrichments
www.linkedtv.eu
36
Editor tool overview – selecting entities and enrichments
www.linkedtv.eu
37
Editor tool overview – editor‘s choices
www.linkedtv.eu
38
The project ends in May 2015. The coming year, we’ll focus on:
•Improving noisy data from automatic enrichments• eg. through aggregating multiple entities and visual cues
•Multiple user studies and iterative development of front-end functionalities and interface based on feedback
•Making enrichments more relevant for users by further refining personalisation technologies
•Further Market studies and business model development
Future work in LinkedTV
www.linkedtv.eu
Online resources!
39
www.linkedtv.eu
40
Do stuff now! Media Fragment Enricher
, Roeland Ordelman1
Eurecom’s tool: http://mfe.synote.org/mfe/
www.linkedtv.eu
Lets explore!
41
multimediacontentanalysis
new distributionplatforms
lots of archive content
www.linkedtv.eu
Lets explore!
41
multimediacontentanalysis
new distributionplatforms
lots of archive content
where the magic happens!
www.linkedtv.eu
42
Thanks!
, Roeland Ordelman1
Find me here:
@johanoomen| [email protected] @lottebelice [email protected]
Credits:§All LinkedTV partners§Slide hat-tip to: Lynda Hardman (CWI), Lyndon Nixon (MODUL), Raphaël Troncy (Eurecom), Bouke Huurnink (Sound and Vision)
§ LinkedTV overview: From SmartTV to LinkedTV. slideshare.net/linkedtv/linked-tv-introduction§ The importance of Linked Media to the Future Web. slideshare.net/linkedtv/www-linked-media-
keynote§ Describing media assets: media fragment specification and description. slideshare.net/troncy/
describing-media-assets-media-fragment-specification-and-description §Heaps of thanks to AVRO for so generously letting LinkedTV use their content.
Links:§www.linkedtv.eu. See demo’s, tools, research, scenario’s and more!§twitter.com/BenGlabs §www.avro.nl