Salzburg Research Forschungsgesellschaft m.b.H.| Jakob-Haringer-Str. 5/III | A-5020 Salzburg T...

Salzburg Research Forschungsgesellschaft m.b.H.| Jakob-Haringer-Str. 5/III | A-5020 Salzburg T +43.662.2288-200 | F +43.662.2288-222 | [email protected] | www.salzburgresearch.at

Towards a Semantic Turn in Rich-Media Analysis

11th International Conferenceon Electronic Publishing

Vienna, Austria - June 15th, 2007

Georg Güntner, Tobias Bürger{georg.guentner,tobias.buerger} @salzburgresearch.at

Version 1.0 (June 14th, 2007)© Salzburg Research, 2007

15.06.2007 2 © Salzburg Research, 2007

Overview

| Motivation for Leveraging Media Semantics| Knowledge Models and Media Semantics| Case study 1: Automatic Semantic Analysis

Smart Content Factory| Case study 2: Semi-automatic Semantic Analysis

LIVE – Live staging of media events| Discussion of Approaches| Conclusions and Recommendations


Motivation for Leveraging Media Semantics

| The market perspective:| Corporate, community based & personal audiovisual

collections (archives, libraries): spanning from YouTube to the big “National Libraries”

| Media producers: broadcasters, film producers, game producers

| Learning applications, industrial applications| „Embedded publishers“| Media analysts

| The „prosumer crisis“ as a driving factor:| Consumers become producers| Content production gets easier: The amount of

manageable assets explodes.| Professional content production is expensive: re-use is

mandatory| Solutions of the prosumer crisis depend heavily on

improvements of the search and retrieval process


Musical forms(z.B. musical genres andforms)

Musical facts(Instances: e.g. works, composers, titles)

Musical Terminology(e.g. synonyms, classification)

Geographic names(places, regions, districts,countries, areas)

Domain Ontologies – an Example


The Dynamont 3D-Matrix

ModelScope

Level of Expressiveness

ModelAcceptance

● which parts of semantics are modeled?● how is the perspective of the ontology

onto the knowledge of the users?

● what kind of semantics is used?● what kind of semantics are required to

fulfill requirements?

● what are the user communities using the ontology?

● which communities accept the ontology? DynamOntAustrian research projectin FIT-IT Semantic Systems(2005 – 2007)


Knowledge Models and Media Semantics

| Glossary, controlled vocabulary

| Finite number of terms, uniquely defined

| Taxonomy| Controlled vocabulary,

hierarchically structured

| Thesaurus| Taxonomy, enhanced by

associations, e.g. “A is_related_to B”

| Database schema| Typed associations,

schema based on a data model

| Formal Ontology| A (meta) model, describing

possible and true predicates which are assured by axioms

© Alistar J. Miles. SKOS Core Guidelines for Migration. http://www.w3.org/


Approaches to Bridge the “Semantic Gap”

| Construction of meaning| Is an act of interpretation based on pre-existing knowledge

(the “context”)| Can usually not be extracted by low-level feature analysis Semantic gap

Solutions: Creation of training data sets Analysis of usage context Automatic, semi-automatic and manual annotation Combinations of above annotation approaches approaches Social Web approaches


Smart Tracks

Time

Metadata Tracks:

Watch

Listen

Read

Face ID

Speaker ID

User Annotation

Speech to Text

Timecode

Keyframes

Closed Caption

Lynne Russell Hillary Clinton

“Clinton spoke with reporters” “Arrived in New York”

“Older citizens who have”

Improving Medicare

Speaking at rally

Clinton running for senate

Hillary ClintonLynne Russell

00:08:40:0000:08:26:12 00:08:34:29

Encoded Video

>>Hillary Clinton Spoke with reporters when she arrived in New York about her proposal for older citizens...

Audio ClassificationSpeech Applause Speech

On-Screen TextSpecial Report Medicare Issues

© Virage


Location Based Navigation (using Google Maps™)


Browsing by Category


Modelling the LIVE Staging Domain

Event

• Human annotators

• Editor

• Video conductor

• Automatic annotators

• Contentformats

• Stream profiles

• Stream interrelation

• Audience profile

IntelligentContentModel

ProductionArchive

ProductionArchive

• Consumer

User

Staging

Content


LIVE System Components Related to Media Semantics


Knowledge(ontologies)

Media Asset(metadata)

Essence(videoclip)

Intelligent Content Models for Leveraging Media Semantics

(1) Resource Level (2) Meta Level (3) Subject Matter Level

Model


Discussion of Approaches (1/2)

| Enrichment of multimedia data| Analysis of usage context| Analysis of different modalities| Human annotation| Use of Web2.0 approaches

| Combination of low-level feature extraction with background knowledge| Rich News combines text extraction with Google searches| Google extracts topics from the subtitle channel and links to

related web pages| In intranets: combination with production metadata


Discussion of Approaches (2/2)

| Using different modalities| MediaMill: e.g. when visual content is not reflected in closed

caption or in the audio track| MISTRAL: “Measurable intelligent and secure semantic

extraction and retrieval of multimedia data“; Austrian FIT-IT project

| Combination of media asset management approaches with document management technologies

| Virage, Autonomy, Convera, etc.| Google, Yahoo, etc.

| Combination of media asset management approaches with social software approaches| Web 2.0 platforms (some of them recently acquired by the big

players in the media community)


A Recipe for the Semantic Turn in Rich-Media Analysis

| Use existing content (URIs)| Define an appropriate content model

| Is the content invariant to the semantic approach? Must the structures change?

| Integrate existing information sources| Production metadata (speakers, editing), archival

data, playout systems, EPGs | Consider creating links to other sources

| Test metadata extraction tools| Consider metadata annotation

| Is social tagging an alternative?| Define an appropriate domain knowledge model

| Scope, expressiveness, acceptance| Implement the semantic indexing process| Integrate the system in the production /

archiving workflow


Conclusions & Recommendations

| Define the scope of your application!| Create persistent URIs (J. Hendler)| Include reliable information sources!| Leave data in place – rather integrate through

an RDF store! (J. Hendler, Semantics 2006)| Use intelligent content models support

semantic search!| Consider social software approaches where

applicable!| Train your staff!| Define the role of the archivist!| „Install“ a knowledge engineer!


Thank you for your Attention!

| Additional information | Salzburg Research: www.salzburgresearch.at| Salzburg NewMediaLab: www.newmedialab.at| Smart Content Factory: scf.salzburgresearch.at| IST-Project LIVE: www.ist-live.org

| Contacts| Güntner Georg

T: +43.662.2288.401, M: [email protected] ResearchProject manager of Smart Content FactoryScientific and technological coordinator of LIVE

Salzburg ResearchJakob Haringer Straße 5/III

A-5020 SalzburgAustria/Österreich

Tel. +43.662.2288.400Fax: +43.662.2288.222

Salzburg Research Forschungsgesellschaft m.b.H.| Jakob-Haringer-Str. 5/III | A-5020 Salzburg T...

Documents

Transcript of Salzburg Research Forschungsgesellschaft m.b.H.| Jakob-Haringer-Str. 5/III | A-5020 Salzburg T...