Combining Multimedia and Semantics (LACNEM2010)
-
Upload
oscar-corcho -
Category
Technology
-
view
1.684 -
download
0
description
Transcript of Combining Multimedia and Semantics (LACNEM2010)
Combining Multimedia and Semantics
Oscar Corcho ([email protected])
Universidad Politécnica de Madrid
http://www.oeg-upm.net/
LACNEM 2010, Cali, ColombiaSeptember 9th 2010
Credits: Adrián Siles, Mariano Rico, Víctor Méndez, Hector Andrés García-Silva, María del Carmen Suárez-Figueroa, Ghislain Atemezing, Raphaël Troncy
Work distributed under the license Creative Commons Attribution-Noncommercial-Share Alike 3.0
http://www.slideshare.net/ocorcho
2Asunción Gómez Pérez
Ontology Engineering Group. Whom we are
•Director: A. Gómez-Pérez•Research Group (37 people)
- 2 Full Professor- 4 Associate Professors - 1 Assistant Professor- 3 Postdocs- 17 PhD Students- 8 MSc Students- 2 Software Engineers
• Management (4 people)- 2 Project Managers- 1 System Administrator- 1 Secretary
• 50+ Past Collaborators• 10+ visitors
Semantic e-Science (Data Integration, Semantic Grid)
Internet of Things
(Social) Semantic
Web
Natural Language Processing
Ontological Engineering
Research Areas
1995
19972000
2004 2008
4
Before we start…
• How many of you have ever heard about the word “Ontology”?
• And how many of you do actually know what it means?
5
Coming to terms with ontologies and semantics
• An ontology is an engineering artifact, which provides: - A vocabulary of terms- A set of explicit assumptions regarding the intended meaning of the
vocabulary. • Almost always including concepts and their classification• Almost always including properties between concepts
• Shared understanding of a domain of interest - Agreement on the meaning of terms- Formal and machine manipulable model of a domain of interest
• Besides...- The meaning (semantics) of such terms is formally specified- New terms can be formed by combining existing ones- Can also specify relationships between terms in multiple ontologies
6
Example: An ontology about satellites
7
Outline
• Introduction- What I will be talking about and what I will not…
There were several options that I explored before selecting the one that you will be hearing in this talk…
8
Option 1: The Semantic Gap
• The lack of coincidence between the information that one can extract from the sensory data and the interpretation that the same data has for a user in a given situation
However, I already assumed that Ebroul would be talking a lot about it in his opening keynote (as he did).
Besides, I have not worked at all on the low-level part, so it may be difficult for me to provide you with a good insight on the (many) open problems in this area
A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain: Content-based image retrieval at the end of the early years, IEEE PAMI, 1349–1380, 2000.
9
Option 2: MPEG-7 and the Semantic Web
• ISO standard since December 2001• Main components:
- Descriptors (Ds) and Description Schemes (DSs)- DDL (XML Schema + extensions)
• Concern all types of media• A good number of ontologies developed around it
Basic datatypes
Links & media localization
Basic Tools
Models
Basic elements
Navigation & Access
Content management
Content description
Collections
Summaries
Variations
Content organization
Creation & Production
Media Usage
Semantic aspects
Structural aspects
User interaction
User Preferences
Schema Tools
User History Views Views
Part 5 – MDSMultimedia Description Schemes
10
Option 2: MPEG-7 and the Semantic Web
However, the talk may:
Be a bit boring and too technical
May lack the mix of state of the art and vision that an invited talk should normally have
And MPEG-7 is not used too much
…So I will cover only some aspects of this later, when I talk about multimedia ontologies.
11
Option 3: Canonical Processes of Media Production (and semantics, obviously)
• For example….- http://www.cewe-photobook.
com
• Application for authoring digital photo books
• Automatic selection, sorting and ordering of photos- Context analysis methods:
timestamp, annotation, etc.- Content analysis methods:
color histograms, edge detection, etc.
• Customized layout and background
• Print by the European leader photo finisher company
Credits: Raphaël Troncy, Lynda Hardman
CeWe Color PhotoBook Processes
• My winter ski holidays with my friendsPremeditate
Construct Message
Credits: Raphaël Troncy, Lynda Hardman
CeWe Color PhotoBook Processes
•
Create
Package
Credits: Raphaël Troncy, Lynda Hardman
CeWe Color PhotoBook Processes
•
Annotate
Credits: Raphaël Troncy, Lynda Hardman
CeWe Color PhotoBook Processes
•
Organize
Query
Credits: Raphaël Troncy, Lynda Hardman
CeWe Color PhotoBook Processes
• Publish
Distribute
Credits: Raphaël Troncy, Lynda Hardman
CeWe Color PhotoBook Processes
Credits: Raphaël Troncy, Lynda Hardman
Semantics can be important in the process
Credits: Raphaël Troncy, Lynda Hardman
19
Option 3: Canonical Processes of Media Production
However, some of you probably attended Raphaël Troncy’s talk last year (available in slideshare)
20
In summary…
• I decided to talk about something that I have been working in for the last couple of years, and which combines- Semantics (of course, this is the key expertise of our group)
• Mainly annotation, Linked Data and a bit of Multimedia Ontology Engineering
- Social networks, collaboration, sharing and collective intelligence• Exploiting home networks and online multimedia sites
- And, obviously, multimedia
And hence I still leave out manyinteresting topics (e.g., semantics in user interfaces)
21
Outline
• Introduction- What I will be talking about and what I will not
• Sem-UPnP-Grid- Sharing multimedia content across homes through semantic
annotations• Credits: Mariano Rico and Adrián Siles (UPM), Víctor
Méndez and José Manuel Gómez-Pérez (iSOCO), José Manuel Palacios and Mónica Pérez (TID)
• Sem4Tags- Tag disambiguation in Flickr
• M3 Ontology (only if time permits)- A semantic backbone for our multimedia-related work
• Conclusions and outlook
22
Internet
Motivation
• Multimedia resources in Web2.0 are stored in centralised servers.
• You lose some of your rights as an author when you upload these resources to these servers.
• Privacy problems.• Poor annotations and metadata.• These resources cannot be shared with other resources in
your home.
UpGrid
23
Multimedia Content Sharing with UpGrid
P2P
Annotation:“Ángel on the beach”
Annotation:“Ángel playing soccer”
Additional semantic information:- “Ángel is my son”
- “Pedro is my brother”
Pedro
Semantic-based query:“multimedia content related to my
nephew”
Additional semantic information:“Juan is my brother”
JuanReasoning:
- “Ángel is my son”- “Pedro is my brother”- “Juan is my brother”
- -------------------------------- Ángel is my nephew
Architecture
Architecture (another view on it)
Snapshots from the application
Check http://www.youtube.com/results?search_query=UPnPGrid
27
Summary
• An effective means for sharing multimedia contents across homes, avoiding Web2.0 sites where your rights may be compromised
• However, it is still a prototype, and no serious usability testing has been done- Much work still needed in order to go into a real system
• And end users find it difficult to provide annotations - Do you imagine your parents and grandparents annotating
photos and videos like that?- Let’s see how this could be ameliorated with the next part of
our presentation.
28
Outline
• Introduction- What I will be talking about and what I will not
• Sem-UPnP-Grid- Sharing multimedia content across homes through semantic
annotations
• Sem4Tags- Tag disambiguation in Flickr
• Credits: Héctor Andrés García Silva
• M3 Ontology (only if time permits)- A semantic backbone for our multimedia-related work
• Conclusions and outlook
Egresado de laUniversidad del Valle
• Social Tagging Systems- Web 2.0 applications - Applications for storing, sharing, and
discovering information resources.- Users assign tags to identify
information resources- Tags are used to search/discover
resources
29
Introduction
• Folksonomy- Emerging classification scheme from
social tagging systems - Folk: People, Taxonomy: Classification- Represented by: Users, Tags, Resources
30
Introduction
Taxonomy
• Top-down• Controlled Vocabulary• Hierarchical structure• Exclusive/Restrictive• Expensive to maintain
Folksonomy
• Bottom-up (user created)• No fixed vocabulary
• No Hierarchical structure• No Exclusive/Flexible• Low cost
Introduction
• Why is tagging so popular?- Reduce cognitive burdens
• it’s easy to use• Users don´t need any special skill or experience
- The benefits of tagging are immediate• Future retrieval• Contribution and sharing• Attract Attention• Self Presentation• Opinion Expression
31
• However- Tags can be ambiguous
• Polysemy: party as a celebration as opposed to party as a political organization
- Synonym: party and celebration
- Morphological variations:
party, parties, partying, partyign • Plurals• Acronyms• Conjugated verbs• Misspelling
- Compound words• Political party, PoliticalParty, Political_party,
Political-Party, etc.
- Detail/granularity level
A general tag as party in contrast to a specific tag as banquet.
32
Introduction
The problem: Morphological variations, synonyms, granularity, and polysemy hamper information retrieval processes based on folksonomies.
33
Motivation
710.659 results8.661.581 Results
Systems ignore resources tagged with morphological variations or synonyms of that tag, as well as the resources tagged with more generic or more specific tags
When searching with polysemous tags, all the resources tagged with that tag are retrieved without taking into account the tag sense the user was looking for.
(e.g., Query flickr with bank results in photos about financial institutions, river edges, fog banks, and sand banks, etc. )
34
Motivation
Motivation
• What if we associate tags with semantic entities?
35
uk, tories, party, conservative, speech party, balloons, colors, bar, crowd
http://morpheus.cs.umbc.edu/aks1/ontosem.owl
#political-party
#political-entity
#organization
#Coalition#federation
#party
#special-occasion
#non-work-activity
#Celebration
#Birthday #Anniversary
We can avoid the
aforementioned pitfalls
State of the Art: Semantic Grounding of Cross-Lingual Folksonomies
36
None of the analyzed approaches deals with multilingual tags
Garcia HA, Corcho O, Alani H, Gómez-Pérez A. Review of the state of the art: Discovering and Associating Semantics to Folksonomies. Knowledge Engineering Review (in press)
Semantic Grounding of Cross-Lingual Folksonomies
• MSR: a Multilingual Sense Repository based on Wikipedia and enriched with semantic information taken from DBpedia.
37
Banco
Banco
Cardumen
Banco de Arena
Sandbank
Swarm
BankTerms and frequency
Terms and frequency
Terms and frequency
http://dbpedia.org/resource/Bank
http://dbpedia.org/resource/Swarm
http://dbpedia.org/resource/SandBank
38
Semantic Grounding of Cross-Lingual Folksonomies
• Sem4Tags: A process for Associating Semantics to Tags.
Bank
EuropeEuro
FinanceCentral bankawesomePic
Nikon ..
http://dbpedia.org/resource/Bank Banco
Dinero,Calle,
Santander,Money,Madrid,Atm, cajero
39
Semantic Grounding of Cross-Lingual Folksonomies
• Disambiguation activity- The candidate senses and the tag context are represented
as vectors. • The vector components are the set of most frequent
terms in each Wikipedia page representing a sense.• For each sense the values of the vector are calculated
using TF-IDF.• For the tag context the values in each position are 1 or 0
if the corresponding term appears in the tag context.
- The tag context vector is compared against each sense vector using the cosine of the angle as similarity measure.
- The most similar sense to the tag context is selected as the one representing the meaning of the analyzed tag
39
40
Semantic Grounding of Cross-Lingual Folksonomies
• Disambiguation activity- We use the information of the wikipedia default sense for a term. - Sim(TagContext, Sensei)= λ*Cosine + β*defaultSense- We experimentally defined β = 0,2 and λ = 0.8
- We attempt to use DBpedia semantic information in the disambiguation activity:
• Sim(TagContext, Sensei)= λ*Cosine + β*defaultSense + δ*SemanticInfo
• Studies have shown that tags in flickr refers mainly to: Locations, Time, Given Names, Potography related subjects among others.
• We use DBpedia and YAGO relations to classify the senses according to this categories.
• However, we found that not all the senses related to a term have the same amount of relations. (e.g. Madrid is not a city)
41
Let’s try it
• http://robinson.dia.fi.upm.es:8080/SemanticTagsWebApp/index.jsp
• What does “bernabeu” mean if its context is…?- estadio, madrid, fútbol
42
Experiment
• Baseline: Directly associate tags with DBpedia resources- Look for spaces and replace them with ' _‘.- For tags in English:
• Create a URI of the form http://en.wikipedia.org/wiki/tag• Query DBpedia using the http://xmlns.com/foaf/0.1/page
relation- For tags in Spanish:
• Create a URI of the form http://es.wikipedia.org/wiki/tag• Query DBpedia using the
http://dbpedia.org/property/wikipage-es relation
43
Experiment
• Approaches:- Baseline: Selection of the sense without a disambiguation
activity.- Sem4Tags: For each sense we use the whole Wikipedia
article as source for frequent terms.- Sem4TagsAC: Same as Sem4Tags including the selection
of the Active Context.- Sem4TagsAbs: For each sense we use the Wikipedia
article abstract (extracted from DBpedia) as source for frequent terms.
- Sem4TagsAbsAC: Same as Sem4TagsAbs including the selection of the Active Context.
44
Experiment
• Initial Data Set- Wide range of Users, photos, and tags.- 764 photos uploaded by 719 users to Flickr that have been
tagged with tags describing tourist places in Spain- 12.4 (+/- 7.85) tags per photo- 9484 tagging activities (TAS) : <user,photo,tag>- 4135 distinct tags where used
• Processed Data Set- From each photo we processed on average 2 tags - 2260 tagging activities (TAS)
45
Experiment
• Evaluation Campaign- 41 Evaluators- Evaluate semantic associations produce by each approach:
<user; tag; photo; DBpedia resource; language>
- Three different evaluators evaluated each semantic association.
- Questions:• Able to identify the tag meaning (known or Unknown)• Tag language (English, Spanish, Both, other)• The tag correspond to a Named entity• According to the identified tag language they evaluate
the semantic association in terms of• Highly related, Related, Not Related.
46
Experiment
• Results- Evaluators identified the semantics of the 87% of TAS
(known)• 62.6 % of TAS were considered in English• 87.7% of TAS were considered in Spanish
- Agreement among evaluators (Fleiss’ kappa statistics):• k=0.76 for highly related• K=0.71 for the related case/highly related case
47
Experiment
• Precision and Recall for Highly Relevant results
Spanish
English
48
Experiment
• Conclusions- Baseline obtained high precision, however it was able to find
semantic resources for just a fraction of the analyzed data set:• Baseline: 27.7% in English and 19.4% in Spanish.• Sem4Tags: 79.1 % in English and 81.4% in Spanish
- All approaches obtained better precision with named entities than with unnamed entities.
- Sem4Tags and Sem4TagsAC are the approaches that obtained the best results in terms of Precision and Recall. • Sometimes Sem4TagsAC obtains better P@1 values but
the improvements are supported by no or low statistical evidence.
- Sem4TagsAbs and Sem4TagsAbs are clearly the worst approaches.
49
Outline
• Introduction- What I will be talking about and what I will not
• Sem-UPnP-Grid- Sharing multimedia content across homes through semantic
annotations
• Sem4Tags- Tag disambiguation in Flickr
• M3 Ontology (only if time permits)- A semantic backbone for our multimedia-related work
• Conclusions and outlook
50
Ontología M3
OntologíaM3
PerspectivaMultimedia
PerspectivaMultidominio
PerspectivaMultilingüe
There are already multimedia ontologies
• MDS Upper Layer represented in RDFS- 2001: Hunter- Later on: link to the ABC upper ontology
• MDS fully represented in OWL-DL- 2004: Tsinaraki et al., DS-MIRF model
• MPEG-7 fully represented in OWL-DL- 2005: Garcia and Celma, Rhizomik model- Fully automatic translation of the whole standard
• MDS and Visual parts represented in OWL-DL- 2007: Arndt et al., COMM model - Re-engineering MPEG-7 using DOLCE design patterns
• However, their requirements are not always clear nor have they been developed with clear methodological guidelines
52
Knowledge Resources
Non Ontological Resource
Reuse
Non Ontological Resource
Reengineering
2
2
2
Non Ontological Resources
Thesauri
DictionariesGlossaries Lexicons
TaxonomiesClassification
Schemas
O. Localization
9
Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation; Configuration Management; Evaluation (V&V); Assessment
1,2,3,4,5,6,7,8, 9
Ontological Resource
Reengineering
4
4
4
O. Aligning
O. Merging
Alignments5
5
5
6
6
6
6
3
Ontological Resource
Reuse
3Ontological Resources
O. Repositories and Registries
FlogicRDF(S)
OWL
Ontology Design
Pattern Reuse
7
O. Design Patterns
Ontology Restructuring(Pruning, Extension,
Specialization, Modularization)
8
O. Specification O. Conceptualization O. ImplementationO. Formalization
1
RDF(S)
OWL
FlogicScheduling
NeOn Methodology
53
Ontology Requirements Specification (I)
Task 1. Identifying the purpose, scope and implementation
language
Task 2. Identifying the intended end-users
Task 3. Identifying the intended uses
Task 5. Grouping functional requirements
Are they valid?
No
Yes
Task 8. Extracting terminology and its frequency
Users and Domain Experts
Users, Domain Experts and ODT
Task 4. Identifying requirements
Users, Domain Experts and ODT
Task 6. Validating the set of requirements
Task 7. Prioritizing requirements
Users, Domain Experts and ODT
Ontology Development Team
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Users, Domain Experts and ODT
ORSDOUTPUT
Set of ontological
needs
INP
UT
Non-functional ontology requirements: Characteristics not related to the ontology content
54
Ontology Requirements Specification (II)
Task 1. Identifying the purpose, scope and implementation
language
Task 2. Identifying the intended end-users
Task 3. Identifying the intended uses
Task 5. Grouping functional requirements
Are they valid?
No
Yes
Task 8. Extracting terminology and its frequency
Users and Domain Experts
Users, Domain Experts and ODT
Task 4. Identifying requirements
Users, Domain Experts and ODT
Task 6. Validating the set of requirements
Task 7. Prioritizing requirements
Users, Domain Experts and ODT
Ontology Development Team
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Users, Domain Experts and ODT
ORSDOUTPUT
Set of ontological
needs
INP
UT
Functional ontology requirements: Content specific requirements referred to the particular knowledge to be
represented by the ontology Requirements in natural language
in the form of CQs in the form of sentences (General Characteristics)
Strategies: (1) Top-Down, (2) Bottom-Up, and (3) Middle out
55
Ontology Requirements Specification (III): Functional Requirements on M3
Perspectiva Multimedia
Perspectiva Multilenguaje
Perspectiva Multidominio
PCsCompetencias
PCs“Multidominio”
PCsActualidad
PCsDeportes
Proceso deAbstracción
56
Ontology Requirements Specification (IV): ORSD
Perspectiva Multidominio
Perspectiva Multimedia
Perspectiva Multilenguaje
M3
57
Knowledge Resources
Non Ontological Resource
Reuse
Non Ontological Resource
Reengineering
2
2
2
Non Ontological Resources
Thesauri
DictionariesGlossaries Lexicons
TaxonomiesClassification
Schemas
O. Localization
9
Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation; Configuration Management; Evaluation (V&V); Assessment
1,2,3,4,5,6,7,8, 9
Ontological Resource
Reengineering
4
4
4
O. Aligning
O. Merging
Alignments5
5
5
6
6
6
6
3
Ontological Resource
Reuse
3Ontological Resources
O. Repositories and Registries
FlogicRDF(S)
OWL
Ontology Design
Pattern Reuse
7
O. Design Patterns
Ontology Restructuring(Pruning, Extension,
Specialization, Modularization)
8
O. Specification O. Conceptualization O. ImplementationO. Formalization
1
RDF(S)
OWL
FlogicScheduling
NeOn Methodology
58
Scheduling using gOntt
Task 1. Selecting the ontology network life cycle model
Task 2. Selecting the set of scenarios
Users, Domain Experts and ODT
Task 3. Updating initial plan
Task 4. Establishing resource restrictions and assignments
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Scheduling for the
Ontology Network
Development
OUTPUT
ORSD
INP
UT
Initial Ontology Network Life Cycle in the
form of a Gantt chart
OUTPUT
Types of potential knowledge
resources to be reused
Modified Ontology
Network Life Cycle in the
form of a Gantt chart
OUTPUT
I need to schedule the development of the M3
ontolgy network
Task 1. Selecting the ontology network life cycle model
Task 2. Selecting the set of scenarios
Users, Domain Experts and ODT
Task 3. Updating initial plan
Task 4. Establishing resource restrictions and assignments
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Scheduling for the
Ontology Network
Development
OUTPUT
ORSD
INP
UT
Initial Ontology Network Life Cycle in the
form of a Gantt chart
OUTPUT
Types of potential knowledge
resources to be reused
Modified Ontology
Network Life Cycle in the
form of a Gantt chart
OUTPUT
Life cycle model selection
Scenarios selection
59
Scheduling using gOntt (II)
Task 1. Selecting the ontology network life cycle model
Task 2. Selecting the set of scenarios
Users, Domain Experts and ODT
Task 3. Updating initial plan
Task 4. Establishing resource restrictions and assignments
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Users, Domain Experts and ODT
Scheduling for the
Ontology Network
Development
OUTPUT
ORSD
INP
UT
Initial Ontology Network Life Cycle in the
form of a Gantt chart
OUTPUT
Types of potential knowledge
resources to be reused
Modified Ontology
Network Life Cycle in the
form of a Gantt chart
OUTPUT
60
Knowledge Resources
Non Ontological Resource
Reuse
Non Ontological Resource
Reengineering
2
2
2
Non Ontological Resources
Thesauri
DictionariesGlossaries Lexicons
TaxonomiesClassification
Schemas
O. Localization
9
Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation; Configuration Management; Evaluation (V&V); Assessment
1,2,3,4,5,6,7,8, 9
Ontological Resource
Reengineering
4
4
4
O. Aligning
O. Merging
Alignments5
5
5
6
6
6
6
3
Ontological Resource
Reuse
3Ontological Resources
O. Repositories and Registries
FlogicRDF(S)
OWL
Ontology Design
Pattern Reuse
7
O. Design Patterns
Ontology Restructuring(Pruning, Extension,
Specialization, Modularization)
8
O. Specification O. Conceptualization O. ImplementationO. Formalization
1
RDF(S)
OWL
FlogicScheduling
NeOn Methodology
61
Reusing Ontological Resources: Comparative Analysis (I)
62
Reusing Ontological Resources: Comparative Analysis
63
Outline
• Introduction- What I will be talking about and what I will not
• Sem-UPnP-Grid- Sharing multimedia content across homes through semantic
annotations
• Sem4Tags- Tag disambiguation in Flickr
• M3 Ontology (only if time permits)- A semantic backbone for our multimedia-related work
• Conclusions and outlook
64
Conclusions and outlook
• We all agree that…- Multimedia UGC has been one of the basis of Web2.0
• The use of semantics can provide…- Better understanding of the domain and of their content
• Heavyweight: addressing the semantic gap automatically• Ligthweight: allowing users to annotate• Middleweight: from free tags to knowledge
- Better exploratory navigation and serendipity• Interconnecting multimedia content with the Linked Data
cloud
• However, privacy issues are still a major barrier for a larger uptake, especially for some population segments- Allowing P2P exchange between “known” homes, while
exploiting semantic-based search
Combining Multimedia and Semantics
Oscar Corcho ([email protected])
Universidad Politécnica de Madrid
http://www.oeg-upm.net/
LACNEM 2010, Cali, ColombiaSeptember 9th 2010
Credits: Adrián Siles, Mariano Rico, Víctor Méndez, Hector Andrés García-Silva, María del Carmen Suárez-Figueroa, Ghislain Atemezing, Raphaël Troncy
Work distributed under the license Creative Commons Attribution-Noncommercial-Share Alike 3.0
http://www.slideshare.net/ocorcho