Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag...
Transcript of Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag...
![Page 1: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/1.jpg)
Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum
Max Planck Institute for Informatics, Saarbrücken, Germany
![Page 2: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/2.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
Q: bicycle in street Q: environment friendly traffic
![Page 3: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/3.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Q: bicycle in street Q: environment friendly traffic
![Page 4: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/4.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Text-only
Q: bicycle in street Q: environment friendly traffic
![Page 5: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/5.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Visual objects: bicycle, bus, car
Text-only
Text + visual
Q: bicycle in street Q: environment friendly traffic
![Page 6: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/6.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Visual objects: bicycle, bus, car
Text-only
Text + visual
Q: bicycle in street Q: environment friendly traffic
“Biking by the river”
Visual objects: train, piano
![Page 7: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/7.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Visual objects: bicycle, bus, car
Text-only
Text + visual
Q: bicycle in street Q: environment friendly traffic
“Biking by the river”
Visual objects: train, piano
Text-only
Text + visual
![Page 8: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/8.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Visual objects: bicycle, bus, car
Text-only
Text + visual
“Riding for a cause.”
Visual objects: person, bicycle
CSK: (riding bicycle, be, environment friendly)
Q: bicycle in street Q: environment friendly traffic
“Biking by the river”
Visual objects: train, piano
Text-only
Text + visual
![Page 9: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/9.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Visual objects: bicycle, bus, car
Text-only
Text + visual
“Riding for a cause.”
Visual objects: person, bicycle
CSK: (riding bicycle, be, environment friendly)
Text/visual-only
Text + visual + CSK
Q: bicycle in street Q: environment friendly traffic
“Biking by the river”
Visual objects: train, piano
Text-only
Text + visual
![Page 10: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/10.jpg)
17/06/2016 1Sreyasi Nag Chowdhury, AKBC 2016
User Query
Concrete Abstract
“Wow! Double-decker buses still run!”
Visual objects: bicycle, bus, car
Text-only
Text + visual
“Riding for a cause.”
Visual objects: person, bicycle
CSK: (riding bicycle, be, environment friendly)
Text/visual-only
Text + visual + CSK
Q: bicycle in street Q: environment friendly traffic
“Biking by the river”
Visual objects: train, piano
Text-only
Text + visual
Our contribution
![Page 11: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/11.jpg)
CSK: Where do we get it from?
CSK: How do we use it?
CSK: How to combine noisy signals?
CSK: Does it help?
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016 2
![Page 12: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/12.jpg)
Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood
CSK: WHERE DO WE GET IT FROM?
317/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 13: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/13.jpg)
Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood
Our corpus: Wiki articles from domain ‘tourism’
CSK: WHERE DO WE GET IT FROM?
317/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 14: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/14.jpg)
Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood
Our corpus: Wiki articles from domain ‘tourism’
“tourism” “be travel for” “recreation, leisure, family, business purposes”
“people” “fall in” “love”
---------------------------------------------------------------------------------------------------
“the bloody hell” “be” “you”
CSK: WHERE DO WE GET IT FROM?
3
Domain-specific
ReVerb triples
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 15: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/15.jpg)
Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood
Our corpus: Wiki articles from domain ‘tourism’
Pruned by Jaccard Similarity
“tourism” “be travel for” “recreation, leisure, family, business purposes”
“people” “fall in” “love”
---------------------------------------------------------------------------------------------------
“the bloody hell” “be” “you”
CSK: WHERE DO WE GET IT FROM?
3
Domain-specific
ReVerb triples
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 16: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/16.jpg)
Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood
Our corpus: Wiki articles from domain ‘tourism’
Pruned by Jaccard Similarity
~22,000 CSK triples
“tourism” “be travel for” “recreation, leisure, family, business purposes”
“people” “fall in” “love”
---------------------------------------------------------------------------------------------------
“the bloody hell” “be” “you”
CSK: WHERE DO WE GET IT FROM?
3
Domain-specific
ReVerb triples
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 17: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/17.jpg)
CSK: HOW DO WE USE IT?
Query string: travel with backpack
CSK to expand query
• t1: (tourists, use, travel maps)
• t2: (tourists, carry, backpack)
• t3: (backpack, is a type of, bag)
417/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 18: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/18.jpg)
Query string: travel with backpack
CSK to expand query
• t1: (tourists, use, travel maps)
• t2: (tourists, carry, backpack)
• t3: (backpack, is a type of, bag)
Document x with features
• Textual: “A tourist reading a map by the road”
• Visual: person, bag, bottle, bus
Text-only systems
Text + visual + CSK systems
417/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
![Page 19: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/19.jpg)
Query string: travel with backpack
CSK to expand query
• t1: (tourists, use, travel maps)
• t2: (tourists, carry, backpack)
• t3: (backpack, is a type of, bag)
Document x with features
• Textual: “A tourist reading a map by the road”
• Visual: person, bag, bottle, bus
Text-only systems
Text + visual + CSK systems
4
CSK bridge vocabulary gap
between query and document
CSK establish relations
between concepts
CSK diminish noise from
modalities – ensemble effect
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
![Page 20: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/20.jpg)
5
A tour group is standing on the grass with ruins in the background.
Group of people standing in front of a stone structure.
Document x
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
![Page 21: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/21.jpg)
5
Document x
Textual features xx
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
A tour group is standing on the grass with ruins in the background.
Group of people standing in front of a stone structure.
![Page 22: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/22.jpg)
5
Document x
Textual features xx
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
A tour group is standing on the grass with ruins in the background.
Group of people standing in front of a stone structure.
Visual features xv : backpack, person
![Page 23: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/23.jpg)
5
Document x
Textual features xx
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
A tour group is standing on the grass with ruins in the background.
Group of people standing in front of a stone structure.
Visual features xv : backpack, bag, container
person, casual agent, organism
![Page 24: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/24.jpg)
5
Document x
Textual features xx
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
A tour group is standing on the grass with ruins in the background.
Group of people standing in front of a stone structure.
Visual features xv : backpack, bag, container
person, casual agent, organism
Query: “group excursion”
Query expansion: (an excursion, be trip by, a group of people)
(organized excursions, book through, a tour company)
CSK features
![Page 25: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/25.jpg)
A tour group is standing on the grass with ruins in the background.
Group of people standing in front of a stone structure.
517/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: HOW DO WE USE IT?
Visual features xv : backpack, bag, container
person, casual agent, organism
Query: “group excursion”
Query expansion: (an excursion, be trip by, a group of people)
(organized excursions, book through, a tour company)
![Page 26: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/26.jpg)
CSK: HOW TO COMBINE NOISY SIGNALS?
6
Mixture LM:
Commonsense-aware LM:
Smoothed LM: , where
Basic LM: , where
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 27: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/27.jpg)
CSK: HOW TO COMBINE NOISY SIGNALS?
6
Mixture LM:
Commonsense-aware LM:
Smoothed LM: , where
Basic LM: , where
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 28: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/28.jpg)
6
Mixture LM:
Commonsense-aware LM:
Smoothed LM: , where
Basic LM: , where
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK triple
CSK: HOW TO COMBINE NOISY SIGNALS?
![Page 29: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/29.jpg)
6
Mixture LM:
Commonsense-aware LM:
Smoothed LM: , where
Basic LM: , where
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
Probabilities based on
word-wise overlaps
CSK: HOW TO COMBINE NOISY SIGNALS?
![Page 30: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/30.jpg)
6
Mixture LM:
Commonsense-aware LM:
Smoothed LM: , where
Basic LM: , where
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
Background corpus –
Co-occurring Flickr tags
CSK: HOW TO COMBINE NOISY SIGNALS?
![Page 31: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/31.jpg)
6
Mixture LM:
Commonsense-aware LM:
Smoothed LM: , where
Basic LM: , where
17/06/2016Sreyasi Nag Chowdhury, AKBC 2016
Textual and visual features
CSK: HOW TO COMBINE NOISY SIGNALS?
![Page 32: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/32.jpg)
717/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
Image Dataset
o Flickr30k
o MS COCO captioned dataset
o Pascal Sentence Dataset
o SBU captioned dataset
![Page 33: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/33.jpg)
717/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
Image Dataset
o Flickr30k
o MS COCO captioned dataset
o Pascal Sentence Dataset
o SBU captioned dataset
Boat trip to see the mythical pink dolphins...
this is John checking in with the office for that day.
![Page 34: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/34.jpg)
717/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
Image Dataset
o Flickr30k
o MS COCO captioned dataset
o Pascal Sentence Dataset
o SBU captioned dataset
A group of tourists is crossing a bridge that connects a walking path to a trail of nature.
Many people cross a very tall footbridge with a tree-covered hill in the background.
This shows a group of people walking over an arched red bridge.
People cross a large bridge to get over the body of water.
People walking over a white and red bridge over a pond.
Boat trip to see the mythical pink dolphins...
this is John checking in with the office for that day.
![Page 35: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/35.jpg)
717/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
Image Dataset
o Flickr30k
o MS COCO captioned dataset
o Pascal Sentence Dataset
o SBU captioned dataset
A group of tourists is crossing a bridge that connects a walking path to a trail of nature.
Many people cross a very tall footbridge with a tree-covered hill in the background.
This shows a group of people walking over an arched red bridge.
People cross a large bridge to get over the body of water.
People walking over a white and red bridge over a pond.
Boat trip to see the mythical pink dolphins...
this is John checking in with the office for that day.
social media post
blog post
![Page 36: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/36.jpg)
717/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
Image Dataset
o Flickr30k
o MS COCO captioned dataset
o Pascal Sentence Dataset
o SBU captioned dataset
~ 50,000 images with captions
A group of tourists is crossing a bridge that connects a walking path to a trail of nature.
Many people cross a very tall footbridge with a tree-covered hill in the background.
This shows a group of people walking over an arched red bridge.
People cross a large bridge to get over the body of water.
People walking over a white and red bridge over a pond.
Boat trip to see the mythical pink dolphins...
this is John checking in with the office for that day.
social media post
blog post
![Page 37: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/37.jpg)
Baselines: Text-only and Text + Visual search approaches
Evaluation metric: Average Precision @ 10
817/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
![Page 38: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/38.jpg)
Baselines: Text-only and Text + Visual search approaches
Evaluation metric: Average Precision @ 10
817/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
47% 64% 85%
Text-only Text + Visual Text + Visual + CSK (Know2Look)
![Page 39: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/39.jpg)
Baselines: Text-only and Text + Visual search approaches
Evaluation metric: Average Precision @ 10
Examples queries:
o Concrete – ball park, bridge road, table home, bicycle road
o Abstract – diesel transport, housing town
o Mixed – old clock, backpack travel, boat tour
817/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
47% 64% 85%
Text-only Text + Visual Text + Visual + CSK (Know2Look)
![Page 40: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/40.jpg)
Baselines: Text-only and Text + Visual search approaches
Evaluation metric: Average Precision @ 10
Examples queries:
o Concrete – ball park, bridge road, table home, bicycle road
o Abstract – diesel transport, housing town
o Mixed – old clock, backpack travel, boat tour
817/06/2016Sreyasi Nag Chowdhury, AKBC 2016
CSK: DOES IT HELP?
47% 64% 85%
Text-only Text + Visual Text + Visual + CSK (Know2Look)
Co-occurring Flickr tags
![Page 41: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/41.jpg)
917/06/2016Sreyasi Nag Chowdhury, AKBC 2016
Text-only Text + Visual Text + Visual + CSK (Know2Look)
Query: “group excursion”
xxj: “A small excursion boat anchored
on the beach at the resort in Mexico. "
xvj: lunar excursion module, conveyance xxj: “A group of people riding camels.“
yk: (an excursion, be trip by,
a group of people)
CSK: DOES IT HELP?
![Page 42: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/42.jpg)
917/06/2016Sreyasi Nag Chowdhury, AKBC 2016
Text-only Text + Visual Text + Visual + CSK (Know2Look)
Query: “group excursion”
xxj: “A small excursion boat anchored
on the beach at the resort in Mexico. "
xvj: lunar excursion module, conveyance xxj: “A group of people riding camels.“
yk: (an excursion, be trip by,
a group of people)
CSK: DOES IT HELP?
![Page 43: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/43.jpg)
Noisy OpenIE triples capture commonsense knowledge
Noisy textual cues + noisy visual object detection + noisy commonsense knowledge ensemble effect better results for multimodal document retrieval
CSK act as bridge between text and vision
1017/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 44: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/44.jpg)
Noisy OpenIE triples capture commonsense knowledge
Noisy textual cues + noisy visual object detection + noisy commonsense knowledge ensemble effect better results for multimodal document retrieval
CSK act as bridge between text and vision
Do word co-occurrences or word embeddings provide similar results?
Do structured commonsense knowledge improve retrieval?
1017/06/2016Sreyasi Nag Chowdhury, AKBC 2016
![Page 45: Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum › ~sreyasi › akbc_slides.pdf · Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1 User Query Concrete Abstract “Wow! Double-decker](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f043df57e708231d40d02bc/html5/thumbnails/45.jpg)
Noisy OpenIE triples capture commonsense knowledge
Noisy textual cues + noisy visual object detection + noisy commonsense knowledge ensemble effect better results for multimodal document retrieval
CSK act as bridge between text and vision
Do word co-occurrences or word embeddings provide similar results?
Do structured commonsense knowledge improve retrieval?
1017/06/2016Sreyasi Nag Chowdhury, AKBC 2016
Thank you