Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4....
Transcript of Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4....
![Page 1: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/1.jpg)
Multimedia Information Retrieval
Prof Stefan Rüger
Multimedia and Information SystemsKnowledge Media Institute
The Open Universityhttp://kmi.open.ac.uk/mmis
![Page 2: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/2.jpg)
Multimedia Information Retrieval
1. What are multimedia queries?
2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation
8. Browsing, search and geography
- Information Visualisation
- User interaction
- Visual search and relevance feedback
- Browsing: lateral and geographic
![Page 3: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/3.jpg)
Alternatives to the list
How else could results be displayed?
![Page 4: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/4.jpg)
Sammon maps
![Page 5: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/5.jpg)
Dendro map
![Page 6: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/6.jpg)
Radial
![Page 7: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/7.jpg)
Query point moving
![Page 8: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/8.jpg)
Query point moving
![Page 9: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/9.jpg)
Weight space movement
10
![Page 10: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/10.jpg)
11
![Page 11: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/11.jpg)
12
![Page 12: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/12.jpg)
13
![Page 13: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/13.jpg)
Connect similar objects
Colour
Text
Structurefocus [with Heesch ECIR 2004, CIVR 2005][with Heesch et al ACM MM 2006,TRECVID 2003-2004][PhD thesis Heesch, nominated forbest UK thesis 2005]
![Page 14: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/14.jpg)
Browsing example
TRECVID topic 102: Find shots from behind the pitcher in a baseball game as he throws a ball that the batter swings at
In your mind
![Page 15: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/15.jpg)
[Alexander May: SET award 2004: best use of IT – national prize][dataset TRECVID 2003]
![Page 16: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/16.jpg)
![Page 17: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/17.jpg)
![Page 18: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/18.jpg)
![Page 19: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/19.jpg)
Empora.com
![Page 20: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/20.jpg)
Empora.com
![Page 21: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/21.jpg)
Geotemporal search
![Page 22: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/22.jpg)
Geography
![Page 23: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/23.jpg)
Wikipedia
[Logo trademark and © by Wikimedia Foundation, Inc]
![Page 24: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/24.jpg)
English language
[with Overell, GIR 2007; MMKM 2008]
![Page 25: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/25.jpg)
![Page 26: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/26.jpg)
English
[with Overell, GIR 2007; MMKM 2008]
![Page 27: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/27.jpg)
Russian
[with Overell, GIR 2007; MMKM 2008]
![Page 28: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/28.jpg)
German
[with Overell, GIR 2007; MMKM 2008]
![Page 29: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/29.jpg)
French
[with Overell, GIR 2007; MMKM 2008]
![Page 30: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/30.jpg)
Spanish
[with Overell, GIR 2007; MMKM 2008]
![Page 31: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/31.jpg)
Portugese
[with Overell, GIR 2007; MMKM 2008]
![Page 32: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/32.jpg)
Chinese
[with Overell, GIR 2007; MMKM 2008]
![Page 33: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/33.jpg)
Japanese
[with Overell, GIR 2007; MMKM 2008]
[with Overell, GIR 2007; MMKM 2008]
![Page 34: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/34.jpg)
![Page 35: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/35.jpg)
![Page 36: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/36.jpg)
Conclusion
Are we all little Steinbergs?
S Overell and S Rüger: View of the world according to Wikipedia: are we all little Steinbergs?International Journal of Computational Science, 2(3), pp 193–197, 2011. DOI: 10.1016/j.jocs.2011.05.006
![Page 37: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/37.jpg)
Mobile interactions
• SnapTell• Spot&Search
![Page 38: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/38.jpg)
SnapTell
![Page 39: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/39.jpg)
Spot&Search: OU Art
![Page 40: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/40.jpg)
Spot&Search: OU Art
![Page 41: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/41.jpg)
Feedback
• Please fill in the feedback form• I want to know what worked (and what didn't!)
![Page 42: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/42.jpg)
Multimedia Information Retrieval
1. What are multimedia queries?
2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation
8. Browsing, search and geography
- Information Visualisation
- User interaction
- Visual search and relevance feedback
- Browsing: lateral and geographic
![Page 43: Multimedia Information Retrieval · 2017. 4. 26. · 3. Metadata & piggy-back retrieval 4. Automated image annotation 5 Visual content-based retrieval I 6 Visual content-based retrieval](https://reader035.fdocuments.us/reader035/viewer/2022071104/5fddd02d07ffbe4a1f091a4b/html5/thumbnails/43.jpg)
Thank you!
• You have been a fantastic crowd• Thank you for your interest and all the interaction• Big thanks to NII, Professor Shin'ichi Satoh,
Professor Noriko Kando and the professional international affairs and educational support team, in particular Ms Tsukushi Inagaki, for being such a great host!