Descriptive Semantic Image Retrieval David Norton Derral Heath.
-
date post
21-Dec-2015 -
Category
Documents
-
view
220 -
download
3
Transcript of Descriptive Semantic Image Retrieval David Norton Derral Heath.
![Page 1: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/1.jpg)
Descriptive Semantic Image Retrieval
David Norton
Derral Heath
![Page 2: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/2.jpg)
Motivation
Retrieve an image based on a descriptive query: “Find me an image that is red, dark, scary, and
beautiful”
![Page 3: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/3.jpg)
Content-Based Image Retrieval
Retrieve an image strictly from image features color texture shape
General semantic based image retrieval is hard “Find me a picture of a piranha”
![Page 4: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/4.jpg)
Emotional Semantic Image Retrieval
Query images matching emotional words or word-pairs “Find me a happy picture”
Usually adjectives
Usually a small orthogonal subset of terms
Query via single words (or pairs)
![Page 5: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/5.jpg)
Descriptive Semantic Image Retrieval
Open to all descriptive words
Query via any number of words “Find me an image that is red, dark, scary, and
beautiful”
![Page 6: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/6.jpg)
Three Components
Extraction of image features
Semantic representation of image
Mapping between visuals and language
![Page 7: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/7.jpg)
Extraction of Image Features
Color (12) Average RGB values Color count
Texture (50) Entropy
Shape (10) Eccentricity
![Page 8: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/8.jpg)
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
114.0887 99.0219 118.648 20 0.1332 8.924632 0.109940296
113.5899 98.5232 118.145 18 0.1411 8.200488 0.101902564
113.6192 98.5358 118.153 20 0.1511 8.015762 0.082569348
![Page 9: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/9.jpg)
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
146.4833 75.8135 1.547 5 0.5259 7.805696 0.064152
145.9641 75.2931 1 4 0.5317 7.654775 0.073264
146.0377 75.3628 1.054 4 0.5496 7.502806 0.054083
![Page 10: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/10.jpg)
Semantic Representation of Image
How do we obtain a description?
What is a descriptive word?
What are the features?
![Page 11: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/11.jpg)
User Input Interface
![Page 12: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/12.jpg)
Bright has 11 Senses 1. (17) bright -- (emitting or reflecting light readily or in large amounts; "the sun was bright and hot"; "a
bright sunlit room") 2. (6) bright, brilliant, vivid -- (having striking color; "bright dress"; "brilliant tapestries"; "a bird with vivid
plumage") 3. (5) bright, smart -- (characterized by quickness and ease in learning; "some children are brighter in one
subject than another"; "smart children talk earlier than the average") 4. (3) bright -- (having lots of light either natural or artificial; "the room was bright and airy"; "a stage bright
with spotlights") 5. (1) bright, burnished, lustrous, shining, shiny -- (made smooth and bright by or as if by rubbing;
reflecting a sheen or glow; "bright silver candlesticks"; "a burnished brass knocker"; "she brushed her hair until it fell in lustrous auburn waves"; "rows of shining glasses"; "shiny black patents")
6. (1) bright -- (splendid; "the bright stars of stage and screen"; "a bright moment in history"; "the bright pageantry of court")
7. undimmed, bright -- (not made dim or less bright; "undimmed headlights"; "surprisingly the curtain started to rise while the houselights were still undimmed")
8. bright, brilliant -- (clear and sharp and ringing; "the bright sound of the trumpet section"; "the brilliant sound of the trumpets")
9. bright -- (characterized by happiness or gladness; "bright faces"; "all the world seems bright and gay") 10. bright, shining, shiny, sunshiny, sunny -- (abounding with sunlight; "a bright sunny day"; "one shining
morning"- John Muir; "when it is warm and shiny") 11. bright, promising -- (full or promise; "had a bright future in publishing"; "the scandal threatened an
abrupt end to a promising political career")
![Page 13: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/13.jpg)
Narrowing Down the Feature Space
Interface: Adjectives from WordNet
Restrict characters
Reduce available senses
![Page 14: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/14.jpg)
Narrowing Down the Feature Space
Post Processing: Use Synsets
Frequent synsets
Fit ORM ontology lexicons
![Page 15: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/15.jpg)
Image ORM Ontology
![Page 16: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/16.jpg)
Mapping between visuals and language
Series of Neural Networks
Bayes Net
Fuzzy Logic
![Page 17: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/17.jpg)
Evaluation
Let machine label images
Let humans label images
Let different humans evaluate machine and human labels
Compare evaluations
![Page 18: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/18.jpg)
Related Work
Aesthetic Visual Quality Assessment of Paintings (2009) Congcong Li and Tsuhan Chen
Labeled impressionistic style landscape paintings as ‘high’ or ‘low’ quality using machine learning.
Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition (October 2008) Ritendra Datta, Jia Li, and James Z. Wang
Overview of research involving predicting the quality class, score, and emotional label of photographs.
![Page 19: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/19.jpg)
Related Work
A Survey on Emotional Semantic Image Retrieval (2008) Weining Wang and Qianhua He
Surveys ongoing Emotional Semantic Image Retrieval research.
Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction (October 2006) Wang Wei-ning, Yu Ying-lin, and Jiang Sheng-ming
Labeled paintings with 12 emotional word pairs. Psychological research involved in choosing word pairs.
![Page 20: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/20.jpg)
Further Motivation
Augment the study of human perception and cognition.
Establish a linguistic-visual foundation for an artificially creative artist.
![Page 21: Descriptive Semantic Image Retrieval David Norton Derral Heath.](https://reader037.fdocuments.us/reader037/viewer/2022103022/56649d585503460f94a3810f/html5/thumbnails/21.jpg)
Questions?