Palette Power: Enabling Visual Search through Colors
-
Upload
wei-di -
Category
Technology
-
view
33 -
download
2
Transcript of Palette Power: Enabling Visual Search through Colors
![Page 1: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/1.jpg)
Palette Power: Enabling Visual Search through Colors
Aug 14, 2013
eBay Research Labs
http://labs.ebay.com
![Page 2: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/2.jpg)
Changing Landscape of Search
2
![Page 3: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/3.jpg)
Visual Search for Fashion
eBay
Inventory
Given an item image,
find similar eBay inventory
Query Image
Similar Items
3
![Page 4: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/4.jpg)
Item Similarity
Fre
quency
Color Distributions
Dots Floral Checks
Patterns & Textures
Styles
4
![Page 5: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/5.jpg)
Approach Overview
Large Image Data Speed Requirements
Take Advantage of Context
Our Approach – The Power of Color Distributions
Color Spaces
𝑑 = 𝑓(𝑖1, 𝑖2) Distance Functions
5
![Page 6: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/6.jpg)
6
Challenges (1/3)
Low contrast between background and foreground
![Page 7: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/7.jpg)
7
Challenges (2/3)
Background Clutter
![Page 8: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/8.jpg)
8
Challenges (3/3)
Lighting Variation
![Page 9: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/9.jpg)
9
Insights From Data
Object localization using spatial priors
Choosing the right color space
![Page 10: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/10.jpg)
Why Object Localization?
10
Cluttered background degrades performance.
State-of-the-art segmentation too expensive.
Need a fast and reliable solution!
Spatial Prior to the rescue!
![Page 11: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/11.jpg)
Understanding Spatial Prior
11
![Page 12: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/12.jpg)
12
Choosing Best Color Space
![Page 13: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/13.jpg)
13
Handling Color Confusion
![Page 14: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/14.jpg)
14
Generating Color Histogram
![Page 15: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/15.jpg)
Faster Lookup via k-center
15
Scaling via backend clustering/indexing.
Potential for semantic/intent diversification - e.g. query t-shirt image where you like style but not colors
Achieves 60x speedup close to 70% overlap!
Median speed-up Median %-overlap
![Page 16: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/16.jpg)
16
Architecture
![Page 17: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/17.jpg)
17
Experiment I – Fashion Dataset
Categories: Women’s Dresses, Tops & Blouses, Coats & Jackets,
Skirts, Sweaters and T-Shirts
Data Sets: 1600 Queries & 1 Million Inventory images, 15 users for
30 days
![Page 18: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/18.jpg)
18
Results - Solid Queries
![Page 19: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/19.jpg)
Results - Pattern Queries
19
![Page 20: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/20.jpg)
Experiment II – Generic ecommerce Dataset
Categories: Toys, Sports, Camera
Data Sets: Query & Inventory sets for each category
Ground Truth: ~15 per query
20
![Page 21: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/21.jpg)
Example Inventory Images
21
Toys
Sports
Camera
![Page 22: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/22.jpg)
22
MAP Performance
![Page 23: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/23.jpg)
23
Experiment III – INRIA Holidays Dataset
Categories: Personal Holidays Photos
Data Sets: 500 Queries (1 per group) & 1491 Inventory Images
Ground Truth: Human Annotations
![Page 24: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/24.jpg)
24
MAP Performance
![Page 25: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/25.jpg)
25
Computational Costs
Feature Extraction Time 10 ms
Retrieval Time 80 ms
Feature Vector Size 196 Bytes
Memory Required 190 MB
Machine Stats: 24 GB RAM, 2.53GHz
Index Size: 1M+
![Page 26: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/26.jpg)
26
Summary
Color a fundamental cue
Spatial Prior can eliminate need for expensive
background removal
Future work to focus on efficient descriptors
![Page 27: Palette Power: Enabling Visual Search through Colors](https://reader030.fdocuments.us/reader030/viewer/2022020211/55ca3e1bbb61eb22488b45e5/html5/thumbnails/27.jpg)
27
Questions?