Lecture_04_Image_Search

8/8/2019 Lecture_04_Image_Search

1/27

Image Searches,

Abstraction, Invariance36-350: Data Mining

2 September 2009

1


2/27

Medical: x-rays, brain imaging, histology (dothese look like cancerous cells?)

Satellite imagery

Fingerprints

Finding illustrations for lectures...

2


3/27

Searching for Images by

Searching for Text Assume theres text accompanying the images

(annotation)

tags

Search those text records with the query phrase

Take images which appear close to the queryphrase on highly-ranked records

This how Google does it

3


4/27

Sometimesthis worksperfectly...

4


5/27

...and

sometimesit doesnt;depends on

the text!

5


6/27

Searching for images by

representing images

For text, we only cared about features, and

only worked with feature vectors

Define numerical features for images andeverything carries over

Abstraction

6


7/27

Abstraction

Remove some of the details but keep others

Kept details = features Then act on abstracta

Hopes:

Simplifies problem Lets you treat many problems similarly

7


8/27

Abstract level: feature vectors

Concrete level: meaningful objects

Text 1 Text 2 Text 3

BoW

BoW

v1 v2 v3 v4 v5 v6

Similaritymatching

DimensionalityReduction

Classification Clustering

BoW

BoW


BoW

BoW

etc.

8


9/27



Topics

v1 v2 v3 v4 v5 v6

Topics


Topics

Topics

Topics

Topics


Similaritymatching


Classification Clustering etc.

9


10/27


Bitmap

v1 v2 v3 v4 v5 v6

Pic. 1 Pic. 2 Pic. 3 Pic. 4 Pic. 5 Pic.6

Bitmap

Bitmap

Bitmap

Bitmap

Bitmap


Similaritymatching



10


11/27


v1 v2 v3 v4 v5 v6

Pic. 1 Pic. 2 Pic. 3 Pic. 4 Pic. 5 Pic.6

Bagofcolors

Bagofcolors

Bagofcolors

Bagofcolors

Bagofcolors

Bagofcolors


Similaritymatching



11


12/27


Network 6

Motifs

v1 v2 v3 v4 v5 v6

Network 1 Network 2 Network 3 Network 4 Network 5

Motifs

Motifs

Motifs

Motifs

Motifs


Similaritymatching



12


13/27


14/27



BoW

BoW

Topics

Pic. 1 Pic. 2

Social

Network

v1 v2 v3 v4 v5 v6

Bagofcolors

Bitmap

Motifs


Similaritymatching



14


15/27

flower1 flower2 flower3

tiger1 tiger2 tiger3

ocean1 ocean2 ocean315


16/27

Euclidean Distance of

Images Image is MxN pixels, each with 3 color

components, so a 3MN vector

Euclidean distance possible, and OK forsome kinds of noise-removal

but hopeless even at grouping flower1 with

flower2

or slight changes in perspective, lighting...

16


17/27

Bag of Colors

If it works, try it some more

For each possible color, count how manypixels there are of that color

Use Euclidean distance on color-countvectors

Too many colors, so quantize them down toa manageable number (like stemming, orcombining synonyms)

17


18/27

flower1

flower2

flower3

flower4

flower5

flower6

flower7

flower8

flower9

tiger1

tiger2

tiger3

tiger4

tiger5

tiger6

tiger7

tiger8

tiger9

ocean1

ocean2

ocean3

ocean4

ocean5

ocean6

ocean7

ocean7

ocean6

ocean5

ocean4

ocean3

ocean2

ocean1

tiger9

tiger8

tiger7

tiger6tiger5

tiger4

tiger3

tiger2

tiger1

flower9

flower8

flower7

flower6

flower5

flower4

flower3

flower2

flower1

1.0 0.5 0.0 0.5 1.0

1.0

0.5

0.0

0.5

1.0

V1

V2

flower1

flower2

flower3

flower4

flower5

flower6

flower7

flower8

flower9

tiger1tiger2

tiger3

tiger4

tiger5

tiger6

tiger7tiger8

tiger9

ocean1

ocean2

ocean3ocean4

ocean5 ocean6

ocean7

flower ocean tiger

Multidimensional scaling

Distances between images MDS plot of images18


19/27

Representation and

Invariance

Invariances of a representation = how can wechange the underlying object without changingthe representation?

What differences does the representation ignore?

19


20/27

Invariants of bags of words

Punctuation and word order

Universal words (exact count of the, of,to, ...), if using inverse document frequency

Word-endings, if using stemming

Grammar, context, word proximity ... Send lawyers, guns and money vs.Sending the Guns lawyers for themoney

20


21/27


22/27

Same color counts, different textures

22


23/27

Non-invariants

Lighting, shadows

Occlusion, 3D effects Blurring

There are good ways to deal with blur(from astronomy)

but full vision is very, very hard

23


24/27

Breaking an invariance is easy

e.g., add features for textures

or sub-divide the image and do color-counts on each part

Adding invariances is hard

often need to go back to scratch andchose a different representation

24


25/27

Similaritysearch

with real

imagesfrom the

web(retrievr,

see notes)

25


26/27

26


27/27

Typically works better with more restricteddomains (actually pretty good for medicalimages)

Lecture_04_Image_Search

Documents

Transcript of Lecture_04_Image_Search