CH 14 Multimedia IR. Multimedia IR system The architecture of a Multimedia IR system depends on two...
-
Upload
calvin-robbins -
Category
Documents
-
view
341 -
download
1
Transcript of CH 14 Multimedia IR. Multimedia IR system The architecture of a Multimedia IR system depends on two...
Multimedia IR system
• The architecture of a Multimedia IR system depends on two main factors– The peculiar characteristics of multimedia
data– The kinds of operations to be performed on
such data
Multimedia IR system
• Support variety of data– Different kinds of media
• Text, images (both still and moving), graphs, and sound
– Mix of structured and unstructured data• Metadata• Semi-structured data
– Data whose structure may not match, or only partially match, the structure prescribed by the data schema
• The system must typically extract some features from the multimedia objects
Multimedia IR system
• Data retrieval– Exploiting data attributes and the content of
multimedia objects– Basic steps
• Query specification– Fuzzy predicates, content-based predicates, object
attributes, structural predicates
• Query processing and optimization– Query is parsed and compiled into an internal form
• Query answer• Query iteration
Multimedia IR system
• Combine DBMS and IR technology– DBMS: data modeling capabilities– IR system: similarity-based query capabilities
Data modeling
• Main tasks– A data model should be defined by which the
user can specify the data to be stored into the system
• Support conventional and multimedia data types• Provide methods to analyze, retrieve, and query
such data
– Provide a model for the internal representation of multimedia data
Object-oriented DBMS
• Provide rich data model– More suitable for modeling both multimedia data type
s and their semantic relationships
• Class– Attributes +operations– Inheritance
• Drawback– the performances of storage techniques, query proces
sing, and transaction management is not comparable to that of relational DBMSs
– Highly non-standard
Object-relational DBMS
• Extend the relational model– Represent complex data types– Maintain the performance and the simplicity of
relational DBMSs and related query languages
– Define abstract data types• Allows one to define ad hoc data types for multime
dia data
Internal representation
• Using attributes is not sufficient
• Feature– Information extracted from objects
• Multimedia object is represented as a set of features
• Features can be assigned manually, automatically, or using a hybrid approach
Internal representation
• Values of some specific features are assigned to a object by comparing the object with some previously classified objects
• Feature extraction cannot be precise– A weight is usually assigned to each feature
value representing the uncertainty of assigning such a value to that feature
• 80% sure that a shape is a square
SQL3
• Support extensible type system– Provide constructs to define user-dependent a
bstract data types, in an object-oriented like manner
• Collection data types– Sets, multisets, and lists– The elements of a collection must have comp
atible types
MULTOS
• MULTimedia Office Server– Client/server– Support filing and retrieval of multimedia objec
ts
• Each document is described by a logical structure, a layout structure, and a conceptual structure
• Documents having similar conceptual structures are grouped into conceptual types
MULTOS
• Conceptual types are maintained in a hierarchy of generalization
• Strong type– Completely specifies the structure of its
instances
• Weak type– Partially specifies the structure of its instances– Components of unspecified type (called
spring component types) can appear in a document definition
Document
Place Date Receiver+ Sender
AddressName
CityStreet Country
AddressName
CityStreet Country
Letter_body
spring component type
Conceptual structure of the type Generic_Letter
Complete conceptual structure of the type Business_Product_Letter
Document
Place Date Receiver+ Sender
AddressName
CityStreet Country
AddressName
CityStreet
Company_LogoImage
Country
Product_PresentationText
Product_DescriptionText
Product_CoseText
Signature
Letter_body
Query languages
• Relational/object-oriented database system– Exact match of the values of attributes
• Multimedia IR system– Similarity-based approach (+exact match)
• Considers the structure and the content of the objects
• Content-based query– Retrieve multimedia objects depending on their globe
content
Query languages
• In designing a multimedia query language, three main aspects require attention– How the user enters his/her request to the
system– Which conditions on multimedia objects can
be specified in the user request– How uncertainty, proximity, and weights
impact the design of the query language
Request specification
• Interfaces– Browsing and navigation– Specifying the conditions the objects of
interest must satisfy, by means of queries
• Queries can be specified in two different ways– Using a specific query language– Query by example
• Using actual data (object example)
Conditions on multimedia data
• Query predicates– Attribute predicates
• Concern the attributes for which an exact value is supplied for each object
• Exact-match retrieval
– Structural predicates• Concern the structure of multimedia objects• Can be answered by metadata and information
about the database schema• “Find all multimedia objects containing at least one
image and a video clip”
Conditions on multimedia data
– Semantic predicates• Concern the semantic content of the required data,
depending on the features that have been extracted and stored for each multimedia object
• “Find all the red houses”• Exact match cannot be applied
Uncertainty, proximity, and weights in query expressions
• Specify the degree of relevance of the retrieved objects– Using some imprecise terms and predicates
• Represent a set of possible acceptable values with respect to which the attribute or he features has to be matched
• Normal, unacceptable, typical
– Particular proximity predicates• The relationship represented is based on the
computation of a semantic distance between the query object and stored ones
• Nearest object search
Uncertainty, proximity, and weights in query expressions
– Assign each condition or term a given weight• Specify the degree of precision by which a
condition must be verified by an object• “Find all the objects containing an image
representing a screen (HIGH) and a keyboard (LOW)”
• The corresponding query is executed by assigning some importance and preference values to each predicate and term
SQL3 query language
• Major improvements of SQL3– Functions and stored procedures
• Allow users to integrate external functionalities with data manipulation
– Active database facilities• Support active rules
– The database is able to react to some system- or user-dependant events by executing specific actions
• Limitation– No IR techniques are integrated into the SQL3
query processor
MULTOS query language
• General formFIND DOCUMENTS VERSION version-clause
SCOPE scope-clause
TYPE type-clause
WHERE condition-clause
WITH component
MULTOS query language
• Three main classes of predicates– Data attributes– Textual components– Images
• The class to which an image should belong• Existence and the number of occurrences of an
object within an image
• Support imprecise query– Associating a preference and an importance
value with the attributes in the query
MULTOS query language
FIND DOCUMENTS VERSION LAST WHEREDocument.Date > 1/1/1998 AND(*Sender.Name = “Olivetti” OR *Product_Presentation CONTAINS “Olivetti”) AND*Product_Description CONTAINS “Personal Comp
uter” AND(*Address.Country = “Italy” OR TEXT CONTAINS “Italy”) ANDWITH *Company_Logo
MULTOS query language
FIND DOCUMENTS VERSION LAST WHERE
(Document.Date BETWEEN (12/31/1998,1/31/98) PREFERRED BETWEEN (2/1/1998,2/15/98) ACCEPTABLE) HIGH AND
(*Sender.Name = “Olivetti” OR
*Product_Presentation CONTAINS “Olivetti”) HIGH AND
(*Product_Description CONTAINS “Personal Computer”) HIGH AND
(*Product_Description CONTAINS “good ergonomics”) LOW AND
(*Address.Country = “Italy” OR TEXT CONTAINS “Italy”) HIGH AND
WITH *Company_Logo HIGH
(IMAGE MATCHES
screen HIGH
Keyboard HIGH
AT LEAST 2 floppy_drives LOW) HIGH
Indexing and searching
• Searching similar patterns• Distance function
– Given two objects, O1 and O2, the distance (=dissimilarity) of the two objects is denoted by D(O1,O2)
• Similarity queries– Whole match– Sub-pattern match– Nearest neighbors– All pairs
Spatial access methods
• Map objects into points in f-D space, and to use multiattribute access methods (also referred to as spatial access methods or SAMs) to cluster them and to search for them
• Methods– R*-trees and the rest of the R-tree family– Linear quadtrees– Grid-files Linear quadtrees and grid files explode exponentially
with the dimensionality
R-tree
• R-tree– Represent a spatial object by its minimum bou
nding rectangle (MBR)– Data rectangles are grouped to form parent n
odes (recursively grouped)– The MBR of a parent node completely contain
s the MBRs of its children– MBRs are allowed to overlap– Nodes of the tree correspond to disk pages
R-tree
• Range query– Specify a region of interest, requiring all the d
ata regions that intersect it– Retrieve
• Compute the MBR of the query region• Recursively descend the R-tree, excluding the bra
nches whose MBRs do not intersect the query MBR
• The retrieved data regions will be further examined for intersection with the query region
Generic multimedia indexing approach
• “Whole match” problem– A collection of N objects: O1, O2,…,ON
– The distance/dissimilarity between two objects (Oi,Oj) is given by the function D(Oi,Oj)
– User specifies a query object Q, and a tolerance ε
– Goal• Find the objects in the collection that are within dist
ance εfrom the query object
GEMINI
• Generic Multimedia object INdexIng
• Ideas– A ‘quick-and-dirty’ test, to discard quickly the
vast majority of non-qualifying objects (possibly, allowing some false alarms)
– The use of spatial access methods, to achieve faster-than-sequential searching
GEMINI
• Example– Database: yearly stock price movements, with one
price per day– Distance function
• Euclidean distance
– The idea behind the quick-and-dirty test is to characterize a sequence with a single number (feature), which help us discard many non-qualifying sequences
• Average stock price over the year, standard deviation, some of the discrete Fourier transform (DFT) coefficients
2/1
1
2][][),(
i
iQiSQSD
GEMINI
• Mapping function– Let F() be the mapping of objects to f-dimensional
points, that is, F(O) will be the f-D point that corresponds to object O
• Organize f-D points into a spatial access method, cluster them in a hierarchical structure, like the R*-trees
• Upon a query, we can exploit the R*-tree, to prune out large portions of the database that are not promising
GEMINI
• Search algorithm (for whole match query)– Map the query object Q into a point F(Q) in fe
ature space– Using a spatial access method, retrieve all poi
nts within the desired tolerance εfrom F(Q)– Retrieve the corresponding objects, compute t
heir actual distance from Q and discard the false alarms
GEMINI
• Lower Bounding lemma– To guarantee no false dismissals for whole-match que
ries, the feature extraction function F() should satisfy the following formula
– Dfeature(): distance of two feature vectors
(mapping F() from objects to points should make things look closer)
2121 ,, OODOFOFDfeature
GEMINI
• GEMINI algorithm– Determine the distance function D() between
two objects– Find one or more numerical feature-extraction
functions, to provide a ‘quick-and-dirty’ test– Prove that the distance in feature space
lower-bounds the actual distance D(), to guarantee correctness
– Use a SAM (e.g., an R-tree), to store and retrieve the f-D feature vectors
GEMINI
• ‘Feature-extracting’ question– If we are allowed to use only one numerical
feature to describe each data object, what should this feature be?
• The successful answers to the question should meet two goals– They should facilitate step 3 (the distance
lower-bounding)– They should capture most of the
characteristics of the objects
One-dimensional time series
• Search a collection of (equal-length) time series, to find the ones that are similar to a desirable series.– ‘in a collection of yearly stock price
movements, find the ones that are similar to IBM’
• Distance function– Euclidean distance
One-dimensional time series
• Feature extraction– The coefficients of the Discrete Fourier Transform (D
FT)
• Lower-bounding– Parseval’s theorem
• The DFT preserves the energy of a signal, as well as the distances between two signals
)1,...,0(frequency th - at thet coefficien DFTpoint -n thedenote let ,1,...,0],[ signal aFor
nFFXnixx Fi
lyrespective and of ansformsFourier tr are and where),(),(
yxYXYXDyxD
One-dimensional time series
– If we keep the first f (f≤n) coefficients of the DEF as the features, we lower-bound the actual distance
yxDyFxFD
yx
YX
YXyFxFD
feature
n
i
ii
n
F
FF
f
F
FFfeature
,,
,
1
0
2
1
0
2
1
0
2
– There will be no false dismissals
One-dimensional time series
• DFT concentrates the energy in the first few coefficients, for a large class of signals, the colored noises. These signals have a skewed energy spectrum (O(F-b))– b=2: random walks (brown noises)
• Model stock movements and exchange rates
– b>2: black noises• Model water level of rivers and rainfall patterns
– b=1: pink noise• ‘interesting signals’: musical scores and other works of art
(white noise is unpredictable, brown noise is too predictable)
One-dimensional time series
• Experiments– Artificially generated random walks
• Sequence length n=1024• Database size N=50~400
– GEMINI vs. sequential scanning• SAM: R*-tree
– Response time
One-dimensional time series
– GEMINI can be successfully applied to time series, and specifically to the ones that behave like ‘colored noise’
– For signals with skewed spectrum, the minimum in the response time is achieved for a mall number of Fourier coefficients (f=1,2,3). The minimum is rather flat, which implies that a suboptimal choice for f will give search time that is close to the minimum
– The success in 1D series suggests that GEMINI is promising for 2D or higher-dimensionality signals, if those signals also have skewed spectrum
Two-dimensional color images
• QBIC (Query By Image Content) (IBM)– Query large online image databases using the
images’ content as the basis of the queries– Content
• Color, texture, shape, position, and dominant edges of image items and regions
• Applications– Medical
• “Give me other images that contain a tumor with a texture like this one”
– Photo-journalism• “Give me images that have blue at the top and red at the
bottom”
Two-dimensional color images
• Two datatypes– Image (≡scene)– Item
• A part of a scene– A person, a piece of outlined texture, an apple,…
• Image feature– Focus on the color features
• K-element color histogram for each item and scene, where k=256 or 64 colors
• Each component on the color histogram is the percentage of pixels that are most similar to that color
Two-dimensional color images
• Distance function
k
i
k
j
jjiiijt
hist yxyxayxAyxyxd ,2
A: color-to-color similarity matrix
• Obstacle of color indexing– Dimensionality curse– Quadratic nature of the distance function
• Cross-talk among the features• Expansive• Precludes efficient implementation of commonly used
spatial access methods
Two-dimensional color images
• GEMINI– Feature extraction
• Average amount of red, green, and blue in a given color image (RGB color space)
P
p
avg
P
p
avg
P
p
avg
tavgavgavg
pBPB
pGPG
pRPR
BGRx
1
1
1
)(/1
)(/1
)(/1
,,
P is the number of pixels in the itemR(P), G(p), B(p) are the red, green, and blue components respectively of the p-th pixel
Two-dimensional color images
– Distance function
– Lower-bounding• Quadratic Distance Bounding Theorem
yxyxyxdt
avg ,2
Two-dimensional color images
• Experiment– N=924 color images– K=256 colors– CPU time and disk accesses
Automatic feature extraction
• GEMINI is useful for any setting that we can extract features from
• Automatic feature extraction methods– Multidimensional Scaling (MDS)– FastMap
• Extracting features not only facilitates the use of off-the-shelf spatial access methods, but it also allows for visual data mining: we can plot a 2D or 3D projection of the data set, and inspect it for clusters, correlations, and other patterns