Concise Papers __________________________________________________________________________________________
An Object-Oriented Fuzzy Data Model forSimilarity Detection in Image Databases
Arun K. Majumdar, Senior Member, IEEE,Indrajit Bhattacharya, and Amit K. Saha
Abstract—In this paper, we introduce a fuzzy set theoretic approach for dealing
with uncertainty in images in the context of spatial and topological relations
existing among the objects in the image. We propose an object-oriented graph
theoretic model for representing an image and this model allows us to assess the
similarity between images using the concept of (fuzzy) graph matching. Sufficient
flexibility has been provided in the similarity algorithm so that different features of
an image may be independently focused upon.
Index Terms—Image databases, fuzzy data model, spatial and topological
relations, similarity detection.
�
1 INTRODUCTION
WITH the advancement of multimedia applications, image data-bases have emerged as a major field of research [7], [9], [10], [11].Unlike the size of a record in a conventional database, the size of animage is usually very large. This has necessitated the developmentof efficient storage techniques for image databases. Moreover, thequeries in an image database often require retrieval based on thesimilarity with given images or based on certain attributes of theobjects present in the images. Thus, it is necessary to supportqueries based on image semantics rather than based on mere pixel-to-pixel matching. The image database systems should thereforeallow adequate abstraction mechanisms for capturing higher levelsemantics of the images in order to support content addressabilityas far as possible.
Over the last decade, there has been considerable interest in
representing spatial and topological relations among objects
present in a scene. Chang and Jungert [6] have proposed 2D
Strings and Interval Projection Strings to represent spatial relation-
ships among objects in a scene. Similarly, Gudivada [1] has
suggested a geometric approach, called �R Strings, for capturing
spatial relationships. These representation schemes using 2D or
�R Strings, however, are not efficient for capturing topological
relations like meet, overlap, cover, contain, etc. Considerable work
has been done in this particular direction by Egenhofer and
Herring [2] with GIS design and developments.In real-life images, it is often not possible to precisely identify
the boundaries of objects in a scene. Thus, object identification byimage processing techniques is difficult and identification ofbinary relations, like left-of, contain, or overlap, inexact. In theimage processing literature, statistical or fuzzy set theoreticapproaches have been advocated to deal with noise or uncertaintyin images. Several algorithms have been developed to filter outnoise and identify boundaries of objects in such unfavorable
environments [14]. In this paper, we will be using a fuzzy settheoretic approach to represent objects and their spatial andtopological relationships. This will enable us to deal with thesemantics of real-life images having objects with noncrispboundaries. Moreover, object relationships, such as left-of orcovered-by, with fuzzy qualifiers, like more-or-less, almost, etc.,can also be handled.
2 SPATIAL AND TOPOLOGICAL RELATIONS
Spatial relations like left, right, above, and below define the spatialorientations of the domain objects with respect to each other andhelp in ascribing meaning to a scene. On the other hand,topological relations deal with the nature of overlap betweenobjects and are invariant under rotations of the scene. Egenhoferand Herring [2] have defined interior ðAoÞ and boundary ð�AÞ of aregion (n-cell) A. The topological relations between two regions(objects) A and B are defined in terms of the intersections (ornonintersections) of the interior and boundary of the regions underconsideration. This model is called the 4-intersection model [2] andcan be concisely represented by a 2� 2 matrix:
RðA;BÞ ¼ Ao \Bo Ao \ �B�A \ Bo �A \ �B
� �:
Egenhofer and Herring used this approach to represent eight
basic topological relations among the regions without holes,namely, disjoint, cover, meet, inside, contains, coveredby, overlap, and
equal. Some pairs such as cover and coveredby are duals of each other
(the same is true for inside and contains). Further, disjoint, meet,
overlap, and equal are symmetric. The four empty/nonempty
intersections describe a set of relations that provides completecoverage. These relations are mutually exclusive so that the union
(OR) of all specifications is true and the intersection (AND) of any
two specified relations is identically false.
3 A FUZZY OBJECT DATA MODEL FOR IMAGES
We have already mentioned that, with imprecise or noisy data, it isoften not possible to say definitely whether a pixel belongs to anobject. Rather, an object may be looked upon as a fuzzy subset ofthe set of pixels. Accordingly, let O be the set of objects present in atwo-dimensional scene. An object A 2 O is treated as a fuzzysubset of pixels of the scene with membership function A. Thus,for a pixel located at a point z in the scene, the possibility of thispixel belonging to the object A is AðzÞ. In the following, we usethe notations zX and zY to denote the X and Y coordinates of apixel z. Let zþ denote the set of 8-neighbors of a pixel at point z.Then, the boundary (�A) of A may also be defined as a fuzzy subsetwith membership function
�AðzÞ ¼ minfAðzÞ;maxu2zþf1 AðuÞgg::::::::ðaÞ:
Following this approach, the definition of membership function
for the interior Ao of an object A is given by
AoðzÞ ¼ minfAðzÞ; 1 �AðzÞg::::::::ðbÞ:
In the proposed framework, the spatial and topological
relations among the objects will also be imprecise and will betreated as fuzzy subsets.
3.1 Fuzzy Spatial Relations
The spatial relations left of , right of , above, or below between a
given pixel and an object would be imprecise and will be treated as
fuzzy subsets. The corresponding relations between any two given
1186 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002
. A.K. Majumdar is with the Department of Computer Science andEngineering, Indian Institute of Technology, Kharagpur, West Bengal,721 302, India. E-mail: [email protected].
. I. Bhattacharya is with the Department of Computer Science, University ofMaryland, College Park, MD 20742. E-mail: [email protected].
. A.K. Saha is with Rice University, 6100 Main, Houston, TX 77005.E-mail: [email protected].
Manuscript received 1 Dec. 1999; revised 9 Jan. 2001; accepted 29 May 2001.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 111010.
1041-4347/02/$17.00 � 2002 IEEE
pixels are, however, precise, i.e., a pixel a is either to the left or
right or has the same X-coordinate as another pixel b. Thus, we
define
leftof ða; bÞ ¼0 if ax � bx1 else;
�ð1Þ
rightof ða; bÞ ¼0 if ax � bx1 else:
�ð2Þ
The spatial relations above and below between two pixels can be
similarly defined by considering the Y coordinates of the pixels.We now define the possibility of a pixel z being to the left-of an
object A by the fuzzy predicate “no pixel of A is to the left-of the
pixel z.”
leftof ðz; AÞ ¼ 1maxvfminðleftofðv; zÞ; AðvÞÞg: ð3Þ
Similarly,
rightof ðz; AÞ ¼ 1maxvfminðrightof ðv; zÞ; AðvÞÞg: ð4Þ
Based on these definitions, it may be seen that the possibility of
A being left-of a pixel z has the same value as that of z being right-
of A. This follows from the observation that
leftof ðA; zÞ ¼ 1maxvfminðleftof ðz; vÞ; AðvÞÞg ð5Þ
and, for any two pixels v and z, rightofðv; zÞ ¼ leftof ðz; vÞ.Hence, for any pixel z and an object A, we have
rightofðz; AÞ ¼ leftofðA; zÞ:::::::ðcÞ:
However, unlike the classical case, the fuzzy left-of and right-of
relations between a pixel and an object are not complementary, i.e.,
leftof ðz; AÞ 6¼ 1 rightof ðz; AÞ::::::::::ðdÞ:
We can now treat the spatial relations left-of and right-of between
two objects (A and B) as fuzzy subsets of O�O. Accordingly, the
fuzzy relation “A left-of B” is defined by the fuzzy predicate “no
pixel of B is to the left-of A.” Thus,
leftofðA;BÞ ¼ 1maxzfminðBðzÞ; leftof ðz;AÞÞg: ð6Þ
Similarly,
rightof ðA;BÞ ¼ 1maxzfminðBðzÞ; rightof ðz; AÞÞ: ð7Þ
The fuzzy spatial relations above or below between two objects can
be similarly defined by extending the same relations between two
pixels.
Lemma 3.1. The left-of (right-of) relation between two objects is
transitive.
leftof ðA;CÞ � maxBfminðleftof ðA;BÞ; leftof ðB;CÞÞg: ð8Þ
The proof could not be included due to space constraints. The
transitivity claim can also be made for the fuzzy relation above
(below) between two objects. Note that, to enforce symmetry for
the left-of and right-of relations between objects, we may modify
the definitions as follows:
leftofðA;BÞ¼min½f1maxzðminðBðzÞ; leftofðz; AÞÞÞg;f1maxzðminðAðzÞ; rightof ðz; BÞÞÞg�:
ð9Þ
3.2 Fuzzy Topological Relations
We are now in a position to define fuzzy extensions of the
topological relations discussed in Section 2.
. disjoint: Two objects A and B are said to be disjoint if allpixels in A are not in B and vice versa.
disjointðA;BÞ ¼ 1maxzf AðzÞ þ BðzÞ 1j jg: ð10Þ
Note that the disjoint relation is symmetric. Moreover,
disjointðA;AcÞ ¼ 1, where Ac is the fuzzy complement of
object A, i.e., Ac ðzÞ ¼ 1 AðzÞ. However, disjointðA;AÞmay not always be zero.
. meet:
meetðA;BÞ ¼ minfð1maxzð AoðzÞ þ Bo ðzÞ 1j jÞÞ;maxvðminð�AðvÞ; �BðvÞÞÞg:
ð11Þ
Here, the first term on the righthand side of the
expression corresponds to the condition that Ao and Bo
are disjoint and the second term is associated with the
requirement that there exists some common boundary
point. The fuzzy relation ”meet” thus defined is symmetric.. equal: Two objects A and B are said to be equal if the
pixels have equal possibility of belonging to either of them.
equalðA;BÞ ¼ 1maxz AðzÞ BðzÞj j: ð12Þ
The fuzzy equal relation thus defined is a resemblance
relation, i.e., it is reflexive and symmetric.. overlap: An object A overlaps another object B if there
exists a pixel in Ao and Bo, AND there exists a pixel in Ao
and �B, AND there exists a pixel in �A and Bo, AND thereexists a pixel in �A and �B. Hence,
overlapðA;BÞ ¼ minfmaxzðminðBo ðzÞ; Ao ðzÞÞÞ;maxzðminðAoðzÞ; �BðzÞÞÞ;maxzðminðBoðzÞ; �AðzÞÞÞ;maxzðminð�AðzÞ; deltaBðzÞÞÞg:
ð13Þ
The fuzzy relation overlap is symmetric.. contains: An object A contains another object B if the
presence of the pixel in B implies that it is in Ao.Accordingly,
containsðA;BÞ ¼ minð1;minzð1þ Ao ðzÞ BðzÞÞÞ: ð14Þ
Note, containsðAo;AoÞ ¼ 1 and containsðA;BÞ ¼ 1, when-
ever 8z; AoðzÞ � BðzÞ. Also, if there exists a pixel in the
boundary of A, with �AðzÞ ¼ 1; containsðA;AoÞ ¼ 0:. inside: This follows from contains.
insideðB;AÞ ¼ containsðA;BÞ:
. covers: An object A is said to cover another object B if, forany pixel z, presence in B implies presence in A and thereexists a pixel in �A which is not in B.
coversðA;BÞ ¼ minfminð1;minzð1þ AðzÞ BðzÞÞÞ;maxvðminð�AðvÞ; 1 BðvÞÞÞg:
ð15Þ
. covered-by: This is the dual of covers.
It may be noted that, with precisely defined objects, i.e., with
binary membership functions, the fuzzy topological relations
reduce to their classical counterparts discussed in Section 2.
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002 1187
4 GRAPH MODEL FOR AN IMAGE AND SIMILARITY
So far, we have considered spatial and topological relations
between a pair of objects in isolation. In an image, many objects
may be present. For a proper description of the image, it is
therefore necessary to consider the spatial and topological relations
among all these objects. With this in view, we treat an image as a
labeled graph with a vertex corresponding to each object. An edge
between two nodes, say A and B, is labeled by the spatial and
topological relations that hold between the objects corresponding
to A and B. Graphical representation of objects in an image has
been proposed by Petrakis and Faloutsos [8]. The proposed
graphical model of an image allows fuzzy spatial and topological
relations between objects to be represented. An image (a two-
dimensional scene) consists of a collection of objects which
correspond to the nodes of the image graph. The edges capture
interobject relationships through the following information:
1. Spatial Information: The spatial relations that holdbetween the object pair associated with the edge. It storesthe fuzzy membership values for these relations.
2. Topological Information: The topological relations andtheir associated membership values that hold between theobjects.
3. Euclidean distance between the centroids of the twoobjects (edge length).
We would like to emphasize that the proposed model allows us
to represent more than one spatial and topological relation
between objects. Thus, one may have a scene in which a car is
left-of and below a house. Again, from a medical picture it may not
be clear whether a tumor is inside the pancreas or is covered by it.
In this case, both the topological relations inside and covered-by
between the objects tumor and pancreas will have nonzero
membership values.
4.1 Similarity Measure
With the logical representation of images thus defined, we are now
in a position to examine the problem of associative retrieval of
images from image databases. In order to support such retrieval, it
is necessary to define a suitable similarity measure between the
query image and the target image stored in the database. In the
image database literature, several such similarity measures have
been proposed for associative retrieval of images [3], [4], [5], [1].
However, adequate importance has not been given to the spatial
and topological relations while defining such similarity measures.
Based on the graph model of an image proposed above, we define
the fuzzy similarity measure (SIM) between two images, I1 (query
image) and I2 (an image stored in the database), as follows: We use
S and T as the sets of spatial and topological relations,
respectively, between two objects, GðOI;EIÞ is the graph of an
image I, where OI is the set of nodes (objects) and EI is the set of
edges. An edge e 2 EI between two objects A;B 2 OI is associated
with the membership values of the spatial Se � S and topological
relations Te � T that hold between A and B and the Euclidean
distance De between their centroids.To find the similarity between two images I1 and I2 with
associated sets of graphs GðO1; E1Þ and GðO2; E2Þ, the similarity
between them, we first need to a find suitable association among
the objects in the two images. Such an association can be expressed
in terms of a graph homomorphism: : GðO1; E1Þ!GðO2; E2Þ. maps a node (object) A 2 O1 to a node ðAÞ 2 O2 and, thereby, an
edge e1 2 E1 to an edge ðe1Þ 2 E2. Once a mapping is selected,
we can estimate the similarity between I1 and ðI1Þ. Such a
similarity estimate is assumed to consist of four components:
. Object Similarity. Determines the extent of similaritybetween the objects A and ðAÞ, 8A 2 O1. Accordingly, wewrite,
Obj Sim ðO1; ðO1ÞÞ¼XA2O1
ðwAobjðA; ðAÞÞÞ=XA2O1
wA;
ð16Þ
where objðA;BÞ is a fuzzy similarity between two objects
A and B. Such a similarity measure can be defined based
on the similarity in color and other attributes of the objects
under consideration, like the distance computation
between color histograms, as used in [12], [13]. The weight
value wA;A 2 O1 indicates the importance given to A
while computing object similarity.. Spatial Similarity. Determines to what extent spatial
relations between the objects in I1 match with thosebetween the corresponding objects in ðI1Þ. This isestimated as follows:
Spatial Sim ðE1; ðE1ÞÞ ¼Xe2E1
Xs2Se
we;sminðsðA;BÞ; sð ðAÞ; ðBÞÞÞ=Xe2E1
Xs2Se
we;s;
ð17Þ
where the edge e 2 E1 is assumed to be between the nodes
(objects) A and B in O1 and sðA;BÞ is the membership
value of the spatial relation s 2 Se between A and B. The
weight we;s represents the importance given to the spatial
relation s associated with the edge e 2 E1.. Topological Similarity. Determines to what extent topo-
logical relations between the objects in I1 match with those
between the corresponding objects in ðI1Þ. Using a similar
notation as above,
Topo Sim ðE1; ðE1ÞÞ ¼Xe2E1
Xt2Te
we;tminðtðA;BÞ; tð ðAÞ; ðBÞÞ=Xe2E1
Xt2Te
we;t:
ð18Þ
. Distance Similarity. Gives the similarity in distancebetween a pair of objects in I1 and that between thecorresponding objects in ðI1Þ. This is estimated as follows:
Dist Sim ðE1; ðE1ÞÞ ¼Xe2E1
wedistðe; ðeÞÞ=Xe2E1
we; ð19Þ
where, for a pair of edges e1 and e2, distðe1; e2Þ ¼ ed and
d is the absolute value of the difference in the length of the
two edges.
These four similarity measures are combined to provide an
estimate of similarity between the image I1 and its homomorphic
map ðI1Þ according to the following expression.
SIM ðI1; ðI1ÞÞ ¼ - Obj Sim ðO1; ðO1ÞÞþ . Spatial Sim ðE1; ðE1ÞÞþ / Topo Sim ðE1; ðE1ÞÞþ � Dist Sim ðE1; ðE1ÞÞ;
ð20Þ
where -þ . þ / þ � ¼ 1.Finally, the similarity between any two pair of images I1 and I2
is defined to be the best mapping,
SIMðI1; I2Þ ¼ max ðSIM ðI1; ðI1ÞÞÞ:
1188 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002
The problem of associative retrieval of images has thus been
reduced to finding a best possible mapping between the graph of
the query image and that of the target images in the database using
the fuzzy membership functions for the relations. Though the
problem of graph isomorphism is in NP, yet, in the present case, it
is simplified by the fact that the vertices are labeled by the names
of the objects. Further, if we consider cases where multiple
instances of the same object are not allowed in an image, the
graph isomorphism problem essentially becomes a �ðn2Þ problem
because then there exists just one possible mapping.
4.2 Implementation
The associative retrieval is based on the image similarity measurediscussed in Section 4 and is computed using (20). The module thatestimates the similarity measure between two image graphs isintegrated with the overall query processing package. Each imagein the database is associated with a graph representation which isidentified by a unique graph_id. While estimating the imagesimilarity, the graph models of the associated images are fetchedfirst. The graph_ids of the images, which have similarity with thequery image (above a given threshold value), are used to fetch thetarget images in order of their similarity. Since a graphrepresentation requires much less storage space than the originalimage, usually a single disk access will be sufficient to fetch thegraph of an image.
Indices are built to support faster retrieval of image graphs. Theindices are based on attribute values (e.g., color histogram) of theobjects (nodes) present in the image, as well as spatial andtopological relations that hold among the objects. Since the samespatial/topological relations may hold between multiple nodes,the index based on such relations will only identify the graph_idsof the images where such a relation holds among some pair ofobjects in the image. It may be noted that the triangle inequalitydoes not necessarily hold for the similarity measure between imagegraphs since the topological relations are not transitive. The systemis being used for associative retrieval from a medical imagedatabase which contains X-Ray, ultra-sonogram, and images ofblood slides and skin patches. A medical database systemdeveloped using the proposed scheme is being used in atelemedicine project for supporting online teleconsultation amongdoctors at remote health centers and referral hospitals. Detailedperformance evaluation of the proposed scheme is under progress.
5 CONCLUSION
We have proposed a fuzzy object-oriented data-model for imagesthat extends Egenhofer’s 4-intersection model to deal with objectshaving imprecise boundaries found in many real-life images. Inorder to support associative retrieval from image, the proposedgraph model has been strengthened with suitable fuzzy similaritymeasures involving object attributes, spatial, and topologicalrelations among the objects. We have shown how, by appropriateselection of the weights, the user can focus on different features ofthe image.
REFERENCES
[1] V.N. Gudivada, “A Geometry-Based Representation for Efficient andEffective Retrieval of Images by Spatial Similarity,” IEEE Trans. Knowledgeand Data Eng., vol. 10, no. 3, pp. 504-512, May/June 1998.
[2] M. Egenhofer and J. Herring, “Categorizing Binary Topological Relationsbetween Regions, Lines and Points in Geographic Databases,” technicalreport, Dept. of Surveying Eng., Univ. of Maine, 1990.
[3] S. Santini and R. Jain, “Similarity Matching,” IEEE Trans. Pattern Analysisand Machine Intelligence, 1999.
[4] S. Santini and R. Jain, “Similarity Queries in Image Databases,” Proc. IEEEInt’l Conf. Computer Vision and Pattern Recognition, (CVPR ’96), June 1996.
[5] Y. Cheng, V. Wu, R. Collins, A. Hanson, and E. Riseman, “Maximum-Weight Bipartite Matching Technique and its Application in Image FeatureMatching,” Proc. SPIE Conf. Visual Comm. and Image Processing, 1996.
[6] S.K. Chang and E. Jungert, Symbolic Projection for Image Information Retrievaland Spatial Reasoning. Academic Press, 1996.
[7] M. Flickener and H. Swahaney, “Query by Image and Video Content: TheQBIC System,” Computer, vol. 28, no. 9, Sept. 1995.
[8] E. Petrakis and C. Faloutsos, “Similarity Matching in Large ImageDatabases,” technical report, Computer Science Dept., Univ. of Maryland,Dec. 1994.
[9] V. Ogle and M. Stonebraker, “Chabot: Retrieval from a Relational Databaseof Images,” Computer, vol. 28, no. 9, pp. 40-45, Sept. 1995.
[10] A. Pentland, R. Picard, and S. Sclaroff, “Photobook: Tools for Content-Based Manipulation of Image Databases,” Int’l J. Computer Vision (IJCV),vol. 18, no. 3, pp. 233-254, June 1996.
[11] M. Christel, S. Stevens, and H. Wactlar, “Informedia Digital Video Library,”Proc. ACM Multimedia Conf., pp. 480-481, Oct. 1994.
[12] J.R. Smith and S.-F. Chang, “Tools and Techniques for Color ImageRetrieval,” technical report, Columbia Univ., June 1995.
[13] M. Swain and D. Ballard, “Color Indexing,” Int’l J. Computer Vision, vol. 7,no. 1, pp. 11-32, 1991.
[14] Home Page of Fuzzy Image Processing, http://ipe.et.uni-magdeburg.de/~hamid/FuzzyImageProcessing.htm, June 1997.
. For more information on this or any computing topic, please visit ourDigital Library at http://computer.org/publications/dlib.
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002 1189
Top Related