Concise Papers __________________________________________________________________________________________

An Object-Oriented Fuzzy Data Model forSimilarity Detection in Image Databases

Arun K. Majumdar, Senior Member, IEEE,Indrajit Bhattacharya, and Amit K. Saha

Abstract—In this paper, we introduce a fuzzy set theoretic approach for dealing

with uncertainty in images in the context of spatial and topological relations

existing among the objects in the image. We propose an object-oriented graph

theoretic model for representing an image and this model allows us to assess the

similarity between images using the concept of (fuzzy) graph matching. Sufficient

flexibility has been provided in the similarity algorithm so that different features of

an image may be independently focused upon.

Index Terms—Image databases, fuzzy data model, spatial and topological

relations, similarity detection.

�

1 INTRODUCTION

WITH the advancement of multimedia applications, image data-bases have emerged as a major field of research [7], [9], [10], [11].Unlike the size of a record in a conventional database, the size of animage is usually very large. This has necessitated the developmentof efficient storage techniques for image databases. Moreover, thequeries in an image database often require retrieval based on thesimilarity with given images or based on certain attributes of theobjects present in the images. Thus, it is necessary to supportqueries based on image semantics rather than based on mere pixel-to-pixel matching. The image database systems should thereforeallow adequate abstraction mechanisms for capturing higher levelsemantics of the images in order to support content addressabilityas far as possible.

Over the last decade, there has been considerable interest in

representing spatial and topological relations among objects

present in a scene. Chang and Jungert [6] have proposed 2D

Strings and Interval Projection Strings to represent spatial relation-

ships among objects in a scene. Similarly, Gudivada [1] has

suggested a geometric approach, called �R Strings, for capturing

spatial relationships. These representation schemes using 2D or

�R Strings, however, are not efficient for capturing topological

relations like meet, overlap, cover, contain, etc. Considerable work

has been done in this particular direction by Egenhofer and

Herring [2] with GIS design and developments.In real-life images, it is often not possible to precisely identify

the boundaries of objects in a scene. Thus, object identification byimage processing techniques is difficult and identification ofbinary relations, like left-of, contain, or overlap, inexact. In theimage processing literature, statistical or fuzzy set theoreticapproaches have been advocated to deal with noise or uncertaintyin images. Several algorithms have been developed to filter outnoise and identify boundaries of objects in such unfavorable

environments [14]. In this paper, we will be using a fuzzy settheoretic approach to represent objects and their spatial andtopological relationships. This will enable us to deal with thesemantics of real-life images having objects with noncrispboundaries. Moreover, object relationships, such as left-of orcovered-by, with fuzzy qualifiers, like more-or-less, almost, etc.,can also be handled.

2 SPATIAL AND TOPOLOGICAL RELATIONS

Spatial relations like left, right, above, and below define the spatialorientations of the domain objects with respect to each other andhelp in ascribing meaning to a scene. On the other hand,topological relations deal with the nature of overlap betweenobjects and are invariant under rotations of the scene. Egenhoferand Herring [2] have defined interior ðAoÞ and boundary ð�AÞ of aregion (n-cell) A. The topological relations between two regions(objects) A and B are defined in terms of the intersections (ornonintersections) of the interior and boundary of the regions underconsideration. This model is called the 4-intersection model [2] andcan be concisely represented by a 2� 2 matrix:

RðA;BÞ ¼ Ao \Bo Ao \ �B�A \ Bo �A \ �B

� �:

Egenhofer and Herring used this approach to represent eight

basic topological relations among the regions without holes,namely, disjoint, cover, meet, inside, contains, coveredby, overlap, and

equal. Some pairs such as cover and coveredby are duals of each other

(the same is true for inside and contains). Further, disjoint, meet,

overlap, and equal are symmetric. The four empty/nonempty

intersections describe a set of relations that provides completecoverage. These relations are mutually exclusive so that the union

(OR) of all specifications is true and the intersection (AND) of any

two specified relations is identically false.

3 A FUZZY OBJECT DATA MODEL FOR IMAGES

We have already mentioned that, with imprecise or noisy data, it isoften not possible to say definitely whether a pixel belongs to anobject. Rather, an object may be looked upon as a fuzzy subset ofthe set of pixels. Accordingly, let O be the set of objects present in atwo-dimensional scene. An object A 2 O is treated as a fuzzysubset of pixels of the scene with membership function A. Thus,for a pixel located at a point z in the scene, the possibility of thispixel belonging to the object A is AðzÞ. In the following, we usethe notations zX and zY to denote the X and Y coordinates of apixel z. Let zþ denote the set of 8-neighbors of a pixel at point z.Then, the boundary (�A) of A may also be defined as a fuzzy subsetwith membership function

�AðzÞ ¼ minfAðzÞ;maxu2zþf1 AðuÞgg::::::::ðaÞ:

Following this approach, the definition of membership function

for the interior Ao of an object A is given by

AoðzÞ ¼ minfAðzÞ; 1 �AðzÞg::::::::ðbÞ:

In the proposed framework, the spatial and topological

relations among the objects will also be imprecise and will betreated as fuzzy subsets.

3.1 Fuzzy Spatial Relations

The spatial relations left of , right of , above, or below between a

given pixel and an object would be imprecise and will be treated as

fuzzy subsets. The corresponding relations between any two given

1186 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002

. A.K. Majumdar is with the Department of Computer Science andEngineering, Indian Institute of Technology, Kharagpur, West Bengal,721 302, India. E-mail: akmj@cse.iitkgp.ernet.in.

. I. Bhattacharya is with the Department of Computer Science, University ofMaryland, College Park, MD 20742. E-mail: indrajit@cs.umd.edu.

. A.K. Saha is with Rice University, 6100 Main, Houston, TX 77005.E-mail: amsaha@rice.edu.

Manuscript received 1 Dec. 1999; revised 9 Jan. 2001; accepted 29 May 2001.For information on obtaining reprints of this article, please send e-mail to:tkde@computer.org, and reference IEEECS Log Number 111010.

1041-4347/02/$17.00 � 2002 IEEE

pixels are, however, precise, i.e., a pixel a is either to the left or

right or has the same X-coordinate as another pixel b. Thus, we

define

leftof ða; bÞ ¼0 if ax � bx1 else;

�ð1Þ

rightof ða; bÞ ¼0 if ax � bx1 else:

�ð2Þ

The spatial relations above and below between two pixels can be

similarly defined by considering the Y coordinates of the pixels.We now define the possibility of a pixel z being to the left-of an

object A by the fuzzy predicate “no pixel of A is to the left-of the

pixel z.”

leftof ðz; AÞ ¼ 1maxvfminðleftofðv; zÞ; AðvÞÞg: ð3Þ

Similarly,

rightof ðz; AÞ ¼ 1maxvfminðrightof ðv; zÞ; AðvÞÞg: ð4Þ

Based on these definitions, it may be seen that the possibility of

A being left-of a pixel z has the same value as that of z being right-

of A. This follows from the observation that

leftof ðA; zÞ ¼ 1maxvfminðleftof ðz; vÞ; AðvÞÞg ð5Þ

and, for any two pixels v and z, rightofðv; zÞ ¼ leftof ðz; vÞ.Hence, for any pixel z and an object A, we have

rightofðz; AÞ ¼ leftofðA; zÞ:::::::ðcÞ:

However, unlike the classical case, the fuzzy left-of and right-of

relations between a pixel and an object are not complementary, i.e.,

leftof ðz; AÞ 6¼ 1 rightof ðz; AÞ::::::::::ðdÞ:

We can now treat the spatial relations left-of and right-of between

two objects (A and B) as fuzzy subsets of O�O. Accordingly, the

fuzzy relation “A left-of B” is defined by the fuzzy predicate “no

pixel of B is to the left-of A.” Thus,

leftofðA;BÞ ¼ 1maxzfminðBðzÞ; leftof ðz;AÞÞg: ð6Þ

Similarly,

rightof ðA;BÞ ¼ 1maxzfminðBðzÞ; rightof ðz; AÞÞ: ð7Þ

The fuzzy spatial relations above or below between two objects can

be similarly defined by extending the same relations between two

pixels.

Lemma 3.1. The left-of (right-of) relation between two objects is

transitive.

leftof ðA;CÞ � maxBfminðleftof ðA;BÞ; leftof ðB;CÞÞg: ð8Þ

The proof could not be included due to space constraints. The

transitivity claim can also be made for the fuzzy relation above

(below) between two objects. Note that, to enforce symmetry for

the left-of and right-of relations between objects, we may modify

the definitions as follows:

leftofðA;BÞ¼min½f1maxzðminðBðzÞ; leftofðz; AÞÞÞg;f1maxzðminðAðzÞ; rightof ðz; BÞÞÞg�:

ð9Þ

3.2 Fuzzy Topological Relations

We are now in a position to define fuzzy extensions of the

topological relations discussed in Section 2.

. disjoint: Two objects A and B are said to be disjoint if allpixels in A are not in B and vice versa.

disjointðA;BÞ ¼ 1maxzf AðzÞ þ BðzÞ 1j jg: ð10Þ

Note that the disjoint relation is symmetric. Moreover,

disjointðA;AcÞ ¼ 1, where Ac is the fuzzy complement of

object A, i.e., Ac ðzÞ ¼ 1 AðzÞ. However, disjointðA;AÞmay not always be zero.

. meet:

meetðA;BÞ ¼ minfð1maxzð AoðzÞ þ Bo ðzÞ 1j jÞÞ;maxvðminð�AðvÞ; �BðvÞÞÞg:

ð11Þ

Here, the first term on the righthand side of the

expression corresponds to the condition that Ao and Bo

are disjoint and the second term is associated with the

requirement that there exists some common boundary

point. The fuzzy relation ”meet” thus defined is symmetric.. equal: Two objects A and B are said to be equal if the

pixels have equal possibility of belonging to either of them.

equalðA;BÞ ¼ 1maxz AðzÞ BðzÞj j: ð12Þ

The fuzzy equal relation thus defined is a resemblance

relation, i.e., it is reflexive and symmetric.. overlap: An object A overlaps another object B if there

exists a pixel in Ao and Bo, AND there exists a pixel in Ao

and �B, AND there exists a pixel in �A and Bo, AND thereexists a pixel in �A and �B. Hence,

overlapðA;BÞ ¼ minfmaxzðminðBo ðzÞ; Ao ðzÞÞÞ;maxzðminðAoðzÞ; �BðzÞÞÞ;maxzðminðBoðzÞ; �AðzÞÞÞ;maxzðminð�AðzÞ; deltaBðzÞÞÞg:

ð13Þ

The fuzzy relation overlap is symmetric.. contains: An object A contains another object B if the

presence of the pixel in B implies that it is in Ao.Accordingly,

containsðA;BÞ ¼ minð1;minzð1þ Ao ðzÞ BðzÞÞÞ: ð14Þ

Note, containsðAo;AoÞ ¼ 1 and containsðA;BÞ ¼ 1, when-

ever 8z; AoðzÞ � BðzÞ. Also, if there exists a pixel in the

boundary of A, with �AðzÞ ¼ 1; containsðA;AoÞ ¼ 0:. inside: This follows from contains.

insideðB;AÞ ¼ containsðA;BÞ:

. covers: An object A is said to cover another object B if, forany pixel z, presence in B implies presence in A and thereexists a pixel in �A which is not in B.

coversðA;BÞ ¼ minfminð1;minzð1þ AðzÞ BðzÞÞÞ;maxvðminð�AðvÞ; 1 BðvÞÞÞg:

ð15Þ

. covered-by: This is the dual of covers.

It may be noted that, with precisely defined objects, i.e., with

binary membership functions, the fuzzy topological relations

reduce to their classical counterparts discussed in Section 2.

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002 1187

4 GRAPH MODEL FOR AN IMAGE AND SIMILARITY

So far, we have considered spatial and topological relations

between a pair of objects in isolation. In an image, many objects

may be present. For a proper description of the image, it is

therefore necessary to consider the spatial and topological relations

among all these objects. With this in view, we treat an image as a

labeled graph with a vertex corresponding to each object. An edge

between two nodes, say A and B, is labeled by the spatial and

topological relations that hold between the objects corresponding

to A and B. Graphical representation of objects in an image has

been proposed by Petrakis and Faloutsos [8]. The proposed

graphical model of an image allows fuzzy spatial and topological

relations between objects to be represented. An image (a two-

dimensional scene) consists of a collection of objects which

correspond to the nodes of the image graph. The edges capture

interobject relationships through the following information:

1. Spatial Information: The spatial relations that holdbetween the object pair associated with the edge. It storesthe fuzzy membership values for these relations.

2. Topological Information: The topological relations andtheir associated membership values that hold between theobjects.

3. Euclidean distance between the centroids of the twoobjects (edge length).

We would like to emphasize that the proposed model allows us

to represent more than one spatial and topological relation

between objects. Thus, one may have a scene in which a car is

left-of and below a house. Again, from a medical picture it may not

be clear whether a tumor is inside the pancreas or is covered by it.

In this case, both the topological relations inside and covered-by

between the objects tumor and pancreas will have nonzero

membership values.

4.1 Similarity Measure

With the logical representation of images thus defined, we are now

in a position to examine the problem of associative retrieval of

images from image databases. In order to support such retrieval, it

is necessary to define a suitable similarity measure between the

query image and the target image stored in the database. In the

image database literature, several such similarity measures have

been proposed for associative retrieval of images [3], [4], [5], [1].

However, adequate importance has not been given to the spatial

and topological relations while defining such similarity measures.

Based on the graph model of an image proposed above, we define

the fuzzy similarity measure (SIM) between two images, I1 (query

image) and I2 (an image stored in the database), as follows: We use

S and T as the sets of spatial and topological relations,

respectively, between two objects, GðOI;EIÞ is the graph of an

image I, where OI is the set of nodes (objects) and EI is the set of

edges. An edge e 2 EI between two objects A;B 2 OI is associated

with the membership values of the spatial Se � S and topological

relations Te � T that hold between A and B and the Euclidean

distance De between their centroids.To find the similarity between two images I1 and I2 with

associated sets of graphs GðO1; E1Þ and GðO2; E2Þ, the similarity

between them, we first need to a find suitable association among

the objects in the two images. Such an association can be expressed

in terms of a graph homomorphism: : GðO1; E1Þ!GðO2; E2Þ. maps a node (object) A 2 O1 to a node ðAÞ 2 O2 and, thereby, an

edge e1 2 E1 to an edge ðe1Þ 2 E2. Once a mapping is selected,

we can estimate the similarity between I1 and ðI1Þ. Such a

similarity estimate is assumed to consist of four components:

. Object Similarity. Determines the extent of similaritybetween the objects A and ðAÞ, 8A 2 O1. Accordingly, wewrite,

Obj Sim ðO1; ðO1ÞÞ¼XA2O1

ðwAobjðA; ðAÞÞÞ=XA2O1

wA;

ð16Þ

where objðA;BÞ is a fuzzy similarity between two objects

A and B. Such a similarity measure can be defined based

on the similarity in color and other attributes of the objects

under consideration, like the distance computation

between color histograms, as used in [12], [13]. The weight

value wA;A 2 O1 indicates the importance given to A

while computing object similarity.. Spatial Similarity. Determines to what extent spatial

relations between the objects in I1 match with thosebetween the corresponding objects in ðI1Þ. This isestimated as follows:

Spatial Sim ðE1; ðE1ÞÞ ¼Xe2E1

Xs2Se

we;sminðsðA;BÞ; sð ðAÞ; ðBÞÞÞ=Xe2E1

Xs2Se

we;s;

ð17Þ

where the edge e 2 E1 is assumed to be between the nodes

(objects) A and B in O1 and sðA;BÞ is the membership

value of the spatial relation s 2 Se between A and B. The

weight we;s represents the importance given to the spatial

relation s associated with the edge e 2 E1.. Topological Similarity. Determines to what extent topo-

logical relations between the objects in I1 match with those

between the corresponding objects in ðI1Þ. Using a similar

notation as above,

Topo Sim ðE1; ðE1ÞÞ ¼Xe2E1

Xt2Te

we;tminðtðA;BÞ; tð ðAÞ; ðBÞÞ=Xe2E1

Xt2Te

we;t:

ð18Þ

. Distance Similarity. Gives the similarity in distancebetween a pair of objects in I1 and that between thecorresponding objects in ðI1Þ. This is estimated as follows:

Dist Sim ðE1; ðE1ÞÞ ¼Xe2E1

wedistðe; ðeÞÞ=Xe2E1

we; ð19Þ

where, for a pair of edges e1 and e2, distðe1; e2Þ ¼ ed and

d is the absolute value of the difference in the length of the

two edges.

These four similarity measures are combined to provide an

estimate of similarity between the image I1 and its homomorphic

map ðI1Þ according to the following expression.

SIM ðI1; ðI1ÞÞ ¼ - Obj Sim ðO1; ðO1ÞÞþ . Spatial Sim ðE1; ðE1ÞÞþ / Topo Sim ðE1; ðE1ÞÞþ � Dist Sim ðE1; ðE1ÞÞ;

ð20Þ

where -þ . þ / þ � ¼ 1.Finally, the similarity between any two pair of images I1 and I2

is defined to be the best mapping,

SIMðI1; I2Þ ¼ max ðSIM ðI1; ðI1ÞÞÞ:

1188 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002

The problem of associative retrieval of images has thus been

reduced to finding a best possible mapping between the graph of

the query image and that of the target images in the database using

the fuzzy membership functions for the relations. Though the

problem of graph isomorphism is in NP, yet, in the present case, it

is simplified by the fact that the vertices are labeled by the names

of the objects. Further, if we consider cases where multiple

instances of the same object are not allowed in an image, the

graph isomorphism problem essentially becomes a �ðn2Þ problem

because then there exists just one possible mapping.

4.2 Implementation

The associative retrieval is based on the image similarity measurediscussed in Section 4 and is computed using (20). The module thatestimates the similarity measure between two image graphs isintegrated with the overall query processing package. Each imagein the database is associated with a graph representation which isidentified by a unique graph_id. While estimating the imagesimilarity, the graph models of the associated images are fetchedfirst. The graph_ids of the images, which have similarity with thequery image (above a given threshold value), are used to fetch thetarget images in order of their similarity. Since a graphrepresentation requires much less storage space than the originalimage, usually a single disk access will be sufficient to fetch thegraph of an image.

Indices are built to support faster retrieval of image graphs. Theindices are based on attribute values (e.g., color histogram) of theobjects (nodes) present in the image, as well as spatial andtopological relations that hold among the objects. Since the samespatial/topological relations may hold between multiple nodes,the index based on such relations will only identify the graph_idsof the images where such a relation holds among some pair ofobjects in the image. It may be noted that the triangle inequalitydoes not necessarily hold for the similarity measure between imagegraphs since the topological relations are not transitive. The systemis being used for associative retrieval from a medical imagedatabase which contains X-Ray, ultra-sonogram, and images ofblood slides and skin patches. A medical database systemdeveloped using the proposed scheme is being used in atelemedicine project for supporting online teleconsultation amongdoctors at remote health centers and referral hospitals. Detailedperformance evaluation of the proposed scheme is under progress.

5 CONCLUSION

We have proposed a fuzzy object-oriented data-model for imagesthat extends Egenhofer’s 4-intersection model to deal with objectshaving imprecise boundaries found in many real-life images. Inorder to support associative retrieval from image, the proposedgraph model has been strengthened with suitable fuzzy similaritymeasures involving object attributes, spatial, and topologicalrelations among the objects. We have shown how, by appropriateselection of the weights, the user can focus on different features ofthe image.

REFERENCES

[1] V.N. Gudivada, “A Geometry-Based Representation for Efficient andEffective Retrieval of Images by Spatial Similarity,” IEEE Trans. Knowledgeand Data Eng., vol. 10, no. 3, pp. 504-512, May/June 1998.

[2] M. Egenhofer and J. Herring, “Categorizing Binary Topological Relationsbetween Regions, Lines and Points in Geographic Databases,” technicalreport, Dept. of Surveying Eng., Univ. of Maine, 1990.

[3] S. Santini and R. Jain, “Similarity Matching,” IEEE Trans. Pattern Analysisand Machine Intelligence, 1999.

[4] S. Santini and R. Jain, “Similarity Queries in Image Databases,” Proc. IEEEInt’l Conf. Computer Vision and Pattern Recognition, (CVPR ’96), June 1996.

[5] Y. Cheng, V. Wu, R. Collins, A. Hanson, and E. Riseman, “Maximum-Weight Bipartite Matching Technique and its Application in Image FeatureMatching,” Proc. SPIE Conf. Visual Comm. and Image Processing, 1996.

[6] S.K. Chang and E. Jungert, Symbolic Projection for Image Information Retrievaland Spatial Reasoning. Academic Press, 1996.

[7] M. Flickener and H. Swahaney, “Query by Image and Video Content: TheQBIC System,” Computer, vol. 28, no. 9, Sept. 1995.

[8] E. Petrakis and C. Faloutsos, “Similarity Matching in Large ImageDatabases,” technical report, Computer Science Dept., Univ. of Maryland,Dec. 1994.

[9] V. Ogle and M. Stonebraker, “Chabot: Retrieval from a Relational Databaseof Images,” Computer, vol. 28, no. 9, pp. 40-45, Sept. 1995.

[10] A. Pentland, R. Picard, and S. Sclaroff, “Photobook: Tools for Content-Based Manipulation of Image Databases,” Int’l J. Computer Vision (IJCV),vol. 18, no. 3, pp. 233-254, June 1996.

[11] M. Christel, S. Stevens, and H. Wactlar, “Informedia Digital Video Library,”Proc. ACM Multimedia Conf., pp. 480-481, Oct. 1994.

[12] J.R. Smith and S.-F. Chang, “Tools and Techniques for Color ImageRetrieval,” technical report, Columbia Univ., June 1995.

[13] M. Swain and D. Ballard, “Color Indexing,” Int’l J. Computer Vision, vol. 7,no. 1, pp. 11-32, 1991.

[14] Home Page of Fuzzy Image Processing, http://ipe.et.uni-magdeburg.de/~hamid/FuzzyImageProcessing.htm, June 1997.

. For more information on this or any computing topic, please visit ourDigital Library at http://computer.org/publications/dlib.

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 14, NO. 5, SEPTEMBER/OCTOBER 2002 1189