������������������������������������������ �!�����#"%$'& �(&��!)+* ,��-.�)�/0&213������& �4���5"���6

B.G.Prasad,K.K.BiswasandS.K.GuptaDepartmentof ComputerScience& Engineering

IndianInstituteof Technology, New Delhi, INDIA7bgprasad,kkb,skg8 @cse.iitd.ernet.in

9;:=A@CBED�>In this paperwe presenta techniqueto retrieve imagesby regionmatchingoncolor, shapeandlocation.Weextractup to threedom-inantregionsin theimageusingcolor clustersin RGB space.Eachregion is indexedwith integratedcolor, shapeandlocationfeaturesaswell asvariouscombinationsof region featuresarealsoindexed.Theresultingindicesandtheirrelatedmetadataarestoredin aHashstructure.Imagesareretrievedbasedon thecolor, shapeandloca-tion similarity with thatof thequeriedimage.Theshapefeatureisnormalizedto make it invariantto translation,rotationandscaling.Theretrieval processis non-cascading andimagescanberetrievedbasedon color or shapeor locationor basedon a combinedcolor-shapeor color-shape-locationindex. Resultsobtainedshow thatthe retrieval effectiveness increasesin non-cascaded region-basedqueryingby combinedcolor-shape-locationindex, comparedto ourearliercascadedretrieval methodbasedoncolor followedby shapematching.

Keywords: Imageindexing,Content-basedimageretrieval,Color-shape-locationindex

F GIH >A@CJ�K�L=D=>NMOJ HTherehasbeena lot of interestin content-basedimageretrieval(CBIR) usingvisual featuresover the lastdecade. An overview ofthe work canbe found in [Goodrum2000][GudivadaandRagha-van1995][Marsicoiet al. 1997][Ruiet al. 1999]. Due to the largedependenceon the useof low-level visual features,thereis scopefor automatedand efficient solutionsto this problem. Content-basedimageretrieval is likeaninformationfilter processandhenceshouldprovide a high percentage of relevant imagesfor the useruponquerying. It shouldconformto humanperceptionof visualsemantics.In general,imagefeaturestendto captureonly oneofmany aspectsof imagesimilarity andhenceit becomesdifficult toclearlyspecifywhator how ausershouldinitiatequeries.Thepres-enceof largevolumesof digital repositoriesleadsto many schemesof indexing andretrieval of suchdata(e.g. QBIC [Flickner et al.1995], Netra [Ma andManjunath1997], VisualSEEk[Smith andChang1996],etc). In all thesecases,theuseris interestedin seek-ing theimagessimilar to hisquery.

Color is one of the most important image indexing featuresemployed in CBIR [Flickner et al. 1995][Ma and Manjunath1997][Ravishankaret al. 1999][Smithand Chang1996]. A per-ceptualtechniquebasedoncolorcardinalityis dealtin [Androutsoset al. 1999]. Binary signaturesusingbit string representationforcolor featureis usedin [Chitkaraet al. 2000]. A grid is superim-posedon theentireimageto obtainpartition-basedimageretrieval,but is not rotationinvariant. No accessstructureis usedto supportthesignature-basedrepresentationof color. Theproposalof usingvariablenumberof color histogramsfor color representation,de-pendingonnumberof colorsin theimage,is discussedin [Stehlinget al. 2000]. It is shown to bebetterthanusingglobalhistograms.In [Ko et al. 2000],a flexible sub-blockimageretrieval algorithm

robustto translation,lighting change andobjectappearanceis pro-posed.They usea reducedcolor spaceto overcomethe problemsof global color histograms.Due to the limitations of usingcoloraloneasa global feature,usuallyotherfeatureslike shape,textureandspatiallocationareaddedto the featurespaceto enhancetheretrieval efficiency andeffectiveness.Also, theglobalcolor-basedmethodssuffer from problemsof non-invarianceandlargestoragerequirements.

Shapeis an important feature for perceptual object recogni-tion andclassificationof images.Many techniques suchaschaincode, polygonal approximations,curvature, fourier descriptors,grid-region based,radii methodandmomentdescriptorshave beenproposedandusedin variousapplications. A region-basedshaperepresentationandindexing schemethatis translation,rotationandscaleinvariantis proposedin [Lu andSajjanhar1999]. Comparedto Fouriermethod,it is shown to give betterretrieval performancebut works only on binary images. Shape-basedindexing of colorimagesis proposedin [Prasadet al. 2001b] that alsousessimilarshaperepresentationbut differentindexing andsimilarity measuressoasto obtainbetterretrieval resultsfor color imageregions.Fur-ther, along with color, shapehasbeenusedas an additionalfea-ture to index the extractedregionswithin the images. The queryis sequentiallycascadedfirst by color and thenby shape[Prasadet al. 2001a],which producesbetterresultscomparedto matchingby coloror shapealone.

In thispaperweproposeto useregion-wiselocationinformationadditionallyto index imagesto enableimageretrieval basedonspa-tial locationof regions.It alsoallowsfor matchingimagesbasedonrelative locationpositionswith respectto regionswithin animage.

To computelocationindex, we divide the imagespaceinto 3x3grid cellsandnumberthem1 to 9. Dependingon thecell numberthatis maximallycoveredby aregion, it is allocatedanindex equalto thecell number. This is thencombinedwith thecolor andshapeindicesto provide an integratedcolor-shape-locationindex allow-ing for a non-cascadedmatchingon thesecombinedfeatures.Thecompositeindexing methodis simple,computationallylessinten-siveandshown to performbetter.

Thepaperis organizedasfollows. In thenext section,webrieflyexplain the featureextraction and imagerepresentationapproachadopted. Section3 describesin detail the indexing methodologyusedto index imagesbasedon individual aswell asthecombinedcolor-shape-locationfeatures.TheHashdatastructureandsimilar-ity matchingcriteriausedareexplainedin section4. In section5,we depict the experimental resultsandperformanceevaluation ofoursystem.Conclusionsabouttheemployedschemearediscussedin section6.

P Q+R BE>AL0@ RTS�U >A@CB3D=>NMOJ H B H K G�V BEW RYXZR�[ @ R < R�H=\>NBE>NMOJ H

Our indexing approachemploys color clusteringthat is carriedoutin RGB-spaceto extract up to threedominantregions in imagesbasedon humanperceptual colors [Ravishankaret al. 1999]. It

hasbeenshown that the cardinalityof imagesby color is four ontheav] erage[Stehlinget al. 2000]. We resortto a weaksegmenta-tion, sinceimageretrieval doesnotentailsolvingthegeneralimagerecognitionproblem. It is sufficient thata retrieval systempresentsimilar imagesin someuser-definedsense.

To segmentimagesbasedon dominantcolors,a color quantiza-tion in RGB-spaceusing25perceptualcolorcategoriesis employed[Ravishankaret al. 1999]. Fromthesegmentedimageregions,wefind theenclosingminimumboundingrectangle(MBR) of eachre-gion, its location,area,color information,numberof regions,etc.All theseare storedin a metafileand usedfor matchingimagesbasedoncolor.

For eachof thedominantregionswecomputethemajoraxis,mi-nor axisandthecentroid.It is thennormalizedsuchthatthemajoraxis is parallelwith the horizontalaxis andthe width occupies96pixels. Thena grid of 96x96pixels is placedover the normalizedshaperegionwith thecentroidof theregioncoincidingwith thecen-terof thegrid. Thegrid is dividedinto 8x8cells(eachof size12x12pixels). Depending on coverageof thepixelswithin eachcell, it isassigneda binary value of 1 or 0. Thus the imageregion trans-latesinto a 64 bit uniquesequenceof 1’s and0’s. Theeccentricity(measuredin termsof thenumberof rows the region occupies)ofthe imageregion is usedto index theshapeandconstructthehashstructurebasedon color andshape.Thenwhile matching,thecov-eragealong rows and columnsare usedto pruneout non-similarregionsbasedonsomethresholdvalue.

For locationindexing, the imagespaceis divided into 3x3 gridnumbered1 to 9. The region is likely to overlapnumberof cellsin the imagespace.The index assignedis the cell numberthat ismaximally coveredby the region. If a region overlapsmorethanonegrid cell, thenthecentralareaof theregionis usedto determinethelocationindex. Theclassificationis accordingto a locationmapcontaining9 regionsof theimagespaceasshown in Figure1.

Figure1: Illustrationof Find location.

A programsegmenthighlighting the find locationprocedureisgiven below. We have consideredimagesizeof 192 x 128 pixelsin our work. The four intersectingcornersof the 9 sub locationsdependingon imagesizeareinitializedat thebeginningof thepro-gram.

^_^a`cb?dfeNb?gihkjmlfefh�loncpqprd2s?tin?u;v_dNwmgfprtmdxnyd!s;gzb?lfertmdon`_{_|}vNt_wz~?dNtfu;gNj_j_txe!nc?dNjNtxprtmdon3__2tincpqiN!+imc=5?!�xm=?!mc=c_N!+cmN_=5_N�xm=_NN_=tincp2`�dNjfN=tfs=_bO�zm_bEO?qzm_^_^zm{}v_vfv_l!smptfsNbEO�zma_bEO?qzmNz^N^z?do`;?l!smp`�dNjf� l_vNjml;tfs=_bEO�2m_bEO?q¡2m_z^N^z?l!smp¢a|}lfp2htfu_ucv_lqgon?u'p?dx`

tfs=¤£rgfpm¥¦go|�jAx¢fbEOm2¡!2£rgfp!¥¦go|�jA§x¢fbEO??_`}dNjfl_vNjmla`�dNjfm+ l_vNjml¨tfs=_§bEOx¡!mm_§bEO?qzcr_2^_^2©clon?p?lfbq?l!smp`}dNjfm+ l_vNjml¨tfs=_§bEO�zcr_bEO?q¡zcr_2^_^z?l!s_p¢a|}lfpzhtfu_ucv_l¨gon?uz|}dfp_p?dihtfs=¤£rgfpm¥¦go|�jAcN¢fbEOm2¡!2£rgfp!¥¦go|�jA§cN¢fbEO??_`}dNjf_+l_vNjmla`�dNjfNª= l_vNjml¨tfs§bEOx¡!mcrz^_^z«?dfpNp?dih?l!smp`}dNjfNª=  q^_^_lon?u;d!s¨sf{}v_vfv_l!smpl_vNjml¨tfs=bEO�¡!'??z^_^¬!{�v_vfqbrtxe!¥cp____ q^_^qlon?u;d!sq!{}v_vfqbrtxe!¥cpl_vNjml¨tfs=_bE�zm_bO?q¡zm_z^_^¬«?l!pm®rl_lonyv_l!smpgon?u¯wmloncp?lfb____ q^_^qlon?u;d!sq«?lfpm®rl_lonyv_l!smp¯gon?uwmloncp?lfbl_vNjml¨tfs=_bE�z??_bO?q¡z??_z^_^¬«?l!pm®rl_lon;brtxe!¥cpgonruwmloncp?lfb____ q^_^qlon?u;d!sq«?lfpm®rl_lon;brtxe!¥cp¯gon?uwmloncp?lfb q^_^_lon?u;d!s;gNjNj_txe!nc?dNj_txprtmdon¨`cb?dNwml!uf{cb?l

° G�H > R W�@�BE> R K±D=J² J³@ \ NMµJ H M H K R+U¶3·O¸ ¹³º»¼º}½�¾¼¿AÀAÁrÂThe color spaceis groupedinto 25 perceptualcolor categoriesin-dexed as 1,2,3, ... 25. The imageis segmentedand n dominantregionsareselected.Theoverall color index is formedasfollows:

CI Ã ∑niÄ 125n Å iCiwhereCi is theindex of the i

th region. Imageswith similar overallcolor indicesarestoredin thesamehashentryof hashindex struc-ture.Along with it, othermetadataabouttheregionarealsostored.For our studywe have consideredonly thefirst threedominantre-gions.

¶3· Æ Ç+ÈAÉ}Ê=Á¾¼¿AÀAÁrÂTheshapecategoriesSi rangefrom 1 through8 basedoneccentric-ity of thedominantregions.Theshapeindex for n dominantregionsis formedasfollows:

SI à ∑niÄ 18n Å iSiwhereS1 Ë S2 Ë S3 ˧ÌÍÌ¼Ì aretheshapeindicesof thedominantregionsindecreasingorderof size(S1 is themostdominantregion).

¶E· ¶ Îcº�ÏcÉ}Ði¾¼º¿2¾Ñ¿AÀÁrÂIt is usually desiredto retrieve imagesbasedon regions/objectspresentat particular spatial locations within the image space.Hence,we determinethe grid cell that the extractedimageregionmaximally covers in the imagespaceand assignthe correspond-ing cell numberasits locationindex. Theoverall locationindex isformedasfollows:

LI à ∑niÄ 19n Å iLiwhereL1 Ë L2 Ë L3 Ë§Ì¼Ì¼Ì arethelocationindicesof thedominantregionsin decreasingorderof size.

¶E· Ò ¹³ºÓ Ê=ºÔo¾¼ÐIÁÏNº»Ñºr½CÕCÔoÈAÉ}Ê=Á¾¼¿AÀAÁrÂCascadedretrieval of images,indexing first by color andthenbyshape,resultsin someof the imagesnot beingretrieved from thedatabase.Soa uniqueindex for imageregionsbasedon compositecolor and shapeis proposed. The compositeindex is formed asfollows:

CSI à ∑niÄ 1 Ö 25n Å iCi ×rØ 210 Ù Ö 8n Å iSi ×whereC1 Ë C2 Ë C3 Ë§Ì¼Ì¼Ì andS1 Ë S2 Ë S3 ˧ÌÍÌ¼Ì arethecolorandshapeindicesrespectively of the dominantregions in decreasingorder of size.Theabove indexing methodconsiderstheeccentricityof theregionand its homogeneous color category to form the combinedcolor-shapeindex for representationof the imageregions. Imagesarematchedfor shapesimilarity usingtherow andcolumncoverageasthediscriminatingfactorwithin somethreshold.A detailedproce-dureof theabove indexing methodis explainedin our earlierworkoncascaded retrieval by colorandshape[Prasadet al. 2001a].

¶E· Ú ¹³ºÓ Ê=ºÔo¾¼ÐIÁÏNº»Ñºr½CÕCÔoÈAÉ}Ê=ÁrÕC»¼º�ÏcÉ}ÐI¾Ñº¿'¾¼¿AÀAÁrÂThe previous combinedindex canbe extendedto provide an inte-gratedcolor-shape-locationindex to increasethematchingperfor-mance.Theintegratedindex is formedasfollows:

CSLI à ∑niÄ 1 Ö 25n Å iCi ×}Ø 220 Ù Ö 8n Å iSi ×rØ 210 Ù Ö 9n Å iLi ×whereC1 Ë C2 Ë C3 ˧ÌÍÌ¼Ì , S1 Ë S2 Ë S3 Ë§Ì¼Ì¼Ì and L1 Ë L2 Ë L3 Ë§Ì¼Ì¼Ì are the color,shapeandlocationindicesrespectively of thedominantregionsindecreasingorderof size.

Û Ü>cJ³@�BEW R B H K V BE>ND0´=M H WYD0@�MO> R @CMµBÒE·µ¸ ÝÞÉ}ÔoÈ'ÇßÐi½CàAÏNÐiàA½CÁAn imagestructurebasedon hashingtechniqueis usedto storethecombinedindex of the imageregion-wise. At query time, onlythoseimageregions that are in the samehashbucket as thoseofthequeriedimageregionsarecomparedfor similarity, thusreduc-ing thesearchspaceandtime.

A hashstructureis usedto storethekeys of all imagesbelong-ing to the samegroup in separatebuckets. An instanceof hashtableis shown in Figure2. Along with thecompositeimageindex,featuredatalike numberof dominantregions, color information,color index, area,locationindex, shapestring,shapeindex, columncoveragetotalandrow coverage totalof shaperepresentationof in-dividual regionsaswell asfor regionstakenpair wisearestoredasa vector for eachimageentry in the hashtable. Additionally, thepathnameof eachimageis alsostoredin thevector. Eachof theseconstitutesa featurevectorbasedon the combinedfeatures.Themappingof region informationto thehashindex is shown in Figure3 for all combinationsof regionswithin theimage.

An illustrationof index formationfor a singleregion is giveninthefollowing example:

Figure2: Instanceof hashtable.

Figure3: Hashindex structurefor compositecolor-shape-locationindex.

áCSL â Index à 16 Ø 220 Ù 8 Ø 210 Ù 1 à 16785409

is thecolor-shape-locationindex for a regionhaving acolor= ’16’,shape= ’8’ andlocation= ’1’. All imageregionshaving a simi-lar compositeindex will bemappedto thehashbucket with value’16785409’. Similar analysisfor imageswith multiple regions isdoneto obtainuniqueindicesfor all combinations consideringupto 3 dominantregionsin agivenimage.

Ò3· Æ Ç+¾ÑÓã¾¼»ÑÉc½C¾ÑÐCäãÓ ÁrÉ}ÔoàA½CÁ åº}½ÔoÈAÉ}Ê=ÁÓãÉ}ÐiÏNÈA¾Ñ¿AæTo find the bestmatchbetweenthe query imageand the imagesindexed to samehashlocation, we make useof row and columncoverageof individualregions.Thesimilarity measureis computedasfollows:

1. Calculatetherow coverageRC j andcolumncoverageCC j ofthe regions in the query imageby countingnumberof cellsalongeachdirection.

2. Calculatetherow coverageRCç j andcolumncoverageCCç j oftheregionsin theimageextractedfrom databaseby countingnumberof cellsalongeachdirection.

3. Find therow andcolumndifferencebetweenqueryimagere-gionsandregionsin the imageselectedfrom databaseusingtheequations:

Rd à ∑8j Ä 1 è RC j é RCç j è and Cd à ∑8jÄ 1 èCC j é CCç j è4. If Ö Rd Ù Cd ×ê T (asuitablethreshold),thentheimagesmatch.ë S�Uì[ER @�M V¨R�H >cBÞ²�@ R N

Figure4: Retrieval resultson databaseof flowers,fruits, vegetables andsimulatedfiguresbasedon (a) color (1st col, 1st & 2nd row), (b)shape(2ndcol, 1st& 2ndrow), (c) location(3rd col, 1st& 2ndrow) (d) color-shapeindex (3rd row, left) and(e) color-shape-locationindex(3rd row, right).

Figure5: Retrieval resultson databaseof flagsbasedon (a) color (1stcol, 1st& 2ndrow), (b) shape(2ndcol, 1st& 2ndrow), (c) location(3rdcol, 1st& 2ndrow) (d) color-shapeindex (3rd row, left) and(e) color-shape-locationindex (3rd row, right).

Figure6: Retrieval resultsoncombineddatabasebasedon (a) color (1stcol, 1st& 2ndrow), (b) shape(2ndcol, 1st& 2ndrow), (c) location(3rdcol, 1st& 2ndrow) (d) color-shapeindex (3rd row, left) and(e) color-shape-locationindex (3rd row, right).

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Rec

all÷

Precision

Reg-ColorColor

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Rec

all÷

Precision

Reg-ShapeShape

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Rec

all÷

Precision

Reg-ColShLocReg-ColShape

ColShape

Figure 7: Comparative graphbetweenfull-image cascadedmatchand region-basednon-cascadedmatchfor database of flowers, fruits,vegetablesandsimulatedfigureson (a)Color (b) Shapeand(c) compositeColor-Shape-locationindex.

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Rec

all÷

Precision

Reg-ColorColor

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Rec

all÷

Precision

Reg-Shape Shape

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Rec

all÷

Precision

Reg-ColShLocReg-ColShape

ColShape

Figure8: Comparativegraphbetweenfull-imagecascadedmatchandregion-basednon-cascaded matchfor databaseof flagson(a)Color (b)Shapeand(c) compositeColor-Shape-locationindex.