[IEEE 2014 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP)...

A new ontology for semantic annotation

of remotely sensed images

Wassim messaoudi, Imed Riadh Farah

RIADI Laboratory, national school of computer science, University of Manouba, 2010 Manouba, Tunisia

[email protected] [email protected]

Basel Solaiman

I.T.I Departement, Telecom Bretagne Technopôle Brest Iroise CS 83818,

29238 Brest Cedex France [email protected]

Abstract—The remotely sensed images are considered as an

important source of information which used in several domains such as environmental monitoring, disaster management and military intelligence. However, the appropriate information has to be extracted and represented to make efficient decision processes. This research presents an ontology based methodology for remote sensing imagery annotation and interpretation. We propose a fuzzy spatio-spectro-temporal ontology representing spatial knowledge about remote sensing images and natural risks domain. The proposed ontology takes into account the imperfection on spatial data, temporality of objects in natural scene and specificities of RS images. Besides, we propose to apply the developed ontology to annotate remote sensing images. The proposed process is validated using LANDSAT and SOPT images representing the kef area, situated in north-east of Tunisia.

Keywords— Remote sensing; ontology; semantic annotation;

I. INTRODUCTION

Remote sensing image interpretation becomes an active area of research for several years. However, ontology approach is more used to reduce the semantic gap between low-level features and high-level image semantics [1]. Its role is to capture domain knowledge in a generic way and to provide a commonly agreed upon understanding of a geographic domain. In ontology, concepts are the fundamental units for specification, and provide a foundation for information description. In general, each concept has three basic components: terms, attributes and relations. Terms are the names used to refer to a specific concept, and can include a set of synonyms that specify the same concepts. Attributes are features of a concept that describe the concept in more detail. Finally relations are used to represent relationships among different concepts and to provide a general structure to the ontology [2].

In this work, we propose to exploit the power of ontology in modeling and reasoning to annotate remote sensing images for reducing the semantic gap existing between image features and their semantics. Thus, we proposed an approach for image annotation basing on new ontology taking account of characteristic of the remote sensing image such as spectral, spatial and temporal dimensions. Besides, our ontology represents the imperfections about remote sensing imagery such as ambiguities of features values.

Then, we proposed an ontology development process basing on (1) the reuse of the existing ontological resources, (2) the learning from the remote sensing images, and (3) the formalization of the knowledge of domain (natural risks, knowledge on scenes, sensors of acquisition, etc.). The proposed ontology is applied to generate a meta-information describing the semantic content of the image.

The paper is organized as follow: Section 2 gives an overview of existing geographic ontology. The proposed ontology is presented in section 3. The proposed approach of image annotation is detailed in section 4. The section 5 presents an experimental study, and finally the section 6 presents our conclusions.

II. RELATED WORK

A. Geographic ontology review

In this section, we present some previous works on ontology-based image annotation. Ontology is designed to capture shared knowledge and overcome the semantic heterogeneity among domains.

In Adnani et al. (2004) [3], authors proposed a generic multi-layered ontology to describe key features of urban applications. The ontology contains layers which are composed of a generic functional structure and one or more domain ontology.

In Huang et al. (2009) [4], authors mentioned that geo-ontology should consist of such primitives as geographic concept, geographic relationship, geographic axioms and geographic individuals.

Abadie (2009) [5] specified that the spatial temporal ontology contains a geographic concepts located in space (such as river, bridge, house, and desert) and time.

Bach et al. (2007) [6] proposed to construct geo-ontology basing on schema matching between geographic maps and geographic database.

The ontology presented in Durand et al. (2007) [7] attempts to represent urban objects in satellite images. This ontology is composed of 91 concepts, 20 attributes (in total) and 66 leaf concepts. The depth of the ontological tree is 6. Each concept of this ontology is characterized by a contextual,

1st International Conference on Advanced Technologies for Signal and Image Processing - ATSIP'2014 March 17-19, 2014, Sousse, Tunisia IFI-100

978-1-4799-4888-8/14/$31.00 ©2014 IEEE 36

spatial and spectral attributes which are enriched by a priori values.

The ontology of the DAFOE platform [8] containing concepts associated to data stemming from the European database of land biophysical occupation “Corine Land Cover”, enriched by other concepts and semantic relations. It covers agricultural areas, land use, water areas, etc. In addition, it offers three families of spatial relationships between its concepts, namely topological, distance and directional relations.

Indeed, Hydrology is an ontology that covers topographical features involved in the retention and transport of inner and surface water such as slopes, roads, coastlines, floodplains, etc. FusionTopoCarto2 [9] is an ontology whose concepts are derived from BDTopo and BDCarto databases specifications. The concepts of this ontology are divided into two main categories: the artificial topographic features and natural geographic features. It offers a large number of concepts related to hydrography, topography and agriculture.

B. Discussion

By analyzing the cited state of the art, we deduce these comments:

The existing ontological resources don’t represent the sensor of acquisition, and specially, the temporal characteristic of remote sensing images.

The remote sensing images are characterized by uncertainty which come from several sources. However, the uncertainty is not taken into account in the concepts of ontology.

The remote sensing images are not used as information in the ontology development process.

The majority of the existing ontologies are not rich in axioms and in rules which represent knowledge of the domain. Axioms and rules make mechanisms of inférentiel reasoning in ontology.

III. PROPOSED ONTOLOGY

In this study, we propose a new ontology for remote sensing imagery annotation and interpretation. Our ontology has these properties:

Spatio-spectro-temporal: in order to take account of the specificities of remote sensing imagery such as spectral, spatial and temporal information.

Fuzzy: in order to deal with imperfections about remote sensing imagery such as ambiguities of features values.

Heavy: enriched by a set of axioms that allows inferential reasoning about natural risk assessment.

Evolutionary: the ontology can be adaptable in several domains by reusing and exploiting ontological resources.

A. Ontology development process

We proposed in (Messaoudi et al., 2011) [10] an ontology development process based on (1) the reuse of the existing ontological resources, (2) the learning from the remote sensing images, and (3) the formalization of the knowledge of domain (natural risks, knowledge on scenes, sensors of acquisition, etc.). The development process is composed by three steps as shown in “figure 1”.

Figure 1. Proposed ontology development process

Step 1: Choose of the core ontology

The first step in the process is to choose the core ontology which represents basic knowledge about remote sensing image domain. According to the literature review made on the existing geographical ontologies, we propose (1) to reuse the ontology presented of Durand et al. (2007) [7], and (2) to consider it as core ontology. Indeed, this ontology represents the spatial and the spectral specificities of objects detected in a satellite image.

Step 2: Evolution of the core ontology This step attempts to evolve the selected core ontology to

deal with imperfections, and to represent spectral and temporal dimension. In this context, Ji (2003) [11] was defined a Gaussian fuzzy membership function for representing spectral band of an information class, expressed as Yen and Langari, (1999) [12] :

Selected Core Ontology

Step 2: Evolution of the core ontology

Step 3: Enrichment of the core ontology - Conceptual level - Intentional level - Relational level - Axiomatic level

Step 1: Choose of the Core Ontology

Result: Enriched Ontology

37

A(x)= e-((x-mA

)/A

)², (A>0) (1) with m denote the mean of spectral data for one band of a feature, and the variance.

Thus, we propose to adopt this Gaussian membership function, and to adapt it in our ontology in order to representing time and sensor dimension as shown in “equation 2”.

(2)

with c : class ; b : spectral band ; s : sensor ; t : time; m: mean of spectral data for one band of a class; s: variance

Thus, the concepts of the ontology may have a new fuzzy-spatio-spectro-temporal representation as shown in “table 1”.

Step 3: Enrichment of the core ontology The enrichment process is established at conceptual,

intentional, relational and axiomatic level.

1) Conceptual enrichment: The conceptual enrichment is based on the reusing of

existing ontological resources in order to enrich the core ontology by new concepts. We adopted a process of enrichment composed by these steps as presented in (Nefzi et al., 2013) [13]:

a) Discovery of candidates: identification of existing ontological resources to integrate in the core ontology

b) Evaluation: determination of reusable concepts from each ontological resource. This step is based on the mapping between the core ontology to be enriched and each candidate resource. The alignment is made at the linguistic and structural level.

c) Integration and placement: placement of the selected concepts (from the previous step) in the core ontology basing on the extracted mappings between concepts.

d) Quality and coherence check: check of the coherence of the ontology during its enrichment.

2) Intentional enrichment: In this enrichment level, we try to enrich ontology concepts

by a priori values from remote sensing images. This enrichment allows guiding the semantic annotation of remote sensing images. “Figure 2” shows the process of this enrichment.

3) Relational enrichment: The spatial relations between objects allow maximizing

semantics of the satellite scene. Therefore, we propose to reuse the fuzzy ontology of spatial relation proposed in [14]. In fact, this ontology is enriched with a fuzzy representation representing topological, distance and directional relations between objects, basing on operators of the mathematical morphology to extract fuzziness zones of every object, and on fuzzy operators (t-standard, t-conormes, etc.) to establish the spatial relations.

4) Axiomatic enrichment: This level allows enriching the ontology by axioms and

rules to maximize it ability on reasoning. Thus, we propose to add a set of axioms which represents knowledge related to the natural risks and phenomena such as forest fire, erosion, flood, etc. Next, we present some rules as follow:

R1: If the agricultural ground is neighboring to a mountain having a slope>10°, Then erosion risk is Strong.

R2: If the agricultural ground is neighboring to a mountain having a slope<2°, Then erosion risk is Low.

Figure 2. Intentional enrichment of the core ontology

B. Formalism of the proposed ontology

Our ontology represents three levels of the remote sensing image interpretation:

Pixel level: through the intensions of the concepts; Region level: through the concepts; Decision level: through axioms and rules of domain;

C1

C2

Core Ontology

Image Classe C1

Time t

s1 sn

t1

tn

Sensor S

Image Classe C2

Time t

s1 sn

t1

tn

Sensor S

tn

Image Classe C1

Time t

s1 sn

t1

Sensor S

µP1_S1_b1_tn

µP2_S1_tn

µP7_S1_b1_tn

µP1_S1_b1_t1

µP2_S1_t1

µP7_S1_b1_t1

µP1_Sn_b1_tn

µP2_Sn_tn

µP7_Sn_b1_tn

µP1_Sn_b1_tn

µP2_Sn_tn

µP7_Sn_b1_tn

Image Classe C2

Time t

s1 sn

t1

Sensor S

µP1_S1_b1_tn

µP2_S1_tn

µP7_S1_b1_tn

µP1_S1_b1_t1

µP2_S1_t1

µP7_S1_b1_t1

µP1_Sn_b1_tn

µP2_Sn_tn

µP7_Sn_b1_tn

µP1_Sn_b1_tn

µP2_Sn_tn

µP7_Sn_b1_tn tn

Intentional enrichment

38

TABLE I. PROPOSED FUZZY-SPATIO-SPECTRO-TEMPORAL REPRESENTATION OF CONCEPT OF ONTOLOGY

The ontology is formalized as follow: O = {C, P , R, A , C } où : C : set of concepts of domain

P = {Pf |pf (c, α, µp,t,s , π)} où

o cC, α is the class of the attribute : geometric, spectral…

o µp,t,s is the membership function defining a priori values of attributes basing on time t and sensor s,

o π is the weight attributed to the attribute.

R = {Rf |Rf (ci, cj, t, µRf)} où

o ci, cjC,

o t : type of relation

o µRf membership function defining the fuzzy spatial relation

C : C x C is an hierarchical relation between concepts

A is a set of axioms defined in fuzzy language.

C. Experimental study

We presented in [13] an experimental study done on conceptual enrichment by reusing existing ontological resources. Initially, the selected core ontology of Durand et al. (2007) [7] contained 91 concepts. “Table 2” summarizes the result of enrichment steps at conceptual, intentional, relational and axiomatic levels.

TABLE II. EXPERIMENTAL RESULT OF ENRICHMENT PROCESS

Enrichment level Enrichment result Conceptual level Adding new concepts :

- 34 concepts from ontology of Hydrology - 65 concepts from FTT-01 ontology

Intentional level Creating of base of 60 images class : - multi-dates : January and July - multi-sensors: Spot and LandSat. Generating of 60 Gaussian fuzzy membership functions representing 60 classes.

Relational level Adding of 15 fuzzy spatial relations Axiomatic level Formalization of 100 rules modeling

the natural risk of erosion

IV. ONTOLOGY-BASED IMAGE ANNOTATION APPROACH

According to the proposed ontology, this approach allows annotating remote sensing image by meta-information describing it semantic content, and take into account imperfections related to the image. The process is described as follow:

Step 1: Region extraction

Concept « Ci » of the ontology Attributes Sensor S1 … Sensor Sn

Band1(b1) … Bandn(bn) Band(b1) … Bandn(bn)

Radiometry (P1) t1: µP1_S1_b1_t1 … tn: µP1_S1_b1_t2

… t1: µP1_S1_bn_t1 … tn: µP1_S1_bn_t2

t1: µP1_Sn_b1_t1 … tn: µP1_Sn_b1_t2

… t1: µP1_Sn_bn_t1 … tn: µP1_Sn_bn_t2

NDVI (P2) t1: µP2_S1_t1 … tn: µP2_S1 _t2

t1: µP2_Sn_t1 … tn: µP2_Sn _t2

MNDWI (P3) t1: µP3_S1_t1 … tn: µP3_S1 _t2


SBI (P4) t1: µP4_S1_t1 … tn: µP4_S1 _t2


Texture (P5) t1: µP5_S1_b1_t1 … tn: µP5_S1_b1_t2




Area (P6) t1: µP6_S1_b1_t1 … tn: µP6_S1_b1_t2




Perimeter (P7) t1: µP7_S1_b1_t1 … tn: µP7_S1_b1_t2




39

a) extraction of regions from the request remote sensing image using clustering algorithms such as fuzzy C-Means, region growing, etc.; b) extraction of features related to each region such as radiometry values, texture, shape, etc. c) computing the spatial relations between extracted regions. Step 2: Region classification

In this step, we propose a fuzzy-matching process between extracted regions and concepts of the ontology. The goal is to find the best concept which describes the requested region. This process is described as follow:

a) we compute the similarity index S(Ri, Ci) between a region R={Feature1, Feature2,…} of the image and the concepts C={Attribute1, Attribute2, …} of the ontology as explained in “equation 3” :

with, is the membership degree of the feature Fi in the fuzzy set Ai which represents the attribute of concept C.

b) The concept which has the higher membership degree is considered as a class for the request region.

Step 3: Generating annotation

In this step, we apply the fuzzy matching process for all regions in order to generate the global annotation of the requested image. The generated annotation has the following structure:

Aim(Oim, Rim, Cim, Ctim) : Oim: set of objects of the image; every object has a score

of similarity S(Ri,Ci) with the corresponding concept in the ontology.

Rim: set of spatial relations between objects. Cim: set of image features. Ctim : set of image constraints.

Figure 3. Annotation structure in XML format

<Annotation> <Characteristic>

<Sensor> </Sensor> <Bands> </Bands> <Resolution> </Resolution> <Geo-Ref></Geo-Ref>

</ Characteristic > <Objects> <Object>

<id_objet> </id_objet> <concept_ontologie></concept_ontologie> <features> <Radiometry> <Radiometry> <Texture> </Texture> <Form> </Form> </features>

</Objet> </Objects> <SpatialRelations> </ SpatialRelations> </Annotation>

V. EXPERIMENTAL STUDY

In this section, we present an experimental application showcasing the ontology-based image annotation process:

a) Let be an image representing a scene acquired by a SPOT sensor in March 2000 “figure 3.a”.

b) Extraction of regions: supervised segmentation using ISODATA algorithm “figure 3.b”.

c) Extraction of features of regions using ENVI tool “table 3”

d) Fuzzy matching between regions and concepts of ontology “table 4”.

e) Generated annotation is shown in “figure 5”.

Figure 4. (a) SPOT image acquired in March 2000; (b) extracted regions from the image

(a) (b)

TABLE III. FUZZY FEATURE OF IMAGE REGION

Region A

Feature Feature value

Radiometry µR_S _B1_P µR_S _B2_P µR_S _B3_P

NDVI µNDVI_S _B1_P µNDVI_S _B2_P µNDVI_S _B3_P

Texture µTex _B1_P µTex_S _B2_P µTex_S _B3_P

TABLE IV. RESULT OF THE FUZZY-MATCHING PROCESS

Region Fuzzy-Matching Returned concept

A S(A, lake)=0.8314 S(A, forest)=0.3279 S(A, ground) = 0.01958 …

Lake, Score = 0.8314

B S(B, Wheatfield)=0.74565 S(B, forest)=0.4987 …

Wheatfield, Score = 0.74565

C S(B, ground)=0.6980 S(B, forest)=0.2153 …

Ground, 0.6980

(3)

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

40

Figure 5. Extract of generated annotation in XML format

<Annotation> <Characteristic>

<Sensor> SPOT </Sensor> <Bands> 3 </Bands> <Resolution> 20 metres </Resolution> <Geo-Ref>

<Proj> UTM, Zone 32N </Proj> <Datum> Carthage </Datum>

</Geo-Ref> </ Characteristic > <Objects> <Object>

<id_objet>A</id_objet> <Concept_ontology> <Name_concept> Lake </Name_concept> <MembershipDegree> 0.8314 </MembershipDegree> </ Concept_ontology > <features> <Radiometry>

<band name = “XS1”> 59 </band> <band name = “XS2”> 41 </band> <band name = “XS3”> 16 </name>

<Radiometry> <Texture> … </Texture> <Form> … </Form> </features>

</Objet> <Object> <id_objet>B</id_objet> <concept_ontologie> <Nom_concept> Wheatfield </Nom_concept> <MembershipDegree > 0.74565 </MembershipDegree > .. </Objects> <SpatialRelations> <SpatialRelation> <type>Adjacent </type> <Object> A </ Object> <Object> B </ Object> <MembershipDegree> 0.7856</MembershipDegree> </SpatialRelation> … </ SpatialRelations > </Annotation>

VI. CONCLUSION

In this paper, we presented an ontology approach for remote sensing image annotation. We proposed an ontology development process basing on the reuse of existing ontological resources, and knowledge of domain. In fact, the enrichment process is established at several levels such as conceptual, intentional, relational, and axiomatic level. The main goal of this ontology is to reduce the semantic gap between low-level image features and their semantics. Thus, we proposed a fuzzy matching process between regions of an image and concepts of the ontology. The goal is to find the best concept which describes image regions. This process is simulated on images representing natural scene.

REFERENCES

[1] V. Mezaris, I. Kompatsiaris, M. G. Strintzis, “Region-Based Image Retrieval Using an Object Ontology and Relevance Feedback,” in Journal on Applied Signal Processing, pp. 886–901, 2004.

[2] T.R. Gruber, “A translation approach to portable ontologies,” in Knowledge Acquisition, vol.5 no.2, pp.199–220, 1993.

[3] M. El Adnani, K. Yétongnon, D. Benslimane, “A multiple layered functional data model to support multiple representations and interoperability of GIS: application to urban management systems,” in Proceedings of the 9th ACM international symposium on Advances in geographic information systems,” Atlanta, Georgia, USA, 2001, pp. 70–75.

[4] Y. Huang, G. Deng, “Research on Representation of Geographic Spatio-temporal Information and Spatio-temporal Reasoning Rules Based on Geo-ontology and SWRL,” in International Conference on environmental Science and Information Application Technology, Wuhan, 2009, pp. 381–384,.

[5] N. Abadie, “Schema Matching Based on Attribute Values and Background Ontology,” in 12th International Conference on Geographic Information Science (AGILE'09), Hanovre, Germany, June 2009 pp.2–5.

[6] B. T. Le, R. Dieng-Kuntz, and F. Gandon, On ontology matching problems for building a corporate semantic web in a multi-communities organization”, in Proc. of the Sixth International Conference on Enterprise Information Systems, 2004, pp. 236–243.

[7] N. Durand, S. Derivaux, G. Forestier, C. Wemmert, P. Gançarski, O. Boussaid, A. Puissant, “Ontology-based Object Recognition for Remote Sensing Image Interpretation,” in 19th IEEE International Conference on Tools with Artificial Intelligence, 2007, pp 472–479

[8] J. Charlet, S. Szulman, N. Aussenac-Gilles, A. Nazarenko, N. Hernandez, N. Nadah, E. Sardet, J. Delahousse, V. Teguiak, and A. Baneyx, “Dafoe : une plateforme pour construire des ontologies à partir de textes et de thésaurus,” in 10ième Conférence Internationale Francophone sur l’Extraction et la Gestion des Connaissances, Hammamet, Tunisie, 2010.

[9] S. Mustière, N. Abadie, N. Aussenac-Gilles, M.N. Bessagnet, Kamel M., Kergosien E. , Reynaud C., Safar B., GéOnto : Enrichissement d’une taxonomie de concepts topographiques, in Spatial Analysis and GEOmatics, Paris, France, 25-27 Novembre 2009 pp.1-17.

[10] Messaoudi W., Farah I. R., Solaiman B., “Vers une ontologie spatio-temporelle, floue et lourde de l'imagerie satellitale,” in 4èmes Journées Francophones sur les Ontologies, Montréal, Canada, 22–23 Juin 2011, pp. 253-258.

[11] M. Ji, “Using fuzzy sets to improve cluster labelling in unsupervised classification,” in International Journal of Remote Sensing, vol.24, no4, pp.657–671, 2001.

[12] J. Yen, R. Langari, “Fuzzy logic: Intelligence, control, and information,” Prentice Hall, 1999.

[13] H. Nefzi, W. Messaoudi, M. Farah, I.R. Farah , “Vers une ontologie riche dédiée à l'imagerie satellitaire par réutilisation de ressources existantes,” in Traitement et analyse de l’information, Méthodes et applications (Taima’2013), Hammamet, Tunisie, 13-18 Mai 2013.

[14] C. Hudelot, J. Atif, I. Bloch, “Ontologie de relations spatiales floues pour l’interprétation d'images,” in Rencontres francophones sur la Logique Floue et ses Applications, LFA-2006, Toulouse, France, 2006.

41

[IEEE 2014 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP)...

Documents

Transcript of [IEEE 2014 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP)...