Super-Resolution Mapping of Multiple-Scale Land Cover Features Using aHopfield Neural NetworkA. J.Tatem1,2, H.G. Lewis3, P.M. Atkinson2 and M.S. Nixon11Dept. Electronics & Computer Science, 2Dept. Geography, 3Dept. Aeronautics & AstronauticsUniversity of Southampton,Southampton SO17 1BJ, U.K.A.J.Tatem@soton.ac.ukAbstractSoftclassificationtechniqueshavebeendevelopedtoestimatetheclasscompositionofimagepixels,buttheiroutputprovidesnoindicationofhowtheseclassesaredistributedspatiallywithinthepixel.SeparateHopfieldneuralnetworktechniquesforproducingsuper-resolutionmapsfromimageryoffeatureslargerandsmallerthanapixelhavebeendeveloped.However,thetechniqueshaveyettobecombinedinordertoproducesuper-resolutionmapsofmultiple-scalelandcoverfeatures.This paper presents the first results from combining thetwoapproaches.Theoutputfromasoftclassificationandpriorinformation of sub-pixel feature arrangement is used to constraina Hopfieldneural network formulatedasanenergyminimisationtool.Theenergyminimumrepresentsabestguessmapofthespatialdistributionofclasscomponentsineachpixel.ThetechniquewasappliedtosimulatedSPOTHRVimageryandtheresultantmapsprovidedanaccurateandimprovedrepresentation of the land covers studied.I. INTRODUCTIONTheproductionoflandcovermapshas,inthepast,beenatimeconsumingtaskfraughtwithdifficulties.Theadventofremote sensing afforded the opportunitytoproducesuchmapsquickly and efficiently. However, several problems still arise intheproductionoflandcovermapsfromsatellitesensorimagery.Themostsignificantissuerelatestothespatialresolutionofthesensorused,whichhaspreviouslyrestrictedthe size of object able to be distinguished on the ground.Traditionaltechniquesofmapproductionfromremotelysensedimageshavefocusedonhardclassificationwherebyeachpixelisassignedtoasinglelandcoverclass.However,withinmostsatellitesensorimagery,themajorityofpixelsrepresentamixoflandcovers,leadingtoinaccuracieswithinsuch classifications [1]. More recently, softclassificationhasenabled the contents of each pixel to be estimated, producing amore informative classification. Sub-pixel class composition isestimatedthroughtheuseoftechniquessuchasspectralmixturemodelling,nearestneighbourclassifiers,multi-layerperceptronsandsupportvectormachines.Theoutputofsuchtechniquesgenerallytakestheformofasetofproportionimages, each of which displays the proportion of a certain classwithineachpixel.However,thelocationofthelandcovercomponents within each pixel still remains unknown, hinderingtheproductionofaccuratemapsfromremotelysensedimagery.ThispaperdescribestheapplicationofaHopfieldneuralnetwork,designedasanoptimisationtool,tomapthelocationof land cover class components at the sub-pixel scale.(a)(b)Fig 1. (a) 2x2 pixel image, p and q represent the image dimensions, x and yrepresent the image co-ordinates; (b) Representation of the Hopfield networkfor the image in (a), i and j represent the neuron co-ordinates (int = integervalue).II. USING THE HOPFIELD NEURAL NETWORK FOR SUPER-RESOLUTION LAND COVER MAPPINGTheHopfieldneuralnetworkisafullyconnectedrecurrentnetwork.Like the popular, feed-forward neuralnetworks, eachneuronismodelledusinganinputfunctionand(typically)asigmoidalactivationfunction.However,intheHopfieldnetwork,neuroninputsaretheoutputsofallotherneuronsinthenetwork.Thus,fromasetofinitialneuronactivations,thestateofthenetworkvarieswithtimeuntilconvergencetoastablestate,whereneuronactivationsstopvaryingwithtime.Weightsandbiasesdeterminetheactivationsatthisstablestate.TheHopfieldnetworkcanthereforebeusedforenergyminimizationproblems,iftheweightsandbiasesarearrangedsuchthattheydescribeanenergyfunction,andtheminimumof energy occurs at the stable state [2].An energyfunction canbedefinedasagoalandseveralconstraints.TheHopfieldnetworkwillconvergetoasolutionofferingacompromisebetweenthegoalandconstraints.Mappingthespatialdistributionofclasscomponentswithineachpixelisthereforeformulated as a constraint satisfaction problem with an optimalsolution determined by the minimum of the cost function.ThearchitectureoftheHopfieldnetworkrepresentsafinerresolutionimagewitheachneuroncorrespondingtoapixelinthisimage(Fig.1).Inaddition,eachneuronalsorepresentsasub-pixelpointintheoriginal,coarse-resolutionsatelliteimage.Thezoomfactor,zdeterminestheincreaseinspatialresolutionfromtheoriginalsatellitesensorimagetothenewfine-resolutionimage.Afterconvergencetoastablestate,theneurons represent a binary classification of the land coveratafinerspatialresolution.Fig.1showshowco-ordinatesaretransformed linearly from the image space to the network.Previousworkhasfocusedontwoseparateareasofsuper-resolutionmapping: 1. Land cover features larger thanapixel,2.Landcoverfeaturessmallerthanapixel.Tatemetal.[3]examinedtheproblemofproducingsuper-resolutionmapsof0-7803-7033-3/01/$17.00 (C) 2001 IEEE0-7803-7031-1/01/$17.00 (C) 2001 IEEE 3200singletargetfeatureswhichwerelargerthanapixel.Byutilizinginformationcontainedinsurroundingpixels,thelandcoverwithineachpixelwasmappedusingasimplespatialclusteringfunction,C,codedintotheenergyfunctionofaHopfieldneuralnetwork.Anothersimplefunction,P,intheenergy function ensured that thepixelclassproportionsoutputfromasoftclassificationwereretained.Thistechniquewasextendedtomultiplelandcoverclassmappingin[4].Extralayersofneuronswereaddedtothenetworkandacorrespondingconstraint,M,intheenergyfunctiontoensurenogapsoroverlapsbetweenclasseswereintroduced.Thisenabledtheproductionofsuper-resolutionlandcovermaps,such as the one in Fig. 2.Thefocusof[3]and[4]onsuper-resolutionmappingoffeatures larger thanthescaleofapixel(e.g.agriculturalfieldsin SPOT HRVimagery),enablestheutilisationofinformationcontainedinsurroundingpixels.However,thissourceofinformationisunavailablewhenexaminingimageryoflandcoverfeaturesthataresmallerthanapixel(e.g.housesinLandsatTMimagery).Therefore,[5]presentedatechniquethatattemptedtoovercomethisproblemusingaHopfieldneuralnetworkagain.Themethodwasbasedonpriorinformationonthespatialarrangementoflandcover,intheform of avariogram [6].Asimplefunction,SV,tomatchlandcoverdistributionwithineachpixeltothispriorinformationwascodedintotheenergyfunctionofaHopfieldneuralnetwork. This enabled the production of super-resolutionmapsoftargetfeaturesthatwereoriginallyofsub-pixelscale,suchas the result shown in Fig. 3. This paper presents initial resultsfromcombiningtechniquesofsuper-resolutionmappingoffeatures larger (as described in [4]) and smaller (as described in(a) (b)(c) (d)Fig.2.(a)SPOTHRVImageofanagriculturalareanearBath,UK(Spatialresolution: 20m);(b)Verification imagederivedfromaerialphotographs;(c)Traditionalmaximum-likelihoodhardclassification(Spatialresolution:20m,RMS error = 0.23); (d) Hopfield neural network prediction (Spatial resolution:2.9 m, RMS error = 0.14). From [4].(a) (b)(c) (d)Fig.3.(a)LandsatTMImageofanareaofhousinginBath,UK(Spatialresolution30m);(b)Buildingclassverificationimagederivedfromaerialphotographs;(c)Traditionalmaximum-likelihoodhardclassification(Spatialresolution:30m);(d)Hopfieldneuralnetworkprediction(Spatialresolution:4.3.m). From [5]. [5]) than a pixel, into asingleapproach.Theapproachshouldbecapableofsuper-resolutionlandcovermappingfromimageryofanyspatialresolution,containinganyscaleoffeature.Thiswasundertakenbycombiningthefunctionsdescribedpreviouslyintoasingleenergyfunction(equation(1)), and weighting their influence on certain classes.M P SV C E + + + = .(1)Forexample, Cwasweightedstronglyforfeatureslargerthanthepixelsize,whereas,SVwasgivenmoreinfluencewhensub-pixel scale featureswere dominantinaclass.Throughout,P and Mweregiven the strongestweightings to ensure correctclass proportions were maintained,withoutgaps or overlaps inthe final map.III. RESULTSThenewnetworkset-upwastestedonsimulatedSPOTHRVimagery.Fig.4(a)showsanaerialphotographofthechosentestarea,whichcontainedbothalargeareaofwoodland,andlonetreesamongstgrassland.Theverificationimage in Fig. 4(c) was degraded, using a square mean filter, toproducethreeclassproportionimagesthatprovidedinputtotheHopfieldnetwork.Inaddition,avariogram(Fig.6(a))wascalculated from a small section of Fig. 4(c) to provide the priorspatialinformationonthelonetreeclass,requiredbytheSVfunction.After1000iterationsofthenetworkwithz=7,themapshowninFig.5(b)wasproduced,andatraditionalhardclassificationwasundertaken for comparison. Bothmapswerecomparedtotheverificationdata,andaccuracystatisticscalculated.Theseincludedcorrelationcoefficientbetweenclasses (CC), area error proportion (AEP), closeness (S) (from0-7803-7033-3/01/$17.00 (C) 2001 IEEE3201 (a) (b) (c)Fig.4.(a)AerialphotographofanagriculturalareanearBath,UK(b)Simulated SPOT HRV Image (Spatial resolution 20 m); (c) Verification imagederived from aerial photographs.

(a) (b)Fig.5.(a)HardclassificationoftheimageshowninFig.4(b).(b)ResultofHopfield neural network prediction(Spatial resolution 2.9 m). [7]) and root mean square error (RMSE), all shown in tables Iand II.IV. DISCUSSIONTheresultsshowclearlythatthesuper-resolutiontechniqueprovidesanincreaseinaccuracyovertraditionalhardclassification.VisualinspectionofFig.5revealsthathardclassificationhasfailedtoidentifythelonetreeclass,andproducedanunevenwoodlandboundary.Incontrast,theHopfieldnetworkpredictionappearstohaveidentifiedandmappedbothfeaturescorrectly.Thisisconfirmedafterinspectionoftheaccuracystatisticsandvariograms.Whilethere is little difference between the AEP values in tables I andII,showingthatbothtechniquesmaintainedclassareatoasimilardegree,theotherstatisticsshowhowsuccessfultheHopfieldnetworkwas.Thewoodlandclasswasmappedaccurately, with a correlation coefficient of 0.985, compared tojust0.887usinghardclassification.Overallimageresultsalsoshow an increase in accuracy, with closeness and RMSE valuesof just 0.052 and 0.229respectively.Theonlylowaccuracyisforthelonetreeclass,withacorrelationcoefficientofjust0.43. However, as the aim of the SV function is to recreate thespatialarrangementofsub-pixelscalefeatures,ratherthanaccuratelymaptheirlocations,thisisexpected.Thebestwayto test the performance of this function is therefore, to comparethe shape of thevariograms inFig.6,whichdoconfirmthatasimilarspatialarrangementoftreestothatoftheverificationimage has been recreated.V. CONCLUSIONSThis study has shown that a Hopfield neural network can beTABLE IACCURACY ASSESSMENT RESULTS: HARD CLASSIFICATION_______________________________________________________________Class CC AEP S RMSE_______________________________________________________________Lone Trees N/A 1.0 0.0627 0.251Woodland 0.887 0.0095 0.0476 0.218Grass 0.761 -0.101 0.11 0.331Entire Image 0.00434 0.073 0.271TABLE IIACCURACY ASSESSMENT RESULTS: HOPFIELD NETWORK_______________________________________________________________Class CC AEP S RMSE_______________________________________________________________Lone Trees 0.43 -0.161 0.0721 0.269Woodland 0.985 0.0085 0.0066 0.0809Grass 0.831 0.0136 0.0786 0.28Entire Image 0.00711 0.052 0.229 (a) (b)Fig.6.(a)Variogramoflonetreeclassinverificationimage(Fig.4(c)).(b)Variogram of lone tree class in Hopfield network prediction image (Fig.5(b)).usedtoestimatethelocationoflandcoverclassproportionswithinpixels.TheHopfieldneuralnetworkusedinthiswayrepresentsasimple,robustandefficienttoolforsuper-resolutionmappingofmultiple-scalelandcoverfeaturesfromremotely sensed imagery.REFERENCES[1] P.Fisher,Thepixel:asnareandadelusion,Int.J.Rem.Sens.,vol. 18, pp.679-685, 1997.[2] J. Hopfield and D.W. Tank, Neural computation of decisions in optimization problems, Biol. Cybernetics, vol.52, pp.141-152, 1985.[3] A.J. Tatem, H.G. Lewis, P.M. Atkinson and M.S. Nixon, Super-resolution target identification from remotely sensed images using aHopfield neural network, IEEE Trans. Geosci. & Rem. Sens., vol. 39, in press, 2001.[4] A.J. Tatem, H.G. Lewis, P.M. Atkinson and M.S. Nixon, Land cover mapping at the sub-pixel scale using a Hopfield neural network, Int. J. Applied Earth Obs. & Geoinf., in press.[5] A.J. Tatem, H.G. Lewis, P.M. Atkinson and M.S. Nixon, Super-resolution land cover pattern prediction using a Hopfield neural network, Rem. Sens. of Env., in press.[6] P.J. Curran and P.M. Atkinson, Geostatistics and remote sensing, Prog. in Phys. Geog., vol. 22,pp.61-78, 1998.[7] G.M. Foody, Approaches for the production and evaluation of fuzzy land cover classifications from remotely-sensed data, Int. J. Rem. Sens., vol. 17, pp. 1317-1340, 1996.00.050.10.150 2 4 6 8LagSemivariance00.050.10.150 2 4 6 8LagSemivariance0-7803-7033-3/01/$17.00 (C) 2001 IEEE3202