Instrumentation and MeasurementTechnology Conference - IMTC 2007Warsaw, Poland, May 1-3, 2007

Ant Colony-Based Reinforcement Learning Algorithm for Routing in WirelessSensor Networks

Reza GhasemAghaeil, Md. Abdur Rahman', Wail Gueaieb2, Abdulmotaleb El Saddik''Multimedia Communication Research Laboratory (MCRLab)

2Machine Intelligence, Robotics, and Mechatronics (MIRAM) LaboratoryUniversity of Ottawa

800 King Edward Avenue, Ottawa, Ontario, Canada, KIN 6N5Fax: +1-613-562-5664, Email: {reza, rahman, abed}@mcrlab.uottawa.ca, wgueaiebWsite.uottawa.ca

Abstract - The field of routing and sensor networking is animportant and challenging research area of network computingtoday. Advancements in sensor networks enable a wide range ofenvironmental monitoring and object tracking applications. Routingin sensor networks is a difficult problem: as the size of the networkincreases, routing becomes more complex. Therefore, biologically-inspired intelligent algorithms are used to tackle this problem. Antrouting has shown excellentperformancefor sensor networks. In thispaper, we present a biologically-inspired swarm intelligence-basedrouting algorithm, which is suitable for sensor networks. Ourproposed ant routing algorithm also meet the enhanced sensornetwork requirements, including energy consumption, success rate,and time delay. The paper concludes with the measurement data wehavefound.

Keywords - ant routing, sensor network.

I. INTRODUCTION

Recently, advances in minimization, low-power, low-cost,efficient and multi-functional wireless communicationequipment, and improved small-scale energy supplies make anew hi-tech dream achievable: wireless sensor networks [1],[2]. Since the 1970s, wireless networks have been showinghuge interest to the computing industry and the researchcommunity [3]. Wireless sensor networks contain a largenumber of nodes with sensing capabilities such as vibration,temperature, radiation, light, sound etc. and have simplewireless communication capabilities. Many routing protocolshave been specifically designed for the wireless sensornetworks where several factors such as energy efficiency, lowlatency, and success rate are necessary to take into account[2], [4], [5]. Among the above factors, energy consumption isone of the important concerns in a sensor network becauseeach sensor node has to be in active mode during routing,communicating and monitoring the environment [6], [7].

Routing is elementary in all wireless sensor networks. Themain task of such a routing algorithm is to maintain data flowfrom the source to destination sensor nodes, maximizenetwork performance, and build routing tables - one for eachsensor node in the network, which helps incoming datapackets to choose any efficient outgoing link to continue theirtravel towards the destination node [8]. Existing routingprotocols that are not designed for wireless sensor networksshow poor performance when implemented in sensor

networks. Ant inspired intelligent algorithms show promisingresults in solving routing problems in sensor network [8]. Byusing ants, bees and other social swarms as models, we cancreate software agents that can solve complex problems, suchas rerouting of traffic in a busy telecommunication network[9]. Swarm intelligence, which is revealed by such naturalbiological swarms, has various great properties popular inmany engineering systems, for instance in network routing[10]. Swarm intelligence systems refer to complex behaviors,typically invented from some simple agents cooperating withone another and with their environment [ 11], [ 12]. One of themost successful swarm intelligence techniques is called AntColony Optimization (ACO) [13]. ACO is an optimizationalgorithm that can be used to find approximate solutions todifficult combinatorial optimization problems. In ACOartificial ants find solutions by moving on the problem graphand imitating real ants. They leave artificial pheromones onthe graph in such a way that future artificial ants can findbetter solutions. ACO has been successfully applied to aremarkable number of optimization problems. Ants usereinforcement learning to discover the best way. Inreinforcement learning, the intelligent system is just given agoal to reach. The system then adopts the goal by a trial anderror interaction with the environment. For the interactionsthat take the system close to the target, a positive reward isreceived while going away from the target, a negative rewardis assigned. Computer scientists have addressed thereinforcement learning of an artificial system by introducing aconcept called pheromone decay. When the chemicalevaporates rapidly, longer paths will have trouble maintainingstable pheromone trails. This has been used fortelecommunication networks [9]. Artificial ants continuouslyexplore different paths, and pheromone trails to providebackup plans. Thus, if one link breaks down, a pool ofalternatives already exists [14], [15].

In [8], the authors show an adaptive distributed, mobileagent-based system, which is inspired from the ant colonymetaphor for solving optimization problems and targeted forcommunication networks. Although their basic ant routingalgorithm makes the basic foundation of our work, it is notoptimized for routing in wireless sensor networks. Zhang etal. [16] proposed three ant-routing algorithms for sensornetworks: the Sensor-driven Cost-aware Ant Routing (SC),the Flooded Forward Ant Routing (FF) algorithm, and the

1-4244-0589-0/07/$20.00 ©2007 IEEE 1

Flooded Piggybacked Ant Routing (FP) algorithm. The SCalgorithm is energy efficient but suffers from a low successrate. The FF algorithm has shorter time delays; however, thealgorithm creates a significant amount of traffic. Despite highsuccess rate shown by the FP algorithm, it is not energyefficient. Although the algorithms show significant results,none of them are ideal for routing in wireless sensor networksfor the above-mentioned reasons. Moreover, none of the threealgorithms take into consideration the concept ofreinforcement learning. An Adaptive ant-based DynamicRouting (ADR) algorithm using a novel variation ofreinforcement learning was proposed by Lu et al. [17]. Theauthors used a delay parameter in the queues to estimatereinforcement learning factor. Although queues are an integralpart of communication network, is not an essential part ofsensor network. Also, they did not address the suitability oftheir algorithm for wireless sensor networks.

In this paper, we propose two adaptive routing algorithmsbased on swarm intelligence: the Adaptive Routing (AR)algorithm and the Improved Adaptive Routing (IAR)algorithm. To check the suitability of ADR algorithm in thecase of sensor networks, which was originally intended forpacket-switched communication networks [17], we modifiedthe ADR algorithm (removing the queue parameters) and usedtheir reinforcement learning concept with the routingalgorithms proposed in [16]. We name it the AR algorithm.However, AR algorithm did not result in optimum solution.Therefore, we propose the IAR algorithm by adding acoefficient, the cost between the neighbor node and thedestination node, to further improve the AR algorithm, whichis described in details in Section 2. Both algorithms aredesigned to optimize several criteria such as energy, energy-efficiency, latency and success-rate. The remainder of thepaper is organized as follows: in section 2 we introduce ourproposed algorithms. In Section 3 we analyze the performanceof the proposed algorithms. We finally conclude the paper, insection 4, with a few concluding remarks and possible futureresearch avenues.

II. PROPOSED ALGORITHMS

The proposed AR and IAR uses probability distribution ateach sensor node in the network, where a routing table ismaintained (see Figure 1). Each routing table contains jNk *Nentries where INkl correspond to the set of neighbors of node k,and N is the number of sensor nodes in the network. Ants usethis routing table to choose the next favorable neighbor node.

Fig. 1. Routing table structure for node k.

Each row in the table depicted in figure 1 corresponds to aneighbor and each column refers to a destination. We definethe probability P1, of a node k as the probability of desirabilityof choosing the node i as the neighbor of k to go to thedestination node j. Based on this probability distribution, antsexplore new and better routes. Once new routes are found, thenext-hop probabilities are updated in the routing table, so thatthe next set of ants can follow this route. The followingprobability constraint must be satisfied by all routing tableentries [8]:

Z find =1, i neighbors(k), d E [1, N] (1)

We initialize all the routing tables with equal probabilities.In our proposed algorithms, we employ two types of ants [8]:(i) forward ant (Fant), which travels from the source node (s)to the destination node (d) and (ii) backward ant (Bant), whichis generated by Fant, when Fant reaches the destination d. TheBant will then come back to s by using the information alreadysupplied by Fant. However, the Bant uses a reinforcementlearning algorithm [18], [19] in order to get a better and moreefficient route than the one chosen by the Fant and at the end itupdates the routing tables of the reverse-visited sensor nodes.

First of all we elaborate the proposed AR algorithm. Fromeach sensor node k, having INkl neighbors, Fant selects the nexthop i with a probability of P i,d , which is computed asfollows:

p', _ Jd +/JXC, (2)

1+8xVA 1)where Pid iS the normalized sum of the probabilistic entry

Pid of the routing table with a correction factor ci and thecoefficient ,B.

ci is the cost from the current node k to the neighbor node ithat is calculated as follows:

ci=1 DkI (3)j=l kj

where Dkj is the distance between current node k and itsneighbor node i.

As shown in equation (3), the heuristic correction ci isproportional to the distance Dk i. The value of, in equation (2)weights the desirability of the correction factor ci with respectto the probability values (Pi,d) stored in the routing table. Thecoefficient ,B has a value between zero and one. Thesimulation results found with the AR algorithm was notsatisfactory with regards to the requirements of sensornetworks, which is shown in Section 3.

Now, we present the modification we made with the ARalgorithm, which we call IAR algorithm. Here we propose amodified heuristic correction factor Ai,d that is the cost fromthe neighbor node i to the destination node d to find thefollowing probability P i,d:

2

P'd Ax Id +

where the coefficient Ai,d is given by:

(4)

A IL

idA X Ci,d

In the above equation Ci,d is the distance from the neighbornode i to the destination node d and i is a coefficient, whichhas a value between zero and one. For example, forgeographic routing, we may estimate the node's cost by:

/(Xd -X)2 + (Yd y )2 (6)

where (xi, yi) is the location of the current node and (Xd, Yd)is the location of the destination.

When a Fant moves toward its destination d, it remembersthe list of nodes it has visited while updating a global twodimensional matrix. Subsequent forward ants use this list toavoid visiting the same node. Also, if the visited nodes by aFant are larger than half of the total number of nodes, the Fantdies, which indicates that the path it was following was notefficient.

The Bant works similarly except that during the backwardtravel, every intermediate node is treated like a destinationand its associated probabilities are changed. The function'change of probability' is a negative exponential, with theexponent proportional to the cost between the current node kand the destination node d through the neighbor node i:

AP ee *di,d (7)where y is a coefficient and has a value between zero and

one.

,i,d in the AR algorithm is Cki, which is the cost betweenthe current node k and the neighbor node i, and d,i,d in IARalgorithm is Ck,i + Cid, that is the cost between the currentnode k and the neighbor node i plus the cost between theneighbor node i and the destination node d. When a Bantarrives at node i, the entry in the routing table correspondingto the node i from which the ant has just come is increased byAP as following [8], [17]:

Pd+ APid AP

(8)

Similar to [8] and [17], the other entries in the routing tableof that node are decreased as following:

Pi',dI1 p±P i'e neighbors(k),i'# i (9)

Figure 2 shows the pseudo-code of the proposed IARalgorithm.

III. PERFORMANCE EVALUATION

We used Java-based simulation to evaluate the proposedAR and the IAR algorithms as depicted in Figure 3. BecauseADR [17] is not suitable for sensor networks, we onlycompare our proposed AR and IAR algorithms with thefollowing four routing algorithms: Basic Ant Routing [8],Sensor-driven Cost-aware Ant Routing, Flooded PiggybackedAnt Routing, and Flooded Piggybacked Ant Routing [16], allpresented with detailed explanations and results.We constructed an event driven simulator to evaluate the

performance of the algorithms such as success rates, latency,energy consumption and energy efficiency as performancemetrics. We can select the sensor networks both orderly andrandomly. To better compare the results with respect to [16],we run the simulation for 200 seconds having total number of7 X 7 sensor node grid.

In each evaluation, we choose a value for J, y and Xbetween 0 and 1. We chose the increment factor of 0.1 foreach of the variables and conducted the simulation. Among allthe combinations of the three variables, P=30.1, y=0.5 and)=0.7 resulted in optimum results that are shown in Figure 4,5, 6 and 7. Figure 4 shows the success rate comparison amongthe algorithms.

Algoiithm 1: The IPrp1sdRALAl t:ldiu ftrFxrwardAnt)t~w IlXi iS)l{itoIl() l i I lS'it'.-l {, {iiv lIcHi uni ilc'l{

beginif tihttinlion is Traihid 1thenL (qite at IMwkmirdw -1-it ;i1Co( rix -fst tokar(ImdI_Lst

elseL Add llwi10d

if Lss hr1hlttfrof!th visited-hen

Lf selectm ;efib z uiittxl thenLL~ dl)I i 1O1)

else,L Fi Fowis tXni

end

Algorithm 2; The Propo1sed tARtAIgokrill Bkward-Andi

beginif SOUM2 Zs naneha4 th nL RuukimsirdAt;w tI\

ekefUr all n{ Jhhor nosd do

if mdeehlu neiujhbo un d ihiie by 1nvantiAudI t1iwnIl" "I.+A, "L A

eAse

L i' E+ 4pt;LRAnnov firsi. Ac{ini of &f'wwd-fist-l

Send,(l MEicuskwavngl-AnL4 t. to1 sehml.X-tedif elihlElor:node4lend

Fig. 2. The proposed IAR algorithm

3

- - .,. - -

Fig. 3. Randomized sensor network environment.

Success rate is the total number of packets received at thedestination vs. the total number of packets sent from thesource [16]. As shown in the figure, in the case of IAR, thesuccess rate is almost close to 1, which means least numberof packets is lost in the network. After IAR, the AR showsbetter success rate in comparison with the FF, FP, Basic andthe SC algorithms.

O.25Late 2.0n

y .'

0 05*.i4jmm

. X X );trvvXvX zt' <SX X

f - - - E E3 E--

1 2 3 4 5 6 7 8

Simuilation Time Basic 4FF

tLatrncy:Th tim delay bfa pa.cket fromn the sourc to the detinatior AR.

01 -

1 2 3 4 5 6 7

SimLJlation Time

Sucbess rate =: (Numbrer of Received PacketDI(Number of end Packet)

8 9 1O SecBasic: o SC x

FF. FPiAR. IAR o

Fig. 4. Evaluation of success rate.

Figure 5 shows the comparison of time delay of a packetbetween the source and the destination (latency) among thealgorithms. We assume that in a wireless sensor network, the

Fig. 5. Evaluation of latency.

sensory data/packet processing time within a sensor node(including receiving and sending) is significantly larger thanthat of the delay in wireless channel. In other words, visitingmore number of nodes in a route by a packet will result inhigher latency. We assume that the time taken between any oftwo neighbour nodes and the processing time of packet ineach sensor node are uniform. Therefore, we can ignore thetime unit and calculate delay as the division of 'number ofvisited nodes' by the 'number of routes found', which resultsin 'average number of nodes visited per each route'. Asshown in Figure 5, in the case of IAR, the latency is the

4

Wm

!- 0

-t_i'V R .. _v D t v t

B

9 10 SecSC x

FP,IAR a

lowest, followed by the AR, the FF, the FP, the Basic and theSC respectively.

The comparison of energy consumption among thealgorithms is shown in Figure 6. Similar to [16], we alsoassume each transmission consumes one energy unit.Therefore, we define the total energy consumption isequivalent to the total number of packets sent to the network.As shown in the figure, in the case of IAR, the least numberof packets are sent in the network, thereby consuming theleast energy. AR consumes a bit higher energy than IARfollowed by the FF, the Basic and the SC respectively. TheFP consumes the highest energy.

*_ l= x

5000

E 4000nerg 300&

2000

1000

1 2 3 4 5 6

SiMulation TiMe

Enery= otbal number of packets sent in the network

F7 8 9 10 Sec

Basic o SC x

FF* FP,AR, IARa

Fig. 6. Energy Evaluation.

The IAR algorithm also exhibits the most energyefficiency in comparison with the others. Energy efficiency isthe ratio between the numbers of packets received at thedestination vs. the total energy consumption in the network[16]. As shown in the figure, in the case of IAR, the energyefficiency is almost 0.32, the highest among all, which in thecase of AR 0.13, for FF 0.8, for Basic 0.44, for SC 0.35 andfor FP 0. 19.

least number of nodes, sends the least packets and finds betterpaths rapidly. In IAR, we also take into consideration thedistance between the neighbor sensor node and thedestination sensor node while we calculate heuristiccorrection factor Aid. Therefore, the packets always tend togo in a route, which is closer to the destination. This ensuresthat the number of packets sent in the network is less; thelatency decreases and the success rate increases.

IV. CONCLUSION

Routing in sensor networks is growing rapidly. In thispaper, we have presented a new scheme based on antalgorithm for routing in wireless sensor networks. We haveproposed two new algorithms, the Adaptive Routing (AR)algorithm and the Improved Adaptive Routing (IAR)algorithm. Our proposed algorithms are able to deal, reactand adapt themselves to changes in the network. We haveused reinforcement learning features for the algorithms. Forevaluating the performance of the AR and the IARalgorithms, we compared them with the four other routingalgorithms- Basic Ant Routing, Sensor-driven Cost-awareAnt Routing, Flooded Piggybacked Ant Routing, andFlooded Piggybacked Ant Routing proposed in [16].Among the algorithms, the AR shows good performance

results, and the IAR shows the best performance results in allthe tested conditions. Adopting the AR and the IARalgorithm for routing in wireless sensor network can result inlow energy consumption, high-energy efficiency, lesslatency, and a high success rate.

In our future work, we would like to implement thisalgorithm on real sensor networks to see the suitability of ouralgorithm.

REFERENCES

E 0nF

yEff O.2

e

y

1 2 3 4 5 6 7 8Simulation Time Basic 0

FFeuEnergy Efcieneg-(NumberbfR3ereive-dPacklet)f(Number bfVisited Nbde) AFR

9 10 Sec

FP,

IAR b

Fig. 7. Evaluation of energy efficiency.

In summary, the IAR algorithm has many advantagessimilar to the advantages found with the algorithm proposedin [17]. In all the cases, the IAR algorithm exhibits superiorresult in comparison with that of the AR algorithm. The IARalgorithm also requires the least energy consumption and thehighest success rate among rest of the algorithms. It visits the

[1] I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci,"Wireless Sensor Networks: a Survey". J. Comput. Networks(Elsevier), 38, 4, pp. 393-422, 2002.

[2] H. Karl and A. Willig, "A short survey of wireless sensor networks".TKN Technical Reports, Technical University Berlin, Berlin, Oct. 2003.

[3] E. M. Royer and C. Toh. "A review of current routing protocols for adhoc mobile wireless networks". IEEE Personal Communications, pp.46-55, April 1999.

[4] J. N. Al-Karaki and A. E. Kamal, "Routing techniques in wirelesssensor networks: a survey". IEEE Wireless Communications, pp. 6-28,Dec. 2004.

[5] K. Akkaya and M. Younis, "A survey on routing protocols for wirelesssensor networks". Elsevier Ad Hoc Network Journal, 3, 3, pp. 325-349,2005.

[6] R. Muraleedharan and L. A. Osadciw, "A Predictive Sensor NetworkUsing Ant System," In Proceedings ofSPIE, 2004.

[7] R. Muraleedharan and L. A. Osadciw, "Sensor CommunicationNetwork Using Swarm Intelligence," IEEE Upstate New YorkNetworking Workshop, Syracuse University, Syracuse, NY, 2003.

[8] G. D. Caro, and M. Dorigo, "AntNet: Distributed Stigmergetic Controlfor Communications Networks". J. of Artificial Intelligence Research,vol. 9, Dec. 1998, pp. 317-365.

[9] Bonabeau, E., and Theraulaz, G., "Swarm Smarts". Scientific AmericanInc., pp. 72-79, March 2000.

[10] I. Kassabalidis, M. A. El-Sharkawi, R. J. Marks II, P. Arabshahi and A.A. Gray, "Swarm Intelligence for routing in communication networks,"GLOBECOMIEEE, 2001.

5

t I .

I 1 f74

MmmI.S

[11] C. Shen and C. Jaikaeo, "Ad Hoc Multicast Routing Algorithm withSwarm Intelligence". J. ofMobile Networks and Applications, vol. 10,pp.47-59,2005.

[12] T. White, B. Pagurek, and F. Oppacher, "Connection Managementusing Adaptive Mobile Agents," In Proceedings of InternationalConference on Parallel and Distributed Processing Techniques andApplications (PDPTA'98), 1998.

[13] E. Bonabeau, M. Dorigo, and G. Theraulaz, Swarm Intelligence: FromNatural to Artificial Systems, Oxford University Press, 1999.

[14] M. Heusse, D. Snyers, S. Gu'erin, and P. Kuntz, "Adaptive agent-driven routing and load balancing in communication networks,"D'epartment Intelligence Artificielle et Sciences Cognitives, ENSTBretagne, 1998.

[15] R. Schoonderwoerd, 0. Holland, J. Bruten, and L. Rothkrantz, "Ant-Based Load Balancing in Telecommunications Networks". J. ofAdaptive Behavior, vol. 5, no. 2, pp. 169-207, 1996.

[16] Y. Zhang, L. D. Kuhn, and M. P. J. Fromherz, "Improvements on AntRouting for Sensor Networks," International Workshop on Ant ColonyOptimization and Swarm Intelligence, Sep. 2004.

[17] Y. Lu, G. Zhao, and F. Su, "Adaptive Ant-based Dynamic RoutingAlgorithm," In Proceedings of the 5th World Congress on IntelligentControl and Automation (IEEE, Hangzhuo, China, June 2004), 2694-2697.

[18] L. P. Kaelbling, M. L. Littman and A. W. Moore, "ReinforcementLearning: A Survey". J. ofArtificial Intelligence Research, 4, pp. 237-285, 1996.

[19] D. Subramanian, P. Druschel, and J. Chen, "Ants and reinforcementlearning: A case study in routing in dynamic networks," In ProceedingsofIJCAI, pp. 832-838,1997.

6