Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong...

11
44 Int. J. Computational Science and Engineering, Vol. 9, Nos. 1/2, 2014 Copyright © 2014 Inderscience Enterprises Ltd. TernCam: an automated energy-efficient visual surveillance system Chia-Pang Chen, Cheng-Long Chuang, Tzu-Shiang Lin, Chun-Yi Liu and Joe-Air Jiang Department of Bio-Industrial Mechatronics Engineering, National Taiwan University, 10617 Taipei, Taiwan E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National Taiwan University, 10617 Taipei, Taiwan E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] *Corresponding author Abstract: The Matsu Islands in Taiwan are an archipelago of islands surrounded by the sea. Some of the islands are chosen as the breeding places by many kinds of migratory birds. Among these sea birds, one of the critical endangered species is the Chinese Crested Tern (Thalasseus bernsteini). In the past, its habitual ecological behaviour has been inspected by manual observation with trivial information. Thus, we developed a wireless real-time visual surveillance system to monitor the terns, so the in-situ situation could be captured immediately. It is expected that the system can unravel the mysterious ecological behaviour of the tern. Meanwhile, in order to be suitable for the circumstances of wild islands, the system is designed with a specific packet transmitting strategy capable of increasing image validity while decreasing power consumption. The experimental results show the system is feasible for ecological monitoring and able to effectively preserve the image completeness. Keywords: TernCam; visual surveillance system; tern surveillance; Chinese Crested Tern; wireless communication; energy efficiency. Reference to this paper should be made as follows: Chen, C-P., Chuang, C-L., Lin, T-S., Liu, C-Y., Jiang, J-A., Yuan, H-W., Chiou, C-R. and Hong, C-H. (2014) ‘TernCam: an automated energy-efficient visual surveillance system’, Int. J. Computational Science and Engineering, Vol. 9, Nos. 1/2, pp.44–54. Biographical notes: Chia-Pang Chen received his PhD in Bio-Industrial Mechatronics Engineering from National Taiwan University, Taipei, Taiwan in 2011. He is currently a Postdoctoral Research Fellow with the Department of Bio-Industrial Mechatronics Engineering at National Taiwan University. He is also an Adjunct Assistant Professor in the Department of Computer Science at National Taipei University of Education, Taipei, Taiwan. Cheng-Long Chuang received his two PhDs in Biomedical Engineering and Bio-Industrial Mechatronics Engineering from National Taiwan University, Taipei, Taiwan in 2010. He is currently a Post-Doctoral Research Fellow with the Department of Bio-Industrial Mechatronics Engineering at National Taiwan University. He is also an Adjunct Assistant Professor in the Department of Computer Science at National Taipei University of Education, Taipei, Taiwan. Tzu-Shiang Lin is currently a doctoral candidate in the Department of Bio-Industrial Engineering at National Taiwan University. His research foci are routing protocols, medium access control protocols and localisation algorithms for wireless sensor networks. His research interests also include wireless sensor networks applied to environmental, ecological, and urban air quality monitoring.

Transcript of Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong...

Page 1: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

44 Int. J. Computational Science and Engineering, Vol. 9, Nos. 1/2, 2014

Copyright © 2014 Inderscience Enterprises Ltd.

TernCam: an automated energy-efficient visual surveillance system

Chia-Pang Chen, Cheng-Long Chuang, Tzu-Shiang Lin, Chun-Yi Liu and Joe-Air Jiang Department of Bio-Industrial Mechatronics Engineering, National Taiwan University, 10617 Taipei, Taiwan E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] E-mail: [email protected]

Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National Taiwan University, 10617 Taipei, Taiwan E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] *Corresponding author

Abstract: The Matsu Islands in Taiwan are an archipelago of islands surrounded by the sea. Some of the islands are chosen as the breeding places by many kinds of migratory birds. Among these sea birds, one of the critical endangered species is the Chinese Crested Tern (Thalasseus bernsteini). In the past, its habitual ecological behaviour has been inspected by manual observation with trivial information. Thus, we developed a wireless real-time visual surveillance system to monitor the terns, so the in-situ situation could be captured immediately. It is expected that the system can unravel the mysterious ecological behaviour of the tern. Meanwhile, in order to be suitable for the circumstances of wild islands, the system is designed with a specific packet transmitting strategy capable of increasing image validity while decreasing power consumption. The experimental results show the system is feasible for ecological monitoring and able to effectively preserve the image completeness.

Keywords: TernCam; visual surveillance system; tern surveillance; Chinese Crested Tern; wireless communication; energy efficiency.

Reference to this paper should be made as follows: Chen, C-P., Chuang, C-L., Lin, T-S., Liu, C-Y., Jiang, J-A., Yuan, H-W., Chiou, C-R. and Hong, C-H. (2014) ‘TernCam: an automated energy-efficient visual surveillance system’, Int. J. Computational Science and Engineering, Vol. 9, Nos. 1/2, pp.44–54.

Biographical notes: Chia-Pang Chen received his PhD in Bio-Industrial Mechatronics Engineering from National Taiwan University, Taipei, Taiwan in 2011. He is currently a Postdoctoral Research Fellow with the Department of Bio-Industrial Mechatronics Engineering at National Taiwan University. He is also an Adjunct Assistant Professor in the Department of Computer Science at National Taipei University of Education, Taipei, Taiwan.

Cheng-Long Chuang received his two PhDs in Biomedical Engineering and Bio-Industrial Mechatronics Engineering from National Taiwan University, Taipei, Taiwan in 2010. He is currently a Post-Doctoral Research Fellow with the Department of Bio-Industrial Mechatronics Engineering at National Taiwan University. He is also an Adjunct Assistant Professor in the Department of Computer Science at National Taipei University of Education, Taipei, Taiwan.

Tzu-Shiang Lin is currently a doctoral candidate in the Department of Bio-Industrial Engineering at National Taiwan University. His research foci are routing protocols, medium access control protocols and localisation algorithms for wireless sensor networks. His research interests also include wireless sensor networks applied to environmental, ecological, and urban air quality monitoring.

Page 2: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

TernCam: an automated energy-efficient visual surveillance system 45

Chun-Yi Liu is pursing his MS in Bio-Industrial Mechatronics Engineering from National Taiwan University.

Joe-Air Jiang received his MS and PhD in Electrical Engineering from National Taiwan University, Taipei, Taiwan in 1990 and 1999. Currently, he is a Professor in the Department of Bio-Industrial Mechatronics Engineering at National Taiwan University, Taiwan. He is the Principal Investigator of several large-scale integrative projects funded by the National Science Council and the Council of Agriculture of the Executive Yuan, Taiwan. His specialties in power transmission systems are computer relaying, solar generation systems, fault detection, fault classification, fault location, power quality event analysis, and smart grid systems. Besides, his areas of interest are diverse, and cover wireless sensor network technologies, biomechatronics, neuro-engineering, bio-effects of electromagnetic wave, automatic systems for agroecological monitoring with WSN, and low-level laser therapy.

Hsiao-Wei Yuan received her PhD in Natural Resources from Cornell University, and in Forest Science from Colorado State University, respectively. She is Professor and Associate Professor in the School of Forestry and Resource Conservation at National Taiwan University.

Chyi-Rong Chiou received her PhD in Natural Resources from Cornell University, and in Forest Science from Colorado State University, respectively. She is Professor and Associate Professor in the School of Forestry and Resource Conservation at National Taiwan University.

Chung-Hang Hong is pursuing his PhD in School of Forestry and Resource Conservation at National Taiwan University.

This paper is a revised and expanded version of a paper entitled ‘Energy-efficient visual eyes system for wildlife’ presented at The Workshop on Compiler Techniques for High-Performance and Embedded Computing (CTHPC 2011) and The International Workshop on Embedded Multi-Core Computing and Applications (EMCA 2011), June 2–3, 2011 and September 2–4, 2011 at Taiwan and Canada, respectively.

1 Introduction

The traditional techniques to monitor wildlife are labour intensive and cost greatly. The use of camera systems in wildlife monitoring, however, allows large data collection and increases the size of the sampled area without human presence or requiring additional observers. Moreover, using cameras also offers biologists a glimpse into the secret lives of wildlife by capturing the breeding, feeding and migrant habits. Thus, the cameras can provide the information regarding habitat preferences and population fluctuation. The goal of protecting wildlife from poachers who go undetected can also be achieved by using cameras as surveillance tools.

With the advance in manufacturing technologies of electronic devices, digital cameras have been applied to different domains, such as mass public transportation, enterprise security, industry safety, habitat surveillance, natural resource surveillance, and even military defenses. Among these domains, wildlife surveillance has been at the centre of attention in the recent years. One of the reasons is that many studies indicate that wildlife-based ecotourism would affect animals (Nevin and Gilbert, 2005; Walker et al., 2006). In general, however, monitoring systems for wildlife do not work in a real-time manner. These systems shoot photos/videos and store them into a storage memory. The photos/videos can be manually retrieved from the memory after a certain period of time. Such an off-line recording method, however, cannot immediately trace the information regarding wildlife dynamics.

Many studies have presented non-real-time off-line systems for wildlife surveillance. For example, Wawerla et al. (2009) utilised the solar panels, a battery, and a camera to build a camera system to monitor grizzly bears at the Ni’iinlii Njik (Fishing Branch) Park in Canada. The captured video was periodically forwarded to the base station and was stored in hard drives via a radio signal of 2.4 GHz. Song and Goldberg (2006) built a camera system mounted on a power line pole to assist the research of the ivory billed wood-pecker. The video frames were automatically analysed using a motion detection technique. Sabine et al. (2005) proposed a time-lapse video photography system to observe avian nest activities and depredation using a 12-volt deep cycle battery and a recorder. In another study (Newbery and Southwell, 2009), the authors emphasised both on effective long-term surveillance and on automated operating procedures in a remote area such as Polar Regions. They designed a solar-powered system composed of a digital single lens reflex (DSLR), a flexible solar panel, a battery, and a camera controller. All captured images were stored into memory cards and were able to be downloaded when visiting the site in person. They thought that building wireless connections was is a very complicated and energy consumed task, so it was difficult to achieve the goal of real-time image surveillance through wireless communication in wild areas. Furthermore, other non-real-time image surveillance systems (He, 2009; King et al., 2011; Proudfoot, 1996) have been proposed. For

Page 3: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

46 C-P. Chen et al.

instance, He (2009) designed an energy-efficient portable video communication system to record what animals see and stored the images in memory cards. Once the animals came within the communication range of a router, the compressed images stored in memory cards were going to be uploaded to the router through wireless transmission, and thus the storage space of memory cards would be released for further use.

Employing wireless communication technologies to carry out image monitoring tasks is a prevalent method (He, 2009; King et al., 2011; Wawerla et al., 2009), since wireless connection is able to improve the flexibility of deploying a monitoring system and the immediacy when transmitting surveillance data. Among the wireless technologies, global system for mobile (GSM) communications is a very popular telecommunication technology all over the world, and its service has expanded to more than 214 countries in the world. The GSM service has an extensive coverage, since thousands of base stations have been deployed as dense as possible. Especially in Taiwan, the density of GSM base stations is rather high. With an additional service like general packet radio service (GPRS), the data transfer rate can be up to 48 kbps. Hence, it is appropriate to transfer images by GPRS. Using GPRS, not only can labour costs caused by collecting the data from storage devices be reduced, but also real-time surveillance for wildlife can be achieved through promoting the immediacy of image transmission from event observation. More importantly, GPRS can overcome the difficulty encountered in physical wired deployment. Given the benefits brought by GPRS, many studies or commercial organisations have developed GPRS-based cameras (Colorado Video, Inc., 2012; Duval Security Ltd., 2012; TELTONIKA Ltd., 2009). However, we have found that most of these existing commercial GPRS-based cameras are suitable for the indoor use. Few can be practically applied to the wildlife surveillance by far. Hence, it is necessary to develop a feasible GPRS-based camera/visual surveillance system that is capable of steadily operating for a long time under severe weather conditions and harsh environments.

The Chinese Crested Tern (Thalasseus bernsteini) is one of the least known and possibly the rarest seabirds, because the entire population was reported to be fewer than 50 and many thought that the seabird became extinct approximately 30 years ago (BBC Online News, 2000; Candido, 2006; Tacific Wildlife Foundation, 2010). The seabird is among critically endangered species and was first described in 1863. It was spotted only a few times over a hundred years (Liang et al., 2000; The Pacific Wildlife Foundation, 2010). Fortunately, it was found on a tiny islet of Matsu Islands near Taiwan in 2000 (iMail newspaper 8th August 2000). Afterwards, the seabirds’ nests were discovered on four islets. These islets and their surrounding ones are listed as a wild bird sanctuary by the Taiwanese government. From 2000 to 2010, the bird was found breeding on the islets from

April to September each year. In order to reduce the interference during ecological observation, observing from a distance is typically used to investigate the ecological behaviours of the Chinese Crested Tern during its breeding period, but this method may not provide important information regarding the breeding of the seabird, because the observers cannot get too close to the seabird. Hence, it is necessary to develop an efficient and effective observation method without bringing a negative impact on the seabird.

In order to assist the government in conducting research on the Chinese Crested Tern and initiating protecting programmes for the seabird, we design and implement a less expensive solar-powered GPRS image surveillance system for the precious tern. The proposed system is able to provide real-time in-situ images via the GPRS communication technique, which is different from general data-log-based cameras with a deficiency of timeliness. Based on the mechatronic integration technique, we integrate a digital camera, a FPAG circuit used to control the camera, and a GPRS modem, with a MSP430 microcontroller. All the designs aim at achieving energy efficiency, because there is no electric power source near the habitat of the Chinese Crested Tern. Furthermore, retaining the completeness of received packets is a critical issue as the signal strength of GPRS may attenuate or become unstable, especially in a wild region with a weaker GPRS signal coverage. To address this issue, we design an energy-efficient mechanism based on the file transfer protocol (FTP). The mechanism is able to effectively reduce both energy consumption and data loss rates.

According to the results of performance evaluation and practical tests in the field during the period from May 2011 to October 2011, the proposed system has successfully accomplished the image surveillance tasks. We also successfully record the images of breeding activities of the Chinese Crested Tern through the proposed system.

This paper is structured as follows. In Section 2, we introduce the system architecture of the proposed visual surveillance system (TernCam). Section 3 illustrates the energy-efficient mechanism used to control the operation of the system, including managing the scheduling process, snapping and transmission process, and composing process. In Section 4, the remote control centre is introduced. In Section 5, we discuss several performance evaluation results and present the visual surveillance results. Finally, Section 6 concludes the study.

2 System architecture of the proposed TernCam

The main objective of the visual surveillance system is to automatically snap images and transmit them to a distant control centre via the GPRS communication technique. The TernCam will establish the Internet connection first and then connect to the FTP server built in the control centre.

Page 4: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

TernCam: an automated energy-efficient visual surveillance system 47

Through the connection with the FTP server, a number of smaller sized images generated by the TernCam can be uploaded and stored. The smaller sized files comprise the image data and specified header information. Afterwards, the LabVIEW-based management programme will synthesise the images using these stored files using the interpolation technique.

Numerous challenges have to be overcome when the TernCam is deployed and operates in the Matsu tern sanctuary consisting of several rocky outcrops. The challenges are:

x very serve weather conditions and harsh environments (e.g., heavy rain, high temperature, high humidity, high salinity)

x no available electrical power source

x very weak GPRS signal to achieve real-time monitoring

x how to minimise the impact brought by the proposed system on the wildlife on the islets.

2.1 Hardware design The proposed system is composed of several components as shown in Figure 1. The MSP430 microcontroller (TI Corp., 2012) is responsible for managing all the operating procedures and communicating with all of the external devices. The MSP430 microcontroller features the low energy consumption and the stable operation performance, so it is very suitable for the long-term usage in outdoor environment, such as Jiang et al. (2008) has presented an automatic ecological monitoring system of oriental fruit fly using the same TI MSP430 microcontroller. The FPGA circuit responsible for controlling the digital camera is able to receive the commands from the MSP430 microcontroller and transmit the image to the microcontroller via a RS232 interface. Connected with the brightness detection circuit with a photo-resistor, the FPGA circuit can detect the environmental brightness and then change the exposure time to acquire photos with better quality. Moreover, the GPS receiver (San Jose Technology, Inc., 2012) receives the exact location of the deployed system and the time of detection from the system and forwards the information to the MSP430 microcontroller.

For the goal of conserving energy, a power switch circuit is used to turn on/off the power switch to control the photography devices (the FPGA circuit, the digital camera and the brightness detection circuit) which consume the majority of energy. The circuit receives the control signal from the MSP430 microcontroller to determine whether the photography devices are deactivated. By doing so, the energy consumption could be effectively reduced. Since there is no electrical power source on the islets, the proposed system is powered by the solar energy.

Figure 1 The hardware architecture of the proposed energy-efficient TernCam (see online version for colours)

2.1.1 Power source – solar energy Power is one of the main challenges in this study. How to maintain the operation of the TernCam for a longer time is the important issue. Solely relying on batteries to sustain system operation is not a good idea, since the whole observation time for the breeding activity of the Chinese Crested Tern is around four to six months. Thus, we develop a solar energy supply system equipped with four 12 V100AH batteries and two solar panels. The batteries can be recharged by transforming the solar energy into the electrical power.

2.1.2 FPGA circuit and digital camera A high-resolution digital camera module (the TRDB_D5M camera module made by Terasic Technologies, Inc., 2012) is used in this study. However, because of the limited RAM storage and executing speed of the MSP430 microcontroller, we employ the FPGA circuit (the DEO board made by Terasic Technologies, Inc.) to develop a bridge controller to handle the data processing between the camera module and MSP430 microcontroller. On the other hand, the FPGA circuit is able to automatically adjust the exposure time of the camera module by obtaining the illuminance value from the brightness detection circuit in order to avoid the overexposure or underexposure.

3 The energy-efficient mechanism

For the TernCam, we design an energy-efficient mechanism to control all operations. The mechanism consists of three processes. They are

1 the scheduling process

2 the snapping and transmission process

3 the composing process.

The first two processes are implemented on the front-end TernCam, and through the processes in-suit images are periodically acquired and transferred to the remote FTP

Page 5: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

48 C-P. Chen et al.

server using an image decomposition approach. With the image decomposition, the entire image would be completely preserved after wireless transmission. Unlike the first two processes, the composing process is carried out on the LabVIEW platform (NI, 2012) installed on the FTP server in the base station. In the composing process, the decomposed images received from FTP are recombined, and the bilinear interpolation (Sonka et al., 2007) is used to replace the missing image data. The detailed illustration regarding these processes is provided as follows.

3.1 The scheduling process The scheduling process will determine when to initiate the snapping and transmitting process according to the predefining time schedule. When the system time acquired by the GPS receiver meets the predefining time, the devices (the photography devices and the GPRS modem) will be first powered on, and then the snapping and transmitting process will be performed. Otherwise, the system switches to a standby mode to conserve energy. After the transmission of an image completes, the photography devices and the GPRS modem will be turned off until next snapping and transmitting process begins. The flowchart of the scheduling process is shown in Figure 2.

Figure 2 The flowchart of the scheduling process of the visual surveillance system (see online version for colours)

3.2 The snapping and transmission process Due to the limited memory space and the operating ability of the MSP430 microcontroller, we designed a process to help maintain stable and smooth data transfer from the FPGA circuit to the base station. The MSP430 microcontroller communicates with the FPGA circuit through the five data pins comprising Enable, Start, Next, Repeat, and Tx as shown in Figure 3. Among the pins, the pin Tx is in accordance with the RS232 specification (the baud rate used here is 115,200 bps). The pin Enable is used to activate the photography devices, and the pin Start is used to initiate the snapping process designed for the photography devices. More importantly, the FPGA circuit decomposes the images that come from the camera into several data blocks, and every block consists of m bytes of image data. Hence, each image can be divided into 921,600/m blocks, because the size of each image is 640 × 480 × 3 bytes (24-bit RGB system). It is known that the overhead will increase if m is smaller. Once the FPGA circuit detects the logic ‘0’ in the pin. Next during the process, the FPGA circuit will transfer next block. Similarly, if the logic ‘0’ is detected in the pin Repeat, the FPGA circuit will repeat transferring the current block. By doing so, the problem of limited memory space of the MSP430 microcontroller can be solved, and the completeness of image data can be improved as the MSP430 microcontroller can determine whether the FPGA circuit needs to transmit the received block again. The actions corresponding to different logic combinations of Next, Repeat, and Start are summarised in Table 1.

Figure 3 The pins used to make a communication between the FPGA circuit and the MSP430 microcontroller (see online version for colours)

Table 1 The corresponding actions of logic combinations of the pins enable, start, next, and repeat

Action Logic of enable

Logic of start

Logic of next

Logic of repeat

Inactive mode 1 1 1 1 Snapping a photo 0 0 1 1 Transmitting a new data block

0 1 0 1

Repeating transmitting the current data block

0 1 1 0

Awaiting mode 0 1 1 1 Error mode 0 1 0 0

The MSP430 microcontroller will form a new packet according to the received block and transmit the packet to the FTP server. The packet format is defined in Figure 4. In the packet, the block is enclosed by the Start Byte and End

Page 6: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

TernCam: an automated energy-efficient visual surveillance system 49

Byte, and a unique ID is included. Only the packet with such a structure is regarded as a valid packet. A cyclic redundancy check (CRC)-16 code is generated and attached to the block in order to ensure that the received data is correct. If the regenerated CRC code is not equal to the received CRC code, the block will be discarded. The missing pixel data will be interpolated during the composing process.

Figure 4 The designed packet format used for FTP (see online version for colours)

In the FTP transmission, we use n binary files to store different parts of an image. The availability of the GPRS connection will be checked when every binary file is created by the TernCam on the FTP server. Once the disconnection occurs, a new connection will be established again. Otherwise, the continuing transmission will lead to the result that the data becomes invalid, since the connection does not exist. According to such approaches, it is obvious that the transmission overhead will decrease when n × m is larger, but the data loss rate may increase, since the frequency of checking for the GPRS connection will decrease. It can also be found that the number of binary files influences the effectiveness of the composing process, and further discussions will be seen in the next section.

3.3 The composing process According to the definition of the packet in the Section 3.2, each binary file consists of 921,600/(n × m) blocks. The composing process implemented on the LabVIEW platform will compose the images from the binary files stored in the FTP server using the bilinear interpolation, a linear interpolating function of two variables (x and y) on a regular grid. Because every packet has a unique ID, it is easy to find the corresponding position of every pixel in an image with a 640 × 480 resolution. After finding the available pixels from the received binary files, the bilinear interpolation will be used to replace the missing pixels. The bilinear interpolation takes the four pixels around every missing pixel into consideration, that is, the interpolation will be performed along the x-direction and then again along the y-direction. An illustration is shown in Figure 5, where Q is the pixel with an unknown RGB value, and J11, J12, J21, J22 are the possible four pixels with known RGB values. The interpolation formula is defined by

� � � � � � � �

� � � � � � � �

� � � � � � � �

11 12 21 22

11 12 21 22

11 12 21 22

( )

( )

( ) ,

R R R RR

G G G GG

B B B BB

f J f J f J f Jf Q

uf J f J f J f J

f Qu

f J f J f J f Jf Q

u

� � �

� � �

� � �

(1)

where u denotes the number of available pixels (1 d u d 4), and fR(Q), fG(Q), and fB(Q) denote the R, G, and B value of the pixel Q at coordinate (x, y) (x, y � Z, 1 d x d 640, 1 d y d 480), respectively. If each of the four pixels is invalid, the fR, fG, and fB are equal to zero, and the value of u will decrease by one. Note that the interpolation will not be performed when all of the four pixels are invalid.

The bilinear interpolation can effectively repair the image with a number of missing pixels caused by the unstable communication. In Figures 6(a) and 6(b), for example, the white lines represent a segment of invalid pixels. By contrast, Figures 6(c) and 6(d) show the outcomes after employing the bilinear interpolation. Hence, the image can be effectively restored through the composing process.

Figure 5 An illustration of the bilinear interpolation used in the composing process (see online version for colours)

Figure 6 Two samples that show the images before and after the bilinear interpolation, (a) and (b) are the original images; and (c) and (d) are the interpolated images (see online version for colours)

(a) (b)

(c) (d)

Note: White lines indicate the pixels lost.

Page 7: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

50 C-P. Chen et al.

4 The remote control centre

The control centre is mainly responsible for receiving all the image data transmitted from the filed TernCams and composing them. The centre is equipped with an FTP server controlled by the LabVIEW management programme (LMP). The LMP will activate the composing process if it detects binary image files stored in the FTP server as shown in Figure 7. In addition, the LMP can also filter the abnormal image files. On the other hand, when the sudden malfunction occurs, we allow the LMP to issue a reset command to the distant TernCams via the short message service (SMS) based on the GSM communication technique. It is necessary to take measures to reset the TernCams, because severe weather may influence the normal operation of the TernCams, e.g., higher temperature may cause the system to crash. In the long run, the reset mechanism may also extend the lifetime of the system.

Figure 7 The LabVIEW-based management programme in the control centre (see online version for colours)

Figure 8 The practical deployment of two TerCams on an islet inside the Matsu sanctuary (see online version for colours)

Notes: The control centre was located at the campus of

National Taiwan University. The distance between the TerCams and the remote control centre was near 250 km.

5 Performance evaluation

In this section, we present the performance evaluation results of the proposed TernCam. The performance evaluations were implemented on a wild island of Taiwan (E 119°59’31.456”, N 26°15’28.865”) and in an indoor setting. First of all, two TernCams (no. 1 and no. 2) about five metres apart were placed on a wild islet inside the Matsu sanctuary for Chinese Crested Tern observation from May 12 to October 24, 2011. The arrangement and layout of the two practically deployed TerCams are shown in Figure 8. The photo resolution of TernCam 1 is 1280 × 700, which is higher than that of TernCam 2(640 × 480). In order to evaluate the performance of the proposed energy-efficient mechanism, the pixel loss rate (%) for every received image is examined. On the other hand, energy consumption and transmission time are also the foci of the evaluation. In the indoor experiment, both the energy consumption and time required for transmitting a 640 × 480 image were tested.

5.1 The evaluation of the pixel loss rate Because the TernCams were placed on a wild island to monitor the migrant seabirds where the signal strength of GPRS was usually weak, retaining a steady and effective image transmission was vital to the success of the proposed system. Figure 9 shows the poor GPRS signal quality over the island, based on the service information provided by the telecom operator (Taiwan Mobile Co., Ltd., 2012). In order to increase the successful transmission rate, the proposed mechanism is able to determine whether the network connection is available when every binary file is created in the FTP server. If the connection failed, a new connection will be established. In the following experiments, we first examine the pixel loss rate for each returned image and then find the influence of the number of created binary files on the pixel loss rate, i.e., the influence of value of n on the pixel loss rate.

Figure 9 The quality of the GSM signal (see online version for colours)

Source: Evaluated by the Taiwan Mobile

Co., Ltd. (2012)

Page 8: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

TernCam: an automated energy-efficient visual surveillance system 51

5.1.1 Statistics of the data loss rate for the two TernCams

TernCam 1 transmits five images to the remote control centre each day at 06:00, 09:00, 12:00, 15:00 and 18:00, while TernCam 2 transmits seven images to the remote control centre each day at 06:00, 08:00, 10:00, 12:00, 14:00, 16:00 and 18:00. They have been deployed on the islet from May 12 to October 24, 2011. Because TernCam 2 has a lower image resolution, the number of binary sub-files to be transmitted (i.e., n) is set to 10, but we let n equal to 100 for TernCam 1 with a higher resolution. This is because the high resolution corresponds to a larger file size. Thus, letting n be larger, i.e., increasing the number of connection check, can avoid a large loss of pixels.

The frequency distribution of the pixel loss rate for all images generated by TernCam 1 is shown in Figure 10. There are 302 images successfully received by the remote control centre. Among the images, only four images have a 90%~100% pixels loss. The pixel loss rates of most of the 302 images are below 10%. This suggests that the proposed mechanism is able to preserve the completeness of image data and achieve a higher success rate of image transmission.

Figure 11 shows the frequency distribution of the pixel loss rate for all 403 images transmitted from TernCam 2. From Figure 10, we can know that the pixel loss rates are below 20% in 76.67% of the images. However, it is obvious that the proportion of the images with a higher pixel loss rare (70%~100%) is larger than that generated by TernCam 1, although the cameras are only 5 m apart. One of the possible reasons is that the value of n is smaller (n = 10), so the frequency of connection check decreases. Since the TernCam is placed in an area with a weaker GPRS signal, increasing the frequency of connection check is able to increase the stability of transmission, but the time required for transmission will also increase.

Figure 10 The frequency distribution of the pixel loss rate (%) of 302 images transmitted from TernCam 1 (from May 12 to October 14, 2011) (see online version for colours)

Figure 11 The frequency distribution of the pixel loss rate (%) of 403 images transmitted from TernCam 2 (from May 12 to October 14, 2011) (see online version for colours)

5.1.2 Impact of sub-files on the pixel loss rate In another experiment that examined the impact of different numbers of transmitted sub-files on the pixel loss rate, the TernCam totally transmitted 32 images with a 640 × 480 photo resolution. Half of these images were separated into ten binary files and forwarded to the base station, i.e., n = 10. And the rest of images were separated into 100 files, i.e., n = 100. Figure 12 shows that the average rates of pixel loss under both conditions, either n = 10 or n = 100, are 19.14 and 10.03, respectively. Note that only the packet includes pixel data and a correct corresponding CRC-16 code is regarded as a valid packet. The results show that the pixel loss rate is smaller when n = 100 compared to when n = 10. Obviously, using larger separated binary files to transmit images has a better effect on preserving the completeness of images, because the frequency of checking whether the network connection is available would increase. By doing so, however, the whole transmission time will increase since checking the connection requires a longer time.

Figure 12 The box plot regarding the pixel loss rate (%) when n = 10 and n = 100 (see online version for colours)

Page 9: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

52 C-P. Chen et al.

5.2 The evaluation of energy consumption and transmission time

In the experiment, we varied m from 300 to 900 with an increment of 150. Each case was tested 30 times, and n = 10. During each transmission of an image, both the total energy consumption and required transmission time were recorded. From Figure 13, it can be observed that not only the energy consumption but also the transmission time decrease as m increases, because the overhead of packet transmission decreases.

In general, increasing the value of m leads to the reduction of both energy consumption and transmission time, but a significant increase of the pixel loss rate may occur as long as the network connection fails. This is because a larger amount of data encompassed in a packet will be abandoned once the LabVIEW fails to check the CRC code during the composing process. Thus, appropriately determining m as well as n is very vital for the TernCam. Specifically, it is necessary to preset more proper parameters depending on a variety of monitoring environments.

According to the above experimental result, we can summarise that improving image completeness is accomplished by setting a larger value of n, and that energy efficiency and transmission time reduction are achieved by employing a larger value of m. In other words, setting a larger n and a smaller m may avoid data loss, especially when monitoring systems are deployed in the wild areas with a weak GPRS signal, e.g., secret forest, remote mountain, sea island, etc. On the other hand, a larger m and a smaller n are suitable for the system located in the areas with a stronger and stable GPRS signal, e.g., urban districts, residential areas, etc. Through the parameter adjustment, the proposed system becomes very flexible, and can be applied to different environments.

Figure 13 The averaged energy consumption and averaged time for transmitting one image under five different conditions: (m, n) = (300, 10), (450, 10), (600, 10), (750, 10), and (900, 10) (see online version for colours)

300 450 600 750 9000.220.240.260.280.300.320.340.360.380.400.420.440.460.480.500.520.54

Tran

smis

sion

tim

e pe

r im

age

(sec

ond) Energy consumption (AH)

Transmission time per image

Ener

gy c

onsu

mpt

ion

(AH

)

m

2040

2060

2080

2100

2120

2140

2160

2180

2200

5.3 Results of visual surveillance The Chinese Crested Terns usually nest with other species including the Great Crested Terns (Sterna bergii), Roseate Tern (S. dougallii), Bridled Tern (S. anaethetus), Blacknaped Tern (S. sumatrana), and Black-tailed Gull (Larus crassirostris). In addition, they look like the Great Crested Terns, although the former have smaller size, black tips, and paler upper parts. Hence, they are difficult to identify. After the two TernCams were deployed, we found that Some Chinese Crested Terns laid eggs on May 26, 2011, as shown in Figure 14. During the experimental period, we totally spotted the Chinese Crested Tern three times through the images collected by the TernCams, as shown in Figures 15 to 17. The red arrows in the figures indicate the Chinese Crested Tern. The other terns that look like the Chinese Crested Tern are all Great Crested Terns.

Figure 14 The returned image on May 26, 2011, from TernCam 1 (see online version for colours)

Figure 15 The returned image on May 26, 2011, from TernCam 2 (see online version for colours)

Page 10: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

TernCam: an automated energy-efficient visual surveillance system 53

Figure 16 The returned image on May 27, 2011, from TernCam 2 (see online version for colours)

Figure 17 The returned image on May 30, 2011, from TernCam 1 (see online version for colours)

6 Conclusions

In this paper, we design and deploy real-time TernCams for wildlife surveillance on the islets inside the Matsu sanctuary. By integrating the digital devices with the proposed energy-efficient mechanism, the TernCams successfully complete the image-monitoring task. According to the experimental results in the field, it is found that increasing the number of files transmitted to the base station would decrease the pixel loss rate, and that increasing the size of designed packets would reduce the energy consumption and the required time for the transmission of one image. This useful information will help us to improve the performance of the TernCams deployed in different monitored areas and will serve as a design reference for other researchers to develop similar image surveillance systems. Moreover, the TernCams actually capture the images of the precious species of tern, the Chinese Crested Tern, during the experimental period. We expect that this visual surveillance system can assist the ecological research on the Chinese Crested Tern in the future.

Although the primary experimental result shows that the TernCam is a promising monitoring tool, there is still some room for improvement. For example, it is possible to

integrate image compression technologies with the proposed system. With the technologies, the amount of transmitted data can be significantly reduced, so the energy consumption will be decreased and transmission time will also be drastically shortened.

Acknowledgements

This work was financially supported by the Council of Agriculture, Taiwan, under contract no. 101AS-13.3.1- FB-e1, 101AS-7.1.2-BQ-B2, and 101AS-7.1.2-BQ-B1, the National Science Council, Taiwan, under contracts no. NSC 100-2218-E-002-005, the National Taiwan University, and Intel Corporation under grant no. NSC 100-2911-I-002-001, and 10R70501. The authors would also like to thank the Lienchiang County Government for the official permission of experiments inside the tern conservation district.

References BBC Online News (2000) ‘Chinese seabird returns from

extinction’, (online) available at http://news.bbc.co.uk/2/hi/science/nature/852849.stm (accessed on 7 January 2012).

Candido, E.P.M. (2006) ‘Chinese Crested Tern: observations on juveniles in the Matsu Archipelago of Taiwan’, Birding ASIA, Vol. 6, pp.34–35.

Colorado Video, Inc. (2012) ‘Remote mobile GSM-GPRS wireless internet snapshot camera – observer IV’, (online) available at http://www.colorado-video.com/observer.html (accessed on 6 January 2012).

Duval Security Ltd. (2012) ‘Rapid deployment GSM wireless camera’, (online) available at http://www.gprscamera.co.uk/GSMCameras/RD3GSMFeaturesSpecif%20ications/tabid/67/language/en-GB/Default.aspx (accessed on 6 January 2012).

He, Z. (2009) ‘Energy-efficient integrated camera and sensor system design for wildlife activity monitoring’, Proceedings of the IEEE International Conference on Multimedia and Expo, New York, USA, pp.1580–1581.

Jiang, J-A., Tseng, C-L., Lu, F-M., Yang, E-C., Wu, Z-S., Chen, C-P., Lin, S-H., Lin, K-C. and Liao, C.S. (2008) ‘A GSM-based remote wireless automatic monitoring system for field information: a case study for ecological monitoring of the oriental fruit fly, Bactrocera dorsalis (Hendel)’, Computers and Electronics in Agriculture, Vol. 62, No. 2, pp.243–259.

King, D.I., DeGraaf, R.M., Champlin, P.J. and Champlin, T.B. (2011) ‘A new method for wireless video monitoring of bird nests’, Wildlife Society Bulletin, Vol. 29, No. 1, pp.349–353.

Liang, C-T., Chang, S-H. and Fang, W-H. (2000) ‘Little known oriental bird: discovery of a breeding colony of Chinese Crested Tern’, OBC Bulletin, Vol. 32.

National Instruments (NI) (2012) LabVIEW, available at http://www.ni.com/labview/ (accessed on 6 January 2012).

Nevin, O.T. and Gilbert, B.K. (2005) ‘Perceived risk, displacement and refuging in brown bears: positive impacts of ecotourism?’, Biological Conservation, Vol. 121, No. 4, pp.611–622.

Page 11: Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* · 2017-04-24 · Hsiao-Wei Yuan, Chyi-Rong Chiou and Chung-Hang Hong* School of Forestry and Resource Conservation, National

54 C-P. Chen et al.

Newbery, K.B. and Southwell, C. (2009) ‘An automated camera system for remote monitoring in polar environments’, Cold Regions Science and Technology, Vol. 55, No. 1, pp.47–51.

Proudfoot, G.A. (1996) ‘Miniature video-board camera used to inspect natural and artificial nest cavities’, Wildlife Society Bulletin, Vol. 24, No. 3, pp.528–530.

Sabine, J.B., Meyers, J.M. and Schweitzer, S.H. (2005) ‘A simple, inexpensive video camera setup for the study of avian nest activity’, Field Ornithol, Vol. 76, No. 3, pp.293–297.

San Jose Technology, Inc. (2012) ‘Mini GPS locator’, (online) available at http://eni.so-buy.com/ezfiles/eni/img/img/110267/GM-44-FB.pdf (accessed on 6 January 2012).

Song, D. and Goldberg, K. (2006) ‘Acone: automated collaborative observatory for natural environments’, (online) available at http://www.c-o-n-e.org/acone/ (accessed on 6 January 2012).

Sonka, M., Hlavac, V. and Boyle, R. (2007) Image Processing, Analysis, and Machine Vision, 3rd ed., Thomson Engineering, Toronto, Canada.

Taiwan Mobile Co., Ltd. (2012) ‘Coverage inquiry system’, (online) available at http://m-internet.taiwanmobile.com/internet/cover_map.jsp (accessed on 6 January 2012).

TELTONIKA Ltd. (2009) ‘EDGE camera (MVC100) user’s manual’, (online) available at http://www.farnell.com/datasheets/491562.pdf (accessed on 6 January 2012).

Terasic Technologies, Inc. (2012) ‘TRDB_D5M camera module’, available at http://www.terasic.com.tw/cgi-bin/page/archive. pl?Language=English&No=281 (accessed on 21 May 2012).

The Pacific WildLife Foundation (2011) ‘Introduction of the Chinese Crested Tern’, (online) available at http://www.pwlf.org/chinesecrestedtern.htm (accessed on 7 January 2012).

TI Corp. (2012) ‘MSP430 microcontroller’, (online) available at http://www.ti.com/lsds/ti/microcontroller/16-bit_msp430/overview.page (accessed on 7 January 2012).

Walker, B.G., Boersma, P.D. and Wingfield, J.C. (2006) ‘Habituation of adult Magellanic penguins to human visitation as expressed through behavior and corticosterone secretion’, Conservation Biology, Vol. 20, No. 1, pp.146–154.

Wawerla, J., Marshall, S., Mori, G., Rothley, K. and Sabzmeydani, P. (2009) ‘Bearcam: automated wildlife monitoring at the arctic circle’, Journal of Machine Vision Applications, Vol. 20, No. 5, pp.303–317.