1

ACTIVITY-LEVEL MONITORING USING A DOPPLER RADAR SENSOR PLATFORM

By

GABRIEL A. REYES

A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT

OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE

UNIVERSITY OF FLORIDA

2011

2

© 2011 Gabriel A. Reyes

3

ACKNOWLEDGMENTS

This thesis could not have been completed without the support of many people in

my personal life and professional career. First, I would like to thank my mentor and

advisor, Dr. Jenshan Lin, for giving me the opportunity to join the Radio Frequency

Circuits and Systems Laboratory as an undergraduate student at University of Florida

and for providing me with guidance and support over the past few years as a research

assistant. Special thanks to my thesis committee members, Dr. Xiaolin „Andy‟ Li and Dr.

Sumi Helal, for taking the time to meet with me and provide valuable feedback and

ideas for this project. I have enjoyed sharing and learning from them as a researcher

and student in their Cyber-Physical Systems and Mobile & Pervasive Computing

courses.

I would like to acknowledge all those who have worked with me on this project,

including Di Wang, Rakesh Nair, Changzhi Li and Xiaogang Yu. I also appreciate my

colleagues and friends at the Radio Frequency Circuits and Systems Laboratory for

supporting my research and sharing ideas and suggestions. Finally, I would like to thank

my parents, my sister and the rest of my family for their love, encouragement and

support.

4

TABLE OF CONTENTS

page

ACKNOWLEDGMENTS .............................................................................................. 3

LIST OF TABLES ........................................................................................................ 6

LIST OF FIGURES ...................................................................................................... 7

ABSTRACT ................................................................................................................. 9

CHAPTER

1 INTRODUCTION ................................................................................................ 11

Motivation ........................................................................................................... 11 Contributions ....................................................................................................... 14

Thesis Organization ............................................................................................ 14

2 RELATED RESEARCH ...................................................................................... 16

Non-Contact Sensing Technologies ................................................................... 16

Health and Activity Monitoring Systems .............................................................. 17 Behavior Imaging ................................................................................................ 21

3 DOPPLER RADAR SENSOR ............................................................................. 24

Technology Overview ......................................................................................... 24

Sensor Specifications ......................................................................................... 27

4 HEALTH MONITORING PLATFORM ................................................................. 29

System Architecture ............................................................................................ 29

Platform Design .................................................................................................. 30 Activity-Level Monitoring ..................................................................................... 33 Computer Vision ................................................................................................. 39

Object Detection ........................................................................................... 39 Object Tracking ............................................................................................ 41

Low-Power Communication Link ........................................................................ 45

802.15.4 – ZigBee Protocol .......................................................................... 45 X-CTU Software ........................................................................................... 47 Communication Architecture ........................................................................ 47

Microprocessor Development ............................................................................. 48 Requirements ............................................................................................... 48 Capabilities ................................................................................................... 49

Web Services ...................................................................................................... 53

5

5 SYSTEM INTEGRATION TESTING ................................................................... 56

Vital Signs Radar Sensor Results ....................................................................... 56 Activity-Level Monitoring Results ........................................................................ 57

Platform Monitoring Results ................................................................................ 62

6 CONCLUSION AND FUTURE WORK ................................................................ 63

LIST OF REFERENCES ........................................................................................... 66

BIOGRAPHICAL SKETCH ........................................................................................ 69

6

LIST OF TABLES

Table page 4-1 XBee communication addresses. ................................................................... 48

4-2 Features of Texas Instruments‟ MSP430F1232IPW. ...................................... 49

7

LIST OF FIGURES

Figure page 1-1 Typical two bedroom floor plan ....................................................................... 13

1-2 System level block diagram of activity-level monitoring platform .................... 13

2-1 System architecture for mobile-phone based accelerometry study of human physical activity levels .................................................................................... 20

2-2 Video capture and manual annotation tool ..................................................... 22

3-1 Baseband spectrum detected by Doppler radar sensor .................................. 25

3-2 Block diagram and picture of radar antenna board ......................................... 28

3-3 Doppler radar sensor boards .......................................................................... 28

4-1 Activity-level sensing platform system overview ............................................. 30

4-2 Activity-level sensing platform mounted from ceiling for experiments ............. 32

4-3 System level block diagram of activity-level monitoring platform .................... 32

4-4 Screenshot of National Instruments LabVIEW activity-level moving average . 35

4-5 Screenshot of National Instruments LabVIEW activity-level monitoring back-end programming environment and code ....................................................... 36

4-6 Screenshot of National Instruments LabVIEW activity-level monitoring user interface and visualization .............................................................................. 38

4-7 Screenshot of National Instruments LabVIEW activity-level monitoring user interface and furniture layout .......................................................................... 38

4-8 Screenshot of OpenCV C++ face detection application .................................. 40

4-9 Screenshot of OpenCV C++ face detection application output ....................... 41

4-10 Object tracking and servo movement based on face detection ...................... 43

4-11 GWS S125 3T sail winch servo ...................................................................... 44

4-12 Servo wiring connected to microcontroller PWM ............................................ 44

4-13 BDMICRO MAVRIC-IIB microcontroller board top view ................................. 51

4-14 BDMICRO MAVRIC-IIB microcontroller board top view ................................. 52

8

4-15 BDMICRO MAVRIC-IIB microcontroller board layout ..................................... 52

4-16 LabVIEW web services architecture ............................................................... 54

4-17 LabVIEW web services address ..................................................................... 54

4-18 Snapshot of LabVIEW web services from Google Chrome browser ............... 55

5-1 Vital signs radar sensor detected heartbeat comparison with fingertip transducer....................................................................................................... 56

5-2 Activity-level user interface in state of “inactivity” ........................................... 58

5-3 Activity-level user interface in state of “low” activity ........................................ 58

5-4 Activity-level user interface in state of “high” activity ...................................... 59

5-5 Furniture layouts within user interface ............................................................ 59

5-6 Historical activity-level data collected using LabVIEW .................................... 60

5-7 Historical activity-level data collected using LabVIEW .................................... 60

5-8 Historical activity-level data collected using LabVIEW .................................... 61

5-9 Historical activity-level data collected using LabVIEW .................................... 61

5-10 Activity-level interface with web camera overlay............................................. 62

5-11 OpenCV library face and error detection ........................................................ 62

9

Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science

ACTIVITY-LEVEL MONITORING USING A DOPPLER RADAR SENSOR PLATFORM

By

Gabriel A. Reyes

August 2011 Chair: Jenshan Lin Major: Electrical Engineering

With recent developments in the field of radar sensing and computer vision

algorithms, new and practical assistive living applications based on these technologies

are being developed in research and commercial sectors. The need for assistive living

and smart home technologies will see greater demand arising from an increase in the

aging population around the world. Non-contact sensing technologies have the potential

to revolutionize home healthcare delivery and research on diseases such as dementia,

autism and sleep apnea.

Based on developments in the area of Doppler radar sensing at the University of

Florida‟s Radio Frequency Circuits and Systems Research Laboratory, an activity-level

monitoring platform has been designed and developed. Relevant information on how

active people are during the day based on their movement is obtained using software

algorithms and a non-contact Doppler radar deployed in a hospital or living space. A

continuous 5.8 GHz radar wave emitted by the sensor is phase-modulated by the

physiological chest wall movement of the person observed. The range of the radar

sensor is on the order of 1.8 to 2 meters. The physiological movements are transformed

into activity-level data by observing the magnitude and frequency of the radar signals

10

over time. The radar sensor is deployed as part of a sensing platform equipped with

wireless data capabilities, a mounted video camera and servo motors to detect and

track the person as they move throughout the sensing space. The rotating platform

mounted on the ceiling uses computer vision algorithms to adjust the position of the

radar antenna and ensures accurate positioning to effectively capture the person‟s

movements. The sensor data collected is transmitted wirelessly via ZigBee IEEE

802.15.4 protocol to a central monitoring station capable of storing historical

measurements and computing the person‟s activity levels. Web services, a graphical

user interface and mobile phone access to data allow caregivers and relatives to easily

visualize a person‟s activity levels over time.

The thesis will discuss the development of the activity-level monitoring hardware

and software platform. Particular focus is placed on the algorithm used to extract activity

levels from non-contact biosensor data, digital processing considerations, and wireless

communication links. In addition, an overview of vital signs sensing techniques, related

healthcare monitoring systems and behavior imaging tools is presented.

11

CHAPTER 1 INTRODUCTION

Motivation

According to the United Nations in 2005 [1], 20 percent of the world‟s population

was aged 60 years or over. It is estimated that by 2050 that proportion will increase to

32 percent. In the developing world, the proportion of the population aged 60 or over is

expected to rise from 8 percent in 2005 to 20 percent in 2050. The percentage of the

elderly population is increasingly rapidly and the US Census Bureau estimated that by

2050 the percentage of people aged 65 and older in the United States will exceed the

total number of children less than five years for the first time in history [2].

These aging trends have major implications on the future of healthcare delivery,

policy and quality of life for people around the world. With the growing emphasis on the

adoption and impact of health-related information technology, researchers and

practitioners are increasingly focusing on the design of interactive systems for a variety

of sensing and monitoring capabilities. Despite this progress, however, obstacles still

remain to achieve the vision of ubiquitous sensing and computing. Limited power

availability, network connectivity, deployment and maintenance costs are among a few

of the major problems hindering the widespread use of embedded sensors.

Autonomous health care monitoring is becoming an essential part of modern

medical systems. Much of modern medicine would simply not be possible or cost-

effective without sensors such as the thermometers, blood pressure monitors, glucose

monitors and electrocardiography. Major motivations for automated healthcare

monitoring relating to this project include:

12

Improved monitoring of patients in the emergency room, as well as in work and home environments using embedded sensor technologies.

An increasing aging population. An elder‟s vital signs need to be monitored during daily living to address emergency situations and improve quality of life.

Enable early detection of a variety of symptoms leading to prevention efforts against the onset of chronic diseases.

Account for shortage of nursing staff in hospitals and caregivers at home potentially leading to under-monitored patients.

With a non-contact radar sensor and the current advances in system-on-a-chip

technology, a sensor platform capable of detecting a person‟s vital signs while at rest

and their activity levels while in motion is poised to make a very strong impact on the

engineering and medical community. Because these devices can be deployed in a

variety of locations (home, hospital, emergency room, office space, etc.), activity levels

can be monitored in real-time to update the health and lifestyle conditions of a person or

group of people, and report any variations in activity behavior observed over time.

Figure 1-1 is a representation of a typical two bedroom floor with an overlay of sensors.

Sensing is highly directional with static sensors (i.e. kitchen, bedroom) and

omnidirectional using a rotating sensor platform tracking human movements (i.e. living

room). Ultimately, the health information collected can help prevent the onset of chronic

diseases and may also encourage people to lead a more active and healthy lifestyle.

The overall block diagram of the activity-level monitoring Doppler radar sensor platform

can be found in Figure 1-2.

13

Figure 1-1. Typical two bedroom floor plan. Includes overlay of sensors deployed in bedroom, living room and kitchen to detect activity levels.

Figure 1-2. System level block diagram of activity-level monitoring platform.

14

Contributions

The objective of this research is to utilize a Doppler radar sensor designed to

detect a person‟s vital signs, namely heart rate and respiration rate, and develop a

human activity-level monitoring platform to understand behaviors over time and

encourage a healthy lifestyle. The battery-powered sensor node is capable of detecting

human activity levels up to 2 meters away and transmits data, via a low-power wireless

communication link, to a central monitoring station. The specific goals of the research

provided in this thesis are to:

Present related research in non-contact sensing and health-related IT systems, as well as introduce behavior imaging and its potential impact in treating and understanding dementia, autism and other diseases.

Develop a low-power sensor platform to detect human activity levels based on a person‟s physiological movement captured using a non-contact Doppler radar sensor.

Utilize and enhance OpenCV computer vision algorithms to detect a person‟s presence within the sensor platform‟s observation range and adjust the platform‟s position to ensure accurate physiological movement detection.

Develop a software algorithm to transform the biosensor data collected by the Doppler radar sensor into activity levels and design a flexible user interface to visualize a snapshot of historical and real-time activity levels.

Design the microcontroller systems and wireless communication links to enable sensor platform integration and data signal processing, as well as perform a full system integration and test monitoring activity levels.

Thesis Organization

Chapter 2 introduces related research work in the areas of non-contact sensing

technologies, health and activity monitoring systems, and behavior imaging. Chapter 3

presents an overview and the specifications of the Doppler radar technology used to

collect vital signs data. Chapter 4 details the overall health and activity monitoring

15

platform, including system architecture, computer vision algorithms, microcontroller

system, wireless communication links, and the software development. Chapter 5

emphasizes the full system integration of the platform and includes results from the vital

signs radar sensor, activity-level monitoring results, and a case study results using the

monitoring platform. Finally, a project summary and future work related to the expansion

of sensing capabilities, integration with third-party networks, as well as other hardware

and software enhancements are detailed in Chapter 6.

16

CHAPTER 2 RELATED RESEARCH

The areas of related research concerning this project span a wide branch of

topics, including smart home technologies, radar sensor technologies, behavioral

science and behavior imaging. The project focus is divided into two areas of interest.

The first area is the individual sensor technology being used to collect vital signs data

while a person is at rest and to collect activity-level information while the person is in

motion. The second area is the integration of the sensor into a robust and integrated

system to take advantage of the information collected within the human environment.

The following sections will describe related research in non-contact vital signs sensors,

health and activity monitoring systems, and on-going projects using behavior imaging

tools to understand, recognize and classify human activities.

Non-Contact Sensing Technologies

Current medical techniques for monitoring heart rate, respiration rate and other

phenomena use electrodes attached to the body. These methods are impractical for

patients recovering from surgery or the elderly who are encouraged and in many cases

required to maintain an active lifestyle. A number of techniques for non-contact sensing

technologies are presented.

A capacitive sensor for detecting heartbeat rate without direct contact with the

skin is investigated by Oum et al. [3]. Precordial movements change the capacitance

between patch electrodes embedded in a person‟s clothing and modulate the frequency

of a Colpitts oscillator. Heartbeat and respiration data can be obtained by demodulating

the oscillating signal. In addition, heartbeat signals are extracted from the demodulated

signal and a bandpass filter is used to separate the harmonics of heartbeat frequency.

17

A non-invasive pneumatic sensor is presented by Watanabe et al. [4] and is

placed under the bed mattress to measure human vital functions. The small movements

attributable to a person‟s vital signs are measured as changes in pressure using a

pressure sensor having an almost flat frequency response from 0.1 to 5 kHz and a

sensitivity of 56 mV/Pa. Using the newly developed system, heartbeat, respiration,

apnea, snoring and body movements are measured.

Non-contact sensors and wireless sensor networks have also been used for

surveillance systems, a form of activity-level measurement, within the home, office, and

other indoor environments. Passive infrared motion sensors (PIR sensors) are ideal

because they do not require any signal or devices on the object or person to be tracked

and they can function in dark environments as well. Song et al. [5] analyze the

performance and the applicability of the PIR sensors for security systems and propose a

region-based human tracking algorithm in a real environment.

Health and Activity Monitoring Systems

Increasing demand on public healthcare services due to the aging population has

become a major problem in developed and underdeveloped countries. In parallel with

the advances in ubiquitous computing technologies, extensive research is being carried

out in using sensor networks and automated healthcare systems for home and hospital

care environments [6]. There are various related healthcare monitoring and information

technology systems working to solve the problems of healthcare delivery which utilize

many of the vital signs sensing techniques discussed in the previous section.

Jeonggil et al. [7] MEDiSN is an emergency room monitoring system. In hospital

scenarios, there are large groups of patients requiring medical treatment. However, due

to limited hospital capacity, priority of treatment must be determined and assigned.

18

MEDiSN is a system that assists doctors, nurses and caregivers with calculating triage

according to patients‟ vital signs. A similar system could be deployed in an assistive

living facility and combined with non-contact sensing techniques to monitor elderly

residents.

New born infants are also highly susceptible to illness and infection, but one of

the leading causes of infant mortality is Sudden Infant Death Syndrome (SIDS). SIDS

strikes without warning causing unexplained deaths in infants from one month to one

year of age. The SleepSafe system [8] uses infant clothes-embedded SHIMMER sensor

nodes and another base station to detect an infant‟s sleeping position and reduce the

risk of sudden death. Baby Glove [8] is another solution to monitor infant health and

encompasses two integrated sensor plates which contain a thermostat temperature

sensor, along with electrodes that monitor the child‟s pulse rate and hydration. The

system monitors the vital signs information from sensors via a data acquisition module,

organizes the measurements into packets, and transmits them wirelessly to the second

mote connected to the base station computer for processing.

Fernandez-Lopez et al. [9] propose a monitoring system based on spatially

distributed ZigBee networks. „Health Monitoring for All‟ is a system prototype consisting

of body wearable health monitoring sensors, a ZigBee network and Wi-Fi backbone

infrastructure to implement seamless continuous monitoring of patients. The system‟s

main contribution is the use of ZigBee networks to gather data from body sensors and it

uses a ZigBee gateway to connect to a Wi-Fi connection. This system enables effective

backbone architecture deployment and inspired the use of ZigBee communication

protocols for this project.

19

System interaction and connectivity with mobile phones and applications is also a

growing trend. A remote healthcare monitoring system for Congestive Heart Failure

(CHF) prevention is presented by Suh et al. [10]. The sensor system consists of various

sensors, including Bluetooth-based weight scales, blood pressure monitors, Personal

Activity Monitors (PAMs), traditional cell phones, an Apple iPhone smartphone, and an

SMS message server system used to monitor heart failure patients remotely. Another

system presented by Wood et al. [11] collects weight, blood pressure, and collects daily

SMS surveys. Whenever patients measure their weight and blood pressure, data is

transmitted to a web server via Bluetooth and an Internet connection. The responses to

daily SMS questionnaires and calorie expenditure values are also stored in the

database. The data is accessible through a custom-built web application or through an

iPhone smartphone application. The system allows physicians to monitor patients in

real-time and provides daily feedback on vital signs and other important information.

Searching through the literature, it is clear that there is a large research focus on

recognizing and classifying activities. However, monitoring and capturing activity levels

could prove to be extremely valuable for anticipating disease and health patterns over

time. It seems activity-level data is generally not easily collected or

20

Figure 2-1. System architecture for mobile-phone based accelerometry study of human physical activity levels [12].

closely studied. Accelerometry studies have been reported for gait and activity analysis

among elderly subjects. Hynes et al. [12] introduced a system solely utilizing

accelerometers embedded in off-the-shelf cellular handsets to remotely monitor activity

characteristics of elderly patients at home and in the community. The system provides a

non-intrusive and potentially easily accepted methodology to monitor and analyze a

patient‟s daily activity characteristics and present historically data collected over time.

21

Behavior Imaging

Behavior imaging (BI) is concerned with the capture, process, analysis, and

visualization of behavior in order to support the professional practices of behavior

analysts. Behavior imaging encompasses a set of tools and information technology that

researchers and caregivers may use to better understand and treat patients and loved

ones. Behavior imaging tools bring together knowledge from various research areas

including information visualization, digital signal processing, networking and ubiquitous

computing.

Researchers have used behavior imaging to develop methods for measuring,

recognizing, and quantifying children‟s social and communicative behavior. Data

acquisition has been accomplished using video, audio, and wearable sensors. The most

important objective of behavior imaging tools is to provide parents, doctors and

caregivers with a way of capturing the effect and the cause of certain behavioral

responses in children and adults.

The goal of behavior imaging as it relates to this project is to capture behavioral

incidents in a person‟s natural environments. For example, an interesting task would be

to monitor the activity levels of a person recovering from surgery at home or an elderly

person living alone. Reporting this information back to a doctor, caregiver or relative

could be extremely useful in not only treating the patient but also minimizing the effects

of aging in place, the onset of dementia and the development of other chronic diseases.

Researchers believe that “human behavior is inherently multi-modal, and individuals use

eye gaze, hand gestures, facial expressions, body posture, and tone of voice along with

speech to convey engagement and regulate social interactions."

22

Figure 2-2. Video capture and manual annotation tool. The tool supports richer reflection on behaviors and therapy. [13]

The National Science Foundation‟s „Expeditions in Computing‟ program supports

the initiatives of the Center for Computational Behavioral Science and a group of

researchers working on developing novel computation methods for measuring and

analyzing the behavior of children and adults during face-to-face social interactions. By

applying technology to highly complex problems in the area of human behavior,

behavior imaging tools will allow for modeling, analyzing and visualizing health

conditions, developmental disorders and communicative behaviors. Most importantly, BI

will provide the framework for researchers and doctors to scan through vast amounts of

data collected in the hospital, home, office and school environments [14]. A video

capture tool is shown in Figure 2-2 and the faces of the people in the video have been

blurred for privacy concerns.

In contrast to sensor-based home healthcare systems, extensive research has

been conducted to investigate the use of computer vision techniques and low cost video

23

systems for monitoring and assessing daily activities of occupants [15]. Audio and video

have been major technologies used to enable studies and direct observation by highly

trained specialists in the behavior imaging field. Brumitt et al. proposed a vision-based

activity monitoring system which can be used for homecare applications [16], Nait-

Charif et al. presented a simple vision system in a supportive home environment for

activity recognition and fall detection [17], and Gao et al. proposed fusing motion

segmentation with tracking to understand eating behaviors of patients [18]. With the

increasing number of elderly relying on homecare, improved monitoring and analysis

systems are crucial for maintaining and improving the quality of life for the elderly.

24

CHAPTER 3 DOPPLER RADAR SENSOR

Technology Overview

The remote non-contact detection of vital signs signals, namely heart rate and

respiration, based on microwave Doppler radar and the phase modulation effect has

been reported in the literature for many years [19]. The technique was initially used to

enable non-contact detection of vital signs of humans or animals from a distance,

without the use of any attached sensors [20], [21]. An observation of microwave Doppler

radar under linear approximation is that the higher the carrier frequency, the shorter the

wavelength and thus the higher the detection sensitivity. Based on this rule, microwave

vital sign detectors have been designed from 450 MHz [22] to 2.5 GHz [23].

The sensor emits an unmodulated radio frequency signal which is transmitted

toward the human body. The signal is phase-modulated at the human body by the

periodic physiological movement and reflected back to the receiver antenna. The

reflected RF signal is captured by a receiver antenna and amplified by a low-noise

amplifier (LNA) and a two-stage variable gain preamplifier. The local oscillator signal

and the received signal are mixed together, amplified by a two-stage baseband

amplifier. The non-contact vital sign detection technique consists of sensing

physiological movement in the millimeter or centimeter range.

Signal processing on the radar board is kept to a minimum and the data is

transmitted to a central base station for analysis. The biodata is collected using

LabVIEW to determine activity-levels and is analyzed in the time domain for simplicity

and speed. Spectrum estimation methods such as the fast Fourier transform (FFT) are

25

Figure 3-1. Baseband spectrum detected by Doppler radar sensor. Displays the respiration peak, harmonics in signal and heartbeat peak. [25]

also performed within LabVIEW to determine the baseband spectrum which provides a

more detailed look at breathing rate, harmonics in the detected signal and heartbeat.

The baseband spectrum captured by the Doppler radar sensor is shown in Figure 3-1.

One of the major issues with the technique is the presence of random body

movement which produces a significant source of noise for accurate detection.

However, random body movement cancellation techniques were explored [24] by

sensing from the front and back of the human body.

Presently, the majority of monitoring systems require contact detection, forcing

patients or elders to wear uncomfortable body sensors. Contact body sensors lead to a

variety of issues and might cause patients to feel unnecessary anxiety or concern.

26

There are medical cases in which patients‟ skin is extremely sensitive and easily

irritable, so a non-contact monitoring technique is preferred. Ubiquitous sensing and

non-contact monitoring have the potential to provide invisibility in healthcare delivery

systems and are a promising approach to improving quality of life, vital signs and

activity-level measurements.

Normally, non-contact techniques require targets to be stationary for identification

and measurement purposes. The added inconvenience prevents non-contact vital sign

monitoring from being deployed in a dynamic real-world clinical environment. Computer

vision provides an additional dimension to both contact and non-contact sensing

technologies. Algorithms capable of face detection, blob and object tracking provide a

powerful means of performing localization, activity recognition and episode sensing. The

key to the adoption of assisted living technologies lies in providing patients with secure,

private and invisible technologies that are able to fuse data from multiple sensing

sources and intelligently make decisions to the user‟s benefits.

Hu et al. [24] also proposed an intelligent non-contact wireless patient monitoring

system using Doppler radar. By detecting the Doppler shift, the system is able to

continuously monitor a patient‟s vital signs using non-contact and non-invasive

techniques. By using the same principle, the system may determine whether the target

is moving or not.

Non-contact sensors and wireless sensor networks have also been used for

surveillance systems, a form of activity-level measurement, at home, office, and other

environments. Passive infrared motion sensors (PIR sensors) are ideal because they do

not require any signal or devices on the object or person to be tracked and they can

27

function in dark environments as well. However, compared to either infrared or visible

light, microwave has greater penetration capabilities through concrete and other

construction materials, which creates attractive capabilities for both civilian and military

applications. Similarly, compared to commercially available motion sensors, microwave

alternatives may also be used for detection in total darkness, provide greater sensitivity

in measurements, and are optimized for vital signs sensing.

Sensor Specifications

The Doppler radar sensor used in this project operates at 5.8 GHz and is a low-

gain version of the system. The single Rogers printed circuit board (RO4350B)

integrates quadrature transceivers, a two-stage baseband amplifier, and the power

management circuit within a board size of 6.8 cm × 7.5 cm. To reduce the hardware

cost and the requirement of signal processing speed, the amplified baseband output

signals were converted using a separate analog-to-digital converter and

microprocessor. A low sampling rate of 20 Hz is used, which is sufficient for the vital

sign signal typically less than 2.5 Hz. Figure 3-2 below shows the block diagram and the

low-gain circuit board used for the experiments. Figure 3-3 includes two identical vital

signs sensor boards. The board requires a 6-9 volt input and the outputs are two-

channel baseband analog signals, since it is quadrature detection system. To adjust the

operation range (i.e., the distance between the radar and the subject being monitored),

a gain block is used. The receiver chain contains a low noise amplifier (LNA), two

stages of adjustable gain block, and the down-conversion mixer, which is a compact I/Q

mixer utilizing two standard double balanced mixer cells and a 90 degree hybrid

fabricated in a GaAs MESFET process.

28

Figure 3-2. Block diagram and picture of radar antenna board. A) Block diagram and

b) photo of the low-gain circuit board. [25]

Figure 3-3. Doppler radar sensor boards. Two identical vital signs Doppler radar boards and patch antennas displayed next to a quarter. [25].

29

CHAPTER 4 HEALTH MONITORING PLATFORM

The target application for the vital signs radar sensor is home healthcare. With

microelectronics design and low-cost manufacturing, the vital sign sensor size could be

greatly reduced and embedded in a sensor platform for a variety of applications. For

example, monitoring sleep apnea of infants and adults is a potential high impact

application. Lack of information and awareness of sleep apnea is widespread. Many

adults suffer from sleep apnea without knowing it. An integrated sensing platform

capable of detecting a person‟s vital signs while at rest and their activity levels while in

motion is could significantly improve quality of life and provided valuable information on

a person‟s health and well-being.

System Architecture

At a high level, the sensor platform consists of four main components: a sensing

component, object detection and tracking component, server component, and web

services component. The respective functions of each section are detailed here:

Sensing: responsible for monitoring target‟s vital signs and activity levels using Doppler radar, as well as sending biosensor data wirelessly to central monitoring station for data processing.

Object Detection and Tracking: responsible for observing the sensor area to detect a patient or person in the camera‟s video frame. Based on feedback from the server, a servo motor controls the platform to track a person‟s movements.

Server: interfaces with the sensing component, object detection and tracking component, and the web services components as the backbone infrastructure of the system platform.

30

Web Services: external component to visualize server user interface over the web and responsible for sharing data with third-party groups, including hospital

networks, doctors, caregivers or relatives.

Figure 4-1. Activity-level sensing platform system overview.

The components above are broken down into individual parts that make up the

platform‟s overall architecture. These individual components will be discussed in detail

in the following sections.

Platform Design

The mobile sensing platform has been constructed of 0.220” thick transparent

Plexiglas. A two tier platform was designed using a 17” x 11” piece for the top level and

a 10” x 8” piece for the bottom level. The bottom and top layers of Plexiglas are

connected using aluminum brackets and the following components are mounted on the

lower level of the platform: MSP430 microprocessor and XBee transmitter board, a

second XBee transmitter board, MAVRIC-IIB microcontroller board, 1 x 6V battery,

31

on/off switch, and the GWS S125 3T servo. The servo is placed in an inverted position

and attaches to the easy antenna tracker pan/tilt kit. The wooden assembly kit is

designed to hold the Doppler radar sensor and the wireless web camera. A picture of

the activity-level sensing platform is shown here in Figure 4-2. The block diagram of the

platform is included in Figure 4-3.

32

Figure 4-2. Activity-level sensing platform mounted from ceiling for experiments.

Figure 4-3. System level block diagram of activity-level monitoring platform.

33

Activity-Level Monitoring

Accurate and reliable information concerning the physical activity levels of

individuals is often of great importance to medical professionals during the diagnosis

and treatment of certain disorders as well as a patient‟s recovery period after

undergoing a medical procedure. The general well-being of an elderly patient is

reflected in the amount and levels of activities (e.g. walking) that they undertake during

their normal daily life. Elderly patients recovering from surgery are also typically

assigned an exercise regime, which they must adhere to when released from hospital

care in order to maximize recovery rates.

Monitoring of energy expenditure levels in people who suffer from diabetes in

their home environment is important in controlling the progression of the disease. In

addition, the progression of many degenerative neurological and cognitive disorders

from which the elderly may suffer impacts both gait and general activity patterns.

Existing research has shown that with the use of simple contact sensors combined with

RFID, basic behavior profiling can be achieved for the care of the elderly. However, the

general lack of richer movement information means that precursors to certain adverse

events cannot be detected.

The activity-level computation occurs in the server component of the mobile

platform based on the collected biosensor data. Various averaging algorithms are used

to determine activity and it is possible for the user to adjust the sensitivity of the system

to certain activities.

A moving average is a type of finite impulse response filter used to analyze a set

of data points by creating a series of averages of different subsets of the full data set

[26]. Given an array of biosensor data collected from the vital signs sensor, the moving

34

average can be obtained by first taking the average of the first subset of n data points.

The parameter n is determined experimentally based on the response of the vital signs

sensor. Best approximations for responsiveness and sensitivity to a variety of constant

activity levels over time and spontaneous activities are 25 to 100. The fixed subset size

is then shifted forward, creating a new subset of numbers, which is averaged. This

process is repeated over the entire data series. The plot line connecting all the (fixed)

averages is the moving average. A moving average is a set of numbers, each of which

is the average of the corresponding subset of a larger set of data points.

A moving average is used with time series data from the vital signs sensor to

smooth out short-term fluctuations and highlight longer-term trends or cycles. The

threshold between short-term and long-term depends on the application, and the

parameters of the moving average will be set accordingly and may be modified

programmatically by the caregiver or system administrator. Mathematically, a moving

average is a type of convolution and so it can be viewed as an example of a low-pass

filter used in signal processing When used with non-time series data, a moving average

filters higher frequency components without any specific connection to time, although

typically some kind of ordering is implied.

A cumulative moving average consists of averaging all the data recorded up until

a certain point. As each new data point is collected, the resulting average is computed

until the specified number of data points to average has been reached. The number of

data points to average is specified by the user and affects the sensitivity and volatility of

the resulting average.

35

The cumulative moving average equation used is:

Viewed simplistically it can be regarded as smoothing the data. An example is shown in

Figure 4-4. The green data points are collected from the vital signs sensor, while the red

line is the resulting moving average calculation using n=25 data points. It is possible to

observe that as the frequency of high magnitude radar data is collected, the cumulative

moving average increases as well.

Figure 4-4. Screenshot of National Instruments LabVIEW activity-level moving

average. Display of activity-level moving average signals captured.

The moving average calculation is completed using National Instruments‟

LabVIEW software. LabVIEW is a graphical programming environment used by

engineers and scientists to develop sophisticated measurement, test, and control

systems using intuitive graphical icons and wires that resemble a flowchart. It offers

extensive integration with thousands of hardware devices and provides hundreds of

36

built-in libraries for advanced analysis and data visualization. The LabVIEW platform is

scalable across multiple targets and Oss and is known as an industry leader [27]. A

screenshot of the activity-level monitoring software code is shown in Figure 4-5.

Figure 4-5. Screenshot of National Instruments LabVIEW activity-level monitoring back-end programming environment and code.

LabVIEW also contains a comprehensive collection of drag-and-drop controls

and indicators to quickly and easily create user interfaces for a variety of applications

and effectively visualize results without integrating third-party components or building

views from scratch. The quick drag-and-drop approach does not come at the expense

of flexibility. Power users can customize the built-in controls via the Control Editor and

programmatically control user interface (UI) elements to create highly customized user

experiences. The control user interface displays the vital signs sensor time domain data

37

(top left), as well as the resulting activity-level moving average (left) is shown in Figure

4-6. The interface also includes a custom-built control that dynamically adjusts

magnitude from red to green, representing inactivity to high levels of activity based on

the sensor data. In addition, custom furniture layouts of a home, office, or hospital

space can be easily integrated into LabVIEW to represent the interactivity between the

sensor in the physical world and activity-levels computed in the digital world. Concentric

circles are used to visualize activity in a particular area of the layout, as observed in

Figure 4-7, and it is easy to duplicate and overlay activity-levels on multiple locations on

the layout if multiple sensors are deployed in the space. Due to the quadrature nature of

the radar detection system, the user interface also includes observed heartbeat and

respiration from two channels I and Q. The I/Q signals are collected in the form of

voltage from the vital signs sensor and the magnitude and frequency patterns are used

to compute the estimated heartbeat and respiration rates observed (top right). A simple

filter is used to display to display zero readings for heartbeat and respiration when the

magnitude of collected is zero or nearly zero.

38

Figure 4-6. Screenshot of National Instruments LabVIEW activity-level monitoring user interface and visualization.

Figure 4-7. Screenshot of National Instruments LabVIEW activity-level monitoring user interface and furniture layout. The display includes medium activity (red to orange) on the left and high activity (red to green) on the right.

39

Computer Vision

OpenCV (Free Open Source Computer Vision) is a library composed of

functions, which are used to support real time computer vision analysis. OpenCV is

under BSD license for distribution and free use in any commercial or research

purposes. The framework has a C and C++ interface which can be used for all

development purposes. The prototype in this project uses the C++ based interfaces.

OpenCV has a wide variety of uses which includes human-computer interaction (HCI),

object identification, segmentation and recognition, face recognition, gesture

recognition, motion tracking, among others.

Object Detection

The sensing component which includes the Doppler radar sensor provides best

results obtaining a person‟s vital signs when the antenna is directed towards the

person‟s chest wall. However, it is impossible to assume that a person will be within the

observation range of the activity-level platform at all times. Hence, it is necessary to

incorporate a mechanism which allows for the platform, as well as the antenna, to

realign once a person is detected and their motion is captured. The goal is for the

Doppler radar sensor to realign with the person‟s chest wall and thus provide

continuous monitoring of health and activity levels.

The goal of detecting a person and successfully tracking them is completed using

computer vision technologies. In the system‟s current implementation, OpenCV libraries

are used to detect a person‟s face, track the person‟s movement, and realign the

monitoring platform to maintain visual contact and continuously monitor vital signs and

activity levels. An example of a person‟s face detected in the view of the web camera is

40

shown in Figure 4-8. The blue circle represents the detected face and the person‟s face

is blurred for privacy concerns.

Figure 4-8. Screenshot of OpenCV C++ face detection application.

Facial recognition is an active research area specializing on how to recognize

faces within images or videos. Face recognition compliments face detection. Face

detection is the process of finding a 'face' within images or videos and face recognition

is the process of matching the detected 'face' to one of many faces known to the file

system or stored in a database. The OpenCV library used in this project is for face

detection to classify if a „face‟ has been detected and utilizes a Haar Cascade classifier.

The classifier performs the operation of analyzing the video feed as a series of

static images, while determining featured within each image and classifying as a "face"

or "not face". Haar-like features [28] are digital image features used in object

recognition. A Haar-like feature considers adjacent rectangular regions at a specific

location in a detection window, sums up the pixel intensities in these regions and

41

calculates the difference between them. This difference is then used to categorize

subsections of an image, such as the eyes, nose, cheeks, and other facial

characteristics for recognition.

The key advantage of a Haar-like feature over most other features is its

calculation speed. Due to the use of integral images and stored features, a Haar-like

feature of any size can be calculated in constant time (approximately 60 microprocessor

instructions for a 2-rectangle feature). Based on experimental results and as observed

in Figure 4-9, the detection time using the Haar cascade classifier is less than 100

milliseconds.

Figure 4-9. Screenshot of OpenCV C++ face detection application output. Detection time remains fairly constant less than 100 milliseconds.

Object Tracking

For the purpose of tracking the motion of a person, previously detected in a

frame, a 360 degree servo motor is used to rotate the monitoring platform. The platform

42

includes the camera and the Doppler radar sensor. As soon as the face detection and

tracking module detect that a person, previously detected, has moved from their current

position, the system determines the direction of movement and adjusts the platform in

that direction.

After a face has been detected, the detection and tracking module, calculate a

bounding circle with its center being the center of the detected face as shown in Figure

4-8. For every frame obtained from the camera, the application determines the

coordinates for the position of the center circle. If it is varying, it is indicative of human

motion.

If the person is moving in a horizontal direction the variation will only occur for the

X-coordinate of the subsequent centers. If the motion is in the vertical direction the

variation will be only for the Y-coordinate. The center of the frame is (320,240) pixels

and the monitoring platform is centered such that the center of the bounding circle

remains as a reference point at (320,240) pixels. As the person moves, the application

will track the person‟s movement and the center of the bounding circle will move

accordingly. If the new coordinates are found to be different to the last known

coordinates and the difference is beyond a predetermined threshold of 40 pixels, the

application will request the servo motor to adjust the monitoring platform and center

along the new coordinates for the detected face.

As soon as it is detected that the person has moved towards the left, the

detection and tracking module sends a trigger to the servo, signaling it to move towards

the left. In the first step, the detection and tracking module sends out a signal to a

microcontroller chip along with the information about the direction to move the servo, i.e.

43

left or right. In the second step, the microcontroller propagates the shift instruction

information to the servo causing the desired movement. The servo is programmed to

move incrementally in small steps of 10 degrees until the center of the bounding circle,

constituting the human face detected, equals the new center of the frame located at

(320,240) pixels. A schematic of object tracking and servo movement left to right is

shown in Figure 4-10.

Figure 4-10. Object tracking and servo movement based on face detection.

The reason for adjusting the direction of the monitoring platform is to ensure that

the Doppler radar sensor always points directly at the chest wall of the person in the

frame for continuous monitoring of vital signs and activity levels. The selected servo for

this project is the GWS S125 3T D. With a torque to speed ratio of 0.3 second per 60

degrees at 6 volts, this servo is commonly used for a variety of robotic applications and

model sail boats. The servo is capable of 3-turn rotation and is used to realign the

monitoring platform in the direction of the patient being monitored by the Doppler radar

sensor.

44

Figure 4-11. GWS S125 3T sail winch servo. Shown along with U.S. quarter for size reference.

Most standard radio control servos have three wires, each a different color.

Usually, they are black, red, and white, or they are brown, red, and orange/yellow:

brown or black = ground (GND, battery negative terminal)

red = servo power (Vservo, battery positive terminal)

orange, yellow, white, or blue = servo control signal line

Figure 4-12. Servo wiring connected to microcontroller PWM.

45

Low-Power Communication Link

ZigBee is a low-cost, low-power, wireless mesh networking standard. First, the

low-cost allows the technology to be widely deployed in wireless control and monitoring

applications. Second, the low power-usage allows longer life with smaller batteries.

Third, the mesh networking provides high reliability and more extensive range. [29]

802.15.4 – ZigBee Protocol

ZigBee is a wireless technology developed as an open global standard to

address the unique needs of low-cost, low-power wireless mesh-to-mesh networks. The

ZigBee standard operates on the IEEE 802.15.4 physical radio specification and

operates in unlicensed bands including 2.4 GHz, 900 MHz and 868 MHz. The 802.15.4

specification upon which the ZigBee stack operates gained ratification by the Institute of

Electrical and Electronics Engineers (IEEE) in 2003. The specification is a packet-based

radio protocol intended for low-cost, battery-operated devices. The protocol allows

devices to communicate in a variety of network topologies and can have battery life

lasting several years. [29]

The ZigBee protocol is designed to communicate data through hostile RF

environments that are common in commercial and industrial applications. ZigBee

protocol features include:

Support for multiple network topologies such as point-to-point, point-to multipoint and mesh networks Low duty cycle – provides long battery life Low latency Direct Sequence Spread Spectrum (DSSS)

46

Up to 65,000 nodes per network 128-bit AES encryption for secure data connections Collision avoidance, retries and acknowledgements

A key component of the ZigBee protocol is the ability to support point-to-point

networking. In a point-to-point network, nodes are preprogrammed to connect with other

nodes. One of the key reasons for selecting ZigBee over any other protocol is the ease

in which additional sensors can be integrated into the network. However, there are

some security concerns where an unwanted outside receiver node on the same channel

as any transmitting sensor node could intercept sensor data to compromise the network

and raise issues related to data privacy and security.

ZigBee enables broad-based deployment of wireless networks with low-cost, low-

power solutions. It provides the ability to run for years on inexpensive batteries for a

host of monitoring and control applications. Smart home technologies, automatic meter

reading, lighting controls, building automation systems, tank monitoring, HVAC control,

medical devices and fleet applications are just some of the many spaces where ZigBee

technology is making significant advancements.

Digi is a member of the ZigBee Alliance and has developed a wide range of

networking solutions based on the ZigBee protocol. XBee and XBee-PRO modules and

other XBee-enabled devices provide an easy-to-implement solution that provides

functionality to connect to a wide variety of devices. XBee is the brand name from Digi

International for a family of form factor compatible radio modules. The first XBee radios

were introduced under the MaxStream brand in 2005 and were based on the 802.15.4-

47

2003 standard designed for point-to-point and point-to-multipoint communications at

over-the-air baud rates of 250kbps. [30]

X-CTU Software

The X-CTU software provided by Digi is used to configure and test ZigBee radio

modems. The application supports all XBee (formerly known as MaxStream) products,

displays the Receive Signal Strength Indicator (RSSI), provides firmware updating

capabilities, and contains a terminal window allowing for quick debug and testing of a

variety of modems. The software is easy to use and allows customers to test the radio

modems in the actual environment with just a computer and the items included with the

radio modems. [31]

Communication Architecture

The XBee modules in this project were used as follows. The XBee base module

is located at the central monitoring station (laptop or desktop) and is accessed through

a serial port in the LabVIEW program. The XBee base module is able to send and

receive data as required.

The Doppler radar sensor collects information via its radar antennas and

processes the information on its own integrated circuit board. The sensor data is

transmitted to a microcontroller that handles the analog-to-digital conversion and sends

the data to the XBee transceiver module. The module is responsible for transmitting the

digital data wirelessly to the XBee base module connected to LabVIEW.

A third XBee module in receive mode has been placed on the mobile platform

and is attached to Atmega128 microcontroller on the MAVRIC-IIB board. The three-way

communication between the Doppler radar sensor and the base station, and

48

communication between the base station and the mobile platform is accomplished as

shown in Table 4-1. The addresses used to configure the XBee modules are included.

Table 4-1. XBee communication addresses.

XBee Module Destination Address Source Address

01 Base Station 07 (to servo platform) 01 (from vital signs) 02 Radar Board 03 Servo Platform

01 (to base station) 11 (value other than 01 or 09)

09 (value other than 07 or11) 07 (from base station)

Microprocessor Development

Requirements

The proper selection of microcontrollers was essential to the success of the

design. The system is comprised of two microcontrollers on the sensor platform.

The first microcontroller required an on-board ADC with enough resolution to track the

changes of the Doppler radar sensor and capability of conditioning and processing the

data received from the sensor interface. Enough on-board memory is needed to retain

both the runtime code as well as store the data sampled by the ADC. There is also the

requirement for the microcontroller to have the ability to encode and send sensor data

to the XBee wireless transceiver via a serial output. The system would be optimized if

the XBee chip itself included an on-board ADC, eliminating the need for the first

microcontroller altogether.

The second microcontroller requires pulse-width modulation to control servos, a

USART for serial data transmission as well as open I/O ports for debugging and LCD

connectivity. There also exists the requirement for the microcontroller to be easily

reprogrammable and consume a minimal amount of power. Because the initial goal for

a truly mobile and low-power system, an emphasis must be placed on the assumption

49

that the bulk of power will be consumed in the active states of the microcontroller and

transmitters.

Capabilities

Texas Instruments‟ MSP430F1232IPW was selected for the first microcontroller.

The microcontroller was chosen because it‟s a proven favorite for embedded sensor

applications, has a large knowledge base of users and example code, and TI offers

quality assistance and sample parts. Table 4-2 provides the relevant information of the

microcontroller.

Table 4-2. Features of Texas Instruments‟ MSP430F1232IPW.

Parameter Description

Type of Program Memory Program Memory RAM I/O Pins ADC Interface Supply Voltage Range Active Mode Standby Mode # of Power Saving Modes

Flash 8 kB 256 Bytes 22 pins 10-bit SAR 1 Hardware SPI/UART, Timer UART 1.8 V – 3.6 V 200 uA @ 1 MHz, 2.2 Vsupply 0.7 uA 5

The BDMICRO MAVRIC-IIB is a powerful microcontroller board based on the

ATmega128 MCU and it was selected as the second microcontroller. The board is fully

programmable using C and is an extremely popular board for robotics enthusiasts and

the do-it-yourself electronics community. Most importantly, the MAVRIC-IIB provides the

necessary fusion of sensors and digital I/O needed for the activity-level sensing

platform.

50

The board has 2 on-board UARTs both of which are level-shifted to provide true

RS232 levels, or they may be used using the unshifted TTL levels if that is more

convenient. In addition to all that, 6 high resolution PWM outputs are available for

controlling servos directly used to rotate the sensor platform as needed based on

feedback from the computer vision algorithms. With up to 51 digital I/O pins, the

MAVRIC-IIB can handle even the most complex and demanding control tasks.

An auxiliary power input is also provided to power the servos separately from the

microcontroller electronics which protects the on-board electronics from the high current

drain and associated brown-outs that can result from powering motors and electronics

from the same power supply. The on-board standard programming headers make

development easy using the standard 10-pin JTAG header for use of the JTAGICE for

programming. The features of the MAVRIC-IIB board are numerous:

Atmel ATmega128 MCU

128K Program FLASH

4K Static RAM

4K EEPROM

Dual level shifted UARTs

6 R/C servo headers

6 PWM channels

Up to 51 digital I/O pins

I2C interface

Watchdog timer

Advanced, low drop-out voltage regulator on-board accepts 5.5-15V input

Small size at 2.2 x 3.6 inches

51

The BDMICRO MAVRIC-IIB layout and top-view picture are presented in Figures

4-13 and 4-14. The board layout is presented in Figure 4.15 including all relevant

connections from the microprocessor to the I/O pins and on-board components. An

Integrated Development Environment (IDE) for developing Atmel 8-bit AVR applications

is available for the MAVRIC-IIB board. The IDE supports all Atmel tools that support the

8-bit AVR architecture, including the JTAGICE programmer. AVR Studio 4 includes a

debugger that supports run control with source and instruction-level stepping and

breakpoints; registers, memory, and I/O views; target configuration and management;

and full programming support for standalone programmers.

Figure 4-13. BDMICRO MAVRIC-IIB microcontroller board top view.

52

Figure 4-14. BDMICRO MAVRIC-IIB microcontroller board top view.

Figure 4-15. BDMICRO MAVRIC-IIB microcontroller board layout.

53

Web Services

Web services enable the invocation of a method on a remote target using

standard Web-based protocols. A client sends a request to a remote server, which

processes the request and replies with a response, which is then interpreted and

displayed by the client application.

The following bullets are all components of a Web service:

Server – An application responsible for parsing a request, executing the appropriate method or action, and sending a response to the client. The server is the LabVIEW application running on the central monitoring station.

Client – An application that sends a request to the server and waits to receive a response, which is then interpreted by the client. The client is any web browser (mobile/desktop) connected to the Internet.

Standard protocols – Web-based protocols such as HTTP route data over physical networks from the client to the appropriate server method and then back to the client.

Network – The physical layer, such as Ethernet or IEEE 802.11, over which data is transmitted.

LabVIEW web server allows users to deploy VIs (LabVIEW‟s native design files)

as Web services, which may be invoked via a request from a client using standard

HTTP protocols. The advantage of using LabVIEW web services is that the clients you

build to communicate with a deployed VI do not require the LabVIEW run-time engine,

which means it is possible to use any Web-based client technology, mobile or desktop

operating system or browser without LabVIEW being installed on the device. In this

case, the LabVIEW interface is available on any mobile and desktop browser on the

market. The LabVIEW web services architecture and address space are shown in

Figure 4-16 and Figure 4-17, respectively.

54

Figure 4-16. LabVIEW web services architecture. [33]

A Web service VI must be configured, deployed, and managed from the Build

Specifications section of the LabVIEW Project Explorer. The localhost refers to the IP

address of the central monitoring station running LabVIEW and the user interface.

Figure 4-17. LabVIEW web services address. [33]

For example, the web services address for the LabVIEW user interface in this

project was configured as: http://192.168.0.104:8000/ActivityMonitor3.htm

The LabVIEW web server is configured to display as a continuously updating

monitor. Every one second, the client will receive an updated screenshot of the user

interface and visualization observed at the central monitoring station. The same applies

for access through a mobile device allowing caregivers to remotely view the collected

data but not manipulate the data. This functionality of the web server allows for data to

be consumed as read-only, while protecting the data from being shared or manipulated

in any way. This method of sharing via web services enhances privacy

55

Figure 4-18. Snapshot of LabVIEW web services from Google Chrome browser.

and security to protect the patient and still enables doctors, relatives and caregivers to

obtain the sensed data. A view of the client user interface from within the Google

Chrome browser is shown here in Figure 4-18.

56

CHAPTER 5 SYSTEM INTEGRATION TESTING

Vital Signs Radar Sensor Results

C. Li et al. [25] showed a comparison of the vital signs radar sensor heartbeat

results with a fingertip sensor transducer as a reference. The detected heartbeat and

the reference heartbeat were within 2% of each other, as shown in Figure 5-1. In this

case, it was deemed unnecessary to further collected vital signs sensor results using

the same radar sensor.

Figure 5-1. Vital signs radar sensor detected heartbeat comparison with fingertip transducer.

57

Activity-Level Monitoring Results

A new user interface was created to display the activity-level results computed

from the vital signs sensor data collected. On the left hand side, the I/Q signal chart is

the time domain data collected from the Doppler radar. The graph on the right displays

activity levels over time. In this case, there is no activity captured by the sensor so the

I/Q signal is a flat line (light blue) and the activity-level results are also a flat line (color

red) observed at the edge of the horizontal axis. On the left hand side, there is a vertical

colored indicator bar used to indicate the magnitude of the activity level detected. A high

activity is represented by the color green to encourage positive feeling and an active

lifestyle. Labels for medium (yellow/orange), low (orange/red) and inactive (red) states

are displayed along the indicator bar. The quadrature radar system uses two channels, I

and Q, to detect biosensor data. The resulting time domain data is converted into the

frequency domain using a fast Fourier transform, then it is filtered accordingly using a

high-pass filter (for heartbeat) and low-pass filter (for respiration). An additional filter is

used to detect if the amplitude of the square of the detected time domain data is less

than 0.1. In this case, the biosensor data is filtered as noise and the heartbeat and

respiration indicators are set to zero. Finally, a threshold of an activity level of interest

may be set by the user. A timer may also be set by the user to measure time spent

above a specified activity-level. A green light indicator will light up once the user has

maintained a set level of activity for a specified amount of time. Figures 5-2 through 5-9

shows via the National Instruments LabVIEW user interface various data results

collected using the activity-level monitoring system.

58

Figure 5-2. Activity-level user interface in state of “inactivity”.

Figure 5-3. Activity-level user interface in state of “low” activity. Physiological movement is observed from the time domain data (top left). Activity-level data over time is the red line shown (top right).

59

Figure 5-4. Activity-level user interface in state of “high” activity. Rapid physiological movement is observed from the time domain data (top left). Activity-level data over time is the red line shown (top right).

Figure 5-5. Furniture layouts within user interface. Furniture layouts are added to the UI to interactively display activity-levels around the home or in a collaborative space. Two sensors are deployed in each of the office space (left) and home environment (right).

60

Figure 5-6. Historical activity-level data collected using LabVIEW. The figure shows activity recorded between 12:12am and 12:20am. The user exhibits brief high activity twice around 12:14am. No other activity level is detected. Y-axis represents activity level in volts.

Figure 5-7. Historical activity-level data collected using LabVIEW. A closer look at Figure 5-6‟s historical activity-level data is displayed here. Note that the time axis has been modified and the data presented spans 1 hour from 12am to 1am on 05/18. Y-axis represents activity level in volts.

61

Figure 5-8. Historical activity-level data collected using LabVIEW. Closer look at Figure 5-6‟s data. Time axis has been modified and spans a 6-hour period from 12am-6am. Y-axis represents activity level in volts.

Figure 5-9. Historical activity-level data collected using LabVIEW. Data collected from 12am and 1am shown. #1 represents two burst of instantaneous activity; #2 represents burst of high activity (person walking past sensor) and returning at #3; #4 represents 5 minute period of high activity (jumping) within sensor‟s view.

#1 #2 #3 #4

62

Platform Monitoring Results

The Doppler radar sensor was integrated on the activity-level sensing platform.

The recorded video feed is shown with the user interface in Figure 5-10.

Figure 5-10. Activity-level interface with web camera overlay. No activity detected here and inactive state is shown.

Figure 5-11. OpenCV library face and error detection. Human face detected (right). Slight tilt of face leads to error detecting face (left).

63

CHAPTER 6 CONCLUSION AND FUTURE WORK

Monitoring and assistive living techniques are becoming a popular topic.

Researchers, policy makers and the medical community is beginning to pay close

attention to the issues associated with an increasing aging population, wireless patient

monitoring devices and technologies for sustainable quality of life. While there is a

wealth of research on specific aspects of vital signs monitoring systems, there are still

opportunities to advance the field of ubiquitous heath monitoring using minimally

invasive technologies without the need for user intervention. Research related to non-

contact sensing, health IT system and behavior imaging are presented.

A non-contact activity-level monitoring system using a Doppler radar sensor is

proposed. The system is able to detect a person‟s vital signs while the person is

stationary and the person‟s activity levels when the person is in motion (within sensing

range and field of sight of the sensor). The Doppler radar sensor is able to detect a

person‟s physiological movements and compute activity levels without physical contact.

Currently, the system requires patients to be stationary in order to detect their vital signs

accurately. More advanced signal processing and sensing techniques are required in

order to completely eliminate random body movement and interference possibly

enabling continuous non-contact monitoring in the future.

Another application of interest is to use the radar sensor platform as an obesity

management system. The fact that the system is sensitive to detect the slightest

movement from the user provides an opportunity to use the system as integrator of all

energy signals emitted by an individual over time. Over a period of time, the person‟s

total movements could be measured and integrated. Using a caloric estimator, caloric

64

expenditure could be assessed. High precision instruments provided by the School of

Nursing and Medicine could be used to measure actual caloric expenditure to provide a

direct validation and ground truth measurement.

Another possible future application is the development of a frequency spectrum

signature map for activity and gesture recognition. This idea is investigative and it is

unclear whether the radar sensor is capable of detecting activities based on frequency

spectrum characteristics. The idea is to use the system to determine a set of gestures

and/or activities after the system goes through a training period. During the training

period, the user is asked to perform a set of low level activities (i.e. 40 activities, each

repeated 10 times consecutively). Then, the biosensor data signals collected in the

frequency spectrum would be analyzed and isolated to create frequency domain map

for the given activities. The frequency map and database of activities could be used to

analyze an incoming bio-signal and classify and recognize activities. Activity recognition

modeling will be needed (i.e. Conditional Random Fields or Hidden Markov Models) and

the system will offer these models a set of "virtual sensors" which are the identified

(occurring) low level activities in the frequency domain map.

OpenCV computer vision algorithms were enhanced to detect a person‟s

presence and track the person as they move within the living space to continuously

detect their physiological movements. Current sensing experiments and computer vision

algorithms only focus on detecting a single human target. Detecting multiple targets and

distinguishing people within a crowd could be accomplished using more advanced

computer vision algorithms. These tasks are highly challenging but attractive next steps

for this project.

65

Other improvements within the algorithm and software component of the project

are also required. A historical moving average algorithm was used to compute the

activity levels based on measured physiological movements. Future work will focus on

designing a more robust data storage mechanism to visualize and collect data

continuously for long periods of time. Advanced machine learning and pattern

recognition techniques could be used in the future to perform activity recognition and

enable a variety of more advanced activities. Furthermore, other sensing capabilities

(thermometer, infrared, pressure, etc.) could be added to the sensing platform to

perform more sophisticated sensing tasks. Application programming interfaces and web

services could be further exploited to make all the medical data collected available (with

the person‟s permission) to third party networks, including hospitals, doctors and

researchers conducting investigations into the cause and effects of diseases such as

autism, dementia or other chronic diseases.

Finally, the thesis presented an overview of the integrated monitoring system

focus on the platform integration of the Doppler radar sensor, microcontrollers for signal

processing, wireless communication links, and other components. It was demonstrated

that Doppler radar could be combined with other technologies as a platform to record

indoor activity levels for home healthcare applications.

66

LIST OF REFERENCES

1. World Population To Increase By 2.6 Billion Over Next 45 Years [http://www.un.org/News/Press/docs/2005/pop918.doc.htm]

2. Projected Population of the United States, by Age and Sex: 2000 to 2050 [http://www.census.gov/population/www/projections/usinterimproj/natprojtab02a.pdf]

3. J.H. Oum, S.E. Lee, D.-W. Kim, S. Hong, “Non-contact heartbeat and respiration detector using capacitive sensor with Colpitts oscillator,” Electronics Letters, vol. 44, no. 2, pp. 87-88, Jan. 2008.

4. K. Watanabe, T. Watanabe, H. Watanabe, H. Ando, T. Ishikawa, K. Kobayashi, “Noninvasive measurement of heartbeat, respiration, snoring and body movements of a subject in bed via a pneumatic method," IEEE Trans. Biomed. Eng., vol. 52, no.12, pp. 2100-2107, Dec. 2005.

5. B. Song, H. Choi, H.S. Lee, “Surveillance Tracking System Using Passive Infrared Motion Sensors in Wireless Sensor Network,” in Proceedings of International Conference on Information Networking (ICOIN 2008), Jan. 2008., pp.1-5.

6. I. Korhonen, J. Parkka, M. Van Gils, “Health Monitoring in the Home of the Future”, IEEE Engineering in Medicine and Biology Magazine, pp. 66-73, May-Jun. 2003.

7. J. Ko, J.H. Lim, Y. Chen, R. Musvaloiu-E, A. Terzis, G.M. Masson, T. Gao, W. Destler, L. Selavo, R.P. Dutton, “MEDiSN: Medical emergency detection in sensor networks,” ACM Trans. Embed. Comput. Syst., vol. 10, no. 1, Aug. 2010.

8. C.R. Baker, K. Armijo, S. Belka, M. Benhabib, V. Bhargava, N. Burkhart, A. Der Minassians, G. Dervisoglu, L. Gutnik, M.B. Haick, C. Ho, M. Koplow, J. Mangold, S. Robinson, M. Rosa, M. Schwartz, C. Sims, H. Stoffregen, A. Waterbury, E.S. Leland, T. Pering, P.K. Wright, “Wireless Sensor Networks for Home Health Care,” International Conference on Advanced Information Networking and Applications Workshops (AINAW 2007), vol. 2, pp. 832-837, May 2007.

9. H. Fern ndez-L pez, J.A. Afonso, J.H Correia, R. Sim es, “HM4All: A vital signs monitoring system based in spatially distributed ZigBee networks,” Pervasive Computing Technologies for Healthcare (PervasiveHealth 2010), pp. 1-4, Mar.

2010.

10. M. Suh, L.S. Evangelista, C.A. Chen, K. Han, J. Kang, M.K. Tu, V. Chen, A. Nahapetian, M. Sarrafzadeh, “An automated vital sign monitoring system for congestive heart failure patients,” in Proceedings of the 1st ACM International Health Informatics Symposium (IHI '10), New York, NY, Nov. 2010, pp. 108-117.

67

11. A. Wood, J. Stankovic, G. Virone, L. Selavo, H. Zhimin, C. Qiuhua, D. Thao, W. Yafeng, L. Fang, R. Stoleru, “Context-aware wireless sensor networks for assisted living and residential monitoring,” IEEE Network, vol. 22, no. 4, pp. 26-33, Jul.-Aug. 2008.

12. M. Hynes, H. Wang, E. McCarrick, L. Kilmartin, “Accurate monitoring of human physical activity levels for medical diagnosis and monitoring using off-the-shelf cellular handsets”, Personal and Ubiquitous Computing, pp. 1-12, Dec. 2010.

13. Autism Presentation at Georgia Tech [http://autism.cc.gatech.edu/presentations.htm]

14. Computational Behavioral Science Laboratory [http://www.cbs.gatech.edu/]

15. B.P. Lo, J. L. Wang, and G.-Z. Yang, “From imaging networks to behavior profiling: Ubiquitous sensing for managed homecare of the elderly,” in Adjunct Proc. of 3rd Int. Conf. on Pervasive Computing, May 2005.

16. B. Brumitt, M. Brian, J. Krumm, A. Kern, S. Shafer, “Easy Living: Technologies for Intelligent Environments,” Handheld and Ubiquitous Computing, Sep. 2000.

17. H. Nait-Charif, S. McKenna, “Activity Summarisation and Fall Detection in a Supportive Home Environment,” Intl. Conf. on Pattern Recognition, vol. 4, pp. 323-326, Aug. 2004.

18. J. Gao, H. Wactlar, A. Hauptmann, “Combining Motion Segmentation with Tracking for Activity Analysis,” Intl. Conf. Automatic Face and Gesture Recognition, pp. 699-704, May 2004.

19. J. C. Lin, "Noninvasive Microwave Measurement of Respiration," in Proceedings of the IEEE, Oct. 1975, vol. 63, no. 10, pp. 1530.

20. J. C. Lin, “Microwave sensing of physiological movement and volume change: A review,” Bioelectromagnetics, vol. 13, pp. 557–565, 1992.

21. K. M. Chen, Y. Huang, J. Zhang, A. Norman, “Microwave life-detection systems for searching human subjects under earthquake rubble and behind barrier,” IEEE Trans. Biomed. Eng., vol. 47, no. 1, pp. 105–114, Jan. 2000.

22. A.D. Droitcour, O. Boric-Lubecke, V.M. Lubecke, J. Lin, G.T.A. Kovac, "Range correlation and I/Q performance benefits in single-chip silicon Doppler radars for noncontact cardiopulmonary monitoring", IEEE Trans. Microwave Theory and Techniques, vol. 52, pp. 838-848, Mar. 2004.

23. C. Li, J. Lin, “Complex signal demodulation and random body movement cancellation techniques for non-contact vital sign detection,” in IEEE MTT-S Int. Microw. Symp. Dig., Atlanta, GA, Jun. 2008, pp. 567–570.

68

24. C. Li, J. Cummings, J. Lam, E. Graves, W. Wu, “Radar remote monitoring of vital signs,” IEEE Microw. Mag., vol. 10, no. 1, pp. 47–56, Feb. 2009.

25. Wikipedia - Moving Average [http://en.wikipedia.org/wiki/Moving_average]

26. LabVIEW Software [http://www.ni.com/labview/whatis/]

27. Haar-like features [http://en.wikipedia.org/wiki/Haar-like_features]

28. Datasheet: GWS S125 3T Sail Winch Servo [http://www.hobbydream.com/gws-s125-3t-digital-sail-winch-s125-3t-d_p11891.html]

29. ZigBee Protocol [http://www.digi.com/technology/rf-articles/wireless-zigbee.jsp]

30. X-CTU Software [http://www.digi.com/support/kbase/kbaseresultdetl.jsp?kb=125]

31. XBee [http://en.wikipedia.org/wiki/XBee]

32. Datasheet: ATmega128 microcontroller datasheet [http://www.atmel.com/dyn/resources/prod_documents/2467S.pdf]

33. Web Services in LabVIEW [http://zone.ni.com/devzone/cda/tut/p/id/7350]

34. Datasheet: BDMICRO MAVRIC-IIB Microcontroller Board [http://www.bdmicro.com/mavric-iib/mavric-iib.pdf]

69

BIOGRAPHICAL SKETCH

Gabriel Reyes received his Bachelor of Science degree in electrical engineering

from the University of Florida in 2008. He continued his studies at the University of

Florida to graduate in 2011 with double Master of Science and Master of Arts degrees in

electrical engineering and international business, respectively. His research interests

include ubiquitous computing, radio frequency and wireless sensing applications,

human-computer interaction, cyber-physical and embedded systems.