1. HYPOTHESIS and OVERVIEW - Rochester Institute of...
Transcript of 1. HYPOTHESIS and OVERVIEW - Rochester Institute of...
1
1. HYPOTHESIS and OVERVIEW
We hypothesize that an environmentally-aware, network connected, robotic assistive device will be able
to prevent a number of problem events (particularly falls) and reduce mean response times to several
classes of key health problems by way of risk assessment and unobtrusive monitoring. This will
significantly reduce costs and
lighten the burden on existing
health care resources. Figure 1
depicts the overall proposed
system which will incorporate a
robotic smart walker, an in-
house base station, and a remote
analytics server. The walker will
provide environment-aware
mobility assistance as well as
recording sensor data about the
user and home activities. This
data will be passed wirelessly to
the base station, which will
compress and transmit data over
a wired Internet connection for
analysis, visualization, and secure
storage at the remote server, as well as storing navigational knowledge for later use, by the walker.
In order to ensure maximum accuracy in fall detection, health condition prognosis, and
environment awareness, the sensors involved will include at least visible and infrared light imaging,
omnidirectional and pan/tilt directional audio, accelerometers, and pulse/heartbeat sensors. Research
performed will include cutting-edge feature preprocessing and adaptation, navigation in dynamic
cluttered environments, multimodal signal compression, and statistical models of day-to-day health.
These will support generation of symptomatic alerts of both long-term problems (tremor progression,
changes in respiration or sweating, altered voiding regimen) and immediate problems (falls, distress).
Patient health histories will be stored in compliance with emerging medical standards.
2. OBJECTIVES FOR PERIOD OF PROPOSED WORK
Listed here are the objectives for the proposed project, with additional description in Sections 3 and 4:
Research, design, and test a novel unobtrusive smart walker and a base station
Research, implement, and test navigation algorithms for the smart walker based upon
unobtrusiveness criteria
Research, implement, and test novel algorithms for smart walker environmental awareness
Research, implement, and test biosignal modeling and person-to-person adaptation
Research, implement, and test novel algorithms for biosignal processing
Research, implement, and test novel post-fall detection algorithms based upon audiovisual event
detection and visual pose estimation
Research, implement, and test novel medicine dosing event detection algorithms based upon
audiovisual object recognition and visual pose estimation
Research, implement, and test novel boosted features based upon SLAM and visual context
Research, implement, and test novel long- /short-term biosignal alerts based upon sensor history
Develop an implementable architecture that lays out the major alternatives for partitioning of
work among processing nodes and associated information exchange among nodes
Figure1. Proposed system (smart walker, base station, server)
2
Develop implementable criteria for processing of sensed data for local handling versus
forwarding downstream to meet effectiveness and efficiency objectives of the overall architecture
Develop implementable approach to metadata characterization of patient data compatible with
existing health information technology standards
3. INTELLECTUAL MERIT
Research performed will include cutting-edge feature preprocessing and adaptation, navigation in
dynamic cluttered environments, activity recognition, fall risk assessment based upon pose, audiovisual
fall detection, statistical models of day-to-day health, symptomatic alerts of long-term (medicine
noncompliance) and immediate problems (falls), and health data management. Results in all areas are
expected to build upon the state of the art and improve the fields of robotics and health monitoring in the
context of service-oriented architectures for ambient assisted living.
4. LITERATURE REVIEW AND RESEARCH CHALLENGES
As this proposal is a collaborative proposal, there are four main components (teams) of the proposal:
Geriatric Need, Unobtrusive Smart Walker Design, Event Understanding and Alert Generation, and
Service Oriented Architecture for Health IT.
4.1 Geriatric Need (Rochester General Hospital System – RGHS)
The population of the United States is an aging population: the US Census Bureau estimates that the
population of people age 65 years or older will increase from 39.6 million (12.9%) in 2009 to 72.1 million
(19.3%) by 2030 [1]. With this shift in demographics comes an increased demand for health care resources,
with the availability of these resources recognized as a key indicator of well-being [2]. Clinical studies
shows that 33% of U.S. adults aged 65 of greater suffer falls in a given year [3, 4], with almost one-third
requiring medical attention and further falls occurring for close to 60% within a year. Direct and indirect
costs of falls are on the order of billions annually (e.g., $19 billion in 2000 [5]), some or all of which can be
prevented through risk factor minimization and immediate responder notification [6, 7, 8]. Likewise,
nonadherence to prescribed medical regimens is estimated to be as high as 38% [9], in spite of existing
methods of determining capacity to manage medications [10], resulting in reduced effectiveness and
potential for relapse. Our proposal is aimed at minimizing high-cost independent living health risks (such
as falls and medication nonadherence) for the elderly through robotics.
As it stands, a large proportion of the elderly population makes use of assistive technology (26% of
the chronically disabled without personal care, 58% with some personal care [11]). Mobility devices are
by far the most common, surpassing all anatomical, hearing, and vision devices. Therefore, we propose
the research and development of a robotic system that duplicates the benefits of a mobility device
(exercise, stability) while unobtrusively identifying risks and maximizing information for medical staff.
This research will advance the state of the art across several fields while meeting a key societal need.
4.2 Unobtrusive Smart Walker Design (Electrical and Microelectronic Engineering, RIT)
4.2.1 Assistive Robotics
Assistive devices have remained crucial throughout the advancement of human kind. Canes, eye glasses,
hearing aid, and work tools have been instrumental in technological and intellectual development.
Recently, intelligent robotic systems are being used for helping the elderly, disabled, blind, and injured.
Research has been done on aiding the blind through other senses than vision via a mobility aid called
a Smart Blind Cane that uses a smart phone based collision warning system (built around sensors like
ultrasonic range and temperature) and text -to- speech to notify the user of their surroundings [37]. Other
related devices include a thermal digital device that lets visually impaired individuals visualize works of
3
art by touch, glasses that measure time between blinks, and a device that uses sensors to smell [39]. For
elderly visually impaired people that have difficulty using guide dogs or long canes, walkers or walking
frame with wheels are designed to help them navigate; one such device is called PAM-AID [38].
In general, there have been many devices and robots researched for walking assistance and helping
the elderly in nursing homes and hospitals, with wheelchairs and walkers the most commonly used.
Researchers have tried to augment these walking aids with robots and sensors to make them smart and
autonomous. Exoskeleton devices [42] and robotic suits [55] [56] [57] have also been used as mobility and
rehabilitation aids. In one of these experiments, an anthropomorphic approach was used by mounting a
robotic Manus® arm [41] on a wheelchair and letting the user control the arm through a user friendly
interface [40]. Along with walking, assistance is also needed in standing and sitting. Devices have been
prototyped that help with all three operations to prevent injuries like fractures and dislocation of bones
while trying to stand or sit [43] [45]. Intelligent wheelchairs with obstacle avoidance and independent
control can be used to carry the elderly and/or disabled or move independently in a smart environment
as a sensorial extension, and have been implemented on at least one occasion [44]. Another such
implementation is a call-to-service walking helper robot that comes and provides assistance by locating
the user in a smart environment [35]. But, in many cases, the user desires assistance that respects their
freedom and control within their ability, yet also within an envelope of safety [49, 50]. For this reason,
semi-autonomous mobility aids have been designed that take into account intent [46, 47] and also
interactive [48]. Figure 2 shows some of these devices.
Figure 2. Examples of assistive walking and posture devices, left to right [35, 26, 49, 50]
Further examples explicitly requiring patient effort include exoskeletons supporting leg alignment
[42], full robotic suits [54, 55], robotic arms and exoskeletons for rehabilitation of upper limbs with weak
muscles [58] and patients post-stroke [52], orthotic ankle devices [53] and pelvic assistance [59] for gait
disorders, and cable-driven machines for rehabilitation [56]. Also developed is an exoskeleton for eating
that simulates upper limb motor patterns in a healthy person and can be controlled by wrist action [51].
Many such-assisted elderly/disabled live in nursing homes or in assisted living facilities with
constant assistance and monitoring. Moreover, caregivers need help in managing assistive devices. For
this reason, there has been work in robotics to push and steer wheelchairs on a fixed path [60] or come
when called [35], and dual-arm robots to help dress patients [64]. Models have been suggested to make
homes and/or hospitals context-aware environments sensitive and responsive to the presence of humans
using embedded systems, sensor networks, and robots [62]; based on feedback from health care
organizations, effort is being made on improving the behavior, appearance, and functionalities of such
systems [59]. Other related efforts include mapping fleets of helper robots, camera-based patient tracking
[61] and audiovisual human-robot interaction [65]. Studies on the social and psychological effects of
robotic-assisted activities on elderly people [66, 67] show improved moods and less depression. Research
on socially-acceptable robots [68] and graphical user interfaces for the elderly are underway [69]. This
extends to wearable life sign sensor vests for prognosis [73], shoes with acceleration sensors to detect fall
4
and gait disabilities [74], and floors with sensors to determine positioning and falls [63]. Children with
disabilities and suffering from autism or cerebral palsy have also benefited from robots and robotic toys
that help them learn and interact [70, 71, 72].
Research Challenge
There have been several efforts in developing smart assistive mobility devices for the elderly. However,
these efforts have been limited to one or two aspects of care and are obtrusive. In aiding the elderly,
special care needs to be taken in designing socially-acceptable devices which are minimally obtrusive.
Furthermore, most prototypes have not been networked, limiting the use and reporting. In an effort to
assist elderly without disrupting their daily lives, the assistive device needs to be designed so as to be
unobtrusive, semi-autonomous, and networked. Thus, in this proposed work we plan to design a walker
which is functionally similar to a manual walker but also networked to collect, monitor, and report life
signs.
4.2.2 Mobile Robot Navigation
In mobile robotics, navigation is one of the most fundamental and challenging tasks in robot autonomy;
hence, a lot of research has been done to address challenges such as positioning and orientation,
analyzing surroundings (mapping), conveying this information to the robot, and choosing a subsequent
appropriate action or path to take. Navigation is twofold: local and global. Local navigation, often called
reactive control, is how the robot learns or plans local paths using the sensory inputs without prior
complete knowledge of the environment. Global navigation, also called deliberate control, is where the
robot learns or plans the global paths based upon complete knowledge about the environment [26].
Additionally, challenges in navigating indoors and outdoors are addressed differently. An indoor
environment can be made smart by using a network of sensors or by placing identifying landmarks.
However, in an outdoor environment, all the sensors generally have to be on the robot and hence the
robot has to be independent of the environment to navigate.
Sensors generally used include RFID (Radio Frequency Identification) tags, sonar or ultra-sonic range
sensors, cameras, magnetic sensors and laser or infrared scanners. Vision-based navigation is the most
common and many research efforts use image processing with range sensing (odometry). For example, in
an unknown indoor environment, local navigation is achieved either through monocular vision-based
simultaneous localization and mapping (SLAM) [16] [17] by mounting a camera on the robot facing
upwards and extracting features like lamps, doors and corners and laser scans for odometry [12] [13]; or
by using stereo vision and extracting 3D features from static and dynamic obstacles [14] [15]. For
navigation in unknown environments without vision, networks of RFID tags or passive RF sensors [18]
[19], magnetic landmarks [22] and ultrasonic sensors [23] have been used. A sensor network can also be
used for SLAM by using triangulation localization methods [20] [21], including coupling with limited
sensing [24]. SLAM variants include complete coverage navigation (CCN) where the robot passes
through all points in a changing environment [25], hierarchical Q- learning for coupled global and local
navigation for very accurate tasks [26]; predictive SLAM (P-SLAM) for exploration [27]; uncertainty-
based algorithms [28]; algorithm for probabilistic localization using indoor GPS and laser scanners [29],
topological model [31], or occupancy maps [30], and many more. Further research includes algorithms for
multiple robots such as reconnaissance and surveillance for a large urban environment [33] and adaptive
road mapping [32].
Research Challenge
The main challenge in navigation of the Smart Walker is to move around with precision with passive
motion where the patients holding/pushing the walker. In addition, environmental awareness is very
critical for the walker to make decision on active versus passive motion. Some of the applications of
5
mobile robot navigation are call-to-service mobile robots for elderly patients (Walbot), that use Zigbee
sensor networks for localization and tracking the caller [35], service robots for transportation in hospitals
[34] and an autonomous vehicle for indoor surveillance [36].
4.2.3 Biosignal Processing and Classification
Analyzing and understanding human biosignals has been an important research area with practical
applications in everyday life. Brain computer interfacing (BCI) is a research area that studies controlling
devices by processing and understanding the brain via Electroencephalogram (EEG) signals [75 – 78].
Similarly, assistive robots [79] are being developed using eye [80] or muscle signals [81 – 87] to provide
control inputs. The efficiency for all of these applications depends on being able to process and classify
biosignals. Generally speaking, the most frequently used biosignals are Electroencephalogram (EEG),
Electroocculogram (EOG), and Electromyogram (EMG). Recorded EEG/EMG/EOG signals can be used to
control inputs for real-time systems such as BCI-based assistive devices or detection systems for brain
abnormalities [76]. Therefore, most of the applications that utilize biosignals require careful consideration
in terms of collection, processing and information extraction [79, 80, 85 – 95]. Biosignal applications
include human computer interfaces [75, 76, 96], emotional state classification [88], mobility device control
[97], prosthetic arm control [81, 82, 86], and detecting driver drowsiness [92].
In preprocessing, filtering is commonplace, coupled with feature extraction methodologies including
Principle Component Analysis (PCA) [77, 78, 93], Analysis of Variance (ANOVA) [77], Linear
Discriminant Analysis (LDA) [86, 93], Independent Component Analysis (IDA) [84, 93], Waveform
Length Transforms [85, 89, 90, 92], and clustering [87, 94, 95]. Other techniques appearing are Mean
Absolute Value, Root Mean Square, Variance, Standard Deviation, Zero Crossing, Slope Sign Change,
and Singular Value Decomposition (SVD) [84, 86]. Common machine learning methods are Support
Vector Machines (SVM) [77, 79, 83, 84, 86, 90, 93], Radial Basis Function Networks (RBFN) [79, 87, 94, 95],
Artificial Neural Networks (ANN) [81, 82, 85, 89, 90, 91, 97], C4.5 [78], Fuzzy Logic [81, 82, 87, 92, 94, 95],
Genetic Algorithms (GA) [81, 82], Particle Swarm Optimization (PSO) [87, 94, 95], K-Nearest Neighbor
Classifier [88], Random Forest Classifier [88], Decision Trees [89], and the Gaussian Classifier [96].
Research Challenge
Based on the literature, success rates are nearing the 80-95% range [87, 94 – 96] for EEG, 90-97% for EMG
[83], and above 95% for hybrids [92, 96]. While not directly applicable to this task, successes by PI Ferat
Sahin and his students inspire the use of biosignals for vital sign tracking [87, 94, 95].
4.3 Event Understanding and Alert Generation (Sustainability Institute at RIT)
4.3.1 Fall Detection Algorithms
Visual fall detection is a well-established field of study, yet maintains poor detection in clutter and
insufficient false alarm rates [98]. Current state-of-the art is based around estimating human pose, using
the positioning of body parts to tell if a fall is occurring or has occurred. While the data used may vary
from traditional monocular, visible-light imaging (including infrared [99] and 360° imaging [100]), the
procedure remains largely the same: identify features of human body parts, fit a parameterized model of
human structure to these feature positions, and perform classification based upon the resulting model
parameters, often making use of context cues to weigh the final decision [101 – 103].
Of note is the division between research that identifies fall events as they occur and research that
identifies humans in a post-fall position. While the former [104, 105] suggest a shorter response time, low
false alarm rate requirements have pushed a significant portion of the research community towards
detecting immobilization following a fall [106 – 108], as it is possible to build effective confidence metrics
based upon long-term observation, rather than relying upon a limited window of activity.
6
Research Challenge
For the purposes of this project, we will focus on post-fall detection, both for reduction of false alarm
rates and to allow for investigation of suspected falls that may be outside of the robot’s field of vision.
This last point emphasizes the utility of the proposed system as well as one of our directions of
research: active, multimodal investigation. Much of the current state-of-the-art relies on static, large
baseline deployment for 3D pose estimation, and only makes use of visible light. While there is a push
for the inclusion of audio in sensor networks [146, 147], it has been used only to segment event data [148]
and requires static sensor placement [149]. Instead, we are interested in mobile investigation of potential
fall events, as we can obtain a variety of viewpoints to confirm/reject suspected fall events that occur
within or outside of the robot’s field of vision. Investigation is via floor vibration through accelerometer
readings, coupled with an omnidirectional microphone and stereo video (multiple wavelengths, visible
and otherwise), identifying suspicious occurrences whereupon all data available will be studied to obtain
a more complete picture of the room context, human pose (prone, standing, etc.), and signs of panic. The
research challenges posed, therefore, are: identification of suspicious events (visible and out-of-view),
visual/auditory/vibration/biosignal context, and multi-sensor fusion in a supervised fall/no fall classifier.
In total, work on fall detection algorithm development for our system is expected to take 1.5 to 2
years, and serve as the research cornerstone for a doctoral candidate in computer vision and machine
learning at the Rochester Institute of Technology, with the practical aspects of said research thoroughly
explored by the final year of human subject experimentation.
4.3.2 Activity Recognition Algorithms
Noncompliance with prescription medication is a serious problem for independent-living seniors, and is
an indicator of declining ability to take care of oneself. In order to determine compliance with medication,
we propose the use of activity recognition through video. By identifying when patients do not take
medication on schedule, we can generate noncompliance alerts for medical staff.
Eating/drinking/dosing events, not often the focus of human activity recognition [109], still benefit
from general video-based recognition improvements [110]. For instance, high-res., real-world training
data [111] has allowed data-driven semantic clustering and labeling of repeated activities [112], replacing
traditional parametric algorithms. We will build upon this, emphasizing context and sensor fusion.
Top activity recognition techniques today make use of spatio-temporal context to distinguish
between activities with similar subcomponents [113, 114] and discount activities that are unlikely in a
location or in conjunction with particular objects [115, 116]. By training models of context from a variety
of recorded video, it is possible to automatically learn behavior patterns across days and subjects [117].
Research Challenge
In the proposed independent living domain, we intend to focus on the isolation of prescription dosing
from normal eating and drinking by way of novel context models, particularly the repetition of normal
eating and the spatial/object context of drinking.
In addition to traditional visual feature-based context discernment of dosing, we intend to make use
of additional modalities of sensor data, particularly audio. Intuitively, bottles of pills make sounds that
are distinct from foodstuffs and beverages, allowing us to identify them [150]. The proposed work will
expand upon current techniques in audio context modeling, emphasizing the variety in whole-house
independent living, the presence of other humans and animals, and robot positioning during recording.
Dosing detection algorithms are expected to take 1 to 1.5 years of research and development, serving
as a key focus for a doctoral candidate in audiovisual signal processing at the Rochester Institute of
Technology which, coupled with research work in biosignal anomaly detection and bolstered by
extensive human subject experimentation in year four, will serve as the foundation for their thesis.
7
4.3.3 Vision Preprocessing
In order to maximize the effectiveness of activity recognition and fall detection algorithms, it is necessary
to perform preprocessing of input video data to identify the location of the robot relative to the rest of the
world and to point out recognizable objects. Each of these tasks is well-studied in the literature, and
software solutions could be quickly bootstrapped from existing open-source components.
Beyond odometry (covered in 4.2.2), we are interested in pure vision-based SLAM [118, 119].
Solutions run the gamut of waypoint identifiers and reasoners [120, 121], but the most popular approach
is FastSLAM [122 – 125], which finds the posterior likelihood of an estimate by factorization into
conditional likelihoods for each landmark and path continuity. In particular, earlier research on SLAM in
homes [126 – 128] has led to FastSLAM with added tracking of moving objects for people detection [129].
We will build upon this, adding meta-reasoning that integrates feature extraction for object detection.
Recent advances in feature detection for learnable tasks (such as the detection of particular, pre-
known objects) have hinged upon the immense success and popularity of Adaptive Boosting [130] in
constructing highly-descriptive, complex image features from many simple features [131 – 137].
Research Challenge
Our work will make use of boosted features learned experimentally, pushed further: since we will be
performing SLAM, we can use spatial maps to associate features from multiple frames/viewpoints with
each object, feeding back to improve feature selection across viewings. Research will focus on this
process, especially learning rate and exact feedback mechanisms in the presence of real-world data.
This combined system will be assembled and trained in 6 months to 1 year for use in fall detection
and activity recognition. Doctoral candidates working on those components would collaborate on this
effort, with the publication of results serving as an introduction to the publication and review process.
4.3.4 Vital Sign Anomaly Detection Algorithms
Standard medical practice holds with four key vital signs: body temperature, blood pressure, pulse rate,
and respiratory rate; these four signals are indicative of health and reveal symptoms of health problems.
As the proposed system is intended to inform clinical staff, vital signs will be recorded (when available)
and logged. Temperature, for example, can be captured via infrared thermography [138 – 141], with high
values indicating fever [142 – 145]. Beyond recording of biosignal, it is possible to detect anomalies that
will alert attending doctors to any sudden or long-term extremes or changes in vital signs.
Research Challenge
The proposed system will include the detection of short-term (sudden) changes in vital signs as well as
long-term (gradual) changes indicative of declining health.
For sudden changes, we will build algorithms around signal anomaly detection [151], particularly
feed-forward neural networks [152]. Popular in network intrusion detection [153 – 157], neural networks
have yet to be extensively studied for use in medical anomalies, despite health models serving to
motivate network research [158, 159]. Generally, neural network approaches are based upon the
deviation of inputs from past observations and training data, such as the residuals of a replicator neural
network [160, 161].
We propose the study of network topologies for vital sign signals (number and size of neuron layers,
transfer function design, connectivity), as well as the use of a single network on all modalities
simultaneously. Of particular note is that success in this domain would also allow for improvements in
automatic patient alerts other elderly care settings, including hospital, nursing home, and hospice care.
For longer-term changes, such as gradual reduction in pulse or breathing rate, we intend to create
algorithms that model history to predict future readings. This research will span avenues of study,
8
particularly sequential Monte Carlo simulation [162 – 167]. Like other alerts, we are interested in a single
model across signal types; in this case, most novel work will be in kernel design and selection, focusing
on the automatic learning of kernel combinations [168, 169] and patient-by-patient adaptation thereof.
Research and development in biosignal anomaly detection is expected to take place for six to nine
months, as a joint effort of the two doctoral candidates earmarked for fall detection and activity
recognition (as mentioned earlier). This work is intended to serve as an expansion of research interests,
with publication increasing awareness of studies in sensor reliability in the presence of noisy data.
4.4 Service Oriented Architecture for Health IT (RTSync Inc. of Arizona)
Our overall proposal can be viewed as providing Ambient Assisted Living (AAL), active telehealth for
elderly based upon point-of-care health management within a well-defined service oriented architecture.
4.4.1 Architectural Framework
AAL encompasses technical systems to support elderly people to allow an independent and safe lifestyle
for as long as possible [170]. These systems make use of inhomogeneous technologies such as wireless
networks, machine learning algorithms and sensory devices. Kurschl et al. [171] advocate Model Driven
Architectures [172] that support both expert-based top-down representation of the envisioned system as
well as support developers in bottom-up synthesis [173]. Neßelrath et al. [174] present a related vision
based on open standards and architectures in a smart home.
Research Challenge
Although the Model-Driven Architecture approach advocated by Kurschl et al. [171] is a good starting
point, it does not provide tools for simulated model representation of architecture alternatives with the
ability to test prior to implementation. We will employ proprietary RTSync DEVS-based modeling and
simulation environment [206] to develop, test, and automatically generate our AAL code.
Network Layer: The Internet of Things expands today's Internet to connect smart objects capable of
identifying, locating, sensing, and communicating in support of AAL [175]. Martin et al [176] propose the
combination of ZigBee technology with a service oriented architecture. Pal et al. [177] advocate using
compressed sensing [178] for data processing in low-power networks in the presence of noise.
Services Layer: Service-Oriented Architecture (SOA) is a concept for integration/implementation of
networked systems [179] which separates functions into independent, reusable services that flexibly
interoperate together [180 – 183]. SOA is starting to infiltrate into health care and clinical decision
support [184-186]. Ongenae et al. show how a combination of expert systems and SOA is able to support
physicians in continuing patient care [187].
Data and Modeling Layer
Personal Models: Modeling of the assisted person’s particular characteristics is essential for monitoring
and adaptive intervention [188]. One challenge of AAL is recognition of the assisted person's routine
[189]. Hossain et al. [190] develop an approach to modeling the assisted person's current activity for
detection of changes of capabilities and preferences, including user’s preferred time, location, and level of
effort. Based on World Health Organization’s framework for an individual person or population [191,
192], patient capabilities focus on impairments of body functions, body structures and limitations on
certain activities. Frey et al. model medication calendar tasks for patients with cognitive disabilities [193].
9
4.4.2 Electronic Health Records
Beyond activity models, individualization has been sparse [194], a situation likely to change with
widespread use of Electronic Health Records (EHR) and the push for “meaningful use” of Health
Information Technology [195]. Currently, the most popular HIT standard for EHRs is HL7 v2 [196].
Unfortunately, HL7 v2 messages have no explicit information model and poor definitions for many
fields. To comply with meaningful use criteria, applications are expected to implement all interactions of
the role they conform to. Standards expertise is needed to keep abreast of the moving target for
meaningful use and to enable AAL to comply with emerging EHR standards.
Research Challenge
Given evolution in federal standards for medical records, our data archiving must meet current standards
while still retaining flexibility and fidelity for future analytics. We will use the data engineering approach
of [203, 204, 205] to provide metadata characterization and representation of data for archiving that meet
emerging HIT standards. This ontology-based approach will also support investigation of data handling
principles and implementation in networked environments mediated by our simulation environment.
4.4.3 Multi-sensor information fusion and data engineering
Multi-sensor data fusion, including Kalman filtering, Bayes estimation, Dempster-Shafer evidential
reasoning, fuzzy logic and neural networks, is often used for modeling in robot control [197]. The most
common medical application is diagnostics, albeit with little fusion at levels higher than raw signal, such
as features or classifiers [198]. Use in AAL is in its infancy and offers a chance to apply mature concepts
from other domains [199]. Blach [200] extended military data fusion [201] to account for consumer needs
for processing. Kokar introduced web ontology methodology to support situation awareness using
fusion [202]. Lee [203] applied the information exchange framework based on the System Entity
Structure (SES) ontology [204, 205, 207] to extract / fuse data for commanders with different requirements.
Research Challenge
Challenges arise in the proposed system due to the nature of the walker-generated data: its composition
(motion/vision/audio/bio signal), dynamic, and intermittent nature. Two principals will be investigated:
Early characterization of data quality/utility. For example, data that will not have value to
downstream analytic processing should be identified at the base station and discarded
Early processing of data to generate alerts. For example, using individual patient models, the base
station may be able to off-load work from the analytic server, saving communication bandwidth.
Detailed investigation of these principles in the proposed AAL system will feed back to the architectural
decomposition of work and assignment of processing/communication roles to system nodes.
4.4.4 Targeted information flow with selection among data transmission/distribution technologies
The proposed AAL architecture will focus on the following components:
Data products: walker-generated (motion/vision/audio/bio) data, walker-generated alerts (shouts),
data retained for analysis, base station-generated alerts (phone call, text, email), server-generated
alerts (analytic content displayed on console screens), archived data/metadata, patient health records
Consumers of data products: assisted person, emergency responders, clinical/medical staff
Platforms or processing nodes that must be considered in the architecture: walker, base station, analytic
server, fixed computer workstations, mobile hand held devices (PDAs, smart phones, etc.)
Data transmission media include wireless, internet, phone lines, social media (e.g., Twitter)
A new design will be developed which delivers data products to the consumers over media to meet the
challenges spelled out earlier. Our design will follow the data engineering SES ontology framework via
10
our modeling and simulation environment. We will exploit the ability of SES to represent complex data
rigorously while affording flexibility to transform the structure according to a set of operations; for
example, pruning structure that is not relevant to interests of a consumer based on a specified “pragmatic
frame”. We will also use the implementation of the SES as an XML schema and its generated data as
XML documents to implement on XML-compliant data exchange/processing platforms.
The heart of the design will be a master SES that characterizes data of interest to all consumers and
operations that can be used to tailor the data forwarded to each consumer thereby reducing bandwidth
and downstream processing. Thus, the walker is the primary producer while the base station and the
server are secondary producers that turn processed data into information. The information requirements
of such producers and consumers will be expressed as pragmatic frames able to appropriately tailor the
master SES to meet such requirements. Based on this model, we will investigate information technologies,
to build upon the SOA Web Server-Client, Data-distribution Service/Publish/Subscribe technologies in
the literature for implementing AAL systems.
4.4.5 Acceptability categorization of sensed data for downstream transmission
As a primary consumer of walker-generated data, the server should receive only data that support its
diagnostics. Thus, the pragmatic frame for the server should distinguish actionable and non-actionable
data, transmitting only the former. Criteria for non-actionable data include absence of people, no signal
change, and signal corruption. We will study the fusion of data to implement these criteria. To detect
absence of people, we will use portions of infrared images containing "people" heat signatures so that
thresholds can be developed for human occupancy. For constancy of signals, we will adapt value
quantum methods [208], generalized to accommodate the multi-source walker data and account for noise
using rapid change detection [209]. To handle signal corruption, we will develop models of good and
bad data, parameterized from design and improved experimentally. Having discarded non-actionable
data, the base station will take the data, package it in terms of the standardized XML representation
derived from the pragmatic frame formulation, compress it, and transmit it through an Internet
connection to the server. We will investigate compression methods such as [177] and their interaction
with quantization.
4.4.6 Data-driven modeling, model-based prioritized generation of alerts.
Having established a baseline implementation, we will investigate alternative assignments of processing.
One possibility is the detection of a subset of short-term biosignal anomalies at the base station and
generation of alerts to emergency personnel; such anomaly detection will be based on individualized
models. We will investigate activity capability and preference models [190] for suitability to support this
change. We will also develop new methods for deriving normal, individual biosignal limits. Detection of
values outside these ranges will generate the mentioned alerts intended for emergency responders.
5. STATEMENT of WORK and TIMELINE
Phase I (year 1) of a Four Year Program
1. Purchase and set up analytics server hardware, write data storage software and test with junk data.
2. Graduate students study state-of-the-art in Simultaneous Localization and Mapping (SLAM) and
visual feature detection and state-of-the-art in assistive robotics.
3. Design “smart walker” robotic system and base station, including programmability, power use and
storage, sensor selection and placement, mobility, and communication.
4. Design data representation for storage and transmission.
5. Purchase hardware components for “smart walker” and base station.
6. Construct initial “smart walker” and base station hardware prototypes.
11
7. Implement existing SLAM and feature detection algorithms for server and test with junk data.
8. Start writing SLAM / feature detection paper for scholarly publication.
9. Develop detailed Phase II plan
Phase II (year 2) of a Four Year Program
1. Gather sample sensor data using sensors detached from smart walker.
2. Design, implement, and test novel SLAM and feature detection algorithms, experiment with sample
sensor data, and correct algorithms accordingly.
3. Finish writing SLAM / feature detection paper.
4. Implement existing data triage and compression algorithms for base station.
5. Graduate students study state-of-the-art in mobile robot navigation and environment awareness, fall
event detection and pose estimation.
6. Implement existing robot navigation and environment awareness algorithms for “smart walker”, fall
event detection and pose estimation algorithms for server, and test with sample sensor data.
7. Design, implement, and test novel robot navigation, environment awareness, fall event detection and
pose estimation algorithms, experiment with sample sensor data, and correct algorithms accordingly.
8. Write robot navigation paper.
9. Start writing environment awareness, fall event detection / pose estimation (fall detection) papers.
10. Develop detailed Phase III plan.
Phase III (year 3) of a Four Year Program
1. Finish writing environment awareness and fall detection papers.
2. Refine “smart walker” and base station hardware prototypes.
3. Test and refine data compression, transmission, and representation based upon algorithm use.
4. Graduate students study state-of-the-art in short-term biosignal modeling, person-to-person biosignal
adaptation, risk analysis based upon pose and obstacle detection, long-term biosignal modeling and
specific object and activity recognition.
5. Implement existing short-term biosignal and risk analysis algorithms for “smart walker”, long-term
biosignal, object and activity recognition algorithms for server, and test with sample sensor data.
6. Design, implement, and test novel short-/long-term biosignal, risk analysis, object and activity
recognition algorithms, experiment with sample sensor data, and correct algorithms accordingly.
7. Write short-term biosignal (emergency detection) paper.
8. Start writing object and activity recognition (medicine noncompliance) paper.
9. Develop human subject experiment plan, including specific activity script.
10. Run experiment plan with graduate students and refine accordingly.
11. Develop detailed Phase IV plan.
Phase IV (year 4) of a Four Year Program
1. Write long-term biosignal (health degradation) paper.
2. Confirm all software complete and operational on graduate student experimental data.
3. Run experiment plan with PI/co-PIs and refine accordingly.
4. Finalize human subject experiment plan.
5. Run experiment plan with first set of elderly human subjects.
6. Update algorithms based upon elderly subject experimental data.
7. Run experiment plan with second set of elderly human subjects.
8. Analyze results from both sets of elderly subject experimental data.
9. Write human subject experiment paper.
10. Write follow-up journal papers for any or all earlier papers.
12
11. Compile all findings in final project documentation.
Figure 3 presents the timeline and the main task division among the four partners.
Figure 3. The timeline of the research efforts for four years and four partners
6. RESULTS FROM PRIOR NSF SUPPORT
The PI Sahin has previous worked with colleagues at Rochester Institute of Technology through NSF
funding. He was a co-PI on an MRI grant titled “MRI: Acquisition of an Optical Profiler for Surface
Characterization and Dynamic Analysis of MEMS Devices” (ECS#0619676, PIs: Hirschman, Raisanen,
Sahin, Boedo, Fuller, Walter (Senior Personnel); $223,717, 9/13/2006 – 11/29/2009). This award resulted in
the acquisition of a Veeco Wyko NT1100 DMEMS system, with final reporting on 11/29/2009. The system
is used extensively for research and education at RIT, with almost unlimited availability for device
characterization and otherwise-impossible publications [210 – 214]. The PI Sahin was also a Co-PI of an
NSF grant titled, “MRI: Acquisition of a High Resolution X-Ray Diffractometer for Engineering Research
and Education” (MRI # 0521341, PIs: Gupta, Raisanen, Sahin, Hirschman, Kurinec; $253,746, 9/1/05-
9/1/06). The equipment is also used by doctoral students in Microsystems Engineering PhD Program
where PI Sahin serves in the core faculty, resulting in numerous publications [215 – 218].
7. PRELIMINARY WORK DONE
The Multi-Agent Bio-Robotics Laboratory (MABL), led by PI Ferat Sahin, has completed two studies
related to the proposed work: mobile robotics for hospitals [219] and biosignal classification [87, 94, 95].
7.1 Autonomous robot design for automatic loading and unloading of medical carts in hospitals
In this corporate-funded project, an autonomous mobile robot was designed for loading and unloading of
medical carts in hospital washing machines. The robot was designed to work with the hospital system
and washing machines through wireless communication. RFID tags and cameras were used as
13
navigational sensors to precisely locate the robot and the medical carts. In addition, a barcode scanner
was used to determine the medical cart location and type. The robot was successful in the operation in
the test area, and is planned for inclusion with the washing machines Getinge Inc. [220] is manufacturing.
7.2 Real-Time Robot Control with the Designed BCI using EEG signals
Recent publications include a transaction [94] and two conference papers [87, 95] in classification
techniques for biosignals and demonstrating a real-time application of a robot control using biosignal
acquisition tools: ClevMed Bioradio 150 [221] and Emotiv Epoc Headsets [222]. Currently, we are
working on controlling a wheelchair using EEG signals. In this work, we propose to employ our novel
biosignal classification techniques [94] as well as recent hybrid BCI techniques [96]. Based upon the
methods in [87, 94, 95], PSO- and RBFN-based classifiers present the best opportunity for real-time
classification through reduced computational complexity post- training. In order to train and test the real-
time system, a graphical user interface was created in Matlab. This GUI instructs subjects to think “left”
and “right” while visual stimuli by EmoCube [222] is presented. Once data regarding left and right
thinking is collected, it is fed into PSO-RBFN classifier for learning the subjects’ left and right thinking
patterns. A block diagram describing the system is shown in Figure 4. The RBFN outputs during real-
time EEG processing are converted into serial port inputs for a walking robot called Hexapod, equipped
with wireless communication; commands are sent from Matlab. The real-time control of the robot with
two commands was successful in about 90% of the subject trials [94].
Figure 4. Real-time robot control with biosignals.
8. MEDICAL RESEARCHERS WORKING ON THE PROJECT
In his role as co-PI, Dr. Steven Rich, the Medical Director of Long Term Care & Senior Services at
Rochester General Health Systems will be providing medical oversight. This work will include an
evaluation of design documents in accordance with known symptoms, confirmation of human subject
protections, guidance in the selection and study of individuals suffering from geriatric syndromes that
generally require intervention, and a general assessment of the practically and economics of robot
deployment based upon prototype design and construction. As an initial example of this oversight, the
statistics-driven design emphases on early fall intervention and observation of prescription
14
noncompliance were confirmed by high statistical observations during the Director’s 25 years of
experience in geriatric medicine in the Rochester community.
This oversight is of particular emphasis for human subject experimentation in years 3 & 4, ensuring
compliance with federal, state, regional, and organizational guidelines in subject selection, experimental
control, and protection from health and other hazards. During these years, the Director will work with
all personnel to ensure correct experimental setup, completion, and evaluation.
9. RELATION OF PROPOSED WORK TO LONGER-TERM GOALS OF THE PIs
The proposed work will support the long-term goals of the PIs to develop state-of-the-art technological
solutions for an aging population to supplement traditional medicine. This project is intended as one the
first step in an extensive collaboration between Rochester General Health System, Rochester Institute of
Technology, and their new industrial partner, RTSync. Proposed work will provide initial opportunities
in elderly human subject experimentation in the Rochester area, for this and future projects.
10. RELATION OF PROPOSED WORK TO WORK UNDER OTHER SUPPORT
Rochester Institute of Technology (RIT) and Rochester General Health System (RGHS) have recently
established the RIT/RGHS Alliance to capitalize on the synergies between medicine and technology in
quality and cost-efficient health care. The PI / co-PIs are in application for seed funding for smart walker
prototyping, and have begun sensor selection and robot design in anticipation of continued collaboration
within and beyond the scope of this proposal. Related work by co-PIs also includes work in progress in
non-biological (complex system and vehicle) health monitoring under Department of Defense and
Department of Transportation funding, portions of which will be leveraged in the proposed system.
11. BROADER IMPACTS
We will engage K-12 and college students through culturally popular and low-threshold applications
(robotics) and workshops, expand existing K-12 outreach activities at RIT and boost mentoring of
National Technical Institute for the Deaf (NTID) and Engineering students in Senior Design projects.
11.1 Culturally Popular and Low Threshold Application to Engage Students in STEM
With recent low-cost, miniaturized components and increasing computational power, Lego-based
robotics kits have great appeal for children to work, play, and learn with robots [223]. The Intelligent
Brick (later: Lego Mindstorm [224 – 226]) by MIT first received widespread attention. The Carnegie
Mellon University developed Robotics Engineering I and II
curricula [227] that have been widely used to teach science and
engineering concepts to K-12 and college students through Lego
Mindstorm kits in a fun and innovative environment [224, 228 –
235]. There are several national competitions [236] that use Lego
robots (e.g. First Lego Leagues [237]) or other robots (e.g. US First
Robotics [238], Robotics Soccer [239], Firefighting Robots [240]) to
promote interest in STEM fields [233 – 235, 241]. ROBOLAB [247]
and Lego Mindstorm Education NXT software (powered by
LabView [243]) are programming environments well-received by K-
12 students. The PI Ferat Sahin has been using Lego Mindstorm kits
to offer a two-day weekend workshop, called RoboWeekend (See Figure 5) [244] and summer camps
called RoboCamps [244], for K-12 students. These programs use the CMU Robotics Engineering I&II
curriculum and in 2009-10 offered training to 66 students (grades 2-10) from urban, suburban, and rural
districts, with more than a quarter being females. Students found the workshops to offer highly (83%) or
Figure 5. K-12 students at
RoboWeekend.
15
somewhat (17%) valuable knowledge about robotics. Students responded with “definite” (56%) and
“maybe” (42%) to a question of whether they would want to return; similar student feedback was
received from the 2010-2011 workshops. The Robocamps, first run in Summer 2011 for total of 75 middle
and high school students, are designed to have three levels: beginner, advance, and expert. The
satisfaction levels for the beginners, advance, and expert RoboCamps were 95%, 93%, and 100%
respectively. Students stated that they had enjoyed, learned, and would like to study in STEM fields.
We intend to integrate the proposed smart walker prototypes into these RoboWeekend and
RoboCamp workshops.
RoboWekeends: In a 2-day Beginner RoboWeekend workshop, students work in teams of two to learn
basic building blocks of the Lego Mindstorm NXT kit through instructional videos, tutorials, and
presentations from the CMU Robotics Engineering I curriculum. The program design meets many of the
NETS standards [245] to promote STEM interest. In the first day, students first build their robots by
attaching motors, sensors, and other components. By construction and programming of the robot,
students prepare for a challenge in the second day when they build as a team without major supervision.
In Advance RoboWeekends, students build and program sensor and actuator units from the Lego
Mindstorm Kits (sound sensor, infrared sensor, speakers) and third-party sensors (gyroscope and
camera). The students also explore complex programming concepts to handle data from multiple sensors.
RoboCamps: Beginner and Advanced RoboCamps are designed based upon Beginner and Advanced
RoboWeekend workshops, tailored to students with limited experience in robotics and Lego kits. In the
Beginner RoboCamp, students learn the basics of Lego kits and robotics concepts such as sensor
interfacing, line following, navigation, and programming. In the Advanced RoboCamp, similar topics are
covered with harder challenges and more complex programming. In Expert RoboCamps, students
design a servo-based mobile robot controlled by an Arduino microcontroller and infrared (IR) proximity
sensors. Sensor interfaces are explored, such as infrared and ultrasonic proximity and touch sensors. In
addition, students will experiment with the Smart Walkers from this project.
In Fall quarters, we will have four Beginner RoboWeekend workshops. In Winter and Spring, we will
run four Beginner and two Advance RoboWeekend workshops. Thus, we will reach out to 220 K-12
students per year. In summers, there will be six weeks of RoboCamp workshops (three Beginners (15
students per session, two Advance and one Expert camp (10 students per session)) for a total of 75
students. In four years, we will reach out to 1005 K-12 school students. We will provide registration fee
support for 200 underrepresented and minority students.
11.2 Integration of Smart Walker prototypes with other existing K-12 outreach activities at RIT
The WE @ RIT [246] organization holds extensive on-campus and weekend activities for middle- and
high-school girls throughout the academic year and summer. Programs encourage and inspire girls and
young women in grades 4-12 to consider engineering as a career. Several hundred students (K-12),
educators, and parents participate in WE@RIT outreach programs each year. In a one-day program for
grades 10-12, girls learn about emerging engineering topics. Other RIT K-12 programs, such as E3 Fair
[247], and Traveling Engineering Activity Kits [248], will also introduce our Smart Walker prototypes.
11.3 Structured mentoring environment for deaf or hard of hearing students interested in robotics
The National Technical Institute for the Deaf is the world’s largest technical college for deaf students and
is one of eight colleges at RIT. Hearing-impaired students in engineering will be recruited for Senior
Design teams and as undergraduate researchers; funds have been specifically allocated in the budget for
this. Formal training will be created for these undergraduate students with a focus on graduate school
preparation. Training plans will be developed in conjunction with academic advisors at NTID.
16
12. COORDINATION PLAN
While the proposed system will function as a single whole, the final responsibility of each component is
divided based upon area of expertise.
Dr. Sahin (and his students) will be responsible for the software and hardware design and
implementation of the robotic “smart walker” subsystem, including: recording signal data, navigating
autonomously, maintaining awareness of all observable people and animals, and identifying imminent
risks of falling.
Drs. Ardis and Nenadic (and their students) will be responsible for the software and hardware design
and implementation of the analytics server subsystem, including: maintaining infrastructure for
continuous alert generation, generating fall detection alerts, generating medicine noncompliance alerts,
generating biosignal anomaly alerts, and generating biosignal degradation alerts.
Drs. Zeigler and Seio will be responsible for the software and hardware design and implementation
of the base station subsystem, including: managing data transmission between subsystems,
standardizing data representation, generating biosignal “extreme value” alerts, and minimizing data
transmission (during inactivity or sensor dropout).
Dr. Rich will be responsible for the medical oversight of subsystem design and final human subject
experimentation.
Some of these responsibilities, particularly medical oversight of design and standardization of data,
will require coordination between personnel to ensure compatibility of subsystems. For these
responsibilities, the following steps will be taken:
1.) A wiki will be constructed for continuous discussion and refinement of research work. This wiki
will be partitioned by subsystem, with moderation of each subsystem the responsibility of the
corresponding PI/co-PI(s). Login will be limited by unique usernames and passwords for all
personnel (including students), and the terms of usage will include continuous logging of edits
and restriction of data discussions to cross-subject/non-subject statistics without revealing any
raw signal data (pre- or post-anonymization). This wiki will be maintained by Drs. Ardis and
Nenadic, in addition to their responsibilities above, running on the server subsystem hardware.
2.) Meetings between all senior personnel will occur every second week, on a schedule yet to be
determined. These meetings will include presentations of recent progress, discussion of existing
problems, brainstorming of solutions, and confirmation of deadlines. In order to accommodate
personnel being located in different parts of the country, these meetings will be conducted via
Skype ™ and FreeConference ™ with audio logged for later reference (accessible through the
wiki). Raw signal data will not be presented at these meetings.
3.) Meetings between senior personnel and their students and staff will occur at least once every
week, on a schedule yet to be determined, for all senior personnel with such support. These
meetings will parallel the senior personnel meetings, but with an emphasis on work interplay
within each subsystem and a finer granularity of progress. These meetings will not have
recorded audio, and raw signal data may be presented and discussed freely.
4.) Meetings between all personnel will occur once every six months, on a schedule yet to be
determined. These meetings will parallel the other meetings, but with an emphasis on significant
research milestones achieved and any issues requiring additional collaboration beyond the scope
of the above meetings. All-personnel meetings will take place at the Rochester Institute of
Technology, with personnel traveling as necessary. Presentation materials for these meetings
will be made available through the wiki, along with audio logged for later reference. Raw signal
data will not be presented, although its transfer will be allowed in person (offline, encrypted).
5.) Irregular communication between personnel will be available through email and phone calls, as
needed. Raw signal data will not be discussed through these means.
17
In addition to the above steps to perform cooperative work on the project, we intend to publish all
findings via a series of presentations in scholarly conferences (with follow-up articles in applicable
journals), at the annual Imagine RIT Innovation & Creativity Festival, and at local K-12 schools within the
Rochester, NY area. Furthermore, we intend to make all design and presentation documents public via
web hosting, including software source and libraries (as described in the Data Management Plan).