Design and Implementation of SEE-Phone in

SEES (Smart Environment Explorer Stick)

M. Yusro1,3, K.M. Hou1, E. Pissaloux2, K. Ramli3, D. Sudiana3, Z. Lizhong1 and H.L. Shi1 1Laboratoire d'Informatique de Modélisation et d'Optimisation des Systèmes, Université Blaise

Pascal, France, yusro@isima.fr, kun-mean.hou@isima.fr and hongling.shi@isima.fr 2Institut des Systèmes Intelligents et de Robotique, Université Pierre et Marie Curie

(UPMC, PARIS 6, Paris-Sorbonne), France, Edwige.Pissaloux@upmc.fr 3Department of Electrical Engineering, Faculty of Engineering, Universitas Indonesia, Indonesia,

k.ramli@ui.ac.id and dodi.sudiana@ui.ac.id

Abstract – The mobility is a complex, yet not well

understood, human cognitive process; it subtends several

different tasks such as walking and orientation. The visually

impaired people (VIP) mobility heavily depends on the white

stick (or white cane) which however does not provide an

efficient support for walking and orientation. Indeed, these

two later functions implementation requires the usage of high

technology which should be conveniently integrated to the

white stick in order to obtain the smart stick. This paper

proposes to fill in this gap, and outlines the SEES (Smart

Environment Explorer Stick). This paper focuses on SEE-

phone as a part from whole SEES system. The first prototype

of SEE-phone, a part of the SEES system, is presented.

Keywords: smart phone, GPS, Google map, web service,

SEE-Phone, smart-stick, ubiquitous SEES, Visually Impaired

People (VIP), assistive device

I. INTRODUCTION

Blindness and visual impairment are a major hindrance

in daily life for accessing information, mobility, finding a

way, interaction with the environment, communities and

the others activities. Both of them became a very serious

problem in health and social security. In addition, the other

important issue for VIP is independent ability to move

outdoor. Usually, they are able to move independently

only along the routes which they have already learned

together with a sighted guide [1]. On the other hand, they

have a strong need to walk around by themselves, even

when they are in new location or accidentally lost their

way.

In general, navigation problems in the blind population

have risen regarding mobility and orientation [2-3].

Mobility is the ability to travel safely, comfortably,

gracefully, and independently [2]. Mobility of the visually

impaired can also be interpreted as an activity that

combines earlier defined capabilities (space perception,

orientation, way-finding, navigation and obstacle

avoidance). The human autonomous mobility is a

capability to reach a place without assistance of any other

person.

The below definition raises several problems which

should be cognitively solved [4-5]; two of them are self

orientation and space self awareness. The concept of self

orientation means the knowledge of the relative spatial

position between our current position and targeted spatial

location. Two basic questions which subtend the

orientation are: “where I am?” and “how to reach the

destination from my current position?”. Consequently, the

orientation allows to planning a specific route (path) to

reach the targeted location from the current point.

Space awareness is the capability to know about urban

and social data in our peri-personal space (“space around

our body”), and our navigational space [6]. Some

questions which subtend the space awareness might be:

“what are the nearest streets located one with respect to

the other and to me ? what is the traffic light at the street I

like to cross ? where is the library I like to get in ? “

This paper proposes a support for orientation and space

awareness using the high level technologies, which should

be integrated in the VIP mobility assistive device. More

precisely, the SEES system (or ‘Smart Environment

Explorer Stick’) is defined. The SEES system integrates: a

global remote server (iSEE), an embedded local server

(SEE-phone) and a smart stick (SEE-stick). In this paper,

we focus on SEE-phone and its useful features such as

GPS, camera, TTS (Text-to-Speech), voice command, etc.

Therefore, the paper is organized as follows: Section 2

provides a state-of-the-art on the existing commercial and

academic solutions for VIP navigation and focus on their

supports of orientation and space awareness concepts.

Section 3 defines the SEES concept and outlines its

implementation in the first prototype. Section 4 proposes

the SEE-Phone concept and design. Section 5 addresses

the results of the first SEE-Phone prototype experimental

evaluation. The final section 6 indicates some future works

toward the fully operational of SEES prototype.

II. STATE OF THE ART

Two complementary tracks are investigated for

navigation assistance for VIPs design are GPS-based and

vision-based solutions.

A. GPS-Based Navigation Technology for VIP

Since 1960 various assistive devices for the navigation

system dedicated to VIP have been developed, ranging

from the low to high technology devices. In 1991,

Golledge and team were persons who first proposed for

using of GIS, GPS, speech and the sound sensor (sonic

sensors) to navigate the blind [7]. A sample of navigation

system for VIP is Personal Guidance System (PGS) and an

early model was developed in 1993 [8].

In general, it is possible to materialize the concept of a

navigation system for VIP as a set of the following

modules (figure 1): a) GPS for determination the

traveller’s orientation and position, b) Geographic

Information System (GIS), a software for route planning,

c) The spatial database for relating the traveller’s

orientation and GPS coordinates to the surrounding

environment and d) the user interface.

Fig. 1. Functional Components of PGS System [8]

The system usually provides a range of functions which

generally contains simulation of a journey. It displays the

route (path), searching in the database from a starting point

to targeted (end) point. The provided in real-time audio

instructions enable users to follow the path.

The PGS system above consists of three modules [9].

The first module is used for user location and orientation.

A GPS receiver with differential GPS receiver (DGPS) is

used to determine location and to trail the path of travel.

The DGPS configuration also gives absolute positional

accuracy on the order of 1m root-mean-squared error

given good satellite availability. The second module is the

GIS components that consist of a digitized base map and

software design to trail the traveller’s path, select routes,

recommend the traveller about local features and

landmarks, correct for signal error or loss, and control

dynamic access to the database. The module also contains

the commands needed to navigation aid. The spatial

attributes might concern some criterion of system

centrality, feature or population density, connectivity,

nearest neighbours, distances to other features and etc. The

third module is the user interfaces that provides for two-

way communication between the GIS and the user. In this

part, there are two display options for GIS to user

communication, namely a virtual acoustic display using

binaural earphones and conventional speech display with

an earphone or speaker.

There are some devices of GPS based navigation

systems has been designed or modified, namely: Trekker

of Human Ware, BrailleNote GPS, MoBIC, Loadstone

GPS, Wayfinder Access and Mobile Geo [10]. The system

provides a range of functions which generally contains

simulation. It could display the route on the way, the

search of database from a certain point and real-time

instruction (audio) which enables users to follow the track

to a specific destination.

Trekker of Human Ware is a personal digital assistant

(PDA) or pocket PC adapted for the blind and visually

impaired with talking menus, talking maps and GPS

information. This device has a good functional

possibilities and built-in TTS (Text to Speech). It can

expand to accommodate new hardware platforms and

more detailed geographic information. This device does

not provide a route remote tracking for monitoring the user

location and it cannot detect the obstacle.

BrailleNote GPS is a portable tool with speech and

braille output. It is like a combination of a personal digital

assistant, map-quest software and a mechanical voice. A

BrailleNote has a GPS receiver, maps, and points of

interest database that provide spoken and/or braille access

to location information in any outdoor environment. This

device does not have a route remote tracking for

monitoring the user location and it cannot detect the

obstacle.

MoBIC (Mobility for Blind and elderly people

Interacting with Computers) is designed to allow a blind

person access to information from many sources. The

resulting system has two parts: the pre-journey system,

and the outdoor system. The output system is in the form

of spoken messages. The planning system helps blind

people to study and plan their routes in advance, indoors.

This device does not provide a route remote tracking for

monitoring the user position and it does not have function

to the obstacles detection.

The Loadstone GPS project develops open source

software for satellite navigation for blind and visually

impaired people. The device uses a smart phone, operated

with screen reader, limited possibilities and unstable

operation. Like with the previously device, the Loadstone

GPS does not provide a route remote tracking for

monitoring the user location and it cannot be used to

detect the obstacle.

Wayfinder Access is a navigation system for mobile

phones specially developed for visually impaired. This

application is only for symbian phones, operated by screen

reader and requiring a GPS maps. This device does not

have a route remote tracking for monitoring the user

location and it cannot detect the obstacle.

Mobile Geo is a product designed to convey most of the

information displayed on commercial GPS receivers and

location databases to people with visual disabilities. This

device is used only for smart phones, PDA or Pocket PC

with MS Window Mobile. It is operated with screen reader

and requires GPS maps.

In general as conclusion, several systems above do not

provide assistance for VIP space awareness for

independent navigation. Those systems also do not have a

route remote tracking for monitoring the user location and

do not have the ability for detecting an obstacle (e.g. pole,

traffic lights, building, etc.).

B. Navigation System Based on Vision System for VIP

It is important in independent mobility to provide an

ability to detect a physical obstacle. The ability to know

the presence of a physical object completely can be

performed by vision system. Some of researches on

navigation systems for VIP have been developed the new

devices by combining the orientation function (GPS and

GIS) and vision system. The vision-based mobility

systems an image is captured then conveyed to the VIP via

different sensory channels such as sound or touch.

Concept of using video camera as vision sensors was

introduced through portables systems by Peter Meijer,

namely vOICE. The vOICE system [11

mounted single video camera and translates the image into

sounds (headset). The scene in front of the user is

via head movements, from up to down, and

data are cognitively integrated and interpreted

user. The system does not provide nor orientation neither

space awareness assistance to the VIP.

The NAVIG system [12] attempts to combine

artificial vision and visual geo-located landmark

allow completion of the path using the GPS

user current position. The NAVIG system has

modules: the objects location is used to allocate the

necessary objects for the user and the u

function is used to detect visual objects that are not

displayed to the user, but it is used to improve GPS

positioning. The system used Spike-Net NAVIG

that provides a fast algorithm based on visual research

human.

Jie Xu, et.al [13] proposed a system -

which integrates GPS and vision on a PDA devic

system is made to provide contextual cues for

navigate safely in dynamic situations. The

captures the dynamic information environment and

information landmarks along the route of the map data

using a Visual Studio eSuperMAp.

Do-Hoon Kim and Heung-Gyoon Ryu [

navigation system for VIP which combines the

camera and ultrasonic sensors. This system gets

information about the environment using GPS and

distance data using ultrasonic sensors, and then inform the

user. The camera is used to assess the situatio

users as a color. In the case of short-range obstacles,

vibrators transmit sound and vibration to the user. This

system used a bluetooth communication for tra

data.

Conclusions from several systems above are

systems have combined the orientation function and vision

function, the systems use camera and ultrasonic to detect

an obstacles, but the systems do not provide

monitoring for tracing the user location.

III. THE SEES SYSTEM

A. SEES Concept.

The SEES system aims to provide assistance for VIP

mobility. This paper addresses the SEES support for VIP

orientation and space awareness as defined in Section 1.

The main innovation of SEES is internet connection

which enables the VIP to get help and be

monitored [15], therefore it is possible to consider

ubiquitous smart stick.

The SEES system (figure 2) contains three main

components: a global remote server (iSEE), an embedded

local server (SEE-phone) and a smart stick (

stick). iSEE is a global server providing the web services

for the VIP such as remote real-time hint and help and

remote monitoring (trace the VIP location).

The SEE-phone is based on a commercial smart phone.

It is used as an embedded local server and provides the

local services for the SEE stick such as route vector and

video camera as vision sensors was

introduced through portables systems by Peter Meijer,

] uses a head

translates the image into

sounds (headset). The scene in front of the user is scanned

and all auditory

rpreted by the end

The system does not provide nor orientation neither

to combine the

located landmarks that will

the GPS provided end-

The NAVIG system has two main

is used to allocate the

ser positioning

function is used to detect visual objects that are not

sed to improve GPS

NAVIG library

that provides a fast algorithm based on visual research

- audio guide -

a PDA device. The

contextual cues for VIP to

The PDA camera

dynamic information environment and

along the route of the map database

Gyoon Ryu [14] proposed a

navigation system for VIP which combines the GPS,

. This system gets

information about the environment using GPS and

d then inform the

. The camera is used to assess the situation in front of

range obstacles,

transmit sound and vibration to the user. This

system used a bluetooth communication for transmitting

several systems above are these

systems have combined the orientation function and vision

the systems use camera and ultrasonic to detect

an obstacles, but the systems do not provide a route remote

The SEES system aims to provide assistance for VIP

mobility. This paper addresses the SEES support for VIP

orientation and space awareness as defined in Section 1.

The main innovation of SEES is internet connection

t help and be internet

t is possible to consider it as an

contains three main

components: a global remote server (iSEE), an embedded

phone) and a smart stick (wheeled SEE-

stick). iSEE is a global server providing the web services

time hint and help and

remote monitoring (trace the VIP location).

phone is based on a commercial smart phone.

rver and provides the

local services for the SEE stick such as route vector and

internet access to iSEE. SEE-phone is always connected to

SEE-stick through Wi-Fi.

Each sub-system provides independently

walking and to orientation of the VIP

data for walking, while SEE-phone is the key device for

orientation.

The system provides two channel feedback to the VIP

tactile (point-wise) and audio. The

elaborated by fusing several working in parallel

Fig. 2. Concept of SEES System

Notice that 6LoWPAN (IPv6 over Low Power Personal

Area Network and) and RPL routing protocol are adopted

to implement the SEE-stick thus according to the context

the SEE-stick can connect P2P with the iSEE and

transportation system [16-17]. Therefore,

can be considered as the implementation of ITS

‘Intelligent Transportation System’ concept

this system can be used in the near future

urban transportation system such as car, bus and train.

The mobility cues collected by SEES sensors will be

transformed into high level knowledge (passed to the

via voice or tactile feedbacks) in order to: 1) estimate or

predict the status on an object (for example:

traffic light); 2) obtain the accurate location data (what

allows to track the user and check if (s

correct way); 3) obtain a correct

environment conditions such as obstacles, status of the

traffic light and status of the walking surface; and 4) send

quickly error/alerts messages to VIP

mistake occurs.

B. Considered Mobility Cues and SEES

SEES targets to assist some walking

functions. The considered walking sub

following: obstacle detection, distance to obstacle

estimation, obstacle shape recognition,

estimation, surface roughness estimation

(position and status) detection.

The considered orientation sub-functions are

estimation to the targeted location and end

location/position estimation.

The assisted functions have a direct impact on selected

sensors. The SEES available sensors are:

sensor, camera, wheel encoder, accelerometer

compass.

phone is always connected to

independently assistance, to

of the VIP. SEE-stick collects

phone is the key device for

two channel feedback to the VIP:

These feedbacks are

working in parallel sensors.

Concept of SEES System

IPv6 over Low Power Personal

Area Network and) and RPL routing protocol are adopted

stick thus according to the context

stick can connect P2P with the iSEE and future

Therefore, the SEES system

implementation of ITS

concept. It means that

in the near future by VIP to access

em such as car, bus and train.

SEES sensors will be

transformed into high level knowledge (passed to the VIP

via voice or tactile feedbacks) in order to: 1) estimate or

tatus on an object (for example: status of the

traffic light); 2) obtain the accurate location data (what

allows to track the user and check if (s)he is on his/her

information about

environment conditions such as obstacles, status of the

the walking surface; and 4) send

VIP when orientation

SEES Sensors.

walking and orientation

sub-functions are the

distance to obstacle

obstacle shape recognition, walked distance

estimation, surface roughness estimation and traffic light

functions are: direction

and end-user current

a direct impact on selected

SEES available sensors are: ultrasonic

, camera, wheel encoder, accelerometer and

The ultrasonic sensor is used to detect the obstacles

front of the VIP in very near distance (such as

etc.) with provide the precision.

The camera in SEE-stick is used to identify and

recognize the type of road junctions (e.g.

corner-road, side-road, etc.). In addition, the camera is also

used to read the environmental conditions,

illumination (day and night) of traffic light status

Status of the traffic lights (figure 3) is detected

the smart phone camera (in SEE-phone). The

function dedicated to detect traffic lights status

VIP gets the information from SEE-stick about

lights position.

Fig. 3. The Traffic Lights Model

The wheel encoder sensor is mounted on

wheels, and is used to estimate VIP travel distance

the starting place.

In association with camera information,

data is used to estimate walking surface roughness

the special road pad for VIP (figure 4).

Fig. 4. The Special Road Pad for Blind People

The compass sensor is used to detect a moving direction

of the VIP. Data from the compass will be integrated with

the GPS data and wheel encoder data to enhance the

precision of VIP location and distance estimation

The orientation function will use two main

sensors: camera and GPS (includes a Google

The GPS is used to define the current location

VIP, while the Google map allows to plans the user

journey in order to reach his/her spatial target.

the GPS data will be sent to the database server for

monitoring the VIP, displaying and checking

itinerary in real-time.

Table 1 summarizes the different sensor

mobility functions supported by the SEES system

the obstacles in

(such as tree, wall,

to identify and

(e.g. crossroad,

In addition, the camera is also

used to read the environmental conditions, such as

of traffic light status.

is detected also with

phone). The camera has a

status once the

stick about traffic

The Traffic Lights Model

mounted on the SEE-stick

VIP travel distance from

In association with camera information, accelerometer

walking surface roughness, such as

The Special Road Pad for Blind People

moving direction

compass will be integrated with

the GPS data and wheel encoder data to enhance the

location and distance estimation.

will use two main SEE-phone

Google map).

the current location of the

map allows to plans the user

spatial target. Periodically

server for remote

, displaying and checking his/her

sensors and the

the SEES system.

TABLE 1: THE TYPE AND FUNCTION OF SENSORS

Name of Sensor

SEES System

SEE-

stick

SEE-

phone

Ultrasonic

Wheel Encoder

Accelerometer Compass

Camera

GPS

V

V

V V

V

V

V

V

V

Data collected from different SEES sensors will

combined as follows in order to provide more reliable

outputs (mobility cues): 1) Location data from GPS

phone and SEE-stick will be combined

current accurate location. Location

with data from encoder sensor (distance measurement); 2)

Image data (color and obstacle) from camera

and SEE-stick will be combined in accurate

recognition and object detection

accelerometer data will also be fused

information; 3) Direction data from SEE

stick (compass sensor) will be combined

direction. In case of disagreement a message may be sen

to get help from the iSEE global server; and

data from ultrasonic sensor, distance data

sensor, and surface roughness from accelerometer sensor

will be used as raw data for further processing

The mobility cues will be the inputs to the SEES layer

which will transform them into high level knowledge

useful for mobility; this knowledge which can be

conveyed to the VIP. Figure 5 shows the model of

interconnection in SEES system.

Fig. 5. Sensors Interconnection Model

C. SEES Running Models (Yusro 2013)

SEES is a modular system at the physical level

transmission between the SEE-phone

performed via Wi-Fi protocol. Table

SEES modes. Each mode can be selected

(s)he will go to a targeted location.

TABLE 2: THE SEES MODES

Mode SEES System

SEE-stick SEE-phone iSEE

Mode 0

Mode 1

Mode 2

Mode 3

V

V

V

V

V

V

V

V

THE TYPE AND FUNCTION OF SENSORS

Output Data

Obstacle/object

Travel distance

Surface roughness Direction

Color/obstacle

Location/position

collected from different SEES sensors will be

as follows in order to provide more reliable

Location data from GPS SEE-

combined into the end-user

data communicates

r sensor (distance measurement); 2)

) from camera SEE-phone

accurate result color

recognition and object detection. Ultrasonic and

be fused with camera

SEE-phone and SEE-

s sensor) will be combined into an accurate

In case of disagreement a message may be sent

elp from the iSEE global server; and 4) Obstacle

, distance data from encoder

from accelerometer sensor

further processing.

The mobility cues will be the inputs to the SEES layer

nto high level knowledge

which can be directly

shows the model of sensors

Model in SEES System

(Yusro 2013).

at the physical level. Data

phone and SEE-stick are

Table 2 shows all detailed

selected by the VIP when

THE SEES MODES

Description iSEE

V

V

Basic mode

Phone mode

Local mode con

Complete mode

The SEES interface to VIP offers four assistance

1) Mode 0 (always active, SEES minimal

only the SEE-stick is active). This mode will

the VIP to walk without a smart phone.

2) Mode 1 or phone mode (the SEES work

SEE-phone and iSEE). This mode will allow

stay moving without a smart stick. The navigation

be done using smart phone sensors (GPS, camera and

compass) and iSEE server.

3) Mode 2 or local mode connexion. In this mode

VIP can get a remote help from person

the server as the SEES provide the

specific hints and helps;

4) Mode 3 or the complete mode. In this mode, the VIP

can get a remote help as the SEES provide

selected travel specific hints and helps.

IV. SEE-PHONE

The SEE-Phone is a smart phone that be

two main tasks are orientation and walking

true independent mobility. SEE-phone is the key device

for orientation; indeed, the SEE-phone communicates with

the GPS, through the web server accessing to the map

database and with the others mobile devices.

A. SEE-Phone as Orientation Function

As a function of orientation, the SEE-phone is used to

give direction to destination location, create a new route or

guide the way of the route that has been created previously

(route remote tracking). To implement the navigation

function, SEE-phone uses GPS on smart phone.

In general, the GPS on smart phone

(World Geodetic System 1984) as a reference system.

WGS84 regards the earth as a spheroid object is

determined based on the satellites observation satellites in

earth orbit. The WGS84 major axis is 6,378,137.0 meters

and a minor axis is 6,356,752.3 meters [18].

The tracking of a VIP during his/her travel can be

decomposed in the two following elementary operations:

1) Detection the VIP’s current position and

2) VIP’s current position real-time tracking with

and Google map.

The implementation of these tasks have

follows: 1) to develop an application on Android

phone for detecting the VIP location; 2) to develop an

application for tracking the VIP position in real time with

GPS; 3) to connect the developed application to the server

in the others place (campus), 4) to monitor VIP position

from map display on monitor PC/notebook server; and 5)

to display the VIP position by server.

Figure 6 shows the access model from GPS receiver (in

smart phone) to database server. The VIP current location

data is obtained from smart phone GPS and sent to the

web server to be stored in database server. The last, the

data can be shown on the monitor as line track/route.

assistance levels:

, SEES minimal configuration,

his mode will allow

.

the SEES work by using the

his mode will allow a user to

stick. The navigation will

(GPS, camera and

In this mode, the

persons who monitor

selected travel

or the complete mode. In this mode, the VIP

the SEES provides the

selected travel specific hints and helps.

that be functioned to

walking functions for a

phone is the key device

phone communicates with

the GPS, through the web server accessing to the map

others mobile devices.

phone is used to

give direction to destination location, create a new route or

been created previously

To implement the navigation

phone.

uses WGS84

) as a reference system.

WGS84 regards the earth as a spheroid object is

determined based on the satellites observation satellites in

earth orbit. The WGS84 major axis is 6,378,137.0 meters

].

The tracking of a VIP during his/her travel can be

ing elementary operations:

the VIP’s current position and

time tracking with GPS

s have goals as

: 1) to develop an application on Android smart

or detecting the VIP location; 2) to develop an

application for tracking the VIP position in real time with

GPS; 3) to connect the developed application to the server

in the others place (campus), 4) to monitor VIP position

otebook server; and 5)

shows the access model from GPS receiver (in

) to database server. The VIP current location

GPS and sent to the

database server. The last, the

data can be shown on the monitor as line track/route.

Fig. 6. The Global Concept of SEE

B. SEE-Phone for Color Detection

The implementation of the traffic light status detection

with SEE-phone requires the followin

operations:

1) Traffic lights detection,

2) Traffic light (color) status recognition

3) Status data feedback to the VIP

Figure 7 shows a possible SEES

namely: the camera on smart phone will detect

lights (through two ways detection models, namely:

traffic lights position is detected by the camera sensor

the stick (SEE-stick) and the traffic

detected based on GIS data in databa

the image data from the traffic lights status; the Android

smart phone will process the image; finally, the

phone will produce the audio/voice output to VIP.

Fig. 7. SEE-Phone for Traffic Lights

The whole process on Android system

steps (figure 8): image acquisition, image pre

color filtering and thresholding.

Fig. 8. The Image Processing on Android

The thresholding function will return a binary

which can be easily “translated” into audio data for the

VIP.

C. User Interfaces on SEE-Phone

Navigation guidance using verbal description and

instruction is considered as efficient way

VIPs [19]. Two smart phone applications will be

TTS and STT.

The mobile TTS (Text-to-Speech)

smart phone screen message reading and

for the VIP [20]. Figure 9 shows an overview of a typical

TTS System.

. The Global Concept of SEE-Phone

of the traffic light status detection

requires the following elementary

(color) status recognition; and

VIP.

a possible SEES implementation,

will detect the traffic

detection models, namely: the

detected by the camera sensor in

traffic lights position is

database server), and take

the image data from the traffic lights status; the Android

will process the image; finally, the smart

output to VIP.

Lights Detection

on Android system requires four

image acquisition, image pre-processing,

. The Image Processing on Android

return a binary image

which can be easily “translated” into audio data for the

Navigation guidance using verbal description and

efficient way to pass data to

phone applications will be used:

) is an application for

and audio translation

shows an overview of a typical

Fig. 9. Overview of a typical TTS System

The speech recognition application (STT,

Text) allows a machine or program to receive and interpret

dictation, or to understand and carry out spoken

commands [20]. In practice, the size of a voice

program's effective vocabulary is directly related to the

random access memory capacity of the computer in which

it is installed.

In SEE-Phone, the instruction inputs or commands will

be given in voice. The VIP will speak to run a specific

application with his/her voice (STA, Speech

The SEE-phone output also produces the voice or sound

to be heard by VIP. Figure 10 shows the scheme of voice

command on SEE-Phone implementation.

Fig. 10. The Voice Command on SEE-Phone

The TTS application on SEE-Phone will be

to give the information to VIP about the real

situation captured by camera smart phone;

TTS will translate the text of traffic lights status to

voice/sound. Figure 11 shows the scheme of TTS on SEE

Phone implementation.

Fig. 11. The TTS Application on SEE-

V. EXPERIMENTAL EVALUATION OF SEE

Two or three functions have been implemented in

Phone which are useful for assistance of mobility

color detection using smart phone (for traffic l

detection) and remote route tracking (for continuous check

of the VIP itinerary). Experiments have been realized

Android emulator system (running on personal

and on smart phone. Their goal was to evaluate the

technical performance of the SEE-phone.

. Overview of a typical TTS System

STT, Speech-to-

a machine or program to receive and interpret

dictation, or to understand and carry out spoken

In practice, the size of a voice-recognition

ctly related to the

random access memory capacity of the computer in which

Phone, the instruction inputs or commands will

be given in voice. The VIP will speak to run a specific

Speech-To-Action).

phone output also produces the voice or sound

shows the scheme of voice

Phone (STA).

will be developed

to give the information to VIP about the real-condition or

; therefore, the

of traffic lights status to

shows the scheme of TTS on SEE-

-Phone

EXPERIMENTAL EVALUATION OF SEE-PHONE

have been implemented in SEE-

assistance of mobility, namely:

(for traffic light status

detection) and remote route tracking (for continuous check

have been realized on

ersonal computer)

Their goal was to evaluate the

A. Traffic Lights Status Detection (Yusro 2013)

In this experiment, a camera of smart phone

SIII Model GT-I9300 has been used to capture the color

image. This process uses simple application program to

produce HSV level of color image. Table

HSV level from three colors image (red, yellow

in this experiment.

The average HSV is a reference data for the image

captured by the camera (SEE-phone). When the value of

the colors (traffic lights) is captured by the SEE

camera that was in the range of reference data, the

program will call the voice application. In this experiment,

the voice function used TTS (Text To Spe

The TTS converted the result of images detection as voice,

(“red stop here”, “yellow slowly run” and “

run away”).

TABLE 3: SAMPLE DATA OF HSV LEVEL

Color Images HSV min

Red1

Red2

Red3

Red4

175, 236,169

175, 234,201

175, 242,242

175, 240,246

Average of Red 175, 238, 215

Yellow1

Yellow2 Yellow3

Yellow4

29, 240, 242

26, 245, 243 27, 242, 244

30, 247, 252

Average of Yellow 28, 244, 245

Green1

Green2

Green3

Green4

89, 244, 238

89, 228, 239

88, 233, 229

90, 218, 249

Average of Green 89, 231, 239

The results of this experiment are displayed in figure

12. The upper images (left to right)

lights status for pedestrian. The lower images (left to right)

show results of color detection by SEE

experiment, android application program

colors: red, yellow and green.

Fig. 12. The Color Detection on Android Emulator

B. Route Remote Tracking with Database

2013)

Figure 13 shows map view from the

application program (AndroidTrack) tested on

emulator and (b) Android smart phone

(Yusro 2013)

mart phone Samsung

I9300 has been used to capture the color

application program to

of color image. Table 3 shows data of

red, yellow and green)

The average HSV is a reference data for the image

phone). When the value of

the colors (traffic lights) is captured by the SEE-phone

camera that was in the range of reference data, the

program will call the voice application. In this experiment,

Text To Speech) concept.

The TTS converted the result of images detection as voice,

yellow slowly run” and “green please

OF HSV LEVEL

HSV max

179, 255, 255

179, 255, 255

177, 255, 255

177, 255, 255

178, 255, 255

32, 255, 255

32, 255, 255 32, 255, 255

32, 255, 255

32, 255, 255

91, 255, 255

91, 255, 255

90, 255, 255

91, 255, 255

91, 255, 255

periment are displayed in figure

The upper images (left to right) are signs of traffic

The lower images (left to right)

of color detection by SEE-phone. In this

experiment, android application program can detect three

on Android Emulator

Database Server (Yusro

shows map view from the Android

tested on (a) Android

smart phone.

In this experiment, Android Emulator Google APIs,

Platform 4.1.2, Level 16 and Smart phone Samsung SIII

Model GT-I9300, Android Version 4.1.1 were used.

Figure 14 shows a map of sample route in this

experiment. The user walks carrying a smart phone (SEE-

phone) from G point (start-point) to D point (end-point).

When the user walks, the Android application will run and

the GPS receiver is in “on” position. A GPS receiver of

the smart phone will receive data from GPS satellite and

SEE-phone will send the data to database server via Wi-Fi

network.

(a) (b) Fig. 13. Map View on Android Emulator and on SEE phone.

Fig. 14. The Sample of Route Map

Data locations from SEE-phone collected by database

server are shown in Table 4.

TABLE 4: LOCATION DATA FROM SEE-PHONE

Host:localhost Database : android

Generated by : phpMyAdmin 3.5.2.2/MySQL 5.5.27

No Latitude Longitude Time

1 2

3

4 5

6

7

45.75892 45.75858

45.75850

45.75846 45.75846

45.75847

45.75850

3.11157 3.11185

3.11197

3.11212 3.11231

3.11248

3.11262

17:25:16 17:25:56

17:26:10

17:26:16 17:26:26

17:26:36

17:26:49

Figure 15 shows results of tracking route on monitor

server (display). Red line on figure is route track from user

with 7 points of data location.

Fig. 15. Display of Track Route on Monitor

C. TTS and Voice Command on SEE-Phone

In this experiment, we have tested the Google voice

commands to run some functions. The experiments have

run on smart phone Samsung SIII Model GT-I9300.

The system starts to work once the user said ‘on’, until

to ‘off’ word recognition.

The ‘new’ command will create the new route when

user moves and sends the track data to database server.

The ‘stop’, finishes the application.

In the system SEES current version 2 menus (two

classes of functions) are available:

1) ‘new route’ menu, for making the new route on

database server, and

2) ‘follow route’ menu, for guiding the user

displacements based on data that already existing in

the database server.

Figure 16 below shows a display on database server

when user walks on the wrong way (track in ‘green’ line).

The database server will send the alert message to SEE-

phone through voice (audio) that be heard by user.

Fig. 16. User Walks in the Wrong Way

VI. CONCLUSION

This paper has introduced the design and

implementation of the SEE-Phone which is a part of SEE-

system, an assistive device for the VIP to improve their

orientation and walking skills.

The SEE-Phone has been technically evaluated in two

preliminary conditions, namely: 1) detection of the color

status of traffic lights; 2) conversion of traffic light data

into speech for an audio feedback to VIP, and 3)

development of the remote tracking with database server

for remote monitoring the user during its walk.

The original concept of the SEES system integrates

three main devices, iSEE, SEE-phone and SEE-stick

which complements each other. The SEES system will be

built as an Open Source platform and will integrates

several sensors. Finally, the system will be validated with

end users for its pertinence to targeted assistance and

appropriation. Its technical performances will be evaluated

and compared to these existing systems.

ACKNOWLEDGMENT

This work has been sponsored by the Indonesian and French governments, and French government research program "Investissements d'avenir" through the IMobS3 Laboratory of Excellence (ANR-10-LABX-16-01), by the

European Union through the program Regional Competitiveness and Employment 2007-2013 (ERDF–Auvergne region), and by the Auvergne region.

REFERENCES

[1]. Kamiński Ł, Stepnowski A and Demkowicz J (2011) Wearable

System Supporting Navigation of the Blind. International Journal of

Geology, Isssue 2, Volume 5, 2011, pp. 34-40.

[2]. Strumillo P (2010) Electronics Interface Aiding the Visually

Impaired in Enviromental Acces, Mobility and Navigation. The 3rd

International Conference on Human System Interaction (HSI) 2010,

May 13-15, 2010, pp. 17-24.

[3]. Brilhault A, et al (2011) Fusion and Artificial Vision and GPS to

Improve Blind Pedestrian Positioning, New Technologies, Mobility

and Security (NTMS). The 4th IFIP International Conference,

February 7-10, 2011, pp. 1-5. [4]. Pissaloux E (2009) Assistive Technologies to Support Independent

Mobility of Visually Impaired. Conference and Workshop on

Assistive Technologies for People with Vision and Hearing

Impairments, Past Successes and Futures Challenges, CVHI-2009,

M.A. Hersh (ed.) (tutorial). http://cwst.icchp.org/files/tutorialbook_tut3_part2.pdf

[5]. Pissaloux E (2011) Mobilité de Déficients Visuels : Concepts,

Modèles Computationnels et Assistances Technologiques. The 2rd Int. Conf. Sur l'Accessibilité et les Systèmes de Suppléance aux

personnes en situations de Handicap, Paris, CNRS, 17-18 January

2011, pp. 306-315

[6]. Tversky B (2005) Functional Significance of Visuo-spatial

Representations, in Shah, P., Miyake, A., "Handbook of higher-

level visuospatial thinking", Cambridge: Cambridge University

Press, 2005.

[7]. Golledge R.G., et al (1991) Designing a Personal Guidance System

to Aid Navigation without Sight: Progress on The GIS Component.

International Journal of Geographical Information Systems, Taylor

and Francis Ltd, London.

[8]. Loomis J.M., Golledge R.G. and Klatzky R.L. (1998) Navigation System for the Blind: Auditory Display Modes and Guidance.

Precense, Vol.7, No.2 April 1998, pp. 193-203.

[9]. Golledge R.G., et al (1998) A Geographical Information System for a GPS based Personal Guidance System. International Journal of

Geographical Information Science, 1998, Vol.12, No.7, pp. 727-

749.

[10]. Hersh M.A. and Johnson M.A. (2010) A Robotic Guide for Blind

People part 1: A Multi-National Survey of the Attitudes,

Requirements and Preferences of Potential End-User. Dept of

Electronics and Electrical Engineering, University of Glasgow,

Scotland.

[11]. Balakrishnan G, et al (2006) A Stereo Image Processing System for

Visually Impaired. International Journal of Information and

Communication Engineering, 2:3, 2006, pp. 136-145.

[12]. Katz B.F.G., et al (2009) NAVIG: Navigation Assisted by Artificial Vision and GNSS. The Project was supported by the French

National Research Agency (ANR).

[13]. Xu J, et al (2010) An Outdoor Navigation Aid System for the Visually Impaired. International Conference on Industrial

Engineering and Engineering Management (IEEM), December 7-

10, 2010, Macao.

[14]. Kim D.H and Ryu H.G (2011) Obstacles Recognition System using

Ultrasonic Sensor and DRFC (Duplex Radio-Frequency Camera)

for the Visually Impaired Person. International Conference on

Advanced Communication Technology (ICACT), February 13-16,

2011.

[15]. Yusro M, et al (2013) SEES: Concept and Design of a Smart

Environment Explorer Stick. The 6th International Conference on

Human System Interaction (HSI) 2013, June 6-8, 2013, Sopot,

Poland. [16]. Yibo C, et al (2011) 6LoWPAN Stacks: a Survey. The 7th Int'l

Conf. on Wireless Communications, Networking and Mobile Computing WiCOM -2011, September 23-25, 2011 Wuhan, China.

[17]. Shi HL, et al (2011) Energy Efficient and Fault Tolerant Multicore

Wireless Sensor Network: E2MWSN. The 7th Int'l Conf. on Wireless Communications, Networking and Mobile Computing

WiCOM-2011, September 23-25, 2011 Wuhan, China.

[18]. Modern GPS coordinate systems, about WGS84.

http://www.ordnancesurvey.co.uk/oswebsite/gps/information/coordi

natesystemsinfo/guidecontents/guide4.html. Accessed in January 2013.

[19]. Liao C.F, Rakauskas M and Rayankula A., Development of Mobile

Accesible Pedestrian Signals (MAPS) for Blind Pedestrians at

Signalized Intersections, Center for Transportation Studies,

Department of Civil Engineering, Minnesota Traffic Observatory,

University of Minnesota.

[20]. TTS (Text-to-Speech) Application on Smart phone.

http://searchmobilecomputing.techtarget.com/definition/text-to-

speech. Accessed in March 2013.