Teleoperacion Mobil de Un Robot Mobil

7/30/2019 Teleoperacion Mobil de Un Robot Mobil

1/109

Lappeenranta University Of TechnologyFaculty of Technology Management

Department of Information Technology

Masters Thesis

Ehab Aboudaya

Mobile Teleoperation of a Mobile Robot

Examiners: Professor Ville KyrkiProfessor Heikki Klviinen

Supervisor: Professor Ville Kyrki


2/109

ABSTRACT

Lappeenranta University of TechnologyFaculty of Technology ManagementDepartment of Information Technology

Ehab Aboudaya

Mobile Teleoperation of a Mobile Robot

Thesis for the Degree of Master of Science in Technology

2010

109 pages, 67 gures and 8 tables.

Examiners: Professor Ville KyrkiProfessor Heikki Klviinen

Keywords: Mobile Teleoperation, Mobile Devices, Nokia N770, Maemo 2.2, ARIA,Graphical User Interface.

This thesis describes the design and implementation of a graphical application on amobile device to teleoperate a mobile robot. The department of information tech-nology in Lappeenranta University conducts research in robotics, and the main mo-tivation was to extend the available teleoperation applications for mobile devices.

The challenge was to port existing robot software library onto an embedded de-vice platform, then develop a suitable user interface application that provides suf-cient functionality to perform teleoperation tasks over a wireless communicationnetwork.

This thesis involved investigating previous teleoperation applications and conductedsimilar experiments to test and evaluate the designed application for functional ac-tivity and measure performance on a mobile device which have been identied andachieved.

The implemented solution offered good results for navigation purposes particularlyfor teleoperating a visible robot and suggests solutions for exploration when noenvironment map for the operator is present.


3/109

PREFACE

It has been a long experience preparing this project, working and studying does notalways meet but I thank Allah, then all my family back home in Libya for theirsupport and my wife Gamira for living this experience with me.

I wish to thank my supervisor for his patience and guidance, Mr Janne Laaksonenfor his lab help and time spent testing this project with me. I also would like towarmly thank Ms Susanna Koponen for her assistance and rescue in ofcial matters.

I hope this material can be useful for someone in the near future.

Tampere, September 1st, 2010

Ehab Aboudaya


4/109

4

Contents

1 INTRODUCTION 15

1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.2 Objective and Scope . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.3 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . 16

2 FUNDAMENTAL CONCEPTS 17

2.1 Mobile Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.2 Teleoperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.3 Mobile Teleoperation . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.4 Mobile Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.5 Data Communication . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.6 Graphical User Interfaces . . . . . . . . . . . . . . . . . . . . . . . 28

3 RELATED WORK 31

3.1 Stationary Teleoperation . . . . . . . . . . . . . . . . . . . . . . . 33

3.1.1 Desktop Client Interfaces . . . . . . . . . . . . . . . . . . . 34

3.1.2 Web-Based Interfaces . . . . . . . . . . . . . . . . . . . . 38

3.2 Mobile Teleoperation . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4 REQUIREMENTS AND SPECIFICATION 61


5/109

5

4.1 Overall Description . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.2 Functional Requirements . . . . . . . . . . . . . . . . . . . . . . . 61

4.3 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.4 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

4.5 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

4.6 Teleoperating Environment . . . . . . . . . . . . . . . . . . . . . . 63

4.7 Dependencies and Scope . . . . . . . . . . . . . . . . . . . . . . . 64

4.8 Design and Implementation Constraints . . . . . . . . . . . . . . . 66

4.9 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.9.1 Connection Settings . . . . . . . . . . . . . . . . . . . . . 69

4.9.2 Connect To ARIA Server . . . . . . . . . . . . . . . . . . . 70

4.9.3 Disconnect from ARIA Server . . . . . . . . . . . . . . . . 71

4.9.4 Command . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.9.5 Stop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.9.6 Get Image . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.9.7 Move . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.9.8 Get Robot Status . . . . . . . . . . . . . . . . . . . . . . . 77

4.9.9 Get Environment Map . . . . . . . . . . . . . . . . . . . . 78

4.10 Non-Functional Requirements . . . . . . . . . . . . . . . . . . . . 79

4.10.1 Performance and Quality Requirements . . . . . . . . . . . 79


6/109

6

4.10.2 Safety Requirements . . . . . . . . . . . . . . . . . . . . . 80

5 IMPLEMENTATION 81

5.1 UI Wire Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5.2 Deployment Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.3 Architectural Design . . . . . . . . . . . . . . . . . . . . . . . . . 86

5.4 Class Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.5 Sequence Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . 88

5.6 Teleoperation Command Buttons . . . . . . . . . . . . . . . . . . . 91

5.7 Development Tools . . . . . . . . . . . . . . . . . . . . . . . . . . 91

5.8 Interface Screen Shots . . . . . . . . . . . . . . . . . . . . . . . . 93

5.8.1 RTMU Startup and Idle State . . . . . . . . . . . . . . . . . 93

5.8.2 RTMU Menu And Settings Dialog . . . . . . . . . . . . . . 93

5.8.3 Teleoperation With Environment Map . . . . . . . . . . . . 94

5.8.4 Teleoperation With No Map . . . . . . . . . . . . . . . . . 94

5.8.5 RTMU Hidden Toolbar . . . . . . . . . . . . . . . . . . . . 95

5.8.6 RTMU Ratio Increase / Decrease Indicator . . . . . . . . . 95

5.8.7 RTMU Robot Image Feedback . . . . . . . . . . . . . . . . 96

5.9 Installation and Help system . . . . . . . . . . . . . . . . . . . . . 96

6 TESTS AND RESULTS 97

6.1 Functional Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 97


7/109


8/109

8

List of Figures

1 Pioneer 3-DX Mobile Robot . . . . . . . . . . . . . . . . . . . . . 17

2 Typical ARIA application structure . . . . . . . . . . . . . . . . . . 19

3 Human operator for mobile robots . . . . . . . . . . . . . . . . . . 20

4 Teleoperation system architecture sample . . . . . . . . . . . . . . 21

5 Remote driving with PdaDriver . . . . . . . . . . . . . . . . . . . . 22

6 Maemo N770, 2005 Nokia . . . . . . . . . . . . . . . . . . . . . . 24

7 Maemo platform Key Components . . . . . . . . . . . . . . . . . . 24

8 Minimum WLAN setup for intended teleoperation communication. . 26

9 Sample of WLAN connectivity conguration on the N770. . . . . . 27

10 MSC for client/server communication . . . . . . . . . . . . . . . . 27

11 Different dialog GUI but same functionality . . . . . . . . . . . . . 28

12 MaemoPad GUI application . . . . . . . . . . . . . . . . . . . . . 29

13 MVC Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

14 Novel UI for 3D map Exploration . . . . . . . . . . . . . . . . . . 31

15 Virtual joystick gestures . . . . . . . . . . . . . . . . . . . . . . . 32

16 Fixed Location Operator Teleoperation . . . . . . . . . . . . . . . . 33

17 RobotUI Desktop client UI . . . . . . . . . . . . . . . . . . . . . . 34

18 RobotUI, GUI Architecture . . . . . . . . . . . . . . . . . . . . . . 35

19 The Advantage of Mobility - Desktop GUI design . . . . . . . . . . 36


9/109

9

20 KhepOnTheWeb Web Interface . . . . . . . . . . . . . . . . . . . . 38

21 WebDriver system architecture . . . . . . . . . . . . . . . . . . . . 39

22 WebDriver user interface . . . . . . . . . . . . . . . . . . . . . . . 40

23 WebDriver map sample . . . . . . . . . . . . . . . . . . . . . . . . 40

24 WebDriver ImageManager . . . . . . . . . . . . . . . . . . . . . . 41

25 SWATS - Web Services Teleoperation . . . . . . . . . . . . . . . . 42

26 Java Based Teleoperation System . . . . . . . . . . . . . . . . . . . 43

27 Java Based Teleoperation Web UI . . . . . . . . . . . . . . . . . . 43

28 Finger gesture interface . . . . . . . . . . . . . . . . . . . . . . . . 45

29 PdaDriver: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

30 PdaDriver System Architecture . . . . . . . . . . . . . . . . . . . . 46

31 PdaDriver Screen modes . . . . . . . . . . . . . . . . . . . . . . . 48

32 PdaDriver Vision Screen . . . . . . . . . . . . . . . . . . . . . . . 49

33 PdaDriver Sensor Modes . . . . . . . . . . . . . . . . . . . . . . . 49

34 PdaDriver Combined Screen Mode . . . . . . . . . . . . . . . . . . 50

35 The Advantage of Mobility - PDA Interface . . . . . . . . . . . . . 52

36 The Advantage of Mobility - Simulated map teleoperation . . . . . 53

37 The Advantage of Mobility - Visibility of Robot . . . . . . . . . . . 54

38 The Advantage of Mobility - PDA and Desktop covered area . . . . 55

39 The Advantage of Mobility -Mean and Standard Deviation Visibil-ity Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55


10/109

10

40 Teleoperation Common Tasks and UI elements. . . . . . . . . . . . 58

41 RTMU overall system entities. . . . . . . . . . . . . . . . . . . . . 61

42 P3DX with laptop . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

43 Teleoperating Environment Map . . . . . . . . . . . . . . . . . . . 64

44 Screen shot of MobileSim robot simulator. . . . . . . . . . . . . . . 65

45 Operator manages robot hardware and ARIA server software. . . . . 67

46 RTMU application use cases. . . . . . . . . . . . . . . . . . . . . . 68

47 RTMU Command Use Case Sequence Diagram. . . . . . . . . . . . 73

48 Hildon Application Views . . . . . . . . . . . . . . . . . . . . . . 82

49 Single Toolbar, 360 or 420 pixels . . . . . . . . . . . . . . . . . . . 82

50 RTMU Menu Items. . . . . . . . . . . . . . . . . . . . . . . . . . . 83

51 RTMU Settings Dialog. . . . . . . . . . . . . . . . . . . . . . . . . 83

52 RTMU Auxiliarly Popups . . . . . . . . . . . . . . . . . . . . . . 84

53 RTMU Deployment Diagram. . . . . . . . . . . . . . . . . . . . . 85

54 RTMU Class View. . . . . . . . . . . . . . . . . . . . . . . . . . . 86

55 Connect SD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

56 Move or Stop Command . . . . . . . . . . . . . . . . . . . . . . . 89

57 Request Robot Image . . . . . . . . . . . . . . . . . . . . . . . . . 90

58 RTMU Startup and Idle State . . . . . . . . . . . . . . . . . . . . . 93

59 Menu And Settings Dialog . . . . . . . . . . . . . . . . . . . . . . 93


11/109

11

60 Connected to ARIA Server with map . . . . . . . . . . . . . . . . . 94

61 Connected to ARIA server with no map . . . . . . . . . . . . . . . 94

62 RTMU connected to ARIA Server with map and hidden Toolbar. . . 95

63 RTMU connected to ARIA Server and velocity banner visible. . . . 95

64 RTMU connected to ARIA Server and Requested Image. . . . . . . 96

65 Teleoperation Test Arena . . . . . . . . . . . . . . . . . . . . . . . 99

66 Robot Home Position . . . . . . . . . . . . . . . . . . . . . . . . . 99

67 Robot Goal Position . . . . . . . . . . . . . . . . . . . . . . . . . . 100


12/109

12

List of Tables

1 PDA-Based Human-Robotic trail and task times . . . . . . . . . . . 51

2 PDA-Based Human-Robotic goal achievement . . . . . . . . . . . . 51

3 PDA-Based Human-Robotic Best Performance . . . . . . . . . . . 56

4 Advantages and Disadvantages of teleoperation using mobile devices. 57

5 Recommendations for Mobile Teleoperation and GUI Components. 60

6 RTMU Class Descriptions. . . . . . . . . . . . . . . . . . . . . . . 87

7 RTMU physical Nokia N770 command buttons . . . . . . . . . . . 91

8 Functional Testing Results . . . . . . . . . . . . . . . . . . . . . . 98


13/109

13

List of Abbreviations

AI Articial intelligence

ARIA Advanced Robot Interface for Applications

ARMEL Advanced RISC Machine, RISC architecture distribution

CCD charge-coupled device

DLL Dynamic-link library

GPS Global Positioning System

GUI Graphical User Interface

HTML HyperText Markup Language

HW Hardware

IO Input Output

IT Information Technology

JPEG Joint Photographic Experts Group

LAN Local Area Network

LUT Lappeenranta University of Technicology

MSC Message Sequence Chart

MVC Model View Controller

OS Operating SystemP3-DX Pioneer 3-DX

PC Personal Computer

PDA Personal Digital Assistant

RTMU Robot Teleoperation Maemo User Interface

SA Situational Awareness

SD Sequence Diagrams

SDK Software Development Kit


14/109

14

SW Software

TCP Transmission Control Protocol

UI User Interface

USB Universal Serial Bus

WLAN Wireless Local Area Network


15/109

15

1 INTRODUCTION

1.1 Background

In major industry elds such as oil production, power plants and health sectors tele-operation is practiced regularly. This practice typically involves a human operatorusing a specic interface to carry out tasks on remote locations, the distance be-tween the human operator and remote machine can vary and can stretch as far asanother plant.

The interface is expected to provide sufcient controls and enough feedback infor-mation to the operator in order to complete the assigned task efciently, and byadding autonomous features on remote machines like a mechanism to avoid obsta-cles or use solar power if present etc .. it will enhance the teleoperation experiencebut the main research topic remains the human operator interface.

With current technology advances in wireless communication and mobile devices itis benecial to research the possibility of utilizing those components for teleopera-

tion applications. In addition to the importance of the topic, the Department of In-formation Technology in Lappeenranta University Of Technology robotic researchis active and encourages development in this area.

1.2 Objective and Scope

The objective of this thesis is to design and implement a graphical application ona mobile device to wirelessly teleoperate a mobile robot. The application interfacemust be able to perform the following tasks in real time:

Connect and disconnect to/from the remote robot.

Display the robots environment map and plot its locations.

Move the robot in four directions forward, backward, left and right.

Display robot feedback information.

Request and display an image from the robots mounted camera.


16/109

16

The application will be developed on the Nokia N770 mobile device and use thePioneer P3-DX as the robot hardware, those can be requested and borrowed from

the IT departments laboratory. In the next chapter and in chapter 4 more applicationrelated details are specied.

1.3 Structure of the thesis

This thesis contains 8 chapters:

Chapter 2 presents fundamental concepts by discussing briey teleoperation andmobile device platforms, WLAN ( Wireless Local Area network ), software imple-mentation practices, then concludes with a section for GUI interfaces on mobiledevices. Chapter 3 discusses previous studies on similar topic and concludes with ageneral list of desired functionalities that will be considered in thesis project work.

In Chapter 4 requirements are identied and use cases are described where such in-formation is used as a bases for the project development phase. Chapter 5 explainsarchitecture and implementation design including GUI design proposal. Then onChapter 6 the implementation is tested with functional and teleoperation experi-ments, remaining sections shows results and discusses result ndings. Chapter 7concludes the study outcome with some experiences and nally suggests possiblefurther work. Chapter 8 is reserved for references.


17/109

17

2 FUNDAMENTAL CONCEPTS

2.1 Mobile Robots

A mobile robot is a machine that has the ability to move within an environment andcan perform certain operations to accomplish a specic task. Research in this eldbegan after the introduction of radio communication where the rst master-slavemanipulator appeared in 1948 by Ray Goertz, U.S. Atomic Energy Commission.

Mobile robots may be classied according to the following features [ 1]:

Environmental usage on land, aerial or underwater robots.

Locomotive devices with legs, on tracks or using wheels for movement.

A popular mobile robot produced by MobileRobots Inc. shown in Fig. 1 is used foracademic purposes, this particular model is useful for research elds that involve

mapping, teleoperation and robot localization studies [ 2].

Figure 1 . Pioneer 3-DX Mobile Robot [ 3].


18/109

18

The hardware specication can be summarized in the following list [ 2] :

Weight 9kg, Payload 25kg; Dimensions: Length 45cm, Width 40cm, Height24cm.

Run time with on board PC 3-4 hrs, using 12V batteries with recharge timeof 6 hrs.

Mobility wheels consists of 2 foam-lled wheels and 1 rear caster for balance.

Max speeds of 1.6 meters per second, Translate in 1,400 mm/sec, rotate of

300 deg/sec. Sensors on front and back using ultrasonic sonars and 500-tick wheel en-

coders.

Controls and Ports through Serial, Auxiliaries, Motors and Microcontrollerand joydrive.

This thesis mobile robot hardware will be based on the P3-DX (Pioneer 3-DX)model and extensive work is applied using ARIA (Advanced Robot Interface forApplications) software library. ARIA comes with every MobileRobots product, itis an open-source software development kit based on C++ programming language.

Developing robot application with ARIA is a straight forward practice due to itsclient/server methodology, in Fig. 2 the main classes infrastructure are illustrated.In addition to a supported simulator that matches the same functional behaviour asthe real robot hardware.

The IT department of LUT ( Lappeenranta University of Technicology) providesthe robot hardware and robot server software used for this thesis project, thus focusand objectives are related to the client end of teleoperation.


19/109

19

Figure 2 . Typical ARIA application structure [4].


20/109

20

2.2 Teleoperation

Teleoperation indicates operation of a machine at a distance [ 5], and it is mostlyassociated with mobile robots performing a certain task. If such a device has theability to perform autonomous work, it is called a telerobot [ 6].

Teleoperation systems can be considered as a master/slave relationship where a hu-man operator executes a task from a remote environment simplied in Fig. 3.

Figure 3 . Human operator for mobile robots [ 7].

Teleoperation systems can be composed of a basic textual input/output terminalconnected with a joystick for robot navigation. The success factor depends highly

on their control system, additional environment feedback information enhances theoperators decision making, therefore interface research is necessary.

Safe navigation can be realized using haptic interfaces. A study [ 8] for measuringforce feedback information from a special haptic joystick resulted in less obstaclecollisions. The joystick was used to control velocity of a minirover by applyingforce and pressure. In addition to other sensory calculations it demonstrated betternavigation and improved remote control performances results.

Another example of a haptic device is a touch screen, where tactile feedback tech-nology creates the perception of pressing physical buttons [ 9] on the screen surface.Nokia corporation in 2005 released an internet tablet model N700 using similartechnology called Haptikos [ 10]. This particular model is an ideal candidate formobile teleoperation and was selected for this thesis mobile operator device. Whilerobots can operate autonomously an operator will require an intelligent interfaceless for control and more for monitoring and diagnosis [ 11].

Teleoperation systems share a common architecture design depicted in Fig. 4 andsimilarly most operator application interfaces are found to offer robot movementcontrols, execute a dened function e.g. pick up a box, including sending feedback


21/109

21

of specic environment data e.g. temperatures. While extensive studies on localiza-tion has been carried out, this thesis will focus on the designs of teleoperation user

interfaces.

Figure 4 . Teleoperation system architecture sample [ 12].

2.3 Mobile Teleoperation

Teleoperation is inherently remote and while xed location control centers can serve

the main purpose, developing a mobile system will increase SA (situational aware-ness) for the operator, offer easy deployment, eliminate installation costs, increaseoperation communication area, and by designing intuitive interface training time isreduced signicantly.

To address the need of mobility, handheld devices are used for the following advan-tages:

Portable, lightweight and affordable.

Easy to operate and extend.

Availability of haptic screen and keys designs.

Wireless communication ready.

Offer extra communication services such as GPS (Global Positioning Sys-tem).

Variety of operating systems and multitasking support.

Wide range of development platforms available.


22/109

22

At the same time teleoperation on handheld devices may not be suitable for thattasks that require extensive computing calculations, wide screen display and large

memory usage, in addition to prolonged power usage.

In a study of mobile teleoperation [ 13] a PDA (Personal Digital Assistant) was usedfor remote driving a vehicle. The system was called PdaDriver and uses multiplecontrol modes to make vehicle teleoperation fast and efcient Fig. 5.

Figure 5 . Compaq iPAQ PocketPC and remote driving with PdaDriver [13].

Mobile handhelds are the most attractive devices for mobile teleoperation and in thenext section we will introduce the Nokia internet tablet N770.

2.4 Mobile Devices

Most mobile devices are closely related to wireless technology, cell phones are agood example of mobile devices. In this thesis we aim to teleoperate a mobile roboton a mobile device which should essentially meet the following criteria:

The teleoperating hardware is a mobile device or a smartphone.

The mobile device can run the P3-DX robot software library.

Wireless communicate network must be used between the mobile device androbot.


23/109

23

C/C++ programming language will be the development language.

The mobile device should have appropriate screen size to display a map.

The mobile device battery power can support teleoperation for at least 15mins.

Hardware button keys on the mobile device provide good user control andshould be easily programmable on the mobile device.

The mobile device is available in markets and can be borrowed from the ITdepartment lab.

Power saving modes can be switched on or off on the mobile device.

The Nokia N770 internet tablet in Fig. 6 has been selected for this thesis mobiledevice because it meets all of the above conditions. The Nokia N770 is an elec-tronic mobile device that can access internet through WLAN/Wi-Fi or Bluetoothconnections [ 14] and its main specication can be described as follows:

Price : Starting from $250 US dollars [ 15].

Physical Properties : weight 230 g , size dimensions 141 x 79 x 19 mm [ 16].

Display : High-resolution (800x480) or 4.13 inches touch screen with up to65,536 colours.

Processor & Memory : TI OMAP, 244.1 MHz, 64 MB, ash memory RS-MMC.

Operating System : Internet Tablet 2006 edition, Maemo OS (Operating Sys-tem) 2.2.

Networking : WLAN (IEEE 802.11b/g).

Battery Life : 1500mAh, 7 days standby time, 3 hours browsing time.

Other : support for JPEG, USB device mode for PC connectivity, bundledwith multiple useful applications, Bluetooth is also available [ 17].

What makes the N770 attractive is its use of haptic screen keyboard and programmablehardware keys such as directional scroll keys, zoom key, escape and menu keys.


24/109

24

Figure 6 . Maemo N770, 2005 Nokia [ 18].

One can think of the scroll keys as a 2D virtual joystick, the scroll keys make a

clicking sound when the operator keeps pressing thus, giving the impression of pressure feedback.

The N770 offers Maemo as the operating system, Maemo is mostly based on opensource software. At the kernel level it is Debian Linux distribution which not onlysuitable for network development but also for rebuilding available libraries fromscratch. Fig .7 illustrates the key architectural components of the N770 Maemoplatform.

Figure 7 . Key Components of the a Maemo platform [19].

Linux based device was another key factor for selecting the N770 and the followingitems describe structure layer roles from Fig. 7 [19]:

Linux kernel is the core and heart of Maemo platform that controls and facil-itates usage of hardware such as memory, process, networking, le and other


25/109

25

management services.

The ARM/OMAP-based Linux kernel implements hardware-specic deviceand bus drivers.

Debian Package Management: Follows Debian system design.

System Libraries: Standard GNU C library for various development uses.

System Services: Provides system-wide services to applications and end userssuch as SQLite database library interface.

Maemo Launcher: Specialized in launching all Maemo applications, it pro-

vides initializations and startup data.

GTK+: A framework of C libraries responsible for graphical interface de-signs, it is based on call back events which triggers execution of specicfunctions.

Hildon UI Framework: Provides the X Windows System using GTK+ UI(User Interface) framework developed in the GNOME project. Hildon addsmore components such as control panel, status bar and home applets etc.

Maemo SDK: The SDK (Software Development Kit) Contains tools neededfor application development, C++ can be used with a cross compilation en-vironment called Scratchbox. Scratchbox is used to solve host and targetenvironment compilation problems by isolating them.

2.5 Data Communication

Communication between the operator and robot will be established by WLAN(Wireless Local Area Network). The N770 already comes equipped with a WLAN/Wi-Fi embedded adapter, it is required to connect the device on a network access pointfor operation. This task is a straight forward procedure and very transparent to theend operator.

WLAN is a standard and compliant with the IEEE 802.11b or 802.11g specication.Data exchange rate can reach up to 54 Mbits/s, large enough to stream continuousvideo and heavy network communication. An access point is usually a router usedas a bridge [20] to a wired system or can provide a standalone private network. Theminimum data communication setup needed is illustrated in Fig. 8.


26/109

26

Figure 8 . Minimum WLAN setup for intended teleoperation communication.

Indoor connectivity coverage can expands to few tens of meters and extension ispossible by adding an extra WLAN access points. This gives the operators moremobility to move around and remain connected to the network within a broad cov-erage area [21].

Network security can be established using encryption systems such as WPA (Wi-FiProtected Access), when joining a secure wireless network a key is expected to beentered and upon verifying the router grants a distinct IP network for the device. Forthis thesis work the mobile device and robot are joined in a wireless private network with security turned off. Fig. 9 shows the interface for connecting the N770 devicewith an available WLAN ( Wireless Local Area Network ) access point.

Data communication for teleoperation will follows the same protocol states of anyclient/server application depicted in Fig. 10 sample MSC ( Message SequenceChart ). This chart captures a scenario in which a user (U) send a request to an

interface (I) to gain access to a resource R. The interface in turn sends a requestto the resource, and receives "grant" as a response, after which it sends "yes" to U[22].

ArNetworking ( MobileRobots Advanced Robotics Networking Infrastructure ) isused to add networking services to a robot control program. The client requests aservice from the server in a form of a command or the client can request informationfeedback at a specied rate [ 4]. The command is executed on the robot hardwareto accomplish its dened task, then the outcome can be sent back to the requestingclient.


27/109

27

Figure 9 . Sample of WLAN connectivity conguration on the N770.

ArNetworking interface library is the main building block for communication be-tween the client application used by the operator and the robots server. In the im-plementation chapter of this thesis more details will be identied.

Figure 10 . Typical MSC chart for client/server communication [ 22].


28/109

28

2.6 Graphical User Interfaces

Software applications represent information in different forms ranging from simpletext outputs to sophisticated 3D graphs, this representation is encapsulated withina form called an interface. GUI (Graphical User Interface) are the most commonfeatures of todays applications running on common computing devices.

Software applications are expected not only to meet functional requirements butalso offer visual coherency with simple command layouts easy enough for an av-erage user to get accustomed to. Major software vendors start the GUI interface

trend which is picked up and used by other developers becomes very common andeventually some features become standards of next GUI interfaces application gen-erations.

Fig. 2.6 shows two different GUI designs with the same functionality. The differ-ence is how they are arranged, where the right window groups properties, on the leftwindow properties are layed out horizontally, it is harder and becomes increasinglycomplex to maintain. Thus, designing a GUI should aim at simplicity, grouping,usability and consistencys.

Figure 11 . Same functional dialog but different GUI design [ 23].

Designing interfaces on mobile devices is effort demanding due to the limited re-sources of display size, communication costs and power consumption. User interac-tion must also be kept to a minimum and for this reason some mobile GUI designssuch as in Fig. 12 for MaemoPad the keyboard can be popped out or hidden whennot needed.

Successful software development can be practiced using design patterns such asMVC pattern ( Model View Controller ) simplied in Fig. 13. The controller is the


29/109

29

Figure 12 . Sample of a Maemo application with virtual keyboard [ 19].

middle layer where user actions received from the viewer are passed to the model forprocessing, then model outcome is read by the controller via callbacks and is nallyreected on to the view. This type of pattern is very useful for different types of platforms and because details are separated MVC pattern offers an efcient methodof development. Most modern mobile platforms consists of a standard developmentinterface framework but it is also necessary to investigate if it includes the required

components.

Figure 13 . MVC Pattern

On the Nokia N770 Maemo platform the desktop interface is based on the Hildonapplication framework built using GTK+. The Hildon GTK+ is more suitable forembedded devices interface design as in Fig. 12, it is also binary-compatible with a


30/109

30

normal GTK+ applications without the native Maemo look [ 24].

GTK+ is a multi-platform toolkit for creating graphical user interfaces. It offersvery complete set of widgets to use when building applications, GTK+ was initiallydeveloped for and used by The GIMP, The GNU Image Manipulation Program.Today GTK+ is used by a large number of applications, and is the toolkit used bythe GNU projects GNOME desktop [ 25].

In the next chapter we shall review previous studies related to teleoperation GUIdesigns, and Hidlon GTK+ will be selected for the thesis project development plat-form.


31/109

31

3 RELATED WORK

This chapter contains a summary of several studies related to teleoperation systemswith supervisory graphical interface and controls. The selections are based on appli-cations designed for desktop PCs (Personal Computers) and mobile devices. Effortis focused to cover common interface elements, shared architectural concepts andtest scenarios for system evaluation. The chapter concludes with a set of guidelinesand recommendations to aid the design in this thesis.

The P3-DX robot used for this thesis is basically a vehicle robot Fong et al. identi-

ed the following teleoperation interfaces [ 26]:

Direct manual interfaces: the user is presented with a direct feedback fromthe camera and uses a joystick for robot locomotion.

Multimodal/-sensor interfaces: the user is presented with multiple feedbacksin a single view.

Supervisory control interfaces: the user issues high level commands and mon-

itors the remote environment. Novel interfaces: the user can teleoperate using haptic gestures or a noval

state-of-the-art future technology.

An interesting interface category would be the novel interface designs because thedesign can be mixed with new concepts for example, Fig. 14 adopted from [ 26], [27]illustrates teleoperation in a virtual 3D world that demonstrated improved results

compared to a 2D interface.

Figure 14 . 3D map Exploration, [ 27], 2006, novel interface example.


32/109

32

In Fig. 15 the operator is using gestures to drive a robot, the operator is acting moreof a virtual joystick where the left hand used to activate the gesture system and right

hand to specify direction and magnitude [ 28]. Due to complexity of such systemsand limited system resources on mobile devices, direct and sensor interfaces areconsidered for this project GUI design.

Figure 15 . Visual gesturing for vehicle teleoperation [ 28], 2000


33/109

33

3.1 Stationary Teleoperation

Stationary teleoperation are systems designed for operators in a xed location thattypically reside in a room or at a control center like the one shown in Fig. 16. Theoperator is using a realtime application with a set of control tools and some feedback information to carry out teleoperation tasks. It is necessary to investigate how suchsystem were designed for the following reasons:

1. Collect common commands and the necessary common user controls for tele-operation systems.

2. Recognize the types of GUI elements used to successfully represent feedback information.

3. Tests and experiments conducted to evaluate the designed systems.

4. Identify perhaps shortcomings when porting existing systems onto mobiledevices platforms.

Figure 16 . Standard and Wireless teleoperation systems [ 29].


34/109

34

3.1.1 Desktop Client Interfaces

Graphical Teleoperation Interface For A Mobile Robot , Jukka Turunen 2006[30], developed a GUI application in Fig. 17 to teleoperate a P3-DX robot, theinterface was based on a property software called MobileEyes by MobileRobotsInc., the implementation architectural design followed a MVC pattern where hepresented robot sonar and map using GTK and OpenGl libraries implemented on aLinux based operating system.

Figure 17 . RobotUI [30].

The interface is composed of a rich set of feedback information using map drawingof the remote environment, video images, specic robot data such as velocity andbattery level. The user can move the robot using directional buttons and rotatecamera angles. Sample of the user interface architecture can be seen in Fig. 18.

Among other features included is the ability to send text to the robot server and a

use a pre-installed synthesizer called ArFestival-speech that offers text to speechconversion. The communication between RobotUI and the robot was done usingLAN (Local Area Network) or WLAN networks.


35/109

35

Figure 18 . RobotUI , GUI Architecture [ 30].

The application functionality outcome performed very well passing 11 out of 15tests, the failed or could not be tested cases were results of video technical limita-tions, requesting safe driving from ARIA library was not taking affect, and sonarmeasurements had to be estimated for plotting on the map.

The study concluded with successful teleoperation matching the quality of Mo-bileEyes application, in addition to the usability of synthesizer and communication

on WLAN network. The author also mentions that MVC pattern architecture wasbenecial and aids in further development.

A nal recommendation was to update the MVC interface for better observationsand suggested map interface improvement [ 30]. This particular study served as thebase reference for implementing this thesis project because the same ARIA robotserver from LUT IT department laboratory will be used for thesis project.


36/109

36

The Advantage of Mobility: Mobile Tele-operation for Mobile Robots [31]2001, this study is presented in two parts, in this section the desktop interface is

discussed then the later is available on the mobile teleoperation section. The inter-esting aspect of this study is it compares two interfaces PDA and desktop againstteleoperation effectiveness. The study also considered the optimal task distributionbetween a hand-held and a desktop stationary operations [ 31].

The authors suggest that using a PDA interface can boost teleoperation effectivenessfor situations that cannot be xed to a particular location. Another advantage forusing mobile devices is that they offer on-eld information and partial view of theactual robot environment not retrievable by the robot sensors.

The desktop interface in Fig. 19, was designed for tasks related to exploration, nav-igation and mapping issues, it can also support a team of robots by the offeredswitching mechanism.

Figure 19 . The Advantage of Mobility - Desktop GUI design [ 31].


37/109

37

The interface windows is divided into different panels customized for Navigation,Autonomy Levels, Robot Tools and Settings described as follows [ 31]:

Navigation panel provides a comprehensive zone view by displaying localand global maps. The map is constructed by using laser range sensor andSLAM (Simultaneous Localization and Mapping) odometry. An additional3D viewer is added to give egocentric perspective of the scenario.

Autonomy levels panel is a mode switcher between safe and non-safe tele-operation, having safe mode on can prevent colliding into obstacles. Shared

control "operator clicks robot destination on map" and autonomy "wandermode" each with different velocities (slow, normal and speedy) that have dif-ferent heuristics to explore the environment.

Robot tools panel is for robot kinematics readouts. There is a speedometer,a chronometer to keep track of the mission length, and a gyroscope directlyembedded in the robot within the 3D View (yellow directional arrow).

Settings Panel is divided into two views, consists of the Interface Settings andthe Robot Settings.

Several experiments were conducted on exploration and navigation using differentvariations of locations and situations. The study observed independent variables of interface effectiveness and operation time measurements.

The results demonstrated that it is more useful to drive the robot in narrow spacesusing laser view navigation than using the constructed map alone. Additional data

analysis and results will be described in the mobile teleoperation section of samestudy.


38/109

38

3.1.2 Web-Based Interfaces

Web based teleoperation started back in 1994 [ 32], [33], and Olivier Michel et al. in1997 introduced KhepOnTheWeb [ 34] web based teleoperation system. The studymotivation was due to the recognized benet of a single application utilized for mul-tiple remote operators. In addition to the similar client/server architectural design.Fig. 20 shows the KhepOnTheWeb interface and operates in this manner:

Robot state is represented in a static HTML (HyperText Markup Language)document with a still image and numerical data.

The operator enters and sends unsupervised commands.

The robot executes the commands without user supervision or additional in-teraction.

Command outcome is sent back with generated result after some delay.

Figure 20 . KhepOnTheWeb Web Interface[ 34].

The results indicated that it was difcult to introduce complex systems, web latencycaused delay in completion of teleoperations tasks. On the positive side web in-teractivity was successful and by introducing Java technology to handle responses


39/109

39

delays were reduced. The suggested further work described adding robot actionmonitoring tools that can increase web teleoperation effectives.

Effective Vehicle Teleoperation on the World Wide Web , WebDriver [ 11] 2001,was designed for safe and reliable web based vehicle teleoperation. Targeted forlow bandwidth usage and the interface supported both novice and experts users.The system design is depicted in Fig. 21 which consisted of 3 elements:

Figure 21 . WebDriver system architecture [ 11].

User Interface required browser support for Java applets, where the user canissue commands and able to monitor continuos feedback from the robots

sensors.

Base Station is responsible for user communication with the interface, imageprocessing, and high-level robot controls.

Robot was a teleoperated vehicle equipped with on-board sensors (includinga pan-tilt camera) and a motion controller. It is connected to the base stationvia a radio modem and analog video transmitter [ 11].

The user interface like shown in Fig. 22, served two primary tools, the dynamic mapand the image manager allowed the user to generate commands and to receive feed-back. The interface controls can manipulate the display and pan angle the robots


40/109

40

on-board camera. An added proximity light indicated distance between the robotand nearby obstacles [ 11].

Figure 22 . WebDriver user interface [ 11]

The dynamic map in shown in Fig. 23 is constructed using ultrasonic sonar androbot position. The sensor data is ltered, stored and then displayed as coloredpoints: gray for sensed obstacles and red for robot position. Color points reectcondence value where the darker tones indicate higher condence.

Figure 23 . WebDriver map sample [ 11]

The design displayed status and supported touch commands, on the map imageswere stored as (blue circles) by the image manager [ 11] that can be recalled later.


41/109

41

This reduces communication and unnecessary processing. The image manager dis-played in Fig. 24 is also touch responsive and can send images at regular rate.

Figure 24 . WebDriver ImageManager [ 11].

The base station functionality compresses data and exchanged communication withthe user interface. The image server returns images in JPEG format via TCP (Trans-mission Control Protocol). High-level robot control was established by Saphiraplatform built on top of ARIA SDK by MobileRobots Inc.

The study indicate that the architecture was reliable and robust particularly usingARIA safe driving mode, data communication was compressed and some infor-mation was pre-processed which reduced network usage. Operators explored dif-ferent rooms with ease and camera images helped navigation along the corridors.Although limitations were related to map plotting because obstacle detection waslimited to small amount of proximity sensors (ultrasonic sonar) even by addingmore ultrasonic sonar sensors may not solve this limitation due to added noise and

imprecise readouts.

The study mentions future work can be put in adding higher level of autonomy,proximity sensors may be increased to avoid obstacles and use radio beacons orcarrier-phased GPS for better map positioning. The conclusion stated that success-ful and robust teleoperation results were obtained and the feedback GUI design wasplausible and efcient.


42/109

42

Automated Teleoperation of Web-Based Devices Using Semantic Web Services ,SWATS [ 35] 2005, is a design proposal that employs Semantic Web Services tech-

nology and AI (Articial intelligence) planning techniques to achieve task-orientedautomated teleoperation for Web-based devices.

The implementation addition to typical teleoperation architectures was a semanticlayer shown in Fig. 25 and by encoding useful tasks into web ontology languagesis expected to provide the operator agents an automatic plan for process execution,thus reducing laborous work and increase automaticity. The solution presented wasa step forward for adding local intelligence on remote ends.

Figure 25 . Architecture of SWATS Web services Teleoperation [ 35].

Internet-based Robotic Systems for Teleoperation , [36] 2001, aimed to design anintuitive user interface for inexperienced users and carry out fundamental researchon multi-agent/multi-robot cooperation. The study motivation listed multiple com-

mercial benets gained using multi-agent/multi-robot cooperation in elds of tele-manufacturing, tele-training, tele-repairing, remote surveillance, distributed servicerobots for ofce, hospital and elderly care research value can be realized.

The architecture in presented in Fig. 26 and the interface in Fig. 27 exhibits itsmuch usage of Java web interface components. Experiments and results summa-rized several observations that can be considered for further improvements that isby increasing image quality, adding more autonomy for complex environments andreducing frequent network transmission delays.


43/109

43

Figure 26 . System Structure for Java Based Teleoperation [36].

Figure 27 . Interface sample for Java web based teleoperation [ 36].


44/109

44

The following list summarizes characteristics of web based interfaces:

Internet web applications can operate with high expectancy of communicationdelay, this limitation prompts to use minimum and effective packet exchange.Data loss is also expected thus recovery methods are required [ 36].

Web GUI elements are somewhat limited compared to desktop vast range of available widgets which suggests more UI design consideration.

The nature of the web is stateless and servers are responsible for maintain-ing session information among other localization and other autonomous al-gorithms should not impact performance.

Web based applications are platform independent and can be exposed to awider set of operators in multiple locations.

The advantage of web ubiquity [ 37] can help develop teleoperations researchby using effective methods for multi operators "agents" on multi-robots [ 36].

The interface design architecture can be used on mobile devices with some

limitation only if it is considered in advance but the ability to access webapplications on mobile devices is possible.

Web based teleoperation is not suitable if collaboration between the user andthe robot is required [ 32].


45/109

45

3.2 Mobile Teleoperation

In this section we will review solutions and designs for a operator typically locatedin close proximity but not limited to the remote environment Fig. 28 shows a mobiledevice that can operate a football robot player using only nger gestures for move-ments this inspires and motivates other possibilities to do research on teleoperationusing mobile devices.

The main reasons for using an alternative to stationary teleoperation can be summa-rized as:

Mobile devices can serve as a possible stationary teleoperation backup.

When xed locations are not possible due to high costs, technical limitationsor other preventive constraints.

When human/robot collaboration is needed or if some feedback not presenton remote robots.

Figure 28 . Finger gesture interface, Walky 2009 [ 38].


46/109

46

PdaDriver: A Handheld System for Remote Driving , [13] 2003, is a design for aPDA device to teleoperate a vehicle like shown in Fig. 29. The main project features

include: practical deployment, consumes low network bandwidth and the interfaceis easy to use with little training.

Multiple control modes, sensor fusion displays, and safeguarded teleoperation tomake remote driving fast and efcient are offered. PdaDriver is intended to en-able any operator (novice and expert alike) to teleoperate a mobile robot from anylocation and at any time [ 13].

Figure 29 . PdaDriver: user interface (left), remote driving a mobile robot (right) [ 13].

The PdaDriver system architecture is divided into three main components displayedin Fig. 30 and described as follows:

Figure 30 . PdaDriver system architecture [ 13].


47/109

47

1. Pioneer2-AT "Tarbot" Robot is equipped with on-board pc, a variety of sen-sors (a pan/tilt/zoom color CCD (charge-coupled device) camera, wheel en-

coders, GPS, 3-axis orientation module, an ultrasonic sonar ring), wirelessethernet and a pivoting training target.

2. Safeguarded Controller acts as the central control software running on the Tar-bot, it processes user commands and outputs robot states, it connects to othermodules such as sensors, map making (histogram-based occupancy grid), ob-stacle detection and avoidance, motion control, and localization units.

3. PDA GUI was primarily designed to increase situational awareness and aimed

for ease of use, it was implemented with Personal Java of Sun Microsystems,Inc and runs it on a Casio Cassiopeia E-105 Palm-size PC. The interfacescreens can be seen in Fig. 31, the following descriptions and results havebeen adopted from [ 13]:

Video mode displays images from the robot-mounted camera, horizontallines overlaid on the image indicate the projected horizon line and therobot width at different depths. The user is able to position (pan and tilt)the camera by clicking in the lower-left control area.

The user drives the robot by clicking a series of way points on the imageand then pressing the go button. As the robot moves from point to point,the motion status bar displays the robots progress. This image-basedway point driving method was inspired by STRIPE [ 39].

Map mode displays a map (a histogram-based occupancy grid constructedwith sonar range data) registered to either robot (local) or global (world)coordinates. As in video mode, the user drives the robot by clicking a se-ries of way points and then pressing the go button. As the robot moves,

the motion status bar displays the robots progress.

Command mode provides direct control (relative position or rate) of robot translation and rotation. The user commands translation by click-ing on the vertical axis. Similarly, the user command rotation by click-ing on the horizontal axis. A scale bar (located to the right of the posebutton) is used to change command magnitude. The centered circle in-dicates the size of the robot and is scaled appropriately.

Sensors mode provides direct control of the robots on-board sensors.The user is able to directly command the robots camera (pan, tilt, zoom),enable/disable sonars and to activate movement detection triggers.


48/109

48

Figure 31 . PdaDriver Screen modes for video, map, command, sensors [ 13].

Experiment and tests conducted were on different eld environments indoors andoutdoors (paved roads, off-road benign terrain, uncluttered indoor) suggests that thePdaDriver has high usability, robustness, and performance [ 40].

The results state ease of interface use by novice and expert users, safe mode drivingwas enabled through out testing which allowed users to concentrate on exploringusing different screen modes. The screen stylus is less used because it proved to bedifcult while walking or running.

The study concluded with further work suggestions by using sensor fusion displaysin addition to adding collaboration control dialogs between operator and robot hard-ware, that is the operator express intent to the robot when context information ismade available.


49/109

49

Objective Data Analysis for a PDA-Based Human-Robotic ,[40] 2004, the studyobjective was to implement a touch screen PDA device for teleoperation. The in-

terface design offered three different background screens and on the foreground itdisplays command buttons to control directions of the robot including a stop com-mand.

Vision Screen provides the robots front xed camera image seen in Fig. 32.

Figure 32 . PdaDriver Vision screen mode [ 40].

Sensor Screen provides 180 laser view and ultrasonic sensor nder. Therectangles in the Fig. 33 represent objects detected by the ultrasonic sonarand the connected lines represent objects detected by the laser range nder.

Figure 33 . PdaDriver Sensor screen mode [ 40].


50/109

50

Vision with Sensory Overlay Screen provides the image and the sensory in-formation in concert Fig. 34

Figure 34 . PdaDriver Vision and Sensor screen mode [40].

Operator evaluations were performed to determine which interface screen was the

most understandable and facilitated decision-making. The evaluation collected objective information regarding the task completion times, number of precautions,ability to reach the goal location, as well as the number and location of screentouches. The participants also had to complete a post task questionnaire after eachtask and a post trial questionnaire after each trial.

Trial Round 1 : The vision only screen tests required lower workload than the visionwith sensory overlay. On the questionnaire results; participants rated the Vision-only screen easier to use than the other screens.

Trial Round 2 : The vision only screen was signicantly easier to use than the visionwith sensory overlay screen. Participants rated the sensor-only screen was ranked aseasiest to use. The results across screens during trial two indicate that no signicantrelationship existed.

The fastest task completion time displayed in Table 1 for using the sensor screen,this was because it involved the shortest path and least visual processing.

Table 2 provides the average goal achievement accuracy for all tasks across bothtrials. The signicant increase occurred across trials for the vision task may beattributed to learning the interface and how to control the robot. Screen processing


51/109

51

Table 1 . Completion times by trial and task [40].

Trial One Trial TwoMean St. Dev. Mean St. Dev.

Vision task 4:18 0:54 4:00 0:49

Sensor task 3:36 1:23 1:52 1:12

Vision & sensory task 4:51 0:23 4:39 0:35

for all layers caused a delay which resulted in 50% unnished goals within theallotted time.

Table 2 . Accuracy percentage of goal achievement by trial and task [ 40].

Vision Task Sensor Task Vision & sensory Task

Trial One

Reached 12 14 3

Almost Reached 9 6 4

Passed 2 2 0

Not Reached 7 8 23

Trial Two

Reached 23 26 7

Almost Reached 4 1 3

Passed 1 2 2

Not Reached 2 1 18

The study did not anticipate poor performance of longest time and lowest goalachievement by using all screen layers overlaid, this is attributed to the PDA pro-cessing delay. On contrary when participants were allowed to see directly the robottask completion was the fastest using only the sensory screen, this screen providedprecautions while goal achievement was the highest.

In conclusion the study presented three different interfaces and experiment mea-surements were processed using objective data analysis. Results indicate the delay

caused by PDA screen processing reduces performance. Best results were obtainedusing sensor screen and direct robot view. The authors also suggest extended datanormalization can be utilized for different purposes.


52/109

52

The Advantage of Mobility: Mobile Tele-operation for Mobile Robots [31]2001, part one of the same study was described in previous stationary desktop tele-

operation section. In this section the same application is considered and presentedfor a PDA mobile device.

Only map and laser interfaces was implemented for the PDA due to size limitationand processing capability, the settings interface merged autonomy level and theagent mode panels together. In Fig. 35 the laser view identies obstacles in theremote environment and the map view allows robot directing by point and click, theautonomy level settings controls robot modes.

(a) Laser View (b) Map View (c) Autonomy Settings View

Figure 35 . PDA Interface [ 31].

Three experiments conducted to study the main differences between desktop andPDA teleoperation interfaces with one simulated and two on real environments de-scribed next:

First Experiment : For twenty minutes using a Player/Stage simulator [ 41] partic-ipants had to explore an environment shown in Fig. 36 without colliding into thewall or other objects. For data calculation the independent variable was dened asInterface Type { PDA interface, desktop interface} , and the dependent variablewas the covered area by robot in square meters.

Second Experiment : On a real indoor environment operators try to navigate re-motely a P2AT robot to a target point. The test environment was a 15 meters path


53/109

53

Figure 36 . Simulated Experiment Environment Map [31].

composed of cluttered corridors and narrow spaces. The dependent variable wasnavigation time measured in seconds and the independent variable was interfacetype.

Third Experiment : Test environment was outdoors divided into three differentzones, all realized using reclining panels and cartons for the following:

1. Maze: using single entrance and exit.

2. Narrow Spaces: limited area and directions where robot must pass throughthem.

3. Cluttered Areas: robot must navigate through several obstacles but can choosedifferent directions.

Participants were allowed into the environment but not within the arena, robot isonly visible from a window like seen in Fig. 37 and can be completely hidden forhalf of the path.

Participants had to complete the path measured in seconds for both variable types.The independent used path variation on Space Type {Maze, Narrow Spaces, Clut-tered Areas} , in addition to visibility variable as Operator View Degree { Total Visibility, Partial Visibility} . The dependent variable was set for interface type,

and other factors related to robot congurations, wireless signal strength remainedconstant to guarantee replicability.


54/109

54

Figure 37 . Robot partly visible for PDA operator [ 31].

Preliminary Hypothesis and Data Analysis: The operators used the laser and map

view for path planning and survey knowledge. Path planning reduces obstacle col-lision but still depends on the operator spatial awareness so an egocentric referencesystem was used to decide the direction of movement.

Survey knowledge for way nding depends on the operators location awareness,which is generally considered an integrated form of representation with fast androute independent access to selected locations, this was achieved by using an allo-centric coordinate system [ 31].

Surveying and navigating operations depends highly on information provided fromthe interface, thus, on the PDA performance is expected less due to switching viewswhere on the desktop interface views are always available. So it was priorly hy-pothesized for better performance and operators will favour the desktop interface.

In the third experiment the PDA performance was expected to do better than onthe desktop because the operator was allowed to see at some stage the robot so itrequired less information accessability and may counterbalance the results.

For each experiment the collected data was analyzed using ANOVA (Analysis of Variance) statistical methods. The description can be referred to the study [ 31]. Re-sults for the rst experiment are shown in Fig. 38 where the area is mostly coveredby using the desktop interface and performs considerably better then the PDA forexploration.

On the second experiment navigation times for driving resulted in no signicance

for both interfaces. For the third experiment results are shown in Fig. 39, it indicatesfaster times by using the PDA interface with partial and full visibility.


55/109

55

Figure 38 . Covered area in square meters by the operator using the PDA (bottom curve)and the operator using the desktop interface [31].

More ANOVA test were applied on the partial visibility data due to variable spacewhich also revealed it was faster and visibility independent in comparison with thedesktop interface Fig. 39a. But for partial visibility results are shown in Fig. 39bthe desktop interface results indicates faster navigation in comparison with PDA formaze space type.

(a) Full visibility (b) Partial visibility

Figure 39 . Mean and Standard Deviation Times for Visibility experiment three [ 31].

Discussion: The experiment results conrms different performance effectivenesscan be gained from different interfaces displayed in Table 3. The exploration task corresponds to how operators estimates the state of unknown environment when thenavigation applies to following a certain path. In case of the maze environment bothexploration and navigation were used while driving in the cluttered environment.


56/109


57/109

57

3.3 Conclusions

Studies in this chapter assert and have demonstrated successful teleoperation usinga mobile device, while each served a specic purpose common GUI elements canbe found along with certain architectural designs. Table 4 lists the advantages anddisadvantages of using a mobile devices for teleoperation.

Table 4 . Advantages and Disadvantages of teleoperation using mobile devices.

Advantages Disadvantages

Small, lightweight, easy to use andtransport [40].

Battery recharging is necessary for pro-longed usage.

Cost effective, serves as a mobilebackup [29].

Small screen display size with limitedcomputing hardware resources.

Provides solutions when human/robotcollaboration is needed [ 32].

Cannot support complex teleoperationtasks such as multiple user / robots col-laboration [ 42]

Highly recommended for eld work when stationary is not possible.

Susceptible to frequent jams and com-munication losses.

Platform and model independent, eachmodel contains features may not sup-port teleoperation requirement e.g.power saving modes cannot be con-trolled.

Performs well for certain tasks e.g.navigation, observing or for emergencyintervention only [ 31].

In some operations e.g. exploration canbe limited or not usable.

Novice operators training can bestraight forward with basic knowledgerequired.

Limited to interface controls and com-ponents.

Development time takes longer thandesktop platforms and is more chal-lenging to overcome some simple op-erations such as quick display updates.

Haptics and gesture based interactionscan enhance the practice but must notbe a decision factor.

Requires innovative GUI designs thatrequire less interactive steps to com-mand the robot.


58/109

58

There were also challenges related to communication, processing latency and limi-tation of environment feedback information. Suggestions for general teleoperation

improvements can be achieved by using additional autonomy tools or by addingremote back end intelligent system. The experiments and tests conducted can bereproduced on other designs. Presented interfaces suggest simplicity would over-come device limitations and should provide the maximal viewing of information.The ow chart displayed in Fig. 40 shows common elements used for different tele-operation tasks.

Figure 40 . Teleoperation Common Tasks and UI elements.


59/109

59

Table 5 identies general recommendation and advises when considering teleoper-ation implementation for mobile devices. For testing purposes several experiments

may be required to evaluate the system, task versus interface operated by experi-enced and novice users can be such variables. Although test cases can vary, thefollowing list gives some initial ratings for different teleoperation activities:

1. Time measurements, task accuracy variable, collision frequency count.

2. Exploration and navigation effectiveness with different feedback informationusing ( 2D map, sonar and video or images) and in combination.


60/109

60

Table 5 . Recommendations for Mobile Teleoperation and GUI Components.

Robot Software Platform: Open source distributions such as ARIA or any ven-dor libraries that can support porting to differentmobile device platforms.

Effective Practice: Best suited for full or partial robot viability. Collab-oration can be achieved when operator is in closeenvironment proximity.

Necessary Controls: Robot directional movement, speed control, stopand reset commands preferably as device physicalbuttons. In addition to safe driving mode to reduceobstacle collision.

Feedback Information:1. Robot status information, battery power, avail-

able communication signal.

2. Sonar sensors read outs/displayed helps reducesobstacle collisions.

3. 2D environment map for exploration and naviga-tion.

4. Environment video or images for visual feed-back.

Mobile Device HW: Screen display should be large enough to show agood portion of the 2D environment map. Strongwireless communication support, power manage-ment can be controlled with long lasting battery life,feasible costs. Several programmable physical andergonomic buttons.

Mobile Device SW: The OS should be responsive and preferably sup-

ports threading.Development tools are available, arobot simulator is a must have tool. The devicecan supports common graphical drawing librariesfor faster implantation times.

Supporting HW & SW: GPS for precise location reading, haptic devices forsensitive operations e.g. robotic surgery. Situationautonomy tools and speed acceleration control e.g.mining robots.

Notes and Issues: Communication latency can occur specially if us-

ing video streaming, data packets can be lost thus,recovery methods are necessary in implementationdesign.


61/109

61

4 REQUIREMENTS AND SPECIFICATION

This chapter outlines the system requirements and functional specications used toimplement a mobile application for teleoperating a mobile robot. It also describesthe systems hardware and software components part of the thesis practical work.The designed application will be referred to as RTMU (Robot Teleoperation MaemoUser Interface) throughout the remaining chapters.

4.1 Overall Description

RTMU can be dened as a GUI application running on a mobile Nokia N770 devicewith a sufcient set of commands and GUI layout to teleoperate the P3-DX mobilerobot via WLAN. The visual system entities are depicted in Fig. 41 and each partwill be explained in more detail.

Figure 41 . RTMU overall system entities.

4.2 Functional Requirements

Functional requirements dene the expected system features, lists the main objec-tives and serve as a reference for identifying design specications. The followinglists the required application functionality.

RTMU is a client oriented application that will connect and use services of-fered by the robot server for teleoperation tasks.

RTMU shall use GUI components to achieve teleoperation tasks.


62/109

62

The necessary teleoperation tasks involve moving the robot in four directions,display image from remote environment and present robot status feedback

information. The required robot feedback status information are position coordinates, bat-

tery level, rotation angle, velocity speed and WLAN signal strength.

The GUI is expected to display a 2D map. The map can be requested fromthe robot server.

Hardware buttons are used for directional commands ( forward, backward,right and left ) that will move the robot accordingly. Also a stop command is

required. Those HW buttons must be easily usable on the mobile device.

IT must be able to request and display the robots environment image. If possible, overlay the image under the robots 2D map.

The user can save connection settings for later connections, with out the needto enter them again.

4.3 Hardware

Mobile Robot: The P3-DX shown in Fig. 42 will be used as the robot hardware.It is available from the IT department. Onboard there is a mounted camera anda laptop. The laptop is connected to the robot hardware via USB (Universal SerialBus) or an RS-232 connection and runs a server that can offer teleoperation serviceswhen started and connected to a WLAN network.

Figure 42 . P3DX robot with camera and ARIA server laptop [ 3]


63/109

63

Mobile Device: The Nokia N770 will be used as the mobile teleoperating device.It is available from the departments lab. It can connect to the IT department WLAN

and the factory settings can be updated with newer OS versions, and it is permittedif needed to install essential softwares.

4.4 Communication

Communication between the robot server and mobile device can be achieved usingthe IT department WLAN and must be established prior to any teleoperation tasks.

It is also possible to use a single wireless router for more connectivity and network bandwidth support.

4.5 Software

ARIA Server: The laptop onboard the P3-DX in operational mode, connected toWLAN and running a special application called ARIA server, is responsible for

communicating with ARIA clients and with the robot hardware. Using a specialJPEG image frame grabber, ARIA server can offer images to clients requestingcurrent environment images. The frame grabber is a separate application that needsto be running and identied with the ARIA server.

Nokia N770: The Nokia N770 operates on a Maemo OS and its SDK environmentwill be used for implementing and running the future RTMU application. RTMUis essentially an ARIA client and the major development tools taken in use will

be Maemo 2.2 SDK, Scratchbox, ARIA client library. Other helper applications notlimited to Slickedit, VirtualBox, Xming, Total Commander, VncViewer, RapidSVNand Sparex Enterprise Architect. RTMU GUI wire frames will be presented in theimplementation chapter.

4.6 Teleoperating Environment

The robot will be teleoperated within the IT departments corridors like shown inmap Fig. 43, the RTMU operator can follow the robot or be located in a xed posi-tion. The environment can contain obstacles and forbidden areas / lines, in addition


64/109

64

to home and goal location points.

Figure 43 . Map of the Lappeenranta University of Technology phase 6, Floor 5 [43]

4.7 Dependencies and Scope

Communication between the mobile device and robot is established via WLANTCP/IP, and is available on the teleoperating site.

The N770 and P3-DX are preferably connected on the same network group.

The robot is congured and running ARIA server in a listening to clients

state. This includes starting the JPEG image frame grabber and ARIA is ableto read and can send images.

ARIA server is accessible to RTMU by an IP and port number, which aregiven to the RTMU operator including any required user credentials neededfor connectivity.

Prior to RTMU development ARIA server additional commands or specialfeatures are made available.

ARIA API documentation and code examples are available and can be recom-piled within scratchbox.


65/109

65

A robot hardware simulator MobileEyes shown in Fig. 44 can be used fordevelopment purposes. The P3-DX robot will be used for real system testing.

Client ARIA API library sources can be ported to Maemo N770 ARMELtarget platform.

Figure 44 . Screen shot of MobileSim robot simulator.


66/109

66

4.8 Design and Implementation Constraints

The application implementation should follow the MVC design pattern. Linux is the base operating system for both the server laptop and the N770

device, thus GNU GCC C/C++ development language and compiler will beused.

Development on N770 requires Maemo 2.2 SDK, the package and sourcescan be downloaded from the Nokia web site. The SDK offers Hildon andGTK+ libraries for GUI design and application development that is used ac-cordingly.

ARIA package includes source code released under GNU public license thatcan be recompiled with at least GCC 3.4 on Linux based OS. This recompila-tion is needed with scratchbox and for Maemo 2.2 ARMEL target platforms.Porting source code maybe required.

The actual robot can be simulated using MobileSim [ 44] application likeshown in Fig. 44 and can read the departments map, the ARIA examplescan work with the simulator and development should follow same logic owof connectivity and control example code.

The C/C++ coding convention shall follow the IT departments recommenda-tion style.

For documentation purposes Doxygen [ 45] will be used to produce HTMLles that contains usage instructions and implementation documentation.


67/109

67

4.9 Use Cases

Use cases describe the users interaction with the system and assist in identifyingnecessary functional requirements for system development. This chapter presentsthe recognized use cases for the RTMU application.

Robot Startup : The robot operator is responsible for starting the robot hardwareand the ARIA server like shown in Fig. 45. ARIA server listens to incoming con-nections via WLAN connectivity and is responsible to send commands to the robothardware and retrieve its status.

Figure 45 . Operator manages robot hardware and ARIA server software.

RTMU application will connect to the ARIA server using the laptop IP and thelistening port number. The IP and port numbers can be obtained from the laptopsdisplay or can already be noted from the robot operator. Without this information

no teleoperation can be accomplished. The robot operator is presumed to be the ITdepartment lab assistant.


68/109

68

RTMU : The identied overall use cases can be grouped in Fig. 46 and are ex-plained individually in the upcoming subsections. It is assumed the robot hardware

and ARIA server are running. On the N770 device RTMU application is launchedand is ready to perform the use case presented.

Figure 46 . RTMU application use cases.


69/109

69

4.9.1 Connection Settings

Actors: RTMU operator

Summary: The actor chooses to manage the necessary ARIA serverconnection settings elds.

Preconditions: RTMU is running.

Postconditions: ARIA server connection settings updated or not.

Flow Description:1. Actor selects settings dialog.

2. Application settings dialog is displayed and consists of current IP, Port, User Name and Password elds.

3. Actor modies values by typing into corresponding en-try elds.

4. Actor saves or cancels changes.

5. If saved connection settings are retained for later usesessions.

Alternatives Flows: None

Exceptions: None

Priority: High

Frequency of Use: First time or when such eld values need to be updated.

Notes and Issues: None


70/109

70

4.9.2 Connect To ARIA Server

Actors: RTMU operator, P3-DX Robot

Summary: This use case allows the RTMU operator to connect to theARIA server, it is the initial step and the connection mustremain open during the teleoperation session.

Preconditions: The N770 device is connected to a WLAN network routable to ARIA server. RTMU application is runningand connection settings are vaild. No connection to ARIAserver is already active.

Postconditions: A successful connection will initializes UI and robot mapif possible. Otherwise the RTMU operator will be notiedwith failure.

Flow Description:1. RTMU operator selects the connect command.

2. A notication of attempting to connect is displayed.

3. If connection establishes it will hide the previous noti-cation and updates the interface by populating robotstatus values and try to display the robots environmentmap.

4. Otherwise if a connection attempt has failed then anotication dialog is displayed with connection failedmessage and exists case.

5. RTMU application waits for RTMU operator com-mand.

Alternatives Flows: RTMU operator selects the connect command but no con-

nection settings have been set then a notication to editsettings elds is displayed and exists case.

Exceptions: Only one connection is allowed from the RTMU applica-tion to the ARIA server. The operator is expected to dis-connect after each connection established and when notneeded anymore.

Priority: High

Frequency of Use: For every teleoperation session a connection must be es-tablished.

Notes and Issues: None


71/109

71

4.9.3 Disconnect from ARIA Server

Actors: RTMU operator , P3-DX Robot

Summary: The actor chooses to disconnect from the ARIA serverthus closing the active connection and ends the teleopera-tion session.

Preconditions: RTMU is running and connected to ARIA server.

Postconditions: Connection is closed and interface is cleared from robotstatus values and map display, also enables the connectcommand for further requests to connect again.

Flow Description:1. RTMU operator selected disconnect command that

closes the connection.

2. Interface is cleared from robot status values and envi-ronment map.

3. Interface enables the connect command.

4. RTMU application waits only for exit or connect usecase.

Alternatives Flows: RTMU operator can select the exit command but it implic-itly close connection if its already open.

Exceptions: None.

Priority: Medium

Frequency of Use: After each connection the RTMU operator should closethe connection.

Notes and Issues: If RTMU crashes ARIA server is expected to close theconnection, if the crash does not terminate RTMU thenit is necessary to kill the process using the OS softwaremanager or such tools.


72/109

72

4.9.4 Command

This is an abstract generalized use case, its main purpose is to provide the possibilityof extending the system commands and act as a pattern for implementation. Mostcommands expect a response from the server implemented as an output callback and is described separately.

Actors: RTMU operator , P3-DX Robot

Summary: The RTMU operator selects a specic command from theinterface and is sent to the ARIA server. If the commandis supported then it is executed and the outcome is sentback to the RTMU application. Responses are typicallyexecuted within output call back implementations.

Preconditions: RTMU is running and connected to ARIA server.

Postconditions: Command output call back is executed.

Flow Description:1. RTMU operator selects a specic command from the

interface.

2. Interface is updated or a notication is displayed.3. RTMU application waits for operator next command or

was requested to exit.

Alternatives Flows: RTMU operator can select the same command representedin UI or by using a physical button located on the N770e.g. move robot forward can be accomplished by click-ing/tapping UI up arrow scroll up key.

Exceptions: In some cases a single command can perform several tasksi.e: move forward and update display.

Priority: High - Medium

Frequency of Use: Most commands are handled this way.

Notes and Issues: Timeout and recovery can be challenging to implement. Itis a good practice to che

Teleoperacion Mobil de Un Robot Mobil

Documents

Transcript of Teleoperacion Mobil de Un Robot Mobil