May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Evaluation of a robot as embodied Evaluation of a robot as embodied interface for Brain Computer interface for Brain Computer

Interface systemsInterface systems

L. Tonin, F. Piccione, S. Silvoni, K. Priftis, L. Tonin, F. Piccione, S. Silvoni, K. Priftis, Emanuele MenegattiEmanuele Menegatti

IRCCS San Camillo,Venice, Italy

Dept. Of General Psychology

University of Padua

Intelligent Autonomous Systems Lab University of Padua

2May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

What is BCI?What is BCI?

• There are four general steps in a BCI system:

1. Signal acquisition2. Classification and recognition3. Interaction with device4. Feedback to Subject

•According to Birbaumer et all.(1999): “Brain Computer interfaces are systems that allow to translate in real-time the electrical activity of the brain, in commands to control devices“

• Our target for the use of BCI system are patients severely paralyzed (locked-in) due to injury or disease

3May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Original P300-based BCI of IRCCS San CamilloOriginal P300-based BCI of IRCCS San Camillo

• This BCI system is based on P300 wave detection

• P300 is event-related potential

• Visual interface gives to patient stimulus to activate P300 signal

• The P300 detection will control the movement of cursor

• Directional cues are represented by the single flash of one out four target-direction arrows

• Participants focus their attention on the flashing arrow corresponding to the desired direction of motion

• Validated on 5 tetraplegic patients and 7 healthy patients [Piccione et al. Clinical Neurophysiology 2006]

4May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

P300-based BCI of IRCCS San CamilloP300-based BCI of IRCCS San Camillo

Focusing own attention, patient will generate P300 signal:

5May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Goals of our projectGoals of our project

In this project we aim at investigating two main issues:

• Giving to locked-in patients the control of a physical actuactor to improve their rehabilitation experience

• Developping telepresence of the patient with the body of the robot as medium

The questions are:

Can the robot lead patient to full and better control of BCI via a higher patient engagement in the task?

Might patients identify them-selves into the robot and think of it as an extension of their perceptions and actions (i.e. their body)?

6May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Robot as a physical actuatorRobot as a physical actuator

• To be able to move a body again, might give new strength and will to the locked-in patients

1. Custom built holonomous robot with hexagonal structure and three omnidirectional wheels

2. The holonomous nature of the robot is very important to replicate the motion and the apparence of the interface used by our BCI

The robot is fitted with:• Omnidirectional camera• Audio board• WiFi connectivity

7May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

BCI and robot connectionBCI and robot connection

BCI system SERVER

RobotClient

Command

Action result

• Interaction between robot and BCI is based on client-server system with TCP-IP protocol via IEEE 802.11

Security about connection between BCI and robot is guaranteed by:

• System protocol and network protocol check data received;

• Virtual Private Network between BCI and robot with autentication and encryption

8May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Test and DemoTest and Demo

We carried out many tests to integrate BCI and robot. These tests accomplished at IRCCS San Camillo in Venice, and in IAS Laboratory ofUniversity of Padua:

• Robot and BCI computer connected with wired network

• Testing healthy subject performances to move robot

•BCI and robot connected in larger network with VPN connection(robot was in Padova, and BCI system in Venice)

9May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Test and DemoTest and Demo

During the scientific festival Discovery on Film in Rovereto (May 2008).

A healthy subject at San Camillo in Venice was able to move the robot using the BCI at museum of Rovereto (about 200 Km from Venice).

The test was successful and subject was able to reach the target.

You can see it at: www.sperimentArea.tv the web TV of

the Museo Civico of Rovereto

10

May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Robot as actuator and generator of interfaceRobot as actuator and generator of interface

Robot is able to send back to patient images and sounds about environment.

The images token by omnidirectional camera can be integrated in BCI interfaceto allow patient to see through robot’s camera.

A new integrated interface for BCI is under clinical validitation by prof. Priftis of Psychology Department of University of Padua

11

May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Robot as actuator and generator of interfaceRobot as actuator and generator of interface

Figure 6: classification accuracy (%) of the 5 healthy subjects who performed Experiment 1, and classification accuracy (%) of one subject who performed Experiment 2.

12

May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Robot sensesRobot senses

13

May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Robot as a Telepresence mediumRobot as a Telepresence medium

• The patient is able to see and listen with the senses of the robot and so he is able both to extend own social life and communicate own needs

• Telepresence for museum visits: more autonomous navigation,obstacle avoidance, new graphical user interface

• Telepresence for household rehabilitation: consumer robot Rovio holonomous robot with frontal camera for remote perception.

14

May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

Robot as a Telepresence for museum visitsRobot as a Telepresence for museum visits

TCP/IP

The BCI Server

Internet

Patient in Venice, robot in Agrigento, Sicily, 1467 km

The Client robots developed The Client robots developed at University of Palermo, at University of Palermo, Sicily, ITALYSicily, ITALY

Pioneer3-AT (Outdoor Robot)

PeopleBot (Indoor Robot)

ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 1: Choose of the real robot

The Graphical Interface

ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 2:The Session Start and Sensors and Actuators are Activated

The Graphical Interface

ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 2.1:

The Applet for the visualization of the navigation environment through the robot camera

The Graphical Interface

ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 3: Robot Movements Activation through the

use of the 4 directional arrows

The Graphical Interface

ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

The Graphical Interface

ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

21

May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

• Next step: fully autonomous robot, robot has a map of the environment, robot presents four topological choices (e.g. the four closest masterpieces), subject selects destinations

Robot as a Telepresence for museum visitsRobot as a Telepresence for museum visits

22

May 12, 2009 BCI Workshop ICRA 2009, Kobe

Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems

• Next step: fully autonomous robot, robot has a map of the environment, robot presents four topological choices (e.g. the four closest masterpieces), subject selects destinations

Robot as a Telepresence for museum visitsRobot as a Telepresence for museum visits

The Experimental Results: Simulation

The BCI-Robot moving in the DINFO Department.Time T1: Command Go Forward

BCI- Web_Interface Snapshot of Robot Position

BCI- Web_Interface

Snapshot of Robot Position

Time T10: Command Stop

Many Thanks to You and to: Dept. of Information Engineering, University of Palermo, Palermo,

ItalyRosario Sorbello: sorbello_rosario@unipa.itAntonio Chella: chella@unipa.itSalvatore Anzalone: anzalone@dinfo.unipa.itFrancesco Cinquegrani: cinquegrani@dinfo.unipa.it

Dept. of Information Engineering, University of Padova, Padova, Italy

Emanuele Menegatti: emg@dei.unipd.it

Dept. of Neuro-rehabilitation, S.Camillo Hospital, Venice, ItalyLaboratory: labeeg.nfsancamillo@virgilio.it

Dept. of General Psychology, University of Padova, Padova, ItalyKonstantinos Prifitis: konstantinos.priftis@unipd.it