Dynamical Robotic Interaction Using High-speed Visual Feedback

and High-speed Robot Hand

Yuji Yamakawa1,2 and Masatoshi Ishikawa3

Abstract— In this paper, we focus on dynamical roboticinteraction with high-speed visual feedback and high-speedrobot hand. This paper describes two human–robot interactivetasks; the first one is Janken (rock-paper-scissors) robot as anexample of non-contact type human–robot interaction and thesecond one is carrying object robot as an example of contacttype human–robot interaction. To achieve these tasks, we brieflyexplain the proposed method and show experimental results.

I. INTRODUCTION

Robot technologies for manipulation and human–robot in-

teraction have been proposed. These technologies contribute

to automation of object handling by a robot and assistance

of manual handling by a human. Recently, a fusion of

manipulation and human–robot interaction has been actively

studied for improving the working efficiency and the task

performance. In the conventional technologies, the camera

recognition of the object and the manipulation by the robot

takes a long time, and these factors are the disadvantages of

the conventional technologies.

Thus we apply the high-speed robot technology to the

manipulation and the human–robot interaction. In particular,

this paper describes the concrete applications in the human–

robot interaction.

A. Related Work

Until now, there has been a lot of researches on human–

robot interaction. Especially, in order to carry an object

in the human–robot interaction, the task was successfully

performed by Yokoyama et al. [1], an assistive system was

developed by Hayashibara et al. [2], and a guidance system

was proposed by Stuckler and Behnke [3].

High-speed robot systems using a high-speed image pro-

cessing technique with a high-speed camera (1,000 fps),

which is beyond human eye (30 fps), were developed in our

laboratory. Also, dynamic manipulations and other applica-

tions were achieved [4], [5]. We considered that high-speed

robot technology can be applied to human–robot interaction.

B. Purpose

We describe Janken (rock-paper-scissors) robot as one

example of non-contact type human–robot interaction. Also

we explain object carrying robot as one example of contact

type human–robot interaction. These tasks, which require

1Institute of Industrial Science, The University ofTokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japany-ymkw@iis.u-tokyo.ac.jp

2JST, PRESTO, Kawaguchi, Japan.3Graduate School of Information Science and Technology, The University

of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan

Spee

d

Degree of support

Targetarea

Guidance robot Power assist robot

Janken robot Object carrying robot

Fig. 1. Concept of this research.

the high-speed performance, are considered to be dynamical

robotic interaction using high-speed visual feedback and

high-speed robot hand (Fig. 1). Moreover, we show the

possibility of the applications of the high-speed robot system

to dynamic human–robot interaction, which means the robots

reacts to the human action immediately.

II. HIGH-SPEED ROBOT SYSTEM

Our robot system consists of a high-speed robot hand,

a high-speed vision system and a real-time controller. The

real-time controller receives the image-processing result and

controls the high-speed robot hand at 1 kHz.

A. High-speed Robot Hand

We developed the high-speed robot hand [6] whose joints

can be closed at a speed of 180◦/0.1 s. The high-speed robot

hand has three fingers, and each finger has a top link and a

root link. In the development of the high-speed robot hand,

a miniature and high-power servo motor was also developed

based on a new design concept of the servo motor.

B. High-speed Vision System

We used a high-speed vision system with a high-speed

camera and high-speed image processing technique to recog-

nize the state of a measurement target in real time. Only the

image-processing results are sent to the real-time controller.

By proposing the appropriate image processing to each task,

the high-speed image processing can be performed.

III. JANKEN (ROCK-PAPER-SCISSORS) ROBOT

We show Janken (rock-paper-scissors) robot as the first

application. This task is considered to be non-contact type

human–robot interaction.

A. Method

We divided the task into the following two steps:

1) The hand shape (rock, paper or scissors) of the human

is recognized using high-speed vision in real time.

2) The robot hand is quickly controlled to win the human.

(a) (b) (c) (d)

(e) (f) (g) (h)

Fig. 2. Janken (rock-paper-scissors) robot.

By repeating the above two steps every 1 ms, the robot can

always win the human.

1) Hand Shape Recognition: In order to appropriately rec-

ognize the shape (rock, paper or scissors) of the human hand

at high-speed, we proposed a new algorithm for recognizing

the shape [7]. In the proposed method, we can recognize the

hand shape robustly and quickly.

2) Robot Hand Control: According to the recognition

result of the hand shape, we give the reference joint angles

of the robot hand and control the robot hand so as to beat

the human.

B. Result

Fig. 2 shows continuous photographs of the experimental

result. Also, a video of the experimental result is available on

our web site [8]. From the experimental result, we achieved

the Janken (rock-paper-scissors) robot with 100% winning

rate.

IV. OBJECT CARRYING ROBOT

Here we explain an object carrying robot as the second ap-

plication. This task is considered to be contact type human–

robot interaction via an object to be handled. The details of

our proposed method are given in [9].

A. Method

We also divided the task into the following two steps:

1) The position and orientation of the object are measured

using high-speed vision in real time.

2) The robot hand is quickly controlled according to the

position and the orientation of the object.

By repeating the above two steps every 1 ms, the object

carrying task can be successfully achieved.

1) Measurement of Position and Orientation of Object: In

order to measure the position and orientation of the object,

we tracked four corners of the object (a rectangle board)

using the high-speed vision and the high-speed tracking al-

gorithm [10], in which a marker was attached to each corner.

From the information of the four corners, we measured the

position and the orientation of the object.

2) Robot Hand Control: According to the position and

the orientation of the object, the robot hand was controlled

using PD (proportional-derivative) control. The reference

joint angles of the robot hand were obtained by the inverse

kinematics.

(c)

(g)

(a) (b) (d)

(e) (f) (h)

Fig. 3. Object carrying robot [9].

B. Result

Fig. 3 shows continuous photographs of the experimental

result. Also, a video of the experimental result is available on

our web site [11]. From the experimental result, we achieved

the object carrying task using the high-speed robot system.

V. CONCLUSION

In this paper, we focused on dynamical robotic interaction

with a high-speed robot system. Our system is specialized

in the speeds of robots, visual feedback rate and reaction of

robot. The proposed method was also executed by a high-

speed vision system including high-speed image processing

and a high-speed robot hand. As examples of the human–

robot interaction, we developed and demonstrated Janken

(rock-paper-scissors) robot and object carrying robot.

ACKNOWLEDGMENT

This research was supported in part by JST, PRESTO

Grant Number JPMJPR17J9, Japan.

REFERENCES

[1] K. Yokoyama, H. Handa, T. Isozumi, Y. Fukase, K. Kaneko, F.Kanehiro, Y. Kawai, F. Tomita and H. Hirukawa, Cooperative worksby a human and a humanoid robot, 2003 IEEE Int. Conf. on Roboticsand Automation, pp.2985–2991, 2003.

[2] Y. Hayashibara, T. Takubo, Y. Sonoda, H. Arai and K. Tanie, Assistsystem for carrying a long object with a human-analysis of a humancooperative behavior in the vertical direction, 1999 IEEE/RSJ Int.Conf. on Intelligent Robots and Systems, pp.695–700, 1999.

[3] J. Stuckler and S. Behnke, Following human guidance to cooperativelycarry a large object, 11th IEEE-RAS Int. Conf. on Humanoid Robots,pp.218–223, 2011.

[4] http://www.k2.t.u-tokyo.ac.jp/fusion/index-e.html[5] http://www.hfr.iis.u-tokyo.ac.jp/index-e.html[6] A. Namiki, Y. Imai, M. Ishikawa and M. Kaneko, Development

of a high-speed multifingered hand system and its application tocatching, 2003 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems,pp.2666–2671, 2003.

[7] K. Ito, T. Sueichi, Y. Yamakawa and M. Ishikawa, Tracking andRecognition of a Human Hand in Dynamic Motion for Janken (rock-paper-scissors) Robot, 2016 IEEE Int. Conf. on Automation Scienceand Engineering, pp.891-896, 2016.

[8] http://www.k2.t.u-tokyo.ac.jp/fusion/Janken/index-e.html[9] Y. Matsui, Y. Yamakawa and M. Ishikawa, Cooperative Operation

between a Human and a Robot based on Real-time Measurement ofLocation and Posture of Target Object by High-speed Vision, 2017IEEE Int. Conf. on Control Technology and Applications, pp.457–462,2017.

[10] I. Ishii, Y. Nakabo, and M. Ishikawa, Target tracking algorithm for 1ms visual feedback system using massively parallel processing, 1996IEEE Int. Conf. on Robotics and Automation, pp.2309–2314, 1996.

[11] http://www.hfr.iis.u-tokyo.ac.jp/research/collaboration/index-e.html