INTRODUCTION

Generally, after stroke, patient usually has cerebral cortex functional barrier, for example, the impairment in the following capabilities, namely, language capability (listening, speaking, reading, writing and understanding capability), personality structure (character and response to a thing), memory power, abstract thinking, computing power and judgment power, etc..

Virtual reality is effective method when it is used in rehabilitation training, and researchers find good results in developing related applications.

1

RESEARCH METHOD

A. Experimental architecture

The system is mainly designed with rehabilitation action practice, then together with virtual reality, the simulation of the operation of ball receiving can then be achieved. In the task, Kinect sensor device is used to detect the limb action of the patient, to perform face recognition, and to identify voice command.

2

RESEARCH METHOD

B. System setup

The system uses Kinect sensor device to interact with the virtual environment, which can be used to recognize the upper limb action of the patient. Body sensing technique is totally different to the traditional game player in the control system, in traditional game player, joystick is mostly used to perform button type operation, and that interface is not good in simulating the action in real life. However, in Kinect, human action is used as the basis, and three types of contents can be acquired in one time, namely, color image, 3D depth image and voice signal (figure 1).

3

Figure 1. Color image, 3D depth image and voice signal .

RESEARCH METHOD

B. System setup

Kinect itself has three lens. The central lens is the commonly seen RGB color camera, the left lens is IR transmitter, and the right lens is IR CMOS camera. Three of them forms together 3D depth sensor as (figure 2).

4Figure 2. 3D depth sensor architecture diagram.

RESEARCH METHOD

B. System setup

In order to let Kinect equipment be able to operate in Windows platform, currently, the treatment and sequence is as in the following: 1/ Install Open Natural Interaction (OpenNI), which has cross-platform open source framework. Application Programming Interface (API) has defined the syntax of voice, posture and body action.

5

Figure 3. Three layer architecture of OpenNI.

RESEARCH METHOD

B. System setup

2/ Install Sensor Kinect driver program as provided by Prime Sense, Support OpenNI bottom layer. 3/ Install NITE as provided by Prime Sense. Support OpenNI medium layer and analyze the scene and human skeleton structure (figure 4 and figure 5). 4/ Implement the default calibration posture as defined by NITE.

6Figure 4. Analyze the scene structure.Figure 5. Analyze human skeleton structure.

RESEARCH METHOD

B. System setup

The main objective of the game is to implement the practice of the upper limb action. In the virtual reality environment, one virtual character is receiving ball within it. The person under test must let Kinect be able to detect his action through the extension of the arm, and the controlling of the ball receiving direction by the virtual figure is then corresponded.

7

Figure 6. Kinect Reaching Ball / Figure 7. Kinect Catch Ball.Figure 8. The user interacts with the virtual environment.