Towards a real-time, configurable, and affordable

system for inducing sensory conflicts in a virtual

environment for post-stroke mobility rehabilitation:

vision-based categorization of motion impairments

Babak Taati, Jennifer Campos, Jeremy Griffiths, Mona Gridseth, Alex Mihailidis

Sept 11, 2012ICDVRAT

Outline

Motivation

Background

Problem Definition

Tracking Technologies

Method

Experimental Results

Conclusions & Future Work

2

3

Post-Stroke Rehabilitation

15 million people suffer stroke each year

65% of stroke survivors have difficulty using their upper limbs

The economic cost of stroke is ~$6.3 billion a year

Training exercises that provide patients with visual feedback on the movement of the affected limb facilitate rehabilitation

Motivation

Mirror Box Illusion

View of the affected arm is replaced by the mirror reflection of the healthy arm

Simultaneous motion, or attempt of motion of both arms results in artificial visual feedback which reinforces neurorehabilitation

Mirror box therapy is effective in restoring mobility

4

Background

http://en.wikipedia.org/wiki/Mirror_box

(Ramachandran et al, Nature,1995)

Rubber Hand Illusion

5

http://www.thefrogwhocroakedblue.com/illusions.asp

(Botvinick and Cohen, Nature, 1998)

Simultaneous brushing of the rubber hand and the real hand generates a sensory conflict in the brain between the location of the visual vs. proprioceptive signal

Continued brushing typically results in a sensory takeover of proprioception and the person would feel ownership of visually perceived plastic arm

Background

Objective

Reproduce and combine the Mirror Box and the Rubber Handillusions in a virtual environment

• Computer vision: markerless skeleton tracking

• Computer graphics: visualization + pose augmentation

• Machine learning: classification + latent space projection

6

I felt my arm was here …

… but I see it here!

Problem Def.

Milestones

Duplicate the Rubber Hand Effect

• Visual skeleton tracking & visualization

• Synchronized tactile stimulation

Combine the Rubber Hand with the Mirror Box Effect

• Categorize motion impairments

• Apply a movement gain (normalize / exaggerate)

7

Problem Def.

8

Human Pose Tracking

8

[Kanaujia, 2011]

Tracking Tech.

9

Human Pose Tracking (cont’d)

Marker-based• Elaborate setup

Markerless (vision-based)• Accuracy vs. computational efficiency

9

[CMU Motion Capture Database]

[Flickr: beratus]

[Lee & Nevatia, 2009]

Tracking Tech.

10

Consumer Depth Cameras

10

KinectTM (Microsoft) WAVI Xtion (ASUS / PrimeSense)

Real-time color & depth sensing

Real-time skeleton tracking software libraries

• Microsoft SDK or NITE

Tracking Tech.

11

Challenges

Noise / inaccuracies / systematic bias Missing information Ground truth annotation Evaluation of categorical time series prediction

11

Tracking Tech.

True Labels

Predicted Labels

12

Dataset

7 healthy adults

Simulated 2 of the most common upper limb flexion synergies present in the post-stroke population

• Elbow flexion, shoulder flexion, shoulder abduction

Stereotypical post-stroke movement synergy: Rotating the Trunk

• Elbow flexion

Stereotypical post-stroke movement synergy: Elevating the Shoulder

10x normally, 10x simulating an impaired motion

Method

13

Algorithms

Features• Displacement Vectors• Principal Components Analysis (PCA)

Multi-class categorization• Logistic Regression• Support Vector Machines (SVM)• Random Forest (RF)

Cross validation• 7-fold leave-one-subject-out

Method

Dominant Modes of Motion 1

14

Results

Reaching Across(Elbow flexion, shoulder flexion, shoulder abduction)

With shoulder elevation“Normal”

Dominant Modes of Motion 2

15

Results

Reaching Up(Elbow flexion)

With trunk rotation“Normal”

Classification Accuracy

ClassifierAction N Logistic Reg. SVM RF

Reach Across1 54.3 97.1 97.93 61.4 85.7 97.1

Elbow Flexion1 64.3 95.7 90.03 59.3 85.0 92.9

16

Results

# of PCA components

17

Summary

Inherent biases were observed in the tracking of an elevated shoulder

Despite accuracy and tracking bias issues, it was possible to separate impaired motions

A single most dominant mode of motion, as identified by PCA, was sufficiently discriminative

SVM classifier obtained consistent detection accuracies >95%

Conclusions

18

Future Work

Exaggerate / normalize motions

• Latent space projection

Combine the Rubber Hand and Mirror Box illusions

Real data (real post-stroke patients)

• Transfer learning

Conclusions

19

Acknowledgements

Toronto Rehab (iDAPT research facilities)

20

Home Environment Laboratory (HEL) Challenging Environment

Assessment Laboratory (CEAL)