OI Interface technology: Overview of the MPL Experience to date

Courtney Moran (Courtney.moran@jhuapl.edu)Robert Armiger (robert.armiger@jhuapl.edu)

Revolutionizing Prosthetics Milestones

2005-2007

•Prototype 1 •TMR Integration

2008-2009

•Prototype 2•Neural control and stimulation in primate model

2010-2011

•MPL – Gen1•ECoG Human Control

2012-2013

•First TMR surgery @ JHMI

•First Human Cortical Control of highly dexterous prosthetic

2014-2015

•MPL Gen 2/3 Sensorization

•First TMR+OI subject controlling dexterous prosthetic limb

•First Human Bi-directional Cortical Control

2016-2017

•Significant advancement in Targeted Sensory Reinnervation

•Take Home Evaluation Study

•AR/VR/Autonomy MPL Integration

•MindFlight

2018-2019

•MPL adapted for take-home use

•first demonstration of individual piano playing

Revolutionizing Prosthetics

• JHU/APL has been a world leader in next generation prosthetics:– Highly dexterous prosthetic arms that integrate seamlessly with the

body– Advanced control techniques that enable natural ‘thought-based’

control and feedback– Dynamic measurement of real world use cases

• Overview • Path to OI integration• Significant Accomplishments• Take home study overview• Lessons learned • Next steps

Enabling Osseointegrated Control

Interface technology…• Wireless electrodes• Quick Disconnect hardware for Compress• Embedded Pattern Recognition Controller

Assuring safety… • Sensorized (Load Cell) interface measuring torque• Dynamic ‘Impedance’ control (modulates limb strength)

Capturing Outcomes…• Advanced data logging and connectivity for monitoring• End-user ‘Mobile App’ for limb training• Integrated assessments for performance evaluation

Interface and Wireless Control

• Load cell monitor forces (strain) applied to implant

• Torque limit through rotational slip

• Torque limit through material based failure points

• Open Source pattern recognition controller– Prototype runs on $35 Raspberry Pi– Custom/final version on-board MPL– Wireless signal acquisition– Real time (50Hz) machine learning

classification

Interface and Wireless control

Quick Disconnect safety connector

Myo ® Armband(s)(Custom interface)

Osseointegrated MPL with

embedded controller

Mobile App (web-based)

Bluetooth Low Energy

(2.4 Ghz)

Wifi 802.11(2.4 Ghz)

Usage Data Logging

• Raw EMG• Residual limb position• Implant torque• Elbow / Wrist Joint Position & Torque• Battery Levels• Finger torques• Fingertip tactile forces

End-user interface: Mobile App

• Provides end-user interface for:– Training Limb System using

Machine Learning Algorithms– Interactive / User-driven training

process– Configuring Limb Speeds and motion– Providing periodic user ‘assessments’– Logging and Uploading usage data,

configuration info, and upgrades

Summary

Smarter prosthetics can enable and inform outcomes• OI presents new challenges with removal of socket• New interface technologies developed • Data usage and monitoring tools implemented• Advanced safety features can help minimize risk

and improve understanding of OI process

Lessons learned: Osseointegration

• Significant opportunities– Allows load transfer without interfering with EMG signal acquisition– Supports both surface and implanted motor and sensor interfaces– Promotes embodiment of the device

• Significant challenges– Mechanical: Develop standardized preclinical mechanical testing strategies– Functional: Develop functional evaluation methods through standardized clinical

testing, patient reported outcomes, and patient preference– Biological: Develop standardized methods for the measurement, classification,

and reporting of infections

Next Steps• Successful prosthetic outcomes

require advancements across all levels of the prosthetic device

• Take home study designed for longitudinal evaluation

• Advanced Sensory Capabilities

• Focus on outcome assessments for comparative effectiveness of prosthetic technologies

Body Attachment

Arm Dexterity

Bidirectional Biointerface

Completed

In progress

TAKE HOME STUDY-BACK UP/DATA

Study Design - Assessments• Bimonthly Assessments at Walter Reed

– Box and Blocks (Gross Motion)– Range of Motion – Clothespin task – Positional assessments on force plate – Jebsen Taylor Hand Function (Hand Movement)– ACMC (Activities of Daily Living)– UEFS (Ease of use functional survey)– TAPES-R (Psychosocial survey)

• Home/Weekly assessments:– Daily Journal Entries– TAC-1 and TAC- 3 assessments on Mobile App 2X/day – Quick DASH Survey

• Quarterly Home visits – Evaluation/observation of daily home use– Clinical feedback and recommendations for effective use– Review of metric completion and consistency– Collection of completed journal and surveys since last visit

Target Achievement Control Test

0 1 2 3 4 5 6 7 8

Time, sec

-50

0

50

100

Join

t Ang

le, d

eg

TAC-1 Angle Time History

16

Target Achievement Control Test• TAC-1 and TAC-3 adapted for

Mobile App• Graphic Interface showing position

indicator for each joint• Log Files automatically saved

17

Number of classes trained by date

1-3 grasps trained

Individual finger training aligns with the participants focus on the piano playing task

User trains approximately 8 total classes

Training accuracy of motion sets over time

This scatter plot shows increasing training accuracy for all motions, grasps and individual fingers over time indicating improved proficiency in training consistency over time.

The plot accounts for all recorded training sets and includes sets in which the participant may have chosen to retrain or add data to improve control.

Technical Challenges

20

• User reports frequent finger ‘drop-outs’– Mitigated with App-based soft reset– Mitigated with backup hand– Less prevalent without glove

• Battery System inconsistent- mitigated• Wireless comms are challenging in ‘noisy’ environments

– Limited range on App; Inconsistent EMG streaming over Bluetooth• Embedded System

– Intermittent Clock resets– Mitigated with mobile app based clock setting

• Mechanical Fuses Break frequently– Mitigated with resettable’ fuse

• Thermal Management – Mitigated with fan integration

Additional Back up slides

Targeted Muscle Reinnervation and Osseointegration:Enabling new levels of prosthetic control

Wireless Electrodes• Socket-free interface• Commercial off the shelf

(25 $/channel vs ~400 $/channel)

• Data Sampling• 8 bit• Up to 300 Hz raw EMG• Up to 4 devices simultaneously

• Built-in IMU for orientation

Dynamic Impedance• Dynamically Controlled

Joint Stiffness

• Reduces peak torque from 40 ft-lbs to

Hand and Tactile Sensors

OI Interface technology: Overview of the MPL Experience to dateRevolutionizing Prosthetics �MilestonesRevolutionizing ProstheticsSlide Number 4Enabling Osseointegrated ControlInterface and Wireless ControlInterface and Wireless controlUsage Data LoggingEnd-user interface: Mobile AppSummaryLessons learned: OsseointegrationNext StepsSlide Number 13Take home study-Back up/DataStudy Design - AssessmentsTarget Achievement Control TestTarget Achievement Control TestNumber of classes trained by dateTraining accuracy of motion sets �over timeTechnical ChallengesSlide Number 21Additional Back up slidesTargeted Muscle Reinnervation and Osseointegration:�Enabling new levels of prosthetic controlWireless ElectrodesDynamic ImpedanceHand and Tactile Sensors