TOWARDS CLINICALLY RELEVANT NEURAL PROSTHESES A ...shenoy/Theses/Nuyujukian.pdf · towards...

TOWARDS CLINICALLY RELEVANT NEURAL PROSTHESES

A DISSERTATION

SUBMITTED TO THE DEPARTMENT OF BIOENGINEERING

AND THE COMMITTEE ON GRADUATE STUDIES

OF STANFORD UNIVERSITY

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS

FOR THE DEGREE OF

DOCTOR OF PHILOSOPHY

Paul Nuyujukian

December 2012

http://creativecommons.org/licenses/by-nc/3.0/us/

This dissertation is online at: http://purl.stanford.edu/xm026gn9792

© 2012 by Paul Herag Nuyujukian. All Rights Reserved.

Re-distributed by Stanford University under license with the author.

This work is licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License.

ii



http://purl.stanford.edu/xm026gn9792

I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.

Krishna Shenoy, Primary Adviser


Kwabena Boahen


Jaimie Henderson

Approved for the Stanford University Committee on Graduate Studies.

Patricia J. Gumport, Vice Provost Graduate Education

This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file inUniversity Archives.

iii

Abstract

Neural prostheses translate signals from the brain into useful control signals, manipulating

end-effectors such as computer cursors or robotic arms. Their aim is to offer greater inter-

action with the world for patients suffering from limb dysfunction due to spinal cord injury,

neurodegenerative disease, and other conditions leading to limb paralysis. Prior intracor-

tical electrode neural prosthesis studies have demonstrated compelling proof-of-concept

systems, but barriers to successful clinical translation still remain, such as performance and

robustness. Measures of performance include the speed, accuracy, and bitrate of the sys-

tem. Robustness refers to the sustained performance of the system within a day and across

days. The work presented here demonstrates algorithms and advances for neural prostheses

that increase both performance and robustness. The recalibrated feedback intention trained

Kalman filter (ReFIT-KF) increased performance by at least twofold compared to previ-

ously reported decoders, approaching the speed of natural arm movements. It achieved

bitrates of up to 4.5 bits per second (bps) and communicationrates of up to 10 words per

minute (wpm) when used on a typing task. These results were reliable and repeatable for

hours at a time across 4 array-years between two subjects. Utilizing neural spike threshold

crossings as a signal source, the ReFIT-KF algorithm also demonstrated sustainable per-

formance without any changes to decoder weights for one yearwith a degradation rate of

0.05 bps per month.Performance further increased when the ReFIT-KF was combined with

an HMM state decoder for the detection of clicks, eliminating the need for hold periods.

This combined ReFIT-KF and HMM decoder achieved bitrates of up to 6.5 bps and 15

wpm. Taken together, these findings may help advance neural prostheses closer to clinical

viability.

v

Acknowledgements

For my parents, in recognition of their unyielding support.

vi

Contents

Abstract v

Acknowledgements vi

1 Introduction 1

1.0.1 Neural prostheses. . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.0.2 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Monkey Models 5

2.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 Rhesus monkey models. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3.1 Arm restrained, no EMG modulation. . . . . . . . . . . . . . . . 7

2.3.2 Arm restrained, no EMG measurement. . . . . . . . . . . . . . . 8

2.3.3 Arm not restrained, not visible. . . . . . . . . . . . . . . . . . . . 8

2.3.4 No arm movement, nerve block. . . . . . . . . . . . . . . . . . . 9

2.3.5 Optogenetics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3.6 Freely moving . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.6 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 A Closed-Loop Human Simulator 13

3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

vii

3.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.3 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.3.1 Neural Prosthetic Experimental Hardware. . . . . . . . . . . . . . 20

3.3.2 Neural Prosthetic Experimental Task. . . . . . . . . . . . . . . . . 23

3.3.3 Human Experiments. . . . . . . . . . . . . . . . . . . . . . . . . 27

3.3.4 Animal Experiments. . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3.5 Generating Synthetic Neural Activity. . . . . . . . . . . . . . . . 30

3.3.6 Decoding Neural Activity . . . . . . . . . . . . . . . . . . . . . . 33

3.3.7 Performance Analysis. . . . . . . . . . . . . . . . . . . . . . . . 34

3.4 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.4.1 Online Prosthetic Data. . . . . . . . . . . . . . . . . . . . . . . . 37

3.4.2 Comparing Online Analysis to Offline Analysis. . . . . . . . . . . 42

3.4.3 Comparing OPS to BMI. . . . . . . . . . . . . . . . . . . . . . . 46

3.5 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

3.5.1 The OPS for general system design questions. . . . . . . . . . . . 50

3.5.2 The OPS for specific algorithmic questions. . . . . . . . . . . . . 52

3.5.3 The OPS for neuroscientific questions. . . . . . . . . . . . . . . . 54

3.5.4 Cautionary notes regarding the OPS. . . . . . . . . . . . . . . . . 55

3.5.5 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57


4 ReFIT-KF Neural Prosthesis 58

4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3.1 Performance Overview. . . . . . . . . . . . . . . . . . . . . . . . 60

4.3.2 Generalization & Robustness. . . . . . . . . . . . . . . . . . . . . 63

4.3.3 Two Innovations . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.3.4 Innovation 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.3.5 Innovation 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

viii

4.5 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.6 Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

5 Task Optimization 76

5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.2 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5.3.1 Grid Task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

5.3.2 Radial Task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.3.3 Sustained intraday performance. . . . . . . . . . . . . . . . . . . 86

5.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

6 Sustained performance & typing 87

6.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

6.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

6.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

6.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

6.5 Methods Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94


7 Hidden Markov model ReFIT-KF 97

7.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

7.2 Decoder Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

7.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

7.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

8 Conclusion 104

A Publications 106

Journal papers 107

Conference papers 109

ix

B Patents 111

Bibliography 112

x

List of Tables

2.1 Current and emerging monkey models. . . . . . . . . . . . . . . . . . . . 8

3.1 Categorization of the data sets collected and analyzed inthis paper. The

Results show offline and offline analyses (blue and red in this table and in

subsequent figures), human and monkey subjects (left and right columns

in this table), synthetic and real neural data (identical toOPS and BMI

modes, the top two and bottom two rows here), and finally the continuous

and interleaved task variants (grouped within each cell of this table). These

permutations allow us to perform a breadth of comparisons and controls to

answer the key questions of this study.. . . . . . . . . . . . . . . . . . . . 28

4.1 Performance of ReFIT-KF based control with observation based algorithm

training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5.1 Acceptance region size as a function of targets on the screen for the grid task84

xi

List of Figures

3.1 Concept figure for Online Prosthetic Simulator opportunity. The x-axis

shows four testing paradigms in terms of increasing realism. Offline data

analysis is perhaps the least reasonable proxy to eventual user mode, as it

entirely neglects the closed-loop control. On the other endof the spectrum

is the human clinical trial, which is precisely the eventualuser mode. Left

axis (blue) shows the difficulty associated with testing each algorithm or

algorithmic parameter setting. Right axis (red) shows the number of algo-

rithm and parameter choices that are reasonably testable, given costs and

other constraints.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2 Experimental rig which can be used both for Online Prosthesis Simulator

(OPS) and real neural Brain Machine Interface (BMI) experiments. A hu-

man (shown in gray) or monkey subject reaches (red trace) in a3D volume

obscured from view. An overhead position tracker tracks endpoint kine-

matics. Control PCs process those data and render the subject’s real reach

(in control trials) or a prosthetic decoded reach (in OPS or BMI trials, the

red trace). Two monitors (blue) project a stereo 3D image onto mirrors

(virtual 3D environment shown at right). Also displayed is athe subject’s

reach target (green sphere).. . . . . . . . . . . . . . . . . . . . . . . . . 21

xii

3.3 OPS and BMI schematic.Left Panel: OPS Mode - (a) A healthy hu-

man subject or a monkey (human shown) makes real arm reaches in a 3D

reaching environment (see Figure 3.2). (b) Recorded kinematics of that

real reach are used to generate synthetic neural activity, which is then input

to the (c) Kalman filter neural decode algorithm. The decode algorithm

decodes synthetic neural activity into physical reaching behavior, and (d)

the decoded reach is then rendered back to the subject in the 3D visual en-

vironment, which allows the subject to bring to bear all of his/her online,

closed-loop control strategies to drive a desired reach.Right Panel: BMI

Mode - (a) A monkey makes real arm reaches. (b) The real neural activ-

ity associated with that reach is recorded, which is then input to the (c)

Kalman filter decode algorithm. The decode algorithm decodes real neural

activity into physical reaching behavior, and (d) the decoded reach is then

rendered back to the monkey in the 3D visual environment, which allows

similar closed-loop control as in OPS mode. It is important to note that the

OPS mode of the left panel can be used both by a human and a monkey

(this will provide an important validation of the OPS).. . . . . . . . . . . 25

xiii

3.4 Performance metrics for Human Online Prosthetic Simulator (Human OPS)

decode trials.Left Panel: Continuous Task variant - (a) Failure Rate: the

percentage of trials where the subject did not successfullyacquire and hold

the reach target. (b) Time to Target: the time required to reach the target for

successful trials. In both panels, data from five subjects are shown in light

colors. The average of all trials is shown in dark blue. This average shows

in both cases a significant linear trend indicating that smaller bin widths

will lead to better performance. (c) To compare offline and online data, we

use a third metric: integrated distance to target (failure rate and time to tar-

get can not be calculated for offline data, as offline failure rate approaches

100% without online feedback, which is again telling of the inappropriate-

ness of offline analysis). This metric is normalized by triallength (total

trial time). Right Panel: Interleaved Task variant - These show the same

metrics as the left panel, reinforcing with another task variant these trends.

In all panels, 95% confidence intervals (A: binomial distribution, B, C:

Gaussian) are shown as error bars.. . . . . . . . . . . . . . . . . . . . . . 38

xiv

3.5 Performance metrics for Monkey Brain Machine Interface (Monkey BMI)

and Online Prosthetic Simulator (Human OPS) decode trials.Left Panel:

BMI Continuous Task variant - (a) Failure Rate: the percentage of trials

where the subject did not successfully acquire and hold the reach target.

(b) Time to Target: the time required to reach the target for successful

trials. In both panels, data from five subjects are shown in light colors.

The average of all trials is shown in dark blue. This average shows in

both cases a significant linear trend indicating that smaller bin widths will

lead to better performance. (c) To compare offline and onlinedata, we use a

third metric: integrated distance to target (failure rate and time to target can

not be calculated for offline data, as offline failure rate approaches 100%

without online feedback, which is again telling of the inappropriateness of

offline analysis). This metric is normalized by trial length(total trial time).

Middle Panel: BMI Interleaved Task variant - These show the same

metrics as the left panel, reinforcing with another task variant these trends.

Right Panel: OPS Interleaved Task variant- The monkey can also run

the OPS task with synthetic neural data. This panel shows thesame metrics

as other panels, and these metrics all show similar trends. Bycomparing

BMI and OPS within subject on the same task (middle and right panels), we

have an indication that the OPS is providing a valuable proxyto real neural

prosthetic systems. In all panels, 95% confidence intervals(A: binomial

distribution, B, C: Gaussian) are shown as error bars.. . . . . . . . . . . . 41

xv

3.6 Comparing offline analysis to online analysis in Human OPSmode. Left

Panel: Continuous Task variant- The blue error bars are replicated from

Figure 3.4, Panel C. The dark blue line is a linear fit of that data, show-

ing a significant performance trend indicating that shorterbin widths imply

better performance. A light blue quadratic fit (not visibly or statistically

different from a line) to the same data is obscured by the linear fit. In

red, we perform the same analysis offline, using the real reaching sets from

these users. We perform offline decodes at each bin width, allowing us to

generate the same mean integrated distance to target metric. These error

bars are the offline analogs to the blue error bars. Not surprisingly, offline

analysis has worse performance than online, since there wasno benefit of

feedback. More significantly, however, is the shape of the data. This offline

analysis suggests statistically significant performance optima of roughly

100-150ms. This can be seen in the significant quadratic fit tothe data

(dark red). The linear fit (light red) does a clearly poorer job of fitting

the data. Note that the characteristic “U-shape” of the offline data tells a

very different story than the online analysis, indicating the important dif-

ferences between these two testing paradigms.Right Panel: Interleaved

Task variant - The same analysis and implication holds true as in the left

panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

xvi

3.7 Comparing offline analysis to online analysis in Monkey BMIand Monkey

OPS modes.Left Panel: BMI Continuous Task variant - The blue error

bars are replicated from Figure 3.4, Panel C. The dark blue line is a linear fit

of that data, showing a significant performance trend indicating that shorter

bin widths imply better performance. A light blue quadraticfit (not visibly

or statistically different from a line) to the same data is obscured by the

linear fit. In red, we perform the same analysis offline, usingthe real reach-

ing sets from these users. We perform offline decodes at each bin width,

allowing us to generate the same mean integrated distance totarget metric.

These error bars are the offline analogs to the blue error bars. Not sur-

prisingly, offline analysis has worse performance than online, since there

was no benefit of feedback. More significantly, however, is the shape of

the data. This offline analysis suggests statistically significant performance

optima of roughly 150-200ms. This can be seen in the significant quadratic

fit to the data (dark red). The linear fit (light red) does a clearly poorer

job of fitting the data. Note that the characteristic “U-shape” of the offline

data tells a very different story than the online analysis, indicating the im-

portant differences between these two testing paradigms.Middle Panel:

BMI Interleaved Task variant - The same analysis and implication holds

true as in the left panel.Right Panel: OPS Interleaved Task variant-

Again, the same analysis and implication holds true as in theleft panel.

Taken together, these three panels show that both the BMI and OPS modes

tell the same story: shorter bin widths imply better performance, whereas

the offline analyses indicate an incorrect trend that leads to varying and

misleading performance optima.. . . . . . . . . . . . . . . . . . . . . . . 45

xvii

3.8 Summary of online/offline differences; summary of OPS/BMI similarities.

In the previous figures, we fit quadratic curves to both the offline and online

performance data. Here, we plot those quadratic regressioncoefficients

and their 95% confidence intervals. Any confidence interval overlapping

0 should be read as, ”this data is not significantly differentthan a linear

trend.” This figure shows that nearly all online analyses in monkey and hu-

man do not have significant quadratic terms, agreeing with the implication

from all previous data that shorter bin widths lead to betterperformance.

Comparing that to the red error bars, again the central point is reiterated:

online analysis and offline analysis (whether in real neuraldata or in syn-

thetic neural data) tell very different stories. Furthermore, in addition to

online and offline being very different, we see that OPS and BMIare in

fact very similar. The OPS gives a valuable means by which to sweep per-

formance based on this algorithmic parameter.. . . . . . . . . . . . . . . 47

4.1 Performance comparison.Native arm control shown in blue, ReFIT-KF

in red, and Velocity-KF in green. (a) Representative traces of cursor path

during center-out-and-back reaches. Dashed lines (not visible to the mon-

key) are the demand boxes for the eight peripheral targets and the central

target, shown as translucent green circles. (b) Bar graphs plotting maxi-

mum deviation from a straight-line path to the target on eachsuccessful

trial (mean± SE). (c,d) Histograms of time to target for successful trials

are shown as line graphs for monkeys J and L. The inset bar graphs plot the

time to target (mean± SE). (e,f) Line graphs plotting the mean distance to

the target as a function of time. The inset bar graph plots themean± SE

of the dial-in time, or the time required to finally settle on the demand box,

after first acquired, to successfully hold for 500 ms. The thickened portion

of the line graphs also indicate dial-in time, beginning at the mean time of

first target acquire, and ending at mean trial duration minus500ms. Data

from experiments J & L 20101027-29, 20101102.. . . . . . . . . . . . . . 61

xviii

4.2 Performance of ReFIT-KF control across 4 years.Performance is mea-

sured by the Fitts’ law metric. Data from monkey J and monkey Lare

shown as 94 orange circles and 182 cyan squares, respectively. Each point

plots the performance of the ReFIT-KF algorithm trained on that experi-

mental day. The eight filled data points (four for each monkey) are calcu-

lated from the same datasets used to generate Figure 4.1. Linear regression

lines for data from monkey J (orange) and monkey L (cyan) are shown. The

slope of the regression line for monkey J is not statistically significant from

zero (p>0.34). The slopes for monkey L is statistically significant from

zero (p<0.001). For all datasets shown, the trial success rate was>90%. . . 62

4.3 Performance comparison of native arm vs. ReFIT-KF for the pinball

task. Native arm control is in blue, ReFIT-KF in red. In this task, each tar-

get location is selected from a uniform distribution spanning the workspace.

(a) Each column shows data from 20-minute segments. The top rows are

randomly selected cursor traces for 4 subsequent target acquisitions. Target

demand boxes are shown as dashed lines and target sequence isindicated

from 0 to 4. The bottom row shows normalized histograms of time to target

for successful trials. Arrows below the plot indicate average time. (b) Tar-

get acquisition rate per minute throughout the sessions is shown. The sharp

rate drop indicates when the monkey lost interest in the task. A histogram

of acquisition rate across the sessions is inset. The nativearm and ReFIT-

KF sessions (L-2010-04-01 and L-2010-04-12) were on two separate days,

within 11 days of each other, when the monkey demonstrated a high degree

of motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

xix

4.4 Performance comparison of native arm vs. ReFIT-KF for the obstacle

avoidance task. Native arm control shown in blue, ReFIT-KF in red. In

this task the monkey had to move the cursor from the initial target (labelled

0) to the final target (labelled 1, demand box shown as dashed line) with-

out hitting the magenta-colored barrier. One representative cursor trace is

shown from each of the four principle observed movement types: curve un-

der, curve over, straight (no barrier), and collision into barrier. These data

are from experiment J-2010-03-09.. . . . . . . . . . . . . . . . . . . . . . 65

4.5 Illustrations of the online neural control paradigm and the ReFIT-KF

training methodology. (a) The input to the control algorithm at timet is a

vector of spike counts,yt , from implanted electrodes.yt is translated into

a velocity output,vt , to drive the cursor. (b) ReFIT-KF is trained in two-

stages. Initially, cursor kinematics and neural activity are collected during

arm control or during an observation phase in which cursor movement is

automated (i). These arm movement or observed cursor kinematics (ii) are

regressed against neural activity to generate an initial control algorithm.

Then, a new set of cursor kinematics and neural activity are collected using

the initial algorithm in closed loop (iii). The kinematics collected during

neural control (red vectors in (iv)) are used to estimate intention by rotat-

ing the velocities towards the goal (cyan vectors in (iv)). This estimate of

intended kinematics is regressed against neural activity to generate and run

ReFIT-KF (v). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.1 Experimental setup. Monkey was placed in front of a Wheatstone stereo-

graph [133] such that hand was not visible and presented witha virtual en-

vironment within which to interact. The monkey’s objectivewas to acquire

the target lit in green. Neural data was recorded simultaneously during the

task. Under neural prosthesis operation, the movement of the cursor was

determined by the decoder.. . . . . . . . . . . . . . . . . . . . . . . . . . 78

xx

5.2 Task layouts.a Grid task layout in a 25 target configuration. The accep-

tance windows are adjacent and non-overlapping.b Radial task layout for

8 targets. In the radial task, there were regions that were left as gaps in

the workspace to reduce inadvertent selection. In both tasks, the dotted

lines that are not shown to the monkey outline the acceptanceregion for a

particular target.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5.3 Experimental data of grid task. Monkey J: a-c. Monkey L: d-f. a Acquire

time histogram of grid task with task parameters of 25 targets and 450 ms

hold time.b Bitrate heatmap of task parameters swept for grid task. Each

point in black represents a tested pair of task parameters.c Information

plot of sustained performance across a day for grid task. Thegray line

plots the average bitrate as a function of time and the black line represents

the cumulative information transmitted over the duration of the experiment.

d Acquire time histogram of grid task with 25 targets and 450 mshold time.

eBitrate heatmap as in b.f Information plot as in c.. . . . . . . . . . . . . 82

5.4 Experimental data of radial task. Monkey J: a-c. Monkey L: d-f. a Acquire

time histogram of radial task with task parameters of 8 targets and 9 cm

distance.b Bitrate heatmap of task parameters swept for radial task. Each

point in black represents a tested pair of task parameters.c Information plot

of sustained performance across a day for radial task. The gray line plots

the average bitrate as a function of time and the black line represents the

cumulative information transmitted over the duration of the experiment.d

Acquire time histogram of radial task with 8 targets and 7 cm distance.e

Bitrate heatmap as in b.f Information plot as in c.. . . . . . . . . . . . . . 83

5.5 Bitrate plots for the grid task when the number of targets was varied.a

Monkey Jb Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

5.6 Bitrate plots for the grid task when the hold time was varied. a Monkey J

b Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.7 Bitrate plots for the radial task when the number of targets was varied.a

Monkey Jb Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

xxi

5.8 Bitrate plots for the radial task when the distance to the targets was varied.

a Monkey Jb Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

6.1 Decoding without retraining.a Daily performance plot of 14 day long ex-

periment with Monkey J. The gray line plots the instantaneous, smoothed

information rate across time. The black line plots the cumulative informa-

tion transmitted across the experiment. The x-axis indicates the time the

experiment was run, each tick mark notes an hour in the day.b Similar

performance plot for Monkey L. Plots are from datasets J110926-J111014

& L110912-L110925.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

6.2 Typing task.a Plot of instantaneous, smoothed typing rate across time for

Monkey J while typing an article. Gaps in trace represent times where the

monkey paused and was not typing.b Histogram of words per minute for

Monkey J, same data as in a.c & d Same as in subpanels a & b, but for

Monkey L. Typing data from from datasets J111021 & L111103.. . . . . . 91

6.3 Historical models.a Longitudinal study showing performance of Mon-

key J with a static decoder over one year. The filled in circlesdenote

performance from the first month of experiments with a singledecoder

(Fig 6.1a). That decoder, held constant, was tested over a year. The solid

line was regressed against the performance across a year andis the equa-

tion: Information rate= −0.034· (# months)+3.78bpsb Cross-sectional

historical study of decoder performance from previously recorded data in

Monkey J. Each column of points represents a single model tested. Each

color/symbol represents an experimental day. The peformance at time 0

represents the performance of the decoder on that experimental day. The

black line shows the average performance for each model.c Same as

in subpanel b, but for Monkey L. Panels b & c from datasets J120117-

J120120, L120207-L120210, & L120214.. . . . . . . . . . . . . . . . . . 93

xxii

6.4 Tuning Analysisa Tuning curves of top 5 contributory channels for a 6

month-old decoder for Monkey J. The number to the left of the axis denotes

the channel. Each colored curve represents a different experimental day.

The black curve represents the tuning on the day the decoder was generated

(J110722). The straight lines represent the channel’s cosine fit preferred

direction. b Same tuning curves as in subpanel a, but for Monkey L. The

original decoder was from L110805.c Bar graph of contribution-scaled,

per-channel rotation in tuning for Monkey J. The first columnplots the

unsigned weighted summed rotation of all channels while thesecond plots

the signed net rotation.d Same plot as in subpanel e, but for Monkey L.e

Plot of individual contribution of top 30 channels for Monkey J (red) and L

(blue). f Plot of cumulative sorted per channel contribution to decoder. All

plots derived from same datasets as in Figure 6.3.. . . . . . . . . . . . . . 95

7.1 Two state transition graph. Graphical model of the transition flow for the

two state HMM used with Monkey J. The system starts in the movestate

and can stay in that state at the next timestep. It can likewise transition to

the stop state if there is sufficient information from the neural emissions

matrix to push it to that state. If the system stays in the stopstate for two

timesteps, the HMM notes a click and immediately transitions back to the

move state. The state is locked in the move state for a lockouttime (around

250 ms), before it is allowed to transition out of the move state again. . . . 99

xxiii

7.2 Four state transition graph. Graphical model of the transition flow for the

four state HMM used with Monkey L. The system starts in the slow state.

The system can stay in any state (except stop) for an arbitrary number of

timesteps and can transition to the idle state from any state. Transitions

from the movement based states (slow 1, fast, slow 2) are onlyallowed to

proceed in a stepwise fashion. From the slow state, if the kinematic profile

(magnitude velocity) is above the specified threshold, the system transitions

to the fast state. Once the kinematic profile falls below the specified thresh-

old, the state transitions to the second slow state. The system can transition

to the stop state Only from the second slow state. Upon transition to the

stop state, the system communicates a click signal and immediately returns

to the first slow state. As in the two state HMM, there is a lockout time

enforced before a second stop state can be entered.. . . . . . . . . . . . . 100

7.3 Bitrate for Monkey J with HMM ReFIT-KF decoder. Plot of bitrate

over time for two different days(a) and aggregate bitrate histogram(b) for

the two state HMM ReFIT-KF decoder with Monkey J on a randomized

target grid task. The task was a 7x7 grid with 49 targets and was taken

from datasets J120405 dn J120406.. . . . . . . . . . . . . . . . . . . . . . 101

7.4 Bitrate for Monkey L with HMM ReFIT-KF decoder. Plot of bitrate

over time for three different days(a) and aggregate bitrate histogram(b) for

the four state HMM ReFIT-KF decoder with Monkey L on a randomized

target grid task. The task was a 5x5 grid with 25 targets and was taken from

datasets L120426 L120427 and L120430.. . . . . . . . . . . . . . . . . . 101

7.5 Typing rate for Monkey J with HMM ReFIT-KF decoder. Plot of typing

rate over time(a) and typing rate histogram(b) for the two state HMM

ReFIT-KF decoder with Monkey J. The task was a 6x6 grid with 36 targets

and was taken from datasets J120409.. . . . . . . . . . . . . . . . . . . . 102

7.6 Typing rate for Monkey L with HMM ReFIT-KF decoder. Plot of typ-

ing rate over time(a) and typing rate histogram(b) for the four state HMM

ReFIT-KF decoder with Monkey L. The task was a 6x6 grid with 36 targets

and was taken from datasets L120501.. . . . . . . . . . . . . . . . . . . . 102

xxiv

Chapter 1

Introduction

1.0.1 Neural prostheses

Neural prostheses are systems that translate neural signals into control signals for pros-

thetic devices. These neural signals could be from electroencephalography, [5,72,82] elec-

trocorticography, [8, 78, 84, 105, 106] or the intracortical electrodes as presented in this

work. [12, 57, 58, 94, 102, 109, 124, 130, 132] The target prosthetic devices include com-

puter cursors, keyboards, robotic arms, or any other end effector that could be manipulated

by these control signals. There have been compelling proof-of-concept demonstrations of

these systems, but there are still significant barriers to clinical translation. [52, 63] These

include issues of limited performance, poor robustness, and the need for frequent system

retraining. Performance is measured by values such as trialsuccess rate, straightness of

movements, and information theoretic measures such as bitrate. Robustness refers to the

sustained performance of the system in two domains: sustainable performance within days

(intraday) and across days (interday). Neural prostheses to date have shortcomings in all of

these domains that may need to be improved to make the successful translation to the clinic.

Presented here is the development of an intracortical neural prosthesis system evaluated in

a rhesus monkey model that may address some of these issues. The recalibrated feedback

intention trained Kalman filter (ReFIT-KF) improves the performance of neural prostheses

in a two-dimensional cursor task by at least twofold over current systems, enabling faster,

straighter, and more accurate cursor movements. When evaluated on an optimized grid task

1

2 CHAPTER 1. INTRODUCTION

that enabled the measurement of throughput via achieved bitrate, the ReFIT-KF achieved

rates of up to 4.5 bits per second (bps). This high performance was sustainable for several

hours at a time. Further, when evaluated for performance without retraining, the ReFIT-KF

can sustain performance for a year without any changes to thedecoder with degradation

rate of 0.05 bps per month, demonstrating long-term robustness. The ReFIT-KF achieved

a typing speed of up to 10 words per minute (wpm) when used to simulate typing on a grid

keyboard layout. When combined with an HMM used to decode a click state, performance

further increased. This combined ReFIT-KF HMM achieved a throughput of 6.5 bps and

a typing speed of 15 wpm. Taken together, these advances may increase the performance

and robustness of neural prostheses and help increase theirclinical viability.

1.0.2 Outline

In compiling this dissertation, the intent was to present a coherent and meaningful story

about the development of high-performing neural prostheses developed in the Shenoy Lab.

Over the course of the past four years, the lab has developed novel techniques and al-

gorithms for neural prostheses that enabled higher performing systems. This outline will

detail the chapters of the dissertation, briefly introducing the main ideas and providing con-

text as how how concepts and ideas from the prior chapters were incorporated in subsequent

chapters.

Monkey models

This first chapter details the rationale behind the use of monkeys as an appropriate

animal model when developing clinical neural prostheses. It discusses the different

monkey animal models that have been used and their relative benefits and shortcom-

ings. It also details why the work presented in this dissertation has been based off of

the single arm-restrained monkey model, as it provides the least constraints on the

dimensions that the neural state can explore.

A closed-loop human simulator

This second chapter describes a series of offline simulations and online experiments

using both humans and monkey subjects to evaluate the behavior of neural prosthe-

ses. These experiments highlighted the discrepancy between offline predictions and

3

optimal online performance. Specifically, using this simulator system, it was deter-

mined that shorter neural bin times yield higher performingsystems. The findings of

this study helped lay the foundations for subsequent decoder development.

ReFIT-KF neural prosthesis

This chapter details the development and testing of the ReFIT-KF. This algorithm

performed at least twice as well as prior continuously controlled endpoint decoders.

The ReFIT-KF is comprised of two innovations that led to this increase in perfor-

mance. The first is the cursorGoal correction in which the training set is manipulated

based on assumed kinematic intentions before building the decoder. The second is a

modification to the decoder in which the position of the cursor on screen is accounted

for and removed from the uncertainty of the state estimationin the decoder. Taken

together, these two innovations lead to increased performance on a cursor task and

also enable repeatable performance over years that is sustainable for hours a day.

Task optimization

This chapter describes two new keyboard-like tasks that were used to evaluate the

achievable bitrate from the ReFIT-KF. The two tasks presented were a grid keyboard

task and a radial keyboard task. For each task, the two primary parameters that govern

the task layout were manipulated in an effort to find the optimal task parameters that

maximize bitrate. It was found that the overall bitrate of the decoder did not vary

largely between tasks, as the maximum bitrate for a given subject was similar in both

tasks, with a maximum bitrate of 4 bps. These optimal task parameters were then

used as the basis for future experiments.

Sustained performance and typing

This chapter describes the performance of the ReFIT-KF on a grid keyboard task

using the optimal parameters found from the prior chapter. Specifically, the exper-

iments looked at sustained performance without retrainingsessions. One subject

demonstrated sustained performance of a single set of decoder weights for a year,

achieving a maximum rate of 4.5 bps with a degradation rate of0.05 bps per month.

Further, when the ReFIT-KF was used to transmit text, it achieved rates of up to

10 wpm.

4 CHAPTER 1. INTRODUCTION

HMM-ReFIT-KF decoder

This chapter details the decoding of a click signal using a hidden Markov model state

decoder running in parallel to the ReFIT-KF that controls cursor velocity. Using these

two decoders simultaneously enables the elimination of thehold time requirement

present in all the work presented thus far. Eliminating holdtime speeds up the rate

of selection and could increase performance. On a grid task,bitrates of up to 6.5 bps

were achieved and typing rates of up to 15 wpm. These represent a 40-50% increase

in performance over the ReFIT-KF decoder when used alone.

Chapter 2

Monkey Models

This chapter was a conference paper at the 33rd annual IEEE EMBS Conference in 2011

titled “Monkey models for brain-machine interfaces: the need formaintaining diversity.”

It was authored by Paul Nuyujukian,* Joline Fan,* Vikash Gilja, Paul Kalanithi, Cynthia

Chestek, and Krishna Shenoy. The author’s contribution to this work was in drafting and

writing the manuscript.

2.1 Abstract

Brain-machine interfaces (BMIs) aim to help disabled patients by translating neural signals

from the brain into control signals for guiding prosthetic arms, computer cursors, and other

assistive devices. Animal models are central to the development of these systems and have

helped enable the successful translation of the first generation of BMIs. As we move toward

next-generation systems, we face the question of which animal models will aid broader

patient populations and achieve even higher performance, robustness, and functionality.

We review here four general types of rhesus monkey models employed in BMI research,

and describe two additional, complementary models. Given the physiological diversity

of neurological injury and disease, we suggest a need to maintain the current diversity of

animal models and to explore additional alternatives, as each mimic different aspects of

injury or disease.

5

6 CHAPTER 2. MONKEY MODELS

2.2 Introduction

Neurological injury and disease often result in the permanent loss of motor and sensory

function. In some cases the disability is so severe that it isnot possible to feed oneself or

even communicate. BMIs are a new class of electronic medical systems that aim to improve

the quality of life for disabled patients. These systems interface with the central nervous

system and use neural signals from the brain to control prosthetic devices (e.g., [52]). In

recent years, first-generation BMIs developed with rhesus monkey animal models have

translated from the laboratory into an FDA Phase-I clinicaltrial focused on tetraplegics

and ALS patients (BrainGate I & II, [31,56,57,69,70,117]).

Next-generation BMIs aim to further improve the quality of life of disabled patients

and expand to larger patient populations, such as amputees and paraplegics. These next-

generation systems may meet this goal if performance, robustness, and functionality can

be increased substantially [44, 98]. But what types of monkey models are best suited to

develop next-generation BMIs? It is currently unclear if a single animal model can capture

all aspects of injury and disease or mimic different physiological conditions observed in

patient populations. Thus, we suggest that maintaining andexpanding the diversity of

available monkey models is critical.

We briefly review four major existing rhesus models, and briefly describe two emerging

models. We note that there are several other models not covered here, that either exist or

are possible. These models may employ pharmacological, lesioning, or electrical micro-

stimulation methods. Together, this set of animal models should help provide a diverse

resource for laboratory research being conducted, which may prove essential for the suc-

cessful translation of next-generation BMIs.

2.3 Rhesus monkey models

There appear to be four widely used rhesus monkey models. These have been employed

in recent BMI laboratory experiments, and are listed in Table2.1. We note that the animal

model used while training these systems may be different from the animal model used

while evaluating BMI performance. However, for simplicity,we have categorized these

2.3. RHESUS MONKEY MODELS 7

references based only on the animal model used in evaluationof BMI performance.

2.3.1 Arm restrained, no EMG modulation

In the first model, the monkey’s contralateral arm is restrained and monitored for task-

modulated muscle activity1. The absence of this muscle activity does not require enforcing

electromyography (EMG) silence, but rather that EMG activity not change within or across

behavioral trials. This model has been employed by Schwartzand colleagues, where it was

used in a control experiment to see if neural cursor control could be maintained without arm

movements or EMG modulation [124]. This model has also been employed by Nicolelis,

Carmena, and colleagues [12, 38] as well as by Andersen, Shenoy, and colleagues, al-

beit measuring EMG in separate experiments to assure no EMG modulation during an

“instructed-delay” period [88,102].

This monkey model “looks like a paralyzed patient,” which may be beneficial to BMI

research insofar as there are no arm movements or muscle contractions that would lead

to sensory signals. These signals, including vision, somatosensation, and proprioception,

could feed back to the cortical area driving the BMI and could influence BMI control.

Such sensory input could be viewed as a potential confounding factor, as it is presumably

not present in paralyzed patients as a potential source of useful information, or it could be

viewed as an important opportunity for increasing performance [122]. There also appear

to be two open questions with this model. The first is whether the range and pattern of

possible neural activity is constrained, by virtue of the animal being restricted to not move

the arm or modulate EMG activity. Paralyzed patients would presumably not have this

neural constraint since the injury or disease prevents neural activity from reaching muscles,

regardless of its range or pattern. Second, we ask whether all relevant muscle groups can be

monitored to assure no task-relevant EMG modulation given that the homunculus is highly

fractured and individual neurons may respond with respect to multiple muscle groups. This

is a practical concern, but one that is brought into focus by recent studies highlighting how

individual neurons in the nominal arm area of primary motor cortex contain considerable

1The contralateral arm with respect to the electrode-array implant. The ipsilateral arm is also often re-strained since motor cortical activity is also related to ipsilateral arm movements (e.g., [39]) However theipsilateral arm’s EMG activity is seldom if ever measured.


Table 2.1: Current and emerging monkey models

Current Monkey Models ReferencesA) Arm restrained, no EMG modulation [12,38,88,102,124]B) Arm restrained, no EMG measurement[36,116,130]C) Arm not restrained, not visible [46,47,87,91,99,109,116,124]D) No arm movement, nerve block [86,96]

Emerging Monkey Models ReferencesE) Optogenetics [9,28,44,53]F) Freely moving [16,34,45,61,83]

hand and finger movement related activity [129].

2.3.2 Arm restrained, no EMG measurement

The second model again restrains the monkey’s contralateral arm, but does not measure

EMG activity. Some finger, hand, and arm movement as well as force generation is

therefore allowed. This model has been employed by Schwartz, Vaadia, and colleagues

[36,116,130].

This monkey model “looks a bit less like a paralyzed patient”because there may be

some visible movement. This allowed movement may be beneficial, as described above,

because the neural population under observation may be lessconstrained. This second

model also recognizes, implicitly, that it is challenging to record from all relevant muscles

to confirm that there is no EMG modulation. Instead, small movements and small amounts

of force production are allowed. An open question with this model is whether muscle co-

contraction, isometric force production against the arm restraint, and/or small movements

are producing sensory signals that could feed back to the cortical area driving the BMI.

2.3.3 Arm not restrained, not visible

The third model does not restrain the monkey’s contralateral arm. It also does not allow

the arm to be viewed by the animal, as was the case for the first two monkey models. This

2.3. RHESUS MONKEY MODELS 9

model has been employed by Schwartz, Donoghue, Andersen, Vaadia, Batista, Yu, Shenoy,

and colleagues [46,47,87,91,99,109,116,124].

This monkey model “looks less like a paralyzed patient” because there is frank visi-

ble (to an observer, not the animal) movement. This movementmay again be beneficial

because the neural population under observation may be lessconstrained or altogether un-

constrained. There appear to be two open questions with thismodel. The first is how

to interpret the presumed presence of proprioceptive and somatosensory signals that feed

back to the cortical area driving the BMI, as a result of arm movements. Second, several

groups have observed that sometimes monkeys stop moving their arms and the BMI con-

tinues to operate. It would appear that monkeys can continueto operate what is in essence

the first or second animal model after voluntarily transitioning to keeping the arm motion-

less [12,87,109]. This may suggest that the difference between these three animal models

is not large from the perspective of BMI algorithm design and operation.

2.3.4 No arm movement, nerve block

The fourth monkey model employs a local anaesthetic to blockmuch, if not all, of the

efferent motor signals going to the arm as well as afferent sensory signals coming from the

arm. This is a newer model and has been employed by Fetz, Miller, and colleagues [86,96].

This monkey model “looks like a paralyzed patient” since thearm, hand, and fingers

are (temporarily, reversibly) paralyzed and sensory information is substantially attenuated.

As such, this model may closely mimic a spinal cord injury patient. This model may also

be beneficial, as described above, because the neural population under observation may be

less constrained or altogether unconstrained. There appear to be several open questions

with this model, largely due to its recent development. The first stems from the observation

that nerve blocks are peripheral nerve interventions, whereas spinal cord injury patients

have lesions in the central nervous system. Second, the degree of afferent activity block is

presumably difficult to verify (i.e., daily nerve conduction studies are not feasible). Third,

nerve block is a temporary, acute intervention, whereas spinal cord injury is a chronic

condition. As a result, adaptive or deteriorative changes in the cortex of a patient with

spinal cord injury are presumably not modeled. The daily novelty of paralysis may also


be a distraction to the animal and could even present cue conflict (i.e., can see the arm,

but can’t move or feel it). These factors could lead to different cortical neural activity than

would be present in the chronic patient case. Fourth, the pharmacokinetics of the local

anaesthetic result in a gradual return of motor and sensory function over the course of a

long experiment. This is potentially complicated by the difficulty of verifying the degree

of sensory block and mitigating its day-to-day variability.

Finally, the technical complexities are non-trivial, including surgically implanting a

nerve cuff and drug reservoir, injection-filling the reservoir periodically, and animal hus-

bandry following experiments. While not necessarily a limitation, this additional technical

complexity is a consideration when selecting animal models. As discussed below, optoge-

netic and freely moving monkey models also have additional technical complexity.

2.3.5 Optogenetics

One emerging model that could potentially (temporarily, reversibly) emulate paralysis,

stroke, or other disorders is an “optogenetic monkey model.” Optogenetics provides a

method for exciting or inhibiting neural activity while simultaneously recording, and does

so with high spatial (individual neuron, specific neuron types, specific projection pattern)

and temporal (millisecond timescale) resolution. Severaloptogenetic methods have re-

cently been translated to rhesus monkeys [9,28,53]. With these methods it may be possible

to use light to reversibly mimic injury and disease: “synthetic cord injury” (e.g., inhibit

activity in descending motor fibers with NpHR), “synthetic stroke” (e.g., inhibit activity

within an illumination-defined volume of gray matter with NpHR), and “synthetic spastic-

ity” (e.g., excite activity at specific rhythms with ChR2, or elevate baseline activity with

SFO). It may also be possible to use these methods as part of the BMI system itself to, for

example, “write in” artificial sensory information coming from sensors built in to prosthetic

hands [44].

2.3.6 Freely moving

Another emerging model that could potentially be used to understand more naturalistic

and freeform movements, and is relevant for designing BMIs toassist amputees who live

2.4. DISCUSSION 11

active lives, is a “freely moving monkey model.” It is now possible to build miniature head-

mounted systems to record from multi-electrode arrays, transmit these data wirelessly to

a nearby receiver, and use modern computer vision and markerless motion tracking tech-

niques for high-precision behavioral measurement [16, 45, 83]. In combination, this en-

ables monkeys to move freely and perform a variety of naturalistic tasks while studying

how populations of neurons control movement. This could lead to a more comprehen-

sive understanding of cortical motor control across a wide range of naturalistic movements

and behavioral contexts, and thereby better mimic the lifestyle of arm amputees as well as

model healthy humans [34]. The freely moving monkey model also enables studies explor-

ing chronic neural stimulation, including stimulation contingent on specific neural activity,

and has been employed to demonstrate motor plasticity [61].

2.4 Discussion

A range of rhesus monkey models is currently being employed to help advance BMI de-

sign, and their translation to disabled patients. First-generation BMIs have shown substan-

tial promise and are currently part of a clinical trial. Withthe desire to provide even greater

benefit to the current patient populations, as well as to helpmore patients who suffer from a

wider range of disabilities, next-generation systems are coming into focus. A central ques-

tion to this new endeavor is what types of rhesus models are currently available, and what

new models might be needed. We have attempted to provide a brief review of the existing

models, categorized into four broad types, as well as an overview of two emerging monkey

models currently under development. Two additional pointsare worth highlighting.

First, animal models are essential and appear to be working fairly well. As such, this

diversity of models ought to be maintained. BMIs are still in their early days, and consider-

able additional basic science, basic engineering, and pre-clinical testing are essential. The

current animal models have already led to a clinical trial and, equally importantly, the basic

BMI system architecture appears to work with little design modification when switching

between animal models. This suggests that rhesus monkey models are both appropriate

to translational BMI efforts, and that such a diverse set of animal models may map well

across a range of physiological conditions. It could be thatthe different animal models


each mimic a different neurological injury or disease, and having a set of BMIs that each

operate well with one or more of these models is prudent.

Second, we ought to not only maintain the diversity of rhesusmodels, but also continue

to investigate new ones. These new animal models may providethe field with useful plat-

forms for examining various pathological presentations. It is important to recognize that

BMIs aim to assist patients with a variety of different neurological injuries and disease. A

wide variety of pathology can result in debilitating loss ofmotor function while preserving

cortical areas. Although cervical spinal cord injury is theprototypical example, traumatic

injury can occur anywhere along the pathway, from cortex to subcortical structures to the

distal limb. Many other mechanisms for loss of function alsoexist, including neurodegen-

erative diseases, autoimmune conditions, neuropathies, and myopathies. This suggests that

different monkey models may be needed to mimic these varied pathologies. Even within

the sub-class termed “upper spinal cord injury” there is a whole spectrum of actual injuries

and associated dysfunctions that may leave a patient with varied impairments from moder-

ate paresis to full paralysis, with or without concomitant sensory deficits. Similarly, then,

a rich and growing spectrum of monkey models is presumably required to cover the range

of spinal cord injuries as, ultimately, it is unlikely that asingle monkey model will suffice.

2.5 Conclusion

A range of rhesus monkey models currently exists for BMI research, and they can be

broadly categorized into four types. Two additional types of rhesus models that are emerg-

ing were also reviewed briefly. We suggest that this diversity of models is important,

should be maintained, and expanded as part of the overall effort to design and translate

next-generation BMIs.

2.6 Acknowledgments

We thank the members of the Neural Prosthetics Systems Laboratory at Stanford University

for valuable scientific discussions and editorial feedback.

Chapter 3

A Closed-Loop Human Simulator

This chapter appeared as a paper in the Journal of Neurophysiology in 2010 titled “A

closed-loop human simulator for investigating the roles offeedback-control in brain-machine

interfaces.” It was authored by John Cunningham, Paul Nuyuyujukian, Vikash Gilja, Cyn-

thia Chestek, Stephen Ryu, and Krishna Shenoy. The author’s contribution to this work

was in experimental design, data collection and analysis, and assiting in writing the paper.

3.1 Abstract

Neural prosthetic systems seek to improve the lives of severely disabled people by decoding

neural activity into useful behavioral commands. These systems and their decoding algo-

rithms are typically developed “offline,” using neural activity previously gathered from a

healthy animal, and the decoded movement is then compared tothe true movement that

accompanied the recorded neural activity. However, this offline design and testing may

neglect important features of a real prosthesis, most notably the critical role of feedback-

control, which enables the user to adjust neural activity while using the prosthesis. We hy-

pothesize that understanding and optimally designing high-performance decoders require

an experimental platform where humans are in closed-loop with the various candidate de-

code systems and algorithms. It remains unexplored the extent to which the subject can,

for a particular decode system, algorithm, or parameter, engage feedback and other strate-

gies to improve decode performance. Closed-loop testing maysuggest different choices

13

14 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR

than offline analyses. Here we ask if a healthy human subject,using a closed-loop neural

prosthesis driven by synthetic neural activity, can informsystem design. We use this On-

line Prosthesis Simulator (OPS) to optimize “online” decode performance based on a key

parameter of a current state-of-the-art decode algorithm -the bin width of a Kalman filter.

First, we show that offline and online analyses indeed suggest different parameter choices.

Previous literature and our offline analyses agree that neural activity should be analyzed

in bins of 100-300ms width. OPS analysis, which incorporates feedback-control, sug-

gests that much shorter bin widths (25-50ms) yield higher decode performance. Second,

we confirm this surprising finding using a closed-loop rhesusmonkey prosthetic system.

These findings illustrate the type of discovery made possible by the OPS, and so we hy-

pothesize that this novel testing approach will help in the design of prosthetic systems that

will translate well to human patients.

3.2 Introduction

Debilitating conditions like spinal cord injuries can leave a human without voluntary mo-

tor control. However, in many cases, the brain itself maintains normal function. Millions

of people worldwide suffer motor deficits due to these diseases and injuries that result in

significantly diminished ability to interact with the physical world. Indeed, tetrapalegic

humans list regaining “arm/hand function” as the top priority for improving their quality

of life, as restoring this function would allow significant independence [2]. To address this

huge medical need, brain-machine interfaces (BMI, also called neural prosthetic systems

or brain-computer interfaces) seek to access the information in the brain and use that in-

formation to control a prosthetic device such as a robotic arm or a computer cursor. Such

systems, if successful, would have a large quality of life impact for many people living with

these debilitating medical conditions.

In the last decade, advances in neural recording technologies have accelerated research

in neural prostheses. Technologies for neural recording include electroencephalography

(EEG), electrocorticography (ECoG), and penetrating electrode or microwire arrays (see

[76] for a review). To design a prosthetic arm that can be controlled continuously with high

precision, most work has focused on penetrating electrodesimplanted directly into motor

3.2. INTRODUCTION 15

cortical areas [107, 130]. Researchers use nonhuman primates (e.g., rhesus monkeys) or,

increasingly, human participants [57, 69]. There are many medical, scientific, and engi-

neering challenges in developing such a system [76,98,107,108], but all neural prostheses

share in common the need for a decode algorithm. Decode algorithms map neural activity

into physical commands such as kinematic parameters to control a robotic arm.

Much work has gone into this domain, and many experimental paradigms and decoding

approaches have been developed and used [3,6,7,11,12,14,17,30,38,41,42,54,57,67–69,

71,74,77,79,86–88,95,100,102,109,113–115,119–121,124,130–132,138–141]. Most re-

search in decode algorithms has been done with offline data analysis using simulated neural

data [6,67,119–121,131] and/or neural data that was previously recorded from a healthy

animal [3, 6, 7, 11, 17, 30, 41, 42, 54, 68, 77, 87, 95, 102, 109, 113–115, 124, 132, 138–141].

In these studies, the benchmark for success is often how wellthe decoded arm trajec-

tory matches the true arm movement that was recorded in conjunction with the (possi-

bly simulated) neural activity. A smaller number of papers have used an online, closed-

loop paradigm to illustrate that prostheses can be meaningfully controlled by humans or

monkeys [12,14,38,57,69,71,79,86–88,102,109,124,130], but only a few of these pa-

pers [14,71,79] compare closed-loop performance of different algorithmsin monkeys, and

only one of these papers [69] compares the closed-loop performance of two different de-

code algorithms in humans. This reality is at least in part driven by the substantial resources

and effort required for online studies in animals and humans, thereby prohibiting extensive

online algorithmic comparisons.

Despite this abundance of work, our ability to decode arm movements accurately re-

mains limited. To decode an arbitrary reach, one current state-of-the-art algorithm is

perhaps the Kalman filter (introduced nearly fifty years ago in [65], used in this context

in [69,139]), which is the only algorithm that has been vetted in onlinehuman experiments

as having better performance than some competing possibilities such as a linear filter [69]

(though the closed-loop monkey studies of [14, 71, 79] indicate that other algorithms are

also competitive). Current achievable performance is encouraging and we have exciting

proofs of concept [57, 102, 130], but we must advance considerably before these systems

are clinically viable, and further still before we achieve decoded movements with speed

and accuracy comparable to a healthy arm (e.g., near perfect, sub-second accuracy).


In response to this critical need, several groups have proposed more advanced mathe-

matical approaches to neural prosthetic decoding [3,6,7,30,41,67,68,74,79,87,95,100,

113, 115, 119–121, 131, 140, 141]. However, none of these methods has seen widespread

adoption (across research studies or in critical translational work), in part due to uncertainty

of how these methods translate to closed-loop decode performance.

Offline evaluation of algorithms may neglect potentially important features of a real

neural prosthesis, including the user’s ability to modify control strategies to improve pros-

thetic performance. Truly understanding decode performance requires the human learning

machine (the brain and motor plant) to be in closed-loop withthe decode algorithm. In

this online, closed-loop setting, as soon as a prosthesis user sees a decoded arm reach (the

action of a robotic arm or the path of a cursor on a computer screen), he/she will bring to

bear all of his/her modification strategies to drive a desirable reach.

As a specific example of offline vs. online evaluation (in anticipation of the experiments

done here), a previous offline study found that prosthetic decode error is minimized when

the time bin over which neural activity is integrated (a windowed spike count in the Kalman

filter) is 200-300ms [139]. This bin width also represents the time step at which the algo-

rithm updates its estimate of the decoded reach. However, itmay be that in a closed-loop

experiment, when reaches last only roughly 1000ms, the intermittent “hopping” behavior

of a decoded reach will frustrate the user. Perhaps better control could be gained with a

more frequent update, where feedback-control would compensate for the increased noise

in the decode. Indeed, in [69], shorter bin widths (50 and 100ms) were used in online

human experiments. It would appear that shorter bin widths were found to be better in

initial online testing, though perhaps not optimized online, which may be in part due to the

overall difficulty/challenges of testing with disabled human participants. Thus, it remains

unclear how this and other parameters should be set in futurestudies. This simple ques-

tion, motivated in part by the work of [69,139], can be answered with current algorithmic

technologies, but it requires closed-loop validation. Thefield should investigate the extent

to which the subject can, for a given decode algorithm, engage online control strategies

to improve decode performance. Closed-loop testing may suggest different priorities for

algorithmic development than offline analyses.


101

102

103

104

100

101

102

103

104

100

Offline Analysis Human OPS Animal Model Human Trial

Cost per

tested

algorithm/

parameter

choice

(risk, time,

financial,...)

Number of

algorithm/

parameter

choices that

can be tested

Unexplored

Increasing Realism

Figure 3.1: Concept figure for Online Prosthetic Simulator opportunity. The x-axis showsfour testing paradigms in terms of increasing realism. Offline data analysis is perhapsthe least reasonable proxy to eventual user mode, as it entirely neglects the closed-loopcontrol. On the other end of the spectrum is the human clinical trial, which is preciselythe eventual user mode. Left axis (blue) shows the difficultyassociated with testing eachalgorithm or algorithmic parameter setting. Right axis (red) shows the number of algorithmand parameter choices that are reasonably testable, given costs and other constraints.

Addressing this problem is highly challenging, since fullydoing so would imply val-

idating every algorithmic choice, ideally, in a human clinical trial. Algorithmic choices

include both the structure of the algorithm itself and the parameter settings that should be

optimized, resulting in thousands of decode possibilities. Given the invasiveness and re-

source requirements of a full neural prosthetic clinical trial, this approach is infeasible. To

address this challenge, the field has employed an appropriate animal model such as a rhesus

monkey. However, given the large resource and temporal requirements of awake behaving

intracranial experiments, such an approach to widespread algorithm design is still imprac-

tical. Faced with this reality, most algorithmic work has been done in offline neural data

(simulated or real).

We ask here if a healthy human subject, using an entirely noninvasive prosthetic device

driven by synthetic neural activity, can meaningfully inform the design of prosthetic decode

algorithms. This system, which we call an Online ProsthesisSimulator (OPS), represents a

middle ground between simple (but perhaps less realistic) offline testing and more realistic


(but difficult and resource intensive) animal model and human clinical trials. We detail the

concept of this proposed system in Figure3.1. This figure shows in blue the dramatically

increasing complexity of testing each algorithmic choice (algorithm or parameter setting

within a single algorithm) as researchers move towards animal studies or human clinical

trials. This figure also shows (in red) the corresponding dramatic decrease in the number of

algorithmic choices that can be meaningfully tested. Here we ask two questions: first, how

different are offline and online analyses; and second, how similar are the OPS and closed-

loop animal model BMI? The OPS, by analogy to flight simulatorsor silicon integrated

circuit simulation software like “SPICE,” may allow more realistic evaluation of current

and future prosthetic decode approaches.

The creation of the OPS and the messages of this paper have important connections to

previous work. First, we note that previous literature has used signal sources other than

motor cortex to control an external BMI-like interface. In [97], the authors noted that EMG

may serve as a useful proxy to motor cortical signal for a BMI (apossible extension to

the OPS which we discuss at length in the Discussion). While connections to BMI design

were discussed, this study primarily investigated the ability of the motor system to learn

a non-intuitive control mapping (connecting to an important “fundamental neuroscience”

aspect of the OPS that we describe in the Discussion). From that study we draw the key

message that aspects of BMI design can be studied without the need for an invasive BMI.

More closely, [25] used a sensored glove to map hand movements to cursor control and

to study how subjects learned in the presence of an adaptive control mapping. They draw

connections to “co-adaptation” for BMI, which has been a question of interest since [124].

These two papers support the notion that a simulation systemlike the OPS can be used to

study non-trivial interface controllers. While these studies focus primarily on neuroscien-

tific aspects of human motor learning, the OPS is distinct by being designed specifically to

investigate neural prosthetic algorithm and system designchoices.

Second, two recent related works have investigated online vs. offline analysis in a spe-

cific BMI algorithm setting. First, [14] compared two BMI algorithms in standard offline

analysis and showed stark performance differences. However, when the same two algo-

rithms were then analyzed under closed-loop control in a monkey experiment, the perfor-

mance differences between these two algorithms became considerably less. One significant


message of that study is that certain types of error (directional biases) are readily learnable

in an online context, and so algorithms should not necessarily be discarded because of bi-

ases that impact offline performance negatively. Their second work [71] then compared

more algorithmic features, such as directional biases (again finding a discrepancy between

offline and online performance implications) and trajectory smoothness (which seemed to

matter in both online and offline contexts). This work is important in supporting the dis-

tinction between offline and online, but the OPS is distinct in that it offers a non-invasive

simulation environment for rapidly testing such algorithmic features in true closed-loop.

One abstract [81] did use humans non-invasively in this offline vs. online context, with a

similar spirit to the OPS. Here subjects used a joystick or EMG to control a virtual arm, and

the authors revealed the negative impact of systems that accentuate low-frequency control

signals, insomuch as such signals slow online feedback-control.

The studies [14, 71] also included simulation of one difference between onlineand

offline control and supported the value of further simulations. However, as the authors

noted, this simulation model did not include online correction and the use of feedback-

control. They rightly noted that such a computer simulationwould be considerably more

involved and heavy with assumptions. Here, we introduce theOPS as a system to accurately

model the human feedback-control system - using an actual human in closed-loop with the

decoding algorithm. Furthermore, the OPS system allows verification of these findings and

extensions to new algorithmic questions without full animal model experiments.

The remainder of the paper is as follows: in Methods, we describe the experimental

hardware and software platform that allows us to test the OPSin humans, the OPS in mon-

keys, and a real neural prosthetic BMI in monkeys (we term these testing scenarios “human

OPS mode,” “monkey OPS mode,” and “monkey BMI mode,” respectively). We describe

two variants to a simple center out reaching task that we had both humans and monkey

perform, and we detail relevant data analysis methods. In Results, data from humans and

monkey demonstrate that the subjects using the OPS paradigmshow significant perfor-

mance differences when using different bin widths of a Kalman filter decode algorithm.

We then compare these online results to offline analysis to make the first major point of

this paper: offline analysis does not provide an accurate picture of online performance. As


a second major point, we then show using the monkey data that the OPS paradigm accu-

rately reflects these trends, indicating similar algorithmic choices as does real BMI mode.

Finally, in Discussion, we discuss the implications of these results.

3.3 Methods

In this work, we performed human and animal experiments, offline and online analyses, in

OPS and BMI modes, and with two different task variants. To manage the description, we

break up the methods below as follows. First, we describe therelevant experimental hard-

ware that was used for all experiments. Second, we describe the reaching task performed

by both the animals and humans. Third, we describe human-specific protocols. Fourth, we

describe animal-specific protocols (including surgery andneural data acquisition). Fifth, as

the ability to generate sensible synthetic neural data is a key aspect of the OPS, we describe

in detail our methods and choices for neural spiking models.Sixth, we describe the decode

algorithm that was used to generate prosthetic cursor control in both the online and offline

cases. Seventh and finally, we describe the methods used to analyze the performance of

these varied experimental conditions.

3.3.1 Neural Prosthetic Experimental Hardware

We first describe the relevant experimental hardware and system. This experimental rig is

diagrammed schematically in Figure3.2.

Importantly, this hardware was used in all experiments, both by the human and animal

subjects, in both OPS and BMI modes, so we describe it here without distinction to the user.

This choice is intentional to further emphasize our effort to make the OPS a close proxy

to real BMI mode. In the human experiments, the subject sat in achair and placed his/her

head comfortably on a chin rest. In the animal experiments, the subject sat with a fixed

head position in a custom chair. In both cases, the subject’snose was positioned directly

in front of a pair of mirrors (at 45 degree angles from the eye). Each mirror reflected an

image from a pair of LCD monitors on either side of the subject’s head. These monitors

displayed identical images, but with a slight (multi-pixel) offset to create a disparity cue

3.3. METHODS 21

control

PCs

position tracker

stereo monitors

(for 3D effect)

mirrors

human or

monkey

reach

target

decoded

arm reach

3D visual percept

3D real

arm reach

(recording, signal

processing, neural

or synthetic neural

decoding)

Figure 3.2: Experimental rig which can be used both for Online Prosthesis Simulator (OPS)and real neural Brain Machine Interface (BMI) experiments. A human (shown in gray) ormonkey subject reaches (red trace) in a 3D volume obscured from view. An overheadposition tracker tracks endpoint kinematics. Control PCs process those data and render thesubject’s real reach (in control trials) or a prosthetic decoded reach (in OPS or BMI trials,the red trace). Two monitors (blue) project a stereo 3D imageonto mirrors (virtual 3Denvironment shown at right). Also displayed is a the subject’s reach target (green sphere).


leading to a stereoscopic depth percept, creating the illusion of a 3D environment (a typical

Wheatstone stereo 3D display [133]).

The subject made arm reaches in the large 3D volume behind themirrors. We recorded

kinematic parameters of the reach endpoint using a small reflective bead on the subject’s

finger, which allowed overhead optical position tracking (Polaris, Northern Digital, Water-

loo, Ontario). Hand position was measured at 60Hz to a resolution of 0.35mm RMS. These

data were recorded at 1000Hz by a generic x86 “behavior” computer running an embedded

operating system (xPC, MathWorks, Natick, MA) executing custom software. This com-

puter processed the kinematic data (for example, in OPS mode, to generate synthetic neural

data and decode that data into a decoded reach, specifics described below) or neural data

(in BMI mode, specifics described below), and transmitted a cursor position via ethernet

to a “visualization” x86 computer running a stripped-down Linux operating system. This

machine executed visualization software (MSMS, USC MDDF, Los Angeles, CA) which

rendered images to the Wheatstone display in near real-time (measured latency and jitter

in our system: 7±4ms). This low-latency system critically allowed us to investigate short

timescale prosthetic design questions. For example, in this work we were interested in

optimizing the integration bin width of the Kalman filter. However, if our system (which

includes the end-to-end integration of behavioral control, neural data streaming and record-

ing, and decoding) had latency and jitter in excess of 25ms, 50ms, or more, we could not

have investigated any bin widths lower than that threshold.This carefully engineered sys-

tem was thus an important enabler of this study.

This virtual environment rendered, against a black background, a reach target (4cm

diameter green sphere) and the subject’s decoded hand/cursor position (as a 4cm diameter

gray sphere, though it is shown in Figure3.2as a red sphere for clarity of illustration). In

control reaching trials, we rendered the subject’s true hand position back to the display as

the gray cursor. In closed-loop prosthetic trials (either OPS or BMI mode), we rendered a

decoded prosthetic reach as the same gray cursor. This system, similar in spirit to the virtual

environment in [124], allowed online, closed-loop neural prosthesis trials appropriate for

the work proposed here.

3.3. METHODS 23

3.3.2 Neural Prosthetic Experimental Task

Having described the hardware in which the subjects made reaches, we here describe the

specific structure of the experimental task. To make close comparisons between the offline

and online analyses, and between the OPS and BMI modes, we heldconstant the task

design across all human and monkey subjects.

All subjects made center-out reaches in the virtual environment described above. Each

reach consisted of the subject moving the gray cursor (undersubject control) to the green

reach target. The green reach target alternated between a center point and a pseudo-

randomly chosen target at one of 8 target locations evenly spaced on a ring of radius 8cm.

Only one target appeared per trial. In all data, only the center-out reaches (those from the

center point to the periphery) were analyzed.

The specific trial timeline proceeded as follows: the trial began when the green reach

target (either center point or one of the peripheral targets) appeared on the screen. As this

was a direct-reach paradigm, the target appearance was the subject’s “go” cue, and after

a short reaction time, the subject moved the gray cursor to the green target. The reach

was successful if the gray cursor was held within a demand box4cm wide around the

reach target. The cursor had to remain within the acceptancewindow for a hold period of

500ms, after which a reward was given (a tone for human subjects, and a tone and drop

of juice for the monkey). If that success criterion was not satisfied within 3000ms, the

trial was considered a failure and timed out. After either a trial success or failure, an inter-

trial interval of 40ms was imposed before the beginning of a new trial. The cursor was

controlled by the subject using one of three modes: real reaching, OPS prosthetic reaching,

and BMI prosthetic reaching, as described in the next paragraph. During all of these control

modes, the subject (monkey and human) was allowed free movement of his/her limb (as has

been done previously in BMI literature, for example [12, 109, 124]). Because the subject

only saw the cursor (the real arm was hidden from view), the subject remained motivated to

complete the task by the reward structure, which rewarded successful control of the cursor.

Though there was a sensory discrepancy between the visual feedback and the real arm’s

proprioception (in prosthetic trials), previous motor neuroscience work [97] and previous

BMI literature [12, 109, 124] suggest that this confound is minor and does not seriously


affect task learning. Whether the true arm is restrained or allowed to reach, there is still a

proprioceptive confound, so this potential limitation is not specifically of the OPS or of this

study, but rather of all able-bodied animal or human BMI studies. We discuss this aspect

of the experimental design in more depth in the Discussion.

There are three modes by which the subject controlled the gray cursor: real reaching,

OPS prosthetic reaching, and BMI prosthetic reaching. In real reaching, the gray cursor re-

flected the kinematics of the true underlying arm reach. Thisdata is that which is typically

collected for offline prosthetic decode analysis, and it wasused for training the parameters

of the prosthetic decode algorithms. Though other trainingparadigms have been used [57],

we chose this simple training method to parallel existing offline and online literature. In

BMI prosthetic reaching (monkey only), neural activity was recorded from the motor cor-

tex, and that data was decoded into kinematics which controlled the gray cursor. In OPS

prosthetic reaching mode (monkey and human subjects), synthetic neural activity was gen-

erated from the kinematics of the subject’s arm, and that neural activity was then decoded

with a decoding algorithm into control commands for the graycursor. The specifics of

recording, generating synthetic activity, and decoding are detailed below.

The parallel between OPS and BMI control modes is detailed in Figure 3.3. In both

the left and right panels of Figure3.3, the subjects make (a) a real reach (red trace), and

then (b) corresponding neural activity is recorded (synthetic neural activity generated from

the real reach in OPS mode - left panel, and real neural activity corresponding to the real

reach in BMI mode - right panel). That data is then input to the (c) decode algorithm,

which decodes the (possibly synthetic) neural activity into kinematics to control the gray

cursor, which is (d) rendered back to the subject. The decoded reach trajectory will differ

from the subject’s intended arm reach, due to noise (from neural spiking) and algorithmic

model mismatch (e.g. the Kalman Filter only approximately models the dependencyof

the neural activity on kinematics). The subject modifies his/her behavioral strategy so that

the prosthetic movement will mimic, as closely as possible,the desired reach trajectory.

Thus, both OPS and BMI modes offer a means to study the relevance of feedback-control

in neural prosthetic system use.

We also tested two variants of the center-out prosthetic reaching task. In the first - the

“continuous” task variant, prosthetic reaches were made both for center-out reaches to the

3.3. METHODS 25

(a) True physical

behavior

(b) Synthetic neural activity

(generated based on

true physical behavior)

(d) Decoded behavior (rendered

in 3D visual display)

Decode Algorithm

(c)

OPS Mode (Human or Monkey) BMI Mode (Monkey Only)

(a) True physical

behavior

(b) Real neural activity

(d) Decoded behavior (rendered

in 3D visual display)

Decode Algorithm

(c)

Figure 3.3: OPS and BMI schematic.Left Panel: OPS Mode - (a) A healthy humansubject or a monkey (human shown) makes real arm reaches in a 3D reaching environment(see Figure3.2). (b) Recorded kinematics of that real reach are used to generate syntheticneural activity, which is then input to the (c) Kalman filter neural decode algorithm. Thedecode algorithm decodes synthetic neural activity into physical reaching behavior, and (d)the decoded reach is then rendered back to the subject in the 3D visual environment, whichallows the subject to bring to bear all of his/her online, closed-loop control strategies todrive a desired reach.Right Panel: BMI Mode - (a) A monkey makes real arm reaches.(b) The real neural activity associated with that reach is recorded, which is then input to the(c) Kalman filter decode algorithm. The decode algorithm decodes real neural activity intophysical reaching behavior, and (d) the decoded reach is then rendered back to the monkeyin the 3D visual environment, which allows similar closed-loop control as in OPS mode. Itis important to note that the OPS mode of the left panel can be used both by a human anda monkey (this will provide an important validation of the OPS).


peripheral targets and for the reaches returning to the center target. We collected full data

sets from this task in the human OPS, and there was a mildly frustrating “drift” effect that

could occur, where, over many reach trials, the cursor wouldbecome increasingly offset

from the true arm reach. Though this never created a problem for the data (performance

was robust to this effect), it was reported by some subjects to be frustrating. As such, we

developed a second task variant. In the “interleaved” task variant, the reaches returning

to the center target were controlled by the real arm kinematics. This interleaving removed

the potentially frustrating drift effect. The center-out reaches (which are the only reaches

analyzed) remained entirely under prosthetic control. Though in eventual neural prosthesis

system use the “return to center” may be controlled by the system itself, this interleaved

variant was a simple way to keep the subject engaged in the task and to avoid confusion.

We ran human subjects in both OPS continuous and OPS interleaved. We ran the monkey

on BMI continuous and BMI interleaved, which presented no difficulty, since the mon-

key had been highly trained using real neural activity in exactly this continuous paradigm

with this BMI controller. For the monkey in OPS mode, we only ran OPS interleaved, as

the moderately frustrating effects reported by human users(with OPS continuous) would

require more extensive training with the monkey, and that variant is of unclear additional

merit on top of the interleaved task.

For each of these variants, we ran 5 full experiments. Here wefirst describe the full

experiment structure and how it allowed fitting and testing of numerous decode models;

the particulars of task training are left to the following ”Human Experiments” and ”Ani-

mal Experiments” subsections. A full experiment is a block structure designed as follows.

First, the subject (human or monkey) performed one block of 200 trials in real reach mode.

These 200 real reaches and the corresponding neural recordings are used to train the de-

coding models (Kalman Filters with different temporal bin widths). The subject was given

a roughly two minute break, during which online prosthetic control mode (either OPS or

BMI) was switched on. The subject then performed 7 blocks of 100 prosthetic reaches

each, where there was a short 15 second break between blocks.These 7 blocks of 100

reaches were then repeated again in a different order. Thus,a “full experiment” included

roughly 1600 reaches - 200 real reaches followed by two runs of 7x100 reaches. Typically

this took subjects about an hour. This experimental structure allowed an algorithm choice

3.3. METHODS 27

or algorithmic parameter choice to be tried at several different values (here 7, but this could

be changed). As discussed, here we are interested in optimizing performance based on the

temporal bin width of the Kalman Filter decode algorithm. Accordingly, we chose 7 bin

widths - 25, 50, 100, 150, 200, 250, and 300ms - and we had the subjects do two blocks

with each bin width (one block of 100 reaches per 7-block run). We randomized the order

of the blocks (within the 7-block run) to control for any learning effects. Each of the 9

human subjects ran one full experiment (either OPS continuous or OPS interleaved, save

for subject PN, who ran each on different days), and thus we have 5 full experiments each

of human OPS continuous and human OPS interleaved. The monkey ran all fifteen full

experiments (5 each of monkey BMI continuous, monkey BMI interleaved, and monkey

OPS interleaved). From the real reaches that comprise the first block of the experiment, we

are able to run all offline decoders, so the offline analyses were built from each subject’s

real reaching blocks.

This general task structure allows us to study online vs. offline analyses, BMI vs. OPS

mode, continuous vs. interleaved, in humans vs. monkey. To clarify, the results section

will show ten different types of prosthetic decode analysis: offline human OPS continu-

ous, offline human OPS interleaved, offline monkey OPS interleaved, offline monkey BMI

continuous, offline monkey BMI interleaved, online human OPScontinuous, online human

OPS interleaved, online monkey OPS interleaved, online monkey BMI continuous, and on-

line monkey BMI interleaved. These data sets are categorizedexplicitly in Table3.1. These

permutations allow us to perform a breadth of comparisons and controls to answer the key

questions of this study. In the following two sections, we describe particular protocols used

for the monkey and human subjects only (not shared across monkeys and humans as the

above task and hardware details are).

3.3.3 Human Experiments

Human protocols were approved by the Stanford University Institutional Review Board.

Nine healthy adult human subjects performed the reaching tasks in both real reach and OPS

modes as described above. These subjects reached in the experimental apparatus previously

described. Five subjects performed a full experiment (200 real reaches plus 2 blocks of 100


Human Subjects Monkey Subjects

Online analysis with

synthetic neural data

Offline analysis with

synthetic neural data

Online analysis with

real neural data

Offline analysis with

real neural data

Online Human OPS Continuous

Online Human OPS Interleaved

(Figures 3.4, 3.6)

Online Monkey OPS Interleaved

(Figures 3.5, 3.7)

Offline Human OPS Continuous

Offline Human OPS Interleaved

(Figure 3.6)

Offline Monkey OPS Interleaved

(Figure 3.7)

No Data

(requires human clinical trial)

No Data

(requires human clinical trial)

Online Monkey BMI Continuous

Online Monkey BMI Interleaved

(Figures 3.5, 3.7)

Offline Monkey BMI Continuous

Offline Monkey BMI Interleaved

(Figure 3.7)

Table 3.1: Categorization of the data sets collected and analyzed in this paper. The Resultsshow offline and offline analyses (blue and red in this table and in subsequent figures),human and monkey subjects (left and right columns in this table), synthetic and real neuraldata (identical to OPS and BMI modes, the top two and bottom tworows here), and finallythe continuous and interleaved task variants (grouped within each cell of this table). Thesepermutations allow us to perform a breadth of comparisons and controls to answer the keyquestions of this study.

3.3. METHODS 29

reaches at each of 7 bin-widths, as described above) of the OPS continuous task, and five

subjects (four new subjects, and one repeated from the OPS continuous) performed a full

experiment of the OPS interleaved task.

Across all experiments, in OPS and real reach modes, human subjects required very

little training (only a few trials, which are not included inthe results and analysis) to un-

derstand the task and control the cursor well. During real reaching trials (200 reaches),

which are used both for training the decode algorithm and foroffline analysis, we asked

the subjects to reach to the targets at a comfortable, normalspeed. Since the virtual path

of the cursor matches perfectly the true kinematics of the arm, the subjects had no trouble

performing these reaches with 100% accuracy. We then pausedthe task and informed sub-

jects that they were being switched into prosthesis mode forseveral blocks of 100 reaches,

where the virtual reach would not perfectly match their trueunderlying reach. We asked the

subjects to try to maintain the same reach speed and accuracyas in their previous reaches,

and we enforced the time-to-target timeout if subjects tooktoo long to reach (3000ms, as

previously noted). Depending on the bin-width being used, subjects were more or less

able to acquire the targets successfully and quickly (theseperformance differences are the

results of the OPS).

These human experiments provide considerable data for testing the difference between

offline and online (OPS) analyses in terms of the subject’s ability to exploit feedback-

control to improve prosthetic reaching performance.

3.3.4 Animal Experiments

Animal protocols were approved by the Stanford University Institutional Animal Care and

Use Committee. We trained one adult male monkey (Macaca mulatta, monkey L) to per-

form center-out reaches as already described. Unlike humans, monkeys were motivated to

complete the reaches with a juice reward. During experiments, the monkey sat in a custom

chair (Crist Instruments, Hagerstown, MD) with the head braced. Hand position recording

and experimental control were as previously described.

To enable BMI trials, a 96-channel silicon electrode array (Blackrock Microsystems,

Salt Lake City, UT) was implanted straddling dorsal pre-motor (PMd) and motor (M1)


cortex (right hemisphere), as estimated visually from local landmarks, contralateral to the

reaching arm. Surgical procedures have been described previously [21,54,102]. Neural sig-

nals were monitored on each channel and high-pass filtered. Athreshold level of -4.5 times

RMS voltage was established for each channel, and all threshold crossings were recorded

as “spike times.” Spike sorting was not performed, as our previous experiments indicated

that doing so did not significantly alter performance to justify the additional computational

load [15] (see also [36,55,103]).

The monkey was trained over several months on the reaching task and BMI continu-

ous task variant [47]. For the interleaved task variant, in both the OPS and BMI modes,

the monkey quickly learned this task in a few days of training, as it was only a mi-

nor extension of his well trained continuous BMI behavior. Multiple data sets of each

task were collected. For the continuous BMI task, the monkey completed five full ex-

perimental sessions (wherein each 200-trial block with an individual bin width was pre-

sented twice in pseudo-random order, as described above) across three days, and those

sessions are denoted asL20091215.C3.1, L20091217.C3.1, L20091217.C3.2, L20091217.C3.3,

and L20091218.C3.1. For the interleaved BMI task, the monkey completed all five ex-

perimental sessions across two days, and those sessions aredenoted asL20091218.IRR3.1,

L20091220.IRR3.1, L20091220.IRR3.2, L20091220.IRR3.3, andL20091220.IRR3.4. For the inter-

leaved OPS task variant, the monkey completed all five full experimental sessions in a sin-

gle day, and those sessions are denoted asL20091221.IRR2.1, L20091221.IRR2.2, L20091221.IRR2.3,

L20091221.IRR2.4, andL20091221.IRR2.5. To preserve trials, real reach trials were only run

once per day for the monkey, not in each experiment.

These animal experiments provide considerable data for again testing the difference

between offline and online analyses (whether comparing to BMIonline or OPS online).

Further, this data allows us to test the extent to which the OPS paradigm accurately reflects

the trends of real BMI mode.

3.3.5 Generating Synthetic Neural Activity

There are many behavioral features correlated with motor cortical spiking activity, includ-

ing features of the arm (hand position and velocity, muscle activity, forces, joint angles,

3.3. METHODS 31

etc.), features of the reach (smoothness, etc.), and features of the eye (visual reference

frames, etc.); see [126] for extensive references and discussion. In this work, we recorded

endpoint kinematics of the subject’s arm, and we used these data to produce reasonable

simulated neural spike trains which were then decoded into kinematics for controlling the

prosthetic cursor.

Certainly generating synthetic neural data requires some uncomfortable assumptions

about neural tuning and spiking. While synthetic neural activity has been used in the past

in prosthesis studies [6,67,119–121,131], we are particularly interested here in creating a

simulation system that translates well into a real neural system. Much care is needed both

in construction and in interpretation (discussed more fully in the Discussion) to ensure the

legitimacy of the results. Accordingly, our approach was topick as simple a construction

as possible that resulted in decoded reaches (both online and offline) that were qualitatively

similar to our core experience in real neural prosthetic experiments [22,23,67,102,141].

To begin, we used cosine-tuning models [42, 85] to map kinematics to neural firing

rates. Under this tuning model, each neuron was defined by a tuning vector, a minimum

firing rate, and a maximum firing rate (these last two are equivalent to setting a mean rate

and a depth of modulation). First, we chose a population sizeof 96 neurons (equivalent

to the 96 electrode channels of real neural data that we recorded with the array). For each

of these synthetic neurons, we sampled tuning vectors uniformly from the unit 3D sphere,

for both position and velocity (so these tuning vectors corresponded to “preferred position”

and “preferred velocity” vectors). We sampled minimum firing rates uniformly between

0 and 20 spikes per second, and we sampled maximum firing ratesuniformly between

that neuron’s minimum rate and 100 spikes per second, in linewith motor cortical neuron

behavior we have previously observed [19,20,22]. These firing rates were then put through

a spiking process, which we chose to be a simple Poisson process with rate equal to the

firing rate in each time bint. This is also a standard choice in literature [26,107].

Mathematically, we say that neuronk had tuning vectorc(k), maximum firing rateλ(k)max,

and minimum firing rateλ(k)min. Then, we definedxt to be the kinematic parameters of the

arm in time bint, wherext ∈ R6, a vector of 3D position and velocity of the hand.xt was

suitably scaled such that the range of kinematics exhibitedby the subject produced firing

rates within the range of minimum and maximum firing rates. Wedefinedλ(k)t ∈ R

K and


yt ∈NK+ to be the neural firing rates and spiking activities at time bin t (each elementλ(k)

t of

λt andy(k)t of yt corresponding to the firing rate and number of spikes of each of theK = 96

synthetic neurons being generated). With these definitions, we write:

λ(k)t = (λ(k)

max−λ(k)min)c

(k) ·xt +λ(k)min (3.1)

yt | λt ∼ Poisson(h(λt)) (3.2)

wherec(k) ·xt represents the inner product between these two vectors, andthe functionh(λt)

clips the rateλt to 0 if the argument is negative and otherwise equals the argumentλt (as is

commonly done in literature [14], though given our parameter choices, this clipping rarely

occurred). In the first generation of the OPS system, we created this tuning to both 3D po-

sition and 3D velocity kinematics, as has been reported considerably in literature [42,85].

While this produced reasonable decoded behavior offline, online use was not qualitatively

similar to real prosthetic use. With strong position dependence, users immediately devel-

oped a strategy of moving the hand to the virtual location of the target, and then holding

that position. Because there was position tuning information, eventually the cursor would

acquire the target for a full hold period. This behavior was qualitatively very different from

the observed behavior in monkey BMI experiments, where subjects make several online

adjustments to the real reach and are rarely able to successfully acquire a target by holding

a hand position for a period of time. To correct this qualitatively inappropriate effect, we

removed the position dependence in the simulated neurons. Instead, firing rate was only

tuned to velocity (soxt and correspondingc(k) are now only three dimensional, including

3D velocity terms). With this model, the qualitative behavior of OPS users was highly

similar to that of a monkey using a real BMI. After substantialtesting to ensure that other

unrealistic strategies were not developed, we determined that this simple model was ade-

quate for testing the decode implications of the Kalman Filter bin width. We note that other

possibilities for noise generation could include adding noise sources to either the firing rate

or to the spike trains themselves (to model recording noise sources, for example), but we

leave those enhancements to future work.

Here we note two important facts about this choice of spikingmodel. First, this spiking

3.3. METHODS 33

model does not match the decode algorithm, in the sense that the Kalman Filter is not an

optimal decoder of neural activity generated in this way. Indeed, the Kalman Filter is also

not an optimal decoder of real neural activity, as the Kalmanfilter does not exactly model

the real neural system. Second, it is important to note that we do not claim that this is what

the neural system is actually doing, but instead it is a choice that allows us to study the

OPS framework and the relevance of feedback-control to the design of neural prosthetic

systems.

Finally, this method for generating synthetic neural activity can be extended in a number

of ways previously seen in literature (e.g., [119–121]), but the simple velocity-tuned model

was so qualitatively and quantitatively similar to real BMI mode that we were satisfied with

this straightforward choice.

3.3.6 Decoding Neural Activity

We decoded neural activity using the Kalman filter, as has been used for neural prosthetic

algorithms previously [69,137–139]. It is important to note that this algorithm can be used

both in online and offline contexts. In the online case, at each time point, the decoded

kinematics controlled the cursor, which the subject could see, use as feedback, and react to.

In the offline case, the neural (or synthetic neural) activity was collected during real reaches

(when the cursor matches the arm kinematics) and later decoded offline. By using identical

decode algorithms (and parameter settings, etc.), we can effectively study the difference

between online and offline control.

The Kalman filter (introduced in [65]) stipulates a linear dynamical system for arm

movements (often called the prior or the trajectory model),which says that kinematics at

timet should look something like kinematics at timet−1, i.e., smoothness. The model also

stipulates a linear observation model for neural activity,which says that observed neural

spiking is a noisy linear transformation of intended kinematics. As before, we definext to

be the kinematic parameters of the arm in time bint, wherext ∈R7, a vector of 3D position

and velocity of the hand (plus a bias/offset term). We also define yt ∈ RK to be the neural

activity at time bint (each element ofyt corresponding to the activity of each of theK = 96

neurons being recorded). The Kalman filter then assumes the following linear dynamical


system:

xt = Axt−1+wt (3.3)

yt = Cxt +qt (3.4)

whereA ∈ Rp×p andC ∈ R

K×p represent the state and observation matrices, andwt and

qt are additive, independent Gaussian noise (denotedwt ∼ N (0,W) andqt ∼ N (0,Q)).

Such a model is standard for Kalman filtering, and it allows very fast inference (decoding)

of kinematics from neural activity. Furthermore, the parameters{A,C,W,Q} can also be

quickly and exactly learned from training data (the standard details of which are left to the

references [65,69,137–139]).

Intuitively, the Kalman filter starts from kinematics at thebeginning of the trial and

proceeds iteratively through time, updating its estimatesof arm state and error covariance

at every time stept. These steps are entirely based on mathematical propertiesof the

Gaussian, and the algorithm is fast and stable. Again, this is precisely the context that has

been used for online and offline neural prosthetic algorithms previously [69,137–139].

One important feature to note within the above description is the time bint. Since the

above dynamical system is discrete-time, we must choose a window of time over which to

integrate neural activity for our prosthetic decode. This time, which we call the “decode

integration bin width” or just “bin width,” is a key parameter that can be chosen by the

experimenters. As noted in the introduction, previously published offline analysis has sug-

gested that a 200-300ms bin width is optimal [139], but this choice has not been validated

online. We will see that different settings of this parameter can have a large performance

effect both qualitatively and quantitatively. Thus, optimizing performance with respect to

this parameter is an important research question and a valuable test for the OPS, which we

present here.

3.3.7 Performance Analysis

Having decoded the neural activity as described in the section above, we must now in-

troduce metrics to quantify the performance of a particularprosthetic mode (task variant,

subject, bin width choice, etc.). Error metrics are numerous and a subject of study in their

3.3. METHODS 35

own right [29], but here we chose a few sensible metrics that have been seenpreviously in

literature. For consistency, we present all metrics aserror and notperformancemetrics;

i.e. lower is always better throughout the results.

First, for online analysis, failure-rate is an obvious and previously used choice (e.g.,

[102]). Failure-rate measures the fraction of trials in which the subject was unable to ac-

quire and hold the reach target within the allotted time. Second, time-to-target, as used

in [57] measures the amount of time the subject required to reach the target (for the last

time, since the subject could pass through the target a number of times before holding the

cursor at the target for the required hold period).

While these straightforward performance metrics offer a quantitative view of online per-

formance, they can not be meaningfully used for offline decode analysis. Offline decoded

reaches rarely reached the target and never successfully completed a trial (this makes sense

with a noisy decoder in the absence of feedback, which is yet another suggestion that online

analysis should perhaps be prioritized), so failure-rate is a vacuous metric in comparing al-

gorithms or algorithmic choices offline. Time-to-target issimilarly broken, and thus can not

be used for offline analyses. Standard metrics for offline analysis include mean-squared-

error (MSE) or similar, and correlation coefficient, and these have been used extensively

(see description in [22]). With these metrics, one compares the true, natural reachto the of-

fline decode of that same reach. Unfortunately, these sensible offline metrics are not useful

online, as there is no correct “true reach” that can be compared.

To meaningfully compare offline analysis with online analysis (the first major point of

this work), we require a metric that is suitable both in the offline and the online context.

As the subject was always motivated to move closer to the target during a reach (whether

in OPS, BMI, or real reaching mode), we can consider the distance to the target. We

define mean-integrated-distance-to-target as the averagedistance to the target during the

time course of the subject’s reach. On triali, if we call the target positionp(i)targ and the

position at timet during the trialp(i)t , the errorE(i) for trial i can be written mathematically

as:

E(i) =1

T(i)

T(i)

∑t=1

‖p(i)t − p(i)targ‖ (3.5)


whereT(i) is the length of triali. Using the mean distance instead of total distance is sensi-

ble because offline and online trials have characteristically different temporal lengths (since

offline trials are based on real reaches, these are generallyquicker than online, noisy pros-

thetic trials). It is important to note that, when comparingoffline and online performance

(and BMI vs. OPS mode performance), less will be made of absolute differences in perfor-

mance between offline and online (and BMI vs. OPS modes); rather, we are mostly inter-

ested in the optimal parameter settings that are suggested by different prosthetic modes.

Mean-integrated-distance-to-target is by no means the only metric possible for com-

paring offline and online trials, so we here give context for why we believe this metric to

be appropriate. First, since the actual task rewards movement of the cursor to the target

both in offline and online settings, having a metric that considers the target is sensible. One

might reasonably consider other metrics that compare the prosthetic reach to some true arm

kinematics such as an “ideal” real arm reach (perhaps gleaned from real reaching trials).

However, the subject is not rewarded for having a particularkinematic profile - the reward

is for task success,i.e. acquiring the target. Quantifying prosthetic reach error based on a

quantity that is unknown to the subject and only somewhat related to task success does not

seem appropriate. Put another way, a subject using a BMI mightdevelop a different suc-

cess strategy (changing his/her kinematics, for example),and “ideal reach” metrics could

arbitrarily penalize such a strategy. On the other hand, thedistance to target is sensible

because, by task design, we know that the subject is always trying to reduce this distance,

and thus motivation and performance are aligned.

Finally, we note that each of these metrics was calculated ona per trial basis, and thus

we could calculate average statistics and confidence intervals. Throughout the results we

use 95% confidence intervals, calculated via the binomial distribution for failure-rate anal-

yses and Gaussian distributions for time-to-target and mean-integrated-distance-to-target

analyses [142]. These performance analyses give us all the quantificationnecessary to

analyze the results gathered via previous methods.

3.4. RESULTS 37

3.4 Results

The results are composed of three pieces, which we describe here in brief to help the reader

navigate the section. We begin by showing the raw performance data across subjects. These

data will demonstrate that all subjects using the OPS and BMI modes show significant per-

formance differences when using different bin widths of a Kalman filter decode algorithm.

In other words, the raw data shows that the bin width does havemeaningful performance

implications and further that this prosthetic setup is a meaningful way to investigate those

implications. Next, we make the first major point of this study: offline analysis does not

provide an accurate picture of online performance. The datashow significant differences

in the shape of these performance curves based on offline or online (BMI/OPS) modes,

thereby suggesting very different algorithmic parameter choices. Finally, we make the sec-

ond point of this paper: the monkey BMI data, the human OPS data, and the monkey OPS

data all show similar performance trends. This importantlyindicates that, at least when it

comes to this one critical algorithm parameter, the OPS framework can stand as a reason-

able proxy to real BMI mode.

3.4.1 Online Prosthetic Data

We begin by showing the performance for both humans and monkey in the online prosthetic

modes, both BMI and OPS. In Figure3.4, we show the error for the human subjects; the

left column shows the OPS continuous task variant, and the right column shows the human

OPS interleaved variant. Within a given column, each panel shows the same data analyzed

with a different error metric: panel A - failure rate, panel B- time to target, and panel C -

mean integrated distance to target. Each of these metrics isdescribed above in Methods. As

a reminder, this third metric - mean integrated distance to target - is required to compare

offline and online analyses. In each panel, each faint line denotes the average error of a

particular human subject over the course of that subject’s full experiment. For example, in

the right column, panel A, the light green trace shows the average failure rate of subject

JH at each of the bin-widths. We see that, at a bin width of 200ms (subject JH did 200

reaches at this bin width, per the block structure protocol described above), JH failed to

acquire and hold the target on roughly 60% of trials. Since weare particularly interested


Human OPS Continuous

0 50 100 150 200 250 3000

20

40

60

80

100

Fa

ilure

Ra

te (

%)

Decode Integration Bin Width (ms)

JC20091213.C2

VG20091213.C2

MF20100318.C2

SK20100319.C2

PN20100321.C2

All Blocks

0 50 100 150 200 250 3001000

1200

1400

1600

1800

2000

Tim

e T

o T

arg

et (m

s)


A

0 50 100 150 200 250 30035

40

45

50

55

60

65

70

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


B

C

0 50 100 150 200 250 3000

20

40

60

80

100

Fa

ilure

Ra

te (

%)


PN20100321.IRR2

CC20100322.IRR2

JH20100322.IRR2

KS20100322.IRR2

RT20100323.IRR2

All Blocks

0 50 100 150 200 250 30035

40

45

50

55

60

65

70

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


Human OPS Interleaved

0 50 100 150 200 250 3001000

1200

1400

1600

1800

2000

Tim

e T

o T

arg

et (m

s)


A

B

C

Figure 3.4: Performance metrics for Human Online Prosthetic Simulator (Human OPS)decode trials.Left Panel: Continuous Task variant - (a) Failure Rate: the percentage oftrials where the subject did not successfully acquire and hold the reach target. (b) Time toTarget: the time required to reach the target for successfultrials. In both panels, data fromfive subjects are shown in light colors. The average of all trials is shown in dark blue. Thisaverage shows in both cases a significant linear trend indicating that smaller bin widths willlead to better performance. (c) To compare offline and onlinedata, we use a third metric:integrated distance to target (failure rate and time to target can not be calculated for offlinedata, as offline failure rate approaches 100% without onlinefeedback, which is again tellingof the inappropriateness of offline analysis). This metric is normalized by trial length (totaltrial time). Right Panel: Interleaved Task variant - These show the same metrics as theleft panel, reinforcing with another task variant these trends. In all panels, 95% confidenceintervals (A: binomial distribution, B, C: Gaussian) are shown as error bars.

3.4. RESULTS 39

in average error at a bin width (and less interested in inter-subject differences other than to

confirm that the trend is the same in all subjects), in dark blue we plot the average across

all subjects, along with 95% confidence intervals (as described in Methods). This color

scheme is consistent across all panels in Figure3.4.

First and foremost, it is immediately apparent across all panels that, in the human OPS,

shorter integration bin widths lead to lower error in the online setting. The raw data gives

confidence that there is meaningful performance differences across this algorithm parame-

ter. By performing a linear regression to the data in each panel (binomial noise model for

Failure Rates in panels A; gaussian noise model for other error metrics in panels B and C),

we can calculate the confidence level that each panel is indeed a positive sloping line (i.e.

shorter is better). For the human OPS continuous, those values are - panel A:p < 10−4;

panel B:p< 10−4; panel C:p< 10−4. For the human OPS interleaved, those values are -

panel A:p< 10−4; panel B:p< 10−4; panel C:p< 10−4. Thus, with high confidence we

can say that the human OPS indicates that shorter integration bin widths will lead to higher

online performance.

There are a few salient features to point out in this data. First, it is encouraging that all

error metrics suggest a similar trend in the data, giving us confidence that we are looking

at a real trend and not an artifact of the summary metric. Thisfact will also become useful

when we compare offline to online OPS, which we can only do withthe metric of panel C

(since failure rate and time to target are not meaningful in offline analysis, as previously

described). Second, it is encouraging that the data is rather insensitive to the human OPS

continuous or OPS interleaved task variants (left and rightcolumns). Recall that the in-

terleaved variant was introduced to prevent user frustration. The data’s robustness to that

frustration indicates that it was not critical to subject performance and did not invalidate

the data. Third, it is encouraging to see that subjects performed rather similarly compared

to one another. If there were substantial inter-subject differences, more investigation might

be required. However, in both OPS continuous and OPS interleaved, we had a few experi-

enced subjects (JC, PN, and VG, who had run several thousand trials in this task on days

prior to the data collection) and several naive subjects (MF, SK, CC, JH, KS, and RT, who

had not run the task before their full experiment). As no difference between these groups is

apparent, we are also given confidence that there are not significant learning/training effects


that need to be controlled in this data. Thus, in summary, Figure 3.4 tells us that shorter

bin widths will lead to lower error rates, and further that this is a highly robust effect in the

human OPS.

In Figure 3.5 we repeat the same presentation as in Figure3.4, but for the monkey

subject instead of the human subjects. As discussed in Methods, here there is only one

subject who performed many full experiments (as opposed to the human subjects, who

each did one full experiment). Accordingly, Figure3.5shows faint lines corresponding to

individual full experiments, and the dark blue correspondsto the average error across all

experiments. Note also that there are now three columns, including real neural prosthetic

mode - monkey BMI continuous and monkey BMI interleaved, and also monkey OPS

interleaved (these choices discussed in Methods). As in thehuman data, the findings are

largely the same. In online prosthetic mode, across two taskvariants, BMI/OPS mode, and

three error metrics, this data indicate that shorter integration bin widths will lead to higher

prosthetic performance. As previously noted in the Methods, we note that less will be made

of absolute differences in performance between offline vs. online and OPS vs. BMI; rather,

we are mostly interested in the optimal parameter settings that are suggested by different

prosthetic modes.

By performing a linear regression to the data in each panel (binomial noise model for

Failure Rates in panels A; gaussian noise model for other error metrics in panels B and C),

we can calculate the confidence level that each panel is indeed a positive sloping line (i.e.

shorter is better). For the monkey BMI continuous, those values are - panel A:p< 10−4;

panel B:p= 0.07; panel C:p< 10−4. For the monkey BMI interleaved, those values are -

panel A:p< 0.01; panel B:p< 10−3; panel C:p< 10−4. For the monkey OPS interleaved,

those values are - panel A:p< 10−4; panel B:p< 10−4; panel C:p< 10−4. Thus, with

high confidence we can say that the monkey BMI and OPS - like the human OPS - indicate

that shorter integration bin widths will lead to higher online performance.

3.4. RESULTS 41

0 50 100 150 200 250 3000

20

40

60

80

100

Fa

ilure

Ra

te (

%)


L20091215.C3.1

L20091217.C3.1

L20091217.C3.2

L20091217.C3.3

L20091218.C3.1

All Blocks

0 50 100 150 200 250 3000

20

40

60

80

100

Fa

ilure

Ra

te (

%)


L20091218.IRR3.1

L20091220.IRR3.1

L20091220.IRR3.2

L20091220.IRR3.3

L20091220.IRR3.4

All Blocks

0 50 100 150 200 250 3000

20

40

60

80

100

Fa

ilure

Ra

te (

%)


L20091221.IRR2.1

L20091221.IRR2.2

L20091221.IRR2.3

L20091221.IRR2.4

L20091221.IRR2.5

All Blocks

0 50 100 150 200 250 3001500

2000

2500

3000

3500

Tim

e T

o T

arg

et (m

s)

Decode Integration Bin Width (ms)0 50 100 150 200 250 300

1500

2000

2500

3000

3500T

ime

To

Ta

rge

t (m

s)


Monkey BMI Interleaved

A

0 50 100 150 200 250 300

40

50

60

70

80

90

100

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


B

C

Monkey OPS Interleaved

A

0 50 100 150 200 250 300

40

50

60

70

80

90

100

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


B

C

0 50 100 150 200 250 300

40

50

60

70

80

90

100

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


0 50 100 150 200 250 3001500

2000

2500

3000

3500

Tim

e T

o T

arg

et (m

s)


Monkey BMI Continuous

A

B

C

Figure 3.5: Performance metrics for Monkey Brain Machine Interface (Monkey BMI) andOnline Prosthetic Simulator (Human OPS) decode trials.Left Panel: BMI ContinuousTask variant - (a) Failure Rate: the percentage of trials where the subjectdid not suc-cessfully acquire and hold the reach target. (b) Time to Target: the time required to reachthe target for successful trials. In both panels, data from five subjects are shown in lightcolors. The average of all trials is shown in dark blue. This average shows in both cases asignificant linear trend indicating that smaller bin widthswill lead to better performance.(c) To compare offline and online data, we use a third metric: integrated distance to target(failure rate and time to target can not be calculated for offline data, as offline failure rateapproaches 100% without online feedback, which is again telling of the inappropriatenessof offline analysis). This metric is normalized by trial length (total trial time). MiddlePanel: BMI Interleaved Task variant - These show the same metrics as the left panel,reinforcing with another task variant these trends.Right Panel: OPS Interleaved Taskvariant - The monkey can also run the OPS task with synthetic neural data. This panelshows the same metrics as other panels, and these metrics allshow similar trends. By com-paring BMI and OPS within subject on the same task (middle and right panels), we have anindication that the OPS is providing a valuable proxy to realneural prosthetic systems. Inall panels, 95% confidence intervals (A: binomial distribution, B, C: Gaussian) are shownas error bars.


3.4.2 Comparing Online Analysis to Offline Analysis

With the several different variants of online prosthetic tasks (OPS vs. BMI, monkey vs.

human, continuous vs. interleaved), and with the real reachtrials that each subject per-

formed as part of the experiment, we now demonstrate the difference between offline and

online analysis of neural prosthetic systems. We noted in the previous section that the on-

line data indicate strongly that shorter bin widths will lead to lower error. As a reminder,

each panel C of Figures3.4 and3.5 shows an error metric that can be calculated for both

offline and online data. Accordingly, we took the real reaches from each subject during

these experiments, and we ran offline analysis as has been previously seen in much litera-

ture [3,6,7,11,17,30,41,54,68,77,87,95,109,113,115,124,132,138–141]. The offline

analysis consists of taking the neural data from a real reaching trial and running the Kalman

filter with a given bin width, thereby producing a decoded reach from the recorded neural

data alone. Because this data is offline, the same neural activity can be decoded at each

bin width; that is, only one block is needed for every seven that were needed for the online

analysis.

In Figure3.6 we show this offline analysis in red. Again, the left column isthe hu-

man OPS continuous task, and the right is the OPS interleaved. For clarity of illustration,

we only show the average data (not the individual experiments, which were shown as faint

traces in Figures3.4and3.5). The dark red points and confidence intervals (95%) represent

the distribution of the error at each bin width. In dark red, we fit a quadratic polynomial to

the data. In faint red, we show the linear regression to that same data. In blue, we show in

the same way the distribution of the online data analysis. Note that these points and con-

fidence intervals are exactly the same as the points and intervals from Figure3.4, Column

C (without the connecting line segments). In dark blue we show the linear regression to

the data, and in faint blue we show the quadratic fit. In both columns, the dark blue line

entirely obscures the quadratic fit (the quadratic term is nearly 0).

Again, a few salient features appear. First, online (blue) has consistently lower error

than offline (red). This is not surprising, as the subjects’ access to feedback information

in the online case should unambiguously improve performance. Nonetheless it is a sanity

check for the data and error metric. Second, and more importantly, we see that the offline

3.4. RESULTS 43

0 50 100 150 200 250 300

40

50

60

70

80

90

100

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


Human OPS Continuous Human OPS InterleavedPLACEHOLDER... TO COME

0 50 100 150 200 250 300

40

50

60

70

80

90

100

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


offline

online

Figure 3.6: Comparing offline analysis to online analysis in Human OPS mode.Left Panel:Continuous Task variant - The blue error bars are replicated from Figure3.4, Panel C.The dark blue line is a linear fit of that data, showing a significant performance trendindicating that shorter bin widths imply better performance. A light blue quadratic fit (notvisibly or statistically different from a line) to the same data is obscured by the linear fit. Inred, we perform the same analysis offline, using the real reaching sets from these users. Weperform offline decodes at each bin width, allowing us to generate the same mean integrateddistance to target metric. These error bars are the offline analogs to the blue error bars. Notsurprisingly, offline analysis has worse performance than online, since there was no benefitof feedback. More significantly, however, is the shape of thedata. This offline analysissuggests statistically significant performance optima of roughly 100-150ms. This can beseen in the significant quadratic fit to the data (dark red). The linear fit (light red) does aclearly poorer job of fitting the data. Note that the characteristic “U-shape” of the offlinedata tells a very different story than the online analysis, indicating the important differencesbetween these two testing paradigms.Right Panel: Interleaved Task variant - The sameanalysis and implication holds true as in the left panel.


error analysis has a characteristic “U-shape” indicating aperformance optimum at 100-

150ms bin widths, adding more evidence that offline and online analyses may not agree. We

note that the U-shape in offline errors makes intuitive sense(and has been shown elsewhere

[139]): integrate too little neural data (short bin widths), andthe estimate is swamped

by noise; integrate too much neural data (long bin widths), and the system updates the

decoded cursor too slowly. Some optimal tradeoff between these two extremes (i.e., a U-

shape) seems reasonable. Contrasting these U-shapes to the blue lines in Figure3.6(online,

where shorter bin widths give lower error), the online case may also make intuitive sense:

indeed shorter bin widths lead to a noisier estimate, but thecritical presence of feedback-

control allows the user to compensate for that noise, resulting in lower error. On the other

hand, longer bin widths still result in a slowly updating prosthesis, so this slow “hopping”

behavior makes feedback-control more difficult and resultsin relatively higher error.

In Figure3.7 we repeat the presentation of Figure3.6, but for the monkey instead of

humans. Offline analyses are shown in red, and online analyses are shown in blue. The raw

data are represented with points and confidence intervals, and linear and quadratic fits are

shown. In all online cases, the linear fit (dark blue) highly overlaps the quadratic fit, though

the quadratic can just be seen in the rightmost column. Again, the monkey results are con-

sistent with the human results. Online analysis in blue produces linear fits suggesting that

shorter bin widths produce lower error, whereas the offline analyses produce characteristic

U-shapes indicating a performance optimum around a bin width of 150-200ms. Note that

the monkey BMI continuous and BMI interleaved offline data (reddata in the left and mid-

dle columns) are identical: in real reaching trials, there is no difference between continuous

and interleaved cases, as all reaches are real. Accordingly, we used the same reaching data

for these two figures. Another interesting note is that, evendisregarding the U-shapes, the

linear fits to offline data are inconsistent: human offline linear analysis says that shorter

bin widths are better (red data in Figure3.6), but monkey offline linear analysis says that

performance either is insensitive to bin width or improves with larger bin widths (red data

in Figure3.7). Contrast that to the simple and consistent story that is seen across all on-

line (blue) data in Figures3.6and3.7 (and so too Figures3.4and3.5): shorter bin widths

will improve prosthetic performance. Thus, these data strongly suggest that offline analy-

sis gives an inconsistent and different answer than online analysis. Since the eventual user

3.4. RESULTS 45

0 50 100 150 200 250 300

60

80

100

120

140

160

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)

Decode Integration Bin Width (ms)0 50 100 150 200 250 300

60

80

100

120

140

160

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


Monkey BMI Interleaved Monkey OPS InterleavedMonkey BMI Continuous

0 50 100 150 200 250 300

60

80

100

120

140

160

Me

an

In

teg

rate

d D

ista

nce

To

Ta

rge

t (m

m)


offline

online

Figure 3.7: Comparing offline analysis to online analysis in Monkey BMI and MonkeyOPS modes.Left Panel: BMI Continuous Task variant - The blue error bars are repli-cated from Figure3.4, Panel C. The dark blue line is a linear fit of that data, showingasignificant performance trend indicating that shorter bin widths imply better performance.A light blue quadratic fit (not visibly or statistically different from a line) to the same datais obscured by the linear fit. In red, we perform the same analysis offline, using the realreaching sets from these users. We perform offline decodes ateach bin width, allowing usto generate the same mean integrated distance to target metric. These error bars are theoffline analogs to the blue error bars. Not surprisingly, offline analysis has worse perfor-mance than online, since there was no benefit of feedback. More significantly, however,is the shape of the data. This offline analysis suggests statistically significant performanceoptima of roughly 150-200ms. This can be seen in the significant quadratic fit to the data(dark red). The linear fit (light red) does a clearly poorer job of fitting the data. Note thatthe characteristic “U-shape” of the offline data tells a verydifferent story than the onlineanalysis, indicating the important differences between these two testing paradigms.MiddlePanel: BMI Interleaved Task variant - The same analysis and implication holds true asin the left panel.Right Panel: OPS Interleaved Task variant- Again, the same analysisand implication holds true as in the left panel. Taken together, these three panels show thatboth the BMI and OPS modes tell the same story: shorter bin widths imply better perfor-mance, whereas the offline analyses indicate an incorrect trend that leads to varying andmisleading performance optima.


mode of neural prostheses is fundamentally online, this first finding calls into question the

validity of offline analysis in informing prosthetic designchoices.

3.4.3 Comparing OPS to BMI

The previous section showed that offline analysis may often be a poor proxy to online use of

a prosthetic system. This effect was apparent in both the OPSand BMI data across Figures

3.6and3.7. Thus the OPS, alongside BMI mode, has already made clear the importance of

feedback-control considerations in prosthetic design. While this first finding is of scientific

value, it does not clarify that the OPS is a strong proxy to BMI mode. We investigate that

question here.

Figure3.8 demonstrates the similarity between OPS and BMI modes in an online set-

ting. In previous analyses we fit both lines and parabolas to the collected online and offline

performance data. Some of those regressions appeared linear, others “U-shaped.” By in-

vestigating the quadratic fit term, we can see which data support a U-shape conclusion, and

which a linear. Again, this distinction is of critical importance because the shape of the per-

formance curve makes fundamentally different statements about how to design a prosthetic

system. We calculate the 95% confidence intervals on the quadratic regression coefficient

for each of the ten data sets, and we plot that data in Figure3.8. The blue points and inter-

vals again represent the online data, and the red represent offline data. Each row indicates

the prosthetic mode (human or monkey, OPS or BMI, continuous or interleaved). The gray

and white stripes serve only to visually distinguish the rows. The black line denoting 0

allows us to draw the key conclusion from this figure. If a confidence interval includes 0,

then we can not say with 95% confidence that the correspondingdata is U-shaped; rather

we conclude that shorter bin widths will indeed lead to better performance.

By considering only the blue data, we see importantly that allOPS and BMI modes

indicate that shorter bin widths are indeed better. The consistency of these findings indi-

cates that OPS and BMI modes are indeed valid proxies for one another. Only one case -

the monkey OPS interleaved - does not have a confidence interval overlapping 0, though it

does overlap the intervals of all other monkey data. In otherwords, that OPS quadratic fit

term is not statistically significantly different from the BMI regression terms. Furthermore,

3.4. RESULTS 47

0

Human OPS

Continuous

Human OPS

Interleaved

Monkey BMI

Continuous

Monkey BMI

Interleaved

Monkey OPS

Interleaved

Quadratic Fit Term (with 95% con!dence interval)

Online O"ine

0.002

Figure 3.8: Summary of online/offline differences; summaryof OPS/BMI similarities. Inthe previous figures, we fit quadratic curves to both the offline and online performance data.Here, we plot those quadratic regression coefficients and their 95% confidence intervals.Any confidence interval overlapping 0 should be read as, ”this data is not significantlydifferent than a linear trend.” This figure shows that nearlyall online analyses in monkeyand human do not have significant quadratic terms, agreeing with the implication from allprevious data that shorter bin widths lead to better performance. Comparing that to the rederror bars, again the central point is reiterated: online analysis and offline analysis (whetherin real neural data or in synthetic neural data) tell very different stories. Furthermore, inaddition to online and offline being very different, we see that OPS and BMI are in factvery similar. The OPS gives a valuable means by which to sweepperformance based onthis algorithmic parameter.


if we instead plotted 99% confidence intervals, the intervalof this monkey OPS interleaved

case would indeed include 0. Thus, this discrepancy is slight and of minor concern to the

interpretation of the data. Figure3.8 restates what was already visible in Figures3.6 and

3.7: blue online data across OPS and BMI modes agree.

In addition, this figure emphasizes the prior finding that offline analysis gives a different

answer than online analysis: the red offline data are all U-shaped with 95% (and indeed

99%) confidence, but the blue online data do not support that claim, and only show that

shorter bin widths are better. This figure also conveys another troubling aspect of offline

analysis, namely the inconsistency of the findings across different subjects and prosthetic

modes (i.e. the red intervals are quite varied amongst themselves).

This second finding suggests that the OPS may allow rapid and simpler testing of many

algorithmic choices and may be a better proxy to clinical usethan offline data analyses. In

this work, we carefully tested that finding by gathering realBMI experimental data also,

but in the future, careful experimental design with the OPS may allow many algorithms,

parameters, and indeed prosthetic systems to be optimized without the risk and difficulty

of implanted BMI subjects. Offering the OPS as an important part of the neural prosthesis

design toolkit is the broad aim of this study. In the following section, we discuss other uses

of the OPS, and we give caveats about ways in which the OPS may not be a valuable proxy

to BMI mode. These discussion points are critical to understanding the role of the OPS in

future neural prosthesis research.

3.5 Discussion

At a simple level, our results make three specific scientific points about optimizing the

performance of a neural prosthetic system by choosing the bin width of a Kalman Filter.

The first point, seen in the raw data of Figures3.4 and3.5, is that the bin width of the

Kalman filter does indeed have meaningful performance implications; that is, it is a critical

parameter that should be optimized in a neural prosthetic system. Second, the results show

that offline analysis is a poor representative of online, closed-loop performance. Figures3.6

and3.7show, across a variety of paradigms, that offline analysis indicates a very different

trend in the data than the actual online usage mode of a prosthetic system. Third, we see

3.5. DISCUSSION 49

that both the OPS and BMI suggest that shorter integration binwidths of 25-50ms will

lead to better performance, presumably reflecting the increased ability of the subject to

incorporate feedback-control strategies.

At a deeper level, our results imply two fundamental findingsabout the design of neural

prosthetic systems. First, we claim that feedback-controlis an essential consideration in

the design of these systems, as demonstrated by the fact thatoffline analysis of algorithms

is an inconsistent and problematic testing scenario. Across Figures3.6, 3.7, and3.8, we

see that offline analyses of various experimental paradigmsindicate parabolic performance

curves with error minima anywhere from 100-200ms (and previous literature has found

even higher optima, 200-300ms [139], though the experimental paradigm and evaluation

metrics were different in that study). Furthermore, the curvature of these parabolas - in

other words how costly it would be to have a suboptimal parameter setting - is inconsistent

across paradigms, as seen in Figure3.8. Most troubling, however, is that these varied

analyses all disagree fundamentally with online, closed-loop prosthetic use, which is in fact

the true usage mode of these systems. All online analyses suggest that performance curves

are linear, and that shorter parameter settings are better,which stands in stark contrast to

the parabolic offline implication. Thus, the first finding of this work is to say that offline

and online analyses are very different for neural prosthetic systems, and further that offline

is perhaps not adequately realistic to justify its continued use in the design of prostheses.

The second broad implication of this work is to present the OPS as a valuable proxy to

a real neural prosthetic system. By replacing recorded neural activity with synthetic neural

activity, the OPS enables analysis of human subjects using an online, closed-loop prosthetic

system. By reproducing the qualitative behavior of real prosthesis use, we hypothesized

that the OPS would produce similar performance curves as thereal BMI mode. We tested

this hypothesis by comparing humans and a monkey using the OPS to a monkey using

a real BMI, and we found that all reproduce similar performance trends in the bin-width

parameter sweep (Figure3.8summarizes this effect). Thus, in terms of this specific design

question for decoding algorithms like the Kalman filter, theOPS appears to be a high-

quality approximation to real online BMI use. However, a bigger question is the extent

to which the OPS can be relied upon to generalize to differentneural prosthetic design

questions in humans. As is portrayed in Figure3.1, having human results here (in addition


to monkey results) is important to present a full human BMI simulator that does not require

the resources of invasive human or monkey experiments. Human results are also important

for generalizing the findings of the OPS to areas that will notnecessarily be addressable

in monkeys. The following points discuss in depth a number ofways in which the OPS

will generalize well and a few ways in which it may not. These points of generalization are

natural steps for future work as well.

3.5.1 The OPS for general system design questions

With our specific finding that online and offline analyses are fundamentally different for

the Kalman filter bin width, we have demonstrated that feedback-control and subject in-

teraction are significant contributors to eventual system performance. This general fact is

a key principle of the field of human/computer interaction [1, 134]. However, the role of

humans in human/neural-prosthesis interaction remains largely unexplored (though see the

Introduction where we discuss the relevance of [25] to this question). The OPS framework

allows systematic study of this important question. Here wedescribe a few examples of

general design questions that can be studied with the OPS.

First, user interface design for neural prostheses will require systematic study. For ex-

ample, in this paper, the subject reached to one of eight targets on a single ring. In a clinical

application, where presumably the goal is to maximize information throughput or user ex-

perience, how many targets should be placed on the screen, and in what configuration? Our

previous work [23] addressed this point algorithmically, but that was in an offline setting.

Testing this online in the presence of feedback-control is important for user interface de-

sign, but it may also be overly time-consuming for a clinicaltrial subject. However, the

OPS, insomuch as it involves a human in closed-loop and has similar qualitative and quan-

titative performance to a real BMI, can readily be used to investigate this and similar user

interface questions.

Second, offline analyses do not explore the sensitivity of prosthesis users to noise (neu-

ral spiking variability, for example). Different systems and algorithms will cope with this

noise in different ways, and it is critical to understand howthese different algorithmic fea-

tures are managed by human subjects. In an extreme case, evenwith an optimal decoder

3.5. DISCUSSION 51

of neural activity, it is not clear at what level of noise a neural prosthetic device becomes

unusable. For example, an important clinical question is, “how many recorded neurons

are needed to control a prosthesis?” Such a question has major bearing on translational

efforts. While offline studies often perform “neuron-dropping” analyses (e.g., [12,79]) to

test the sensitivity of error to neural population size, these studies can not investigate the

neuron count at which a user can no longer control the prosthesis in a satisfactory way. To

our knowledge only one study [38] has done a similar analysis online, and it suggests that

dropping a meaningful percentage of a small number of stableneurons (that study recorded

from 10 and 15 neurons) can have a highly detrimental effect on relative performance. For

larger populations of neurons and different experimental contexts, however, this question

remains unanswered. The OPS allows this analysis: we can vary the number of synthetic

neurons used in the decode to find the point at which the human subject can no longer

successfully complete the task. This and the above featurescan not be tested in offline

data, when the user can not interact with the prosthesis. Furthermore, rigorously testing

this feature in an online, real BMI may be quite difficult giventhe rarity of human clinical

trials and the frustration that such a study would cause a monkey subject.

Third, in terms of the clinical utility of a BMI, the OPS allowscustomization of the

controller based on the specific user application. For example, BMI signals demonstrated

to date would have difficulty controlling a computer mouse ona normal operating system

without the use of specialized accessibility software (see[57]). This software could involve

non-linear mouse acceleration curves [62] or multiple-level mouse selections (selecting

once to zoom in and a second time to click [75]). Such software may be difficult to test

in an animal model due to having many free parameters to sweepand abstract selection

mechanisms that a monkey will not readily learn. Before usingsuch as system in a human

clinical trial, the OPS can serve as a test bench for developing these interfaces and testing

them at a wide range of potential signal quality.

More generally, again by analogy to human/computer interaction, the design of neural

prostheses will require systematic study. In addition, as the field moves towards clinically

viable prostheses, the role of a human in a prosthetic systemis increasingly important.

Indeed, recent years have seen increased importance of human studies for neural prostheses


[57,69,78]. The OPS should help systematize this design process with ahuman in closed-

loop control. We now discuss more detailed algorithmic features that the OPS can address.

3.5.2 The OPS for specific algorithmic questions

Generating synthetic neural activity is the key enabler of the OPS, as it allows many human

subjects to be tested without neural implants. However, generating synthetic neural data is

also the principal concern with the OPS, as this choice callsinto question the relevance of

OPS findings for real BMI systems. In the Results, we demonstrated that OPS and BMI

modes are consistent for optimizing the bin width of the Kalman Filter, showing that indeed

the OPS can be used as a valuable proxy. Nonetheless, for other algorithmic questions,

there remains the risk that the resulting OPS effects will not translate well into a real neural

system. To address this risk and to help ensure legitimacy ofthe result, we here identify

four areas in which the OPS will continue to allow meaningfulinvestigation of algorithmic

design.

First, algorithmic models for arm reaching can be meaningfully studied with the OPS.

The human motor plant imposes significant constraints on thefrequency content, speed,

and extent of a reach. For example, it is known that the motor system naturally produces

smooth movements [112]. This and other constraints have been well studied in humanbe-

havioral studies [43, 112, 135, 136], but they have been largely neglected in the design of

neural prosthetic decode algorithms. Steps in this direction have been taken to acknowl-

edge the point-to-point nature of reaches [74,87,119–121,140,141], but the field has not

produced a model for arm reaching that will generalize to theeventual prosthesis user

mode of unconstrained natural reaching. The Kalman filter used in this study stipulates a

linear prior model for arm reaching (Eq.3.3), but this model appears only in the decode

algorithm, not in the generation of synthetic neural activity. Critically, the generation of

synthetic neural data has no notion of a model of arm reaching. Accordingly, the OPS

can be legitimately used to study the online effect of different algorithmic models for arm

reaching, and performance improvements discovered here should port reliably to real BMI

mode.

Second, the OPS can be used to study the algorithmic mappingsfrom neural activity to

3.5. DISCUSSION 53

kinematics. Many decode algorithms like the Kalman filter stipulate a linear mapping from

neural activity to kinematic reach parameters (Eq.3.4), but there are also many approaches

that use nonlinear mappings. Understanding this algorithmic mapping is often considered

of critical importance, but it has not been exhaustively studied online (though see [79]). The

OPS of this manuscript generates synthetic neural data via alinear mapping (Eq.3.1), so

its current use for this question would be problematic, since the mapping from kinematics

to neural activity is linear by assumption. However, futurework can go further and use a

source of synthetic neural signal that is not directly and simply related to kinematics. We

think EMG offers a nice opportunity. EMG has been shown to have a linear relationship to

neural activity [104], so synthetic motor cortical neural activity can be reasonably generated

from EMG. However, the connection between kinematics and EMG is by no means simple

( [97], and particularly [125] discuss the difficulty of predicting kinematics from recorded

EMG). Thus, by using EMG to create reasonable synthetic neural activity, we can guarantee

a “non-trivial” relationship between the synthetic neuralactivity that we record and the

endpoint kinematics that we want to decode. This approach would allow us to use the OPS

to study the online implications of linear and non-linear algorithmic mappings between

kinematics and neural activity.

Third, we can evaluate very basic assumptions of particulardecode algorithms. In this

study we considered the decode integration bin width of the Kalman Filter. This time win-

dow specifies both the unit of time for output updates (changing the cursor position) and

the unit of time for input integration (integration of neural activity to use as the decode

signal). To remain specifically within the Kalman Filter framework as has been done in

the past, this agreement between update rate and integration rate is required. However, in

practice this assumption can be relaxed. It may be that the difficulty with the large inte-

gration windows has more to do with the hopping or stroboscopic behavior of the cursor

(the slow update rate) than with the long integration rate. Perhaps maintaining a long in-

tegration rate but making a quicker update rate (smoothly varying the cursor between two

decoded positions, for example) could improve performance. Our preliminary investigation

into this question suggests that further performance gainsmay be achievable by optimiz-

ing jointly the cursor-update-rate/neural-integration-rate pair. Since this question inves-

tigates human/neural-prosthesis interaction and not specific neural properties, the OPS is


well suited to address this optimization directly.

Fourth, another basic assumption of a decode algorithm is the time lags it assumes

between recorded neural activity and movement. This parameter is often optimized in BMI

applications (e.g., [139, 141]). However, much like the Kalman Filter bin width of this

study, time lags are rarely optimized online. Since time lags would be an assumption of the

synthetic neural activity in the OPS, discovering the “true” online optimal time-lag is not

likely to be an appropriate question for the OPS. However, understanding the sensitivity of

this choice is indeed readily testable with the OPS. One can choose a set of time lags for

the synthetic neural activity and then test performance sensitivity to different optimization

strategies (for example, studies have included no time lags, a single time lag across all

neurons, or individual lags per neuron). The OPS thus enables us to study how important

time lags are to closed-loop performance.

These four examples (by no means the only examples) show thatthe OPS paradigm

should continue to be valuable to validate algorithmic advances and translational needs.

3.5.3 The OPS for neuroscientific questions

While the OPS may be viewed primarily as a platform for designing closed-loop neural

prosthetic systems, it can also serve as a tool for investigating motor neurophysiology.

Findings in the applied BMI context can often offer insight into the normal-functioning

motor system. As a specific example, we found here that shorter temporal decode updates

(bin widths) lead to smaller decode error. However, as we note above (the third example in

“The OPS for specific algorithmic questions”), future work should aim to understand the

contribution to BMI decode errors between the stroboscopic effect of the cursor updates and

the integration of shorter amounts of neural data. This stroboscopic effect represents a form

of noise to the visual system. By altering that noise independently of other cursor control

parameters, one could study the effects of noise on learning[97], reflex adaptation in the

motor system [35], or multimodal sensory integration (separating visual and proprioceptive

contributions) [51,118,128].

More generally, the OPS can serve as an experimental system for motor control studies,

and it has interesting connections to that substantial literature (e.g., [112,135]). There has

3.5. DISCUSSION 55

been a great deal of research using variants of a robotic manipulandum to study perturba-

tions of normal motor control and to introduce visuomotor discrepancies (see [60] for a

review). These studies ask questions ranging from dynamic learning in the motor system,

to object manipulation, to limb stiffness measurements: a classic example is subjects mak-

ing reaches in the presence of an external force field [111]. These and other studies point

to the fact that the introduction of distortions between intended movement and perceived

movement has been a fundamental aspect of motor research formany years. The OPS is a

similar system that allows novel 3D visuomotor perturbations (in particular perturbations

that are relevant to the applied field of neural prostheses) and manipulations of the sys-

tem’s underlying dynamics at several levels (at the cursor directly, or more implicitly in the

neural encodings). Thus the OPS should be applicable to studies of motor control and 3D

motor adaptation. In addition to the novel perturbations this system may allow, the OPS

may also prove useful as motor control researchers move towards more freely behaving

experimental paradigms (as has been happening, for example, in motor electrophysiologi-

cal studies [101]). For example, due to electrode array shifts and head acceleration events,

recorded neural signals can change abruptly [101]). Studying 3D motor adaptation to such

signal changes in this less constrained context would be beneficial both scientifically in

increasing our understanding of motor learning, and in an applied setting by informing the

field which aspects of motor control are contextually relevant for BMI. Certainly much cau-

tion should be taken in interpreting the neuroscientific implications of a BMI result, but the

OPS should allow investigation of both applied neural prosthesis and fundamental motor

neurophysiology questions.

3.5.4 Cautionary notes regarding the OPS

The Results and our arguments for more systematic online analysis suggest that the OPS

should be used to address a variety of neural prosthetic design choices. However, a number

of questions can not readily be asked with the OPS, and much care should be taken with

any simulation environment to ensure the legitimacy of any findings. Here we describe

important precautions with the OPS.


As previously noted, the biggest potential pitfall of this system is the generation of syn-

thetic neural data. Generating synthetic neural data or asking design questions carelessly

can lead to tautologies that are uninteresting and potentially misleading. In this work, we

specifically optimized the bin width of the Kalman filter. Since the synthetic neural activity

is generated instantaneously and according to a Poisson likelihood model, the bin-width of

the Kalman filter is not related to how the neural activity is generated. Thus, the bin width

optimization was more a question of how the subjects interacted with systems with dif-

ferent noise characteristics. This question could be and was legitimately investigated with

the OPS, as indeed the real BMI experiments validated. However, with the same setup,

one could propose another hypothesis that would prove problematic. Consider a hypothe-

sis to test the plasticity of the mapping between neural activity and kinematics. Certainly

this is an interesting and important question both for scientific reasons and for the design

of neural prosthetic systems (e.g., [38]). However, the answer here is trivial: because we

synthetically created a static mapping between kinematicsand neural activity, indeed this

relationship is by definition without plasticity. So, questions must be asked that do not

directly test an assumption of the generative model for synthetic neural data. Though this

point is perhaps fairly obvious, it is the overarching caveat with the extensibility of the OPS

framework.

Another possible problem with the closed-loop paradigm regards proprioceptive feed-

back. As previously noted, there was a sensory discrepancy between the visual feedback

and the real arm’s proprioception (in prosthetic trials). Whether the true arm is restrained

or allowed to reach (both are regularly done in the BMI literature), there is still a proprio-

ceptive confound. Hence, this potential limitation is not specific to the OPS or this study,

but rather to all able-bodied animal or human BMI studies. Indeed, previous BMI literature

such as [12,109,124,130] have not reported difficulty with this proprioceptive error signal,

and there is no reason to expect that it is any more prominent in the OPS than in those

other able-bodied studies. Furthermore, previous neuroscience research has suggested that

this confound is minor: [97] found that proprioception did not prevent task learning but did

somewhat increase task difficulty. Other work has found thatvision is a stronger sensory

signal than proprioception in certain contexts (e.g., [127]), but this question is still debated

in the neuroscience community. Thus, while this possible limitation is important to bear in

3.6. ACKNOWLEDGMENTS 57

mind for the OPS and all able-bodied BMI studies, the significant progress that has been

made in this field even with this proprioceptive confound suggests that the OPS framework

should continue to be useful for aiding in the design of clinically relevant prostheses.

3.5.5 Summary

The OPS paradigm provides a middle ground between simple (but low reality) offline test-

ing and more realistic (but very involved) clinical trials.We showed here how the OPS can

be effectively used to find a surprising result that stands incontrast to offline analysis and

previous literature: the bin width of the Kalman filter should be decreased in size for online

prosthetic studies. These results showed the substantial disagreement between offline and

online analyses, and the results showed the substantial similarity between OPS and BMI

modes by using nonhuman prosthetic experiments as validation. While these findings are

valuable in their own right, the broader message of the work is that offline analysis may

be a poor proxy to eventual system use, and thus the field should investigate simulation

opportunities to create a principled design engineering process around neural prosthetic

systems. In this example, considering human interaction with the system was of critical

importance, and we speculate that such feedback-control isindeed critical for many other

aspects of prosthetic system design. Ideally, the OPS or similar online simulators may be-

come a piece of the neural prosthesis researcher’s toolkit,to allow rigorous design before

moving to the gold standards of monkey BMI experiments and human clinical trials.

3.6 Acknowledgments

We thank D. Franklin for helpful conversations, M. Risch for veterinary care, D. Haven for

technical support, and S. Eisensee for administrative support.

Chapter 4

ReFIT-KF Neural Prosthesis

This chapter appeared in Nature Neuroscience in 2012 as a paper titled “A high-performance

neural prosthesis enabled by control algorithm design.” The paper was authored by Vikash

Gilja, Paul Nuyujukian, Cynthia Chestek, John Cunningham, Byron Yu, Joline Fan, Mark

Churchland, Matt Kaufman, Jonathan Kao, Stephen Ryu, and Krishna Shenoy. The au-

thor’s contribution was in experimental rig design and setup, assistance in decoder design,

data collection and analysis, and writing the manuscript.

4.1 Abstract

Neural prostheses, also referred to as brain-machine interfaces or brain-computer inter-

faces, translate neural activity from the brain into control signals for guiding prosthetic de-

vices, such as computer cursors and robotic limbs, and thus offer disabled patients greater

interaction with the world. However, relatively low performance remains a critical barrier

to successful clinical translation; current neural prostheses are considerably slower with

less accurate control than the native arm. Here we present a new control algorithm, the re-

calibrated feedback intention-trained Kalman filter (ReFIT-KF), that incorporates assump-

tions about the nature of closed loop neural prosthetic control. When tested with rhesus

monkeys implanted with motor cortical electrode arrays, the ReFIT-KF algorithm outper-

forms existing neural prostheses in all measured domains and halves acquisition time. This

control algorithm permits sustained uninterrupted use forhours and generalizes to more

58


challenging tasks without retraining. Using this algorithm, we demonstrate repeatable high

performance for years after implantation across two monkeys, thereby increasing the clini-

cal viability of neural prostheses.

4.2 Introduction

Neural prostheses have recently demonstrated considerable promise through proof-of-concept

animal experiments [12, 38, 80, 87, 109, 116, 122, 124, 130] and in human clinical trials

[57, 58, 69, 70] for partially restoring motor output in paralyzed individuals. Studies in

this field primarily focus on adapting insight and methods from the basic neuroscience of

cortical motor control to this engineering context. A critical example of this is the use

of motor cortex tuning models, which describe the relationship between single unit firing

rates and arm movement kinematics, to define a mapping for neural control of a computer

cursor in closed loop (e.g. [12, 109, 124]). When such a neural prosthesis is introduced

to a monkey, performance can increase across days through learning [12]. In addition to

controlling computer cursors, these systems have successfully driven robotic end effec-

tors [130]. Additional concepts from motor neuroscience have been incorporated, demon-

strating the potential to augment system performance by modeling neural activity related

to movement preparation [87] and proprioceptive feedback [122]. Recent work also sug-

gests that when the recorded neural population and control algorithm are held constant,

neural prosthetic performance increases over time as a stable neural output map is formed

and multiple mappings, once learned, can be retained and retrieved across different control

contexts [38]. Despite these new insights and additional algorithmic advances (e.g. [70]),

system performance on simple cursor control tasks remains low relative to native arm con-

trol performance, presenting a critical barrier to clinical translation [63].

In an effort to improve the performance of neural prostheses, we chose to focus on a

systems engineering approach. Building on existing methodsin the field, we developed

two key innovations that alter the modeling assumptions made by these algorithms and the

methods by which these algorithms are trained. Additionally, signal conditioning methods,

which transform recorded neural signals into control algorithm input, were chosen in an

effort to improve system stability and performance [18, 24]. As demonstrated in closed

60 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS

loop neural control experiments, these methods result in high performance across multiple

cursor control tasks.

4.3 Results

4.3.1 Performance Overview

We trained monkeys to acquire targets with a cursor controlled by either native arm

movement or neural activity. We developed a novel algorithm, ReFIT-KF, that led to sub-

stantially higher-performance neural prosthetic control. Figure4.1a shows representative

continuous, uninterrupted cursor movements for three different modalities: native arm con-

trol, ReFIT-KF, and a velocity Kalman filter (Velocity-KF), that is state-of-the-art for cur-

rent neural prostheses (e.g., [58, 69, 70]). Monkeys were required to move the computer

cursor to a visual target and hold the cursor within a demand box for 500 ms to successfully

complete a trial and receive a liquid reward. Targets alternated between central and eight

peripheral locations. During online neural control sessions, the contralateral arm was not

restrained, and movement continued. However, the physicalmovement was not stereotyped

and would often attenuate or even stop during some neural control sessions, while retain-

ing performance. In a set of additional control experiments, both arms were restrained, and

little or no arm movement was observed with similar performance (see Table4.1).

The ReFIT-KF algorithm outperformed the Velocity-KF by several measures. First,

cursor movements with ReFIT-KF control were straighter (Fig. 4.1a,b), producing less

movement away from a straight line to the target. Cursor movements produced using the

ReFIT-KF were qualitatively similar to native arm movements(Fig. 4.1a). Second, these

movements were also completed faster. ReFIT-KF cursor control performance, as mea-

sured by the time to successfully acquire the target (Fig.4.1c,d), was 75-85% of native

arm control performance and at least twice Velocity-KF performance. In addition to lower

mean time to target, the variance was substantially smaller, which is important as this sig-

nifies greater movement consistency and fewer potentially frustrating long trials. Critical

4.3. RESULTS 61

Figure 4.1:Performance comparison. Native arm control shown in blue, ReFIT-KF inred, and Velocity-KF in green. (a) Representative traces of cursor path during center-out-and-back reaches. Dashed lines (not visible to the monkey) are the demand boxes forthe eight peripheral targets and the central target, shown as translucent green circles. (b)Bar graphs plotting maximum deviation from a straight-line path to the target on eachsuccessful trial (mean± SE). (c,d) Histograms of time to target for successful trials areshown as line graphs for monkeys J and L. The inset bar graphs plot the time to target(mean± SE). (e,f) Line graphs plotting the mean distance to the target as a function oftime. The inset bar graph plots the mean± SE of the dial-in time, or the time requiredto finally settle on the demand box, after first acquired, to successfully hold for 500 ms.The thickened portion of the line graphs also indicate dial-in time, beginning at the meantime of first target acquire, and ending at mean trial duration minus 500ms. Data fromexperiments J & L 20101027-29, 20101102.


Figure 4.2:Performance of ReFIT-KF control across 4 years.Performance is measuredby the Fitts’ law metric. Data from monkey J and monkey L are shown as 94 orange circlesand 182 cyan squares, respectively. Each point plots the performance of the ReFIT-KFalgorithm trained on that experimental day. The eight filleddata points (four for each mon-key) are calculated from the same datasets used to generate Figure4.1. Linear regressionlines for data from monkey J (orange) and monkey L (cyan) are shown. The slope of theregression line for monkey J is not statistically significant from zero (p>0.34). The slopesfor monkey L is statistically significant from zero (p<0.001). For all datasets shown, thetrial success rate was>90%.

to achieving this lower time to target, ReFIT-KF controlled cursor movements stopped bet-

ter. The ability to stop is a critical differentiator between the three control modes. The

Velocity-KF took only modestly longer to first acquire the target than other control modes,

but often significantly overshot the target, requiring additional time and multiple passes to

stably acquire and hold the target. This overshoot-correction time dominates the overall

time to successful target acquisition for Velocity-KF control (Fig. 4.1c,d) and is captured

by “dial-in time” (Fig.4.1e,f, bar graphs and bold lines): the average time required tomake

the final target acquisition after having first reached the target. Both native arm and ReFIT-

KF control allowed more precise stopping as compared to Velocity-KF (Fig. 4.1e,f; final

distance to target). Across all trials in eight experimental sessions with two monkeys, when

given five seconds to acquire targets, ReFIT-KF achieved a success rate of>99%, while

Velocity-KF had a success rate of 95%. The task difficulty waschosen to achieve a high

success rate for all three control modalities on the first experimental day. When the task dif-

ficulty is increased, the success rate with Velocity-KF can drop relative to the success rate

with ReFIT-KF, and similarly ReFIT-KF success rates and performance can drop relative

to native arm control (see generalization tasks described below).

Experiments across days and years demonstrated consistenthigh performance. As

4.3. RESULTS 63

shown in Figure4.2, performance was stable as measured by throughput on 280 individ-

ual experimental days. These data were collected across 29 months for monkey L and 16

months monkey J, spanning the range of 0.4 to 4.4 years post-array implantation. To ex-

plore the possibility that performance changed with time, we computed least squares linear

fits on these performance data for each monkey. The slopes of both regression lines are

positive, suggesting that performance was stable over the time period of the study and pro-

viding evidence consistent with the hypothesis that intracortical microelectrode arrays may

permit years of high performance neural control [18].

4.3.2 Generalization & Robustness

We also tested additional behavioral tasks to assess generalization of the ReFIT-KF

control algorithm. We fit the ReFIT-KF algorithm with center-out-and-back reaches, as

before, and then tested on a pinball task in which targets could appear at any location in the

2D workspace. Monkeys were again required to move the cursorto the target and hold it

for 500 ms to successfully complete a trial. Figure4.3shows over 90 minutes of data from

monkey L during two continuous pinball reaching sessions, one with native arm control

and one with the ReFIT-KF control. Given two seconds to acquire a target on each trial,

both sessions had success rates>98%. Across the whole session, the mean time to target

for ReFIT-KF control was 72% as fast as native arm control (ReFIT-KF: 710± 317 ms;

native arm: 519± 196 ms). Comparable performance was found for monkey J. Figure 4.3

illustrates that the performance with ReFIT-KF was not only high (over two-thirds as fast

as the natural arm; comparable distribution widths in Fig.4.1 and Fig.4.3a), but was also

sustained without intervention (Fig.4.3b). Sustained performance was typical of ReFIT-

KF control sessions, whereas Velocity-KF control sessionswith the same task parameters

had much lower success rates (<40%), and the monkeys could not be motivated to acquire

targets for more than 30 minutes.

To further test ReFIT-KF control, we trained monkey J to avoidvisually defined obsta-

cles that appeared in the direct path of the target (maze task[19,66]). The monkey reached

from a central starting target to either a left or right peripheral target. As shown in Fig.4.4,


Figure 4.3:Performance comparison of native arm vs. ReFIT-KF for the pinball task.Native arm control is in blue, ReFIT-KF in red. In this task, each target location is selectedfrom a uniform distribution spanning the workspace. (a) Each column shows data from20-minute segments. The top rows are randomly selected cursor traces for 4 subsequenttarget acquisitions. Target demand boxes are shown as dashed lines and target sequenceis indicated from 0 to 4. The bottom row shows normalized histograms of time to targetfor successful trials. Arrows below the plot indicate average time. (b) Target acquisitionrate per minute throughout the sessions is shown. The sharp rate drop indicates when themonkey lost interest in the task. A histogram of acquisitionrate across the sessions is inset.The native arm and ReFIT-KF sessions (L-2010-04-01 and L-2010-04-12) were on twoseparate days, within 11 days of each other, when the monkey demonstrated a high degreeof motivation.

4.3. RESULTS 65

Figure 4.4: Performance comparison of native arm vs. ReFIT-KF for the obstacleavoidance task. Native arm control shown in blue, ReFIT-KF in red. In this taskthemonkey had to move the cursor from the initial target (labelled 0) to the final target (labelled1, demand box shown as dashed line) without hitting the magenta-colored barrier. Onerepresentative cursor trace is shown from each of the four principle observed movementtypes: curve under, curve over, straight (no barrier), and collision into barrier. These dataare from experiment J-2010-03-09.


Figure 4.5: Illustrations of the online neural control paradigm and the ReFIT-KFtraining methodology. (a) The input to the control algorithm at timet is a vector of spikecounts,yt , from implanted electrodes.yt is translated into a velocity output,vt , to drivethe cursor. (b) ReFIT-KF is trained in two-stages. Initially, cursor kinematics and neuralactivity are collected during arm control or during an observation phase in which cursormovement is automated (i). These arm movement or observed cursor kinematics (ii) areregressed against neural activity to generate an initial control algorithm. Then, a new setof cursor kinematics and neural activity are collected using the initial algorithm in closedloop (iii). The kinematics collected during neural control(red vectors in (iv)) are used toestimate intention by rotating the velocities towards the goal (cyan vectors in (iv)). Thisestimate of intended kinematics is regressed against neural activity to generate and runReFIT-KF (v).

on some trials a barrier appeared along with the peripheral target. To successfully com-

plete a trial, the monkey had to use the cursor to acquire and hold the peripheral target for

500 ms without hitting the barrier. This task was difficult, but the monkey successfully

acquired and held the target on 77% of trials with his native arm (Fig. 4.4a) and on 75%

of trials with ReFIT-KF control (Fig.4.4b). Under ReFIT-KF control, mean time to target

for this task was 74% as fast as with native arm control (ReFIT-KF: 1253± 588 ms; native

arm: 932± 709 ms; mean +/- standard deviation). With Velocity-KF control, the monkey

could not complete the task and quickly became frustrated and disengaged. As in the pre-

vious tasks, the ReFIT-KF was fit with center-out-and-back movements and used without

modification for the maze task, demonstrating robustness across behavioral contexts.

4.3. RESULTS 67

4.3.3 Two Innovations

The described cursor control performance was achieved by redesigning the Velocity-KF

algorithm from a feedback control perspective. As shown in Fig. 4.5a, the prosthetic de-

vice constitutes a new physical plant with different dynamical properties than the native

arm. The subject controls this new plant by modulating measured neural signals (yt in

Fig. 4.5a-i), which are then decoded (Fig.4.5a-ii) into a velocity by the control algorithm.

This velocity is used to update the cursor on screen (Fig.4.5a-iii), affecting neural signals

on subsequent time steps. This closed loop control perspective suggests two design innova-

tions. The first innovation is a modification of neural prosthetic model fitting methodology.

The second innovation is an alteration of the control algorithm. These ReFIT-KF innova-

tions produce the neural prosthetic results described above.

4.3.4 Innovation 1

The first design innovation is to fit the neural prosthesis against estimates of intended ve-

locity. Previous algorithms [12,57,87,109] implicitly assume that the subject uses the same

control strategy for moving the native arm and the prosthetic cursor. Since these control

strategies may be quite different, we, in the vein of past studies [36,37,70,80,116,124,130],

evaluated methods that attempt to better capture the subject’s strategy during prosthetic

control. Ideally, the control algorithm would be fit to the subject’s intended cursor velocity

during closed loop neural control. Since we lack explicit access to the monkey’s inten-

tions, we hypothesized that the monkey wished to move directly towards the target; this

resembles movements made by the native arm and is a good strategy for acquiring rewards.

A two-stage optimization procedure (Fig.4.5b) is used to fit the neural prosthetic model

to these estimates of intended velocity during online neural control. In stage 1, the monkey

controls the cursor using his arm (Fig.4.5b-i). An initial model is fit using arm trajectories

and simultaneously recorded neural signals (Fig.4.5b-ii). The monkey controls the neural

prosthesis with this initial model (Fig.4.5b-iii). In stage 2, neurally controlled cursor

kinematics and neural signals are recorded and used to fit a new model with an estimate of

intended cursor velocity. By starting with cursor velocities collected during the previous

online control session (shown in red), these estimates are calculated for model fitting using


two transforms. First, the velocities are rotated towards the target (blue vectors) to generate

the set of estimated intended velocities (Fig.4.5b-iv). Second, if the cursor is on target, the

monkey’s best strategy is to keep the cursor still to satisfythe hold time requirement. Thus,

in the training set, we assume that the monkey’s intention during these hold periods was to

maintain the cursor position by commanding zero velocity. This zero velocity assumption

applied to the fitting of model parameters improves online performance without changing

the control algorithm. These estimated intentions and corresponding neural data are used

to fit the ReFIT-KF control algorithm (Fig.4.5b-v). It is important to note that the intention

estimation is appliedonly to training data:during online control the neural prosthesis has

no knowledge of the task goal or placement of targets (unlikee.g. [87,120,141]).

The aforementioned training protocol utilizes hand-controlled reaches as training data.

In a paralyzed individual, it is not possible to record arm kinematics for this step. Instead,

this training step could rely on the individual imagining a set of instructed movements.

To test this possible strategy, we trained the initial algorithm based on visual cue observa-

tion [70, 122], removing the requirement for arm control in step one. During these trials

the monkey watched a computer controlled training cursor that automatically moves out

to targets. The initial model was fit using automated training cursor trajectories and si-

multaneously recorded neural activity, without using measured arm movement. Table4.1

summarizes ReFIT-KF performance for three experimental sessions from each monkey in

which stage 1 of ReFIT-KF model training is based on observation data instead of arm

movements. The performance, as measured by Fitts’ Law [10], for these sessions is similar

to that attained for the native arm control initiated sessions described in the results section

and summarized in Figures4.1and4.2.

4.3.5 Innovation 2

The second design innovation builds on the observation thatneural activity is correlated

with both the velocity and the position of the cursor. Most existing neural prostheses model

a relationship between neural activity and either velocity[124,130] or position [57,109].

A human clinical trial has shown that neural prostheses modeling velocity have higher per-

formance [69,70]. However, if the control algorithm models only the velocity relationship,

4.4. DISCUSSION 69

Experiment Targetcenterdistance(mm)

Windowsize(mm)

Acquiretime(sec)

Index ofdifficulty(bits)

Fittsbits/sec

SuccessRate

L-2010-08-10 80 40 0.89 1.32 1.49 94%L-2010-08-11 80 40 0.89 1.32 1.48 95%L-2010-08-12 80 40 0.83 1.32 1.59 94%

J-2010-08-10 80 40 0.76 1.32 1.74 97%J-2010-08-16 80 40 0.82 1.32 1.60 97%J-2010-08-17 80 40 0.76 1.32 1.73 98%

Table 4.1: Performance of ReFIT-KF based control with observation based algorithm train-ing.

then position-based changes in firing will confound decodedvelocities. To mitigate this

effect, we explicitly model velocity as the user’s intention and cursor position as. This

modification allows the user to control velocity with measured neural signals while ac-

counting for the influence of cursor position. We explicitlyassume that the current cursor

position, determined by integrating the previous velocityoutput, is encoded in the neural

activity along with the monkey’s current intended velocityoutput. Thus, the expected con-

tribution of position to neural activity is removed, enabling more accurate estimation of

intended velocity (Fig.4.1).

4.4 Discussion

Other studies have noted the potential change in plant and control strategy and have ad-

dressed it by iteratively refining parameters during neuralprosthetic experiments [36, 37,

70,80,116,124,130]. This approach recognizes that control strategies, and therefore model

parameters, are best measured and understood during closed-loop neural prosthetic exper-

iments. However, randomizing initial parameters [36, 124, 130] may create a control al-

gorithm that never attains the highest possible performance, just as optimization problems

can easily become trapped in local optima. Although, if the recorded neural population

and the neural control mapping are held constant, the consequences of the plant mismatch


can be overcome through learning. Such learning was demonstrated with neural control

mappings built to reconstruct arm kinematics, as well as with a neural control mapping in

which neuron identities were shuffled, so the decoder outputwas no longer predictive of

native arm kinematics [38].

The focus of the present study was to obtain high control performance within a single

session by improving the neural control algorithm and optimizing its parameters. Although

the neural prosthesis constitutes a new plant with different properties than the native arm,

the motor cortices are involved in native arm control (e.g.,[64]). Therefore, we hypoth-

esized that initializing a model with the relationship between neural activity and natural

arm movement would allow the second stage of our training method to achieve a higher

level of optimization. Previous studies [36, 130] have relied on manipulating the control

task to refine the neural decoder, such as by providing assisted control. In those studies, an

automated correct answer was mixed with the output of the neural prosthesis. Over suc-

cessive iterations the weight of automated control was decreased by experimenter intuition

until control was driven only by neural activity. Our approach is different, as the control

task remains constant throughout a neural prosthetic session and only the training data are

manipulated between the first and second neural prosthetic sessions.

In a study with quadriplegic humans [69,70], the neural prosthesis was initially trained

with visual cue observation, similar to the control experiments described above. The study

also uses a second neural prosthetic training session to account for differences during online

control. Unlike ReFIT-KF training, both the neural cursor and the automated training cursor

were on screen during the second session. The neural cursor was presented to provide

feedback so that the participant could attempt to alter their neural output to better follow

the training cursor. After this second session, the neural prosthesis was fit with the training

cursor kinematics. Thus, the underlying assumption is thatthe training cursor kinematics

capture the intended kinematics during online control, while ReFIT-KF fitting assumes

that intended kinematics are best inferred by the output of the neurally controlled cursor

and knowledge of the task goals.

In studies with adaptive decoders [80, 116], the kinematics of the neurally controlled

cursor are continuously used to refine neural prosthetic parameters, also allowing them to

4.4. DISCUSSION 71

account for differences when switching to online control. However, they take different ap-

proaches to estimating intended kinematics for retraining. One approach is to use decoder

parameters non-causually, via smoothing, to estimate intended kinematics for retaining

without task or target information [80]. Impressively, this method was shown to slow per-

formance declines in one monkey over 29 days when using static spike sorting. However,

unlike with ReFIT-KF model fitting, initial performance was not surpassed, perhaps be-

cause without incorporating task goals, the method is subject to inaccuracies present in the

initial model fit. In another adaptive study [116], target information was incorporated in

the kinematics used for retraining. Their algorithm was retrained with a weighted aver-

age of decoded trajectory positions and the target positionfor each trial as an estimate of

intended position. In contrast, ReFIT-KF estimates intended velocities based on intuitive

rules applied to cursor position, decoded velocity, and target position.

ReFIT-KF explicitly treats position and velocity differently. The resulting neural pros-

thesis assumes that the monkey is controlling velocity and not position. We structured the

model assuming that velocity intentions evolve smoothly and that the influence of position

is based on the monkey’s internal model of the cursor. Furthermore, we assume that the

control algorithm output and the monkey’s internal belief about cursor position agree. In

reality, there is some mismatch between the control algorithm’s position estimation and

the animal’s internal belief due to inaccuracies in assessing visual information. There are

likely spatial and temporal components to this inaccuracy that are not modeled. The spatial

aspect is an inexact assessment of the last seen location, and the temporal aspect is due to

visual latency. The spatial aspect could be modeled as fixed position uncertainty. To fully

account for the temporal aspect, one could attempt to algorithmically model the animal’s

internal model of cursor dynamics since the last known position of the cursor. In this work,

we chose to start with a simpler model, assuming that this estimation, which is local in

time, is exact. It is possible that augmenting the algorithmto account for the mismatch be-

tween the temporally local forward model and our dynamics model could further increase

control performance. It is important to note that there may be alternative explanations for

the presence of position information in neural output. For example, this information could

be intended cursor position instead of an internal model estimate of current cursor position.

In support of the internal model hypothesis, a recent study suggests that a forward model of


cursor position is used during closed loop control [50]. However, further study of the role

of position information in the neural activity during online control is necessary and could

aid in the development of future control algorithms.

In experimental sessions, ReFIT-KF performance was stable until the monkey appeared

to lose interest in the task (e.g. drop in target acquisitionrate, Figure4.3b). This rapid drop

off is presumably the analogue of when a hypothetical human user is finished using their

neural prosthesis. It is expected that performance will drift over time [63], and methods

for continuous adaptation of neural control algorithm parameters may be necessary. In a

previous study [80], information from the output of the control algorithm was used with a

Bayesian approach to adapt parameters throughout sessions to sustain performance. If task

goals were known throughout neural prosthesis use, the intention estimates defined in this

study could be used in conjunction with these parameter adaptation methods. It may be

possible to estimate these task goals based on features of the neural prosthetic output. For

example, if a click or target selection signal is simultaneously decoded [70], indicating user

intended target selection, intended cursor velocities could be estimated for moments prior to

target selection. Additionally, in future work, it will be important to assess how multi-day

learning [38] affects the performance and robustness when control algorithm parameters

are set as described in this work, based on estimated movement intention, versus existing

methods for parameter initialization. Adapting the methods of this work to enable multi-

day learning and plasticity, such as by providing a consistent controller day over day, may

well lead to even better performance over time.

Long-duration, continuous, high-performance operation is central to successful trans-

lation of neural prostheses to human patients [63]. The above performance depended upon

three specific design choices used by both Velocity-KF and ReFIT-KF, in addition to the

two key innovations defining ReFIT-KF. First, we did not employ spike sorting. The goal

of spike sorting is to separate single channels composed of action potentials from many

neurons into multiple channels of spiking activity from individual neurons. This standard

practice can yield high information per electrode, but requires tracking each sorted action

potential shape over days, which has recently been shown to be extremely difficult for many

electrode channels [15,38,101]. To reduce signal instabilities that can result from imper-

fect spike sorting and neuron tracking, we counted the number of threshold crossings per

4.4. DISCUSSION 73

electrode instead of spike sorting [18,36]. Second, the results reported here were acquired

from arrays 19-40 months (monkey L) and 4-21 months (monkey J) after neurosurgical

implantation [15, 18]. The number of highly distinguishable single neurons on anelec-

trode array tends to decrease over time. Yet, remaining multiunit activity often has neural

prosthesis relevant tuning. By employing these older array implants, which had relatively

few clearly distinguishable single units, we confirm that threshold-crossing-based activity,

together with the ReFIT-KF, provides high performance for months and years after array

implantation (shown in Fig.4.2). Finally, we used a single, relatively short 50 ms neural

integration time window with no additional temporal lag, unlike some neural prosthetic de-

signs that explicitly incorporate neural data with longer histories and additional lags (e.g.,

multiple 100 ms time bin [38]; multiple 50 ms time bins with history as far back as 1

sec [57,122]). This choice was based on experiments with humans using anonline pros-

thetic simulator [24] and on subsequent neural control experiments with monkeys. Both

indicated that shorter time bins are preferable due to reduced closed-loop feedback time.

The sustained performance and robustness of these ReFIT-KF neural prosthetic experi-

ments demonstrate the potential to provide functional restoration for patients with a limited

ability to move and act upon the world due to neurological injury and disease. Although

descending pathways are compromised, motor cortex may be largely intact, enabling this

class of technology [56,57,69,70]. In recent years, brain interface technologies employing

a variety of signal sources, such as intra-cortical arrays described here, electroencephalog-

raphy (EEG) [82], and electrocorticography (ECoG) [106], have been developed. The

neural prostheses research community continues to create options for disabled individuals

and to assess relative risk and benefit [44]. In this report, we have investigated the princi-

pled design of closed loop neural control algorithms, resulting in the development of the

ReFIT-KF and demonstrations of a significant advance in performance and robustness. This

algorithm, closed loop control perspective, and system design methodology may be applied

to other neural prosthetic domains with the potential to considerably increase benefit and

the clinical viability prostheses.


4.5 Methods

All procedures and experiments were approved by the Stanford University Institutional

Animal Care and Use Committee (IACUC). Experiments were conducted with adult male

rhesus macaques (L and J), implanted with 96 electrode Utah arrays (Blackrock Microsys-

tems Inc., Salt Lake City, UT) using standard neurosurgical techniques [102]. Monkeys L

and J were implanted 19-33 months and 4-14 months prior to theexperiments. Electrode

arrays were implanted in the dorsal aspect of premotor cortex (PMd) and primary motor

cortex (M1), as estimated visually from local anatomical landmarks.

The monkeys were trained to make point-to-point reaches in a2D plane with a virtual

cursor controlled by the contralateral arm or by a neural decoder [24]. The virtual cursor

and targets were presented in a 3D environment (MSMS, MDDF, USC, Los Angeles, CA).

Hand position data were measured with an infrared reflectivebead tracking system (Po-

laris, Northern Digital, Ontario, Canada). Spike counts were collected by applying a single

negative threshold, set to 4.5 x root mean square of the spikeband of each neural channel.

Behavioral control and neural decode were run on separate PCs using the Simulink/xPC

platform (Mathworks, Natick, MA) with communication latencies of 3 ms. This system

enabled millisecond-timing precision for all computations. Neural data were initially pro-

cessed by the Cerebus recording system (Blackrock Microsystems Inc., Salt Lake City, UT)

and were available to the behavioral control system within 5± 1 ms. Visual presentation

was provided via two LCD monitors with refresh rates at 120 Hz,yielding frame updates

within 7 ± 4 ms. Two mirrors visually fused the displays into a single 3Dpercept for the

user, creating a Wheatstone stereograph (see Fig.3.2).

Over 180 experimental sessions with monkey L, and over 100 with monkey J, were

conducted. Central results were replicated multiple days ineach monkey, employing a

within-day A-B-A block structure trial design to highlight algorithmic impact and thereby

quantify performance and robustness.

4.6. ACKNOWLEDGEMENTS 75

4.6 Acknowledgements

We thank S. Stavisky for data collection assistance, J. Aguayo, M. Risch, & S. Kang for

surgical assistance & veterinary care, D. Haven & B. Oskotskyfor IT support, S. Eisensee,

B. Davis, & E. Castaneda for administrative assistance, and P.Ortega for mathematical

insight.

Chapter 5

Task Optimization

This work is as of yet unpublished. The experiments were designed with and data were

collected with the assistance of Joline Fan.

Chapters3 and4 detailed the development leading to a high performance neural pros-

thesis. This chapter builds upon those results, using the ReFIT-KF decoder on keyboard-

like tasks with the aim to measure achievable bitrate. The findings in this chapter are

unpublished at the time of this writing. The data shown here was collected in conjunction

with Joline M Fan in 2010.

5.1 Introduction

Communication neural prostheses can be used to transmit meaningful information by meaus-

ing signals from the brain. These could be used to control a mouse-like cursor, as shown

in Chapter4, but could also be used to control a keyboard for typing. Prior work with

non-penetrating signal sources have demonstrated sustained bitrates of up to 1.2 bps. Elec-

troencephalography [90] (EEG) and electrocorticography [105] (ECoG) can be used as

the signal source for neural prosthetics. With EEG bitratesof 0.13 [72] to 1.02 bps [5]

have been achieved and 0.38 [73] to 1.15 bps [8] with ECoG. A study with intracortical

electrodes demonstrated a peak channel capcity of 6.5 bps [102] for short bursts of time,

however these were not actualized bitrates. This prior study achieved this peak channel

capcity by optimizing the task parameters to enable high data throughput. In a similar

76

5.2. METHODS 77

fashion, the task parameters for a keyboard-like task must also be optimized to maximize

the bitrate achievable. Maximizing bitrate is desirable because it will enable the highest

communication rate for a given level of neural prosthesis control. In this study, we opti-

mized the performance of two keyboard-like tasks, a grid task and a radial keyboard task

and exceeded 3 bps in both tasks.

5.2 Methods

Two keyboard-like tasks were designed and tested with two rhesus macaques in an exper-

imental apparatus (Fi5.1) that mapped neural threshold crossing activity (at−4.5× RMS

of channel [18]) into two-dimensional cursor velocity using the ReFIT-KF neural decoding

algorithm (Ch4). [47] The first task, called the grid task, uniformly divided the workspace

into square acquisition regions each containing a potential target in yellow as shown in

Fig 5.2a. On a given trial, a random target was prompted by changing its color to green.

The monkeys task was to move the cursor into the acquisition region and hold over it for a

specified time, receiving a liquid reward if successful. An incorrect selection occurred if the

cursor dwelled on any of the yellow non-prompted targets forthe specified hold time. After

a selection, the next trial started immediately and anotherrandom target was prompted.

The second task had targets arranged circularly, each equidistant from the center of

the workspace (Fig5.2b) in a design inspired from similar human-computer interaction

input systems such as pie menus. [59, 89] In this radial task, selections were made by

moving the cursor from the center of the screen into an acquisition region in the periphery

without a requisite hold time. Immediately after selection, the cursor was reset to the center,

beginning the next trial.

Both the grid and radial tasks are examples of symmetric communication channels.

They can be used to transmit one of many symbols (targets) to the receiver and relay in-

formation. As with all real-world symmetric communicationchannels, the symbols trans-

mitted are not known to be correct and have some probability of error. The maximum

information communicable over the channel is a function of this error rate. The error rate

(along with the symbol transmission rate) can be used to calculate a theoretical channel

capacity: the highest possible bitrate given multi-symbolchannel codes and infinite time.

78 CHAPTER 5. TASK OPTIMIZATION

Figure 5.1: Experimental setup. Monkey was placed in front of a Wheatstone stereograph[133] such that hand was not visible and presented with a virtual environment within whichto interact. The monkey’s objective was to acquire the target lit in green. Neural data wasrecorded simultaneously during the task. Under neural prosthesis operation, the movementof the cursor was determined by the decoder.

5.2. METHODS 79

a b

Figure 5.2: Task layouts.a Grid task layout in a 25 target configuration. The acceptancewindows are adjacent and non-overlapping.b Radial task layout for 8 targets. In theradial task, there were regions that were left as gaps in the workspace to reduce inadvertentselection. In both tasks, the dotted lines that are not shownto the monkey outline theacceptance region for a particular target.

Channel codes are a means of transmitting information using multiple symbols selected in

a manner appropriate to the error rate so that despite symboltransmission errors, the prob-

ability of message loss is small. Although channel capacityis a very useful measure, in this

study, we chose instead to focus on achieved bitrate. Practically, in a clinical setting, no

subject can take advantage of sophisticated multi-symbol channel codes to transmit infor-

mation in real time as a computer would. The length of channelcodes would be limited to

single symbols, akin to a traditional keyboard. We thus limited the length of channel codes

used in this study to a single symbol as well. This created a problem for error detection,

for unless the subject was prompted for the symbol to transmit, there would be no way to

determine if a received symbol was correct. Since the monkeys’ participation in the task

required prompting of targets, we could determine empirically whether the received sym-

bol was correct. The only way to signal an incorrect transmission in a symmetric channel

is to transmit yet another symbol (e.g. the delete key on the keyboard). Thus, all incorrect

selections were required to be corrected by subsequent successful trials and were not incor-

porated into the final bitrate. The formula for calculating bitrate in the grid and radial tasks


is shown in the below equation.

B =log2(N)(Sc−Si)

t(5.1)

whereN is the total number of targets on the screen,Sc is the number of correct selections,

Si is the number of incorrect selections,t is the time elapsed. Inter-trial time and voluntary

breaks by the monkey were counted as part of the elapsed time.If the quantity(Sc−Si)< 0

then the bitrate is set to a floor at 0 bps, since bitrate cannotbe negative. Note that trials

where a selection was not made (i.e. the cursor was out of the workspace) were not counted

in Si since no symbol was transmitted, but were included in the elapsed time and slowed

overall bitrate. Equation5.1 is a conservative measure of evaluating bitrate, as a neural

prosthesis achieving 50% success rate would have an information rate of 0 bps. Despite

this harsh penalty for errors, this approach more accurately measures the clinical utility of a

neural prosthesis, as transmitting text involves similar challenges and penalties when used

in the real-world. Trials in which the subject made no selection and timed out were not

counted as incorrect selections as no symbol was transmitted, however the time spent was

included in the elapsed time for that block.

In both tasks, each trial prompted a single target out of manypotential choices. There-

fore, each selection conveyed information equal to the binary logarithm of the number of

targets on the screen. Viewing the neural prosthesis as a communication channel, the infor-

mation transmitted over timea function of several task parametersrepresents the throughput

of the system. With the aim of maximizing throughput, the task parameters must be set

appropriately. A trial can convey more information by increasing the number of targets,

but this raises the tasks difficulty and lowers the success rate. Thus, the number of targets

was an adjustable parameter in both tasks. In addition, eachtask had a second, unique,

adjustable parameter. In the case of the grid task, this parameter was the hold time required

to select a target. A long hold time would slow the overall rate of target selection whereas a

short hold time would lead to more inadvertent selections while navigating to the prompted

target. In the radial task, the distance to the targets was the adjusted parameter: placing

targets far from the center would slow the selection rate, while placing them close to the

center would increase the error rate. For each task, we foundthe optimal task parameters

5.3. RESULTS 81

that yielded high information throughput, as measured in bits per second (bps), striking the

best balance between success and selection rates.

5.3 Results

As each subject may have had differing cursor control ability, this two-dimensional param-

eter optimization was evaluated empirically for both monkeys since it could not be easily

determined by offline analysis. On a given day for a specific task, different pairs of task

parameters were tested in randomly ordered blocks of continuous and uninterrupted trials.

Each block was approximately 200 trials except when the taskwas too difficult and ter-

minated early. The randomization was repeated for several days to account for possible

ordering biases. Example histograms of the time to acquire the target for a pair of task

parameters for the grid task are shown in Figs5.3a/5.3d and likewise for the radial task in

Figs5.4a/5.4d.

Adjusting the task parameters significantly impacted the success and selection ratesand

thus the resulting information throughput. Figs5.3b/5.3e and Figs5.4b/5.4e show bitrate

heat maps of the optimization sweep for both monkeys on the grid task and the radial task,

respectively. Each point represents a pair of task parameters. For both monkeys across

both tasks the average throughput using the optimal task parameters was at least 3 bps.

Both monkeys had similar optimal parameters of 25 targets and450 ms hold time for the

grid task. For the radial task, the optimal number of targetswas 8 for both monkeys, but the

optimal target distance differed (Monkey J: 9 cm, Monkey L: 7cm). This slight difference

for the radial task represented a decrease of about 1 bps fromeach monkeys maximum,

highlighting the importance of sweeping task parameters tofind high throughput conditions

for each subject. Additionally, since bitrate falls steeply with increasing error rate, these

optimal parameters corresponded to high success rate conditions. These tended to be less

frustrating for the subject, facilitating longer sessionsdue to ease of use.


0 1 20

5

10

2+

Target Acquire Time (s)

Per

cent

of T

rials

0 9 25 49 64

250

450

650

850

Number of Targets

Hol

d T

ime

(ms)

0 30 60 90 120

Time (min)

0

1

2

3

4

Info

rmat

ion

Rat

e (b

its/s

ec)

0

10

20

Tot

al In

form

atio

n (k

bits

)

0 1 20

5

10

2+


Per

cent

of T

rials

0 9 25 49 64

250

450

650

850

Number of Targets

Hol

d T

ime

(ms)

0 30 60 90 120

Time (min)

0

1

2

3

4

Info

rmat

ion

Rat

e (b

its/s

ec)

0

10

20

Tot

al In

form

atio

n (k

bits

)

Monkey J

a b c

Monkey L

d e f

0 3bps

Figure 5.3: Experimental data of grid task. Monkey J: a-c. Monkey L: d-f. a Acquire timehistogram of grid task with task parameters of 25 targets and450 ms hold time.b Bitrateheatmap of task parameters swept for grid task. Each point inblack represents a tested pairof task parameters.c Information plot of sustained performance across a day for grid task.The gray line plots the average bitrate as a function of time and the black line representsthe cumulative information transmitted over the duration of the experiment.d Acquire timehistogram of grid task with 25 targets and 450 ms hold time.e Bitrate heatmap as in b.fInformation plot as in c.

5.3. RESULTS 83

0 1 20

5

10

15

2+


Per

cent

of T

rials

0 2 4 8 12 16

5

7

9

11

13

Number of Targets

Dis

tanc

e (c

m)

0 30 60 90 120

Time (min)

0

1

2

3

4

Info

rmat

ion

Rat

e (b

its/s

ec)

0

10

20

Tot

al In

form

atio

n (k

bits

)

0 1 20

5

10

15

2+


Per

cent

of T

rials

0 2 4 8 12 16

5

7

9

11

Number of Targets

Dis

tanc

e (c

m)

0 30 60 90 120

Time (min)

0

1

2

3

4

Info

rmat

ion

Rat

e (b

its/s

ec)

0

10

20

Tot

al In

form

atio

n (k

bits

)

Monkey J

a b c

Monkey L

d e f0 3bps

Figure 5.4: Experimental data of radial task. Monkey J: a-c.Monkey L: d-f. a Acquiretime histogram of radial task with task parameters of 8 targets and 9 cm distance.b Bitrateheatmap of task parameters swept for radial task. Each pointin black represents a testedpair of task parameters.c Information plot of sustained performance across a day forradial task. The gray line plots the average bitrate as a function of time and the blackline represents the cumulative information transmitted over the duration of the experiment.d Acquire time histogram of radial task with 8 targets and 7 cm distance.eBitrate heatmapas in b.f Information plot as in c.


Table 5.1: Acceptance region size as a function of targets onthe screen for the grid task

# targets Region (cm)9 825 4.836 449 3.42864 3

300 400 500 600 700 800 9000

0.5

1

1.5

2

2.5

3

3.5

4varying target number

hold time (ms)

stre

amB

itrat

e

9 targets25 targets49 targets64 targets

300 400 500 600 700 800 900-0.5

0

0.5

1

1.5

2

2.5

3

3.5varying target number

hold time (ms)

stre

amB

itrat

e

9 targets25 targets49 targets64 targets

a b

Figure 5.5: Bitrate plots for the grid task when the number of targets was varied.a MonkeyJ b Monkey L

5.3.1 Grid Task

The workspace for the grid task (Fig5.2a) was approximately a 20 cm square. The accep-

tance regions were set such that there were no gaps between targets. The more targets on

the screen, the smaller the acceptance region per target. The table below lists the acceptance

window size for varying number of targets.

Figs 5.5 and5.6 are the 2D traces of the bitrate data shown in the bitrate heatmap in

Figs5.3b/5.3e. In Fig5.5 the number of targets was varied, and in Fig5.6, the hold time

was varied. The error bars are bootstrapped standard deviations.

The data for the grid task consisted of four full randomized blocks of trials of all task pa-

rameter pairs, totalling 20716 trials for Monkey J. Blocks 1-3 were from datasets J100913-

J100917 and block 4 is from datasets J100928-J100930. For Monkey L, 17677 trials were

collected. Blocks 1-2 are from datsets L100913-L100917, andblocks 3-4 from L100920,

L100929, and L100930.

5.3. RESULTS 85

10 20 30 40 50 60 700

0.5

1

1.5

2

2.5

3

3.5

4varying hold times

number of Targets

stre

amB

itrat

e

250mm450mm650mm850mm

10 20 30 40 50 60 70-0.5

0

0.5

1

1.5

2

2.5

3

3.5varying hold times

number of Targets

stre

amB

itrat

e

250mm450mm650mm850mm

a b

Figure 5.6: Bitrate plots for the grid task when the hold time was varied.a Monkey JbMonkey L

50 60 70 80 90 100 110 120 130-0.5

0

0.5

1

1.5

2

2.5

3


radial Distance (mm)

stre

amB

itrat

e

2 targets4 targets8 targets12 targets16 targets

50 60 70 80 90 100 110-1

-0.5

0

0.5

1

1.5

2

2.5

3


radial Distance (mm)

stre

amB

itrat

e

2 targets4 targets8 targets12 targets16 targets

a b

Figure 5.7: Bitrate plots for the radial task when the number of targets was varied.aMonkey Jb Monkey L

5.3.2 Radial Task

In the radial task, there were fixed gaps in the acquisition region as depicted in Fig5.2b.

These gaps reduced the error rate of the task by providing space where no selections would

be made.

Figs5.7and5.8are the 2D traces of the bitrate data shown in the bitrate heatmap in Fig

5.4b/5.4e. In Fig5.7plot the number of targets was varied, and in Fig5.8, the distance to

the targets was varied. The error bars are bootstrapped standard deviations.

The data for the radial task were collected from datasets J100831-J100903 and J100906

for Monkey J, totalling 16754 trials. For Monkey L, the bitrate sweep are from datasets

L100901-L100903 and L100906-100907, totalling 14684 trials.


2 4 6 8 10 12 14 16 18 20-0.5

0

0.5

1

1.5

2

2.5

3

3.5varying radius

number of Targets

stre

amB

itrat

e

50mm70mm90mm110mm130mm

2 4 6 8 10 12 14 16 18 20-1

-0.5

0

0.5

1

1.5

2

2.5

3

3.5varying radius

number of Targets

stre

amB

itrat

e

50mm70mm90mm110mm

a b

Figure 5.8: Bitrate plots for the radial task when the distance to the targets was varied.aMonkey Jb Monkey L

5.3.3 Sustained intraday performance

Once the optimal parameters were found, we then evaluated the sustained performance of

the neural prosthesis by testing continuous, uninterrupted sessions. These sessions were

started at the beginning of an experimental day and run cuntil the subject was satiated. Figs

5.3c/5.3f plot the performance of the grid task for both monkeys and Figures 3c/3f plot the

session performance of the optimal parameters for each tasktype. Note that performance

is locally constant, at or above 3 bps, but trends downward due to decreasing motivation.

The monkeys accumulated more reward over the course of the experimental session until

they lost interest in the task and abruptly stopped.

5.4 Discussion

The experiments with these two keyboard-like tasks demonstrate that the ReFIT-KF de-

coder can be used to communicate information at higher ratesthan previously reported

with any neural prosthesis signal source. Rates of 3-4 bps were achieved, but were only

attainable after the task parameters were properly tuned for a specific subject’s level of

control. This suggests that an important aspect in the clinical deployment of neural pros-

theses is the proper optimization of each patient’s task to maximize their performance. It is

assumed, as in the case with the monkeys, that an easier to usetask leads to less frustration

and greater willingness to use the system.

Chapter 6

Sustained performance & typing

This chapter is unpublished. The data was collected with theassistance of Jonathan Kao,

Sergey Stavisky, and Joline Fan.

6.1 Abstract

Neural prostheses translate neural activity into control signals for guiding assistive devices

such as robotic arms and computer cursors, and strive to improve quality of life by pro-

viding greater interaction with the world for people with movement disabilities. Despite

encouraging proof-of-concept demonstrations, [12,32,40,57,58,88,94,102,124,130] sig-

nificant barriers to clinical translation remain. Several reports have described limitations to

adoption of intracortical neural prostheses, highlighting issues of performance [44,52,98]

and robustness [13, 63, 84] as well as the need for daily retraining. [52] Recent studies

from an ongoing clinical trial have shown that intracortical electrodes are viable for at least

five years, although signal quality, and resulting neural prosthesis performance, degrades

over time. [57,58,117] Here we present the results from monkeys using two and four year

old multielectrode arrays. We demonstrate sustained high performance with communica-

tion rates up to 4.5 bits per second during hours-long experiments. This was demonstrated

across weeks, and up to a year in one monkey, without retraining using a keyboard-like,

continuously controlled cursor task with the ReFIT algorithm. [48] This is the first demon-

stration of sustained online control for a year. Additionally, when used to type text, we

87

88 CHAPTER 6. SUSTAINED PERFORMANCE & TYPING

show information throughput exceeding 10 words per minute,the highest reported to date.

Finally, we show that the reason this neural prosthesis can transmit meaningful informa-

tion for many months without retraining is due to the relative stability of the neural signals

recorded and the consistency of the animal’s control strategy with the decoder. These

results suggest that long-term chronically implanted neural prostheses may help disabled

individuals successfully communicate text for hours a day,and do so for weeks and months,

without retraining.

6.2 Introduction

Robust performance is an important factor of neural prostheses and stems from having

stable neural signals. Prior studies have demonstrated thestability of single units over

time [17, 27, 123] and reports of stable decoding both offline [4, 18, 110] and online [38]

have been published, in addition to recent preliminary findings. [33, 93] However, there

have been no reports of robust online control consistent with clinically viable performance.

Robustness is important on at least two timescales: within a day (intraday) and across days

(interday). Neural prosthesis performance should be sustainable on real-world tasks for

hours at a time during a day and minimize or eliminate training time across days. This is

challenging because the number of single units recorded from intracortical arrays tend to

decrease over time. [57,117]

6.3 Results

To test our neural prosthesis across various ages of array conditions, we selected two rhe-

sus macaques implanted with multielectrode arrays two years (Monkey J) and four years

(Monkey L) prior to experiments. It was important to use aging multielectrode arrays in an

attempt to emulate the clinical scenario of chronically implanted patients because there is

currently no known method of predicting, or accelerating, neural signal decline. Monkey

J’s two arrays in M1 and PMd had many arm-movement modulated channels with substan-

tial tuning in all directions of the 2D task. Monkey L’s single array in M1/PMd had fewer

6.3. RESULTS 89

0

4In

form

atio

n R

ate

(b

its/s

ec)

0

500

To

tal In

form

atio

n (

kb

its)

1PM

1

1PM

2

12PM

3 4

1PM

5 6

12PM

7

12PM

8

12PM

9

1PM

10 11

2PM

12

2PM

13

7PM

14Day

0

4

Info

rma

tio

n R

ate

(b

its/s

ec)

0

300

To

tal In

form

atio

n (

kb

its)

4PM

1

3PM

2

3PM

3

2PM

4

2PM

5

3PM

6

3PM

7

4PM

8

3PM

9

3PM

10

3PM

11

2PM

12

6PM

13

7PM

14Day

a

b

Figure 6.1: Decoding without retraining.a Daily performance plot of 14 day long exper-iment with Monkey J. The gray line plots the instantaneous, smoothed information rateacross time. The black line plots the cumulative information transmitted across the experi-ment. The x-axis indicates the time the experiment was run, each tick mark notes an hourin the day. b Similar performance plot for Monkey L. Plots are from datasets J110926-J111014 & L110912-L110925.

arm-movement modulated channels and may mimic a patient with deteriorating neural sig-

nal quality. We first confirmed that the neural prosthesis wasstable within a day for a week

with decoder retraining but without additional hand-controlled sessions. We then evaluated

stability without decoder retraining sessions. Fig6.1 plots the results of this 14 day ex-

periment with both monkeys performing a keyboard-like neural prosthesis task. Decoders

were built using threshold crossings from all available channels under the ReFIT-KF [48]

algorithm. Using this task, an information rate was calculated based on the rate of target

selection and net successful trials. By conservative information-transmission measures,

Monkey J achieved an average rate of 3.4 bits per second (bps)and Monkey L averaged

2.6 bps over the course of the two consecutive weeks. For several minutes during the ex-

periments, Monkey J achieved up to 4.5 bps and Monkey L up to 4 bps. This performance

represents the highest achieved and sustained bitrate witha neural prosthesis.

Throughout this two week period, no additional training time was required other than


the initial decoder training on the first day. Monkey J’s signal quality was sufficiently

high that a single decoder was held constant daily for over a month during days 2-32.

This is the first report of daily, sustained, high performance without any intervention for a

month. Because of his poorer signal quality, a single decodercould not be held constant

for Monkey L, but instead he required seven different decoders over the two week period.

However, none of the six subsequently built decoders required any subject training time.

Additional functional decoders were automatically generated using the last hour of data of

each experimental day and were available if necessary, resulting in a growing collection

of available decoders. On a given experimental day, one of these, usually the one used

successfully the previous day, was selected within the first5 minutes of the experiment

and held for the duration of that day. A clinical system mightfunction in a similar way,

where a number of decoders could be quickly evaluated beforesettling on one for use in

that session. The results from Monkey J demonstrate that a single decoder can sustain

performance for at least a month, and the results from MonkeyL demonstrate that despite

diminished signal quality, high performance can be sustained for weeks without retraining

sessions.

Although bitrate is an accurate measure of performance, it can be difficult to appreci-

ate its meaning without a more practical context. For communication neural prostheses, a

nominally clinically relevant measure is typing speed in words per minute (wpm) or char-

acters per minute (cpm). To evaluate typing speed, we modified the grid task to simulate

the transmission of words and sentences. The monkeys were tasked to select targets in a

specific order that corresponded to the spelling of words as if the grid were a keyboard. The

task was interactive such that incorrect character selections required the subsequent selec-

tion of the delete key before resuming the text. In this manner, an approximation of the

real-world performance of the system could be measured. Monkey J achieved a peak speed

of 13 wpm and averaged 50 cpm over three hours in a single session (Fig. 6.2a & 6.2b),

completing an 8000 character passage during that time. On the same passage, Monkey

L achieved a peak speed of 10 wpm and averaged 36 cpm (Figs6.2c & 6.2d), completing

two-thirds of the passage in about two hours. This is the highest sustained neural prosthesis

performance on a real-world task. The monkeys are not cognizant of the words they are

constructing; instead, success here is the simulation of the pattern of key selection and task

6.3. RESULTS 91

0 60 120 180Time (min)

0

20

40

60

Typ

ing

ra

te(C

ha

racte

rs p

er

min

)0 5 10

0

10

20

Words per min

Pe

rce

nt o

f T

ria

ls

0 60 120 180Time (min)

0

20

40

60

Typ

ing

ra

te(C

ha

racte

rs p

er

min

)

0 5 100

10

20

Words per min

Pe

rce

nt o

f T

ria

ls

a b

c d

Figure 6.2: Typing task.a Plot of instantaneous, smoothed typing rate across time forMonkey J while typing an article. Gaps in trace represent times where the monkey pausedand was not typing.b Histogram of words per minute for Monkey J, same data as in a.c &d Same as in subpanels a & b, but for Monkey L. Typing data from from datasets J111021& L111103.

difficulty which a human subject would face. Moreover, the extent to which the cognitive

load of concurrent sentence formation and letter selectionwould affect performance cannot

be measured here. By combining the findings from Figs6.1& 6.2, on day 32 of the month-

long experiment, Monkey J achieved 10 wpm on a typing task using the same decoder held

constant in Fig6.1a. These findings together demonstrate that a neural prosthesis can be

both robust over weeks and convey meaningful information ata high rate.

These results encouraged us to explore the stability of decoders over even longer peri-

ods of time. Investigating the limits to sustained, robust performance may have implications

regarding the long-term utility of neural prostheses. Since the daily experimental testing

shown in Fig6.1were untenable for months on end, we evaluated long-term robustness in

two fashions: a longitudinal study and a historical cross-sectional analysis. First, in the

longitudinal study, performance of the same decoder used with Monkey J in Fig6.1a was

evaluated over a period of a year, shown in Fig6.3a. This study, performed exclusively in


Monkey J since only he could sustain high performance of a single decoder for multiple

months, is the first reported demonstration of high performance using a static decoder sus-

tained for a year. The regressed line for this model finds thatperformance declines at a rate

of half a bps over the course of a year, suggesting that the model may be stable across time.

Second, in the historical cross-sectional analysis, the performance of several decoders cre-

ated between 3-14 months prior were evaluated over contiguous experimental days. This

was done with both monkeys, summarized by Figures6.3b & 6.3c. The results of the

historical cross-sectional analysis support the findings of the longitudinal study, demon-

strating that performance can be sustained over several months but trends down as the age

of the model grows. Performance also becomes more subject todaily changes in neural

tuning, as not every prior decoder or experimental day worksconsistently across a year.

This experiment also suggests, based on the relative steepness of decline of each monkey’s

performance, that older arrays are less robust across months than newer ones, mirroring

the findings of the distribution of relative channel weightsbetween monkeys. Other factors

besides array age may contribute to signal quality such as surgical conditions, subject age,

health, immune status, and activity level; but these cannotbe easily evaluated due to the

limited number of subjects studied. Additionally, the use of two arrays in Monkey J may

have kept his signal quality high across time relative to Monkey L. Taken together, the lon-

gitudinal study and the historical cross-sectional analysis demonstrate for the first time that

a single decoder can maintain usable performance for several months up to a year.

6.4 Discussion

How is such stability is possible, particularly when it has been assumed that decoder perfor-

mance was very short-lived and required daily retraining? [52] To investigate, we analyzed

the tuning directions of highly contributory channels in the decoder. Figs6.4a & 6.4b plot

the tuning curves of the top 5 electrode channels that originally made the greatest contri-

bution to the 6-month-old decoders in both monkeys. Each color represents the tuning of

that channel on various experimental days, with black linesplotting the tuning on the day

the decoder was built. Surprisingly, among the most contributory channels, neural tuning

6.4. DISCUSSION 93

0 4 8 12Time (months)

0

2

4In

form

atio

n r

ate

(b

its/s

ec)

0000 4444 6666 8888 10101010Time (months)

0

2

4

Info

rma

tio

n r

ate

(b

its/s

ec)

00000 33333 66666 1414141414Time (months)

0

2

4

Info

rma

tio

n r

ate

(b

its/s

ec)a b c

Figure 6.3: Historical models.a Longitudinal study showing performance of Monkey Jwith a static decoder over one year. The filled in circles denote performance from the firstmonth of experiments with a single decoder (Fig6.1a). That decoder, held constant, wastested over a year. The solid line was regressed against the performance across a year and isthe equation: Information rate=−0.034·(# months)+3.78bpsb Cross-sectional historicalstudy of decoder performance from previously recorded datain Monkey J. Each column ofpoints represents a single model tested. Each color/symbolrepresents an experimental day.The peformance at time 0 represents the performance of the decoder on that experimentalday. The black line shows the average performance for each model. c Same as in subpanelb, but for Monkey L. Panels b & c from datasets J120117-J120120, L120207-L120210, &L120214.

across six months is relatively stable. The tuning stability of the most contributory chan-

nels is the enabling mechanism by which the decoders shown inFigs6.1& 6.3maintained

their performance over extended periods of time. However, this is not to say there is no

change in tuning across arrays over time. As shown in Figs6.4c & 6.4d, there is a many-

fold difference between the per-channel, contribution-scaled, absolute-versus-net rotation

of tuning. Thus, although there may be shifts in tuning in channels over time, they appear to

be random and tend to largely cancel one another out for the purposes of two-dimensional

decoding. In this fashion, having many channels for the decoder to draw information from

is beneficial, so that the change in tuning of a single channelwould not significantly af-

fect performance. Further, the smaller net rotation observed in Monkey J may explain why

he, with two newer arrays, could sustain performance with a single decoder for an entire

month, while Monkey L, with a single much older array, required several decoders across

two weeks. In support of this idea, Fig6.4e plots the individual contributions of the top

30 channels to the decoder for both monkeys. Because Monkey L’s array has poorer signal

quality and fewer channels to draw from, the decoder is forced to assign greater weights to


the most informative channels in Monkey L, whereas no channel in Monkey J is assigned

greater than 3% weight. On average, Monkey L’s decoder draws50% of its information

from about a dozen channels, while Monkey J’s decoder distributes that same weight across

more than twice that many channels (around 30). These relationships are further detailed in

Fig 6.4f, a plot of cumulative contribution as a function of sorted channels which details a

steeper rise of contribution per channel for Monkey L vs Monkey J. It is thus less surprising

that Monkey L’s decoder is less stable because it is more sensitive to fluctuations in tun-

ing of a single important channel. As evidenced with Monkey J, the more tuned channels

available for the decoder to draw from, the more robust its performance.

The robustness of the neural prosthesis for extended periods of time and its high perfor-

mance with regard to both bitrate and real-world metrics like words per minute illustrate the

potential for this class of neural prosthesis to provide a functional communication system

for people with movement disabilities. Although prior studies [12,38] had demonstrated de-

coder stability for a few weeks while tracking hand-selected neurons, this is the first study

to demonstrate that high-performance can be sustained for up to year without interven-

tion or monitoring. Recent years have seen the development ofneural prosthesis systems

employing various interfaces including electroencephalography, electrocorticography, and

the intracortical arrays used here. The neural prosthesis research community continues to

explore these methodologies while assessing their relative risk and benefit. While intracor-

tical arrays have been questioned for their assumed short lifetime and limited performance

advantage, here we have demonstrated their sustained, robust, and high performance for

weeks without retraining using 2-4 year old multielectrodearrays. Constructing reliable

neural prosthesis systems that minimize training time while offering high performance on

real world tasks is crucial to successful translation. The results shown here may help bring

these systems closer to clinical viability.

6.5 Methods Summary

All procedures and experiments were approved by the Stanford University Institutional

Animal Care and Use Committee (IACUC). Experiments were conducted with adult male

6.5. METHODS SUMMARY 95

123 61 110 119 129

75 53 89 76 82

Sum Net0

0.5

Rad

ians

Sum Net0

0.5

Rad

ians

0 30Channel0

0.1

Indi

vidu

alC

ontr

ibut

ion

(%)

0 200Number of channels0

1

Cum

ulat

ive

Con

trib

utio

n (%

)

a

b

c d e f

Figure 6.4: Tuning Analysisa Tuning curves of top 5 contributory channels for a 6 month-old decoder for Monkey J. The number to the left of the axis denotes the channel. Eachcolored curve represents a different experimental day. Theblack curve represents the tuningon the day the decoder was generated (J110722). The straightlines represent the channel’scosine fit preferred direction.b Same tuning curves as in subpanel a, but for Monkey L.The original decoder was from L110805.c Bar graph of contribution-scaled, per-channelrotation in tuning for Monkey J. The first column plots the unsigned weighted summedrotation of all channels while the second plots the signed net rotation. d Same plot as insubpanel e, but for Monkey L.ePlot of individual contribution of top 30 channels for Mon-key J (red) and L (blue).f Plot of cumulative sorted per channel contribution to decoder.All plots derived from same datasets as in Figure6.3.


rhesus macaques (L and J), implanted with 96 electrode Utah arrays (Blackrock Microsys-

tems Inc., Salt Lake City, UT) using standard neurosurgical techniques. Electrode arrays

were implanted in the dorsal aspect of premotor cortex (PMd)and primary motor cortex

(M1), as estimated visually from local anatomical landmarks. Monkey L had a single array

implanted, whereas Monkey J had two arrays implanted, one adjacent to the other. Aside

from the intraday stability experiments which used only theM1 array, all data presented

for Monkey J used both arrays. The monkeys were trained to make point-to-point reaches

in a 2D plane with a virtual cursor controlled by the contralateral arm or by a neural de-

coder. The virtual cursor and targets were presented in a 3D environment (MSMS, MDDF,

USC, Los Angeles, CA). Hand position data were measured with aninfrared reflective bead

tracking system (Polaris, Northern Digital, Ontario, Canada). Spike counts were collected

by applying a single negative threshold, set to 4.5× root mean square of the spike band of

each neural channel. [18] Behavioral control and neural decode were run on separate PCs

using the Simulink/xPC platform (Mathworks, Natick, MA) with communication latencies

of 3 ms. This system enabled millisecond-timing precision for all computations. Neural

data were initially processed by the Cerebus recording system (Blackrock Microsystems

Inc., Salt Lake City, UT) and were available to the behavioralcontrol system within 5± 1

ms. Visual presentation was provided via two LCD monitors with refresh rates at 120 Hz,

yielding frame updates within 7± 4 ms. Two mirrors visually fused the displays into a

single 3D percept for the user, creating a Wheatstone stereograph. This animal model was

selected because we felt it most closely mimics the neural state of a patient that would be

employing a BMI in a clinical study. [92]

6.6 Acknowledgments

We thank Mackenzie Mazariegos, John Aguayo, Wendy Kalkus, Erica Morgan, and Clare

Sherman for veterinary care. We thank Sandra Eisensee, Evelyn Castaneda, and Beverly

Davis for administrative support. We thank Boris Oskotsky for information technology

support.

Chapter 7

Hidden Markov model ReFIT-KF

This work is yet unpublished. The systems presented here andthe data shown were created

and gathered in conjunction with Jonathan Kao.

7.1 Introduction

Several neural prostheses have relied solely on the movement of the cursor as the single

control modality. In these tasks, the cursor would move overthe desired target and dwell

over it for a prescribed hold period. Target selection is conveyed at the end of this hold

period. However, this hold time can be undesirable because the duration of hold time can

make up a significant portion of the trial length. This can be up to nearly 50% of the trial

length in some cases. If hold time could be eliminated, the overall throughput of the system

would increase, transmitting more information faster (bitrate) and enabling faster typing

speeds. One common approach to eliminating hold time is to signal target selection using

an alternative decoding scheme, and thus implementing a click signal. Prior reports have

implemented click signals including using naive Bayes (NB) [102] classifiers and linear

discriminant analysis (LDA) [70,117]. These modalities work reasonably well, but the false

positive rate of those decoders was high under certain circumstances. Further, decoding a

click signal is akin to decoding states, and both NB and LDA can decode multiple states,

but provide no structure between states. In both these decoders, state transitions can and

do occur instantly and can be from any one state to another. This can be undesirable when

97

98 CHAPTER 7. HIDDEN MARKOV MODEL REFIT-KF

deployed in the setting of a neural prosthesis because task states often do not switch rapidly

and arbitrarily.

For these reasons, this work utilized an alternative state decoder: a hidden Markov

model (HMM). The advantages of an HMM are that, unlike a NB or LDA, an HMM models

noise of the underlying observations. Modeling noise improves the overall performance

of the system. In addition, it provides flexibility in modeling the number of states and

their transition characteristics. Both of these features were utilized in developing an HMM

state decoder that was run in parallel to the ReFIT-KF. In thisfashion, the ReFIT-KF [49]

controlled the velocity of the cursor while the HMM controlled the state of the click. Using

these two decoders together improved the performance of system as measured by bitrate,

up to a peak rate of 6 bits per second (bps) and a typing rate of 15 words per minute (wpm).

These findings demonstrate that the performance of cursor-controlled neural prosthetics

can be improved by running an HMM state decoder in parallel tothe cursor movement

decoder.

7.2 Decoder Design

All experimental methods were carried out as per Chapter4. The prosthesis’s function was

broken up into two decoders run in parallel. The movement of the cursor was controlled by

the ReFIT-KF as before, however selections were no longer communicated by holding over

the desired target. Instead, the HMM state decoder was run inparallel with the ReFIT-KF,

and handled the task of target selection. As implemented in this study, the HMM’s obser-

vations of the neural data were assumed to follow a multivariate Gaussian distribution, just

as the Kalman filter does. These multivariate Gaussian observations informed the updating

of the hidden state (click vs no click) via an emissions matrix. The hidden state itself was

also given dynamics and had a static transition matrix that was trained off kinematic data.

Tweaking the transition matrix can allow for shaping of the transition states, if desired.

For Monkey J, a two state HMM performed sufficiently well thatthe simple model shown

in Fig 7.2 performed well. However, for Monkey L, this simple model didnot perform

well. The two state model used with Monkey J resulted in too many false positive errors

in Monkey L and would additionally click when Monkey L was taking volitional breaks

7.3. RESULTS 99

move stop

Figure 7.1:Two state transition graph. Graphical model of the transition flow for the twostate HMM used with Monkey J. The system starts in the move state and can stay in thatstate at the next timestep. It can likewise transition to thestop state if there is sufficientinformation from the neural emissions matrix to push it to that state. If the system stays inthe stop state for two timesteps, the HMM notes a click and immediately transitions backto the move state. The state is locked in the move state for a lockout time (around 250 ms),before it is allowed to transition out of the move state again.

from the task. To address these issues, two changes were made. The first was the inclusion

of an idle state, which was trained against neural data collected while the monkey was not

engaged in the task. The second was the break up of the move state into three kinematic

states which were transitioned between based on the magnitude of the velocity of the cursor

in the ReFIT-KF. These two changes together resulted in a system that was controllable for

Monkey L and shown in Fig7.2.

7.3 Results

The HMM was tested in two fashions. The first was by measuring bitrate on a grid task as

in Chapters5 and6. The results of these experiments are shown in Fig7.3 for Monkey J

and Fig7.3 for Monkey L. Note that Monkey J can achieve a maximum bitrateof 6.5 bps

for minutes at a time.

Second, the performance of the system was also evaluated themore real-world typing

task presented in Chapter6. Here again, Monkey J achieved a peak typing rate of 15 wpm.


idle

slow fast slow stop

Figure 7.2:Four state transition graph. Graphical model of the transition flow for thefour state HMM used with Monkey L. The system starts in the slow state. The system canstay in any state (except stop) for an arbitrary number of timesteps and can transition to theidle state from any state. Transitions from the movement based states (slow 1, fast, slow2) are only allowed to proceed in a stepwise fashion. From theslow state, if the kinematicprofile (magnitude velocity) is above the specified threshold, the system transitions to thefast state. Once the kinematic profile falls below the specified threshold, the state transitionsto the second slow state. The system can transition to the stop state Only from the secondslow state. Upon transition to the stop state, the system communicates a click signal andimmediately returns to the first slow state. As in the two state HMM, there is a lockout timeenforced before a second stop state can be entered.

7.3. RESULTS 101

0 30 60Time (min)

0

1

2

3

4

5

6

7

Info

rmat

ion

Rat

e (b

its/s

ec)

0 1 2 3 4 5 6 70

5

10

Bitrate (bps)P

erce

nt o

f Tim

e

Figure 7.3:Bitrate for Monkey J with HMM ReFIT-KF decoder. Plot of bitrate overtime for two different days(a) and aggregate bitrate histogram(b) for the two state HMMReFIT-KF decoder with Monkey J on a randomized target grid task. The task was a 7x7grid with 49 targets and was taken from datasets J120405 dn J120406.

0 30 60 90 120 150

Time (min)

0

1

2

3

4

Info

rmat

ion

Rat

e (b

its/s

ec)

0 1 2 3 4 5 6 70

5

10

Bitrate (bps)

Per

cent

of T

ime

Figure 7.4:Bitrate for Monkey L with HMM ReFIT-KF decoder. Plot of bitrate overtime for three different days(a) and aggregate bitrate histogram(b) for the four state HMMReFIT-KF decoder with Monkey L on a randomized target grid task. The task was a 5x5grid with 25 targets and was taken from datasets L120426 L120427 and L120430.


0 30 60 90 120Time (min)

0

20

40

60

80

Typ

ing

rate

(Cha

ract

ers

per

min

)

0 5 10 150

10

20

30

Words per min

Per

cent

of T

ime

Figure 7.5:Typing rate for Monkey J with HMM ReFIT-KF decoder. Plot of typing rateover time(a) and typing rate histogram(b) for the two state HMM ReFIT-KF decoder withMonkey J. The task was a 6x6 grid with 36 targets and was taken from datasets J120409.

0 30 60 90 120Time (min)

0

20

40

60

80

Typ

ing

rate

(Cha

ract

ers

per

min

)

0 5 10 150

10

20

30

Words per min

Per

cent

of T

ime

Figure 7.6:Typing rate for Monkey L with HMM ReFIT-KF decoder. Plot of typingrate over time(a) and typing rate histogram(b) for the four state HMM ReFIT-KF decoderwith Monkey L. The task was a 6x6 grid with 36 targets and was taken from datasetsL120501.

7.4. CONCLUSION 103

7.4 Conclusion

The results from the HMM based decoder show that using a clicksignal can increase the

performance of the system. The combination of the ReFIT-KF controlling the cursor and

the HMM decoding state was effective in distributing roles to the various decoders despite

one input signal. This suggests that additional information may be encoded in the regions

of cortex being recorded from, and that an appropriate decoder can extract this information.

Further, the success of Monkey L’s mixed kinematic-based HMM suggests that decoders

can be used as signal sources for each other, yielding a layered decoding system that ex-

tract information from multiple levels. Together, these findings should provide a higher

performing and more flexible framework for real-time decoding, helping to increase the

clinical viability of neural prostheses.

Chapter 8

Conclusion

The work presented in this dissertation constitutes a systematic development of neural pros-

thetics. The theoretical underpinnings and applicabilityof the animal model are detailed

in the first chapter, setting the stage for understanding howearly developmental work oc-

curs before testing in human subjects. Such a foundation is necessary to demonstrate why

animal work is believed to hold relevance to future human work. The advances made us-

ing animal models is anticipated to also contribute similarly when used in patients. Thus,

using this animal model, an analysis of how online findings further motivates the need

for experiments of this class. This was the subject of the second chapter. From this, it

is understood that online, closed-loop animal experimentation advances are important be-

fore exposing humans to the risks associated with experimentation. Additionally, the same

chapter discussed how the results from offline testing cannot be trusted as they can yield

measurably incorrect findings. From here, the animal model and online, closed-loop per-

spective were used to develop an algorithm which significantly improved the performance

of a two-dimensional cursor under neural prosthetic control. This new algorithm, ReFIT-

KF, was only possible by inspecting and accounting for the system as a whole, subject

and decoder, and adjusting the decoder with an understanding of how the components of

the system interacted. The ReFIT-KF would not be possible without the attention to the

online, closed-loop nature of decoding, and the results area consequence of the perspec-

tive taken as laid out in the foundational work in the prior chapters. Subsequent chapters

then discussed the optimization and applicability of the ReFIT-KF under real-world tasks,

104

105

with an eye for maximizing meaningful and clinically relevant performance. The issue

of robustness was also addressed, demonstrating decoder stability for weeks and up to a

year. Finally, a hybrid decoder was presented using a combined system with the ReFIT-

KF controlling the cursor velocity and an HMM managing the click state. Taken together,

this development represents the progression from concept and theory to implementation,

advancement, evaluation, and refinement.

These results, however, have been limited exclusively to animal studies and need fur-

ther validation. The true test of these algorithms and advancements lies in the clinical

translation when tested in human subjects. Armed with thesefindings, the performance

and robustness of the neural prosthesis developed here appears high enough that the po-

tential benefit to patients may justify the risk of experimentation. The Shenoy lab fully

believes this and has joined part in the BrainGate clinical trial to take that next step. We

are optimistic that the same advances presented in this workwill carry over to the clinical

realm, where the need lies. At the same time, we recognize thechallenges in translation

and aim to iterate between the human and animal models rapidly. Just as Chapter3 found

that offline findings are a poor predictor of online results, so too will there be instances

where the animal model predictions are not in perfect alignment with the observations in

the clinic. However, keeping the animal model on hand as an experimental site for di-

agnosing, evaluating, and correcting the challenges facedduring translation will likely be

of great benefit in addressing the issues faced. It is an open question whether the human

subjects can achieve comparable performance to that demonstrated by the animal subjects.

Although the initial clinical tests will involve two-dimensional cursor control, additional

end effectors such as robotic arms can be slotted in for evaluation. The aim is to leverage

the engineering foundation and advances presented in this dissertation to the clinic, hope-

fully making neural prostheses more viable and improving the quality of life of the millions

of patients living with paralysis.

Appendix A

Publications

The following pages are listed all peer-reviewed publications published during graduate

training.

106

Journal papers

[1] CA Chestek, V Gilja, P Nuyujukian, JD Foster, JM Fan, MT Kaufman, MM Church-

land, Z Rivera-Alvidrez, JP Cunningham, SI Ryu, and KV Shenoy. Long-term stability

of neural prosthetic control signals from silicon corticalarrays in rhesus macaque mo-

tor cortex.Journal of Neural Engineering, 8(4):045005, Aug 2011.

[2] CA Chestek, V Gilja, P Nuyujukian, RJ Kier, F Solzbacher, SI Ryu, RR Harrison, and

KV Shenoy. Hermesc: low-power wireless neural recording system for freely mov-

ing primates.IEEE Transactions on Neural Systems and Rehabilitation Engineering,

17(4):330–338, Aug 2009.

[3] MM Churchland, JP Cunningham, MT Kaufman, JD Foster, P Nuyujukian, SI Ryu,

and KV Shenoy. Neural population dynamics during reaching.Nature, 487:51–56,

Feb 2012.

[4] JP Cunningham, P Nuyujukian, V Gilja, CA Chestek, SI Ryu, and KV Shenoy. A

closed-loop human simulator for investigating the role of feedback-control in brain-

machine interfaces.Journal of Neurophysiolgy, 105:1932–1949, Oct 2010.

[5] S Farshchi, P Nuyujukian, A Pesterev, I Mody, and JW Judy.A tinyos-enabled

mica2-based wireless neural interface.IEEE Transactions on Biomedical Engineer-

ing, 53(7):1416–1424, Jul 2006.

[6] S Farshchi, A Pesterev, P Nuyujukian, E Guenterberg, I Mody, and JW Judy. Em-

bedded neural recording with tinyos-based wireless-enabled processor modules.IEEE

Transactions on Neural Systems and Rehabilitation Engineering, 18(2):134–141, Apr

2010.

107

108 JOURNAL PAPERS

[7] S Farshchi, A Pesterev, P Nuyujukian, I Mody, and JW Judy.Bi-fi: an embedded

sensor/system architecture for remote biological monitoring. IEEE Transactions on

Information Technology in Biomedicine, 11(6):611–618, Nov 2007.

[8] H Gao, RM Walker, P Nuyujukian, KAA Makinwa, KV Shenoy, B Murmann, and

TH Meng. Hermese: A 96-channel full data rate direct neural interface in 0.13 um

cmos.IEEE Journal of Solid-State Circuits, PP(99):1–13, Feb 2012.

[9] V Gilja, CA Chestek, P Nuyujukian, JD Foster, and KV Shenoy.Autonomous head-

mounted electrophysiology systems for freely behaving primates. Current Opinion

Neurobiology, 20(5):676–686, Oct 2010.

[10] V Gilja, P Nuyujukian, C Chestek, JP Cunningham, BM Yu, JM Fan, JC Kao, SI Ryu,

and KV Shenoy. A high-performance neural prosthesis enabled by control algorithm

design.Nature Neuroscience, 15:1752–1757, 2012.

[11] RR Harrison, RJ Kier, CA Chestek, V Gilja, P Nuyujukian, SI Ryu, B Greger,

F Solzbacher, and KV Shenoy. Wireless neural recording withsingle low-power inte-

grated circuit.IEEE Transactions on Neural Systems and Rehabilitation Engineering,

17(4):322–329, Aug 2009.

[12] D Sussillo, P Nuyujukian, JM Fan, JC Kao, SD Stavisky, SIRyu, and KV Shenoy.

A recurrent neural network for closed-loop intracortical brain-machine interface de-

coders.Journal of Neural Engineering, 9(2):026027, Mar 2012.

Conference papers

[1] W Bishop, CA Chestek, V Gilja, P Nuyujukian, SI Ryu, KV Shenoy,and B Yu. Long-

term decoding stability without retraining for intracortical brain computer interface,

Feb 2012.

[2] CA Chestek, JP Cunningham, V Gilja, P Nuyujukian, SI Ryu, and KV Shenoy. Neural

prosthetic systems: current problems and future directions. In IEEE EMBS, pages

3369–3375, Sep 2009.

[3] CA Chestek, V Gilja, P Nuyujukian, SI Ryu, RJ Kier, F Solzbacher, RR Harrison,

and KV Shenoy. Hermesc: Rf wireless low-power neural recording system for freely

behaving primates. InProc. of IEEE ISCAS, pages 1752–1755, May 2008.

[4] J Dethier, V Gilja, P Nuyujukian, S Elassaad, KV Shenoy, and K Boahen. Spiking

neural network decoder for brain-machine interfaces. InProc. of the 5th International

IEEE EMBS Conference on Neural Engineering, pages 396–399, Jun 2011.

[5] J Dethier, P Nuyujukian, C Eliasmith, TC Stewart, SA Elasaad, KV Shenoy, and

KA Boahen. A brain-machine interface operating with a real-time spiking neural

network control algorithm. In J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F.C.N. Pereira,

and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems

24, pages 2213–2221. Dec 2011.

[6] S Farshchi, P Nuyujukian, A Pesterev, I Mody, and JW Judy.A tinyos-based wireless

neural sensing, archiving, and hosting system. InProc. of 2nd IEEE EMBS Neural

Engineering Conference, pages 671–674, Mar 2005.

109

110 CONFERENCE PAPERS

[7] JD Foster, O Freifeld, P Nuyujukian, SI Ryu, MJ Black, and KVShenoy. Combining

wireless neural recording and video capture for the analysis of natural gait. InProc.

of the 5th International IEEE EMBS Conference on Neural Engineering, pages 613–

616, Jun 2011.

[8] JD Foster, P Nuyujukian, O Friefeld, SI Ryu, MJ Black, and KVShenoy. A frame-

work for relating neural activity to freely moving behavior, 2012.

[9] V Gilja, P Nuyujukian, CA Chestek, JP Cunningham, BM Yu, JM Fan, SI Ryu, and

KV Shenoy. A brain machine interface control algorithm designed from a feedback

control perspective, 2012. in press.

[10] V Gilja, P Nuyujukian, CA Chestek, JP Cunningham, BM Yu, SI Ryu, and

KV Shenoy. High-performance continuous neural cursor control enabled by a feed-

back control perspective. InComputational and Systems Neuroscience, Feb 2010.

[11] RR Harrison, RJ Kier, B Greger, F Solzbacher, CA Chestek, V Gilja, P Nuyujukian,

SI Ryu, and KV Shenoy. Wireless neural signal acquisition with single low-power

integrated circuit. InProc. of IEEE ISCAS, pages 1748–1751, May 2008.

[12] P Nuyujukian, Fan JM, V Gilja, PS Kalanithi, CA Chestek, and KV Shenoy. Monkey

models for brain-machine interfaces: the need for maintaining diversity. InProc. of

the 33rd Annual IEEE EMBS Conference, Aug 2011.

[13] P Nuyujukian, JC Kao, Fan JM, SD Stavisky, SI Ryu, and KV Shenoy. A high-

performance, robust brain-machine interface without retraining, Feb 2012.

[14] RM Walker, H Gao, P Nuyujukian, K Mackinwa, KV Shenoy, TH Meng, and B Mur-

mann. A 96-channel full data rate direct neural interface in0.13um cmos. InProc. of

the 2011 Symposium on VLSI Circuits (VLSIC), Jun 2011.

Appendix B

Patents

Two patent filings were made during graduate training. The first is patent US 2011/0224572

A1, application number 12/932,070, filed on Feb 17, 2011, titled “Brain machine interface”

based on the ReFIT-KF decoder presented in Chapter4. The second is a provisional patent

application number 61/701974, filed 9/17/2012, titled “A Brain Machine Interface Utilizing

a Hidden Markov Model State Detector” based on the ReFIT-KF and HMM combination

decoder presented in Chapter7.

111

Bibliography

[1] P. Adler and T. Winograd.Usability: Turning Technologies into Tools. Oxford Press,

New York, 1992.

[2] K. D. Anderson. Targeting recovery: Priorities of the spinal cord injured population.

J. Neurotrauma, 21:1371–1383, 2004.

[3] P. Artemiadis, G. Shakhnarovich, C. Vargas-Irwin, J. Donoghue, and M. J. Black.

Decoding grasp aperture from motor-cortical population activity. In Proc. of the IEEE

EMBS, pages 518–521, 2007.

[4] W Bishop, Cynthia A Chestek, V Gilja, P Nuyujukian, Stephen IRyu, Krishna V

Shenoy, and Byron M Yu. Long-term decoding stability withoutretraining for in-

tracortical brain computer interface. InComputational and Systems Neuroscience,

2012.

[5] B Blankertz, G Dornhege, C Schafer, R Krepki, J Kohlmorgen, K R Muller, V Kunz-

mann, F Losch, and G Curio. Boosting bit rates and error detection for the classifica-

tion of fast-paced motor commands based on single-trial eeganalysis.Neural Systems

and Rehabilitation Engineering, IEEE Transactions on, 11(2):127–131, 2003.

[6] A.E. Brockwell, A.L. Rojas, and R.E. Kass. Recursive Bayesiandecoding of motor

cortical signals by particle filtering.J Neurophysiol, 91(4):1899–1907, 2004.

[7] E.N. Brown, L.M. Frank, D. Tang, M.C. Quirk, and M.A. Wilson. A statistical

paradigm for neural spike train decoding applied to position prediction from the en-

semble firing patterns of rat hippocampal place cells.J Neurosci, 18(18):7411–7425,

1998.

112

BIBLIOGRAPHY 113

[8] P Brunner, A L Ritaccio, J F Emrich, H Bischof, and G Schalk. Rapid communication

with a p300 matrix speller using electrocorticographic signals (ecog).Frontiers in

Neuroscience, 5(00005), 2011.

[9] L. Buchen. Neuroscience: Illuminating the brain.Nature, 465:26–28, 2010.

[10] S K Card, W K English, and B J Burr. Evaluation of Mouse, Rate-Controlled Iso-

metric Joystick, Step Keys, and Text Keys for Text Selectionon a CRT.Ergonomics,

21(8):601–613, 1978.

[11] J. M. Carmena, M. A. Lebedev, C. S. Henriquez, and M. A. Nicolelis. Stable ensemble

performance with single-neuron variability during reaching movements in primates.

J Neurosci, 25(46):10712–10716, 2005.

[12] J.M. Carmena, M.A. Lebedev, R.E. Crist, J.E. O’Doherty, D.M. Santucci, D.F. Dim-

itrov, P.G. Patil, C.S. Henriquez, and M.A.L. Nicolelis. Learning to control a brain-

machine interface for reaching and grasping by primates.PLoS Biology, 1(2):193–

208, Oct 2003.

[13] Jose M Carmena. Becoming bionic.IEEE Spectrum, 49:24–29, March 2012.

[14] S. M. Chase, A. B. Schwartz, and R. E. Kass. Bias, optimal linear estimation, and the

differences between open-loop simulation and closed-loopperformance of spiking-

based brain-computer interface algorithms.Neural Networks, 22:1203–1213, 2009.

[15] C. A. Chestek, J. P. Cunningham, V. Gilja, P. Nuyujukian, S.I. Ryu, and K. V. Shenoy.

Neural prosthetic systems: Current problems and future directions. InProc. of the

Annual Conference of the IEEE (EMBS), 2009.

[16] C. A. Chestek, V. Gilja, P. Nuyujukian, R. Kier, F. Solzbacher, S. I. Ryu, R. R. Har-

rison, and K. V. Shenoy. HermesC: Low-power wireless neural recording system for

freely moving primates.IEEE Trans. in Neural Systems and Rehab. Eng., 17:330–

338, 2009.

[17] C.A Chestek, A.P. Batista, G. Santhanam, B.M. Yu, A. Afshar,J.P. Cunningham,

V. Gilja, S.I. Ryu, M.M. Churchland, and K.V. Shenoy. Single-neuron stability during

114 BIBLIOGRAPHY

repeated reaching in macaque premotor cortex.J Neurosci, 27(40):10742–10750, Oct

2007.

[18] Cynthia A Chestek, Vikash Gilja, Paul Nuyujukian, JustinD Foster, Joline M Fan,

Matthew T Kaufman, Mark M Churchland, Zuley Rivera-Alvidrez,John P Cunning-

ham, Stephen I Ryu, and Krishna V Shenoy. Long-term stabilityof neural prosthetic

control signals from silicon cortical arrays in rhesus macaque motor cortex.J Neural

Eng, 8(4):045005, Aug 2011.

[19] M. M. Churchland, J. P. Cunningham, M. T. Kaufman, S. I. Ryu,and K. V. Shenoy.

Cortical preparatory activity: Representation of movement or first cog in a dynamical

machine?Neuron, 2010.

[20] M. M. Churchland and K. V. Shenoy. Temporal complexity and heterogeneity of

single-neuron activity in premotor and motor cortex.J Neurophysiol, 97:4235–4257,

2007.

[21] M. M. Churchland, B. M. Yu, S. I. Ryu, G. Santhanam, and K. V. Shenoy. Neural

variability in premotor cortex provides a signature of motor preparation.J Neurosci,

26(14):3697–3712, Apr. 2006.

[22] J. P. Cunningham, V. Gilja, S. I. Ryu, and K. V. Shenoy. Methods for estimating neural

firing rates, and their application to brain-machine interfaces.Neural Networks, 2009.

doi:10.1016/j.neunet.2009.02.004. In Press.

[23] J. P. Cunningham, B. M. Yu, V. Gilja, S. I. Ryu, and K. V. Shenoy. Toward optimal

target placement for neural prosthetic devices.J Neurophysiol, 100(6):3445–3457,

2008.

[24] John P Cunningham, Paul Nuyujukian, Vikash Gilja, Cindy AChestek, Stephen I

Ryu, and Krishna V Shenoy. A closed-loop human simulator for investigating the

role of feedback control in brain-machine interfaces.J Neurophysiol, 105(4):1932–

49, Apr 2011.

BIBLIOGRAPHY 115

[25] Z. Danziger, A. Fishbach, and F.A. Mussa-Ivaldi. Learning algorithms for human-

machine interfaces.Biomedical Engineering, IEEE Transactions on, 56(5):1502 –

1511, 2009.

[26] P. Dayan and L.F. Abbott.Theoretical Neuroscience. MIT Press, Cambridge, 2001.

[27] Adam S Dickey, Aaron Suminski, Yali Amit, and Nicholas GHatsopoulos. Single-

unit stability using chronically implanted multielectrode arrays. Journal of Neuro-

physiology, 102(2):1331–1339, August 2009.

[28] I. Diester, M. T. Kaufman, M. Mogri, R. Pashaie, W. Goo, O.Yizhar, C. Ramakrish-

nan, K. Deisseroth, and K. V. Shenoy. An optogenetic toolboxdesigned for primates.

Nature Neurosci, 14:387–397, 2011.

[29] S. A. Douglas, A. E. Kirkpatrick, and I. S. MacKenzie. Testing pointing device per-

formance and user assessment with the iso9241, part 9 standard. In Proc. of the ACM

Conf. on Human Factors in Computing Systems (CHI 99), pages 215–222. ACM, New

York, 1999.

[30] U.T. Eden, L.M. Frank, R. Barbieri, V. Solo, and E. N. Brown.Dynamic analysis

of neural encoding by point process adaptive filtering.Neural Computation, 16:971–

998, 2004.

[31] Editorial. Is this the bionic man?Nature, 442:109, 2006.

[32] C Ethier, E R Oby, M J Bauman, and L E Miller. Restoration of grasp following

paralysis through brain-controlled stimulation of muscles. Nature, 485(7398):368–

71, May 2012.

[33] R Flint, W Zachary, and M Slutzky. Control of a biomimeticbrain machine interface

with local field potentials: Performance and stability of a static decoder over 200

days. InProc. of the 34th Annual IEEE EMBS Conference, page In press, 2012. in

press.

[34] J. D. Foster, O. Freifeld, P. Nuyujukian, S. I. Ryu, M. J. Black, and K. V. Shenoy.

Combining wireless neural recording and video capture for the analysis of natural

116 BIBLIOGRAPHY

gait.Proc. of the 5th International IEEE EMBS Conf on Neural Eng, Cancun, Mexico,

pages 613–616, 2011.

[35] D. W. Franklin and D. M. Wolpert. Specificity of reflex adaptation for task-relevant

variability. J. Neurosci., 28(52):14165–14175, 2008.

[36] G. W. Fraser, S. M. Chase, A. Whitford, and A. B. Schwartz. Control of a brain-

computer interface without spike sorting.J Neural Engineering, 6(5):1–8, Oct 2009.

[37] Gregory J Gage, Kip A Ludwig, Kevin J Otto, Edward L Ionides, and Daryl R Kipke.

Naive coadaptive cortical control.Journal of Neural Engineering, 2(2):52, 2005.

[38] K. Ganguly and J. M. Carmena. Emergence of a stable cortical map for neuropros-

thetic control.PLoS Biology, 7(7), Jul 2009.

[39] K. Ganguly, L. Secundo, G. Ranade, A. Orsborn, E. F. Chang,D. F. Dimitrov, J. D.

Wallis, N. M. Barbaro, R. T. Knight, and J. M. Carmena. Cortical representation of

ipsilateral arm movements in monkey and man.J Neurosci, 29:12948–12956, 2009.

[40] Karunesh Ganguly, Dragan F Dimitrov, Jonathan D Wallis, and Jose M Carmena. Re-

versible large-scale modification of cortical networks during neuroprosthetic control.

Nat Neurosci, 14(5):662–7, May 2011.

[41] Y. Gao, M.J. Black, E. Bienenstock, S. Shoham, and J.P. Donoghue. Probabilistic

inference of hand motion from neural activity in motor cortex. Advances in NIPS, 14,

2002.

[42] A.P. Georgopoulos, A.B. Schwartz, and R.E. Kettner. Neuronal population coding of

movement direction.Science, 233:1416–1419, 1986.

[43] Z. Ghahramani and D. Wolpert. Modular decomposition invisuoumotor learning.

Nature, 386:392–395, 1997.

[44] V. Gilja, C. Chestek, I. Diester, J. M. Henderson, K. Deisseroth, and K. V. Shenoy.

Challenges and opportunities for next-generation intra-cortically based neural pros-

theses.IEEE Trans Biomed Eng, 58(7):1891–9, Jan 2011.

BIBLIOGRAPHY 117

[45] V. Gilja, C. A. Chestek, P. Nuyujukian, J. D. Foster, and K.V. Shenoy. Autonomous

head-mounted electrophysiology systems for freely-behaving primates. Curr Opin

Neurobio, 20:676–686, 2010.

[46] V. Gilja, P. Nuyujukian, C. A. Chestek, J. P. Cunningham, B. M. Yu, S. I. Ryu, and

K. V. Shenoy. A high-performance continuous cortically-controlled prosthesis en-

abled by feedback control design.Program No. 20.6. Neuroscience Meeting Planner.

San Diego, CA: Society for Neuroscience, Online., 2010.

[47] V. Gilja, P. Nuyujukian, C. A. Chestek, J. P. Cunningham, B. M. Yu, S. I. Ryu, and

K. V. Shenoy. High-performance continuous neural control enabled by a feedback

control perspective.Fronteirs in Systems Neuroscience. Conference Abstract: Com-

putational and systems neuroscience (COSYNE), 2010.

[48] V Gilja, P Nuyujukian, Cynthia A Chestek, John P Cunningham, Byron M Yu, Jo-

line M Fan, Stephen I Ryu, and Krishna V Shenoy. A brain machineinterface control

algorithm designed from a feedback control perspective. InProc. of the 34th Annual

IEEE EMBS Conference, page In press, 2012. in press.

[49] Vikash Gilja, Paul Nuyujukian, Cynthia A Chestek, John P Cunningham, Byon M

Yu, Joline M Fan, Jonathan C Kao, Stephen I Ryu, and Krishna V Shenoy. A high-

performance neural prosthesis enabled by control algorithm design. Nature Neuro-

science, in press, 2012.

[50] Matthew D Golub, Byron M Yu, and Steven M Chase. Internal Models Engaged by

Brain-computer Interface Control. In34th Annual International Conference of the

IEEE EMBS, pages 1327–1330, San Diego, CA, 2012.

[51] M. A. Graziano. Where is my arm? the relative role of vision and proprioception in

the neuronal representation of limb position.PNAS, 96:10418–10421, 1999.

[52] A M Green and J F Kalaska. Learning to move machines with the mind. Trends

Neurosci, 34(2):61–75, Feb 2011.

118 BIBLIOGRAPHY

[53] X. Han, X. Qian, J. G. Bernstein, H.-H. Zhou, G. T. Franzesi, P. Stern, R. T. Bron-

son, A. M. Graybiel, R. Desimone, and E. S. Boyden. Millisecond-timescale optical

control of neural dynamics in the nonhuman primate brain.Neuron, 62:191–198,

2009.

[54] N. Hatsopoulos, J. Joshi, and J. G. O’Leary. Decoding continuous and discrete motor

behaviors using motor and premotor cortical ensembles.J Neurophysiol, 92:1165–

1174, 2004.

[55] D. J. Herzfeld and S. A. Beardsley. Improved multi-unit decoding at the brain-

machine interface using population temporal linear filtering. J Neural Engineering,

7(4), 2010.

[56] L. R. Hochberg. Turning thought into action.N Engl J Med, 359(11):1175–1177,

2008.

[57] L. R. Hochberg, M. D. Serruya, G. M. Friehs, J. A. Mukand, M. Saleh, A. H. Caplan,

A. Branner, D. Chen, R. D. Penn, and J. P. Donoghue. Neuronal ensemble control of

prosthetic devices by a human with tetraplegia.Nature, 442:164–171, Jul 2006.

[58] Leigh R Hochberg, Daniel Bacher, Beata Jarosiewicz, Nicolas Y Masse, John D

Simeral, Joern Vogel, Sami Haddadin, Jie Liu, Sydney S Cash, Patrick van der Smagt,

and John P Donoghue. Reach and grasp by people with tetraplegia using a neurally

controlled robotic arm.Nature, 485(7398):372–5, May 2012.

[59] Don Hopkins. The design and implementation of pie menus. Dr. Dobb’s Journal of

Software Tools, 16(12):16–26,94, Dec 1991.

[60] I. S. Howard, J. N. Ingram, and D. M. Wolpert. A modular planar robotic manipu-

landum with end-point torque control.J. Neuroscience Methods, 181(2):199 – 211,

2009.

[61] A. Jackson, J. Mavoori, and E. E. Fetz. Long-term motor cortex plasticity induced by

an electronic neural implant.Nature, 444:56–60, 2006.

BIBLIOGRAPHY 119

[62] H. D. Jellinek and S. K. Card. Powermice and user performance. Proc of SIGCHI

Conf on Human Factors in Computing Systems, pages 213–220, 1990.

[63] Jack W Judy. Neural interfaces for upper-limb prosthesis control: opportunities to

improve long-term reliability.IEEE Pulse, 3(2):57–60, Mar 2012.

[64] J F Kalaska. From intention to action: motor cortex and the control of reaching

movements.Adv Exp Med Biol, 629:139–178, 2009.

[65] R. E. Kalman. A new approach to linear filtering and prediction problems.J. Basic

Engineering, 82:35–45, 1960.

[66] M T Kaufman, M M Churchland, G Santhanam, B M Yu, A Afshar, SI Ryu, and

K V Shenoy. Roles of monkey premotor neuron classes in movement preparation and

execution.J Neurophysiol, 104(2):799–810, Aug 2010.

[67] C.T. Kemere, K.V. Shenoy, and T.H. Meng. Model-based neural decoding of reaching

movements: a maximum likelihood approach.IEEE Transactions on Biomedical

Engineering – Special issue on Brain-Machine Interfaces, 51:925–932, 2004.

[68] S. Kim, J. C. Sanchez, Y. N. Rao, D. Erdogmus, J. M. Carmena, M. A. Lebedev,

M. A. L. Nicolelis, and J. C. Principe. A comparison of optimalmimo linear and

nonlinear models for brain-machine interfaces.J Neural Engineering, 3:145–161,

2006.

[69] S. Kim, J. D. Simeral, L. R. Hochberg, J. P. Donoghue, and M. J. Black. Neural

control of computer cursor velocity by decoding motor cortical spiking activity in

humans with tetraplegia.J Neural Engineering, 5:455–476, 2008.

[70] S. P. Kim, J. D. Simeral, L. R. Hochberg, J. P. Donoghue, G.Friehs, and M. J. Black.

Point-and-click cursor control with an intracortical neural interface system in humans

with tetraplegia.IEEE Trans Neural Syst Rehabil Eng, 19(2):193–203, Apr 2011.

[71] S. Koyama, S. M. Chase, A. S. Whitford, M. Velliste, A. B. Schwartz, and R. E.

Kass. Comparison of brain-computer interface decoding algorithms in open-loop and

closed-loop control.J Computational Neuroscience, 29(1-2):73–87, 2009.

120 BIBLIOGRAPHY

[72] D J Krusienski, E W Sellers, D J McFarland, Vaughan T M, and J R Wolpaw. Toward

enhanced p300 speller performance.Journal of Neuroscience Methods, 167(1):15–

21, 2008.

[73] D J Krusienski and J J Shih. Control of a visual keyboard using an electrocortico-

graphic brain-computer interface.Neurorehabilitation and Neural Repair, 25(4):323–

331, 2011.

[74] J. E. Kulkarni and L. Paninski. State-space decoding ofgoal-directed movements.

IEEE Sig Proc Magazine, 25(1):78–86, 2008.

[75] M. Kumar, A. Paepcke, and T. Winograd. Eyepoint: Practical pointing and selection

using gaze and keyboard.Proc of SIGCHI Conf on Human Factors in Computing

Systems, pages 421–430, 2007.

[76] M. A. Lebedev and M. A. L. Nicolelis. Brain-machine interfaces: past, present, and

future. Trends in Neurosci, 29, 2006.

[77] M.A. Lebedev, J.M. Carmena, J.E. O’Doherty, M. Zacksenhouse, C.S. Henriquez,

J.C. Principe, and M.A.L. Nicolelis. Cortical ensemble adaptation to represent ve-

locity of an artificial actuator controlled by a brain-machine interface.J Neurosci,

25(19):4681–4693, 2005.

[78] E.C. Leuthardt, G. Schalk, J.R. Wolpaw, J.G. Ojemann, andD.W. Moran. A brain-

computer interface using electrocorticographic signals in humans.Journal of Neural

Engineering, 1:63–71, 2004.

[79] Z. Li, J. E. O’Doherty, T. L. Hanson, M. A. Lebedev, C. S. Henriquez, and M. A. L.

Nicolelis. Unscented kalman filter for brain-machine interfaces. PLoS One, 4(7),

2009.

[80] Z Li, J E O’Doherty, M A Lebedev, and M A Nicolelis. Adaptive decoding for

brain-machine interfaces through bayesian parameter updates. Neural Computation,

23(12):3162–3204, Dec 2011.

BIBLIOGRAPHY 121

[81] A. R. Marathe, K. E. Ward, M. Maryampillai, and D. M. Taylor. Are your move-

ment decoding methods appropriate for online control of neuroprosthetics?Soc. for

Neurosci. Abstr., 2009.

[82] D J McFarland, W A Sarnacki, and J R Wolpaw. Electroencephalographic (eeg)

control of three-dimensional movement.J Neural Eng, 7(3):036007–036007, Jun

2010.

[83] H. Miranda, V. Gilja, C. A. Chestek, K. V. Shenoy, and T. H. Meng. HermesD:

A high-rate long-range wireless transmission system for simultaneous multichannel

neural recording applications.IEEE Transactions on Biomedical Circuits and Sys-

tems, 4:181–191, 2010.

[84] Daniel Moran. Evolution of brain-computer interface:action potentials, local field

potentials and electrocorticograms.Curr Opin Neurobiol, 20(6):741–5, Dec 2010.

[85] D.W. Moran and A.B. Schwartz. Motor cortical representation of speed and direction

during reaching.J Neurophysiol., 82(5):2676–2692, 1999.

[86] C. T. Moritz, S. I. Perlmutter, and E. E. Fetz. Direct control of paralysed muscles by

cortical neurons.Nature, 456:639–642, 2008.

[87] G. H. Mulliken, S. Musallam, and R. A. Andersen. Decodingtrajectories from poste-

rior parietal cortex ensembles.J Neurosci, 28(48):12913–12926, Nov 2008.

[88] S. Musallam, B.D. Corneil, B. Greger, H. Scherberger, and R.A. Andersen. Cognitive

control signals for neural prosthetics.Science, 305:258–262, 2004.

[89] William Newman and Robert F Sproull.Principles of Interactive Computer Graphics.

McGraw-Hill, 2 edition, Jun 1979.

[90] Luis Fernando Nicolas-Alonso and Jaime Gomez-Gil. Brain computer interfaces, a

review. Sensors, 12(2):1211–1279, 2012.

122 BIBLIOGRAPHY

[91] P. Nuyujukian, V. Gilja, C. A. Chestek, J. P. Cunningham, J.M. Fan, B. M. Yu, S. I.

Ryu, and K. V. Shenoy. Generalization and robustness of a continuous cortically-

controlled prosthesis enabled by feedback control design.Program No. 20.7. Neuro-

science Meeting Planner. San Diego, CA: Society for Neuroscience, Online., 2010.

[92] Paul Nuyujukian, Joline M Fan, Vikash Gilja, Paul S Kalanithi, Cindy A Chestek,

and Krishna V Shenoy. Monkey models for brain-machine interfaces: the need for

maintaining diversity.Conf Proc IEEE Eng Med Biol Soc, 2011:1301–5, 2011.

[93] Paul Nuyujukian, Jonathan Kao, Joline M Fan, Sergey D Stavisky, Stephen I Ryu,

and Krishna V Shenoy. A high-performance robust brain-machine interface without

retraining. InComputational and Systems Neuroscience, 2012.

[94] Joseph E O’Doherty, Mikhail A Lebedev, Peter J Ifft, Katie Z Zhuang, Solaiman

Shokur, Hannes Bleuler, and Miguel A L Nicolelis. Active tactile exploration using a

brain-machine-brain interface.Nature, 479(7372):228–31, Nov 2011.

[95] L. Paninski, S. Shoham, M.R. Fellows, N.G. Hatsopoulos,and J.P. Donoghue. Su-

perlinear population encoding of dynamic hand trajectory in primary motor cortex.J

Neurosci., 24:8551–8561, 2004.

[96] E. A. Pohlmeyer, E. R. Oby, E. J. Perreault, S. A. Solla, K.L. Kilgore, R. F. Kirsch,

and L. E. Miller. Toward the restoration of hand use to a paralyzed monkey: brain-

controlled functional electrical stimulation of forearm muscles.PLoS One, 4:e5924,

2009.

[97] S. M. Radhakrishnan, S. N. Baker, and A. Jackson. Learninga Novel Myoelectric-

Controlled Interface Task.J Neurophysiol, 100(4):2397–2408, 2008.

[98] S. I. Ryu and K. V. Shenoy. Human cortical prostheses: Lost in translation?Neuro-

surgical Focus, 27, 2009.

[99] P. T. Sadtler, S. I. Ryu, B. M. Yu, and A. P. Batista. High-performance neural pros-

thetic control along instructed paths.Proc. of the 5th International IEEE EMBS Conf

on Neural Eng, Cancun, Mexico, pages 601–604, 2011.

BIBLIOGRAPHY 123

[100] J. C. Sanchez, J. C. Principe, T. Nishida, R. Bashirullah, J. G. Harris, and J. Fortes.

Technology and signal processing for brain-machine interfaces: The need for beyond

the state-of-the-art tools.IEEE Sig Proc Magazine, 25(1):29–40, 2008.

[101] G. Santhanam, M. D. Linderman, V. Gilja, A. Afshar, S. I. Ryu, T. H. Meng, and

K. V. Shenoy. Hermesb: A continuous neural recording systemfor freely behaving

primates.IEEE Transactions in Biomedical Engineering, 54:2037–2050, 2007.

[102] G. Santhanam, S. I. Ryu, B. M. Yu, A. Afshar, and K. V. Shenoy. A high-

performance brain-computer interface.Nature, 442, 2006.

[103] G. Santhanam, M. Sahani, S. I. Ryu, and K. V. Shenoy. An extensible infrastructure

for fully automated spike sorting during online experiments. In Proceedings of the

26th Annual International Conference of the IEEE EMBS, volume 2, pages 4380–

4384, 2004.

[104] D. M. Santucci, J. D. Kralik, M. A. Lebedev, and M. A. L. Nicolelis. Frontal and

parietal cortical ensembles predict single-trial muscle activity during reaching move-

ments in primates.Eur. J. Neurosci, 22:1529–1540, 2005.

[105] G Schalk and E C Leuthardt. Brain-computer interfaces using electrocorticographic

signals.Biomedical Engineering, IEEE Reviews in, 4:140–154, 2011.

[106] G Schalk, K J Miller, N R Anderson, J A Wilson, M D Smyth, JG Ojemann, D W

Moran, J R Wolpaw, and E C Leuthardt. Two-dimensional movement control using

electrocorticographic signals in humans.J Neural Eng, 5(1):75–84, Mar 2008.

[107] A. B. Schwartz. Cortical neural prosthetics.Ann. Rev. Neurosci, 27:487–507, 2004.

[108] A. B. Schwartz, X. T. Cui, D. J. Weber, and D. W. Moran. Brain-controlled inter-

faces: Movement restoration with neural prosthetics.Neuron, 52(1):205 – 220, 2006.

[109] M.D. Serruya, N.G. Hatsopoulos, L. Paninski, M.R. Fellows, and J.P. Donoghue.

Instant neural control of a movement signal.Nature, 416:141–142, 2002.

124 BIBLIOGRAPHY

[110] Mijail Serruya, Nicholas Hatsopoulos, Matthew Fellows, Liam Paninski, and John

Donoghue. Robustness of neuroprosthetic decoding algorithms. Biological Cyber-

netics, 88:219–228, 2003. 10.1007/s00422-002-0374-6.

[111] R Shadmehr and F. A. Mussa-Ivaldi. Adaptive representation of dynamics during

learning of a motor task.J. Neurosci., 14(5):3208–3224, 1994.

[112] R. Shadmehr and S. P. Wise.The Computational Neurobiology of Pointing and

Reaching. MIT Press, Cambridge, 2005.

[113] G. Shakhnarovich, S. P. Kim, and M. J. Black. Nonlinear physically-based models

for decoding motor-cortical population activity. InAdvances in NIPS 18. MIT Press,

Cambridge, MA, 2006.

[114] K.V. Shenoy, D. Meeker, S. Cao, S.A. Kureshi, B. Pesaran,P. Mitra, C.A. Buneo,

A.P. Batista, J.W. Burdick, and R.A. Andersen. Neural prosthetic control signals from

plan activity.NeuroReport, 14:591–596, 2003.

[115] S. Shoham, L.M. Paninski, M.R. Fellows, N.G. Hatsopoulos, J.P. Donoghue, and

R.A. Normann. Statistical encoding model for a primary motorcortical brain-machine

interface.IEEE Transactions on TBME, 52(7):1313–1322, Jul. 2005.

[116] L. Shpigelman, H. Lalazar, and E. Vaadia (2009). Kernel-arma for hand tracking and

brain-machine interfacing during 3D motor control.D. Koller, D. Schuurmans, Y. Ben-

gio, L. Bottou (eds.) Advances in Neural Information Processing Systems, 21:1489—

1496, 2009.

[117] J. D. Simeral, S.-P. Kim, M. J. Black, J. P. Donoghue, andL. R. Hochberg. Neural

control of cursor trajectory and click by a human with tetraplegia 1000 days after

implant of an intracortical microelectrode array.J Neural Eng, 8(2):025027, Apr

2011.

[118] S. J. Sober and P. N. Sabes. Multisensory Integration during Motor Planning.J.

Neurosci., 23(18):6982–6992, 2003.

BIBLIOGRAPHY 125

[119] L. Srinivasan and E. N. Brown. A state-space framework for movement control to

dynamic goals through brain-driven interfaces.IEEE Transactions on Biomedical

Engineering, 54(3):526–535, 2007.

[120] L. Srinivasan, U. T. Eden, S. J. Mitter, and E. N. Brown. General purpose filter

design for neural prosthetic systems.J Neurophysiol, 98(4):2456–75, Oct 2007.

[121] L. Srinivasan, U. T. Eden, A. S. Willsky, and E. N. Brown.A state-space analysis for

reconstruction of goal-directed movements using neural signals.Neural Computation,

18:2465–2494, 2006.

[122] A. J. Suminski, D. C. Tkach, A. H. Fagg, and N. G. Hatsopoulos. Incorporating

feedback from multiple sensory modalities enhances brain-machine interface control.

J Neurosci, 30:16777–16787, 2010.

[123] S Suner, M R Fellows, C Vargas-Irwin, G K Nakata, and J P Donoghue. Reliability

of signals from a chronically implanted, silicon-based electrode array in non-human

primate primary motor cortex.Neural Systems and Rehabilitation Engineering, IEEE

Transactions on, 13(4):524–541, Dec 2005.

[124] D.M. Taylor, S.I. Helms Tillery, and A.B. Schwartz. Direct cortical control of 3D

neuroprosthetic devices.Science, 296, 2002.

[125] J. Tian and J He. Can EMG machine interface be used to model brain machine

interface? InProceedings of the 25th Annual International Conference of the IEEE

EMBS, pages 1658–1661, 2003.

[126] E. Todorov. Direct cortical control of muscle activation in voluntary arm movements:

a model.Nature Neuroscience, 3:391–398, 2000.

[127] P. Touzalin-Chretien, S. Ehrler, and A. Dufour. Dominance of vision over proprio-

ception on motor programming: Evidence from erp.Cereb. Cortex, 2009.

[128] R. J. van Beers, A. C. Sittig, and J. J. Denier van der Gon. Integration of pro-

prioceptive and visual position-information: An experimentally supported model.J

Neurophysiol, 81(3):1355–1364, 1999.

126 BIBLIOGRAPHY

[129] C.E. Vargas-Irwin, G. Shakhnarovich, P. Yadollahpour, J.M.K. Mislow, M. J. Black,

and J. P. Donoghue. Decoding complete reach and grasp actions from local primary

motor cortex populations.J Neurosci, 30:9659–9669, 2010.

[130] M. Velliste, S. Perel, M. C. Spalding, A. S. Whitford, andA. B. Schwartz. Cortical

control of a prosthetic arm for self-feeding.Nature, 453:1098–1101, Jun 2008.

[131] V. Ventura. Spike train decoding without spike sorting. Neural Computation,

20(4):923–963, Apr 2008.

[132] J. Wessberg, C.R. Stambaugh, J.D. Kralik, P.D. Beck, M. Laubach, J.K. Chapin,

J. Kim, S.J. Biggs, M.A. Srinivasan, and M.A.L. Nicolelis. Real-time prediction of

hand trajectory by ensembles of cortical neurons in primates. Nature, 408:361–365,

2000.

[133] C Wheatstone. Contributions to the physiology of vision- part the first. on some

remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical

Transactions of the Royal Society of London, 128:371–394, 1838.

[134] T. Winograd and F. Flores.Understanding Computers and Cognition: A new foun-

dation for design. Addison-Wesley, Norwood, NJ, 1987.

[135] D. Wolpert and Z. Ghahramani. Computational principles of movement neuro-

science.Nature Neuroscience, 3:1212–1217, 2000.

[136] D. Wolpert, Z. Ghahramani, and M. Jordan. An internal model for sensorimotor

integration.Science, 269:1880–1882, 1995.

[137] W. Wu, M.J. Black, Y. Gao, E. Bienenstock, M. Serruya, andJ.P. Donoghue. In-

ferring hand motion from multi-cell recordings in motor cortex using a Kalman filter.

In SAB’02-Workshop on Motor Control in Humans and Robots: On theInterplay of

Real Brains and Artificial Devices, pages 66–73, 2002.

[138] W. Wu, M.J. Black, D. Mumford, Y. Gao, E. Bienenstock, andJ.P. Donoghue. Mod-

eling and decoding motor cortical activity using a switching Kalman filter. IEEE

Transactions on TBME, 51(6):933–942, Jun. 2004.

BIBLIOGRAPHY 127

[139] W. Wu, Y. Gao, E. Bienenstock, J.P. Donoghue, and M.J. Black. Bayesian popula-

tion decoding of motor cortical activity using a Kalman filter. Neural Computation,

18(1):80–118, Jan. 2006.

[140] W. Wu and N. G. Hatsopoulos. Target-included model andhybrid decoding of

stereotyped hand movement in the motor cortex. InIntl Conf on Biomed Robotics

and Biomech, 2008. (IEEE RAS/EMBS), pages 55–60, 2008.

[141] B. M. Yu, C. Kemere, G. Santhanam, A. Afshar, S. I. Ryu, T. H.Meng, M. Sahani,

and K. V. Shenoy. Mixture of trajectory models for neural decoding of goal-directed

movements.J Neurophysiol, 97:3763–3780, 2007.

[142] J. H. Zar.Biostatistical Analysis. Prentice Hall, New Jersey, 1999.

TOWARDS CLINICALLY RELEVANT NEURAL PROSTHESES A ...shenoy/Theses/Nuyujukian.pdf · towards...

Documents

Transcript of TOWARDS CLINICALLY RELEVANT NEURAL PROSTHESES A ...shenoy/Theses/Nuyujukian.pdf · towards...