ADVANCING MOTOR NEURAL PROSTHESIS ROBUSTNESS AND NEUROSCIENCEshenoy/Theses/Stavisky.pdf ·...
Transcript of ADVANCING MOTOR NEURAL PROSTHESIS ROBUSTNESS AND NEUROSCIENCEshenoy/Theses/Stavisky.pdf ·...
ADVANCING MOTOR NEURAL PROSTHESIS
ROBUSTNESS AND NEUROSCIENCE
A DISSERTATION
SUBMITTED TO THE NEUROSCIENCES PROGRAM
AND THE COMMITTEE ON GRADUATE STUDIES
OF STANFORD UNIVERSITY
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
Sergey D. Stavisky
March 2016
http://creativecommons.org/licenses/by-nc/3.0/us/
This dissertation is online at: http://purl.stanford.edu/hq855qg2603
© 2016 by Sergey Stavisky. All Rights Reserved.
Re-distributed by Stanford University under license with the author.
This work is licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License.
ii
I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.
Krishna Shenoy, Primary Adviser
I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.
William Newsome
I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.
Anthony Norcia
I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.
Jennifer Raymond
Approved for the Stanford University Committee on Graduate Studies.
Patricia J. Gumport, Vice Provost for Graduate Education
This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file inUniversity Archives.
iii
Abstract
The frontier challenges that must be solved before brain-machine interfaces (BMIs) can be used
as clinically useful motor prostheses differ depending on the degree of function being restored.
Two-dimensional cursor control (i.e., for communication) has recently reached high levels of peak
performance in pre-clinical studies (e.g. Gilja et al., 2015), but translation is hampered by less than
reliable performance due to unstable neural signals (despite substantial inroads, e.g., Jarosiewicz
et al. 2015). Meanwhile, control of robotic arms remains poor, despite some impressive glimpses at
what the future could be (e.g. Collinger et al., 2013b), because we lack fundamental understanding
of how the brain incorporates the BMI into its motor schema. This hampers our ability to accurately
decode intended arm movements. My dissertation focused on both sets of problems in pre-clinical
macaque BMI studies. Chapters 2 and 3 provide neural engineering solutions for improving BMI
robustness. Chapter 4 looks ahead to challenges that will be encountered when BMI-controlled limbs
operate in the physical world by describing how movement error signals impact ongoing BMI control.
Chapter 2 describes a machine learning approach to building decoder algorithms that are robust
to the changing neural-to-kinematic mappings that plague translational BMI efforts. We developed
a multiplicative recurrent neural network decoder that could exploit the large quantities of data
generated by a chronic BMI — data that has heretofore gone unused. This novel decoder was
markedly more robust to both artificial and naturally occurring recording condition changes than an
existing state-of-the-art decoder. This robustness benefit came without a performance tradeoff under
more ideal conditions.
Chapter 3 describes a neural engineering approach for increasing the device lifespan of a BMI
by providing high performance control even after losing spike signals. I developed a method for
decoding local field potentials (LFPs) as a longer-lasting alternative or complimentary BMI control
signal. This led to the highest-performing LFP-driven BMI and the first ’hybrid’ BMI which decoded
kinematics from spikes and LFPs together. This study also demonstrated that a decoding method’s
control loop latency can result in dramatic differences between offline and closed-loop performance,
and that this should serve as a cautionary lesson to guide future BMI design.
Chapter 4 focuses on one project that exemplifies my efforts to understand the neuroscience
of BMI use by studying it as a feedback control system. BMIs used in real-world settings will
iv
encounter unexpected perturbations due to, for example, collisions with other objects. Fortunately,
BMI use, like natural movement, is a closed-loop process in which sensory feedback conveying
errors is transformed into motor corrections. But this presents a potential problem in areas such as
motor cortex, which both receive error signals and generate efferent motor commands: error-related
modulation needs to not ”leak out” to the effector unless it would generate the appropriate corrective
movement. While this is the case for natural movement, in which motor cortex indirectly affects
muscles, what happens when motor cortical activity directly drives the effector via a BMI? I tested
this by perturbing the kinematics of monkeys performing a BMI-controlled cursor task. Visual
feedback drove cortical responses as early as 70 ms after the perturbation, but this initial modulation
did not cause deleterious motor feedback. Rather, it was limited to a neural null space in which
activity does not affect the BMI. When neural activity changed in output-potent dimensions starting
115 ms after perturbation, it caused corrective BMI movement. This elegant arrangement may hint
at a broader computational strategy by which error processing is separated from output.
v
Acknowledgments
My graduate training, which has culminated in completing this dissertation, owes its success to many
people. First I would like to thank my advisor, Professor Krishna Shenoy. Krishna is an amazing
person and mentor. In addition to being a brilliant and creative scientist, he is also an extraordinarily
kind and caring mentor who has time after time put his people first, despite everything he’s had to
go through. He’s taught me to be the best scientist I can be, and to do so with integrity, purpose,
courage, and humility. The success of his group, as measured by its scientific contributions and the
subsequent achievements of those who have come through it, are a testament to the amazing research
and teamwork culture that Krishna has cultivated. I am incredibly fortunate, and grateful, to have
had this opportunity to learn from him.
My research also owes a great deal to the prior work, help, and advice of the Shenoy group, past
and present: Jonathan Kao, Dr. Paul Nuyujukian, Dan O’Shea, Dr. Cora Ames, Eric Trautmann, Dr.
Joline Fan, Dr. Justin Foster, Dr. Werapong Goo, Megan Wang, Nir Even-Chen, Xulu Sun, Jessie
Verhein, John Aguayo, Clare Sherman, Erica Morgan, Liana Yates, Dr. Stephen Ryu, Mackenzie
Mazariegos, Michelle Wechsler, Steven Lin, Shannon Smith, Rosie Steinbach, Beverly Davis, Dr.
Boris Oskotsky, Dr. Chandramouli Chandrasekaran, Dr. David Sussillo, Dr. Mark Churchland, Dr.
Matthew Kaufman, Dr. Vikash Gilja, Dr. Cindy Chestek, Dr. Ilka Siester, Dr. Chethan Pandarinath,
and Dr. Paul Kalanithi, as well as our close collaborator, Professor Maneesh Sahani, and his group.
You’re a group of the smartest people I’ve ever been around, and I wouldn’t have learned half as
much as I did without you all. You’re also good, generous, kind, funny and warm people and I
treasure our scientific family. A few of these names deserve additional mention. Paul N. was the
senior student on the prosthetics team when I joined the group, and helped train me to work with our
macaques and understand the experimental rig’s software. Jonathan was my teammate throughout
our graduate experience, and I couldn’t ask for a more brilliant, humble, and helpful colleague. Matt
stepped up as the acting laboratory leader during Krishna’s prolonged health-related absence, and
provided me with crucial mentoring and advice during my second year of graduate school, which
lived up to its notoriety of existential difficulty. Mackenzie is an unmatched laboratory manager and
monkey czar, and I’m extraordinarily grateful for her above-and-beyond care of our research animals.
Knowing she could be called upon to expertly deal with any emergency gave me peace of mind. I
vi
also want to explicitly acknowledge and thank Jonathan, Paul N., David, Stephen, and Krishna, who
were my collaborators on the projects described here and co-authors on the resulting journal papers.
My thesis committee — Professors Bill Newsome, Jennifer Raymond, and Tony Norcia — provided
highly helpful scientific and career advice throughout my graduate training. Thank you for being
my sounding-board and cheerleaders. I’m also grateful to Professor Tom Clandinin for chairing my
defense, and to Professors Steve Baccus and Tony Norcia for mentoring me during my first-year
rotations in their laboratories.
Outside the lab, my science has benefited from the excellent neurosciences community at Stanford.
I’m thankful to Yvette Fisher, Emily Ferenczi, Diogo Peixoto, Hannah Payne, Vince McGinty, David
Kastner, Georgia Panagiotakos, Niru Maheswaranathan, Lane McIntosh, Amy Braun, Becca Krock,
Kiah Hardcastle, Jordan Sorokin, for their feedback, which helped me become a better student and
scientist. My training experience in Stanford’s Neurosciences graduate program has been excellent
not only due to the superb faculty and students, but also because the program is very well run:
its administration is both efficient and shows personalized concerned for each individual’s success.
This excellence is due to the hard work of the program directors (Professors Tony Ricci and John
Huguenard) and staff (Kalai Diamond, Ross Colvin, Katie Johnson, and Sheila Quimpo).
Stanford has been an exhilarating intellectual and professional journey, but the past five and a half
years have also been immensely interesting and joyous outside of the lab. For this, I must thank my
new friends from within my Program and from varied and serendipitous meetings outside it. Thank
you all for making the Bay Area my home. It is a rare thing in academia to be able to stay in the
same area after finishing one’s Ph.D, and I feel extraordinarily fortunate to be able to stay close to
you in this most beautiful of lands.
At the same time, I’ve also been fortunate to keep in close touch with my friends from childhood
and University, most of whom are scattered across the East Coast. It’s been a joy to host you in San
Francisco and to visit you through frequent trips to New York, Boston, and D.C. Though a part of
me is sad not to return east for my postdoc, I look forward to the numerous upcoming weddings
(and occasional vacations) as opportunities to see you all, as I miss you dearly.
Lastly, I thank my family for giving me such a loving foundation from which to strive for the
Ph.D. and a career in science. I am especially grateful to my parents, Natalia and Dmitry Stavisky,
for sacrificing their own scientific careers to emigrate from the Soviet Union and build a new and
better life for us in the United States. It’s humbling to think that in the time it’s taken me to finish
my degree, you’d already built a second career after having finished your scientific education, while
also raising me.
Sergey D. Stavisky
Stanford University, California, USA, March 2016
vii
Contents
Abstract iv
Acknowledgments vi
1 Introduction 1
1.1 The premise and promise of motor neural prostheses . . . . . . . . . . . . . . . . . . 1
1.2 Key challenges to clinically viable motor neural prostheses . . . . . . . . . . . . . . . 3
1.3 Dissertation overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Animal model and general experimental setup . . . . . . . . . . . . . . . . . . . . . . 5
2 Harnessing months of previously collected data to improve brain-machine interface
robustness to future neural variability 8
2.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.1 An MRNN can leverage large amounts of training data to improve decoder
performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2 Robustness to unexpected loss of the most informative electrodes . . . . . . . 15
2.3.3 Robustness to naturally occurring recording condition changes . . . . . . . . 16
2.3.4 High performance BMI using the MRNN decoder . . . . . . . . . . . . . . . . 18
2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.1 A robust and high-performing BMI decoder . . . . . . . . . . . . . . . . . . . 23
2.4.2 Key innovations enabling the MRNN’s robustness . . . . . . . . . . . . . . . 24
2.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5.1 Animal model and neural recordings . . . . . . . . . . . . . . . . . . . . . . . 25
2.5.2 Behavioral tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5.3 Multiplicative Recurrent Neural Network (MRNN) decoder . . . . . . . . . . 27
2.5.4 Training the MRNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5.5 MRNN training data augmentation . . . . . . . . . . . . . . . . . . . . . . . . 29
viii
2.5.6 Controlling the BMI cursor with MRNN network output . . . . . . . . . . . . 30
2.5.7 Feedback Intention Trained Kalman Filter (FIT-KF) decoder . . . . . . . . . 30
2.5.8 Mutual information for determining electrode dropping order . . . . . . . . . 31
2.5.9 Principal angles of neural subspaces analysis . . . . . . . . . . . . . . . . . . 32
2.6 Project acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3 A high performing brain-machine interface driven by low-frequency local field
potentials alone and together with spikes 34
3.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.1 Selection of local motor potential as a control signal based on offline evaluation
of candidate LFP features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.2 Control experiment showing LMP is not a movement artifact . . . . . . . . . 38
3.3.3 Closed-loop BMI performance driven by LMP . . . . . . . . . . . . . . . . . . 42
3.3.4 Fixed LMP decoders work across multiple days without retraining . . . . . . 46
3.3.5 BMI control using a hybrid spikes and LMP decoder . . . . . . . . . . . . . . 46
3.3.6 LMP decoding has a longer control loop latency than spikes decoding . . . . 48
3.3.7 Hybrid decoding can rescue performance when spikes are lost . . . . . . . . . 51
3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.1 Choosing LMP for BMI use based on offline decode performance . . . . . . . 55
3.4.2 Closed-loop LMP-driven BMI performance . . . . . . . . . . . . . . . . . . . 57
3.4.3 Hybrid decoding using both LMP and spikes . . . . . . . . . . . . . . . . . . 59
3.4.4 Decoding LMP to mitigate losing spikes signals . . . . . . . . . . . . . . . . . 60
3.4.5 Differences between offline and closed-loop results . . . . . . . . . . . . . . . 61
3.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.5.1 Subjects and neural recording . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.5.2 Behavioral tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.5.3 Signal processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.5.4 Neural decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.5.5 Performance measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.5.6 Electrode preferred direction and decoder contribution . . . . . . . . . . . . . 70
3.5.7 Comparing the contribution of LMP and spikes in the hybrid decoder . . . . 71
3.6 Project Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4 Effect of Sensory Error Signals on BMI Use 73
4.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
ix
4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.3.1 Perturbations evoked an early motor cortical response
without accompanying cursor velocity output . . . . . . . . . . . . . . . . . . 77
4.3.2 Initial error-related neural activity was confined to a decoder-null space . . . 77
4.3.3 Perturbation-specific information in the early error-related response . . . . . 84
4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.4.1 BMI error correction latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.4.2 Early BMI error-related signals are present in motor cortex but do not interfere
with ongoing BMI control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.3 Computing in an output-null subspace provides a mechanism by which BMI
output can ignore early error signals . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.4 Differences between responses to different jump location conditions . . . . . . 88
4.4.5 Implications for future BMI design . . . . . . . . . . . . . . . . . . . . . . . . 89
4.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.5.1 Cursor jump task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.5.2 Neural recording and decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.5.3 Analysis of responses to cursor jumps . . . . . . . . . . . . . . . . . . . . . . 92
4.5.4 Neural response from the decoder’s perspective . . . . . . . . . . . . . . . . . 93
4.5.5 Electrode neural push direction and decoder contribution weight . . . . . . . 94
4.6 Project Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5 Conclusion 96
5.1 Summary of motor neural prosthesis advances . . . . . . . . . . . . . . . . . . . . . . 96
5.2 Next steps: testing in a human clinical trial . . . . . . . . . . . . . . . . . . . . . . . 97
5.3 Additional graduate work not covered in this dissertation . . . . . . . . . . . . . . . 98
5.3.1 Drawing scientific conclusions about neural population activity without sorting
single neuron activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.3.2 Observed BMI cursor position only weakly affects motor cortical activity . . 100
5.3.3 Single trial motor cortical correlates of a rapidly adapting visuomotor internal
model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.3.4 Rapid but limited adaptation to a change in BMI velocity gain . . . . . . . . 101
5.3.5 Do motor cortical dynamical rules constrain the activity that can be generated
for BMI control? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
x
List of Tables
2.1 Network and training parameters used for the closed-loop MRNN BMI decoder . . . 28
xi
List of Figures
1.1 Schematic of a motor neural prosthesis . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Rhesus macaque motor cortical regions recorded from . . . . . . . . . . . . . 6
2.1 Our strategy for training a decoder robust to recording condition changes 10
2.2 An MRNN decoder can harness large training datasets . . . . . . . . . . . . 12
2.3 Similar neural recording conditions can often be found on other days . . . 14
2.4 Robustness to unexpected loss of the most important electrodes . . . . . . 16
2.5 Artificially dropped electrodes were active in the training data . . . . . . . 17
2.6 Robustness to naturally occurring recording condition changes . . . . . . . 18
2.7 Training data spike rate perturbations improve MRNN performance . . . 19
2.8 Additional tests showing that FIT Old typically performs poorly . . . . . . 20
2.9 The MRNN outperforms the FIT Kalman filter under “ideal” conditions 21
2.10 Closed-loop MRNN decoder performance on the Random Target Task . . 22
3.1 Overview of local motor potential (LMP) BMI signal processing . . . . . . 37
3.2 Comparison of offline decode accuracy using different neural features . . . 39
3.3 Offline decode analysis of additional candidate LFP features . . . . . . . . . 40
3.4 Directional tuning of LMP overlaid on electrode location in motor cortex 41
3.5 LMP-driven BMI performance on the Continuous Random Target Task . 43
3.6 Performance of the LMP decoder on a Radial 8 Task . . . . . . . . . . . . . 44
3.7 Performance on a Continuous Random Target Task with short hold time 45
3.8 Stable performance using a fixed LMP decoder . . . . . . . . . . . . . . . . . 47
3.9 Closed-loop comparisons of different neural control signals . . . . . . . . . . 49
3.10 LMP underperforms online due to longer control loop latency . . . . . . . 50
3.11 Performance of spikes-only and hybrid decoders after electrode dropping 52
3.12 Cumulative electrode contribution to the spikes-only and LMP-only decoders 54
3.13 Closed-loop LMP decoding benefited from half-wave rectification . . . . . 66
4.1 Should we worry about error signals affecting ongoing control? . . . . . . . 74
xii
4.2 BMI cursor jump task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.3 Rapid motor cortical perturbation response without deleterious BMI output 78
4.4 Perturbations evoked both increases and decreases of firing rates . . . . . 79
4.5 Examples of neural push following a cursor jump . . . . . . . . . . . . . . . . 80
4.6 Decoder null to decoder potent neural dynamics hypothesis . . . . . . . . . 82
4.7 Initial error-related modulation is restricted to the decoder’s null space . 83
4.8 Responses in the potent and null space of an observation-trained decoder 84
4.9 Electrode decoder contribution weight curves . . . . . . . . . . . . . . . . . . 85
4.10 Summary schematic of early error response confined to an output-null space 86
xiii
Chapter 1
Introduction
1.1 The premise and promise of motor neural prostheses
Motor neural prostheses are an emerging type of medical device with the potential to restore motor
function to people with paralysis (Figure 1.1). The core premise of motor neural prosthesis, a type of
brain-machine interfaces (BMI), is to decode neural activity to allow a user to directly control an
effector such as a computer cursor (Ganguly & Carmena, 2009; Gilja et al., 2012) or robotic arm
(Carmena et al., 2003; Taylor et al., 2002).
Inspired by initial studies in non-human primates, intracortical BMIs are now being tested in
pilot clinical trials to restore communication (Bacher et al., 2015; Pandarinath et al., 2014) and
movement (Collinger et al., 2013b; Hochberg et al., 2012). This technology can work across a wide
range of neurological damage etiologies because it decodes movement intentions directly from the
brain, upstream of most strokes, neurodegeneration, spinal cord injury, or amputations that cause
movement impairment (Anderson, 2004; Foundation, 2009). The eventual goal is to not only be
able to control external prostheses, but to use stimulation to restore movement of the patient’s own
musculature (Ethier & Miller, 2014; Ethier et al., 2012; Schearer et al., 2014) and to write sensory
information from the robotic arm (or the real arm, if sensory afferents are injured) into the brain
(Bensmaia & Miller, 2014; Dadarlat et al., 2014; O’Doherty et al., 2011; Tabot et al., 2013). In
other words, the long-term vision for motor neural prostheses is to restore near-natural function by
bi-directionally bypassing the injury.
The underlying operational principle of these systems is to identify a mapping between recorded
neural activity and desired movement intention such that when the neural correlates of a particular
movement intention are detected, the appropriate control signal is output to the effector. This
operation is called decoding (Kao et al., 2014) and forms one of the three key neuroscientific and
engineering domains of the broader neural prosthetics endeavor. It is in this domain that I have
focused my work as a neuroscientist and neural engineer. The other two core challenges — acquisition
1
CHAPTER 1. INTRODUCTION 2
Chronically Implanted Sensor
Neural Actvitiy
Sensory Feedback
Prosthesis
Decoded
Movement
Command
Figure 1.1: Schematic of a motor neural prosthesisIn paralyzed or amputee patients, there is damage somewhere along the normal efferent pathwaybetween the brain and muscles. Nonetheless, neural correlates of the person’s movement intentioncan be detected using implanted sensors (here, one 96-electrode BlackRock Microsystems array isshown). This activity is then decoded and transformed into a control signal sent to an effector suchas a robotic arm. The neural prosthesis is a closed-loop system: the user receives sensory feedbackabout the position and movements of the prosthesis.
CHAPTER 1. INTRODUCTION 3
of the neural signals in the first place, and design of the output effector — are the subject of
focused research and development efforts by neurosurgeons, neurologists, materials scientists, bio-
and electrical engineers, prosthetists and roboticists. The scope of motor neural prosthetics is large
and inherently lends itself to multidisciplinary collaboration. This is well reviewed in (Ethier et al.,
2015; Gilja et al., 2011; Green & Kalaska, 2011; Hatsopoulos & Donoghue, 2009; Homer et al., 2013;
Nicolelis & Lebedev, 2009; Schwartz, 2004; Tsu et al., 2015).
Although there are a variety of neural signals that can potentially be decoded from both within
and outside the brain (Andersen et al., 2004; Moran, 2010), to date the highest-performing BMIs
have been driven by decoded action potentials (‘spikes’) and local field potential signals recorded
from within cortex (reviewed in Gilja et al., 2012; Stavisky et al., 2015a). Given the technology
currently available and on the horizon, I believe that accurate control of complex effectors such as
prosthetic arms will only be possible with intracortical systems, and this perspective has directed my
training and research. That said, since intracortical recording necessitates the non-zero risk and cost
of neurosurgery, I would love to be proved wrong.
Although I have motivated the development of BMIs based on their potential medical value,
they have a closely related, but distinct, purpose as a tool for basic motor neuroscience. In a
BMI, the relationship between neural activity and the effector that the brain is controlling (e.g.,
computer cursor) are completely defined by the experimenter. This provides a uniquely rigorous
and parameterized technique for studying how the motor system learns to alter the activity of
BMI-connected neurons in order to generate particular patterns of output for accomplishing a given
task (e.g. Fetz, 1969; Jarosiewicz et al., 2008; Orsborn et al., 2014; Sadtler et al., 2014). My Chapter
4 exemplifies the use of BMIs to study motor neuroscience.
1.2 Key challenges to clinically viable motor neural prostheses
The key challenges that need to be overcome in order for BMIs to become clinically useful motor
prostheses differ depending on the degree of function being restored. Here I will review two sets of
challenges that directly motivated my dissertation research.
Two-dimensional cursor control has recently reached high levels of peak performance in pre-clinical
studies (Gilja et al., 2012; Nuyujukian et al., 2015), but translation to human participants is often
hampered by unreliable performance due to instability of the neural signals from which motor
intentions are decoded These changing recording conditions consist of both moment-by-moment and
day-to-day changes in the recorded spike signals (Aflalo et al., 2015; Perge et al., 2013, 2014; Simeral
et al., 2011; Wodlinger et al., 2015), as well as longer-term degradation of the chronically implanted
electrodes’ ability to record spikes (Barrese et al., 2013; Chestek et al., 2011; Kruger et al., 2010;
Simeral et al., 2011). The former problem interrupts normal use of the BMI and necessitates decoder
recalibration, whereas the latter shortens the useful lifespan of the device. To be clinically useful,
CHAPTER 1. INTRODUCTION 4
motor neural prostheses will need to be more robust both to short-term and long-term neural input
changes. Only then will they provide the reliable, long-lasting benefit necessary to justify their risks.
Recent human studies such as Gilja et al. 2015 and Jarosiewicz et al. 2015 have achieved improved
performance and robustness by adopting and extending methods developed in non-human primates;
these studies demonstrate the utility of working on these problems in an animal model to generate a
pipeline for further translational improvements.
BMI-driven control of robotic arms remains poor both in humans (Hochberg et al., 2012; Wodlinger
et al., 2015) and pre-clinical monkey models (Velliste et al., 2008) because we lack fundamental
understanding of what neural signals mean when a user is trying to control a BMI arm. Factors such
as intended muscle forces (Ethier & Miller, 2014; Evarts et al., 1983; Flint et al., 2012a; Lillicrap
& Scott, 2013; Todorov, 2000), actual and anticipated sensory feedback (Archambault et al., 2011;
Georgopoulos et al., 1983; Omrani et al., 2014; Pruszynski et al., 2014; Shadmehr et al., 2010),
grasp-related activity (Castiello, 2005; Pohlmeyer et al., 2007; Vargas-Irwin et al., 2010, 2015), and
changing expectations about the physical dynamics of the arm (Shadmehr & Mussa-Ivaldi, 1994;
Wolpert et al., 1995) all change moment-by-moment during the complex process of moving a physical
arm that is interacting with other objects. This hampers our ability to accurately decode arm control
intention. Further progress in BMI capabilities (e.g. dexterous reach and grasp) will require greater
scientific understanding of how the brain controls a BMI and incorporates it into its motor schema
and how to best decode prosthesis control signals amidst these many intermixed neural signals.
1.3 Dissertation overview
Chapter 2 addresses the BMI robustness problem due to fluctuating spike input signals. It describes
a machine learning method in which a recurrent neural network decoder is trained to exploit the large
quantity of data generated by a chronic BMI. These data, which have heretofore gone unused, include
a variety of previously encountered neural recording conditions. Our intuition was that a decoder
which has learned a variety of neural-to-kinematic mappings will be more robust to encountering
similar recording conditions in the future. We experimentally tested this novel decoding strategy and
showed that it is indeed more robust than a current state-of-the-art decoder to two different forms of
recording condition changes.
This work is currently under review, with the authors listed as David Sussillo*, Sergey D. Stavisky*,
Jonathan C. Kao*, Stephen I. Ryu, and Krishna V. Shenoy, where the asterisk represents equal
contribution. J.C.K and I previously presented it at the 2014 Society for Neuroscience annual
meeting (Kao,* Stavisky*, Sussillo*, et al., 2014). As co-equal lead author, my contributions included
co-conceiving the study, helping develop the closed-loop MRNN decoder, performing the experiments
with the help of J.C.K. and D.S., and writing the manuscript with help from J.C.K. and D.S. Professor
Shenoy provided mentorship throughout the process.
CHAPTER 1. INTRODUCTION 5
Chapter 3 also addresses BMI robustness, but focuses on how to render the BMI robust to
longer-term loss of the sensor’s ability to record spike signals. To do so, I developed methods for
decoding local field potentials as a longer-lasting complimentary and alternative BMI control signal.
When tested in closed-loop, these techniques resulted in the highest performance to date of a local
field potential-driven BMI, as well as the first closed-loop BMI controlled using spikes and LFPs
together. This work has been published in the Journal of Neural Engineering, with the title “A
high performing brain-machine interface driven by low-frequency local field potentials alone and
together with spikes”, by Sergey D. Stavisky, Jonathan C. Kao, Paul Nuyujukian, Stephen I. Ryu,
and Krishna V. Shenoy (Stavisky et al., 2015a). Prior to that, I presented it at the 2014 IEEE
Engineering in Medicine and Biology Society Conference, which was published as a peer-reviewed
conference proceeding (Stavisky et al., 2014). As lead author, I conceived the study, performed all
analyses, conducted the experiments with the help of J.C.K., and wrote the paper. Professor Shenoy
provided mentorship throughout the process.
Chapter 4 lays the groundwork for robust closed-loop control of prosthetic limbs by studying the
neuroscience of how sensory feedback related to perturbation of the BMI effector effects motor cortical
activity and ongoing BMI control. This feedback presents a potential problem because it could affect
the decoded output commands, resulting in unwanted prosthesis movement due to a deleterious
feedback loop. Anticipating this issue, I designed an experiment that captured the essential elements
of perturbing a BMI in a simpler, more controlled 2D cursor control experiment: I applied a step
perturbation to the position of a BMI cursor and measured how the neural response affected the BMI.
I found that although visual feedback did indeed produce early error-related motor cortical responses,
this initial modulation was limited to ‘output-null’ neural dimensions (as defined in Kaufman et al.
Nat Neurosci 2014) which did not cause deleterious feedback into the BMI output. Only later did
the neural activity enter ‘output-potent’ dimensions that led to the correct BMI output.
This work is currently being prepared for submission, and will have the authors listed as Sergey D.
Stavisky, Jonathan C. Kao, Stephen I Ryu, and Krishna V. Shenoy. I presented a preliminary report
of a similar task at the 2015 IEEE Neural Engineering and Repair meeting, which was published
as a peer-reviewed conference paper (Stavisky et al., 2015b). As the lead author of this study, my
contributions included conceiving the study, implementing the task, co-conducting the experiments
with J.C.K, analyzing the data, and creating the written description of the work that is presented
here. Professor Shenoy provided mentorship throughout the process.
1.4 Animal model and general experimental setup
All three of these research projects were performed using a non-human primate pre-clinical animal
model. Rhesus macaques have a close motor and sensory system homology to humans and are
CHAPTER 1. INTRODUCTION 6
D
V
AP
Primary Motor Cortex (M1)
Dorsal Premotor
Cortex (PMd)
Figure 1.2: Rhesus macaquemotor cortical regionsrecorded fromI recorded from twoclosely-related motor corticalregions, the primary motorcortex (M1) and the dorsalpremotor cortex (PMd) using96-electrode arrays. A =Anterior, P = Posterior, D =Dorsal, V = Ventral.
the best animal model currently available for studying the motor system (Churchland, 2015) and
developing neural prostheses rapidly without endangering humans (Nuyujukian et al., 2011). These
monkeys are capable of learning complex motor tasks, capable of precise repeated behavior, and have
brains large enough to implant the same sensors that are currently used in human BMI clinical trials
(Barrese et al., 2013; Homer et al., 2013). As such, able-bodied monkeys provide the best available
compromise between the unethical use of human research subjects in early research, and the ethical
and veterinary care difficulties posed by a paralyzed monkey model. Still, unlike the target patient
population that this research has in mind for its long-term impact, our monkeys were able-bodied.
This is an important difference that imposes additional uncertainty on how well pre-clinical findings
will translate, but I note that human BMI studies in which participants imagined making movements
found similar patterns of cortical activity to that of able-bodied monkeys reaching (Hochberg et al.,
2006; Aflalo et al., 2015; Gilja et al., 2015; Pandarinath et al., 2015).
In all three projects, BMI control was driven by neural activity from two motor-related brain
areas: the arm/hand related areas of the primary motor cortex (M1) and dorsal premotor cortex
(PMd), diagrammed in Figure 1.2.
Neural activity in these areas is strongly related to movement preparation (Tanji & Evarts, 1976;
Bastian et al., 1998; Crammond & Kalaska, 2000; Churchland et al., 2012) and generation (Evarts,
1968; Kalaska & Drew, 1993; Kandel et al., 2000; Rathelot & Strick, 2009). M1 and PMd neurons
exhibit strong correlations with both kinematics (Georgopoulos et al., 1988; Kalaska et al., 1989;
Moran & Schwartz, 1999) and kinetics (Cabel et al., 2001; Evarts, 1968; Kalaska et al., 1989; Morrow
& Miller, 2003; Suminski et al., 2015) of movement. This richness of movement-related activity,
combined with the surgical accessibility of their surface location, has made M1 and PMd common
sites for recording control signals for motor neural prostheses (e.g. Taylor et al., 2002; Schwartz, 2004;
Ganguly & Carmena, 2009; Hatsopoulos & Donoghue, 2009; Hochberg et al., 2006; Santhanam et al.,
CHAPTER 1. INTRODUCTION 7
2006; Velliste et al., 2008). That said, there are ongoing investigations of decoding neural activity
from other regions (e.g. Andersen et al., 2004; Musallam et al., 2004; Pasley et al., 2012; Graf &
Andersen, 2014; Aflalo et al., 2015).
Although there are distinctions between M1 and PMd, most notably a gradient of increasingly
more motor preparation as one records in more anterior sites, within the cortical areas I recorded
from these differences are more of degree than a stark qualitative difference (Schwartz et al., 1988;
Riehle & Requin, 1989; Wise et al., 1998; Crammond & Kalaska, 1996; Messier & Kalaska, 2000). For
the purposes of the studies described in this dissertation, I did not explicitly distinguish between M1
and PMd activity. This does not preclude there being interesting anterior vs. posterior differences
worthy of future investigation, particularly for the sensory feedback-related analyses (Chapter 4).
Chapter 2
Harnessing months of previously
collected data to improve
brain-machine interface robustness
to future neural variability
2.1 Summary
A major hurdle to clinical translation of brain-machine interfaces (BMIs) is that current decoders,
which are trained from a small quantity of recent data, are rendered ineffective when neural recording
conditions subsequently change. Meanwhile, a typically large quantity of data accumulated from
chronic BMI use has heretofore gone unused. We tested the hypothesis that a decoder trained to
handle a variety of recording conditions sampled from months of previously collected data would
be more robust to future neural variability. To do so, we developed a new BMI decoder that can
successfully learn a large variety of neural-to-kinematic mappings in a single artificial recurrent neural
network. When tested in a non-human primate preclinical BMI model, this decoder was robust under
conditions that disabled a state-of-the-art Kalman filter based decoder. These results validate a new
BMI strategy in which accumulated data history is effectively harnessed, and may facilitate reliable
daily BMI use by reducing the need for decoder retraining downtime.
8
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 9
2.2 Introduction
Brain-machine interfaces (BMIs) can restore motor function and communication to patients with
paralysis (Collinger et al., 2013b; Hochberg et al., 2006, 2012). Progress has been particularly strong
towards enabling two-dimensional computer cursor control, which may allow versatile communications
prostheses (Bacher et al., 2015; Gilja et al., 2015; Jarosiewicz et al., 2015). Cursor control performance
has approached that of the native hand in recent macaque studies (Gilja et al., 2012; Nuyujukian
et al., 2015), but this was accomplished under favorable laboratory conditions where neural recordings
are often stable both during and across BMI sessions (Ganguly & Carmena, 2009; Flint et al., 2012a;
Nuyujukian et al., 2014). In contrast to these pre-clinical studies, one of the major challenges
impeding BMI use by human clinical trial participants is the high degree of within- and across-day
variability in neural recording conditions (Simeral et al., 2011; Perge et al., 2013, 2014; Wodlinger
et al., 2015; Aflalo et al., 2015), illustrated in Figure 2.1a. I use the term ‘recording condition’ to
broadly encompass the combination of factors which together determine the relationship between
observed neural activity and intended kinematics. These factors include the relative position of the
electrodes and surrounding neurons (diagrammed in Figure 2.1b), variability in sensor properties
such as impedance or wiring quality, noise sources, and biological factors such as cognitive state or
medications. Existing neural decoding algorithms are poorly suited to handle this input variability,
resulting in intermittent performance and a need for frequent decoder retraining (Bacher et al., 2015;
Jarosiewicz et al., 2013, 2015; Perge et al., 2013).
The clinical viability of BMIs would be much improved by making decoders robust to recording
condition changes (Huggins et al., 2011; Collinger et al., 2013a), and several recent studies have
begun to focus on this problem. We can broadly divide the conditions that a BMI will encounter
into one of two types: 1) conditions that are completely different from those previously encountered,
and 2) conditions that share some commonality with those previously encountered. For existing
BMI methods, both of these situations necessitate some interruption of function while the decoder is
updated to handle the new condition. One strategy for minimizing this interruption is to use adaptive
decoders rather than collecting new training data for a de novo decoder (Dangi et al., 2013, 2014;
Bishop et al., 2014; Orsborn et al., 2014; Nuyujukian et al., 2014; Pohlmeyer et al., 2014; Shenoy &
Carmena, 2014; Jarosiewicz et al., 2015) . In principle, however, BMI interruption could be avoided
altogether in the second case by a decoder capable of exploiting the overlap between the current and
previously encountered conditions (Figure 2.1c).
We were motivated to try this complimentary strategy because chronic BMI systems do typically
encounter recording conditions in which there is some commonality with past recording conditions
(Ganguly & Carmena, 2009; Chestek et al., 2011; Flint et al., 2013; Perge et al., 2013, 2014; Bishop
et al., 2014; Orsborn et al., 2014; Nuyujukian et al., 2014; Stavisky et al., 2015a). Furthermore,
these systems generate and store months, or even years, of neural and kinematic data as part of
their routine use. Almost all of this past data is left unused in existing BMI systems: decoders are
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 10
D5D
1D
1D
2D
1D
3D
4D
1D
5D
5D
1D
1D
2D
2D
3D
1D
3D
6
?
Traditional
Decoding
day 1 day 2 previous day now
…
Decoder Trained
From Chronic
Recordings
Dall
Time
(lost)
Electrodes
Recording Condition 1 Recording Condition 2 Recording Condition 1 Recording Condition 3
(added)
(broken)
Time (min)10020 40 60 80
0
100
% T
ria
ls C
orr
ect
0
20
Firin
g R
ate
(h
z)
a
b
c
Figure 2.1: Our strategy for training a decoder robust to recording condition changesa) Example data from a BMI clinical trial showing sudden decoder failure caused by a recordingcondition change. The black trace shows the participant’s closed-loop performance over the course ofan experiment using a fixed Kalman filter. An abrupt drop in performance coincides with a reductionin the observed firing rate (red trace) of a neuron with a high decoder weight. Both the neuron’sfiring rate and decoder performance spontaneously recover 40 minutes later. Adapted from Figure 7of (Perge et al., 2013).(b) A cartoon depicting one hypothetical cause of the aforementioned change: micro-motion of theelectrodes leads to Recording Condition 2, in which spikes from the red-shaded neuron are lost.BMI recovery corresponds to a shift back to Condition 1. Over time, further changes will result inadditional recording conditions; for example, Condition 3 is shown caused by a disconnected electrodeand an additional neuron entering recording range.(c) Recording conditions (schematized by the colored rectangles) will vary over the course of chronicintracortical BMI use. We hypothesize that oftentimes new conditions are similar to ones previouslyencountered (repeated colors). Typically, decoders are fit from short blocks of training data and areonly effective under that recording condition (decoders D1,D2, ...). Consider the goal of traininga decoder for use at time “now” (black rectangle on right). Standard practice is to use decoderD1 trained from the most recently available data (e.g. from the previous day or the start of thecurrent experiment). D1 will perform poorly if the recording condition encountered differs from itstraining data. To increase the likelihood of having a decoder that will perform well given the currentrecording condition, we tested a new class of decoder, Dall, trained using a large collection of previousrecording conditions.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 11
trained using the most recently available data, typically from a block of calibration trials at the
start of that day’s experiment, or from a recent previous experiment (Homer et al., 2013). Using
this historical data would be difficult for most state-of-the-art BMI decoders, as they are linear
(e.g. Collinger et al., 2013b; Gilja et al., 2012). Although linear decoders have other advantages
(for instance, their operation is easy to understand and they can be fit quickly), they are prone
to underfitting heterogeneous training sets, such as those that might be sampled from months of
data. To overcome this limitation, an essential aspect of our approach is to use a nonlinear and
computationally powerful decoder capable of learning a diverse set of neural-to-kinematic mappings.
Specifically, we tested whether one could gain traction on the decoder robustness problem by
exploiting this idle wealth of stored data using an artificial recurrent neural network (RNN). We did
this with a three-pronged approach. The first was the aforementioned use of the nonlinear RNN.
The second was to train the decoder from many months of previously recorded data. Third, to
‘harden’ the decoder against being too reliant on any given electrode’s inputs, we artificially injected
additional variability into the data during decoder training. We did this by perturbing the training
data in severe ways, for example by zeroing or doubling the firing rates of the electrodes.
Our strategies to enhance the typical BMI training set provide a large and variable set of training
data that most traditional decoding methods are unable to handle due to their limited computational
power (Kao et al., 2014). We therefore developed a new BMI decoder using a powerful RNN
variant called the multiplicative recurrent neural network (‘MRNN’, Sutskever et al., 2011). Several
properties of an MRNN make it attractive for this application. Its recurrent connectivity leads to rich
nonlinear internal dynamics that are well-suited to decoding the time-varying, complex relationships
between neural firing rates and kinematics (Churchland & Shenoy, 2007; Sussillo et al., 2012). Its
‘multiplicative’ architecture increases computational power by allowing the neural inputs to strongly
influence the internal dynamics of the RNN by changing the connectivity weights (Figure 2.2a).
Loosely speaking, this allows the MRNN to learn a ‘library’ of different neural-to-kinematic mappings
that are contextually appropriate to the current recording conditions (the mathematical details
are described in the Methods). This general nonlinear approach is likely to work well with many
out-of-the-box RNN variants, such as a standard RNN (e.g. Sussillo et al., 2012) or LSTM (Hochreiter
& Schmidhuber, 1997) and is also completely complementary to adaptive decoding.
We evaluated decoders using two non-human primates implanted with chronic multielectrode
arrays similar to those used in ongoing clinical trials. In this chapter, I first show that the MRNN
trained with more data from previous recording sessions leads to improved accuracy when decoding
new neural data, and that a single MRNN can be trained to accurately decode hand reach velocities
across hundreds of days. I next present closed-loop results showing that an MRNN trained with
many days’ worth of data is much more robust than a state-of-the-art Kalman filter based decoder
(the Feedback Intention Trained Kalman filter ‘FIT-KF’, Fan et al., 2014) to two types of recording
condition changes likely to be encountered in clinical BMI use: the unexpected loss of signals from
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 12
1 1024
0
1
1 663
0
1
De
co
de
Accu
racy (
r2)
Recording DayRecording Day
MRNN
FIT LongFIT Sameday
0
1 ***
0
1 ***
1 14 34 47 55
0
1
50 Training Days38 Training Days25 Training Days17 Training Days11 Training Days 7 Training Days 5 Training Days 3 Training Days 2 Training Days 1 Training Day
Training Days
1 7 51 59
0
1
37 Training Days25 Training Days 17 Training Days 11 Training Days 7 Training Days 5 Training Days 3 Training Days 2 Training Days 1 Training Day
Test Day
De
co
de
Accu
racy (
r2)
Test Day
Training Days
vy(t)
vx(t)
px(t)
py(t)
b Monkey R Monkey L
c
a
Figure 2.2: An MRNN decoder can harness large training datasets(a) Schematic of the experimental setup and MRNN decoder. A monkey performed a targetacquisition task while spikes were recorded. Data from many days of this task were used to traintwo MRNNs such that velocity and position could be read out from the state of their respectiveinternal dynamics. These internal dynamics are a function of the binned spike counts; thus, theMRNN can conceptually be thought of as selecting which internal dynamics are active at any giventime based on the input signals. During closed-loop use, decoded velocity and position outputs wereblended together to update the on-screen cursor. (b) The MRNN was better prepared for futuredays’ recording conditions after being trained with more previously collected data. We evaluated itsability to reconstruct offline hand velocity after training with increasing numbers of previous days’datasets. Training data were added by looking further back in time. In monkey R the early test daysalso contributed training data (with the test trials held out). In monkey L, the training datasetsstart with the day prior to the first test day. More training data (darker colored traces) improveddecode accuracy, especially when decoding more chronologically distant recording conditions (latertest days). For comparison, I also plot performance of a traditional decoder (FIT Kalman filter)trained from each individual day’s training data (‘FIT Sameday’, light blue). (c) An MRNN couldsuccessfully learn even larger datasets spanning many more recording days. An MRNN (red trace)was trained with data from 154 (monkey R) and 250 (monkey L) recording days spanning manymonths. Its offline decoding accuracy on held-out trials was compared to that of the FIT Sameday(light blue). We also tested a single FIT-KF trained using the same large dataset as the MRNN(‘FIT Long’, dark blue). Gaps in the connecting lines denote recording gaps of more than ten days.(Insets) mean ± std decode accuracy across all recording days. The MRNN outperformed both typesof FIT-KF decoders on every day’s dataset.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 13
highly-informative electrodes, and day-to-day changes. Finally, I show that this robustness does
not come at the cost of reduced performance under more ideal (unperturbed) conditions: in the
absence of artificial challenges, the MRNN provides excellent closed-loop BMI performance and
slightly outperforms the FIT-KF. To my knowledge, this is the first attempt to improve robustness
by using a large and heterogeneous training dataset: we used roughly two orders of magnitude more
data than in previous closed-loop studies.
2.3 Results
2.3.1 An MRNN can leverage large amounts of training data to improve
decoder performance
We first tested whether training the MRNN with many days’ worth of data can improve offline
decoder performance across a range of recording conditions. This strategy was motivated by our
observation that the neural correlates of reaching — as recorded with chronic arrays — showed
day-to-day similarities (Figure 2.3). For a typical recording session, the most similar recording came
from a chronologically close day, but occasionally the most similar recording condition was found
in chronologically distant data. MRNN decoders were able to exploit these similarities: Figure
2.2b shows that as more days’ data (each consisting of 500 point to point reaches) were used to
train the decoder, the accuracy of reconstructing reach velocities of test datasets increased (positive
correlation between number of training days and decoded velocity r2, p < 0.001). Two results
from this analysis were particularly encouraging. First, using more training data substantially
increased the decode accuracy for the “hard” days that challenged decoders trained with only a few
days’ data (e.g., test day 51 for monkey R). Second, this improvement did not come at the cost of
worse performance on the initially “easy” test days. These results demonstrate that larger training
datasets better prepare the MRNN for a variety of recording conditions, and that learning to decode
additional recording conditions did not diminish the MRNN’s capability to reconstruct kinematics
under recording conditions that it had already “mastered”. There was not a performance versus
robustness trade-off. Finally, it is worth noting that after training with just one day’s data, the
MRNN’s offline performance on the next several days exceeded that of the FIT-KF trained from
held out test day reaches.
We then tested whether the MRNN’s computational capacity could be pushed even further by
training it using data from 154 (250) different days’ recording sessions from monkey R (L), which
spanned 22 (34) months (Figure 2.2c). Across all these recording sessions’ held-out test trials, the
MRNN’s offline decode accuracy was r2 = 0.81± 0.04 (mean ± std, monkey R) and r2 = 0.84± 0.03
(monkey L). For comparison, we tested the decode accuracy of the FIT-KF trained in two ways:
either specifically using reaching data from that particular day (‘FIT Sameday’), or trained on the
same large multiday training dataset (‘FIT Long’). Despite the multitude of recording conditions
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 14
1 252
252
Min
. Prin
cipa
l Ang
le (°
)
3
36Recording Session #1 154
154
Recording Session #
Min
. Prin
cipa
l Ang
le (°
)
2
Rec
ordi
ng S
essi
on #
22
Monkey R Monkey L
b
c
a
1 1024Recording Day 3
36
1 663Recording Day 2
22
Min
. Prin
cipa
l Ang
le (°
)
sess
ion
#2
sess
ion
#52
sess
ion
#103
sess
ion
#153
sess
ion
#2
sess
ion
#84
sess
ion
#168
sess
ion
#251
1 50 100 150
78%
10%
Number of Recording Sessions Apart1 20 40 60 80 100 120O
ccur
ence
Fre
quen
cy
Number of Recording Sessions Apart
62%
14%
Figure 2.3: Similar neural recording conditions can often be found on other daysChronologically close days tend to have more similar neural recordings, but for a given day there areoccasional similar recordings from more distant days. (a) Between-day variability of the structureof neural activity recorded during center-out 8-target reaches over the course of many months (154recording sessions over a 663 day period in monkey R, and 252 sessions spanning 1024 days in monkeyL; these sessions correspond to the same data as in 2.2c). The color at the intersection of row iand column j corresponds to how differently the observed neural activity covaried during recordingsessions i and j. Specifically, we have plotted the minimum principal angle between subspacesspanned by the top 10 eigenvectors of each day’s mean-activity-subtracted covariance matrix (seeMethods). These 10 eigenvectors captured on average 51 (46)% of single-trial variance for monkeysR (L). Sharp “block” structure transitions typically correspond to a long (many weeks’) intervalbetween consecutive recording sessions. (b) Four slices through each monkey’s principal angle matrixshow that for these example days, there were similar recording conditions on other days both nearand far in time. Each color series shows the minimum principal angle between every recording day’sdata and the reference day marked with the arrow of this color in panel a. Note that the horizontalaxis, which spans the same time range as in panel a, is now in units of (calendar) days rather thansession number. Each series’ reference day is marked with a vertical line. (c) Histograms showing thedistribution, across each monkey’s recordings, of how many recording sessions away (either forwardor back in time) we observed the most similar neural correlates of reaching as measured by minimumprincipal angle.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 15
that the MRNN had to learn, on each and every test day each monkey’s single MRNN outperformed
that day’s FIT Sameday filter (monkey R: FIT-Sameday r2 = 0.57 ± 0.05, p < 0.001 paired t-test
comparing FIT-Sameday to MRNN; monkey L: r2 = 0.52 ± 0.05, p < 0.001). Unsurprisingly, a
linear FIT-KF did not benefit from being trained with the same large multiday training set and
also performed worse than the MRNN (monkey R: FIT Long r2 = 0.56, p < 0.001 for a paired t-test
comparing FIT Long to MRNN; monkey L: r2 = 0.46± 0.05, p < 0.001).
2.3.2 Robustness to unexpected loss of the most informative electrodes
We next performed closed-loop BMI cursor control experiments to test the MRNN’s robustness to
recording condition changes. The first set of experiments challenged the decoder with an unexpected
loss of inputs from multiple electrodes. The MRNN was trained with a large corpus of hand reaching
training data up through the previous day’s session (119 - 129 training days for monkey R, 212 - 230
days for monkey L). Then, its closed-loop performance was evaluated on a Radial 8 Task while the
selected electrodes’ input firing rates were artificially set to zero. By changing how many of the most
informative electrodes were dropped (“informative” as determined by their mutual information with
reach direction; see Methods), we could systematically vary the severity of the challenge. Since this
experiment was meant to simulate sudden failure of electrodes during BMI use (after the decoder
had already been trained), we did not retrain or otherwise modify the decoder based on knowledge
of which electrodes were dropped. There were no prior instances of these dropped electrode sets
having zero firing rates in the repository of previously collected training data (Figure 2.5). Thus,
this scenario is an example of an unfamiliar recording condition (zero firing rates on the dropped
electrodes) having commonality with a previously encountered condition (the patterns of activity on
the remaining electrodes).
We found that the MRNN was robust to severe electrode dropping challenges. It suffered only a
modest loss of performance after losing up to the best 3 (monkey R) or 5 (monkey L) electrodes
(Figure 2.4). We compared this to the electrode-dropped performance of a FIT-KF decoder trained
with calibration data from the beginning of that day’s experiment according to previously reported
best practices (Gilja et al., 2012; Fan et al., 2014). This FIT Sameday decoder’s performance worsened
dramatically when faced with this challenge. Across all electrodes-dropped conditions, Monkey R
acquired 52% more targets per minute using the MRNN, while Monkey L acquired 92% more targets.
Although the past datasets used to train the MRNN never had these specific sets of highly
important electrodes disabled, our technique of artificially perturbing the true neural activity during
MRNN training did generate training examples with reduced firing rates on various electrodes (as
well as examples with increased firing rates). The MRNN had therefore been broadly trained to be
robust to firing rate reduction on subsets of its inputs. Subsequent closed-loop comparisons of MRNN
electrode dropping performance with and without this training technique (Figure 2.7) confirmed its
importance.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 16
Monkey R Monkey L
Top N Electrodes Dropped0 3 5 7 10
30
0 5/8 8/11 5/7 5/5 1/43/110/8 0/7 0/5 0/4
Top N Electrodes Dropped
Targ
ets
Per M
inut
e
0 2 3 5 7 10
0
30
6/9 7/7 6/7 2/7 1/7 1/71/9 1/7 1/7 1/70/7 0/7
**
* ***
Figure 2.4: Robustness to unexpected loss of the most important electrodesClosed-loop BMI performance using the MRNN (red) and FIT Sameday (blue) decoders whilesimulating an unexpected loss of 0 to 10 electrodes by setting the firing rates of these electrodes tozero. The mean and SEM of performance across experimental sessions is shown for each decoderas a function of how many electrodes were removed. Stars denote conditions for which the MRNNsignificantly outperformed FIT Sameday across sessions (p < 0.05, paired t-test). The fractionsabove each condition specify for how many of the individual evaluation days each decoder performedsignificantly better according to success rate and time to target. Electrode-dropping order wasdetermined by the mutual information between that electrode’s spike count and target directionduring arm-controlled reaches.
2.3.3 Robustness to naturally occurring recording condition changes
The second set of closed-loop robustness experiments challenged the MRNN with naturally occurring
day-to-day recording condition changes. In contrast to the highly variable recording conditions
encountered in human BMI clinical trials, neural recordings in our laboratory setup are typically
quite stable on the timescale of days (Figure 2.3; Nuyujukian et al., 2014). Therefore, in order to
challenge the MRNN and FIT-KF decoders with greater recording condition variability, we evaluated
them after withholding the most recent several months of recordings from the training data. I refer
to this many month interval between the most recent training data day and the first test day as the
training data “gap” in these ‘stale training data’ experiments. We repeated the experiment with two
different training data gaps for each monkey.
For each gap, we trained the MRNN with a large dataset consisting of many months of recordings
preceding the gap and compared it to two different types of FIT-KF decoders. The ‘FIT Old’ decoder
was trained from the most recent available training day (i.e. the day immediately preceding the gap);
this approach was motivated under the assumption that the most recent data were most likely to
be similar to the current day’s recording condition. The ‘FIT Long’ decoder was trained from the
same multiday dataset used to train the MRNN and served as a comparison in which a conventional
decoder is provided with the same quantity of data as the MRNN. The logic underlying this FIT
Long approach was that despite the Kalman filter being ill-suited for fitting multiple heterogeneous
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 17
Recording Session #
Ele
ctr
od
e
20 40 60 80 100 120 140 160 180 212 231
20
40
60
80
10
20
30
40
50
60
Recording Session #
Ele
ctr
od
e
20 40 60 80 100 129
20
40
60
80
100
120
140
160
10
20
30
40
0 0
1 1
192
96
ExperimentsPrevious Training Data
Ave
rag
e F
irin
g R
ate
(H
z)
Ave
rag
e F
irin
g R
ate
(H
z)
119
ExperimentsPrevious Training Data
Monkey R
Monkey L
Dropped
Electrode
Figure 2.5: Artificially dropped electrodes were active in the training dataThese plots show each electrode’s average firing rate during each dataset used to train theMRNN; electrodes are ordered by descending average firing rate across all recording sessions.Recording sessions numbered in gray were only used for training data. The electrode droppingexperiments (Figure 2.4) were conducted during the sessions numbered in black. Zero firing rates (i.e.non-functional electrodes) are shown in purple for emphasis, while electrodes selected for dropping ona particular day are shown in red (note that although on a given test session we evaluated differentnumbers of electrodes dropped, this plot shows each day’s broadest dropped set). These droppedelectrodes rarely recorded zero firing rates in the training data sessions, and the specific sets ofdropped electrodes used to challenge the decoders never all had zero firing rates in the training data.
datasets, this “averaged” decoder might still perform better than the FIT-KF Old trained using a
single distant day.
We found that the MRNN was the only decoder that was reliably usable when trained with stale
data (Figure 2.6). FIT Old performed very poorly in both monkeys, failing completely in 4/6 monkey
R experimental sessions and 6/6 monkey L sessions. Further tests of additional FIT Old decoders
confirmed that they generally perform poorly (Figure 2.8). FIT Long performed better than FIT
Old, but its performance was highly variable — it was usable on some test days but not on others.
In Monkey R, the across-days average acquisition rate was 105% higher for the MRNN than FIT Old
(p < 0.01, paired t-test). Monkey L’s MRNN did not perform as consistently well as Monkey R’s, but
nevertheless demonstrated a trend of outperforming FIT Long (32% improvement, not significant).
Although monkey L’s FIT Long outperformed the MRNN on one test day, on all other test days FIT
Long was either similar to, or substantially worse than, MRNN. Moreover, whereas the MRNN was
controllable every day, FIT Long was not even capable of acquiring targets on some days. The lack
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 18
Targ
ets
Per M
inut
e
0
30
FIT Old
MRNNFIT Long
Decoder Set 2 Training Data
FIT Old
MRNNFIT Long
Decoder Set 1 Training Data
706 810 964Day 1 Set 1 Test
{ {
Set 2 Test 384 447 558Day 1
0
30
Targ
ets
Per M
inut
e
Arm Control
FIT Sameday
FIT Old
MRNNFIT Long
Decoder Set 2 Training Data
FIT Old
MRNNFIT Long
Decoder Set 1 Training Data
Set 1 Test{ {
Set 2 Test
Monkey R Monkey L
Figure 2.6: Robustness to naturally occurring recording condition changesRobustness to a natural sampling of neural input variability. We created decoder evaluation conditionsin which the neural inputs were likely to be different from much of the training data by withholdingaccess to the most recent several months of data. Each circle corresponds to the mean closed-loopBMI performance using these “stale” MRNN (red), FIT Long (dark blue), and FIT Old (teal)decoders when evaluated on six different experiment days spanning 7 (13) days in monkey R (L).Each test day, these three decoders, as well as a FIT Sameday decoder trained from that day’s armreaches, were evaluated in an interleaved block design. The legend bars also denote the time periodsfrom which training data for each stale decoder came from. We repeated the experiments for asecond set of decoders to reduce the chance that the results were particular to the specific trainingdata gap chosen. The training data periods contained 82 and 92 datasets (monkey R), and 189 and200 training datasets (monkey L). Both monkeys were only able to consistently perform the taskusing the MRNN decoder. To aid the interpretation of these stale decoder performances, we showthe average performance across the six experiment days using arm control (gray dashed line) or aFIT Sameday decoder (blue dashed line).
of consistent usability by any of the FIT-KF decoders demonstrates that having access to a large
repository of stale training data does not enable training a single Kalman filter that is robust to
day-to-day variability in recording conditions. In contrast, an MRNN trained with this large dataset
was consistently usable.
2.3.4 High performance BMI using the MRNN decoder
Finally, we asked whether the MRNN’s robustness to challenging recording conditions came at the
cost of reduced performance under more ideal conditions. We evaluated the MRNN’s closed-loop
performance when trained using several months’ data up through the previous day, without electrode
dropping or any other artificial challenge. In this scenario, the MRNN enabled both monkeys to
accurately and quickly control the cursor. We compared the MRNN’s performance to that of a
FIT-KF Sameday decoder. Figure 2.9a shows representative cursor trajectories using each decoder,
as well as under hand control. The MRNN outperformed the FIT-KF in both monkeys (Figure 2.9b).
Across nine experimental sessions and 4,000+ trials with each decoder, Monkey R acquired targets
7.3% faster with the MRNN (619 ± 324 ms mean ± std vs. 668 ± 469 ms, p < 0.001, two-sided
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 19
Top N Electrodes Dropped0 3 5 7 10
0
30
Top N Electrodes Dropped
Ta
rge
ts P
er
Min
ute
0 2 3 5 7 10
0
30Monkey R Monkey La
b
Block
400
800
1200
1600
Monkey L
0Block
200
400
600
800
0Block
0
200
400
600
800
Tim
e to
Ta
rge
t (m
s)
Monkey R
Training Set 1 Training Set 1Training Set 2
Figure 2.7: Training data spike rate perturbations improve MRNN performance(a) Robustness to electrode dropping. We evaluated the closed-loop BMI performance of the MRNNdecoder trained with (red) and without (brown) the training data spike count perturbations. Bothdecoders were evaluated on the same days with firing rates on the top 0 to 10 electrodes set to zero(similar to Figure 2.4). Each circle corresponds to a decoder’s mean performance on a given evaluationday. Filled circles denote days where there was a significant performance difference (p < 0.05 binomialtest on success rate, followed, if success rate not significantly different, by two-sample t-test of timesto target) between the two decoders tested. The line shows the grand mean performance for eachdecoder. The MRNN trained with perturbed firing rates consistently outperformed the MRNNtrained without these training data perturbations when encountering electrode-dropped neural input.(b) Stale training data experiments. MRNNs were trained without access to recent training data, asin Figure 2.6, either with (red) or without (brown) training data spike rate perturbations. We traineddecoders from both of monkey R’s stale training data periods and from monkey L’s longer staletraining data period. Closed-loop BMI performance using these decoders was then compared on thesame evaluation day in alternating blocks. The plot shows the mean time to target for each block. Forall three pairs of decoders compared, the MRNN with spike rate perturbations had significantly fastertimes to target (p < 0.05, two-sample t-test aggregating trials across blocks). Datasets R.2014.03.21& L.2014.04.04.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 20
Days Between Training Data and Decoder Test155 272 965 978
0
35
Days Between Training Data and Decoder Test
Ta
rge
ts P
er
Min
ute
31 559
0
35 Monkey R Monkey L
FIT O
ld fr
om 2
014-
02-0
3
FIT O
ld fr
om 2
013-
10-2
2
FIT O
ld fr
om 2
013-
07-1
0
FIT O
ld fr
om 2
011-
08-0
4
FIT O
ld fr
om 2
013-
10-0
9
FIT O
ld fr
om 2
013-
11-2
1
FIT O
ld fr
om 2
013-
09-1
9
FIT O
ld fr
om 2
013-
04-1
6
FIT O
ld fr
om 2
012-
09-2
1
Figure 2.8: Additional tests showing that FIT Old typically performs poorlyWe investigated whether our result that three of the four different FIT Old decoders tested in themain stale training data experiments (Figure 2.6) failed was due to a particularly unlucky choiceof FIT Olds. To better sample the closed-loop performance of FIT-KF decoders trained usingold training data, we trained FIT Old decoders from 3 (monkey R) and 2 (Monkey L) additionalarbitrarily chosen arm reaching datasets from the monkey’s prior experiments. We evaluated all 5(4) FIT Old decoders on a number of additional days over the course of the current study (8 totaltest days for monkey R, 13 total test days for monkey L). Each point shows the performance of aparticular FIT Old decoder on one test day. Different days’ evaluations of the same FIT Old decoderare shown in the same color. Black borders denote data points and black underlines denote decodersthat are shared with Figure 2.6.
t-test). Monkey L acquired targets 10.8% faster with the MRNN (743 ± 390 ms vs. 833 ± 532 ms,
p < 0.001, two-sided t-test) across 8 sessions and 2,500+ trials using each decoder. These online
results corroborate the offline results presented in Figure 2.2c; both show that an MRNN trained
from many days’ recording conditions outperforms the current state-of-the-art FIT Kalman filter
trained from training data collected at the start of the experimental session.
A potential risk inherent to a computationally powerful decoder such as the MRNN is that
it will overtrain to the task structure of the training data and fail to generalize to other tasks.
Most of our MRNN training data were from arm reaches on a Radial 8 Task similar to the task
used for evaluation (albeit with 50% further target distance). We therefore also verified that the
MRNN enabled good cursor control on the Random Target Task, in which the target could appear
in any location in a 20× 20 cm workspace (Figure 2.10). Monkey R performed the Random Target
Task on two experimental sessions and averaged a 99.4% success rate over 649 trials, with mean
distance-normalized time to target of 0.068 s/cm. Monkey L performed one session of this task at a
100% success rate across 207 trials with mean normalized time to target of 0.075 s/cm.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 21
0 250 500 750
Trial Time (ms)
0
4
8
0 250 500 750
0
4
8
Trial Time (ms)
Dis
tan
ce
to
Ta
rge
t (c
m)
Monkey R Monkey L
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
1
2
3
4
56
7
8
4 cm
Arm Control MRNN FIT Sameday
b
a
Figure 2.9: The MRNN outperforms the FIT Kalman filter under “ideal” conditions(a) We compared cursor control using the MRNN (red) trained from many datasets up throughthe previous day to the FIT-KF (blue) trained from data collected earlier the same day, withoutany artificial challenges (i.e. no electrodes dropped). Cursor trajectories are shown for eightrepresentative and consecutive center-out-and-back trials of the Radial 8 Task. Gray boxes showthe target acquisition area boundaries, and the order of target presentation is denoted with greennumbers. For comparison, cursor control using the hand is shown in gray. From dataset R.2014.04.03.(b) Mean distance to target, during the Radial 8 Task, as a function of trial time using each cursorcontrol mode. Thickened portions of each trace correspond to ‘dial-in time’, i.e. the mean timebetween the first target acquisition and the final target acquisition. The MRNN and FIT Samedaydata corresponds to the drop 0 electrodes condition in Figure 2.4.
2.4 Discussion
We developed the MRNN decoder to help address a major problem hindering the clinical translation
of BMIs: once trained, decoders can be quickly rendered ineffective due to recording condition
changes. A number of complementary lines of research are aimed at making BMIs more robust,
including improving sensors to record from more neurons more reliably (e.g. Seo et al., 2014); decoding
multiunit spikes (Fraser et al., 2009; Chestek et al., 2011; Nuyujukian et al., 2014) or local field
potentials (Flint et al., 2013; So et al., 2014; Stavisky et al., 2015a) which appear to be more stable
control signals than single unit activity; and using adaptive decoders that update their parameters
to follow changing neural-to-kinematic mappings (Wu & Hatsopoulos, 2008; Dangi et al., 2013, 2014;
Jarosiewicz et al., 2013, 2015; Bishop et al., 2014; Orsborn et al., 2014; Pohlmeyer et al., 2014; Shenoy
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 22
200
1100
20002001-5000
Tim
e to
ta
rge
t (m
s)
fail4 cm
Time to target (ms)0 1000 2000 3000 4000 fail
# T
ria
ls0
20
40
60
80 median = 692 ms
Time to target (ms)0 1000 2000 3000 4000 fail
# T
ria
ls
0
100
200
300 median = 562 ms
Monkey R Monkey La
b
Figure 2.10: Closed-loop MRNN decoder performance on the Random Target TaskBoth monkeys were able to use this decoder to acquire targets across a broad workspace in whichtargets often appeared at locations that differed from the target locations dominating the trainingdatasets.(a) Histograms of times to target (time of final target entry minus target onset time, not includingthe 500 ms target hold period) for both monkeys.(b) Task workspace plots showing the location of each Random Target Task trial’s target. Eachpoint corresponds to the center of one trial’s target, and its color represents the time it took themonkey to acquire this target. The location of the one failed trial (for monkey R) is shown with ablack ‘x’. The acquisition area boundaries of the nine Radial 8 Task targets used for the majorityof the training data are shown as black squares. Monkey R’s data are aggregated across the twoexperimental sessions in which he performed this task. Monkey L’s data are from one session.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 23
& Carmena, 2014). Here we’ve taken the approach of building a fixed decoder that is trained to
be inherently robust to recording condition changes it has previously encountered as well as new
conditions that have some similarity to a previously encountered condition. I stress that these
approaches are complementary in several respects. For example, a decoder that is inherently more
robust to neural signal changes, such as the MRNN, would still benefit from improved sensors and
decoder adaptation, and could operate on a mix of input signal types including single- and multiunit
spikes and field potentials.
2.4.1 A robust and high-performing BMI decoder
We demonstrated the MRNN’s robustness to two types of recording condition changes. These changes
were chosen because they capture key aspects of the changes that commonly challenge BMI decoders
during clinical use. The stale training data experiments showed that the MRNN was usable under
conditions where the passage of time would typically require recalibration of conventional decoders
such as the FIT-KF. I do not mean to suggest that in a clinical setting one would want to — or
would often have to – use a BMI without any training data from the past several months. Rather, we
used this experimental design to model recording condition changes that can happen on the timescale
of hours in human BMI clinical trials (Perge et al., 2013). Possible reasons for the greater recording
condition variability observed in human participants compared to non-human primates include:
more movement of the array relative to the human brain due to larger cardiovascular pulsations
and epidural space; greater variability in the state of the BMI user (health, medications, fatigue,
cognitive state); and more electromagnetic interference from the environment. The MRNN can
take advantage of having seen the effects of these sources of variability in previously accumulated
data; it can therefore be expected to become more robust over time as it builds up a “library” of
neural-to-kinematic mappings under different recording conditions.
The electrode dropping experiments, which demonstrated the MRNN’s robustness to an unexpected
loss of high-importance electrodes, are important for two reasons. Firstly, sudden loss of input
signals, for instance due to a electrode connection failure (Gold et al., 2006; Barrese et al., 2013),
is a common BMI failure mode that can be particularly deleterious to conventional BMI decoders
(Kao et al., 2013). The MRNN demonstrates considerable progress in addressing this so-called
“errant unit” problem. Secondly, these results demonstrate that the MRNN trained with artificially
perturbed neural data can be relatively robust even to a recording condition change that has not
been encountered in past recordings.
The MRNN’s robustness did not come at the cost of diminished performance under more ideal
conditions. This result is nontrivial given the robustness-focused decisions that went into its design
(e.g. perturbing the input spike trains in the training set). Instead, we found that the MRNN was
excellent under favorable conditions, outperforming a state-of-the-art same day trained FIT-KF
decoder. Taken together, these results demonstrate that the MRNN exhibits robustness to a variety of
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 24
clinically relevant recording condition changes, without sacrificing peak performance. These advances
may help to reduce the onerous need for clinical BMI users to collect frequent retraining data.
One disadvantage of this class of nonlinear decoders trained from large datasets, when compared
to traditional linear decoders trained on smaller datasets, is the longer training time required to
fit their parameters. In the present study, which we did not optimize for fast training, this took
multiple hours. This could be substantially sped up by iteratively updating the decoder with new
data instead of retraining de novo and by leveraging faster computation available with e.g. graphics
processing units, cloud computing, or custom hardware.
2.4.2 Key innovations enabling the MRNN’s robustness
The MRNN decoder’s robustness was due to the combination of a large training data corpus,
deliberate perturbation of the training data, and a computationally powerful architecture that was
able to effectively learn this diverse training data. While it may seem obvious that a decoder that
successfully learns more training data is better, this is not necessarily true. Older data only helps if
some of these past recordings capture neural-to-kinematic relationships that are similar to that of the
current recording condition. Our offline and closed-loop MRNN robustness results suggest that this
was indeed the case for the two monkeys used in this study. While there are indications that this
will also be true in human BMI studies (Perge et al., 2014), validating this remains an important
future question. The relevance of old data to present recording conditions also enables a different
robustness-enhancing approach: store a library of different past decoders and evaluate each to find a
decoder well-suited for the current conditions (e.g. Nuyujukian et al., 2014). However, since offline
analyses are poor predictors of closed-loop performance (Fraser et al., 2009; Cunningham et al., 2010;
Koyama et al., 2010; Stavisky et al., 2015a), this approach necessitates a potentially lengthy decoder
selection process. Using a single decoder (such as the MRNN) that works across many recording
conditions avoids switching-related downtime.
In addition to training with months of previous data, we improved the MRNN’s robustness by
intentionally perturbing the training neural data. In the present study we applied random Gaussian
firing rate scaling based on a general assumption that the decoder should be broadly robust to
both global and private shifts in observed firing rates. This perturbation type proved effective, but
this approach (called data augmentation in the machine learning community) can potentially be
much more powerful when combined with specific modeling of recording condition changes that the
experimenter wants to train robustness against. For example, data augmentation could incorporate
synthetic examples of losing a particularly error-prone set of electrodes; the effects of known sources
of electrical noise; recording changes predicted by models of array micro-movement or degradation;
and perhaps even the predicted interaction between kinematics and changes in cognitive state or
task context. I believe this is an important avenue for future research.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 25
I view the success of our specific MRNN decoder implementation as a validation of the more
general BMI decoder strategy of training a computationally powerful nonlinear decoder to a large
quantity of data representing many different recording conditions. This past data need not be have
been collected explicitly for the purpose of training as was done in this study; neural data and
corresponding kinematics from past closed-loop BMI use can also serve as training data (Nuyujukian
et al., 2014; Jarosiewicz et al., 2015). It is likely that other decoding algorithms will also benefit from
this strategy, and that there are further opportunities to advance the reliability and performance of
BMIs by starting to take advantage of these devices’ ability to generate large quantities of data as
part of their regular use.
2.5 Methods
2.5.1 Animal model and neural recordings
All procedures and experiments were approved by the Stanford University Institutional Animal Care
and Use Committee. Experiments were conducted with adult male rhesus macaques (‘R’ and ‘L’),
implanted with 96-electrode Utah arrays (Blackrock Microsystems Inc., Salt Lake City, UT) using
standard neurosurgical techniques. Monkeys R and L were implanted 30 months and 74 months
prior to the primary experiments, respectively. Monkey R had two electrode arrays implanted, one
in PMd and the other in M1, as estimated visually from anatomical landmarks. Monkey L had one
array implanted on the border of PMd and M1. Spike counts were collected by applying a single
negative threshold, set to 4.5 × root mean square of the spike band of each electrode. Behavioral
control and neural decode were run on separate PCs using the xPC Target platform (Mathworks,
Natick, MA), enabling millisecond-timing precision for all computations. Neural data were initially
processed by Cerebus recording system(s) (Blackrock Microsystems Inc., Salt Lake City, UT) and
were available to the behavioral control system within 5± 1 ms.
2.5.2 Behavioral tasks
We trained the monkeys to acquire targets with a virtual cursor controlled by either the position of
the hand contralateral to the arrays or directly from neural activity. Reaches to virtual targets were
made in a 2D fronto-parallel plane presented within a 3D environment (MSMS, MDDF, USC, Los
Angeles, CA) generated using a Wheatstone stereograph fused from two LCD monitors with refresh
rates at 120 Hz, yielding frame updates within 7± 4 ms (Cunningham et al., 2011). Hand position
was measured with an infrared reflective bead tracking system at 60 Hz (Polaris, Northern Digital,
Ontario, Canada). During BMI control, we allowed the monkey’s reaching arm to be unrestrained
(Nuyujukian et al., 2011; Shenoy & Carmena, 2014) so as to not impose a constraint upon the monkey
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 26
that during BMI control he must generate neural activity that does not produce overt movement
(Kaufman et al., 2014).
In the ‘Radial 8 Task’ the monkey was required to acquire targets alternating between a center
target and one of eight peripheral targets equidistantly spaced on the circumference of a circle. For
our closed-loop BMI experiments, the peripheral targets were positioned 8 cm from the center target.
In hand reaching datasets used for decoder training and offline decode, the targets were either 8 cm
or 12 cm (the majority of datasets) from the center. In much of Monkey L’s training data, the three
targets forming the upper quadrant were placed slightly further (13 and 14 cm) based on previous
experience that this led to decoders with improved ability to acquire targets in that quadrant. To
acquire a target, the monkey had to hold the cursor within a 4 cm × 4 cm acceptance window
centered on the target for 500 ms. If the target was acquired successfully, the monkey received a
liquid reward. If the target was not acquired within 5 s (BMI control) or 2 s (hand control) of target
presentation, the trial was a failure and no reward was given.
We measured closed-loop BMI performance on the Radial 8 Task using two metrics. Target
acquisition rate is the number of peripheral targets acquired divided by the duration of the task. This
metric is calculated across all trials of a certain experimental condition (i.e., which decoder was used)
and holistically reflects cursor control ability because, unlike time to target, it is negatively affected
by failed trials and directly relates to the animal’s rate of liquid reward. As a second measure of
performance that is more sensitive when success rates are high and similar between decoders (such
as the “ideal” conditions where we presented no challenges to the decoders), we compared times to
target. This measure consists of the time between when the target appeared and when the cursor
entered the target acceptance window prior to successfully acquiring the target, but does not include
the 500 ms hold time (which is constant across all trials). Times to target are only measured for
successful trials to peripheral targets.
In the Random Target Task each trial’s target appeared at a random location within a 20 cm × 20
cm region centered within a larger workspace that was 40 × 30 cm. A new random target appeared
after each trial regardless of whether this trial was a success or a failure due to exceeding the 5 s
time limit. The target location randomization enforced a rule that the new target’s acceptance area
could not overlap with that of the previous target. Performance on the Random Target Task was
measured by success rate (the number of successfully acquired targets divided by the total number of
presented targets) and the normalized time to target. Normalized time to target is calculated for
successful trials following another successful trial, and is the duration between target presentation
and target acquisition (not including the 500 ms hold time), divided by the straight-line distance
between this target’s center and the previously acquired target’s center (Suminski et al., 2010).
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 27
2.5.3 Multiplicative Recurrent Neural Network (MRNN) decoder
At a high level, the MRNN decoder transforms inputs u(t), the observed spike counts on each
electrode at a particular time, into a cursor position and velocity output. This is accomplished by
having first trained the weights of an artificial recurrent neural network such that when the network
is provided a time series of training neural data inputs, the training data kinematic outputs can be
accurately “read out” from this neural network’s state. The rest of this section will describe the
architecture, training, and use of the MRNN for the purpose of driving a BMI.
The generic recurrent network model is defined by an N -dimensional vector of activation variables,
x, and a vector of corresponding “firing rates”, r = tanh(x). Both x and r are continuous in time
and take continuous values. In the standard RNN model, the input affects the dynamics as an
additive time-dependent bias in each dimension. In the MRNN model, the input instead directly
parameterizes the artificial neural network’s recurrent weight matrix, allowing for a multiplicative
interaction between the input and the hidden state. One view of this multiplicative interaction is
that the hidden state of the recurrent network is selecting an appropriate decoder for the statistics of
the current dataset. The equation governing the dynamics of the activation vector is of the form
suggested in (Sutskever et al., 2011), but adapted in this study to continuous time in order to control
the smoothness to MRNN outputs,
τ x(t) = −x(t) + Ju(t) r(t) + bx. (2.5.1)
The N ×N ×|u| tensor Ju(t) describes the weights of the recurrent connections of the network, which
are dependent on the E-dimensional input, u(t). The symbol |u| denotes the number of unique values
u(t) can take. Such a tensor is unusable for continuous valued u(t) or even discrete valued u(t) with
prohibitively many values. To make these computations tractable, the input is linearly combined
into F factors and Ju(t) is factorized (Sutskever et al., 2011) according to the following formula:
Ju(t) = Jxf · diag(Jfu u(t)) · Jfx, (2.5.2)
where Jxf has dimension N × F , Jfu has dimension F × E, Jfx has dimension F ×N , and diag(v)
takes a vector, v, and returns a diagonal matrix with v along the diagonal. One can directly control
the complexity of interactions by choosing F . Additionally, the network units receive a bias bx. The
constant τ sets the time scale of the network, so we set τ on the order of hundreds of milliseconds to
allow meaningful interactions. The output of the network is read out from a weighted sum of the
network firing rates plus a bias, defined by the equation
z(t) = WTOr(t) + bz, (2.5.3)
where WO is an N ×M matrix, and bz is an M -dimensional bias.
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 28
Monkey R Monkey L∆t 20ms 20-30msτ 100ms 100msN 100 50F 100 50σtrial 0.045 0.045σelectrode 0.3 0.3gxf 1.0 1.0gfu 1.0 1.0gfx 1.0 1.0E 192 96days of training data 82-129 189-230years spanned 1.59 2.77β 0.99 0.99
Table 2.1: Network and training parameters used for the closed-loop MRNN BMI decoder
2.5.4 Training the MRNN
We began decoder training by instantiating MRNNs of network size N = 100 (monkey R) and N = 50
(monkey L) with F = N in both cases (see Table 2.1 for all key parameters). For monkey R, who
was implanted with two multielectrode arrays, E = 192, while for monkey L with one array, E = 96.
The non-zero elements of the non-sparse matrices Jxf , Jfu, Jfx are drawn independently from a
Gaussian distribution with zero mean and variance gxf/F , gfu/E, and gfx/N , with gxf , gfu, and
gfx set to 1.0 in this study. The elements of WO are initialized to zero, and the bias vectors bx and
bz are also initialized to 0.
The input u(t) to the MRNN (through the matrix Ju(t)) is the vector of binned spikes at each
time step. Concatenating across time in a trial yields training data matrix, Uj , of binned spikes
of size E × T j , where T j is the number of times steps for the jth trial. Data from five consecutive
actual monkey-reaching trials are then concatenated together to make one ‘MRNN training’ trial.
The first two actual trials in an MRNN training trial were used for seeding the hidden state of the
MRNN (i.e., not used for learning), whereas the next three actual trials were used for learning. With
the exception of the first two actual trials from a given recording day, the entire set of actual trials
are used for MRNN learning by incrementing the actual trial index that begins each training trial by
one.
The parameters of the network were trained offline to reduce the averaged squared error between
the measured kinematic training data and the output of the network, z(t). Specifically, we used the
Hessian-Free (HF) optimization method (Martens, 2010; Martens & Sutskever, 2011) for RNNs (but
adapted to the continuous-time MRNN architecture). HF is an exact 2nd order method that uses
back-propagation through time to compute the gradient of the error with respect to the network
parameters. The set of trained parameters is {Jxf ,Jfu,Jfx, bx, bz}. The HF algorithm has three
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 29
critical parameters: the minibatch size, the initial lambda setting, and the max number of conjugate
gradient iterations. We set these parameters to one-fifth the total number of trials, 0.1, and 50,
respectively. The optimizations were run for 200 steps and a snapshot of the network was saved every
10 steps. Among these snapshots, the network with the lowest cross-validation error on held-out data
was used in the experiment.
We independently trained two separate MRNN networks to each output a 2-dimensional (M = 2)
signal, z(t). The first network learned to output the normalized hand position through time in both
the horizontal (x) and vertical (y) spatial dimensions. The second MRNN learned to output the
hand velocity through time, also in the x and y dimensions. We calculated hand velocities from the
positions numerically using central differences.
In this study we trained a new MRNN whenever adding new training data; this allowed us to
verify that the training optimization consistently converged to a high-quality decoder. However, it
is easy to iteratively update an MRNN decoder with new data without training from scratch. By
adding the new data to the training corpus and using the existing decoder weights as the training
optimization’s initial conditions, the MRNN will more rapidly converge to a new high-quality decoder.
2.5.5 MRNN training data augmentation
A critical element of achieving both high performance and robustness in the MRNN decoder was
training the decoder using data from many previous recording days spanning many months. When
training datasets included data from more than one day, we randomly selected a small number of
trials from each day for a given minibatch. In this way, every minibatch of training data sampled the
input distributions from all training days.
A second key element of training robustness to recording condition changes was a form of data
augmentation in which we intentionally introduced perturbations to the neural spike trains that
were used to train the MRNN. The concatenated input, U = [ui, · · · ,ui+4] was perturbed by adding
and removing spikes from each electrode. We focus on electrode c of the jth training trial, i.e., a
row vector of data Ujc,:. Let the number of actual observed spikes in Uj
c,: be njc. This number was
perturbed according to
njc = ηjηcnjc, (2.5.4)
where both ηj and ηc are Gaussian variables with a mean of one and standard deviations of σtrial and
σelectrode, respectively. Conceptually, ηj models a global firing rate modulation across all electrodes
of the array (e.g., array movement, arousal), while ηc models electrode by electrode perturbations
such as electrode dropping or moving baselines in individual neurons. If njc was less than zero or
greater than 2njc, it was resampled, which kept the average number of perturbed spikes in a given
electrode and training trial roughly equal to the average number of true (unperturbed) spikes in the
same electrode and training trial. Otherwise, if njc was greater than njc, then njc − njc spikes were
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 30
added to random time bins of the training trial. If njc was less than njc, then njc − njc spikes were
randomly removed from time bins of the training trial that already had spikes. Finally, if njc = njc,
nothing was changed.
The process of perturbing the binned spiking data occurred anew on every iteration of the
optimization algorithm, i.e. in the HF algorithm, the perturbation njc = ηjηcnjc occurs after each
update of the network parameters. Note that these input data perturbations were only applied
during MRNN training; when the MRNN was used for closed-loop BMI control, true neural spike
counts were provided as inputs. Figure 2.7 shows the closed-loop control quality difference between
the MRNN trained with and without this data augmentation.
2.5.6 Controlling the BMI cursor with MRNN network output
Once trained, the MRNNs were compiled into the embedded real-time operating system and run in
closed-loop to provide online BMI cursor control. The decoded velocity and position were initialized
to 0, as was the MRNN hidden state. Thereafter, at each decode time step the parallel pair of
MRNNs received binned spike counts as input and had their position and velocity outputs blended to
yield a position estimate. This was used to update the drawn cursor position. The on-screen position
that the cursor moves to during BMI control, dx(t), dy(t), is defined by
dx(t) = β(dx(t−∆t) + γvvx(t−∆t)∆t) + (1− β)γppx(t) (2.5.5)
dy(t) = β(dy(t−∆t) + γvvy(t−∆t)∆t) + (1− β)γppy(t), (2.5.6)
where vx, vy, px, py are the normalized velocity and positions in the x and y dimensions and γv, γp
are factors that convert from the normalized velocity and position, respectively, to the coordinates of
the virtual-reality workspace. The parameter β sets the amount of position versus velocity decoding
and was set to 0.99. In effect, the decode was almost entirely dominated by velocity, with a slight
position contribution to stabilize the cursor in the workplace (i.e., offset accumulated drift). Note
that when calculating offline decode accuracy (Figure 2.2), we set β to 1 to more fairly compare the
MRNN to the FIT-KF decoder, which decodes velocity only.
2.5.7 Feedback Intention Trained Kalman Filter (FIT-KF) decoder
We compared the performance of the MRNN to FIT-KF (Fan et al., 2014). The FIT-KF is a Kalman
filter where the underlying kinematic state, z(t), comprises the position and velocity of the cursor as
well as a bias term. Observations of the neural binned spike counts, u(t), are used to update the
kinematic state estimate. With ∆t denoting bin width (25 ms in this study), the FIT-KF assumes
the kinematic state gives rise to the neural observations according to the following linear dynamical
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 31
system:
z(t+ ∆t) = Az(t) + w(t) (2.5.7)
u(t) = Cz(t) + q(t), (2.5.8)
where w(t) and q(t) are zero-mean Gaussian noise with covariance matrices W and Q respectively.
The Kalman filter is a recursive algorithm that estimates the state z(t) using the current observation
u(t) and the previous state estimate z(t−∆t). Previous studies have used such decoders to drive
neural cursors (e.g. Kim et al., 2008; Sussillo et al., 2012; Bacher et al., 2015).
The parameters of this linear dynamical system, A,W,C,Q, are learned in a supervised fashion
from hand reach training data as previously reported (Wu et al., 2003; Gilja et al., 2012). The
FIT-KF then incorporates two additional innovations. First, it performs a rotation of the training
kinematics using the assumption that at every moment in time, the monkey intends to move the
cursor directly towards the target. Second, it assumes that at every time step, the monkey has
perfect knowledge of the decoded position via visual feedback. This affects Kalman filter inference in
two ways: first, the covariance of the position estimate in Kalman filtering is set to 0, and secondly,
the neural activity that is explainable by the cursor position is subtracted from the observed binned
spike counts. These innovations are further described in (Gilja et al., 2012; Fan et al., 2014).
2.5.8 Mutual information for determining electrode dropping order
When testing the decoders’ robustness to unexpected electrode loss, we determined which electrodes
to drop by calculating the mutual information between each electrode’s binned spike counts and the
reach direction. This metric produced a ranking of electrodes in terms of how statistically informative
they were of the reach direction; importantly, this metric is independent of the decoder being used.
Let p denote the distribution of a electrode’s binned firing rates, u denote the binned spike counts
lying in a finite set U of possible binned spike counts, M denote the number of reach directions, and
xj denote reach direction j. The set U comprised {0, 1, 2, 3, 4, 5+} spike counts, where any spike
counts greater than or equal to 5 were counted towards the same bin (‘5+‘, corresponding to an
instantaneous firing rate of 250 Hz in a 20 ms bin). We calculated the entropy of each electrode,
H(U) = −∑u∈U
= p(u) log p(u), (2.5.9)
as well as its entropy conditioned on the reach direction,
H(U |X) = −M∑j=1
p(xj)∑u∈U
p(u|xj) log p(u|xj). (2.5.10)
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 32
From these quantities, we calculated the mutual information between the neural activity and the
reach direction as Idrop(X;U) = H(U)−H(U |X). We dropped electrodes in order from highest to
lowest mutual information.
2.5.9 Principal angles of neural subspaces analysis
For a parsimonious scalar metric of how similar patterns of neural activity during reaching were
between a given pair of recording days (used in Figure 2.3), I calculated the minimum principal angle
between the neural subspaces of each recording day. I defined the neural subspace on a recording
day as the top K principal components of the neural coactivations. Put more simply, I asked how
similar day i and day j’s motifs of covariance between electrodes’ activity were during arm reaching.
Specifically, I started with a matrix Ui from each day i consisting of neural activity collected while
the monkey performed 200 trials of a Radial 8 Task (8 cm distance to targets) using arm control;
this task has been run at the start of almost every experimental session conducted using both
monkeys R and L since array implantation. Ui is of dimensionality E × T , where E is the number of
electrodes and T is the number of non-overlapping 20 ms bins comprising the duration of this task. I
next subtracted from each row of Ui that electrode’s across-days mean firing rate (I also repeated
this analysis without across-days mean subtraction and observed qualitatively similar results, not
shown). To obtain the principal components, I performed eigenvalue decomposition on the covariance
matrix UiUTi (note, Ui is zero mean), and defined the matrix Vi as the first K eigenvectors. Vi had
dimensions I ×K, where each column k is the vector of principal component coefficients (eigenvector)
corresponding to the kth largest eigenvalue of the decomposition. Figure 2.3 was generated using
K = 10, i.e., keeping the first 10 PCs, but the qualitative appearance of the data was similar when
K was varied from 2 to 30 (not shown). Finally, the difference metric between days i and j was
computed as the minimum of the K subspace angles between matrices Vi and Vj . Subspace angles
were computed using the subspacea MATLAB function (Knyazev & Argentati, 2002).
2.6 Project acknowledgements
I thank Chethan Pandarinath and the members of the Shenoy group for very helpful discussions;
Mackenzie Mazariegos, Michelle Wechsler, John Aguayo, Clare Sherman, Erica Morgan, and Liana
Yates for surgical assistance and veterinary care; Beverly Davis, Sandy Eisensee, and Evelyn Castaneda
for administrative support; Boris Oskotsky for information technology support.
This work was supported by the National Science Foundation Graduate Research Fellowship
(J.C.K., S.D.S.); NSF IGERT 0734683 (S.D.S.); Christopher and Dana Reeve Paralysis Foundation
(S.I.R. and K.V.S.); and the following to K.V.S.: Burroughs Welcome Fund Career Awards in the
Biomedical Sciences, Defense Advanced Research Projects Agency Reorganization and Plasticity
to Accelerate Injury Recovery N66001-10-C-2010, US National Institutes of Health Institute of
CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 33
Neurological Disorders and Stroke Transformative Research Award R01NS076460, US National
Institutes of Health Director’s Pioneer Award 8DP1HD075623-04, US National Institutes of Health
Director’s Transformative Research Award (TR01) from the NIMH 5R01MH09964703, and Defense
Advanced Research Projects Agency NeuroFAST award from BTO W911NF-14-2-0013.
Chapter 3
A high performing brain-machine
interface driven by low-frequency
local field potentials alone and
together with spikes
3.1 Summary
Current high performance BMIs are driven by action potentials (spikes), but access to this signal often
diminishes as sensors degrade over time. Decoding local field potentials (LFPs) as an alternative or
complementary BMI control signal may improve performance when there is a paucity of spike signals.
Only a small handful of LFP decoding methods have been tested online; there remains a need to test
different LFP decoding approaches and improve LFP-driven performance. Until this project, there
had also not been a reported demonstration of a hybrid BMI that decodes kinematics from both
LFP and spikes. We recorded spikes and LFP from two macaques implanted with multielectrode
arrays in primary and premotor cortex while they performed a reaching task. I first evaluated a
BMI driven by the local motor potential (LMP), a low-pass filtered time-domain LFP amplitude
feature. I then combined decoding of both LMP and spikes to implement a hybrid BMI. We compared
closed-loop BMI control using biomimetic decoders driven by LMP, spikes, or both signals together.
LMP decoding enabled quick and accurate cursor control which surpassed previously reported LFP
BMI performance. Hybrid decoding of both spikes and LMP improved performance when spikes
signal quality was mediocre to poor. These findings show that LMP is an effective BMI control signal
which requires minimal power to extract and can substitute for or augment impoverished spikes
34
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 35
signals. Use of this signal may lengthen the useful lifespan of BMIs and is therefore an important
step towards clinically viable BMIs.
3.2 Introduction
To date, the highest-performing BMIs have been driven by spike activity recorded with intracortical
multielectrode arrays; these have enabled non-human primates to accurately control computer cursors
(Serruya et al., 2002; Ganguly & Carmena, 2009; Gilja et al., 2012), robotic limbs (Carmena et al.,
2003; Velliste et al., 2008), or the subject’s own musculature (Ethier et al., 2012). This research is
now being translated to early-stage clinical studies in people with paralysis (Hochberg et al., 2012;
Collinger et al., 2013b; Bacher et al., 2015). A critical challenge that must be overcome to enable
clinically viable BMIs is to improve the device’s useful lifespan by maintaining high performance over
a long period of time, thereby improving its risk-benefit balance (Ryu & Shenoy, 2009). Chronically
implanted sensors often degrade over time and gradually lose their ability to record action potentials
(Kruger et al., 2010; Simeral et al., 2011; Chestek et al., 2011; Barrese et al., 2013). One approach to
mitigate this is to decode multiunit spikes instead of well-isolated single unit activity (Fraser et al.,
2009; Chestek et al., 2011; Gilja et al., 2012; Hochberg et al., 2012; Nuyujukian et al., 2014). An
alternative — or complementary — strategy is to make use of neural signals other than spikes that
contain information about movement intention and are available from the same sensors. The local
field potential (LFP) is such a signal: it is obtained by low-pass filtering the same raw voltage signal
from which spikes are high-pass filtered, and it carries information about the kinematics of planned
(O’Leary & Hatsopoulos, 2006) and executed (Mehring et al., 2003; Rickert et al., 2005; Scherberger
et al., 2005; O’Leary & Hatsopoulos, 2006; Bansal et al., 2011a) reaching movements.
Despite many offline LFP decoding studies (Pesaran et al., 2002; Mehring et al., 2003; Rickert
et al., 2005; Scherberger et al., 2005; O’Leary & Hatsopoulos, 2006; Asher et al., 2007; Markowitz
et al., 2011; Bansal et al., 2011a; Flint et al., 2012a,b; Zhuang et al., 2010; Hwang & Andersen, 2013;
Wang et al., 2014; Perge et al., 2014) and a few closed-loop demonstrations (Kennedy et al., 2004;
Hwang & Andersen, 2009), only very recently have there been reports of effective closed-loop cursor
control driven by LFP (Flint et al., 2013; So et al., 2014). While encouraging, the performance
achieved by these first forays into LFP-driven BMIs is low compared to spike-driven performance and
leaves open the question of how viable LFP is as an alternative and complimentary control signal.
My study aimed to address this gap by showing considerably improved LFP-driven performance and
by showing, for the first time, that under certain conditions LFP can be beneficially combined with
available spikes to improve closed-loop BMI control. LFP can be processed into a variety of different
features, and these two previous studies made specific design choices to decode LFP power in multiple
frequency bands as well as — in (Flint et al., 2013) — low-frequency LFP amplitude known as local
motor potential (‘LMP’, which has also been referred to as movement-evoked potential in earlier
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 36
studies). Closed-loop performance using other choices of LFP features had yet to be characterized.
Based on the results of my own offline evaluation of decoder performance using various LFP features,
I identified that the LMP was the best candidate feature. We subsequently evaluated the closed-loop
performance of a BMI design that differs from that of previous studies by only extracting from each
electrode a half-wave rectified variant of the LMP, rather than the unrectified LMP or LFP power
in various frequency bands. Two macaques successfully used this LMP-driven BMI to perform a
2D target acquisition task and, to my knowledge, demonstrated higher performance than in any
previously reported online LFP BMI study.
After showing that this LMP feature is an effective alternative BMI control signal, I set out to
test whether combining LMP and spikes decoding could improve closed-loop performance. LFP
reflects a spatial averaging of synaptic and other currents in the vicinity of the electrode and carries
information that is distinct from spikes (Belitski et al., 2008; Katzner et al., 2009; Buzsaki et al.,
2012; Einevoll et al., 2013). Furthermore, LFP can be informative even on electrodes which do not
record spikes (Bokil et al., 2006; Heldman et al., 2006; Flint et al., 2012b; Hwang & Andersen, 2013;
Wang et al., 2014). For these reasons it has long been hoped that decoding LFPs could augment
BMIs, especially in cases where few electrodes have spikes available. I demonstrated the first use of a
hybrid LFP and spikes decoder for continuous BMI cursor control and showed that this approach
can indeed improve performance compared to spikes-only decoding. The benefit of hybrid decoding
increased as we reduced the number of electrodes with informative spike activity. However, we also
found that in our second monkey, who had more electrodes recording tuned spikes activity and worse
overall LMP-driven performance, hybrid decoding was only beneficial if spikes were made unavailable
from the majority of electrodes.
3.3 Results
3.3.1 Selection of local motor potential as a control signal based on offline
evaluation of candidate LFP features
In order to choose which LFP-derived feature(s) to evaluate as the control signal for closed-loop BMI
experiments, I first tested a number of neural features offline to determine how well the velocity
of point to point arm reaches could be decoded from each feature. I compared spike counts in
non-overlapping 50 ms bins, the local motor potential (‘LMP’, i.e. low-pass filtered LFP amplitude),
and LFP power in the 1-4 Hz, 3-10 Hz, 12-23 Hz, 27-38 Hz, and 50-300 Hz bands. I also tested
decode performance using combinations of spike counts and one or more LFP-derived features. For
this analysis I used five datasets of the Radial 8 Task from each of two adult male rhesus macaques.
A velocity Kalman filter was trained to decode 2D hand endpoint velocity from each type of neural
feature or combination of features. Neural data came from 192-electrode recordings from contralateral
M1 and PMd. Half of each dataset’s 500 trials were used for training the decoder and the remaining
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 37
d
cb
a
Analog Filter 0.3 to 7500 Hz bandpass
digital filter500 Hz lowpass
0 1
-250
0
150
time (s)
Vo
lta
ge
(µ
v)
Raw Neural Signal
Decode
Velocity
a
time (s)-0.3 0 1.1
-80
0
80
0
80
vo
lta
ge
(µ
V)
vo
lta
ge
(µ
V)
0°45°90°
135°180°225°270°315° Half-Wave Rectified LMP
50 ms boxcar filterExtract LMP
LMP
vo
lta
ge
(µ
V)
0 1
-150
0
150
time (s)
LFP
LMP
Figure 3.1: Overview of local motor potential (LMP) BMI signal processing(a) A macaque sat in a primate chair and observed a cursor and target in a virtual-reality environment.During arm control, he controlled the cursor position by reaching with his unrestrained arm. DuringBMI control, the cursor was driven by a velocity command decoded from neural activity.(b) Neural signals were recorded from 96-electrode arrays in M1 and PMd. Electrical signals wereamplified, analog filtered from 0.3 to 7500 Hz, and digitized. A 1 s sample of this raw signal is shown;both lower frequency components and high frequency multiunit spikes are visible.(c) The LFP (shown in grey) was then obtained by low-pass filtering the raw digital signal below 500Hz. LMP, shown in black, was extracted by boxcar filtering the most recent 50 ms of LFP.(d) (Top) Example LMPs trial-averaged over 32 arm reaches to each of 8 peripheral targets. Tracecolor corresponds to reaches to targets in the specified direction. The target was presented at time =0. (Bottom) Before decoding, LMP was half-wave rectified: negative voltage samples were rectifiedwhile positive voltage samples were set to 0. Dataset R.2013.07.30, electrode 1 (M1 array).
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 38
half were used to test how accurately the decoder reconstructed the true hand velocity. Kinematics
and neural features were binned every 50 ms, and decode performance using different lags (i.e., neural
data aligned to or leading kinematics) was evaluated; I report how accurately the testing data was
decoded using the best lag for each particular feature as fit from the training data.
Figure 3.2 shows the average of the Pearson’s linear correlation coefficients between true and
reconstructed x- and y-velocities using each neural feature(s). I found that LMP outperformed the
various LFP power features, and was the only LFP-derived feature to match or exceed the offline
decoding accuracy of spikes (monkey R spikes r = 0.73± 0.022 [mean ± standard deviation], LMP
r = 0.78± 0.010, p = 0.007 [paired t-test for a difference in means]; monkey J spikes r = 0.80± 0.032,
LMP r = 0.77± 0.005, p = 0.065). Combining LMP with spikes improved offline decoding accuracy
in both animals compared to decoding spikes alone (monkey R ∆r = 0.11, p < 0.001; monkey
J ∆r = 0.054, p = 0.003). Although decoding both gamma LFP power (50-300 Hz) and spikes
together also led to a small increase in accuracy, this effect was smaller than when combing LMP
and spikes. Furthermore, adding both LMP and gamma to spikes did not substantially improve
the decode accuracy over just using spikes + LMP (monkey R ∆r = 0.0026, p = 0.034; monkey J
∆r = 0.0002, p = 0.78). LMP was also a better offline decode feature than signal amplitude in the
other bands, or acausally filtered power in the other bands (Figure 3.3).
Velocity could be decoded from LMP because this signal evolved differently over time depending
on which direction the animal was reaching. This can be seen in Figure 3.1d, which shows larger
amplitude LMP deflections during reaches towards a subset of the 8 targets on an example M1
electrode. Across electrodes and monkeys I observed a variety of LMP waveforms in terms of shape,
number of peaks, and tuning for velocity (not shown). Figure 3.4 shows that the LMP recorded on
most electrodes was significantly tuned for reach direction, and that the preferred directions (PDs) of
LMP recorded on nearby electrodes tended to have similar tuning. Despite these local correlations,
across all electrodes the PDs spanned both dimensions required for 2D cursor control. I wish to
point out that monkey R’s M1 array is an example of an array that records very poor spikes signals
(offline decode accuracy of r = 0.11± 0.042 [mean ± standard deviation] using just this array’s spikes
to perform the same analysis as in Figure 3.2), but has significant LMP tuning on a number of
electrodes. When using LMP measured only from this array, I observed an improved offline decode
accuracy of r = 0.46± 0.029. Based on these offline results I decided to evaluate LMP alone and
spikes + LMP together as control signals for a closed-loop BMI.
3.3.2 Control experiment showing LMP is not a movement artifact
Prior to conducting closed-loop experiments, I first wanted to convince myself that the LMP we
were decoding was not artifactually correlated with the monkey’s arm movement due to factors such
as microphonic pickup of electronic noise in the room (despite heavily electromagnetically shielded
walls) or mechanical vibration of the recording equipment (despite the ruggedly built experimental
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 39
*** ***
*** ***
***
*** **
*** **
0
0.2
0.4
0.6
0.8
1 Monkey J
spikes
LMP +
50−
300H
zLM
P
power
1−4
Hz
power
3−1
0Hz
power
12−
23Hz
power
27−
38Hz
power
50−
300H
z
LMP
power
1−4
Hz
power
3−1
0Hz
power
12−
23Hz
power
27−
38Hz
power
50−
300H
z
spikes +
**
*** ******
***
***
** ****
***
0
0.2
0.4
0.6
0.8
1
***
spikes
LMP +
50−
300H
zLM
P
power
1−4
Hz
power
3−1
0Hz
power
12−
23Hz
power
27−
38Hz
power
50−
300H
z
LMP
power
1−4
Hz
power
3−1
0Hz
power
12−
23Hz
power
27−
38Hz
power
50−
300H
z
spikes +
Monkey R
*
De
co
de
Accu
racy (
r)
1 2 3 4 5 6 7 8 9 10
0
0.1
0.2
0.3
Frequency (Hz)
No
rma
lize
d V
olta
ge example monkey R M1 electrode
example monkey J PMd electrode
a
c
0 20 40 60 80
−40
−30
−20
−10
0
Frequency (Hz)
Ma
gn
itu
de
(d
B)
LMP
1-4 Hz 3-10 Hz 12-23 Hz 27-38 Hz 50-300 Hzb
Figure 3.2: Comparison of offline decode accuracy using different neural features(a) Two-dimensional hand velocity from Radial 8 Task reaches was reconstructed offline with aKalman filter that decoded either multiunit threshold crossing spikes (red), various LFP features(blue), or the combination of spikes and one (violet) or two (striped) LFP features. The Pearsoncorrelation (r) between true and decoded velocity was averaged across all trials and both x- and y-dimensions. We show the grand mean and standard error of measurement across five datasets foreach monkey. Stars denote signals or combinations of signals whose decode performance differedsignificantly from that of decoding spikes only (paired t-test, * p < 0.05, ** p < 0.01, *** p < 0.001).(b) Amplitude response of the filter cascade used to generate each of the evaluated LFP frequencybands. The periodic dip in the response of LMP is due to the 50 ms boxcar filtering operation. Thehigh frequency cutoff of the 50-300 Hz filter extends beyond the panel; it is flat until 200 Hz androlls off to -4.7 dB at 300 Hz. (c) To illustrate the actual spectral content of the half-wave rectifiedLMP signal, we have plotted the frequency spectra of this feature during closed-loop use of the LMPdecoder from a representative electrode from each monkey. Datasets R.2013.09.09 electrode 1 andJ.2013.09.11 electrode 122.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 40
monkey R monkey J
0
0.2
0.4
0.6
0.8
1
acausal pow
er 3-1
0Hz
De
co
de
Accu
racy (
r)
spike
sLM
P
|LM
P|
power 1−4Hz
bandpass 1−4Hz
power 3−10Hz
bandpass 3-1
0Hz
power 12-2
3Hz
bandpass 1
2-23Hz
power 27-3
8Hz
bandpass 27-3
8Hz
power 50-3
00Hz
bandpass 50-3
00Hz
spike
sLM
P
|LM
P|
power 1−4Hz
bandpass 1−4Hz
power 3−10Hz
bandpass 3-1
0Hz
power 12-2
3Hz
bandpass 1
2-23Hz
power 27-3
8Hz
bandpass 27-3
8Hz
power 50-3
00Hz
bandpass 50-3
00Hz 0
0.2
0.4
0.6
0.8
1
acausal pow
er 3-1
0Hz
Figure 3.3: Offline decode analysis of additional candidate LFP featuresOffline decode analysis of additional candidate LFP features. Solid bars are the same features asin Figure 3.2a, plotted here for comparison; outlined bars with bolded labels show offline decodeaccuracy using additional features. The offline decode was performed using the same data andmethods as in Figure 3.2. None of these additional features performed as well as the LMP feature.‘|LMP|’ is the absolute value of the LMP signal; note that this is not the same as the half-waverectified LMP, which had similar offline decode accuracy to the LMP results shown here. The ‘acausalpower 3-10Hz’ feature was computed by filtering the signal both forwards and backwards in timeto yield a zero phase delay filter. While this improved offline decode accuracy, we note that thisacausal filter would be impractical for closed-loop decoding because of the required delay introducedinto the control loop. We also evaluated the other frequency bands’ power after filtering in thisacausal manner (including the LMP), and found that these features’ offline decode accuracy (notshown) were very similar to their causally filtered counterparts. The ‘bandpass’ features are thetime-domain signal amplitude (sampled every 50 ms) after filtering the LFP using the same filter asthe corresponding band power feature.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 41
Cen
tral
Sul
cus
M1
Precentral
DimplePMd
Central Sulcus
M1
PMd
Precentral
Dimple
M
L
A P
4 mm
Monkey R Monkey J
Figure 3.4: Directional tuning of LMP overlaid on electrode location in motor cortexThe locations of the two arrays in each monkey are drawn along with major anatomical landmarksbased on intraoperative surgical photographs. Each dot with a colored vector coming out of itcorresponds to an electrode and the preferred direction (PD) of the LMP feature recorded on thatelectrode, as computed from 250 center out arm reaches. Both the vector’s color and angle correspondto PD, while the vector length signifies 1− (p value) of tuning (ANOVA). Electrodes that were notsignificantly tuned (p > 0.01) are shown in greyed out color. Note that the PDs shown are in thecoordinate system of the two-dimensional task and are unrelated to the anatomical coordinate systemused to map the physical location of each electrode in the brain. PMd = dorsal premotor cortex, M1= primary motor cortex, A = anterior, P = posterior, L = lateral, M = medial. Monkey R’s M1array quality was poor both for LMP and especially for spikes even from the earliest post-implantrecordings. Datasets R.2013.09.09 J.2013.09.11.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 42
apparatus being connected to massive load-bearing metal walls via structural metal). Both of these
noise sources could appear to be “tuned” to reach kinematics because arm movements can cause
very small but specific movements of the experimental apparatus. We therefore conducted a control
experiment in which similar arm movements were made without a corresponding neural movement
intention. We sedated the monkey with ketamine and dexmedetomidine, gently placed him in
the primate chair, and recorded neural signals while the monkey’s hand was moved by a human
experimenter to complete the same Radial 8 Task. Mean hand velocity reconstruction accuracy using
LMP dropped from r = 0.82 (monkey R), r = 0.79 (monkey J) when the animals were awake and
making volitional reaches, to r = 0.05 (monkey R) and r = 0.01 (monkey J) using the sedated control
data collected later the same day. This indicates that the LMP signal is not an artifact related to
the monkey’s arm movements.
3.3.3 Closed-loop BMI performance driven by LMP
Both monkeys demonstrated high performance using an LMP-driven BMI. For each experimental
session, we trained a velocity Kalman filter using LMP from 500 arm reaches. I evaluated performance
by measuring the success rate, normalized time to target, and path length ratio on a Continuous
Random Target Task in which 5× 5 cm targets appeared anywhere in a 20× 20 cm workspace and
had to be acquired within 8 seconds (Figure 3.5). Monkey R did the task at a 99% success rate
(11,429 trials across 6 experiment sessions) with 0.08 s/cm median normalized time to target and
median path length ratio of 2.09. Monkey J had an 86% success rate using the LMP decoder across
2,065 trials over 8 sessions, with median normalized time to target of 0.095 s/cm and path length
ratio 2.47. Chance performance levels were 21% success rate, 0.33 s/cm time to target for monkey
R, and 34% success rate, 0.31 s/cm time to target for monkey J. Note that I had given monkey J
an easier task (300 ms target hold time compared to monkey R’s 500 ms hold time) to keep him
engaged in the task because he had worse LMP-driven BMI performance.
Because there have only been a few published online LFP BMI studies, we were able to also
measure performance on tasks closely matched to those used in two prior reports to allow for a more
direct comparison. When tested on a Radial 8 Task similar to that of (So et al., 2014), monkeys R
and J achieved considerably higher success rates: 98% for monkey R and 85% for monkey J (Figure
3.6), compared to 78% and 72% success rates reported in that study. In terms of successful target
acquisitions per minute, monkey R’s 16.1 targets/min and monkey J’s 9.8 targets/min compare
favorably to the 10.6 and 5.8 targets/min in (So et al., 2014). We also had both monkeys perform a
Continuous Random Target Task with smaller targets and 100 ms hold times as in (Flint et al., 2013)
(Figure 3.7). Monkey R performed this task with 79% grand mean success rate, normalized time to
target = 0.070 s/cm, index of performance = 2.71 bits/s, and path length ratio = 1.53. Monkey J’s
metrics were: 73% success rate, normalized time to target = 0.090 s/cm, index of performance =
2.16 bits/s, and path length ratio = 1.79. Compared to (Flint et al., 2013), this represents a slightly
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 43
TrialTrial
Su
cce
ss
Ra
teN
TT
T (
s/c
m)
Pa
th L
en
gth
(cm
/cm
)
a
b
c
Monkey R Monkey J
0.86
1 221 490 765 1041 1325 1605 1876 2065 0 0.2 0.4 0.6 0.8 1
1 221 490 765 1041 1325 1605 1876 2065 0 0.1 0.2 0.3 0.4 0.5
1 221 490 765 1041 1325 1605 1876 2065 1 2 3 4 5 6
2.47
0.1530.095
0.99
1 2702 6622 7891 9057 10155 11429 0 0.2 0.4 0.6 0.8 1
1 2702 6622 7891 9057 10155 11429 0 0.1 0.2 0.3 0.4 0.5
1 2702 6622 7891 9057 10155 11429 1 2 3 4 5 6
2.46
0.0810.0810.039
1.18
0.0380.038
1.28
Figure 3.5: LMP-driven BMI performance on the Continuous Random Target TaskEach performance metric is smoothed over the previous 50 trials and is shown across six and eightdays’ experiments for monkeys R and J, respectively. Vertical gray lines separate trials from differentdays. Grand mean (success rate) or median (NTTT and path length) performance for each plot ismarked with a black arrow and value. Dashed gray lines and gray numbers show performance onthis task using hand control.(a) Success rate is the fraction of trials where the target was acquired within 8 seconds.(b) Normalized time to target measures how long it took to successfully acquire the target, normalizedby the straight-line distance between the current and previous targets. The final hold time is notincluded in NTTT.(c) Path length ratio measures the distance the cursor travelled to acquire the target divided by thestraight-line distance; closer to 1 is more direct (better).
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 44
Monkey R
Pe
rce
nt S
ucce
ssfu
l
trial
98.3% overall successful
1 203 458 763 1066 1371 1679
0
100
Monkey J
Pe
rce
nt S
ucce
ssfu
l
trial
85.1% overall sucessful
1 170 385 557
0
1000 4 8
0
0.27
Time to Target (s)
Pro
po
rtio
n o
f tr
ials
Monkey R
0 4 8
0
0.30
Time to Target (s)
Monkey J
Pro
po
rtio
n o
f tr
ials
a b
Figure 3.6: Performance of the LMP decoder on a Radial 8 TaskTargets were pseudorandomly positioned at one of 8 locations evenly spaced along a 13 cm diametercircle. The monkey initiated the trial by first acquiring a central target, after which the targetjumped to a peripheral location. To successfully acquire the target, the cursor had to be movedwithin an acquisition region of width 3.4 cm and held there for a contiguous 300 ms. (a) Runningmean of success rate for the previous 50 trials across 6 days for monkey R and 3 days for monkey J.Vertical grey lines mark transitions between experiment days. Overall percent successful across allexperimental sessions’ trials is shown with a black arrow. (b) Histograms showing the distributionof times to target, also known as reach times, for all trials of this task. The 300 ms hold time is notincluded in this calculation of time to target.
better success rate and over twice as fast target acquisition times. Note that these success rates are
lower than in the standard Continuous Random Target Task because I posthoc counted as failures
those trials in which the cursor exited the target and then successfully reacquired it (which were
successful as far as the monkey was concerned). I did this to be consistent with the methods used by
Flint and colleagues; otherwise, the success rates would have been 99% and 86% for monkeys R and
J, respectively.
Since we evaluated biomimetic decoders trained using arm reaching neural activity, we did not
restrain the monkey’s contralateral (with respect to the arrays) arm during BMI use. I observed that
both monkeys continued to move their arms during LMP-driven cursor control. The monkeys were
not required to keep the reflective bead taped to their fingers visible to the tracking system when
using the BMI; consequently, I was only able to quantify the relationship between hand and cursor
movement for 78% (monkey R) and 53% (monkey J) of LMP-driven time samples. During these
epochs, the grand mean correlation between hand and LMP-driven cursor velocities was r = 0.736
(p < 0.001) for monkey R and r = 0.668 (p < 0.001) for monkey J (averaging across x- and y- velocity,
and across all Continuous Random Target Task sessions).
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 45
a
b
c
d
Monkey R Monkey J
1 2 3
0
0.2
0.4
0.6
0.8
1
1 2 3
0
0.1
0.2
0.3
0.4
0.5
0.6
1 2 3
0
1
2
3
1 2 3
0
1
2
3
4
5
6
Su
cce
ss R
ate
1 2 3 4 5 6
0
0.2
0.4
0.6
0.8
1
NT
TT
(s/c
m)
1 2 3 4 5 6
0
0.1
0.2
0.3
0.4
0.5
0.6
IP (
bits/s
ec)
1 2 3 4 5 6
0
1
2
3
Pa
th L
en
gth
(cm
/cm
)
1 2 3 4 5 6
0
1
2
3
4
5
6
DayDay
Figure 3.7: Performance on aContinuous Random Target Task withshort hold timeTargets randomly appeared anywhere in a20× 20 cm region. The monkey had up to 10seconds to move the cursor into a 4× 4 cmtarget area and hold it there for 100 ms. Tobe consistent with the metrics used in (Flintet al., 2013), we restricted our definition of asuccessful trial to those where the cursorstayed within the target for 100 ms the firsttime they entered the target region; that is,trials with ‘dial-in’ (leaving the target within100 ms and then reacquiring it) were countedas failures. Note that this is a posthocrestriction of what constitutes a successfultrial, and that our monkeys were trained toexpect that dial-in is allowed and performedthe task under this reward contingency.(a) Mean success rate for each day when thistask was run is shown in filled circles. Opencircles shown the success rate had wecounted trials where the target was acquiredafter dial-in as successful.(b) Normalized time to target.(c) IP = Index of Performance, also knownas Fitts bits per second.(d) Path length ratio.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 46
3.3.4 Fixed LMP decoders work across multiple days without retraining
We also evaluated how well a fixed LMP decoder performed without any retraining for just over
three weeks. A decoder was trained on day 1 of these experiments and then loaded and evaluated for
at least 200 trials at the beginning of subsequent days’ experimental sessions. No recalibration or
re-normalization was done; the decoder weights and entire signal processing pipeline was identical
between collection of the training dataset on day 1 and during evaluation of the fixed decoder. The
fixed LMP decoder enabled both monkeys to perform the Continuous Random Target Task across the
entire tested period (Figure 3.8a-d), although with worse performance than using same day-trained
decoders. Monkey R’s daily success rates using the day 1 decoder on subsequent days ranged from
90.4% to 99.8%, with an accuracy of 93.0% on the last day tested (day 22), while monkey J’s daily
success rate ranged from 69.8% to 86.4%, and was 72.7% on day 23.
To investigate whether the ability to use a fixed decoder over several weeks was due to stable
LMP signals or the animal adapting to a potentially mismatched decoder during closed-loop BMI use,
I compared the fixed decoder’s closed-loop performance with its offline decode accuracy. I performed
an offline decode of hand velocities recorded during Radial 8 Task arm reaches at the start of each of
these experimental sessions using either the fixed day 1 decoder or a decoder trained from that same
day’s reaches. The fixed day 1 decoder was only slightly worse than the same day decoder (Figure
3.8e), suggesting that the LMP signals were largely stable across days. For monkey J, who exhibited
a wider divergence in BMI performance between the fixed and same-day decoders, closed-loop day
1 decoder success rates correlated with the day 1 decoder’s offline accuracy (r = 0.79, p < 0.05,
Pearson correlation). For monkey R, who had little variation in fixed decoder offline accuracy, this
regression was not significant against any of the closed-loop performance metrics.
A potential concern is that the monkey’s familiarity with the fixed decoder could have interfered
with his ability to control new decoders provided on subsequent days. Aside from a brief pause of
the task, no explicit cue was provided to indicate that the decoder had been switched. Nonetheless,
we found that the monkeys rapidly adjusted to each day’s newly trained decoder after switching
from the fixed decoder. The data shown in Figure 3.5 begin following a brief (10-20 trial) “transition”
period after the switch from the fixed decoder, and do not show within-session improvement that
would indicate overcoming interference from the first decoder.
3.3.5 BMI control using a hybrid spikes and LMP decoder
We next evaluated whether simultaneously decoding both LMP and spikes would improve performance
compared to a BMI driven by spikes alone. Figure 3.9 shows aggregated data from eight datasets
in monkey R and five datasets in monkey J where the hybrid decoder and spikes-only decoders
were evaluated in an interleaved ‘ABAB’ block format. When LMP features were added to spikes,
the hybrid decoder outperformed the spikes-only decoder in monkey R (12% improvement in mean
normalized time to target averaged over all datasets, p < 0.001; difference significant at p < 0.01
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 47
1 2 3 7 8 9 10 23
0
0.2
0.4
0.6
0.8
1
1 2 3 7 8 9 10 23
0
0.1
0.2
0.3
0.4
0.5
1 2 3 7 8 9 10 23 0
1
2
1 2 3 7 8 9 10 23
1
2
3
4
5
6
1 2 3 4 5 6 10 11 16 20 22
0
0.2
0.4
0.6
0.8
1
1 2 3 4 5 6 10 11 16 20 22
0
0.1
0.2
0.3
0.4
0.5
1 2 3 4 5 6 10 11 16 20 22
0
1
2
1 2 3 4 5 6 10 11 16 20 22
1
2
3
4
5
6
1 2 3 7 8 9 10 231 2 3 4 5 6 10 11 16 20 22
Su
cce
ss
Ra
teN
TT
T
(s/c
m)
IP
(bits/s
ec)
Pa
th L
en
gth
(cm
/cm
)
a
b
c
d
Monkey R Monkey J
Experiment Day Experiment Day
trained same daytrained day 1
e
0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
Offline
De
co
de
Accu
acy (
r)
Figure 3.8: Stable performance using a fixed LMP decoderClosed-loop performance of a fixed LMP decoder trained on day 1 of the experiment and thenevaluated on subsequent days. Each gray point shows performance of this fixed decoder for eachsubsequent day’s experiment. On some days we also trained a new decoder from arm reaches; theperformance of this same-day trained decoder is shown in black. On days 2-6 for monkey R thesame day and day 1 decoders performed similarly, such that the black points obscure the gray points.Same day performance corresponds to the data shown in Figure 3.5. (a) Mean success rate. (b)Median normalized time to target. (c) Median Index of Performance (IP), also known as Fittsbits per second. Faster acquisition times and/or accomplishing a harder task yields higher IP. (d)Median path length ratio. (e) Offline decode accuracy when predicting each day’s arm reachinghand velocities from LMP using either the fixed day 1 decoder (gray) or a decoder trained from thatday’s data with 10-fold cross-validation (black)
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 48
in 5/8 individual sessions). In monkey J, however, the hybrid decoder was slightly worse than the
spikes-only decoder (8% slower normalized times to target, p = 0.007, difference not significant at
p < 0.01 in any individual session).
To provide context for these results, I also compared each monkey’s spikes-only and hybrid
decoding performance to his performance on the same task using either his hand (3 and 4 datasets
from monkey R and J, respectively) or the LMP-only decoder (9 and 7 datasets from monkey R and
J, respectively); these data came from different experimental sessions. Figure 3.9 shows that in both
monkeys, BMI performance was better using a spikes-only decoder than an LMP-only decoder. This
difference was much larger for monkey J (298% faster normalized times to target) than monkey R
(51% faster). Using an LMP-only decoder, both monkeys were slower to initially reach the target
and were worse at holding the cursor inside the target acquisition area for the requisite hold time.
Monkey R’s mean time to initial target entry was 50% slower for LMP than spikes (937 ms vs. 624
ms) and his dial-in time was 61% slower (LMP: 672 ms, spikes: 417 ms). Monkey J’s initial target
entry was 273% slower for LMP than spikes (1789 ms vs. 479 ms), and his dial-in was 313% slower
(LMP: 517 ms, spikes: 125 ms). All of these differences were significant at p < 0.001 (two-sided
t-test). I also note that monkey J had very good spike-driven performance, acquiring targets only
18% slower than with his hand.
3.3.6 LMP decoding has a longer control loop latency than spikes decoding
To better understand why closed-loop hybrid decoding only modestly helped in monkey R and did
not help in monkey J, I compared how spikes and LMP each contributed to the hybrid decoder both
during online BMI control and during offline decoding of neural data recorded while the monkey did
the same Continuous Random Target Task with his hand. At each decode update time step, spike
counts and LMP features from all electrodes were input to the decoder to generate the instantaneous
velocity component (‘neural push’) due to each signal (Figure 3.10a). In both monkeys, LMP
contributed more than spikes to the BMI cursor’s movement, accounting for 67% of the decoded
velocity in monkey R and 54% in monkey J (Figure 3.10b). This ruled out the possibility that LMP
was simply not being substantially used in the hybrid decoder. However, the online decode velocity
component from LMP pointed less directly towards the target than the spikes’ velocity component
(Figure 3.10c). Note that one should not interpret monkey J’s smaller absolute LMP angular error
to mean that his LMP signal was more accurate than monkey R’s; rather, monkey J’s more direct
hybrid-driven cursor trajectories (even if they were straighter largely due to better spikes decoding)
also brought down the LMP’s absolute errors. These online results contrast with their offline decode
counterparts. Offline, decoded LMP aimed the cursor more accurately towards the target than
decoded spikes and, not surprisingly, was weighted heavily by the decoder (Figure 3.10e,f).
This observation that the LMP aimed the decoded velocity more accurately than spikes towards
the target during hand reaches, which are direct and precise, but was less accurate than spikes during
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 49
a
b
d
c
1.4
4
1.6
5
5.6
7
1.6
8
1.7
0
2.6
3
4.0
3
2.1
7
0
1
3
6
1.0
0
1.0
0
0.9
9
1.0
0
***
Su
cce
ss R
ate
0
1
NT
TT
(s/c
m)
Pa
th L
en
gth
(cm
/cm
)
hand spikes LMP hybrid
Monkey R Monkey J
1.0
0
1.0
0
0.8
5
1.0
0
*** 1
hand spikes LMP hybrid
5
4
2
0
1
3
6
5
4
2
0.0
45
0.0
53
0.2
09
0.0
57
0
0.1
0.2
0.0
43
0.0
94
0.1
41
0.0
82
0
0.1
0.2
0
25
12
5
51
7
10
1
250
500
750
0
40
41
7
67
2
24
8
0
250
500
750
Dia
l-in
Tim
e (
ms)
Figure 3.9: Closed-loopcomparisons of different neuralcontrol signalsComparison of spikes-only,LMP-only, and hybrid spikes + LMPdecoders. Mean and SEM of eachperformance metric was computedacross trials from multiple datasetsin which the Continuous RandomTarget Task was performed witheither the hand (gray), or a BMIdriven by spikes (red), LMP (blue),or hybrid spikes + LMP (violet).(a) Success rate. The LMP decoderwas significantly worse than othercontrol modes (p < 0.001).(b) Normalized time to target. Alldifferences significant at p < 0.01.(c) Dial in time. All differencessignificant at p < 0.05.(d) Path length ratio. All differencessignificant at p < 0.05 exceptmonkey J spikes vs. hybrid.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 50
***
0
10
20
30
40
online
err
or
ang
le (d
eg
rees)
spikes
LMP
***
spikes
LMP
0 80
−60
120
Y Position (mm)
Y P
ositio
n (
mm
)a b Monkey R Monkey J Monkey R Monkey J
d
c
e f
***
spikes
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Fra
ctio
n D
eco
de C
ontr
ibutio
n O
nline
***p < 0.001
LMP
spikes
LMP
50ms
100ms
250 0 −200Lag (ms)
50ms
100ms
250 0 −200Lag (ms)
20
90
Err
or
An
gle
(D
eg
ree
s)
Monkey R Monkey J
0
10
20
30
40
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Fra
ctio
n D
eco
de C
ontr
ibutio
n O
ffline
Offline
Err
or
Ang
le (D
eg
rees)
spikes
LMP
spikes
LMP
spikes
LMP
spikes
LMP
Monkey R Monkey J Monkey R Monkey J*** ***
***
***
Figure 3.10: LMP underperforms online due to longer control loop latency(a) Example trial showing BMI cursor position (gray circles) in successive 50 ms steps as it movedtowards the target (black disk). The cursor and target are drawn at half-size. Instantaneouscontributions of the spikes (red) and LMP (blue) signals to decoded velocity are shown as vectorsoriginating from the cursor. Filled cursor positions denote the analysis epoch from 200 ms aftertarget onset until when the cursor first entered the target (dashed box). From dataset R.2013.09.27.(b) Mean and SEM of each signal’s relative contribution to the decoded velocity. (c) Angular errorwas computed between each signal’s decoded velocity and the direction pointing directly towardsthe target. (d) We repeated the angular error analysis but varied the lag between each signal’sinstantaneous neural push and “correct” angle to estimate control loop latency. For example, +100ms lag means that error was computed against a vector pointing towards the target from the cursor’sposition 100 ms prior. Mean and SEM angular error for the velocity decoded from spikes (red) andLMP (blue) are shown for different lags. (e) Relative decode contribution of spikes and LMP signalsin the hybrid decoder was computed as in (b) but using data recorded while the monkey performedthe same task with his hand. (f) Error angle was computed as in (c) but using hand control neuraldata. (b-d) are averaged across all trials from full-channel hybrid decoder datasets. (e,f) are averagedacross all hand control trials.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 51
closed-loop cursor control, which is noisier and requires more online correction, led me to suspect
that the LMP signal was slower than spikes to reflect corrective changes in the animal’s movement
intention. To look for evidence of this, I compared the ‘control loop latency’ for spikes and LMP
during closed-loop BMI control by finding the temporal lag that minimized the error angle between
each signal’s instantaneous neural push and a vector pointing directly towards the target from the
lagged cursor position (Figure 3.10d). In other words, I asked how delayed the response of each
signal appeared to be under the (simplistic) assumption that the monkey’s control strategy was
always to push the cursor directly towards the target. For both monkeys, the LMP control loop
latency estimated in this manner was twice as long as that of spikes (100 ms versus 50 ms). These
error-minimizing lags were the same when closed loop BMI data driven by LMP-only or spikes-only
decoders were analyzed (not shown).
3.3.7 Hybrid decoding can rescue performance when spikes are lost
In the previous section, I reported whether combining LMP with spikes improved BMI performance
when both signals were available from all electrodes. We next investigated whether hybrid decoding
could increase the BMI’s robustness to a loss of available spikes information, as would be encountered
with degrading arrays. To simulate this scenario, we performed channel-dropping experiments
in which we trained hybrid and spikes-only decoders after removing the most contributing spikes
electrodes (see Methods for a description of the electrode ranking metric). Closed-loop performance
using these channel-dropped decoders was then evaluated on the Continuous Random Target Task
(Figure 3.11). Due to the previously described differences in spikes-only, LMP-only, and all-electrodes
hybrid performance between the two monkeys, I treated LMP differently for each monkey’s electrode
dropped hybrid decoders. For monkey J, hybrid decoders used spikes from the reduced subset of
electrodes and LMP from all of the electrodes. This simulates a situation where spikes can no longer
be detected on some electrodes, but LMP is still available. For monkey R, I ran a more aggressive
electrode dropping protocol where both LMP and spikes were removed from dropped electrodes; this
simulates complete failure of the electrodes.
My motivation for using these two different protocols is as follows. Recall that monkey J’s
spikes decoder was much better than his LMP decoder, and his all-electrodes hybrid decoder did
not improve upon spikes-only decoding. Monkey J’s arrays recorded good spikes signals from almost
every electrode, which resulted in a shallow spikes electrode contribution curve (Figure 3.12). Even
with many electrodes removed, his spikes decoder still outperformed the all-electrodes LMP decoder
(Figure 3.11). Given this, and my earlier observation that adding LMP to much better spikes could
be deleterious during closed-loop hybrid decoding, it was not surprising that in pilot studies we
found that dropping electrodes for both signals — even when ordered by spikes contribution — still
resulted in monkey J’s spikes-only decoders always beating hybrid decoders (up to the point where
both failed). I therefore concluded that a more informative monkey J electrode dropping experiment
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 52
*0/5 *3/5 *3/5 *3/5 *5/5
0 40 80 120 160
fail
1
Electrodes Removed
(spikes only)
*5/5 *4/5 *4/5 *5/5
0 40 80 120 160 0
0.35
*0/5
2x 5x
Su
cce
ss R
ate
*3/7 *2/6 *5/6 *7/7 *4/4
0 20 30 40 60 80
fail
1
Electrodes Removed
(both spikes & LMP)
NT
TT
(s/c
m)
*5/8 *5/7 *4/6 *6/6 *6/7 *3/4
0 20 30 40 60 80 0
0.35
3x 3x
0.5 0.5
*0/8
Monkey R Monkey Ja
b
Figure 3.11: Performance of spikes-only and hybrid decoders after electrode droppingSpikes-only (red) and hybrid spikes + LMP (purple) decoders were trained and evaluated with thespecified number of electrodes removed in a “worst case scenario” by descending order of contributionto the spikes decoder. For monkey R, we removed both spikes and LMP features from these electrodes.For monkey J only spikes were removed. Spikes and hybrid decoder performances with a givennumber of electrodes removed were compared in the same session. Their mean performances areshown as a red and purple point on the plot; filled points represent a significant difference (p < 0.01)for the paired comparison between decoders on that day. Starred fractions summarize how oftenthis difference was significant out of the total number of days when that condition was tested. Solidlines show the grand mean performance of each decoder as a function of electrode-drop condition.Days when a decoder “failed” (success rate below 0.5) were not included in this grand mean. Dashedlines extend mean lines into the all-failure conditions for illustrative purposes. All-electrodes LMPperformance metrics (mean from Figure 3.9) are shown with horizontal blue lines spanning thoseconditions where all LMP electrodes were available to the hybrid decoder. (a) Success rate. (b)Normalized time to target.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 53
would ask: will hybrid decoding ever be helpful if his spikes recording degrades but his mediocre
all-electrodes LMP is still available?
For monkey R, removing only spikes electrodes (i.e., the monkey J protocol) would not have
been very informative given the following observations. Monkey R’s spikes performance was roughly
1.5 times better than his LMP-only performance (a much smaller difference than in monkey J)
and relied heavily on a small subset of electrodes (Figure 3.12). Once the best few electrodes
were removed, monkey R’s spikes-only decoder became worse than the all-electrodes LMP decoder
(figure 8). Furthermore, from the all-electrodes hybrid experiments I knew that the two signals
already combined beneficially in monkey R’s hybrid decoder, and that this decoder was dominated
by LMP even with all spikes electrodes available. Thus, applying the protocol used in monkey J
to monkey R would have rapidly converged to comparing a reduced-electrode spikes decoder to an
all-electrodes LMP decoder (plus a minor contribution from remaining spikes). This comparison
can be approximated with just a spikes-only electrode dropping experiment (which the data does
include). By instead comparing reduced electrode spikes-only decoders to hybrid decoders in which
both signals are removed from dropped electrodes, I was able to additionally test how much hybrid
decoding could rescue performance in the more dire scenario of completely losing electrodes that had
the best spikes signals.
We tested monkey J with spikes removed from 0, 40, 80, 120, and 160 electrodes and LMP still
available from all electrodes. We found that with 40 or 80 electrodes removed, the hybrid decoder
performed worse than a spikes-only decoder (Figure 3.11). Hybrid decoding started to outperform
spikes-only decoding with 120 electrodes removed, performing superiorly in 3/5 sessions. This 120
electrodes removed condition also corresponds to when spikes-only decoding became worse than
the average all-electrodes LMP-only performance. Once 160 electrodes were removed, the spikes
decoder consistently failed but the hybrid decoder was consistently usable (77-91% trials successful).
However, I note that in the conditions where hybrid decoding outperformed spikes-only decoding, it
was no better, on average, than LMP-only decoding. Thus, in monkey J hybrid decoding improved
robustness to the loss of many spikes electrodes, but not beyond the improvement afforded by just
decoding LMP.
In monkey R, hybrid decoding substantially improved robustness to a complete loss of signals
from electrodes containing the best spikes. I compared hybrid and spikes-only decoders with 0, 20, 30,
40, 60, or 80 electrodes disabled for both spikes and LMP. Hybrid decoders increasingly outperformed
their spikes-only counterparts as more electrodes were removed. This difference was consistently
significant (6/6 sessions) once 40 electrodes were removed; for this condition, success rate was 25%
higher and times to target were 38% faster (6 sessions’ grand mean) using the hybrid decoder,
not including one session where the spikes decoder failed outright. The hybrid decoder remained
consistently usable even with 80 electrodes removed (77-91% trials successful), whereas the spikes-only
decoder started to become unusable once 40 electrodes were removed. Three factors contributed to
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 54
Monkey R Monkey J
Electrode Rank1 192
0
1
Cu
mu
lative
De
co
de
r C
on
trib
utio
n
Electrode Rank1 192
0
1
0 0.040
0.04 Correlation r = 0.16
Spikes weight
LM
P W
eig
ht
0 0.10
0.1 Correlation r = 0.36
Spikes Weight
LM
P W
eig
ht
Figure 3.12: Cumulative electrode contribution to the spikes-only and LMP-onlydecodersThe contribution of each electrode to a decoder is calculated based on that electrode’s weight inthe steady-state velocity Kalman filter multiplied by the average magnitude of the signal from thiselectrode in the training data set. Electrodes are ranked from greatest to least contribution, and thecumulative decoder contribution is plotted as a function of electrode rank, starting from the rank 1(most contributing) electrode. We show the cumulative electrode contribution curves for spikes-only(dark red) and LMP-only (dark blue) decoders built from the arm reaching training data of eachexperiment day in which an all-electrodes hybrid decoder was evaluated (8 datasets for monkey R, 5datasets for monkey J). Bright red and bright blue lines show the means of these individual days’contribution curves. On average, half of monkey R’s spikes decoder weight was contributed by thetop 13 electrodes, compared to 30 electrodes for the LMP decoder. For monkey J, on average 31electrodes contributed half the spikes decoder weight, compared to 50 LMP electrodes. Inset scatterplots show the fractional contribution of each electrode to the spikes decoder and LMP decoders;each point corresponds to one electrode from one session. The mean Pearson’s correlation betweenelectrodes’ fractional contribution to the spikes and LMP decoders is summarized in the inset title.This figure shows that LMP decoder weights were more distributed across electrodes than spikesdecoder weights, and that a given electrode’s importance in the LMP decoder was poorly correlatedwith its importance in the spikes decoder.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 55
why the improvement afforded by hybrid decoding increased as I removed more electrodes. First,
I dropped electrodes in order of descending spikes decoder contribution; second, there was a low
correlation between how important a given electrode was for spikes and LMP decoding (Figure 3.12
insets); and third, decoder weight was much more evenly distributed across the LMP electrodes than
spikes electrodes (Figure 3.12).
3.4 Discussion
The goal of this study was to address the need for motor neural prostheses to be robust to the
loss of recordable spikes. The present work substantially advances this goal by demonstrating the
effective use of LFP as an alternative or complementary BMI control signal. We decoded a velocity
command either solely from multielectrode LMP, or from a hybrid control signal consisting of both
binned spike counts and LMP. We demonstrated that LMP alone is a viable alternative control signal
that enabled effective cursor control in two monkeys. The LMP BMI did not require daily decoder
recalibration, and its performance substantially exceeded that of previously reported LFP-driven
systems. This was the first study to evaluate BMI control using a single type of LFP feature, in
this case the time-varying LFP amplitude — LMP — which is dominated by power below 5 Hz and
requires minimal electrical and computational power to extract. We presented new evidence that
this signal is not a movement-related artifact by showing that hand velocity could not be decoded
from the LMP of sedated monkeys whose arms were passively moved. Finally, we demonstrated for
the first time continuous cursor control using a hybrid decoder combining spikes and LMP. We found
that while hybrid decoding did not improve BMI performance when many electrodes were able to
detect spikes, if this spike-based information was removed from some electrodes (simulating sensor
degradation), then providing LMP signals to the decoder could rescue performance. Altogether, these
results demonstrate that even if multielectrode arrays eventually lose their ability to record spikes
on some or even all electrodes, an intracortical BMI may be able to continue to function effectively
as long as LMP can still be recorded. The methods described here may therefore extend the useful
lifespan of BMI systems, thereby increasing clinical viability.
3.4.1 Choosing LMP for BMI use based on offline decode performance
There are myriad ways to process LFP signals into decodable features, and it would not have
been experimentally practical to develop and test online BMIs driven by each candidate feature
set. Existing offline studies examining the relationship between LFP and reaching kinematics give
differing recommendations as to which features are best for BMI use. One study (So et al., 2014) has
compared closed-loop performance using LFP power in different frequency bands, but that study
reported that the best frequency band was different for each monkey and did not evaluate LMP.
Therefore, to maximize our chance of choosing the right LFP feature(s), I first performed my own
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 56
offline comparisons which found that LMP was the best of the tested LFP-derived features. This is in
accordance with past studies showing that LMP recorded from primary motor and premotor cortices
is informative about kinematics (Mehring et al., 2003; Rickert et al., 2005; O’Leary & Hatsopoulos,
2006; Stark & Abeles, 2007; Bansal et al., 2011a,b; Flint et al., 2012a,b). Interestingly, both our study
and previous studies (Pesaran et al., 2002; Scherberger et al., 2005; Zhuang et al., 2010; Markowitz
et al., 2011) found that power in low-frequency LFP bands is a poor decode feature. This suggests
that even though low-frequency components dominate the LMP, the loss of phase information when
estimating low-frequency power results in a loss of useful information.
The second best LFP feature I tested, power in the 50-300 Hz band, performed considerably worse
than LMP in the offline decode comparison. This contrasts with (Bansal et al., 2011a; Flint et al.,
2012b) which found that high-frequency LFP power was comparable to or better than LMP. Such
differences might be expected if the relative information contained in LMP versus high frequency
LFP power is sensitive to precisely where the signals are recorded. I am unaware of any study that
methodically mapped how the quality of LMP information changes as a function of recording location,
but one study (Markowitz et al., 2011) showed that decoding performance using higher frequency
spectral features depends strongly on cortical recording depth. Similar studies to optimize recording
location for LMP-based decoding would be very useful.
I were surprised to find that my offline decode accuracy using LMP was comparable to decoding
multiunit spikes. Although previous studies have shown better or equal offline decoding using LFP
features rather than spikes from a small number of electrodes (Pesaran et al., 2002; Mehring et al.,
2003; Bansal et al., 2011a,b; Flint et al., 2012a; Hwang & Andersen, 2013), many of the same studies
(Bansal et al., 2011a,b; Flint et al., 2012a; Hwang & Andersen, 2013) and others (Stark & Abeles,
2007; Flint et al., 2012b), reported that at higher electrode counts, multielectrode spike activity is
more informative than multielectrode LFP. My 192-electrode study falls within the upper electrode
count range of these past reports. However, differences in electrode placement, signal processing and
decoding techniques, and the nature of the kinematics being decoded obfuscate direct comparisons
between studies and may explain why my offline decode of velocity was better using LFP than using
spikes. Consistent with previous reports (Mehring et al., 2003; Stark & Abeles, 2007; Bansal et al.,
2011b; Hwang & Andersen, 2013), I found that offline decoding of both LFP and spikes together led
to a modest but significant accuracy improvement. This improvement was greater using LMP than
50-300 Hz power. Although my results do not speak to this directly, one reason why high frequency
LFP power may not add much information to what can already be decoded from multiunit spikes is
that this signal may itself be contaminated by spiking activity (Asher et al., 2007; Waldert et al.,
2013).
Taken together, my offline decode results led me to conclude that LMP was the most promising
candidate LFP feature both as an alternative and complementary signal to spikes for BMI control.
These data also address several concerns that have been raised about the utility of low-frequency
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 57
LFP signals for BMI use. One such concern is that decoding low-frequency LFP may be hindered by
high correlations of both signal and noise between electrodes (O’Leary & Hatsopoulos, 2006; Bansal
et al., 2011b; Flint et al., 2012b; Hwang & Andersen, 2013). While I did observe that LMP from
nearby electrodes had similar tuning, there were a variety of PDs across the two 96-electrode arrays
and I was able to accurately decode single-trial velocity. A second concern is that the signal is an
artifact caused by the animal’s movement. Our sedated animal decode control experiment adds a new
piece of evidence on top of the already-persuasive case made by prior studies (Hwang & Andersen,
2009; Bansal et al., 2011b) that low-frequency LFP is not merely a mechanical or electrical artifact.
A third concern is that LMP would be ill-suited to serve as a neural prosthetic signal if it lags behind
kinematics (i.e., contains mainly sensory information); this has been reported in motor (Bansal et al.,
2011b) and parietal (Asher et al., 2007) cortices. Contrary to these studies, I found that kinematics
were best decoded when LMP was time-shifted to precede kinematics by approximately 100 ms. My
results may differ from (Asher et al., 2007) because they recorded from a different cortical region, and
from (Bansal et al., 2011b) because they appear to have recorded from an area with more sensory
information (they also found that most single units’ optimal decoding lags followed kinematics).
My results are consistent with a different set of studies which found that that LMP is tuned before
movement onset (Mehring et al., 2003; Hwang & Andersen, 2009; Flint et al., 2012a).
3.4.2 Closed-loop LMP-driven BMI performance
This study demonstrated that decoding LMP from two multielectrode arrays can enable effective
BMI cursor control. It adds to the small existing body of closed-loop LFP-driven cursor control work
(Flint et al., 2013; So et al., 2014) by demonstrating improved performance while also characterizing
the quality of control using a single LFP-derived feature (LMP). Differences in monkeys and signal
recording quality inevitably limit how much can be inferred by comparing the performance of different
BMI methods across studies. Nonetheless, I made a best effort to directly compare the performance
of our LMP-driven BMI to that of two previous closed-loop LFP-driven BMI studies on closely
matched tasks. We observed target acquisition rates roughly 50% higher than (So et al., 2014) and
over twice as fast as (Flint et al., 2013). Thus, the work described here represents, to the best of
my knowledge, the highest-performing LFP-driven BMI system to date. I believe that two design
features contributed to better performance using our BMI system. First, we decoded neural activity
from more electrodes than previous studies. We recorded from two 96-electrode arrays (one in M1
and one in PMd), whereas So and colleagues decoded LFP from twenty electrodes randomly chosen
from a 128-electrode array spanning M1 and PMd, and Flint and colleagues recorded from a single
96-electrode array in M1. Second, we decoded LMP rather than spectral power, which according to
my offline decoding analysis better predicts reach velocity. Flint and colleagues also decoded LMP as
one of their features, but did not implement something akin to our half-wave rectification. If either
of their monkeys exhibited the same deleterious LMP after-potential that I observed in our monkey
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 58
R (perhaps partially compensated for by other decoded LFP power features), this may have limited
performance. So and colleagues’ decoders used multiple LFP power bands between 0 and 150 Hz,
but did not decode LMP.
This substantially better LFP-driven performance serves as an important existence proof that
high-quality BMI control is possible using alternative (non-spike) signals from chronically implanted
multielectrode arrays. Importantly, however, we found that our LMP-driven BMI performance was
not as good as spike-driven performance using the state-of-the-art FIT-KF algorithm (Fan et al.,
2014). Thus, while our results demonstrate that LMP is a viable alternative control signal to spikes,
I do not suggest employing LFP-only decoding if good spikes are available. I do note, however,
that the low sampling rate and minimal signal processing required to extract LMP, combined with
its reasonable performance as a closed-loop control signal, may make LMP attractive for power-
or bandwidth-constrained applications such as fully implanted BMI systems (these constraints are
overviewed in e.g., Dethier et al. 2013). The low-pass filtering properties used to extract LMP in
this study can be approximated with a passive analog filter followed by half-wave rectification via a
very low power analog rectification circuit. Downstream processing would only need to sample the
resulting voltage 40 times per second (the decode update time used in this study). This contrasts
dramatically with the order 103 samples per second used to identify single unit spikes. Generally
speaking, recording sorted spikes has the highest power demands, followed by threshold crossings
and higher frequency LFP band power, and finally LMP.
A desirable property of a neural prosthetic control signal is not just the performance that it
affords, but also its stability. BMI use without daily decoder recalibration has been shown when
decoding well-isolated single units (Ganguly & Carmena, 2009), multiunit spiking (Flint et al., 2013;
Nuyujukian et al., 2014), and a broad set of LFP-derived features including both LMP and higher
frequency power (Flint et al., 2013). We found that our LMP-only decoder can also be used without
retraining for at least three weeks. Unlike in (Flint et al., 2013), we did not see long-term performance
improvement when using a fixed decoder. This may be because we did not hold the decoder stable
for as many days or for as long during each day, or because the higher performance in our study
reduced the pressure on the animal to improve his performance.
Our study was similar to that of Flint and colleagues (Flint et al., 2013), but differed from the So
and colleagues study (So et al., 2014), in that we allowed the monkeys to continue to move their arm
during BMI use. This arm-free BMI animal model is better suited for evaluating biomimetic decoders
trained using arm reaching data (Nuyujukian et al., 2011), and unsurprisingly we observed that
monkeys continued to move their arms when using the LMP-driven BMI. In this regard LMP-driven
BMI use is similar to spike-driven control — both in this and past studies such as (Gilja et al., 2012)
— during which monkeys typically continue to move their arms if allowed to. As with any neural
prosthetics method tested using an able-bodied animal model, it remains to be seen how well our
approach will work in paralyzed or amputee patients. While I am encouraged by a recent report
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 59
of movement intention-evoked transient low-frequency LFP modulation in humans with tetraplegia
implanted with similar arrays (Perge et al., 2014), validation of my LMP decoding approach will
require closed-loop evaluation of this method in clinical trial participants.
3.4.3 Hybrid decoding using both LMP and spikes
Decoding spikes or LFP need not be an either-or decision. While it has often been predicted that
hybrid decoding of both signals could improve BMI performance, this is the first closed-loop study
to decode continuous kinematics from spikes and LFP together. In doing so, I faced two decisions:
how should I combine these two signals in a single decoder, and what spikes-only decoder should I
compare the resulting hybrid decoder against? I felt that it would be most impactful to try to improve
performance and robustness above that of a current state-of-the-art spikes decoder. I therefore used
as our baseline the FIT-KF algorithm (Fan et al., 2014) and then extended it to decode both spikes
and LMP. When both spikes and LMP were decoded from all available electrodes, hybrid decoding
enabled a modest improvement in closed-loop performance in monkey R, but was marginally worse
than spikes-only decoding in monkey J. An analysis of each signal’s contribution to the closed-loop
velocity decode provided insight into why adding LMP did not improve closed-loop performance
as much as might have been expected from offline analyses: when the hybrid decoder is fit, LMP
features are heavily weighted because collectively they are as well as or better correlated than spikes
with the open-loop training data kinematics. During closed-loop BMI use, however, even modest
decode errors must be corrected in order to acquire the target; consequently, the longer control loop
latency characteristic of LMP leads to worse control. As a result, the LMP’s contribution to a hybrid
decoder with excellent spikes signals does little to improve — and can even worsen — closed-loop
cursor control. Consequently, achieving a benefit from online hybrid decoding is not a given, but
rather will depend on the relative closed-loop quality of the spikes and LMP signals.
I note two caveats related to my design choices of building the hybrid decoder by extending the
FIT-KF spikes decoder and then comparing it against the spikes-only variant. First, while combining
the spikes and LMP Kalman filters was a principled application of existing high-performing decoding
techniques to the hybrid decoding problem, I make no claim that the resulting decoder — which
determines the relative weighting of spikes and LMP features based on how well each signal regressed
against training data kinematics and how the resulting residuals covaried — is optimal for closed-loop
BMI use. I view this work as an encouraging first foray into hybrid decoding but recognize a need for
future work looking at whether there are better ways to combine spikes and LFP signals. For example,
there may be an opportunity to improve performance by taking into account the different control
loop latencies of LMP compared to spikes, or by decreasing the decoder weights assigned to LMP
features to account for their inferior closed-loop utility. Second, by comparing the hybrid decoder
to what was already a very high performing spikes-only decoder, I set a high bar for performance
improvement. Monkey J, in particular, had many good electrodes with spikes and controlled the
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 60
cursor almost as well with the all-electrodes FIT-KF spikes decoder as with his hand. With these
caveats in mind, the present results suggest that when many electrodes with spikes information are
available, hybrid decoding offers little to no improvement over state-of-the-art spikes-only decoding.
3.4.4 Decoding LMP to mitigate losing spikes signals
Thus far I have discussed how LMP and hybrid decoding compares to spikes-based decoding in the
case of arrays with excellent (monkey J) or mediocre (monkey R) spike signals. But what about
when spike signal quality is poor and insufficient for accurate BMI control? This is oftentimes the
case when arrays degrade over the years following implantation (Barrese et al., 2013; Wang et al.,
2014) and is one of the major impediments to long-lasting, clinically viable neural prostheses. We
experimentally simulated this situation with electrode dropping experiments, and found that the
performance drop resulting from losing spikes information can be mitigated by making use of LMP.
The degree to which LMP provides robustness to spikes loss depends on the relative quality of the
LMP and remaining spikes, as I will now describe.
In monkey J, LMP was in effect a BMI control signal of last resort. This subject had excellent
spike signals on most electrodes, which enabled much better cursor control using a spikes-driven
BMI than with an LMP-driven BMI. Only after a majority of spikes electrodes were dropped did his
spikes-only control become worse than all-electrodes LMP-only control. At that point, both hybrid
decoding and LMP-only decoding (both of which had LMP available from all electrodes) provided
usable cursor control, whereas the spikes-only decoder failed. Importantly, across the electrode
dropping conditions we tested, the hybrid decoder did not outperform whichever individual signal
type (spikes-only or LMP-only) worked best. It is somewhat counterintuitive that adding additional
LMP information to monkey J’s electrode-dropped spikes signal actually reduced performance in the
conditions where spikes still outperformed all-electrodes LMP. This (suboptimal) hybrid decoding
result can be understood in light of the previously discussed observation that LMP and spikes were
comparable for offline decoding but that LMP provided worse closed-loop control. When monkey J’s
most contributing spikes electrodes were removed, the regression to fit a hybrid decoder increasingly
weighed the LMP over the remaining spikes because of LMP’s high offline explanatory power. This
resulted in a hybrid decoder that was (overly) dominated by the inferior LMP signal rather than the
more controllable remaining spikes.
In contrast, utilizing LMP improved monkey R’s BMI control even when spikes were available
from all electrodes, and the LMP became increasingly important for maintaining performance as
spike signals were lost. Monkey R had worse spikes-driven control and better LMP-driven control
than monkey J, such that all-electrode LMP decoding outperformed spikes-only decoding once twenty
or more spikes electrodes were lost. This suggests that when spikes’ performance advantage over
LMP is not large (and is tenuous insofar as it depends on a handful of key electrodes), a modest
degree of array spikes signal decay will result in LMP decoding becoming a preferred alternative to
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 61
spikes decoding. Furthermore, monkey R’s hybrid decoder outperformed the spikes-only decoder
across all electrode drop conditions tested, demonstrating that when closed-loop LMP-driven control
is comparable to (or better than) spikes-driven control, LMP can complement spikes and improve
BMI performance.
Taken together these results provide experimental validation that utilizing LFP (in this study,
specifically the LMP feature) as a BMI control signal can increase a neural prosthesis’ robustness to
the loss of spikes signal. At a minimum, LMP can provide an alternative control signal to spikes (as
in monkey J’s case), which is useful because there is considerable experimental evidence suggesting
that arrays’ LFP signals persist longer than spikes (Asher et al., 2007; Bokil et al., 2006; Flint et al.,
2012b; Heldman et al., 2006; Hwang and Andersen, 2013; Perge et al., 2014). In the more fortuitous
case where LMP-driven performance is quite good (such as in monkey R), LMP can be used not just
as an alternative to spikes, but as a complementary signal for a hybrid decoder that both improves
maximal performance, and slows down the rate at which performance decays when spikes signals are
lost. I am particularly encouraged that in this study, decoding LMP enabled effective cursor control
in both monkeys under conditions where spikes-only decoding was non-functional.
There are several electrode dropping experiment permutations that this study did not explore
but which are worth considering. I did not test monkey R in the case that spikes were lost from
dropped electrodes but LMP was still available from all electrodes. However, the observation that his
all-electrodes LMP-only decoder was better than any of the electrode-dropped spikes-only decoders,
and that LMP and spikes combined beneficially in the all-electrodes hybrid case, strongly predicts
that hybrid decoding would be even better if more LMP electrodes are available. Since the main
goal of this study was to use available LMP signals to increase robustness to a potential loss of
spikes signals, I also did not explicitly explore how these decoders would perform when, instead,
LMP electrodes are lost (i.e., LMP-only electrode dropping experiments, or hybrid-decoder electrode
dropping ordered by LMP contribution). Based on the high correlations between LMP on nearby
electrodes and the shallow LMP decoder contribution curves, I predict that such experiments would
find that LMP and hybrid decoders would degrade slower (as a function of number of electrodes
removed) than spikes-driven decoders. This prediction remains to be tested in future closed-loop
experiments.
3.4.5 Differences between offline and closed-loop results
My experiences carrying out this work add to the growing body of literature reporting that offline
decoding results do not always predict closed-loop BMI performance (e.g. Chase et al., 2009;
Cunningham et al., 2010). Given the field’s heavy reliance on offline decode analyses, I believe it
is worthwhile to enumerate these differences in order to draw attention to factors that may affect
how offline performance translates to closed-loop results. Although my offline metrics showed that
decoding LMP was comparable to or even better than decoding spikes, during closed-loop evaluation
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 62
spikes-driven cursor control was consistently superior. Qualitatively, the LMP-only decoder appeared
more “ballistic” and less precisely controllable than the spikes-only decoder: if the monkey missed
the target with his initial reach, his corrective response was slower and less smooth. This impression
is supported by several observations: LMP-only decoding had longer times to target and dial-in time;
during hybrid decoding, the decoded LMP aimed the cursor less accurately than spikes did; and
the estimated control loop latency was longer for LMP than for spikes. I therefore attribute LMP’s
reduced closed-loop controllability to this signal being slower to reflect changing movement intention
— a limitation that is likely to be shared by other low-frequency signals. To better predict such offline
versus closed-loop differences in the future, I suggest that the datasets used to evaluate candidate
BMI control signals include conditions in which rapid reach corrections are made. This would
enable estimation of the candidate signal’s control lag. LMP’s inferior closed-loop controllability also
explains a second difference between offline and closed-loop decoding: offline, hybrid spikes and LMP
decoding helped in both monkeys, but during closed-loop control this benefit was only realized in
monkey R.
A third example is that decoding LMP without half-wave rectification worked well offline, but
in one monkey half-wave rectification was needed to enable closed-loop performance. This signal
processing step removed an LMP ‘afterpotential’ (visible at t = 0.6 s in Figure 3.1d) which occurred
shortly after the monkey reached towards the target. Without rectification, this afterpotential
generated a substantial cursor velocity during the time period when the monkey was typically trying
to keep the cursor still over the target. After I discovered this problem during closed-loop BMI testing,
I looked more closely at my offline decode results and noticed that the decoded velocities did slightly
increase at the end of the target hold period. However, this deleterious effect was poorly captured in
my offline decode metrics because the afterpotential happened near the end of the trial. Consequently,
the decoded hand velocity was just beginning to increase when the trial ended, preventing further
accumulation of decode error. The next trial’s decode then began with the Kalman state reset to
zero (a common practice to prevent offline decoding errors from accumulating over the course of
many trials), minimizing the afterpotential’s effect on this next trial. In short, the trial structure of
the data I used for the offline study largely hid the problem, but in closed-loop the afterpotential’s
effect was strong enough to dramatically hinder target acquisition.
Offline studies are invaluable for suggesting promising BMI design approaches, but these differences
underscore the need for careful closed-loop validation. Thus, while I specifically investigated
closed-loop LMP-driven performance because this LFP feature worked best offline, these results
do not rule out that other features cannot yield even better closed-loop performance. The nascent
subfield of LFP-driven BMIs would benefit from more studies evaluating other choices of LFP features
and decoding algorithms in closed-loop experiments.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 63
3.5 Methods
3.5.1 Subjects and neural recording
All procedures and experiments were approved by the Stanford University Institutional Animal Care
and Use Committee. Experiments were conducted with adult male rhesus macaques (monkeys ‘R’
and ‘J’) previously implanted with two 96-channel multielectrode arrays (1 mm electrodes spaced
400 µm apart; Blackrock Microsystems, Salt Lake City, UT) using standard neurosurgical techniques
as described in (Santhanam et al., 2006). The arrays were placed in M1 and PMd contralateral
to the reaching arm. Monkey R’s arrays were implanted 24-25 months prior to these closed-loop
and ‘sedated reaching’ (see next paragraph) experiments and 16-20 months prior to collection of the
offline decode datasets. Monkey J’s arrays were implanted 49-50 months before the closed-loop and
sedated reaching experiments and 24 months prior to the offline datasets.
I used an “arm not restrained, not visible” animal model for BMI study, meaning that the monkey’s
reaching arm was unrestrained, but visually occluded by the display, during both arm controlled and
BMI controlled tasks (Shenoy & Carmena, 2014). I believe that this is a good BMI animal model
because it allows training biomimetic decoders from natural reach data, and subsequently allows
the animal to control the BMI cursor by generating whatever neural activity he wants without an
additional constraint that this activity must not cause movement of the arm (Nuyujukian et al.,
2011). As a control experiment to test if LMP could be merely an arm movement artifact, we also
collected datasets in which each monkey was sedated while an experimenter performed the Radial
8 Task by moving the monkey’s hand from target to sequential target. The animal was sedated
with ketamine 2 mg/kg plus dexmedetomidine 0.04 mg/kg IM and positioned in the experimental
rig in the same fashion as during awake-behaving experimental sessions. The experimenter was
electrically insulated from the animal and moved the monkey’s hand so as to approximately match
the animal’s typical reach trajectories; meanwhile, the behavioral task software was running, thereby
allowing for identical analysis of this control experiment data. Neural and kinematics recording,
signal processing, and subsequent data analysis were identical to that of our standard awake-behaving
reaching experiments.
3.5.2 Behavioral tasks
The monkeys were trained to sit head-restrained in a primate chair and perform reaching tasks with
one hand, as shown in Figure 3.1a. A virtual cursor and target were displayed in a 3D environment
(MSMS, MDDF, USC, CA, USA) in front of the monkey by two 120 Hz LCD displays visually fused
by mirrors. However, all of the cursor tasks in this study were confined to a 2D vertical plane.
Custom experiment control software written on an xPC Target platform (Simulink, The Mathworks,
Natick, MA, USA) controlled the behavioral task and updated the monitors with latency of 7 ± 4
ms. The monkey controlled the cursor position either by moving his hand, which was tracked with
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 64
an infrared reflective bead-tracking system at 60 Hz (Polaris, Northern Digital, Canada), or with a
velocity command from decoded neural activity (see below). In all tasks, the monkey earned a liquid
reward after each successful target acquisition.
At the beginning of each experimental session the monkey performed a block of the Radial 8 Task.
In this task, the location of the target alternated on each trial between the center of the workspace
and a peripheral location pseudorandomly chosen from eight locations equally spaced along a circle of
radius 12 cm. To successfully complete a trial, the monkey had to keep the cursor inside a 4 × 4 cm
acquisition area around the target for a ‘target hold period’ of 500 consecutive ms. A new center trial
immediately followed a successful trial or after failure to acquire the target within a 2 s time limit.
These data were used for decoder training during closed-loop experiments, and previously collected
Radial 8 Task datasets were used for offline analyses comparing different LFP-derived neural features.
Unless stated otherwise, we evaluated closed-loop performance on the Continuous Random Target
Task. At the start of each trial, the target appeared in a random location anywhere in a 20 × 20
cm box centered in a bounded workspace that was itself 40 × 32 cm. The monkey had up to 8 s to
acquire the target by keeping the cursor within a 5 × 5 cm acquisition area for a target hold period
of 500 ms (monkey R) or 300 ms (monkey J).
We also evaluated performance on two additional tasks to compare our BMI to those of prior
LFP studies. The first was a variant of the Continuous Random Target Task with target size reduced
to 4 × 4 cm, trial time limit of 10 s, and hold time reduced to 100 ms, as in (Flint et al., 2013).
Due to differences in animal training, unlike in that study we did not fail a trial if the cursor left
the target within 100 ms and later came back to make a successful acquisition. Instead, I post-hoc
labeled these trials as failures. This is a more punitive metric of our subjects’ performance because
the animals did not have feedback to strongly incentivize them to avoid leaving the target. The
second task was a variant of the Radial 8 Task similar to (So et al., 2014) with 3.4 cm wide targets
located around a 13 cm diameter circle. Acquiring the center target initiated a trial but only reaches
to the peripheral targets were analyzed. I based our variant of the So and colleagues study’s task on
an earlier report (Dangi et al., 2013) and therefore used a 300 ms hold time and 8 second time limit.
I note that in (So et al., 2014) the hold time was actually 400 ms and the time limit was 10 seconds.
3.5.3 Signal processing
Voltage from each of 192 electrodes was initially sampled at 30 ksps by a Cerebus system (Blackrock
Microsystems) and analog filtered from 0.3 to 7500 Hz with a 3rd order Butterworth filter (Figure
3.1b). To extract multiunit spikes (also known as ‘threshold crossings’), the raw neural data from
each electrode was digitally filtered from 250 to 7500 Hz (3rd order Butterworth). A spike was
detected whenever the signal went below a threshold set at the beginning of each recording session
to be -4.5 times the root mean squared value of the electrode’s voltage (Chestek et al., 2011). Spike
counts in non-overlapping bins from each electrode formed a vector of spike features computed at
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 65
each decoder time step. For all offline analyses this decoder time step was 50 ms. Prior studies
have found that such short time steps lead to better closed-loop control than the longer values more
commonly used in offline decoding studies (Cunningham et al., 2010; Gilja et al., 2012). For monkey
R’s closed-loop experiments we used the same 50 ms time step, while for monkey J we used a time
step of 25 ms based on the observation that his spikes-only performance improved with this even
shorter time step. Spikes extraction from disconnected electrodes or electrodes with no spikes was
disabled; depending on the dataset, 168-174 electrodes recorded spikes in monkey R and 188-192 in
monkey J.
LFP was obtained by digitally low-pass filtering the same raw neural data below 500 Hz (4th
order Butterworth) and then downsampling to 2 ksps. For my offline decoding analysis, I evaluated
a number of candidate LFP frequency bands in addition to the time-domain local motor potential
(LMP), which is described in the next paragraph. I created a vector of LFP power features at each
50 ms time step for each channel by taking the mean power of the causally band-pass filtered signal
(3rd order Butterworth) from the previous 50 ms of LFP. Because a 50 ms window does not capture
a full period of the signal for bands with minimum frequency fmin below 20 Hz, for these features
I used overlapping bins of 1/fmin seconds slid every 50 ms. The amplitude response of the filter
cascades used to generate each candidate LFP feature, including the effects of the shared initial
signal processing described above, are shown in Figure 3.2b. These were computed by simulating
the analog and digital filters applied by the Cerebus signal processor, and then analyzing the series
of these filters, plus the LFP frequency band filter of interest, using MATLAB’s cascade and filter
visualization (fvtool) tools.
To obtain the LMP feature from a given channel, LFP samples were first clipped at ±300µV to
mitigate intermittent noise bursts. The LMP was then computed as the sliding mean voltage from
the past 50 ms of LFP (Figure 3.1c). The LMP is thus the low-pass filtered LFP amplitude (Figure
3.2b). For offline decoding the LMP was decoded directly as described above, but in pilot closed-loop
studies in monkey R I found that an additional step was needed to enable effective continuous cursor
control. I observed that on most electrodes, the negative voltage deflection at the start of the reach
was followed by a positive ‘after-potential’, as seen in the top panel of Figure 3.1d. During closed-loop
BMI use this led to the cursor “springing back” in the opposite direction shortly after a movement
was initiated. This markedly impaired performance (Figure 3.13). I therefore applied a half-wave
rectification step where all LMP input values below 0 µV were rectified while positive input values
were set to 0, as illustrated in the bottom panel of Figure 3.1d. While zeroing positive voltages on all
electrodes worked well across the four arrays used in this study, I anticipate that in some situations
the sign of this rectification step may need to be set on a electrode-by-electrode basis depending on
each electrode’s initial LMP polarity. To provide the reader with further intuition about the LMP
signal, Figure 3.2c shows the spectral content of the LMP used to control the BMI on two different
electrodes from representative experimental sessions. These spectra were generated by applying the
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 66
MATLAB fft function to the full duration of each electrode’s LMP during BMI task. The spectra
were binned into 1 Hz bins, with figure tick locations correspond to the maximum frequency of each
bin. The signal magnitude within each frequency bin was then normalized by total magnitude over
all frequencies.
Block Within Session
0
0.2
0.4
0.6
0.8
1
Monkey J
26
0 tria
ls
Block Within Session
0
0.2
0.4
0.6
0.8
1
Su
cce
ss R
ate
Monkey Rwith half-wave LMP rectificationwithout rectification
25
6 tria
ls
25
5 tria
ls
25
8 tria
ls
28
7 tria
ls
25
4 tria
ls
25
7 tria
ls
25
7 tria
ls
22
0 tria
ls
23
7 tria
ls
25
2 tria
ls
26
2 tria
ls
Figure 3.13: Closed-loop LMP decoding benefited from half-wave rectification
We evaluated performance on the Continuous Random Target Task using LMP decoders either with
the half-wave rectification step (black) or without this step (grey). When used, half-wave rectification
was applied both to the training data and during closed-loop use. The two decoders were compared
in alternating blocks within the same experimental session. Success rates for each block are shown
in chronological order within an experiment session. The number of trials in each block is labeled.
Monkey R’s overall success rate was significantly higher with half-wave rectification (p < 0.001,
binomial test), but monkey J’s was not (p > 0.05).
3.5.4 Neural decoding
A velocity Kalman filter (KF) was used to decode cursor velocity from neural activity for both offline
decoding and closed-loop BMI experiments. The details of the velocity KF as applied to BMIs are
well described in Wu et al. (2006); Kim et al. (2008); Gilja et al. (2012); here I enumerate specific
design choices used for the present study. Briefly, the KF estimates the state of a linear Gaussian
dynamical model where the kinematics state at time t, x(t) evolves from the previous time step’s
state and relates to neural activity y(t) according to,
x(t) = Ax(t− 1) + w(t) w(t) ∼ N (0,W) (kinematics state update) (3.5.1)
y(t) = Cx(t) + q(t) q(t) ∼ N (0,Q) (neural observation) (3.5.2)
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 67
where x(t) is the velocity vector [vx(t), vy(t), 1]T (the 1 is an offset term to accommodate non-zero-mean
neural data), y(t) is a vector consisting of the neural feature(s) from every channel at time t, A is the
matrix describing the linear kinematics dynamics, C is the observation matrix mapping kinematics
state to neural activity, and w(t) and q(t) are Gaussian noise. A,C,W, and Q are fit from binned
velocity and neural feature(s), from the start to the end of each successful trial in the training dataset.
Once the model is learned, decoded kinematics can be inferred at every time step using the KF,
which is a closed-form recursion. When regressing neural data against kinematics to fit C, I allowed
the decoder to try different causal lags (i.e., neural data either aligned with or leading kinematics)
for each type of neural feature: spikes, LMP, or LFP power in different frequency bands (which were
computed using either 50 ms bins or, for lower frequency bands, longer bins slid every 50 ms). This
lag was applied globally to all electrodes. For the final decoder I fit parameters with whichever lag
minimized the residual. For LMP decoding the best lag was 100-125 ms (depending on the dataset),
while for spikes decoding it was 50 ms in monkey R and 150 ms in monkey J. However, when fitting
spikes decoders for closed-loop evaluation I used a 0 ms lag so as to be consistent with the methods
used in (Gilja et al., 2012; Fan et al., 2014). The offline spikes decoding difference between using the
best lag and 0 lag was small and not significant: correlation between true and reconstructed velocity
r = 0.73 vs. r = 0.72 for monkey R, and r = 0.80 vs. r = 0.77 for monkey J (p > 0.05 for a difference
in means, paired t-test). For closed-loop LMP decoding, I only applied the 100-125 ms lag to the
training data used to fit the models; no lag was used during closed-loop use. This choice follows with
the general control systems design principle that minimizing control loop lag improves performance
when feedback is used to change the control signal (Kuo, 1987), as is the case for closed-loop BMI
use. Reducing lag has been empirically shown to be best practice in the context of motor control
(MacKenzie & Ware, 1993) and BMI use specifically (Cunningham et al., 2010; Gilja et al., 2012;
Willett et al., 2012). By applying a lag in the training data only, I was trying to better align neural
correlates of movement intention with kinematics. During closed-loop use, I wanted the observation
of similar neural activity to then generate intended cursor velocity with as little delay as possible.
I also applied an additional operation during LMP-driven BMI use to help stabilize the cursor in
the workspace. A ‘centering velocity‘ pointing towards the center of the workspace was always added
to the decoded velocity, with magnitude 0.15 · r s-1, where r is the distance from the cursor to the
workspace center. Thus, the intervention was minimal when the cursor was near the workspace center
but increased when the cursor was near the edges. I emphasize that this assistance is not dependent
on target location; the centering velocity always moved the cursor towards coordinate (0,0), whereas
targets could appear throughout the workspace. This was helpful because a velocity-only decoder can
accrue a position bias over time; the centering velocity ameliorates this bias. The average centering
velocity applied during LMP-driven cursor control was up and to the left in both monkeys (monkey
R: −1.34± 0.19 mm/s x-velocity, 2.33± 0.30 mm/s y-velocity, grand mean ± std across datasets;
monkey J: −1.14 ± 0.23 mm/s x-velocity, 3.32 ± 0.50 mm/s y-velocity. All y-velocity means and
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 68
monkey R’s x-velocity mean were significantly different [p < 0.01] from zero by two-sided t-test). This
indicates that there was a down and to the right decoder bias. The centering velocity gain value of
0.15 was chosen based on a cursory parameter sweep during pilot experiments but was not thoroughly
optimized, either in magnitude or in directionality (i.e., the centering velocity was symmetrical
around the center and thus nave to the particular subject’s typical bias); a more sophisticated bias
correction scheme is a potential avenue for future work. I also sometimes applied this centering
velocity when evaluating hybrid decoders in the electrode-dropping experiment described later. To
prevent this from potentially giving hybrid decoders an undue advantage over spikes-only decoders,
I evaluated spikes-only decoders both with and without the same centering velocity and recorded
the higher performance of the two. Note that this conservative methodology slightly favors the null
hypothesis that hybrid decoding does not perform better than spikes-only decoding.
To evaluate whether decoding both LMP and spikes together would improve closed-loop performance,
I chose to start with a state-of-the-art spikes decoder and then compare it to a hybrid spikes + LMP
decoder. For the spikes-only decoder I used our previously reported Feedback Intention Trained
Kalman filter (FIT-KF) (Fan et al., 2014). The FIT-KF improves upon a standard velocity KF by
adjusting kinematics of the training data to attempt to better match the subject’s true intent: cursor
velocity is rotated to always point towards the target and is set to zero during target hold epochs.
Unlike in (Fan et al., 2014), here I kept a number of design choices from the original ReFIT-KF
decoder (Gilja et al., 2012) from which FIT-KF is descendent: I trained the decoder using Radial 8
Task reaching data, and used data starting from the beginning of each trial.
The hybrid spikes + LMP decoder was built by combining the spikes FIT-KF and LMP velocity
KF models, which were fit from the same training data. This decoder operated on a stacked feature
vector y = [yspikes;yLMP] which was mapped to kinematics by a stacked Chybrid = [Cspikes; CLMP].
Since Aspikes was fit from intention-estimated velocities, it differed slightly from ALMP; I set Ahybrid =
Aspikes and Whybrid = Wspikes. A combined covariance matrix Qhybrid was then calculated from the
training data and Chybrid, yielding the complete KF parameters {Ahybrid,Whybrid,Chybrid,Qhybrid}.Note that during closed-loop use, position feedback subtraction (Gilja et al., 2012) was performed on
the observed spike counts as in FIT-KF, but not on the observed LMP.
3.5.5 Performance measures
The primary metrics I used to quantify closed-loop task performance on the Continuous Target
Acquisition Task were success rate and normalized time to target (NTTT), which directly affect the
animal’s rate of liquid reward. Success rate is defined as the number of trials in which the target was
successfully acquired, divided by the total number of trials. NTTT is the time duration between
trial start and successful target acquisition (not including the required target hold time), divided
by the straight-line distance between this trial’s target center and the previous trial’s target center
(Suminski et al., 2010). NTTT is only computed for consecutively successful trials. Unless otherwise
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 69
stated, when comparing two different decoders, I first compared success rates using each decoder
(binomial test). If success rates were not significantly different, I then compared the NTTT with
two-tailed t-tests for a difference in means. A threshold of p < 0.01 was used for both comparisons.
Chance performance levels were calculated under the null hypothesis that the cursor’s movements
were not goal-directed towards acquiring the displayed target. I did this by replaying the time series
of actual cursor trajectories recorded during the BMI session while randomly selecting virtual target
locations drawn from the set of all targets presented during the experiment. A virtual trial ended
successfully if the cursor stayed within the target for the requisite hold time, or unsuccessfully if the
8 s time limit was exceeded. At the end of each virtual trial the next random target was selected and
the kinematics playback continued. Chance success rate and median NTTT were then calculated
by averaging across 1,000 repeats of this target-shuffled simulation. This method of calculating
chance performance differs from the two methods used in (Flint et al., 2013) and (So et al., 2014); in
particular, it yields a comparatively higher chance success rate when trial time limits are long, which
is tempered by long chance times to target. I chose to use this method because it is less prone to
underestimating chance success rates of undirected kinematics which nonetheless traverse much of
the workspace.
I also computed three additional metrics of successful trials’ kinematics. Index of performance (IP)
is a throughput metric for human-computer interface devices motivated by Fitts’ law (Fitts, 1954)
and is specified in the ISO 9241 standard (Douglas et al., 1999). It was adapted to two-dimensional
tasks (MacKenzie & Buxton, 1992) and adopted by the BMI field (e.g. Simeral et al., 2011; Gilja
et al., 2012). IP is calculated for each trial and rewards faster target acquisition while taking into
account differences in target size and distance:
IP =ID
(time to target), (3.5.3)
where ID is the index of difficulty which depends on this trial’s reach distance, D, which is the
Euclidean distance between the cursor’s position at the start of the trial and its position when the
target is successfully held (Simeral et al., 2011), and the width of the target, W :
ID = log2
(DW
+ 1). (3.5.4)
‘Dial-in time’ is calculated for each trial and measures the time between when the cursor first entered
the target acquisition area and when it last entered the target area prior to successfully staying inside
the target area for the requisite hold time 2012a. A dial-in time of 0 ms means that the monkey
reached and successfully held the cursor inside the target on his first attempt. A longer dial-in time
signifies that the cursor left the target acquisition area too soon, which required the monkey to
reacquire the target. Lower mean dial-in times signify better ability to stop the cursor over the target
and hold it there.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 70
Path length ratio (sometimes also referred to as ‘distance ratio’ or the inverse of ‘path efficiency’)
is the ratio of the distance the cursor actually travelled from the beginning of the trial to when it
entered the target acquisition area, divided by the straight-line distance between the cursor’s position
at the start of the trial and the edge of the target acquisition area. Path length ratios closer to 1
signify straighter cursor trajectories.
3.5.6 Electrode preferred direction and decoder contribution
Data from the hand-controlled Radial 8 Task were used to compute the directional tuning of LMP, as
well as each electrode’s contribution to both the LMP and spikes velocity decoders fit from this data.
To compute a simple measure of preferred direction and tuning significance for a given electrode, I
first averaged this channel’s LMP over an analysis epoch spanning 200 to 600 ms into each trial. For
context, the median and standard deviation of movement start times (defined as when hand speed
reached 10% of the trial’s peak speed) was 144± 90.3 ms for monkey R and 222± 68.6ms for monkey
J. The hand entered the target after 511± 196 ms (monkey R) and 495± 141ms (monkey J). The
electrode’s LMP averaged over the analysis epoch yielded a single data point per trial. I then grouped
trials by reach target and performed a one-way ANOVA to test whether LMP was significantly
different for reaches towards at least one of the eight targets. A measure of each electrode’s preferred
direction was computed by summing eight individual vectors that pointed in the prompted reach
directions, where each vector’s length equaled this channel’s average LMP across all trials where the
monkey reached in that direction.
The magnitude of each electrode’s contribution to the decoder (Nuyujukian et al., 2014) was
used both for determining electrode dropping order and for comparing the cumulative electrode
contribution curves of spikes and LMP decoders. To find each electrode’s contribution to a particular
decoder I first converted that Kalman filter to a closed-form steady-state (Malik et al., 2011),
x(t) = M1x(t− 1) + M2y(t), (3.5.5)
where the jth column of M2 is the decoder weight vector wj = [wx,j , wy,j , 0]T that describes how
yj(t), the neural activity observed on electrode j, contributes to x- and y- velocity. Electrode j’s
decoder contribution is then computed as that electrode’s average effect on cursor velocity:
(contribution of electrode j) =∥∥E[wj · yj(t)]
∥∥ (3.5.6)
where the expectation is taken over time. An electrode’s relative contribution to the decoder is then
calculated by dividing that electrode’s contribution by the sum of all electrodes’ contributions. In
the electrode dropping experiments, for each condition we removed the top n electrodes ranked in
order of most to least contribution to the all-electrodes spikes decoder.
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 71
3.5.7 Comparing the contribution of LMP and spikes in the hybrid decoder
I dissected LMP’s and spikes’ respective contributions in the hybrid decoder using the steady-state
decoder form described above. At each 50 ms time step, the instantaneous contribution to decoded
velocity (‘neural push’) x(t) was divided into:
xspikes(t) = M2 · [yspikes(t); 0], (3.5.7)
xLMP(t) = M2 · [0;yLMP(t)], (3.5.8)
where yspikes(t) and yLMP(t) are the spikes and LMP feature vectors, respectively, and yspikes(t)
incorporates the position subtraction operation of the ReFIT-KF (Gilja et al., 2012). The relative
contribution of each signal (LMP or spikes) was then computed by dividing the magnitude of its
velocity contribution by the sum of both signals’ velocity contribution magnitudes. This metric
was averaged across all time steps from 200 ms after target presentation until when the cursor first
acquired the target.
I also used the partial decoded velocities xspikes(t) and xLMP(t) to find the angular error between
the direction pointing directly from the cursor towards the target and the direction that each signal
pushed the cursor. I report the mean angular error for LMP and spikes averaged across all time steps
included in the decoded velocity contribution analysis described above. This decode error metric
assumes that at each time step the animal’s true intended velocity points directly from the current
cursor position towards the target. However, due to delays in the control loop (of both biological and
technical origin), the neural signals observed at time t likely reflect an intended velocity towards
the target from the cursor’s earlier position at time t− τ . I therefore also computed angular error
as described above across different lag values of τ . I was particularly interested in the difference
in error-minimizing τ between the spikes and LMP signals because this could reveal differences in
how quickly the animal modulates each type of control signal. For this analysis I used each trial’s
data starting from 300 ms after target presentation (rather than 200 ms, as above) to allow testing a
wider range of τ .
For the analyses of closed-loop hybrid decoder use, all experimental sessions evaluating the
all-electrodes hybrid decoder were included. For the matched offline hybrid decoding comparisons of
Figure 3.10e,f,, I analyzed data from experimental sessions where the monkey performed the same
Continuous Random Target Task with his hand while neural data was recorded (812 trials from 2
sessions for monkey R, 1721 trials from 4 sessions for monkey J).
3.6 Project Acknowledgements
This work was supported by National Science Foundation’s Graduate Research Fellowships (SDS, JCK)
and IGERT award 0734683 (SDS), Stanford Medical Scientist Training Program (PN), NIH Pioneer
CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 72
Award 8DP1HD075623 (KVS), NIH T-RO1 award NS076460 (KVS), and DARPA REPAIR award
N66001-10-C-2010 (KVS). I thank Mackenzie Mazariegos, John Aguayo, Michelle Wechsler, Claire
Sherman, Erica Morgan, Liana Yates for expert surgical assistance veterinary care, Chandramouli
Chandrasekaran for Simulink real-time code to access LFP data, Boris Oskotsky for IT support, and
Beverly Davis, Evelyn Casteneda for administrative assistance.
Chapter 4
Effect of Sensory Error Signals on
BMI Use
4.1 Summary
Movement is typically a closed-loop process in which sensory feedback conveying errors is transformed
into motor corrections. This creates a constraint for areas such as motor cortex, which both receive
error signals and generate efferent motor commands: error-related modulation must not leak out to
the effector unless it would generate the appropriate correction. While this is the case for natural
movement, in which motor cortex indirectly affects muscles, what happens when motor cortical
activity directly drives the effector via a brain-machine interface (BMI)? We tested this by perturbing
the kinematics of monkeys performing a BMI-controlled cursor task. Visual feedback drove cortical
responses as early as 70 ms after the perturbation, but this initial modulation did not cause deleterious
motor feedback. Rather, it was limited to an output-null subspace which does not effect the BMI
effector. When neural activity later changed in the output-potent dimensions (115 ms), it caused
corrective BMI movement.
4.2 Introduction
Motor cortex plays a causal role in generating output during both natural motor control and
intracortical BMI control. But, a major difference exists between these two contexts: during BMI use,
cortical activity directly affects the BMI with no intermediate processing except that imposed on the
recorded signals by the experimenter (schematized in Figure 4.1). In contrast, motor cortical output
to the real arm is modulated by other cortical, subcortical, cerebellar, and spinal intermediates (Wise
et al., 1997; Scott, 2004; Ramnani, 2006; Cisek & Kalaska, 2010; Shmuelof & Krakauer, 2011) as
73
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 74
BMI Control
Perturbation
computation
sensoryfeedback
directmotoroutput
computation
sensoryfeedback
Perturbation
indirectmotoroutput
Natural Arm Control
Figure 4.1: Should we worry about error signals affecting ongoing control?During natural motor control, cortical output to the arm is modulated by multiple intermediaries.In contrast, during BMI use motor cortex directly drives the BMI and also participates in ongoingfeedback control computations. The principal question of this study is whether the direct feedbackloop between motor cortex, BMI output, and sensory information about the state of the effector willcause error-related cortical modulation to deleteriously affect ongoing control.
well as internal output-gating mechanisms (Kaufman et al., 2014). What are the consequences of
this difference in light of the fact that motor cortex is also strongly modulated by sensory feedback
carrying motor error signals (Georgopoulos et al., 1983; Archambault et al., 2011; Pruszynski et al.,
2014)? During natural arm use, sensory signals are transformed into the appropriate outputs before
they affect muscles; motor cortical activity related to error signals does not deleteriously affect
ongoing control. Put more plainly, despite motor cortex responding vigorously when something
bumps into the arm, this does not cause immediate and undirected arm flailing.
It remains unknown whether during BMI use, error-related modulation of the same cortical
population driving the BMI output will feed back out to the effector as a maladaptive command.
If this is the case, this unwanted feedback will present an obstacle to real-world use of physical
BMI-driven prostheses such as robotic arms, which will interact and collide with other objects in
the environment. To answer this question, we used the longstanding motor neuroscience strategy
of applying a carefully controlled, discrete perturbation to the motor system and observing how
it responds to — and corrects for — errors (Wolpert & Miall, 1996; Scott, 2004; Shadmehr et al.,
2010; Pruszynski et al., 2011b). We introduced perturbations to the comparatively better-studied
experimental preparation in which monkeys perform a 2D BMI cursor control task. Specifically, we
added a sudden cursor position offset (‘cursor jump’) at different phases of the trial while the monkey
was trying to bring the cursor to a target. Although several groups have studied how the sensorimotor
system responds to a continual perturbation consisting of changing the neural-to-kinematic mapping
of the BMI decoder (Jarosiewicz et al., 2008; Chase et al., 2012; Sadtler et al., 2014; Golub et al.,
2015), to our knowledge this is the first time that the response to a step perturbation has been
studied. These discrete and severe perturbations allowed us to observe how neural dynamics and
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 75
BMI output evolved following the error-causing event and answer key questions about when and how
sensory feedback leads to corrective BMI output.
4.3 Results
We investigated the effect of error signals on ongoing BMI control using a ‘Cursor Jump Task’. Two
monkeys (‘J’ and ‘L’) performed a 2D target acquisition task in which we occasionally perturbed their
neurally-controlled cursor by introducing a large instantaneous 6 cm position offset orthogonal to the
12 cm vector from the workspace center to the target (Figure 4.2a). The perturbation could be in
one of two directions and, if a particular trial was randomly chosen for perturbation, the jump could
happen after one of four ‘jump location conditions’ was met: after the cursor traveled 3, 6, or 10 cm
along the ‘Task Axis’ connecting the center start location to the radial target, or 250-300 ms into the
500 ms target hold period required for successfully completing the trial. Since the context of these
jump location conditions was rather different (starting movement, mid movement, acquiring target,
or trying to keep the cursor still), we analyzed trials for each jump location condition separately.
Both monkeys performed the BMI Cursor Jump Task well, with individual dataset success rates
ranging from 99.0% to 99.9% in monkey J and 96.0% to 99.2% in monkey L. Figure 4.2b shows
perturbed trial cursor trajectories from a representative dataset. Following perturbations during
movement, trajectories initially continue along the Task Axis. After a reaction time delay, they curve
towards the target. After jumps during the hold period, the cursor takes a mostly straight trajectory
back towards the target. Although we will examine the changes in neural activity in more detail later,
these cursor trajectories already suggest that there is little to no deleterious cursor deviation due to
sensory feedback signaling the error interfering with ongoing BMI control. Cursor jumps increased
the time it took the monkeys to acquire the target (Figure 4.2c), with the penalty increasing for the
cursor jump conditions occurring later. Monkey J’s time to target penalty (compared to unperturbed
trials) ranged from 16% (jump after 3 cm) to 58% (jump during hold), and monkey L’s penalty
ranged from 19% to 59%. This ordering by jump location is not surprising, given that the geometry
of the task meant that earlier perturbations allowed the monkeys to “cut the corner” when making
corrections back to the target.
No additional training was needed for these monkeys — who had years of prior BMI-controlled
cursor task experience — to adjust to the Cursor Jump Task; my impression was that they were
completely nonplussed by the perturbations. This is corroborated by there being no significant
correlation between chronological experimental session number and mean time to target for any jump
location condition (p > 0.05, linear regression, for both monkeys).
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 76
12
85
15
27
16
84
20
23
20
44
0
1000
2000
10
43
12
11
13
42
16
46
16
47
Tim
e T
o T
arg
et (m
s) Monkey J Monkey L
X Position (cm)
Y P
ositio
n (cm
)
10630-3-6-10
-10
0
10
Task Axis
Pe
rtu
rba
tio
n A
xis
Threshold Crossings
DecodedVelocity
Cursor Position
VisualFeedback
∑
KalmanFilter
Perturbation
Time
Δ P
ositio
n
{
MotorCortex
a
c
b
No jumpJump at 3 cmJump at 6 cmJump at 10 cmJump during hold
Figure 4.2: BMI cursor jump task(a) We observed how a BMI user respondsto perturbations of their BMI-controlledcursor. Monkeys were implanted withmultielectrode array(s) and trained toperform a 2D cursor acquisition task.Cursor velocity commands were decodedfrom multiunit spikes using a Kalman filter.After decoder training, the monkeysperformed a two-targetcenter-out-and-back task. On a random25% of outward trials, we applied aperturbation consisting of a 6 cm positionstep between successive monitor updates.Perturbations were randomly selected tooccur at one of four location/times duringthe trial, and in one of two directions.(b) Cursor trajectories from one exampledataset showing the cursor jump trials(colored traces) and 100 of the unperturbedtrials (gray traces). The perturbation wasapplied once the cursor had traveled 3 cm(green), 6 cm (orange), or 10 cm (blue)along the Task Axis towards the target, orduring the target hold period (violet).Dataset J.2015.01.19.(c) Perturbations added a modest time totarget penalty that was larger for laterperturbations. Bars show the mean SEMof time to target, not including the finalhold time and not including the “stolen”hold time for jump during hold trials. Alldifferences were significant (p < 1e-5,two-sample t-test) except Jump after 10cm vs Jump During Hold.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 77
4.3.1 Perturbations evoked an early motor cortical response
without accompanying cursor velocity output
We now examine in more detail how motor cortical neural activity changed following a cursor jump
perturbation. Figure 4.3a shows the change in firing rates, across the recorded population, between
perturbed and unperturbed trials to the same target. We could detect statistically significant firing
rate changes (p < 0.001, shuffle test) as early as 69 ms after the perturbation in monkey J and as
early as 111 ms in monkey L. This error-related response took the form of an initial transient which
diminished somewhat before being followed by a larger change that persisted for the duration of the
cursor trajectory correction. These changes involved both firing rate increases and decreases across
the neural ensemble, with a preponderance of increasing rates (Figure 4.4).
As described so far, our analysis of the error-related response captures the magnitude of changes
across all electrodes, regardless of how these changes affected the BMI. We next examined the same
neural activity changes through the lens (projection matrix, to be more literal) of the BMI decoder.
Since the relationship between recorded neural activity and velocity was completely specified, we
could examine exactly how the perturbation-evoked neural response affected decoder output by
analyzing its ‘neural push’ (see Methods). Analysis of this neural push (Figure 4.3b and Figure
4.5) allowed us to make two key observations. First, the early change in firing rates did not affect
decoded velocity: neural push did not deviate from that of unperturbed trials until at least 35 ms
after the early firing rate changes (the precise time of decoder-potent response is calculated in the
next section). Second, once the neural push did start to change, it was a corrective push opposing
the perturbation (positive deflection of the Perturbation Axis traces in Figure 4.3b). The concurrent
negative deflections in neural push along the Task Axis (most visible for monkey L) correspond
to a reduced push along that dimension due to turning towards the target. The exception to this
was monkey J’s jump during hold condition, in which there was a brief and small initial push away
from the target before the error-related response began to push the cursor towards the target. This
maladaptive response, which was nonetheless still well after the earliest firing rate change for this
condition, may reflect a slight “leak” in what otherwise appears to be strong separation between
early error-related processing and BMI output. We now discuss this separation in more detail.
4.3.2 Initial error-related neural activity was confined to a decoder-null
space
Why was the early error-related change in firing rates invisible from the perspective of the BMI
output? We hypothesized that the changing neural activities were canceling out, i.e. occurring in
the null space of the decoder’s mapping from firing rates to velocity (schematized in Figure 4.6a).
Inspecting the neural responses to specific perturbation conditions on individual electrodes (e.g.
Figure 4.6b,c) suggested that different electrodes’ early firing rate changes generate velocity push
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 78
0 200 400 600
−1
0
3
Time After Cursor Jump (ms)
Δ N
eura
l P
ush T
ow
ard
s
Targ
et
(cm
/s)
0 200 400 600
−1
0
6
Task AxisPerturbation Axis
0 200 400 600-2
0
10
111 ms
∆ F
irin
g R
ate
(H
z)
0 200 400 600-2
0
6
69 ms
Time After Cursor Jump (ms)
a
b
Monkey J Monkey L
Figure 4.3: Rapid motor cortical perturbation response without deleterious BMIoutput(a) Change in multiunit firing rates attributable to the cursor jump was computed by subtractingunperturbed trials’ mean firing rates from the responses following the jump. Each trace shows thevector norm of this difference across the 192 (monkey J) or 96 (monkey L) electrodes, averaged acrossall target/jump-direction/dataset conditions. Colors correspond to different perturbation conditionsas in Figure 4.2. Vertical lines mark the earliest detectable change in neural activity (p < 0.001).(b) Change in ‘neural push’ following the perturbation, i.e. how the neural response contributedto change in cursor velocity (the actual cursor trajectory also depended on the momentum termintroduced by the Kalman filter). Neural push is projected separately onto the Perturbation Axis(solid traces) and Task Axis (dashed trace). To aid visual comparison, the vertical lines marking firstfiring rate change from panel a are repeated here. Note that these earliest firing rate changes are notaccompanied by output change as far as the BMI is concerned. Note also that when decoded velocitydoes change, it pushes the cursor towards the target.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 79
Monkey LMonkey J
0 200 400 600
0 200 400 600
0 200 400 600
0 200
0 200 400 600
-10
0
10
0 200 400 600
-10
0
10
0 200 400 600
-10
0
10
0 200
-10
0
10
Time After Cursor Jump (ms) Time After Cursor Jump (ms)
Jump during hold
Jump at 3 cm
Jump at 6 cm
Jump at 10 cm
∆ F
irin
g R
ate
Fo
llow
ing
Pert
urb
atio
n (H
z)
Figure 4.4: Perturbations evoked both increases and decreases of firing ratesEach panel shows a 2D histogram of the distribution of perturbation-evoked firing rate changesfor a given jump location condition. These changes were calculated by subtracting the meanunperturbed response from the mean perturbed response for a given condition, as in Figure 4.3. Agiven dataset/target/jump direction condition (such as the examples shown in Figure 4.5) contributedone datum for each electrode at each time point. These were then counted in bins that are 1 mswide and 1 Hz tall. The color scale maps linearly from 0 (blue) to the maximum number of points inany bin for a given panel (yellow). Distribution means (gray traces) and medians (black traces) areshown for every time point. Distribution tails (|∆firing rate| > 10 Hz) are cut off so that the bulk ofthe distribution can be seen clearly.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 80
0 200 400 600
-1
0
1
2
3
0 200 400 600
-1
0
1
2
3
4
5
0 200 400 600
-2
0
2
4
6
0 200 400 600
-2
0
2
4
6
-1.5
-01
-0.5
0
0.5
1
1.5
0
1
2
3
4
2
4
6
8
10
-2
0
2
4
6
-1
0
1
2
3
4
5
-1
0
1
2
3
-4
-2
0
2
4
-4
-2
0
2
4
X Position (cm)
Y P
ositio
n (cm
)
2-10
-3
14
2-14
-9
5
14-3-9
2
8-2
-14
3a
b
c
d
Time After Cursor Jump (ms)
Δ N
eu
ral P
ush
(cm
/s)
Task A
xis
(c
m/s
)P
ert
urb
atio
n A
xis
(cm
/s)
unpertubed
after perturbation
Jump at 3 cm
J.2015.01.20 J.2015.01.12 L.2015.01.07 L.2015.01.14
Jump at 6 cm Jump at 10 cm Jump During Hold
Δ Neural Push
Figure 4.5: Examples of neural push following a cursor jumpEach figure column corresponds to the neural push analysis of trials from one example condition,which consists of a particular target, cursor jump direction, jump location, and experimental session.All four targets and jump locations are represented, and there are two examples from each monkey.(a) Cursor trajectories for all perturbed trials of each example dataset/condition that met the trialinclusion criteria (see Methods). The cursor started near coordinate (0,0) and proceeded towardsthe target whose acceptance boundaries are denoted by the gray square. (b) The neural pushcomponent along the Perturbation Axis; positive push corresponds to the direction opposing thecursor jump. Unperturbed responses were generated by aligning to ‘faux jumps’, i.e. when the jumpwould have happened in an unperturbed trial had that trial been selected for perturbation. Notethat the duration of the Jump During Hold unperturbed trace is limited because these trials endedshortly after the faux jump. (c) Same as (b) but for the neural push component along the Task Axis.Positive corresponds to push in the target direction (gray arrows in (a)). (d) Difference in neuralpush was calculated as the neural push of jump trials minus the mean neural push for unperturbedtrials to a given target. Neural push is plotted separately along the Perturbation Axis (solid line)and Task Axis (dashed line). Figure 4.3b was generated in the same manner, but averaged across alltarget locations, jump directions, and datasets.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 81
contributions which oppose each — but this intuition had to be verified at the population level. To
do so, we once again leaned on the analytical power afforded by having full knowledge of the mapping
from motor cortex to the BMI output. Figure 4.7a shows the error-related neural response projected
into the decoder-potent subspace (this is the same as the vector norm of the neural push shown in
4.6b) and an orthogonal decoder-null subspace (which is the null space of the Kalman filter gain
matrix). When viewed from this angle, the decoder-potent error-related response (i.e. the corrective
response) began 117 ms after the cursor jump in monkey J (147 ms in monkey L). Decoder-null
error-related responses began significantly earlier: 69 ms after the perturbation in monkey J (111 ms
in monkey L). This suggests that the intuition from Figure 4.6c is correct: there was no net neural
push when early error-related activity was summed across the full neural ensemble contributing to
the BMI output.
We wondered whether this phenomenon of the initial error-related responses being confined to
decoder-null dimensions was a consequence of the ReFIT decoder training protocol (Gilja et al., 2012
and Methods). Perhaps the refitting process optimized the decoder weights such as to minimize the
effect of error-related neural activity evoked during the “jittery” cursor movements of the closed-loop
calibration block? To control for this, we performed the same analysis, now projected into the
decoder-potent and decoder-null subspaces of a velocity Kalman filter fit from the initial passive
observation block. In this first training step, the monkey watched automated, perfectly straight
cursor movements to targets (Figure 4.8). Error-related responses projected into these dimensions
exhibited similar decoder-null followed by decoder-potent dynamics as observed for the final ReFIT
decoder, ruling out the decoder recalibration process as the cause of this effect.
However, the results described thus far could also be explained by a mechanism more trivial than
the “canceling out” arrangement I’ve suggested: there could be a separate neural subpopulation
receiving error signals that is distinct from the BMI-controlling subpopulation. If this were the case,
then the early decoder-null activity could be decoder-null simply because it is restricted to electrodes
that did not affect the BMI. To test for this, I repeated the decoder-potent, decoder-null projection
analysis for a subset of the electrodes that contributed substantially to the decoded velocity (see
Methods for the decoder weight calculation). Depending on the dataset, this data came from 24-30
electrodes in monkey J (cumulative decoder weight 0.5) and 28-38 electrodes in monkey L (cumulative
weight 0.8; Figure 4.9 shows each monkey’s decoder weight curves). Figure 4.7b shows that in monkey
J, the early initial transient was still present in these high-weight electrodes’ decoder-null subspace,
with the first decoder-null activity change occurring 79 ms after perturbation. This effect was weaker
in monkey L, with the first decoder-null activity 137 ms after perturbation, only 10 ms before the
decoder-potent correction. Thus, we can rule out the separate-subpopulations alternate hypothesis
definitively in monkey J but with less confidence in monkey L.
Interestingly, in monkey J removing low-weight electrodes from the analysis also had the effect
of increasing the earliest component of the decoder-potent neural push. This suggests that despite
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 82
0 500
-20
0
80
-30
0
130
Time After Cursor Jump (ms)
Decoder Push Direction
Neuron 2
Neuron 1
Δ F
irin
g R
ate
(H
z)
2 cm
Dec
oder-N
ull
Decoder-P
otent
Correct Left
Correct Right
Δ Firing Rate Neuron 1
Δ F
irin
g R
ate
Neuro
n 2
a
b
c
Figure 4.6: Decoder null to decoder potent neural dynamics hypothesis(a) Cartoon view of a mechanism by which early error-related neural activity could avoid affectingthe BMI. Consider two neurons whose increased firing push the cursor velocity in opposite directions.If both neurons increase or decrease their rates (blue line), the net effect on the decoder output iszero. But if one neuron increases its rate and another decreases its rate (e.g. green line), this willchange the cursor’s velocity. These combinations of neuron 1 and neuron 2 activity occupy orthogonaldecoder-null and decoder-potent dimensions. For a BMI driven by E electrodes with a 2-D velocityoutput, there will be an E − 2 dimensional decoder-null space and a 2-D decoder-potent plane. Wehypothesized that error-related signals first evolved in this decoder-null space, and that changes onlyentered the decoder-potent plane when the activity was such as to generate the desired correctiveBMI output.(b) Perturbed trials’ cursor trajectories for an example dataset/condition (upward target, leftwardperturbation after 10 cm) corresponding to the neural data in (c). From Dataset J.2015.01.14.(c) Hand-picked example perturbation-evoked neural response changes more concretely illustrate thedecoder-potent and decoder-null mechanism hypothesis. The arrows show the direction of decodervelocity contribution due to increases of firing rates on each electrode. Shading illustrates an initialdecoder-null epoch (blue) in which these electrodes’ neural activity roughly cancels out as far as thedecoder is concerned, followed by a decoder-potent epoch (green) when the net output is rightwardneural push. Although real data is shown, it is presented as an illustrative cartoon; the magnitudeand time courses of these two electrodes’ first phase of activity does not fully cancel out. Indeed,it would be unlikely to find two example electrodes that truly exemplify what is in actuality ahigh-dimensional, population-level phenomenon.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 83
0 50 100 150 200 250
Time After Cursor Jump (ms)
0 50 100 150 200 250
Perturbation Direction Differences
76 ms 146 ms
Jump at 3 cmJump at 6 cmJump at 10 cmJump during hold
0
∆ D
eco
der-
Po
tent
(AU
)∆
Deco
der-
Nu
ll (A
U)
High Decoder Weight Electrodes Only
79 ms 137 ms
107 ms 185 ms
69 ms 111 ms
147 ms117 ms
a
b
c
Monkey J Monkey L
Figure 4.7: Initial error-related modulation is restricted to the decoder’s null space(a) The initial neural responses to cursor jump perturbations were confined to the decoder-nullsubspace, and only later entered decoder-potent dimensions. We plot change in neural activityprojected into decoder-potent (upward-going traces) and decoder-null (plotted downward-going forvisual clarity) dimensions. Unperturbed trials’ neural projections into each space were subtractedfrom the condition-matched perturbed trials’ projections, and we plot the vector norm of thesedifferences. Vertical lines mark the first time of significant neural activity change (p < 0.001).(b) Same as (a) but restricted to electrodes with high decoder weights. The initial decoder-nullresponse is still present, indicating that early error-related information is not restricted to a neuralsubpopulation with minimal influence on the BMI output.(c) Instead of plotting the difference between perturbed and unperturbed trials, here we plot the normof the population response difference between cursor jumps in opposite directions. This difference isonly shown in the decoder null space, since decoder-potent jump direction difference is essentiallythe same as decoder-potent perturbation response. This analysis reveals that the early decoder-nulltransient contains direction-specific perturbation information.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 84
Jump at 3 cmJump at 6 cmJump at 10 cmJump during hold
200 25010050 15002500 50 100 150 200
Time After Cursor Jump (ms)
147 ms
113 ms69 ms
120 ms
0
∆ D
eco
der-
Po
tent
(AU
)
∆ D
eco
der-
Null
(AU
)Monkey J Monkey L
Figure 4.8: Responses in the potent and null space of an observation-trained decoderThe data shown in Figure 4.7d were re-analyzed with the decoder-potent and decoder-null subspacescalculated with respect to a velocity Kalman filter trained using data from the first block of the ReFITdecoder training protocol, in which the monkey passively watches automated error-free movements ofthe cursor to targets. Cursor perturbation-evoked neural responses projected into these dimensionsare very similar to those shown in Figure 4.7d, which were projected into the null- and potent-dimensions of the final (closed-loop re-calibrated) ReFIT decoder.
having a low weight in the decoder, these electrodes still played a role in the delicate balance which
minimized the effect of early error-related activity on decoder output during closed-loop BMI use.
4.3.3 Perturbation-specific information in the early error-related response
Finally, to better understand what information was present in this early decoder-null motor cortical
response, we took advantage of having presented the monkeys with cursor jumps in two different
directions at any given perturbation location (e.g., jump up or jump down after traveling 3 cm
towards the rightward target). If this early transient was agnostic to the details of the perturbation
(for instance, if it was a generalized surprise or error signal), we would expect no early difference in
the response to jumps in different directions. If, however, this response carried information about
more specific aspects of the perturbation, such as the cursor’s position or a target-cursor error vector,
then we would expect to see a divergence in responses to jumps in different directions. Figure 4.7c
shows the decoder-null difference in post-jump neural activity between responses following different
jump directions, averaged across all target/jump-location/dataset conditions. In monkey J, there is
clear jump directional information as early as 76 ms after the perturbation. For monkey L, whose
data is noisier, the effect is small, but there was still a significant jump direction difference starting
at 146 ms for the jump after 3 and 10 cm conditions. For the 3 cm condition this was substantially
earlier than the decoder-potent perturbation response.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 85
961 400
0.1
Cu
mu
lati
ve
Deco
der
Weig
ht
Electrode Rank
0
1
1921 100
Fra
cti
on
D
eco
der
Weig
ht
0
0.05
0
1
Monkey J Monkey L
0.5 0.8
Figure 4.9: Electrode decodercontribution weight curvesRanked electrode decoder contributionweights are shown in red, whilecumulative decoder weight for the top Eelectrodes are showed in black.Electrodes are sorted in order ofdescending decoder weight. Thinnerfaded traces show the weight curves forindividual experimental sessions; thickertraces show the means of these individualdays’ weight curves. Dashed vertical linesmark when the mean cumulative weightexceeds the threshold used for inclusionin Figure 4.7b.
4.4 Discussion
When we introduced cursor jump perturbations while monkeys used a BMI, we observed error-related
modulation in motor cortex with latency consistent with the earliest visual feedback to this area. The
presence of this activity in the same neural population directly driving the BMI raised the concern
of a deleterious feedback loop by which this response would “leak out” into the BMI output before
it was appropriately shaped to properly direct the corrective movement. Instead, we found that
the early error-related activity was confined to the decoder’s null space and thus did not interfere
with BMI control. Neural responses became output-potent tens of milliseconds later, at which time
they directed the cursor towards the target. This beneficial phenomenon allowed the monkeys to
effectively correct for BMI perturbations. I will now discuss the significance of these results for
understanding BMI use (and motor control more broadly) as a feedback control process, as well as
implications for the future design of BMIs.
4.4.1 BMI error correction latency
Control loop latency is a critical characteristic of any control system, and provides a lower limit
on how well the system will be able to correct for unanticipated errors. Our results provide the
most direct measurement to date of the control loop latency of a skilled BMI user relying on visual
feedback: 117 and 147 ms for the two monkeys. For simplicity, and given that this was a conservative
estimate limited by our statistical power and monitor refresh latency, we will henceforth refer to our
estimate of earliest BMI corrective response as occurring as early as 115 ms. This measurement is
consistent with the 100 – 133 ms range reported in (Golub et al., 2015), which was calculated based
on reaction time to target presentation rather than perturbation correction. These measurements also
provide a benchmark against which to measure reductions in feedback loop time that will hopefully
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 86
115 ms
X Output DimensionError dim.
Y O
utpu
t Dim
.
Rotate
X Output Dimension
Y Output Dim.Erro
r Dim
ensi
on
70 ms
Unperturbed
Perturbation Up
Perturbation DownX
Y
Neural Trajectory
Cursor Trajectorya
b
c
Figure 4.10: Summary schematic ofearly error response confined to anoutput-null space(a) Cursor trajectory for this example. Arightward cursor movement to the target isperturbed (red circle) either upward (blacktrajectory) or downward (red trajectory).(b) Neural activity viewed such that adecoder-null dimension occupied byerror-related activity is on the vertical axis, anda decoder-potent dimension corresponding tox-axis velocity is on the horizontal. For visualclarity, this horizontal dimension alsoapproximates time (as if the cursor is also beingpushed faster and faster). Modulation in thedecoder-null dimension begins approximately 70ms after the perturbation, with perturbedneural trajectories (red and black) divergingboth from the unperturbed (gray) neuraltrajectory and from each other.(c) The same neural activity, now rotated 90degrees out of the page such that the verticalaxis is a second decoder-potent dimensionwhich corresponds to decoded y velocity. Theerror-related neural modulation does not affecty velocity output until roughly 115 ms after theperturbation, at which point the red and blacktrajectories also diverge along the y velocitydimension. There is a 45 ms period in whichmotor cortical activity is changing (andpossibly performing computations related toerror correction) without “leaking” this activityto the BMI effector.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 87
be achieved by artificially writing proprioceptive and somatosensory feedback from the BMI into the
user’s brain (O’Doherty et al., 2011; Bensmaia & Miller, 2014; Dadarlat et al., 2014).
4.4.2 Early BMI error-related signals are present in motor cortex but do
not interfere with ongoing BMI control
The entire question of whether or not error-related signals interfere with ongoing BMI control hinges
on such signals actually being present in the motor cortical areas from which we decoded the BMI
control signal. The fact that BMI use is a closed-loop process implies that error signals must, of
course, eventually affect motor cortical output. But until this study, it was unknown whether this
activity is present early after the perturbation, or whether it only enters these areas later after it
is fully processed and in the form of a corrective output command. Our data support the former
hypothesis: we observed error-related modulation in motor cortex that arrived as early as 70 ms
after the perturbation, well before BMI output correction. This is consistent with the earliest
visually-driven re-targeting responses reported in motor cortex during arm control (Georgopoulos
et al., 1983; Archambault et al., 2011) but is considerably slower than the 50 ms latency reported for
proprioception-mediated error responses in M1 (Pruszynski et al., 2011a). Fortunately, we also found
that this early response did not affect the BMI output and did not appear to interfere with ongoing
control. We will discuss the how — and speculate about the why — of this in the next section.
4.4.3 Computing in an output-null subspace provides a mechanism by
which BMI output can ignore early error signals
To explain the mechanism — at least at a computational Marr level (Marr & Poggio, 1976) — of
how the initial error-related activity did not affect BMI output, I adapted Kaufman and colleagues’
‘muscle null’ hypothesis for explaining how motor cortical activity during reach planning does not
affect muscles until after the go cue is given (Kaufman et al., 2014). My analysis was made drastically
easier by the fact that, unlike in that study, partitioning neural activity into BMI decoder-potent and
decoder-null subspaces is both much easier and more accurate than approximating a linear mapping
between cortical activity and the muscles. Figure 4.10 summarizes my working model of how BMI
error-related neural activity transforms from an early response confined to decoder-null dimensions
into a later decoder-potent corrective response.
This study adds to a growing body of knowledge suggesting a computational motif in which
balanced changes in motor cortical activity permit network changes related to motor planning
(Kaufman et al., 2014), motor-reported sensory detection (Zagha et al., 2015), or motor learning
(Li et al., 2001) to occur without immediately affecting the efferent output. Although our study
can only directly speak to this computational strategy being used during a motor BMI task, we
speculate that a similar mechanism is at work following perturbations during natural arm control.
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 88
We believe that experiments to test this prediction would be of high value, but note that they will be
made more challenging by the aforementioned difficulty in definitively identifying the muscle-null
and muscle-potent neural dimensions.
The possibility that these output-null to output-potent neural dynamics are shared with natural
motor control also presents a hypothesis about why motor cortex exhibits this advantageous behavior
during BMI use — seemingly without the animals having had to learn to squelch the error-related
signal into output-potent dimensions. A slightly inverted way of asking this question is, why were the
decoder’s weights such that early error-related responses occurred in their null space? I suspect this
was because the decoders used in our study were “biomimetic” in the sense that we sought, to the
best of our ability, to match them to an intuitive pre-existing neural-to-kinematic mapping (Jackson
& Fetz, 2011; Shenoy & Carmena, 2014). In these experiments, the decoder was initiated based on
the patterns of neural activity evoked by the monkeys observing automatic cursor movements and
“following along” with some sort of motor imagery or mental rehearsal. This leads me to speculate
that error-related signals were in the output-null space of the decoder because it is similar to the
output-null space of the muscles. If the motor system has evolved or learned to sequester error
processing into output-null neural dimensions, then biomimetic BMI use might be making use of this
same arrangement. In other words, it could be that the intrinsic manifold (Sadtler et al., 2014) of
intuitive BMI mappings exploits not only the pre-existing output-generation neural machinery, but
also its closely coupled input-processing properties.
However, another possibility which we cannot rule out is that the monkeys’ years of previous
BMI experience, which did not include cursor jumps but did involve spontaneous decoding errors,
led to network changes which minimized undesired effects of sensory feedback. The question of
whether the brain can learn to sequester error-related signals into an output-null subspace could
be answered with a future learning study in which the decoder is intentionally modified such that
the early transient does affect output. This decoder could then be held fixed as the monkey uses it
for a prolonged period, allowing tracking of both BMI output and underlying neural changes as in
(Ganguly & Carmena, 2009; Ganguly et al., 2011).
4.4.4 Differences between responses to different jump location conditions
Deeper investigation of the following two comparisons between how the monkeys corrected BMI
perturbations occurring at different phases of the trial are outside the scope of the current report, but
we feel they are worth remarking upon. First, we were surprised that neural push did not increase
more rapidly for later cursor jump locations, as one might expect under many optimal feedback
control cost functions (Todorov & Jordan, 2002; Scott, 2012). This could be evidence for there
being a limit to how rapidly the motor cortical network can change its firing rates in the service of
error correction goals. On the other hand, it could simply reflect a lack of motivation on the part
of the monkeys to correct these later mistakes “extra hard”, perhaps because the trial time limits
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 89
were generous. Second, decoder-potent activity seemed to ramp up slower (in magnitude slope, not
in earliest detectable deviation) for the jump during hold condition. I speculate that this may be
because the neural network is in a less labile/responsive state when it is explicitly trying to hold the
cursor still than when it is in the dynamic phase of movement to the target.
4.4.5 Implications for future BMI design
The primary focus of this study was to address a previously raised concern (e.g. Green & Kalaska,
2011) that the unnatural arrangement of performing a motoric task while bypassing normal cortical
and subcortical circuits may incur a penalty when trying to make rapid error corrections. Our finding
no deleterious feedback loop, as well as the overall proficiency with which the monkeys corrected for
the perturbation, suggests that this may not be as problematic as one might think — at least for
2D cursor tasks. This came as a pleasant surprise; I had expected to see deleterious feedback and
anticipated having to develop methods to adjust the decoder to mitigate this feedback. However, our
study does demonstrate that early visually-driven, error-related signals are very much present in
homologues of the cortical areas currently being used for human robotic arm control studies (Vogel
et al., 2010; Hochberg et al., 2012; Collinger et al., 2013b; Wodlinger et al., 2015). Our findings should
therefore serve as a warning to be on guard for deleterious error signals, as it as of yet unknown
whether this decoder-null, decoder-potent separation holds up 1) in humans and 2) during high
degree-of-freedom arm control in which the output-potent subspace is larger. That said, our results
also provide hope that these initial error-related signals can be sequestered in an output-null neural
subspace. This helpful arrangement might fall out “for free”, especially if a biomimetic approach is
taken, or it could emerge after longer-term neuroprosthetic skill learning (Ganguly & Carmena, 2009;
Orsborn et al., 2014), perhaps thanks to strategic decoder adjustments to guide beneficial learning
(Shenoy & Carmena, 2014).
A second major point of this study concerns BMI control loop latency. As mentioned earlier, this
latency is a critical control system property with profound consequences for the controllability of
the BMI (Chase et al., 2009; Cunningham et al., 2011; Stavisky et al., 2015a). Although we found
that the monkeys did not begin to correct perturbations for at least 115 ms, error-related motor
cortical signals contained information about the cursor jump direction as early as 76 ms after the
perturbation. This presents an opportunity for future work investigating whether one could decode
this early signal in order to automatically correct for the error faster than the BMI user would
otherwise be able to. This proposed motor performance error auto-correction has parallels to our
ongoing work to detect and correct for (binary) task outcome errors (Even-Chen et al., 2015).
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 90
4.5 Methods
4.5.1 Cursor jump task
All procedures and experiments were approved by the Stanford University Institutional Animal Care
and Use Committee. Two rhesus macaques (monkeys ‘J ’and ‘L’) were trained to sit head-fixed in
a primate chair and perform 2D target acquisition tasks by controlling an on-screen cursor with
either their hand position or via a BMI in order to obtain a liquid reward. The task was displayed in
virtual reality with a latency of 7± 4 ms as described in (Gilja et al., 2012). For this study, we used
a “hand restrained training, hand restrained BMI control” paradigm (Shenoy & Carmena, 2014) in
which the monkey’s arms were gently restrained during open-loop BMI training, closed-loop BMI
training, and the main task.
The Cursor Jump Task at the core of this study was a two-target variant of the traditional
center-out-and-back Radial 8 Task. We used only two targets to increase the number of repetitions of
each target/perturbation condition. The monkey controlled the velocity of a cursor to acquire targets
by bringing the cursor within a 4× 4 cm target acquisition area and holding it there for a continuous
500 ms. The target changed color to signify that the cursor was within the acquisition zone. Leaving
the target during this hold period reset this timer but did not result in immediate failure. If the
monkey acquired the target within a 5 s time limit, he received a liquid reward. Otherwise, he heard
a failure tone and was presented with the center target again. The task always began with the
center target being presented. After a successful center acquisition, one of the two radial targets was
presented. The two possible radial targets were arranged either along the horizontal axis or vertical
axis (the ‘Task Axis’), and the sequence of targets was random. To increase the number repetitions
of a given perturbation type per dataset, the orientation of the Task Axis was fixed within a given
experimental session, but varied across experimental sessions. The next target was presented 20 ms
after either success or failure, and the cursor was under continuous neural control.
The key experimental manipulation was that on a random 25% of trials towards radial targets, a
perturbation was applied. This perturbation consisted of offsetting (“jumping”) the cursor position
by 6 cm perpendicular to the Task Axis (we call this dimension the ‘Perturbation Axis’, see Figure
4.2b). These perturbations were applied on the next decode update time step after one of four
randomly selected and equally likely ‘jump location conditions’ were met. Three of the conditions’
had a spatial criterion: the jump happened after the cursor travelled either 3 cm, 6 cm, or 10 cm
towards the target along the Task Axis. In the fourth condition, the cursor jump happened on the
decode time step prior to 300 ms into the target hold period. Empirically, the mean times when
this happened was 287.0 ms into the hold for monkey J (25 ms decode time steps) and 274.0 ms
for monkey L (50 ms decode time steps). Only one perturbation could happen per trial, and the
different perturbation conditions were interleaved throughout each session. The time when the cursor
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 91
jump command was sent to the display computer was recorded with 1 ms resolution, after which it
appeared at the next 120 Hz monitor update.
The data consist of 8 experimental sessions (1 per day) in monkey J and 9 sessions in monkey L.
We will refer to one experimental session’s data as a ‘dataset’. When dataset identifiers are provided
(e.g., “J.2015.01.19”), they are of the format <monkey initial>.<year>.<month>.<date>.
4.5.2 Neural recording and decoding
Monkeys J and L were implanted with 96-electrode arrays (1 mm electrodes spaced 400 µm apart,
Blackrock Microsystems) using standard neurosurgical techniques. Monkey J had one array implanted
into primary motor cortex (M1) and the other into dorsal premotor cortex (PMd) of the same
hemisphere 64 months prior to these experiments. Monkey L was implanted with a single array at
the M1/ PMd border 83 months prior to the experiments. Voltage signals from each of the electrodes
were band-pass filtered from 250 to 7500 Hz and then processed to obtain multiunit ‘threshold
crossings’ spikes. A spike was detected whenever the voltage crossed below a threshold set at the
beginning of each day to be -4.5 × rms voltage (Chestek et al., 2011). We did not spike sort to
assign spikes to individual putative neurons and instead grouped together threshold crossings on
a given electrode; the population activity will therefore include both single- and multiunit activity.
We believe that this is justified because 1) threshold crossings are the becoming the standard spike
signal used for prosthetics applications (e.g. Gilja et al., 2012, 2015; Collinger et al., 2013b), and 2)
we are making claims about information present in the population activity and lower-dimensional
projections of this activity, rather than specific responses of individual neurons (Trautmann et al.,
2016). All three arrays recorded good signals, with spikes detected on almost every electrode.
We used the ‘recalibrated feedback-intention trained’ Kalman filter (ReFIT) training protocol
and decoding algorithm (Gilja et al., 2012). This involved a two-stage decoder training process to
first initialize and then refine a decoder in the absence of overt arm movements. At the start of each
experiment, the monkey first observed 250 automated straight center-out-and-back cursor movements
between the center target and one of eight radial targets located 12 cm from the workspace center.
When training Monkey L’s decoder, the three targets forming the upper quadrant were placed slightly
further (13 and 14 cm) based on previous experience that this improved his ability to acquire targets
in that quadrant following decoder training. We used the neural and automated kinematic data from
these trials to train a position + velocity Kalman filter. This first-pass decoder was used to complete
an additional 250 center-out-and-back trials of the Radial 8 Target task under BMI control. This
closed-loop data was then used to fit the final ReFIT velocity decoder after several modifications
of the kinematics training data to improve the estimation of the animal’s true intent, described in
(Gilja et al., 2012). However, because we experimentally manipulated cursor position via the cursor
jump, we did not apply the ‘position subtraction’ ReFIT operation (otherwise the cursor jump would
directly affect decoded velocity).
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 92
During decoder training and BMI use, no distinction was made between M1 and PMd electrodes.
For the purpose of the subsequent analyses in this study, they were analyzed together, with the
exception that we also replicated the main decoder-potent vs. decoder-null analyses for monkey J
using just his M1 array and saw similar results (not shown). I emphasize that this does not preclude
there being more nuanced differences based on electrode location, which may be deserving of future
study.
4.5.3 Analysis of responses to cursor jumps
Both monkeys almost always successfully acquired targets in the Cursor Jump Task, with failures
likely attributable to the monkey disengaging from the task. We thus restricted our analysis to
successful trials. We quantified cursor control performance using time to target, which measured how
long it took to acquire the target, not including the 500 ms required final hold time, but including
incomplete earlier holds. For the jump during hold condition, we also did not include the initial hold
time preceding the cursor jump, since this time was “stolen” in the sense that there was no way for
the animal to minimize this penalty. To exclude trials where the monkey is likely to have briefly
disengaged from the task, we removed outlier trials with times to target more than three standard
deviations longer the mean (between 0.7% and 2.0% of trials across individual datasets). For neural
analyses in which responses were averaged across trials of the same condition, we also excluded any
jump-location/target/jump-direction/dataset conditions with fewer than 5 trials. Combining across
monkey J’s eight datasets, we analyzed 8,297 unperturbed trials, 743 jump after 3 cm trials, 726
jump after 6 cm trials, 698 jump after 10 cm trials, and 552 jump during hold trials. Across monkey
L’s nine datasets, we analyzed 5,039 unperturbed trials, 427 jump at 3 cm trials, 399 jump at 6 cm
trials, 440 jump at 10 cm trials, and 370 jump during hold trials.
To measure how neural activity evolved after a cursor jump perturbation, we calculated the firing
rate of all trials using a 25 ms boxcar window slid by 1 ms steps. Whenever we present data, its
timestamp corresponds to the last millisecond of the window, thereby conservatively estimating
neural response latencies. We were interested in the change in neural activity specifically due to the
perturbation. Therefore, when analyzing perturbation-evoked responses, we first subtracted away
the mean responses of otherwise similar unperturbed trials. This served to largely remove underlying
firing rate changes related to performing the BMI task. More specifically, for every perturbation
trial j, we aligned trial j’s firing rate to the time of the cursor jump. We then subtracted away the
mean firing rate of all unperturbed trials to the same target in trial j’s dataset. These unperturbed
trials’ responses were aligned to ‘faux jumps’ by assigned them a jump time based on when a jump
would have happened on that trial if trial j’s jump location condition had been in effect. We then
calculated a mean jump-evoked firing rate difference for each ‘dataset/condition’ by averaging across
all jump trials in a given dataset that were to the same target, had perturbations at the same
jump location, and perturbed the cursor in the same jump direction. Thus, each dataset typically
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 93
contributed (4 jump locations) × (2 targets) × (2 jump directions) = 16 trial-averaged time-varying
neural change traces. Unless otherwise noted, all results presented are averaged across all of the
monkey’s dataset/conditions of a particular jump location. This was done so that every dataset
contributed roughly equally to the final analysis, removing the risk that our results were unduly
weighted by a few dataset/conditions with the most trials.
To calculate overall change of firing rates across the E electrodes, I took the vector norm of
this E-dimensional difference vector at each time point. This captures neural changes due to both
increases and decreases in firing rates. Since difference norms must be non-negative, there will be
some baseline firing rate differences just due to taking differences between firing rate time series that
differ due to single-trial variability unrelated to the difference between unperturbed and perturbed
trials. I estimated this baseline noise with a shuffle test in which I performed the same processing on
data that we knew had no perturbation-related activity: I assigned P unperturbed trials to be labeled
as perturbed trials, where P was the number of actual perturbed trials for this dataset/condition,
and used the remaining unperturbed trials for the mean unperturbed response subtraction. This
shuffled analysis was repeated 10,001 times. The mean of the resulting shuffled difference norms was
subtracted from the true data’s difference norm to yield time series with baseline norms close to
zero as shown in Figure 4.3a. The shuffled data also provided a nonparametric statistical test for
significant post-perturbation firing rate change. Specifically, the p-value of the significance of firing
rate change was calculated at each time point by counting how many of the shuffled norms the real
data was larger than: p < 0.001 denotes that the real neural difference was larger than 10 of the
10,001 shuffled differences.
4.5.4 Neural response from the decoder’s perspective
To assess how these changes in neural activity affected BMI output, I took advantage of the fact that
the closed-loop Kalman filter almost immediately converges to a steady-state linear decoder (Malik
et al., 2011) of the form:
x(t) = M1x(t− 1) + M2y(t), (4.5.1)
where x(t) is a 5× 1 state vector of x- and y- position and 1 for bias offset; y(t) is a E × 1 vector
of each electrode’s spike counts in the current time step; and M1 is a state dynamics matrix that
smooths velocity over time and lawfully integrates velocity to change position. The Kalman gain
matrix, M2, applies no neural contribution to position but has two rows of weights that linearly
map each electrode’s firing rate to an x- and y- velocity component. The effect of observed neural
activity on time step t onto velocity is the summation of these velocity components. We therefore
projected firing rates through the M2 matrix to calculate the ‘neural push’, i.e. how instantaneous
neural activity affects BMI output velocity. Firing rates projected through M2 are decoder-potent
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 94
activity. Note that neural push is an offline analysis measurement and is not quite the same thing
as the true velocity of the closed-loop BMI cursor. This difference has two causes. First, true
cursor velocity was subject to a momentum-like term imposed by the M1 matrix. Second, true
cursor velocity is computed once every (non-overlapping) decode time step, with on-screen velocity
interpolated between decode steps. Offline neural push can be computed for any given millisecond
using firing rates computed using arbitrary time windows, and should be interpreted as showing how
the instantaneous neural activity would affect velocity had the decoder updated at that time. A
useful analogy is that neural push is similar to instantaneous muscle force, which is closely related to
— but is distinct from — the movement of the arm, which is subject to physical dynamics.
Perturbation-evoked changes in neural push along the Perturbation Axis and Task Axis (Figures
4.3a and 4.5) is a signed quantity; therefore, no shuffle baseline subtraction was needed since
random variation in neural push would tend to average out to zero. This is not to be confused with
subtracting the unperturbed trials’ neural push, which we did do as shown in Figure 4.5. However,
the magnitude of decoder-potent activity change (Figure 4.7a,b), i.e. the vector norm of the difference
between perturbed and unperturbed trials’ decoder-potent neural activities, is again a non-negative
quantity. As such, this measurement has the same source of baseline noise as described above for
perturbation-evoked changes in firing rates. We therefore performed the same shuffled baseline
subtraction and nonparametric significance testing. To compute the perturbation-evoked response
changes in the decoder-null subspace, we analyzed the data exactly as for the decoder-potent subspace,
except that instead of projected firing rates through M2, we projected them into the null space of
M2.
We also examined the differences between decoder-null responses to cursor jumps that happened
at the same jump location during movements to the same target, but which displaced the cursor
in opposite directions (Figure 4.7c). To do so, we calculated the difference between projected
jump-aligned neural activity for a given dataset/condition and that of the dataset/condition with the
opposite jump direction. We then took the vector norm of this difference at each millisecond. Since
vector norms are non-negative, we once again did a shuffle control to subtract the baseline noise. We
performed the same analysis after randomly permuting the jump direction labels, repeated 10,001
times. At every time point, the mean vector norm across these permuted datasets was subtracted
from the true norm. Significance of jump-direction related difference was calculated at each time
point by counting how many of the shuffled norms the real data norm was larger than. To aggregate
results across targets and datasets, each dataset/jump condition/target contributed one trial-averaged
time-series, which were then averaged across datasets/targets to yield the final result.
4.5.5 Electrode neural push direction and decoder contribution weight
The decoder training data were used to compute each electrode’s directional “tuning” and its decoder
contribution weight. We used the M2 Kalman gain matrix described earlier, where the jth column of
CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 95
M2 is the decoder weight vector wj = [wx,j , wy,j , 0]T that describes how yj(t), the neural activity
observed on electrode j, contributes to x- and y- velocity. Electrode j’s decoder contribution is then
computed as its average effect on cursor velocity:
(contribution of electrode j) =∥∥E[wj · yj(t)]
∥∥ (4.5.2)
where the expectation is taken over every bin of the training dataset. An electrode’s relative
contribution to the decoder is then calculated by dividing that electrode’s contribution by the sum of
all electrodes’ contributions. Its decoder push direction is determined by the angle specified by wj .
4.6 Project Acknowledgements
I thank the members of the Shenoy group for helpful discussions; Jordan Sorokin for helping examine
earlier cursor jump data during his rotation in the laboratory; Mackenzie Mazariegos, Michelle
Wechsler, Clare Sherman, Liana Yates, Rosie Steinbach, and Shannon Smith for surgical assistance
and veterinary care; Beverly Davis, and Evelyn Castaneda for administrative support; Boris Oskotsky
for information technology support.
This work was supported by the National Science Foundation Graduate Research Fellowship
(J.C.K., S.D.S.); NSF IGERT 0734683 (S.D.S.); Christopher and Dana Reeve Paralysis Foundation
(S.I.R. and K.V.S.); and the following to K.V.S.: Burroughs Welcome Fund Career Awards in the
Biomedical Sciences, Defense Advanced Research Projects Agency Reorganization and Plasticity
to Accelerate Injury Recovery N66001-10-C-2010, US National Institutes of Health Institute of
Neurological Disorders and Stroke Transformative Research Award R01NS076460, US National
Institutes of Health Director’s Pioneer Award 8DP1HD075623-04, US National Institutes of Health
Director’s Transformative Research Award (TR01) from the NIMH 5R01MH09964703, and Defense
Advanced Research Projects Agency NeuroFAST award from BTO W911NF-14-2-0013.
Chapter 5
Conclusion
5.1 Summary of motor neural prosthesis advances
My doctoral research has laid a foundation for addressing several key challenges to clinically useful
motor neural prostheses through experiments in a pre-clinical animal model. For this technology to
provide a substantial quality of life improvement for people with paralysis, it must function reliably
and have a long device lifespan. The recurrent neural network decoding method detailed in Chapter
2 provides a technique for building BMIs that become more robust to changing recording conditions
as they accumulate more data with every passing day. Furthermore, by learning context-dependent
neural mappings and allowing for data augmentation, this technique may also improve decoding
performance across a wider range of tasks and settings. This will help create BMIs that provide
consistently high performance on a minute-by-minute and day-by-day basis.
The local field potential decoding methods detailed in Chapter 3 address the need for longer
device lifespan by providing useful levels of BMI performance even without spike signals. Since local
field potentials can often be recorded long after sensor degradation renders chronically implanted
electrodes unable to record spikes, this can reduce the need for surgical re-implantation of the sensor.
This will help positively tilt the risk-benefit tradeoff of motor neural prostheses, especially since there
does not currently appear to be a path to controlling high-degree-of-freedom effectors such as robotic
arms using non-invasive methods.
The longer-term vision for motor neural prostheses includes restoring control of arm reach-and-grasp
function. Amongst the many additional complexities introduced by controlling physical limbs, rather
than computer cursors, is that they will operate in physical environments where they will encounter
perturbations. The effect of the resulting sensory feedback on neural signals must be understood if
we are to build BMIs that work well when interacting with the physical world. Chapter 4 describes
my detailed investigation of error-related feedback on BMI control. This first-in-the-field study also
96
CHAPTER 5. CONCLUSION 97
revealed a sensorimotor computational mechanism that we might be able to exploit in order to
prevent sensory feedback from deleteriously interfering with BMI output.
5.2 Next steps: testing in a human clinical trial
This research was performed in able-bodied monkeys, a pre-clinical animal model that is as close as
feasible to the human clinical case while still allowing for the higher risk, greater precision, and faster
paced research needed to maximally advance the engineering and science of motor neural prostheses.
Nonetheless, for this work to generate a tangible societal benefit, it will need to be translated to
human clinical studies. I believe that there are clear applications for each of the described projects
that are worth testing in human BMI users.
The MRNN decoder may prove particularly useful for handling the variable recording conditions
often encountered in human BMI users. Ongoing clinical trials are already collecting the necessary
large corpus of previous data; thus, there is a fairly straightforward path to testing whether this
method will improve participants’ BMI reliability. This technique may also prove useful for decoding
control signals for prosthetic arms. In this application, the neural-to-kinematic mapping may vary
depending on the position of the arm and its interaction state with objects. The MRNN might be
well-suited to accommodate these changes because its flexible neural-to-kinematic mapping depends
on the recent history of neural inputs, and can be extended to explicitly depend on the effector state.
Decoding local field potentials may be of high value in participants whose electrode arrays no
longer record good spike signals. Although I hope that recordings conditions will not necessitate
this approach — since this would mean that a participant has lost the better spikes signals — there
is no reason why LFP-only and hybrid decoding can’t be tested in dedicated experiments sooner,
subject only to the constraints of experimental timelines and priority queues. In the meantime, data
already collected can be analyzed with a specific focus on the low-frequency local motor potential
feature that I found performed well in a monkey animal model. That said, my LFP decoding work
specifically emphasized that offline decoding analyses can only go so far in predicting the actual
utility of a BMI method. Online tests will be the gold standard for testing whether LFP decoding
can extend device lifespan.
The two key unknowns related to my study of neural responses to BMI perturbations are 1)
whether the same decoder-null to decoder-potent dynamics hold up as the decoder-potent subspace
grows with the number of controlled degrees-of-freedom (for instance, when controlling prosthetic
limbs), and 2) whether this phenomenon that I discovered in able-bodied monkeys is also present in
movement-impaired people. If we find that this beneficial arrangement does hold up, then not only
would this be a fortuitous turn of events, but we will also have the benefit of understanding why
error-related sensory signals are not interfering with BMI use. If it doesn’t, we are now equipped
with an analytical framework by which to identify these deleterious feedback signals. We can then
CHAPTER 5. CONCLUSION 98
try to adapt our decoders, or encourage neural adaptation, in order to force the error-related activity
into decoder-null dimensions. Either way, the only way to find out what will happen is to forge ahead
with prosthetic limb control experiments while deliberately planning a careful study of perturbation
responses. The experiments and analyses described in this dissertation can serve as a model for how
to go about doing this.
My doctoral training in the Stanford Neurosciences Program, conceiving and carrying out the
research described in this document under the mentorship of Professor Krishna Shenoy, has prepared
me to try these approaches — as well as tackle new challenges and research directions — in the
service of providing increasing motor function to clinical trial participants during my postdoctoral
endeavors.
5.3 Additional graduate work not covered in this dissertation
My Ph.D. training afforded me with the opportunity to explore a variety of topics related to motor
control and motor neural prostheses. Two considerations determined my choice of which three
projects formed the core of this dissertation. First, they were each at a mature phase, being either
published (Chapter 3), in review (Chapter 2), or very near submission (Chapter 4). Second, they have
a thematic relationship: all three of these studies speak to how we can robustly decode movement
intent despite changes in, loss of, or sensory feedback effects upon, the observed neural signals. There
were a number of other projects to which I devoted considerable portions of my graduate work, but
which did not fit into the scope of this dissertation either because they were thematically less close,
or because they are in a less developed stage. In this section I enumerate and briefly describe the
remaining major projects which have a high likelihood of being developed into journal papers.
5.3.1 Drawing scientific conclusions about neural population activity without
sorting single neuron activity
During the course of my graduate work, I collected a large quantity of multielectrode array data
in which many electrodes recorded multiunit ‘threshold crossing’ spikes but not well-isolated single
neuron activity. Since this neural data had sufficient information to drive accurate BMI cursor
control, and since my analyses were providing interesting scientific insight, I wanted to be able to
report my discoveries without objections being raised that I was not using “spike sorted” single unit
activity. It would have been difficult and time-consuming for me to re-collect my data to record
single unit activity: this would require array re-implantation in one of my monkeys (or training a
new monkey), and it would have substantially reduced the number of neural information channels
available from the other monkey. In order to maximize the utility of each research animal, I was
compelled to think about whether I could justify performing neural population analyses on unsorted
threshold crossing data.
CHAPTER 5. CONCLUSION 99
Thinking about the goals of my analyses, I made two key observations. First, I was attempting
to measure and describe the information present in a cortical ensemble, rather than describing the
behavior of individual neurons per se. Second, there is an emerging view in the field (e.g. Shenoy et al.,
2013) which argues that, at least in motor cortex and several other cortical areas, individual neurons’
activity is a distributed and redundant high dimensional “read out” of latent lower-dimensional
dynamics. In light of this, analyzing threshold crossings (which is the equivalent of summing together
the activity of some single neurons as well as adding the measurements of multiple more distant
neurons which otherwise would not have been detected) appears to be theoretically justified as long
as the latent neural state can still be estimated from this (different) high dimensional multiunit
neural projection.
I decided to empirically test whether one would come to the same scientific conclusions about
population neural activity with and without spike sorting, i.e. by analyzing ‘threshold crossings’. I
started this project by taking previous published scientific work from our laboratory (Ames et al.,
2014) and re-analyzing the data without spike sorting to see how this would affect the results. After
I began this project, my colleague Eric Trautmann joined the collaboration. Based on our respective
interests and project portfolio priorities, he is now taking primary charge of this work and will
see it through to publication. Eric re-analyzed data reported in two more prior papers from our
group (Churchland et al., 2012; Kaufman et al., 2014), and presented our results at the 2016 Cosyne
conference, titled “Sortfree: Using threshold crossings to evaluate scientific hypotheses in population
analyses” with authors Eric M. Trautmann, Sergey D. Stavisky, Mattew Kaufman, K. Cora Ames,
Stephen I. Ryu, and Krishna V. Shenoy (Trautmann et al., 2016).
We found that the conclusions drawn from these data were for the most part the same with and
without spike sorting. Although some electrodes’ peri-stimulus time histograms changed noticeably
compared to those of single units recorded from the same electrode, most electrodes showed similar
activity patterns. Many of the “leftover” threshold crossings that had not been spike sorted appeared
to be tuned for movement parameters, suggesting that our method worked not only because relatively
little information was being lost by potentially combining multiple units on a given electrode, but
also because we were now utilizing some previously missed information. More importantly, the
measured population neural dynamics and statistical hypothesis tests run on sorted single unit data
were very similar when run on threshold crossings. In light of this validation and the practical need
to more rapidly analyze growing data quantity created by new, higher density recording technologies,
I anticipate that analyzing threshold crossings in place of sorted spikes will become increasingly
common for drawing conclusions about population activity.
CHAPTER 5. CONCLUSION 100
5.3.2 Observed BMI cursor position only weakly affects motor cortical
activity
Tracking the state of the motor ‘effector’ (also known as the ‘plant’) is a computation strongly
predicted to occur in the sensorimotor system under contemporary feedback control theories (e.g.
Wolpert et al., 1995; Scott, 2004). For example, the arm’s position is well represented in cortical
activity (Caminiti et al., 1990). An early project that emerged from my starting to think about
the neuroscience of BMI use was to look for motor cortical correlates of the BMI cursor’s position.
This information would be consistent with the BMI cursor taking on the role of the which the real
arm fills during natural movements. To investigate this, I analyzed the motor cortical correlates
of 1) hand position during a hand controlled target acquisition task, 2) cursor position during an
arms-restrained BMI task, and 3) hand and cursor position during an arm-free BMI task. I found
that although both hand and cursor position influenced firing rates, this effect was much stronger for
the hand’s position than for the cursor’s position, especially once the correlations between hand and
cursor were accounted for in the arm-free condition. I presented this work at the 2013 Society for
Neuroscience annual conference, in the form of a poster titled “Motor cortical activity tracks the
position of a brain-machine interface cursor”, with authors Sergey D. Stavisky, Jonathan C. Kao,
Paul Nuyujukian, Stephen I. Ryu, and Krishna V. Shenoy.
Given this weak effect of cursor position, and more importantly, the potential confound of the
monkey’s gaze following the cursor’s position, barring new analytical insight I plan to abandon
the scientific aim of this project. However, this work yielded an observation with practical neural
engineering value. The ReFIT decoder (Gilja et al., 2012) that has been the workhorse of our group’s
BMI efforts in recent years includes one decoder innovation called ‘position subtraction’. Inspired
by the hypothesis that cursor position effects neural activity, ReFIT treats this position effect as a
nuisance variable and subtracts its expected firing rate contribution before inferring velocity intent.
However, the utility of this design choice has not been previously tested via direct, back-to-back
closed-loop performance comparisons with and without this position subtraction operation. I did
exactly this, and found that although position subtraction did improve performance when the
monkey’s reaching arm was free, it did not help performance when the monkeys used the BMI without
moving their arms. This suggests that the apparent benefit of position subtract was because this
operation was indirectly accounting for the effect of the arm position (which is correlated with cursor
position), and that this operation is unlikely to be of use in paralyzed BMI users.
5.3.3 Single trial motor cortical correlates of a rapidly adapting visuomotor
internal model
Accurate motor control is mediated by internal models of how neural activity generates movement.
As part of a series of experiments showing that during BMI use the brain maintains an up-to-date
CHAPTER 5. CONCLUSION 101
internal model of the BMI’s properties, I conducted matched experiments in which monkeys controlled
a cursor either with their hand (this project) or with a BMI (see next section) while the gain scaling
between the commanded velocity and the cursor’s movement was varied. The hand-controlled data
were initially intended to provide a point of comparison between BMI and natural arm control
visuomotor adaptation rates. However, I discovered an interesting effect in these data: neural activity
varied during the so-called “baseline” period in between reaches in a way that was evocative of a
neural correlate of the visuomotor internal model. I explored this effect further, which led me to write
up this result as its own manuscript which I anticipate submitting to a journal soon. The authors for
this manuscript are Sergey D. Stavisky, Jonathan C. Kao, Stephen I. Ryu, and Krishna V. Shenoy.
Previous studies of cortical changes during visuomotor adaptation focused on preparatory and
peri-movement epochs and analyzed trial-averaged neural data. In this project, we recorded
simultaneous neural population activity using multielectrode arrays and focused the analysis on
neural differences in the period before the target appeared. This let me ask whether there were
correlates of different internal models being “loaded” in motor cortex. I found that I could estimate
the monkey’s internal model of the gain using the neural population state during this pre-target
epoch. This neural correlate depended on the gain experienced on the previous trial, and it predicted
the speed of the subsequent reach. The ability to observe internal model changes with single-trial
resolution has utility for studying the interaction of internal models, movements, and errors on the
single-trial timescales at which these computations ultimately play out.
Furthermore, the ability to estimate the internal model before movement begins has potential
BMI applications. A brain-controlled robotic arm user will need an accurate internal model of how
her neural commands affect the arm; this is complicated by the arm’s own physical dynamics (which
could vary across joint conformations) and its interaction with objects (e.g., carrying something
heavy). My variable cursor velocity gain task can be viewed as a (much simplified) example of a user
under- and over-estimating the gain between their neural command and the effector. Therefore, to
explore the neural prosthesis utility of being able to estimate the internal model, I performed an
offline analysis of these data which showed that the neural activity can be used to compensate for
upcoming reach extent errors.
5.3.4 Rapid but limited adaptation to a change in BMI velocity gain
In the first part of this project, I used the matched hand- and BMI-controlled changing velocity gain
tasks introduced in the previous section to show that monkeys rapidly adapt to a changing BMI
cursor velocity gain similarly to how they adapt to hand-controlled cursor velocity gain changes. This
result adds to the nascent body of knowledge showing that, similarly to natural motor control, BMI
control employs internal models (Golub et al., 2015). I presented this work at the 2014 Society for
Neuroscience annual conference as a poster titled “Brain-machine interface performance is mediated
CHAPTER 5. CONCLUSION 102
by an internal model of decoder velocity gain”, with authors Sergey D. Stavisky, Jonathan C. Kao,
Stephen I. Ryu, and Krishna V. Shenoy.
Despite rapid initial gain adaptation, I also observed that the monkeys were never able to fully
compensate for the changed BMI velocity gain. Although they increased their velocity command
when adapting to a slow gain and decreased the velocity command when adapting to a fast gain,
these changes were not sufficient to equalize their cursor speed profile to that of the gain 1 condition.
Consequently, their performance, as measured in achieved bitrate on a target grid task, was worse
when the gain deviated from ∼1. The cursor moved too slowly under low gain conditions, which
slowed down acquisition times, or too quickly under high gain conditions, which resulted in more
incorrect selections because it was harder for the monkey to precisely hold and acquire the correct
target. These first experiments were limited to short timescale adaptation in which the monkey
experienced a particular gain for only several hundred trials. However, in a follow-up experiment, we
held the low or high gain constant for an ‘medium’ time scale of more than an hour and 1,000+ trials.
I found that the additional time did not enable further adaptation or improvement. In contrast, when
we performed similar medium time scale experiments in which the manipulation was a visuomotor
rotation, rather than gain scaling, I did observed both short and medium time scale improvements,
consistent with (Jarosiewicz et al., 2008; Chase et al., 2012; Golub et al., 2015). This argues against
the lack of gain adaptation being due to low motivation or general learning inability on the part of
the monkeys.
Rather, my data suggest that there’s something about gain scaling that limits the motor cortical
network’s ability to adapt to and compensate for this perturbation. In theory, if the brain could
just double the firing rates of the neural population driving the BMI during under the gain 0.5
condition, or halve firing rates under the gain 2 condition, this would fully compensate for the changed
neural-to-kinematic gain mapping. The apparent inability of the network to do so implies that there
are constraints on what kind of adaptation it is capable of: in particular, there are limits to how
much firing rates of many neurons can be simultaneously gained up or down. One possible reason for
this is that the network has a limited dynamic range, perhaps optimized for the range of firing rates
useful for generating natural movements. If so, perhaps it is very difficult, if not impossible, for the
network to re-organize in order to operate outside this range — at least on short to medium time
scales. This study informs BMI designers to be aware of these constraints: for example, we should
build decoders that can output the full range of desired commands in response to neural inputs that
fall within the reachable dynamic range of the neural population being recorded from.
5.3.5 Do motor cortical dynamical rules constrain the activity that can
be generated for BMI control?
My inspiration for this study was a conversation I had with a BrainGate clinical trial participant
during my pre-doctoral research. This participant employed a BMI control strategy in which she
CHAPTER 5. CONCLUSION 103
imagined moving a computer mouse with her hand. During one experiment in which the decoder
developed a bias that constantly pushed the cursor in one direction, the participant reported having
difficulty sustaining cursor velocity-generating mental imagery in the opposing direction because in
her mental imagery, her “hand had reached the limit of its range of motion”. In other words, she was
reporting the BMI equivalent of running one’s computer mouse off its mousepad. This anecdote made
me wonder whether motor cortex’s legacy of generating output for natural motor control imposed
constraints on what mental imagery — and by extension, patterns of neural activity — it could
generate. For instance, perhaps one cannot imagine a continual reaching movement, and hence one
cannot sustain the motor cortical correlates of a point-to-point reach.
These constraints are more formally described in an emerging view that motor cortex is a
dynamical system governed by a lawful set of dynamical “laws” (e.g. Churchland et al., 2010, 2012;
Shenoy et al., 2013). I wanted to probe how rigid these dynamics are in the context of BMIs: will
they impose constraints on the neural activity that can be generated to control the BMI? To test
this, I conducted a set of experiments in two monkeys testing whether motor cortex can generate
sustained reach-like activity for much longer than the duration of typical arm reaches. I did this
by dramatically reducing the gain between decoded velocity and cursor velocity; this had the effect
of asking the monkey to generate strong neural push for many seconds, rather than the several
hundred milliseconds that would normally be needed to reach the target. My preliminary analyses of
these data suggest that neural push can be sustained, which would be consistent with firing rates
being able to dwell in a reach-like state when the BMI task demands it. This would suggest that
motor cortex need not always adhere to the rapidly “unwinding” dynamical system rules described
by recent studies — that is, that the time-varying evolution imposed by neural dynamics can be
“paused” rather than rigidly obeyed. However, substantially more analysis is needed to flesh out this
story about whether neural dynamics constrain BMI control.
Bibliography
Aflalo, T., Kellis, S., Klaes, C., Lee, B., Shi, Y., Pejsa, K., Shanfield, K., Hayes-Jackson, S., Aisen,
M., Heck, C., Liu, C., & Andersen, R. A. (2015). Decoding motor imagery from the posterior
parietal cortex of a tetraplegic human. Science, 348, 906–910.
Ames, K. C., Ryu, S. I., & Shenoy, K. V. (2014). Neural Dynamics of Reaching following Incorrect
or Absent Motor Preparation. Neuron, 81, 438–451.
Andersen, R. A., Musallam, S., & Pesaran, B. (2004). Selecting the signals for a brain-machine
interface. Current opinion in neurobiology, 14, 720–6.
Anderson, K. D. (2004). Targeting recovery: priorities of the spinal cord-injured population. Journal
of neurotrauma, 21, 1371–1383.
Archambault, P. S., Ferrari-Toniolo, S., & Battaglia-Mayer, A. (2011). Online control of hand
trajectory and evolution of motor intention in the parietofrontal system. Journal of Neuroscience,
31, 742–52.
Asher, I., Stark, E., Abeles, M., & Prut, Y. (2007). Comparison of direction and object selectivity
of local field potentials and single units in macaque posterior parietal cortex during prehension.
Journal of Neurophysiology, 97, 3684–3695.
Bacher, D., Jarosiewicz, B., Masse, N. Y., Stavisky, S. D., Simeral, J. D., Newell, K., Oakley, E. M.,
Cash, S. S., Friehs, G., & Hochberg, L. R. (2015). Neural Point-and-Click Communication by a
Person With Incomplete Locked-In Syndrome. Neurorehabilitation and Neural Repair, 29, 462–471.
Bansal, A. K., Truccolo, W., Vargas-Irwin, C. E., & Donoghue, J. P. (2011a). Decoding 3D reach
and grasp from hybrid signals in motor and premotor cortices: spikes, multiunit activity, and local
field potentials. Journal of Neurophysiology, 107, 1337–55.
Bansal, A. K., Vargas-Irwin, C. E., Truccolo, W., & Donoghue, J. P. (2011b). Relationships
among low-frequency local field potentials, spiking activity, and three-dimensional reach and grasp
kinematics in primary motor and ventral premotor cortices. Journal of Neurophysiology, 105,
1603–19.
104
BIBLIOGRAPHY 105
Barrese, J. C., Rao, N., Paroo, K., Triebwasser, C., Vargas-Irwin, C., Franquemont, L., & Donoghue,
J. P. (2013). Failure mode analysis of silicon-based intracortical microelectrode arrays in non-human
primates. Journal of Neural Engineering, 10, 066014.
Bastian, A., Riehle, A., Erlhagen, W., & Schoner, G. (1998). Prior information preshapes the
population representation of movement direction in motor cortex. Neuroreport, 9, 315–9.
Belitski, A., Gretton, A., Magri, C., Murayama, Y., Montemurro, M. A., Logothetis, N. K., &
Panzeri, S. (2008). Low-frequency local field potentials and spikes in primary visual cortex convey
independent visual information. Journal of Neuroscience, 28, 5696–709.
Bensmaia, S. J. & Miller, L. E. (2014). Restoring sensorimotor function through intracortical
interfaces: progress and looming challenges. Nature Reviews Neuroscience, 15, 313–25.
Bishop, W., Chestek, C. C., Gilja, V., Nuyujukian, P., Foster, J. D., Ryu, S. I., Shenoy, K. V., & Yu,
B. M. (2014). Self-recalibrating classifiers for intracortical brain-computer interfaces. Journal of
Neural Engineering, 11, 026001.
Bokil, H. S., Pesaran, B., Andersen, R. A., & Mitra, P. P. (2006). A method for detection and
classification of events in neural activity. IEEE Transactions on Biomedical Engineering, 53,
1678–87.
Buzsaki, G., Anastassiou, C. A., & Koch, C. (2012). The origin of extracellular fields and currents
EEG, ECoG, LFP and spikes. Nature Reviews Neuroscience, 13, 407–420.
Cabel, D., Cisek, P., & Scott, S. (2001). Neural Activity in Primary Motor Cortex Related to
Mechanical Loads Applied to the Shoulder and Elbow During a Postural Task. Journal of
Neurophysiology, pp. 2102–2108.
Caminiti, R., Johnson, P., & Urbano, A. (1990). Making arm movements within different parts of
space: dynamic aspects in the primate motor cortex. Journal of Neuroscience, 10, 2039.
Carmena, J. M., Lebedev, M. A., Crist, R. E., O’Doherty, J. E., Santucci, D. M., Dimitrov, D. F.,
Patil, P. G., Henriquez, C. S., & Nicolelis, M. A. L. (2003). Learning to control a brain-machine
interface for reaching and grasping by primates. PLoS biology, 1, 193–208.
Castiello, U. (2005). The neuroscience of grasping. Nature Reviews Neuroscience, 6, 726–36.
Chase, S. M., Kass, R. E., & Schwartz, A. B. (2012). Behavioral and neural correlates of visuomotor
adaptation observed through a brain-computer interface in primary motor cortex. Journal of
Neurophysiology, 108, 624–44.
BIBLIOGRAPHY 106
Chase, S. M., Schwartz, A. B., & Kass, R. E. (2009). Bias, optimal linear estimation, and the differences
between open-loop simulation and closed-loop performance of spiking-based brain-computer
interface algorithms. Neural Networks, 22, 1203–13.
Chestek, C. A., Gilja, V., Nuyujukian, P., Foster, J. D., Fan, J. M., Kaufman, M. T., Churchland,
M. M., Rivera-Alvidrez, Z., Cunningham, J. P., Ryu, S. I., & Shenoy, K. V. (2011). Long-term
stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor
cortex. Journal of Neural Engineering, 8, 045005.
Churchland, M. M. (2015). Using the precision of the primate to study the origins of movement
variability. Neuroscience, pp. 1–9.
Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Foster, J. D., Nuyujukian, P., Ryu, S. I.,
& Shenoy, K. V. (2012). Neural population dynamics during reaching. Nature.
Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Ryu, S. I., & Shenoy, K. V. (2010). Cortical
Preparatory Activity: Representation ofMovement or First Cog in a Dynamical Machine? Neuron,
68, 387–400.
Churchland, M. M. & Shenoy, K. V. (2007). Delay of movement caused by disruption of cortical
preparatory activity. Journal of Neurophysiology, 97, 348–59.
Cisek, P. & Kalaska, J. F. (2010). Neural mechanisms for interacting with a world full of action
choices. Annual Review of Neuroscience, 33, 269–98.
Collinger, Boninger, M. L., Bruns, T. M., Curley, K., Wang, W., & Weber, D. J. (2013a). Functional
priorities, assistive technology, and brain-computer interfaces after spinal cord injury. The Journal
of Rehabilitation Research and Development, 50, 145.
Collinger, Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., McMorland, A.
J. C., Velliste, M., Boninger, M. L., & Schwartz, A. B. (2013b). High-performance neuroprosthetic
control by an individual with tetraplegia. Lancet, 381, 557–64.
Crammond, D. & Kalaska, J. (1996). Differential relation of discharge in primary motor cortex
and premotor cortex to movements versus actively maintained postures during a reaching task.
Experimental Brain Research, 108, 45–61.
Crammond, D. J. & Kalaska, J. F. (2000). Prior information in motor and premotor cortex: activity
during the delay period and effect on pre-movement activity. Journal of Neurophysiology, 84,
986–1005.
Cunningham, J. P., Nuyujukian, P., Gilja, V., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2010). A
Closed-Loop Human Simulator for Investigating the Role of Feedback-Control in Brain-Machine
Interfaces. Journal of Neurophysiology.
BIBLIOGRAPHY 107
Cunningham, J. P., Nuyujukian, P., Gilja, V., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2011).
A closed-loop human simulator for investigating the role of feedback control in brain-machine
interfaces. Journal of Neurophysiology, 105, 1932–1949.
Dadarlat, M. C., O’Doherty, J. E., & Sabes, P. N. (2014). A learning-based approach to artificial
sensory feedback leads to optimal integration. Nature Neuroscience.
Dangi, S., Moorman, H. G., Orsborn, A. L., & Carmena, J. M. (2014). Continuous Closed-Loop
Decoder Adaptation with a Recursive Maximum Likelihood Algorithm Allows for Rapid
Performance Acquisition in Brain-Machine Interfaces. Neural Computation, 1839, 1811–1839.
Dangi, S., Orsborn, A. L., Moorman, H. G., & Carmena, J. M. (2013). Design and Analysis of
Closed-Loop Decoder Adaptation Algorithms for Brain-Machine Interfaces. Neural Computation,
25, 1693–1731.
Dethier, J., Nuyujukian, P., Ryu, S. I., Shenoy, K. V., & Boahen, K. (2013). Design and validation
of a real-time spiking-neural-network decoder for brain-machine interfaces. Journal of Neural
Engineering, 10, 036008.
Douglas, S., Kirkpatrick, A., & MacKenzie, I. (1999). Testing pointing device performance and user
assessment with the ISO 9241, Part 9 standard. Proc. ACMSIGCHI Conf. Human Factors Comput.
Syst. (CHI’99), pp. 215–222.
Einevoll, G. T., Kayser, C., Logothetis, N. K., & Panzeri, S. (2013). Modelling and analysis of local
field potentials for studying the function of cortical circuits. Nature Reviews Neuroscience, 14,
770–785.
Ethier, C., Gallego, J., & Miller, L. (2015). Brain-controlled neuromuscular stimulation to drive
neural plasticity and functional recovery. Current Opinion in Neurobiology, 33, 95–102.
Ethier, C. & Miller, L. E. (2014). Neurobiology of Disease Brain-controlled muscle stimulation for
the restoration of motor function. Neurobiology of Disease.
Ethier, C., Oby, E. R., Bauman, M. J., & Miller, L. E. (2012). Restoration of grasp following paralysis
through brain-controlled stimulation of muscles. Nature, 485, 368–371.
Evarts, E., Fromm, C., Kroller, J., & Jennings, V. (1983). Motor Cortex control of finely graded
forces. Journal of Neurophysiology, 49, 1199–1215.
Evarts, E. V. (1968). Relation of pyramidal tract activity to force exerted during voluntary movement.
Journal of Neurophysiology, 31, 14–27.
BIBLIOGRAPHY 108
Even-Chen, N., Stavisky, S. D., Kao, J. C., Ryu, S. I., & Shenoy, K. V. (2015). An auto-correcting
brain machine interface: Error detection using spiking neuronal activity in the motor cortex. In
37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.
Fan, J. M., Nuyujukian, P., Kao, J. C., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2014). Intention
estimation in brainmachine interfaces. Journal of Neural Engineering, 11, 016004.
Fetz, E. E. (1969). Operant conditioning of cortical unit activity. Science, 163, 955–958.
Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude
of movement. Journal of Experimental Psychology, 47, 381–391.
Flint, R. D., Ethier, C., Oby, E. R., Miller, L. E., & Slutzky, M. W. (2012a). Local field potentials
allow accurate decoding of muscle activity. Journal of Neurophysiology, 108, 18–24.
Flint, R. D., Lindberg, E. W., Jordan, L. R., Miller, L. E., & Slutzky, M. W. (2012b). Accurate
decoding of reaching movements from field potentials in the absence of spikes. Journal of Neural
Engineering, 9, 046006.
Flint, R. D., Wright, Z. A., Scheid, M. R., & Slutzky, M. W. (2013). Long term, stable brain
machine interface performance using local field potentials and multiunit spikes. Journal of Neural
Engineering, 10, 056005.
Foundation, C. &. D. R. (2009). One Degree of Separation: Paralysis and Spinal Cord Injury in the
United States. Tech. rep.
Fraser, G. W., Chase, S. M., Whitford, A., & Schwartz, A. B. (2009). Control of a brain-computer
interface without spike sorting. Journal of Neural Engineering, 6, 055004.
Ganguly, K. & Carmena, J. M. (2009). Emergence of a stable cortical map for neuroprosthetic
control. PLoS biology, 7, e1000153.
Ganguly, K., Dimitrov, D. F., Wallis, J. D., & Carmena, J. M. (2011). Reversible large-scale
modification of cortical networks during neuroprosthetic control. Nature Neuroscience, 14, 662–667.
Georgopoulos, A., Kalaska, J., Caminiti, R., & Massey, J. (1983). Interruption of Motor Cortical
Discharges Subserving Aimed Arm Movements. Experimental Brain Research, 49, 327–340.
Georgopoulos, A., Kettner, R., & Schwartz, A. (1988). Primate motor cortex and free arm movements
to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal
population. Journal of Neuroscience, 8, 2928–37.
Gilja, V., Chestek, C., Diester, I., Henderson, J., Deisseroth, K., & Shenoy, K. (2011). Challenges and
Opportunities for Next-Generation Intra-Cortically Based Neural Prostheses. IEEE transactions
on bio-medical engineering, XX, 1–10.
BIBLIOGRAPHY 109
Gilja, V., Nuyujukian, P., Chestek, C. A., Cunningham, J. P., Yu, B. M., Fan, J. M., Churchland,
M. M., Kaufman, M. T., Kao, J. C., Ryu, S. I., & Shenoy, K. V. (2012). A high-performance
neural prosthesis enabled by control algorithm design. Nature Neuroscience, 15, 1752–7.
Gilja, V., Pandarinath, C., Blabe, C. H., Nuyujukian, P., Simeral, J. D., Sarma, A. A., Sorice, B. L.,
Perge, J. A., Jarosiewicz, B., Hochberg, L. R., Shenoy, K. V., & Henderson, J. M. (2015). Clinical
translation of a high-performance neural prosthesis. Nature Medicine, 21, 1142–1145.
Gold, C., Henze, D. a., Koch, C., & Buzsaki, G. (2006). On the origin of the extracellular action
potential waveform: A modeling study. Journal of Neurophysiology, 95, 3113–3128.
Golub, M. D., Yu, B. M., & Chase, S. M. (2015). Internal models for interpreting neural population
activity during sensorimotor control. eLife, 4.
Graf, A. B. A. & Andersen, R. A. (2014). Brain-machine interface for eye movements. Proceedings
of the National Academy of Sciences of the United States of America, 111, 17630–5.
Green, A. M. & Kalaska, J. F. (2011). Learning to move machines with the mind. Trends in
Neurosciences, 34, 61–75.
Hatsopoulos, N. G. & Donoghue, J. P. (2009). The science of neural interface systems. Annual
Review of Neuroscience, 32, 249–66.
Heldman, D. A., Wang, W., Chan, S. S., & Moran, D. W. (2006). Local Field Potential Spectral
Tuning in Motor Cortex During Reaching. IEEE Transactions on Neural Systems and Rehabilitation
Engineering, 14, 180–183.
Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., Haddadin, S.,
Liu, J., Cash, S. S., van der Smagt, P., & Donoghue, J. P. (2012). Reach and grasp by people with
tetraplegia using a neurally controlled robotic arm. Nature, 485, 372–5.
Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner,
A., Chen, D., Penn, R. D., & Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic
devices by a human with tetraplegia. Nature, 442, 164–71.
Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9,
1735–1780.
Homer, M. L., Nurmikko, A. V., Donoghue, J. P., & Hochberg, L. R. (2013). Sensors and decoding
for intracortical brain computer interfaces. Annual Review of Biomedical Engineering, 15, 383–405.
Huggins, J. E., Wren, P. A., & Gruis, K. L. (2011). What would brain-computer interface users
want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotrophic
Lateral Sclerosis, 12, 318–324.
BIBLIOGRAPHY 110
Hwang, E. J. & Andersen, R. A. (2009). Brain control of movement execution onset using local field
potentials in posterior parietal cortex. Journal of Neuroscience, 29, 14363–14370.
Hwang, E. J. & Andersen, R. A. (2013). The utility of multichannel local field potentials for
brain-machine interfaces. Journal of Neural Engineering, 10, 046005.
Jackson, A. & Fetz, E. E. (2011). Interfacing With the Computational Brain. IEEE Transactions on
Neural Systems and Rehabilitation Engineering, 19, 534–541.
Jarosiewicz, B., Chase, S. M., Fraser, G. W., Velliste, M., Kass, R. E., & Schwartz, A. B.
(2008). Functional network reorganization during learning in a brain-computer interface paradigm.
Proceedings of the National Academy of Sciences of the United States of America, 105, 19486–91.
Jarosiewicz, B., Masse, N. Y., Bacher, D., Cash, S. S., Eskandar, E., Friehs, G., Donoghue, J. P.,
& Hochberg, L. R. (2013). Advantages of closed-loop calibration in intracortical brain-computer
interfaces for people with tetraplegia. Journal of Neural Engineering, 10, 046012.
Jarosiewicz, B., Sarma, A. A., Bacher, D., Masse, N. Y., Simeral, J. D., Sorice, B., Oakley, E. M.,
Blabe, C., Pandarinath, C., Gilja, V., Cash, S. S., Eskandar, E. N., Friehs, G., Henderson, J. M.,
Shenoy, K. V., Donoghue, J. P., & Hochberg, L. R. (2015). Virtual typing by people with tetraplegia
using a self-calibrating intracortical brain-computer interface. Science Translational Medicine, 7,
313ra179–313ra179.
Kalaska, J. & Drew, T. (1993). Motor cortex and visuomotor behavior. Exercise and Sport Sciences
Reviews.
Kalaska, J. F., Cohen, D. A., Hyde, M. L., & Prud’homme, M. (1989). A comparison of
movement direction-related versus load direction-related activity in primate motor cortex, using a
two-dimensional reaching task. Journal of Neuroscience, 9, 2080–2102.
Kandel, E., Schwartz, J., & Jessel, T. (2000). Principles of Neural Science. (New York: McGraw-Hill),
4th edn.
Kao, J. C., Nuyujukian, P., Stavisky, S., Ryu, S. I., Ganguli, S., & Shenoy, K. V. (2013). Investigating
the role of firing-rate normalization and dimensionality reduction in brain-machine interface
robustness. In 2013 35th Annual International Conference of the IEEE Engineering in Medicine
and Biology Society (EMBC), vol. 2013, pp. 293–298. (IEEE).
Kao, J. C., Stavisky, S. D., Sussillo, D., Nuyujukian, P., & Shenoy, K. V. (2014). Information Systems
Opportunities in Brain-Machine Interface Decoders. Proceedings of the IEEE, 102, 666–682.
Katzner, S., Nauhaus, I., Benucci, A., Bonin, V., Ringach, D. L., & Carandini, M. (2009). Local
origin of field potentials in visual cortex. Neuron, 61, 35–41.
BIBLIOGRAPHY 111
Kaufman, M. T., Churchland, M. M., Ryu, S. I., & Shenoy, K. V. (2014). Cortical activity in the
null space: permitting preparation without movement. Nature Neuroscience, 17, 440–8.
Kennedy, P. R., Kirby, M. T., Moore, M. M., King, B., & Mallory, A. (2004). Computer Control
Using Human Intracortical Local Field Potentials. IEEE Transactions on Neural Systems and
Rehabilitation Engineering, 12, 339–344.
Kim, S.-P., Simeral, J. D., Hochberg, L. R., Donoghue, J. P., & Black, M. J. (2008). Neural control
of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia.
Journal of Neural Engineering, 5, 455–476.
Knyazev, A. V. & Argentati, M. E. (2002). Principal Angles Between Subspaces in An A-Based
Scalar Product: Algorithms and Perturbation Estimates. SIAM Journal on Scientific Computing,
23, 2008–2040.
Koyama, S., Chase, S. M., Whitford, A. S., Velliste, M., Schwartz, A. B., & Kass, R. E. (2010).
Comparison of brain-computer interface decoding algorithms in open-loop and closed-loop control.
Journal of computational neuroscience, 29, 73–87.
Kruger, J., Caruana, F., Volta, R. D., & Rizzolatti, G. (2010). Seven years of recording from monkey
cortex with a chronically implanted multiple microelectrode. Frontiers in neuroengineering, 3, 6.
Kuo, B. C. (1987). Automatic Control Systems. (Englewood Cliffs: Prentice-Hall, Inc.), 5th edn.
Li, C.-S. R., Padoa-Schioppa, C., & Bizzi, E. (2001). Neuronal Correlates of Motor Performance and
Motor Learning in the Primary Motor Cortex of Monkeys Adapting to an External Force Field.
Neuron, 30, 593–607.
Lillicrap, T. P. & Scott, S. H. (2013). Preference Distributions of Primary Motor Cortex Neurons
Reflect Control Solutions Optimized for Limb Biomechanics. Neuron, 77, 168–179.
MacKenzie, I. S. & Buxton, W. (1992). Extending Fitts’ law to two-dimensional tasks. In Proceedings
of the SIGCHI conference on Human factors in computing systems - CHI ’92, pp. 219–226. (New
York, New York, USA: ACM Press).
MacKenzie, I. S. & Ware, C. (1993). Lag as a determinant of human performance in interactive
systems. In Proceedings of the SIGCHI conference on Human factors in computing systems - CHI
’93, pp. 488–493. (New York, New York, USA: ACM Press).
Malik, W. Q., Truccolo, W., Brown, E. N., & Hochberg, L. R. (2011). Efficient decoding with
steady-state Kalman filter in neural interface systems. IEEE Transactions on Neural Systems and
Rehabilitation Engineering, 19, 25–34.
BIBLIOGRAPHY 112
Markowitz, D. A., Wong, Y. T., Gray, C. M., & Pesaran, B. (2011). Optimizing the Decoding of
Movement Goals from Local Field Potentials in Macaque Cortex. Journal of Neuroscience, 31,
18412–18422.
Marr, D. C. & Poggio, T. (1976). From understanding computation to understanding neural circuitry.
Martens, J. (2010). Deep learning via Hessian-free optimization. 27th International Conference on
Machine Learning, 951, 735–742.
Martens, J. & Sutskever, I. (2011). Learning recurrent neural networks with Hessian-free optimization.
Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pp. 1033–1040.
Mehring, C., Rickert, J., Vaadia, E., Cardoso de Oliveira, S., Aertsen, A., & Rotter, S. (2003). Inference
of hand movements from local field potentials in monkey motor cortex. Nature Neuroscience, 6,
1253–4.
Messier, J. & Kalaska, J. F. (2000). Covariation of primate dorsal premotor cell activity with direction
and amplitude during a memorized-delay reaching task. Journal of Neurophysiology, 84, 152–165.
Moran, D. (2010). Evolution of brain-computer interface: action potentials, local field potentials and
electrocorticograms. Current opinion in neurobiology, 20, 741–5.
Moran, D. W. & Schwartz, a. B. (1999). Motor cortical activity during drawing movements: population
representation during spiral tracing. Journal of Neurophysiology, 82, 2693–704.
Morrow, M. M. & Miller, L. E. (2003). Prediction of muscle activity by populations of sequentially
recorded primary motor cortex neurons. Journal of Neurophysiology, 89, 2279–88.
Musallam, S., Corneil, B. D., Greger, B., Scherberger, H., & Andersen, R. A. (2004). Cognitive
Control Signals for Neural Prosthetics. Science, 305, 258–262.
Nicolelis, M. A. L. & Lebedev, M. A. (2009). Principles of neural ensemble physiology underlying
the operation of brainmachine interfaces. Nature Reviews Neuroscience, 10, 530–540.
Nuyujukian, P., Fan, J. M., Gilja, V., Kalanithi, P. S., Chestek, C. A., & Shenoy, K. V. (2011).
Monkey Models for Brain-Machine Interfaces: The Need for Maintaining Diversity. In 33rd Annual
International Conference of the IEEE Engineering in Medicine and Biology Society. Boston, MA.
Nuyujukian, P., Fan, J. M., Kao, J. C., Ryu, S. I., & Shenoy, K. V. (2015). A High-Performance
Keyboard Neural Prosthesis Enabled by Task Optimization. IEEE Transactions on Biomedical
Engineering, 62, 21–29.
Nuyujukian, P., Kao, J. C., Fan, J. M., Stavisky, S. D., Ryu, S. I., & Shenoy, K. V. (2014).
Performance sustaining intracortical neural prostheses. Journal of Neural Engineering, 11, 066003.
BIBLIOGRAPHY 113
O’Doherty, J. E., Lebedev, M. A., Ifft, P. J., Zhuang, K. Z., Shokur, S., Bleuler, H., & Nicolelis,
M. A. L. (2011). Active tactile exploration using a brain-machine-brain interface. Nature, 479,
228–231.
O’Leary, J. G. & Hatsopoulos, N. G. (2006). Early Visuomotor Representations Revealed From
Evoked Local Field Potentials in Motor and Premotor Cortical Areas. Journal of Neurophysiology,
96, 1492–1506.
Omrani, M., Pruszynski, J. A., Murnaghan, C. D., & Scott, S. H. (2014). Perturbation-evoked
responses in primary motor cortex are modulated by behavioral context. Journal of Neurophysiology,
112, 2985–3000.
Orsborn, A. L., Moorman, H. G., Overduin, S. A., Shanechi, M. M., Dimitrov, D. F., & Carmena,
J. M. (2014). Closed-Loop Decoder Adaptation Shapes Neural Plasticity for Skillful Neuroprosthetic
Control. Neuron, 82, 1380–1393.
Pandarinath, C., Gilja, V., Blabe, C., Jarosiewicz, B., Perge, J. A., Hochberg, L. R., Shenoy, K. V.,
& Henderson, J. M. (2014). High-performance communication using neuronal ensemble recordings
from the motor cortex of a person with ALS. In American Society for Stereotactic and Functional
Neurosurgery. Washington, DC.
Pandarinath, C., Gilja, V., Blabe, C. H., Nuyujukian, P., Sarma, A. A., Sorice, B. L., Eskandar,
E. N., Hochberg, L. R., Henderson, J. M., & Shenoy, K. V. (2015). Neural population dynamics in
human motor cortex during movements in people with ALS. eLife, 4, 1–9.
Pasley, B., David, S., Mesgarani, N., & Flinker, A. (2012). Reconstructing Speech from Human
Auditory Cortex. PLoS biology, 10, e1001251.
Perge, J. A., Homer, M. L., Malik, W. Q., Cash, S., Eskandar, E., Friehs, G., Donoghue, J. P., &
Hochberg, L. R. (2013). Intra-day signal instabilities affect decoding performance in an intracortical
neural interface system. Journal of Neural Engineering, 10, 036004.
Perge, J. A., Zhang, S., Malik, W. Q., Homer, M. L., Cash, S., Friehs, G., Eskandar, E. N., Donoghue,
J. P., & Hochberg, L. R. (2014). Reliability of directional information in unsorted spikes and local
field potentials recorded in human motor cortex. Journal of Neural Engineering, 11, 046007.
Pesaran, B., Pezaris, J. S., Sahani, M., Mitra, P. P., & Andersen, R. A. (2002). Temporal structure
in neuronal activity during working memory in macaque parietal cortex. Nature Neuroscience, 5,
805–11.
Pohlmeyer, E. A., Mahmoudi, B., Geng, S., Prins, N. W., & Sanchez, J. C. (2014). Using reinforcement
learning to provide stable brain-machine interface control despite neural input reorganization. PloS
one, 9, e87253.
BIBLIOGRAPHY 114
Pohlmeyer, E. A., Perreault, E. J., Slutzky, M. W., Kilgore, K. L., Kirsch, R. F., Taylor, D. M.,
& Miller, L. E. (2007). Real-time control of the hand by intracortically controlled functional
neuromuscular stimulation. In Proceedings of the 2007 IEEE 10th International Conference on
Rehabilitation Robotics, pp. 454–458.
Pruszynski, J., Omrani, M., & Scott, S. (2014). Goal-Dependent Modulation of Fast Feedback
Responses in Primary Motor Cortex. Journal of Neuroscience, 34, 4608–4617.
Pruszynski, J. A., Kurtzer, I., Nashed, J. Y., Omrani, M., Brouwer, B., & Scott, S. H. (2011a).
Primary motor cortex underlies multi-joint integration for fast feedback control. Nature, 478,
387–90.
Pruszynski, J. A., Kurtzer, I., & Scott, S. H. (2011b). The long-latency reflex is composed of at least
two functionally independent processes. Journal of Neurophysiology, 106, 449–59.
Ramnani, N. (2006). The primate cortico-cerebellar system: anatomy and function. Nature Reviews
Neuroscience, 7, 511–22.
Rathelot, J. & Strick, P. (2009). Subdivisions of primary motor cortex based on cortico-motoneuronal
cells. Proceedings of the National Academy of Sciences, 106, 918.
Rickert, J., Oliveira, S., Vaadia, E., Aersten, A., Rotter S, & Mehring, C. (2005). Encoding of
Movement Direction in Different Frequency Ranges of Motor Cortical Local Field Potentials.
Journal of Neuroscience, 25, 8815–8824.
Riehle, A. & Requin, J. (1989). Monkey primary motor and premotor cortex: single-cell activity
related to prior information about direction and extent of an intended movement. Journal of
Neurophysiology, 61, 534–549.
Ryu, S. I. & Shenoy, K. V. (2009). Human cortical prostheses: lost in translation? Neurosurgical
Focus, 27, E5.
Sadtler, P. T., Quick, K. M., Golub, M. D., Chase, S. M., Ryu, S. I., Tyler-Kabara, E. C., Yu, B. M.,
& Batista, A. P. (2014). Neural constraints on learning. Nature, 512, 423–426.
Santhanam, G., Ryu, S. I., Yu, B. M., Afshar, A., & Shenoy, K. V. (2006). A high-performance
brain-computer interface. Nature, 442, 195–8.
Schearer, E. M., Liao, Y.-W., Perreault, E. J., Tresch, M. C., Memberg, W. D., Kirsch, R. F., &
Lynch, K. M. (2014). Multi-Muscle FES Force Control of the Human Arm for Arbitrary Goals.
IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22, 654–663.
Scherberger, H., Jarvis, M. R., & Andersen, R. A. (2005). Cortical local field potential encodes
movement intentions in the posterior parietal cortex. Neuron, 46, 347–54.
BIBLIOGRAPHY 115
Schwartz, A., Ketter, R., & Georgopoulos, A. (1988). Primate motor cortex and free arm movements
to visual targets in three-dimensional space. I. Relations between single cell discharge and direction
of movement. Journal of Neuroscience, 8, 2913–2927.
Schwartz, A. B. (2004). Cortical neural prosthetics. Annual Review of Neuroscience, 27, 487–507.
Scott, S. H. (2004). Optimal feedback control and the neural basis of volitional motor control. Nature
Reviews Neuroscience, 5, 532–46.
Scott, S. H. (2012). The computational and neural basis of voluntary motor control and planning.
Trends in cognitive sciences, 16, 541–9.
Seo, D., Carmena, J. M., Rabaey, J. M., Maharbiz, M. M., & Alon, E. (2014). Model validation of
untethered, ultrasonic neural dust motes for cortical recording. Journal of Neuroscience Methods,
244, 114–122.
Serruya, M., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., & Donoghue, J. P. (2002). Instant
neural control of a movement signal. Nature, 416, 141–142.
Shadmehr, R. & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of
a motor task. Journal of Neuroscience, 14, 3208–3224.
Shadmehr, R., Smith, M. A., & Krakauer, J. W. (2010). Error correction, sensory prediction, and
adaptation in motor control. Annual Review of Neuroscience, 33, 89–108.
Shenoy, K. V. & Carmena, J. M. (2014). Combining Decoder Design and Neural Adaptation in
Brain-Machine Interfaces. Neuron, 84, 665–680.
Shenoy, K. V., Sahani, M., & Churchland, M. M. (2013). Cortical control of arm movements: a
dynamical systems perspective. Annual Review of Neuroscience, 36, 337–59.
Shmuelof, L. & Krakauer, J. (2011). Are we ready for a natural history of motor learning? Neuron,
72, 469–476.
Simeral, J. D., Kim, S.-P., Black, M. J., Donoghue, J. P., & Hochberg, L. R. (2011). Neural control of
cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical
microelectrode array. Journal of Neural Engineering, 8, 025027.
So, K., Dangi, S., Orsborn, A. L., Gastpar, M. C., & Carmena, J. M. (2014). Subject-specific
modulation of local field potential spectral power during brainmachine interface control in primates.
Journal of Neural Engineering, 11, 026002.
Stark, E. & Abeles, M. (2007). Predicting movement from multiunit activity. Journal of Neuroscience,
27, 8387–94.
BIBLIOGRAPHY 116
Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., & Shenoy, K. V. (2014). Hybrid Decoding
of Both Spikes and Low-Frequency Local Field Potentials for Brain-Machine Interfaces. In 36th
Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp.
3041–3044. Chicago, IL.
Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., & Shenoy, K. V. (2015a). A high performing
brainmachine interface driven by low-frequency local field potentials alone and together with spikes.
Journal of Neural Engineering, 12, 036009.
Stavisky, S. D., Kao, J. C., Sorokin, J. M., Ryu, S. I., & Shenoy, K. V. (2015b). System identification
of brain-machine interface control using a cursor jump perturbation. In 2015 7th International
IEEE/EMBS Conference on Neural Engineering (NER), pp. 643–647. (IEEE).
Suminski, A. J., Mardoum, P., Lillicrap, T. P., & Hatsopoulos, N. G. (2015). Temporal evolution of
both premotor and motor cortical tuning properties reflect changes in limb biomechanics. Journal
of Neurophysiology, p. jn.00486.2014.
Suminski, A. J., Tkach, D. C., Fagg, A. H., & Hatsopoulos, N. G. (2010). Incorporating Feedback from
Multiple Sensory Modalities Enhances Brain-Machine Interface Control. Journal of Neuroscience,
30, 16777–16787.
Sussillo, D., Nuyujukian, P., Fan, J. M., Kao, J. C., Stavisky, S. D., Ryu, S., & Shenoy, K. (2012). A
recurrent neural network for closed-loop intracortical brainmachine interface decoders. Journal of
Neural Engineering, 9, 026027.
Sutskever, I., Martens, J., & Hinton, G. (2011). Generating text with recurrent neural networks. In
Proceedings of the 28th International Conference on Machine Learning. Bellevue, WA.
Tabot, G. A., Dammann, J. F., Berg, J. A., Tenore, F. V., Boback, J. L., Vogelstein, R. J., &
Bensmaia, S. J. (2013). Restoring the sense of touch with a prosthetic hand through a brain
interface. Proceedings of the National Academy of Sciences of the United States of America, 110.
Tanji, J. U. N. & Evarts, V. (1976). Anticipatory Activity of Motor Cortex Neurons in Relation to
Direction of an Intended Movement. Journal of Neurophysiology, 39, 1062–1068.
Taylor, D. M., Tillery, S. I. H., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic
devices. Science, 296, 1829–32.
Todorov, E. (2000). Direct cortical control of muscle activation in voluntary arm movements: a
model. Nature Neuroscience, 3, 391–398.
Todorov, E. & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination.
Nature Neuroscience, 5, 1226–35.
BIBLIOGRAPHY 117
Trautmann, E. M., Stavisky, S. D., Ames, K. C., Kaufman, M. T., Ryu, S. I., & Shenoy, K. V. (2016).
Sortfree: Using threshold crossings to evaluate scientific hypotheses in population analyses. In
Cosyne Abstracts, pp. Trautmann, E.M., Stavisky, S.D., Ames, K.C., Kaufm. Salt Lake City.
Tsu, A. P., Burish, M. J., GodLove, J., & Ganguly, K. (2015). Cortical neuroprosthetics from a
clinical perspective. Neurobiology of Disease.
Vargas-Irwin, C. E., Franquemont, L., Black, M. J., & Donoghue, J. P. (2015). Linking Objects to
Actions: Encoding of Target Object and Grasping Strategy in Primate Ventral Premotor Cortex.
Journal of Neuroscience, 35, 10888–10897.
Vargas-Irwin, C. E., Shakhnarovich, G., Yadollahpour, P., Mislow, J. M. K., Black, M. J., &
Donoghue, J. P. (2010). Decoding complete reach and grasp actions from local primary motor
cortex populations. Journal of Neuroscience, 30, 9659–9669.
Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control
of a prosthetic arm for self-feeding. Nature, 453, 1098–1101.
Vogel, J., Haddadin, S., Simeral, J. D., Stavisky, S. D., Bacher, D., Hochberg, L. R., Donoghue, J. P.,
& van der Smagt, P. (2010). Continuous Control of the DLR Light-weight Robot III by a human
with tetraplegia using the BrainGate2 Neural Interface System. In International Symposium on
Experimental Robotics (ISER2010), pp. 1–12. (New Delhi, India: Springer).
Waldert, S., Lemon, R. N., & Kraskov, A. (2013). Influence of spiking activity on cortical local field
potentials. The Journal of Physiology, 591, 5291–5303.
Wang, D., Zhang, Q., Li, Y., Wang, Y., Zhu, J., Zhang, S., & Zheng, X. (2014). Long-term decoding
stability of local field potentials from silicon arrays in primate motor cortex during a 2D center
out task. Journal of Neural Engineering, 11, 036009.
Willett, F. R., Suminski, A. J., Fagg, A. H., & Hatsopoulos, N. G. (2012). Compensating for delays
in brain-machine interfaces by decoding intended future movement. In 2012 Annual International
Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4087–4090. (IEEE).
Wise, S. P., Boussaoud, D., Johnson, P. B., & Caminiti, R. (1997). Premotor and parietal cortex:
corticocortical connectivity and combinatorial computations. Annual Review of Neuroscience, 20,
25–42.
Wise, S. P., Moody, S. L., Blomstrom, K. J., & Mitz, A. R. (1998). Changes in motor cortical activity
during visuomotor adaptation. Experimental Brain Research., 121, 285–99.
Wodlinger, B., Downey, J. E., Tyler-Kabara, E. C., Schwartz, A. B., Boninger, M. L., & Collinger,
J. L. (2015). Ten-dimensional anthropomorphic arm control in a human brainmachine interface:
difficulties, solutions, and limitations. Journal of Neural Engineering, 12, 016011.
BIBLIOGRAPHY 118
Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995). An internal model for sensorimotor
integration. Science, 269, 1880–1882.
Wolpert, D. M. & Miall, R. C. (1996). Forward Models for Physiological Motor Control. Neural
Networks, 9, 1265–1279.
Wu, W., Black, M. J., Gao, Y., Bienenstock, E., Serruya, M., Shaikhouni, A., & Donoghue, J. P.
(2003). Neural Decoding of Cursor Motion Using a Kalman Filter. Advances in Neural Information
Processing Systems 15: Proceedings of the 2002 Conference, pp. 133–140.
Wu, W., Gao, Y., Bienenstock, E., Donoghue, J. P., & Black, M. J. (2006). Bayesian Population
Decoding of Motor Cortical Activity Using a Kalman Filter. Neural Computation, 118, 80–118.
Wu, W. & Hatsopoulos, N. G. (2008). Real-time decoding of nonstationary neural activity in motor
cortex. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 16, 213–222.
Zagha, E., Ge, X., & McCormick, D. A. (2015). Competing Neural Ensembles in Motor Cortex Gate
Goal-Directed Motor Output. Neuron, 88, 565–577.
Zhuang, J., Truccolo, W., Vargas-Irwin, C., & Donoghue, J. P. (2010). Decoding 3-D reach and
grasp kinematics from high-frequency local field potentials in primate primary motor cortex. IEEE
Transactions on Bio-Medical Engineering, 57, 1774–84.