A Memristive Dynamic Adaptive Neural Network Array

(mrDANNA)

Garrett S. Rose, PhDgarose@utk.edu

March 9, 2016

2

Overview

● Memristive Neuromorphic Systems: Where we've been

– Memristor basics

– Programmable Threshold Logic Arrays

– Gradient descent for memristive circuits

● Memristive Neuromorphic Systems: Where we're going

– Background: The NIDA/DANNA architecture

– Reconfigurable structure in memristive neuromorphic systems

– mrDANNA: a Memristive Dynamic Neural Network Array

3

Overview

● Memristive Neuromorphic Systems: Where we've been

– Memristor basics

– Programmable Threshold Logic Arrays

– Gradient descent for memristive circuits

● Memristive Neuromorphic Systems: Where we're going

– Background: The NIDA/DANNA architecture

– Reconfigurable structure in memristive neuromorphic systems

– mrDANNA: a Memristive Dynamic Neural Network Array

4

● A “fundamental device device property”: memristance

● A memristor (“memory resistor”) similar to variable resistor that can be made to operate in one of many states

● Many interesting applications: nanoscale digital logic, memory (next-gen Flash), neuromorphic computing

The Memristor

5

● SUNY Polytechnic—CNSE fabricating HfO2 memristors

● Complete hybrid CMOS/memristor process

– Memristors in via layer between metal1 and metal2

Ref.: N. Cady, et al., “Towards Memristive Dynamic Adaptive Neural Network Arrays,” GOMAC, March 2016.

Hafnium-Oxide Memristors

6

● Threshold logic a natural fit for memristive circuits

● Memristors are used as “artificial synapses”

● CMOS can be used as neurons

Prior Work: Memristive Programmable Threshold Gate Array

Ref.: J. Rajendran et al., “Memristor-based Prog. Threshold Logic Array,” NANOARCH, 2010.

All Nano Memristive Gate Array

● Memristors used as weights in programmable threshold gates● Resonant tunneling diodes used to construct Goto pair latches

– Provides gain and acts as thresholding circuit

Ref.: G.S. Rose et al., “Leveraging Memristive Systems in Construction of Digital Logic,” Proc of IEEE, 2012.

All Nano Memristive Gate Array

● Memristors used as weights in programmable threshold gates● Resonant tunneling diodes used to construct Goto pair latches

– Provides gain and acts as thresholding circuit

Ref.: G.S. Rose et al., “Leveraging Memristive Systems in Construction of Digital Logic,” Proc of IEEE, 2012.

RTD+Memristor Basic Shape Classification

9

● Supervised learning based on stochastic gradient descent

● “Read-monitored-write” approach with incremental changes in memristance

● Hardware still built with global and local training circuits

Training a Threshold Gate Array

Ref.: H. Manem et al., “Stochastic Gradient Descent for CMOS/Memristive Threshold Gate Array,” TCAS-I, 2012.

10

Overview

● Memristive Neuromorphic Systems: Where we've been

– Memristor basics

– Programmable Threshold Logic Arrays

– Gradient descent for memristive circuits

● Memristive Neuromorphic Systems: Where we're going

– Background: The NIDA/DANNA architecture

– Reconfigurable structure in memristive neuromorphic systems

– mrDANNA: a Memristive Dynamic Neural Network Array

11

● Spiking neural network embedded in 3D space

● Simple neuron and synapse design: 2 parameters per element

● Discrete event simulation

NIDA:Neuro-Inspired Dynamic Architecture

Ref.: C. Schuman, J.D. Birdwell, and M. Dean, “Neuroscience-Inspired Dynamic Architectures,” BSEC, 2014.

12

● Array of programmable neural network elements

● Each element can be connected to up to 16 other elements

● Have implemented using FPGA

● Hardware-accurate simulator available – used for offline evolutionary optimization

Dynamic Adaptive Neural Network Array (DANNA)

Ref.: M. Dean, C. Schuman, and J.D. Birdwell, “Dynamic Adaptive Neural Network Array,” Springer, 2014.

NIDA/DANNA Results – Iris Dataset

14

● Maintain salient features of NIDA/DANNA architecture:– Configurable hardware: structure & weights optimized offline– Dynamic adaptation via LTD/LTP mechanism– Simple neuron and synapse design

● Neurons: programmable threshold & refractory period● Synapses: programmable delay & synaptic weights

● Leverage metal-oxide memristors for synapses– High density synaptic arrays– Potential for low-power operation

Toward a Memristive DANNA (mrDANNA)

Goal: A memristor-based neuromorphic system with reconfigurable structure, parameter initialization and dynamic adaptability.

15

• Memristors for synaptic weights• Memristance (memristor resistance) always

positive – one memristor can only represent positive weight

• Pair of memristors used to represent either positive or negative weight

• One memristor (Rp) drives positive current; other (Rn) negative current

Rp < Rn positive weight→Rp > Rn negative weight→

• Neuron follows integrate-and-fire mechanism

• Integrator accumulates charge to reach threshold in integration phase

• Comparator and “firing flop” generate firing spike during firing phase

• When firing, integrator charge clears• Feedback to synaptic inputs for

potentiation/depression during firing

mrDANNA Circuit Elements

16

● Implemented NIDA rule for LTP/LTD

● LTP: Increase weight for synapse causing a post-synaptic neuron firing event

● LTD: Decrease weight for synapse that fires simultaneous to post-synaptic neuron firing event

● Integrate & Fire neurons – integrates until threshold reached, charge resets upon firing event

LTP and LTD in mrDANNA System

17

Functional Weights

3 1 1 1 3

1 3 1 3 1

3 3 1 3 3

3 1 3 1 3

3 3 1 3 3

- - - -é ùê ú- - -ê úê ú- - -ê ú- - -ê úê úë û

Simple Example: Pattern Recognition of Very Basic Images● Four images:

– Triangle– Square– Diamond– Plus

● Circuit tested using both pure and noisy images● Up to 3 noise bits generated randomly● Functional weight matrix – base of network structure● Determining the weight matrix – difference between

weighted triangle image matrix and all others● Simple network allows easy assessment of memristive

synapses and neurons

0 0 1 0 0

0 1 0 1 0

0 1 0 1 0

1 0 0 0 1

1 1 1 1 1

1 1 1 1 1

1 0 0 0 1

1 0 0 0 1

1 0 0 0 1

1 1 1 1 1

0 0 1 0 0

0 1 0 1 0

1 0 0 0 1

0 1 0 1 0

0 0 1 0 0

0 0 1 0 0

0 0 1 0 0

1 1 1 1 1

0 0 1 0 0

0 0 1 0 0

18

• Each column of weight matrix denotes the weights of each neuron

• Optimization of number of neurons reduces the structure complexity

• Threshold calculation consistent with maximum and minimum synaptic weights

Simulation Results

19

● Build better understanding of constraints for memristive circuits– Guide any needed algorithmic refinement

● Optimize peripheral circuitry for initializing memristive elements● Hardware interfaces and software

– Developing FPGA-based DANNA in parallel – improved interfaces– Software: Evolutionary optimization, mrDANNA simulator, “commander”

and other tools for interfacing with the system● Strike a balance: offline network initialization vs. online training

● Far term consideration:– Nanoscale RTD-based neurons?

Moving Forward:Circuit & Architecture Refinement

20

● Memristors essentially artificial synapses

● Straightforward implementations of threshold logic possible

● Depending on materials used, memristors offer low power operation with reasonable delay

● Developing memristive dynamic adaptive neural network array (mrDANNA)

Summary

● Collaborators:– Nathaniel Cady (CNSE)– Joseph Van Nostrand (AFRL)– Mark Dean (UTK)– James Plank (UTK)– Catherine Schuman (ORNL)– Thomas Potok (ORNL)– Robert Patton (ORNL)– Steven Young (ORNL)

● Sponsors:

Acknowledgements

● Students:– Gangotree Chakma– Mesbah Uddin– Adam Disney– Patricia Eckhart– John Reynolds– Austin Wyer– Elvis Offor– Miles Gepner– Daniel Caballero