INTRODUCTION

01/22/2020

COMPSCI0591/691NR0Neural0Networks0and0Neurodynamics

Instructor:0Robert0KozmaTA:0Devdhar Patel0

Course'Description'COMPSCI'591/691NR'covers'various'aspect'of'neural'networks,'from'

fundamentals'to'advanced'concepts.'Topics'include'feed7forward'neural'networks,'kernel7based'approaches,'deep'learning,'recurrent'neural'networks,'Hopfield'networks,'Kohonen'Self7Organized'Maps,'Helmholtz'machines,'MDL,'Grossberg'Adaptive'Resonance'Theory,'space7time'neurodynamics,'with'links'to'computational'neuroscience.'Theoretical'foundations'of'supervised,'unsupervised,'and'reinforcement'learning'are'described.'Advanced'machine'learning'applications'include'image'processing,'speech'recognition,'game'playing,'time'series'prediction,'and'neurocontrol.'The'course'is'self7contained,'preliminary'knowledge'of'neural'networks'basics'is'useful'but'not'required.'Students'at'the'600'level'are'expected'to'complete'a'project'implementing'a'neural'network'to'solve'a'pattern'recognition'task,'while'students'at'the'500'level'will'be'evaluated'based'on'assignments'with'conceptual'designs.'

Comparison*Between*COMPSCI*682*and*COMPSCI*591NR/691NR

COMPSCI*682 teaches'the'engineering'techniques'necessary'to'train'modern'neural'network'architectures'to'achieve'competitive'performance'on'problems'such'as'image'classification'and'sequence'prediction.'This'includes'optimization'techniques,'hyperparameter'setting,'and'debugging'techniques'for'large'neural'networks.'There'is'less'emphasis'on'theory.'

591NR/691NR introduces'a'broader'array'of'neural'network'models'beyond'feed=forward'networks'(such'as'Kohonen'nets,'Hopfield'nets,'Boltzmann'machines,'Adaptive'Resonance'Theory)'and'analyzes'some'of'their'theoretical'properties,'such'as'guaranteed'convergence'and'stability.'In'general,'there'is'more'emphasis'on'theoretical'properties,'and'less'on'engineering'issues.'

Neural'Networks'…'as

• Key$part$of$AI$(Deep$Learning)• Models$of$nervous$system• Models$of$behavior,$cognition• Statistical$tool• (Universal)$computational$tool• Mathematical$object$(graph,$network,…)• Electrical$circuit• Model$of$physical$objects$(crystal,$spinFglass)

What%IS a%Neural%Network%??

• A"system"containing"a"large"amount"of"‘simple’"components

• The"components"are"connected in"a"generally"complicated"way

• The"behavior"of"the"system"as"a"whole"is"not"obviously"present"in"parts;>emergence

“The%Computer%&%the%Brain”by%John%Von%Neumann%(1958)

• Design(Computers(by(modeling(brains• Silliman(lectures,(Yale,(New(Haven• Analog(vs.(digital(machines• Parallel(vs.(serial(schemes• Need(for(specialized(memory(unit• Control(by(sequence(of(commands• Precision(and(speed(issues(• Automaton(code

• Complete(code• Short(code(:(imitate(a(meaningful(behavior(• Turing(test((1937);(Decide(intelligence

• Language(of(the(brain(>>(Not(mathematics(!!John%Von%Neumann%

(190311957)

AlphaGo vs.+Human+Go+World+Champion+Lee+Sedol loses+(2016)

Alpha&Go&is&Cutting&Edge&!Progress'since'IBM'Deep'Blue'(1997)

!Not'just'executing'existing'rules,'also'learning'from'examples'of'human'players.

Deep&Learning&(DL)!DL'is'today’s'best'ML'approach.

!Multilayer,'feedFforward'architecture'with'supervised'learning,'and'unsupervised'and'reinforcement'components.

!Huge'demand'on'computing'and'data'resources

Tasks&Ahead!Limitations'due'to'expanding'resource'demand.!Combine'advantages'of'DL'with'learning'from'biological'principles

Future&Progress&of&AI• Where&does&AI&leads&us?&• Cyborgs?• Clone&army?• Mutual&destruction?• How&can&AI&be&trusted?

Elon Musk

?

https://www.tensorflow.org/performance/benchmarks

blog.openai.com/ai6and6compute/

Computational+Demand+Deep$Learning$versus$Brains$

A"petaflop/s,day"(pfs,day)"consists"of"performing"1015 neural"net"operations"per"second"for"one"day,"or"total"approx"1020 operations

Hardware #neuronTotal.Power.Use.W

Power.W.per.neuron

Power.W.for.

20x10^9.neurons.

Biology Human,-Neocortical 20x10^9 10.00, 5x10^-10 10.00

Neuromorphic

TrueNorth,-Izhikevich

1,048,567 0.15 1.4x10^-7 2.8x10^3

Analog Neurogrid 1x10^6 3, 3x10^-6 6x10^4

FPGA SNAVA,-Izhikevich 12,800 0.625 4.88x10^-5 9.7x10^5

ARM SpiNNaker,–SNN 18x10^3 1, 5.5x10^-5 1.1x10^6

GPU NCS6,-Izhikevich 1x10^6 2847, 2.84x10^-3 5.6x10^7

CPU

K,computer,-SNN88,128,

processors

1.86x10^9

12.6,x,10^6 6.77x10^3 1.3x10^8

AlexNet! AlphaGo Energy+Use+of+Chips+&+Brains

Computers*and*BrainsSeason 2

0 Learning*the*Secret*of*Brain*Efficiency*0

High%precision-hardware.--------------------------------------- Messy-wetwareHigh-power-consumption------------ Low-power-needs-Millions-of-Watts(MW) A-few-Watts-(W)-

Supercomputer

New$Computing$Paradigm

• Natural$processes$are$complex$and$essentially$nonlinear• they%cannot%be%described%by%traditional%math• usual%tools%also%insufficient%(rely%on%calculus)• 55>%discrepancy%between%modern%computers%and%%%

today’s%computational%tools• Potential$solution

• NNs%as%distributed%parameter%systems%• 55>%free%of%a%lot%of%assumptions• more%general%than%Differential%Equations• high%level%of%knowledge%also%can%be%extracted%using%AI,%

rules,%...%

Computational+paradigm+shift+(cont’d)

• Key+role+of+:+connectivity+• instead(of(functional+relationship(for(300+(

years

• Features+of+connectivity• domain:(spatial(444 temporal• intensity:(weak(444 strong• extent:(short(4 medium(4 long4range• nature:(local(444 partial(444 overall

• Intermediate+effects+(partial+connectivity)• Rules(&(AI(= partial(connectivity• can(generate(low(dimensional(structures• structures(express(a(delicate(balance(between(

infinite(dilution(of(full(coupling and(extreme(fragmentation(of(locally(interacting(parts

Coupled(Oscillators(Multi4level(representation

with(or(w/o(external(driving(force

Data$Processing$Basics• Measurement

• collecting)data)on)an)object• active)/ passive,)quantitative)/ qualitative

• Experiment• active)measurement)for)hypothesis)test• goal:)achieve)repeatability)under)known)conditions

• Control• achieve)a)desired)system)state• measured)inputs,)adjust)control,)target)output

SYSTEM Outputvariables

Inputvariables

Controlparameters

Concepts)in)Systems)Analysis

• Detectability• of,known,changes,in,the,system,state

• Repeatability• of,system,behaviour

• Controllability• to,reach/maintain,target,system,state

Data$Processing$Steps

1.#Preprocessingworking#with#raw#dataconvert#them#into#suitable#format

2.#Model#buildingmaking#assumption#on#the#systeme.g.,#mathematical,#statistical,#connectionist

3.#Evaluation#of#the#developed#modeltesting#with#new#datadetermine#the#limitations#of#the#model

Data$Processing$Scheme.

OBJECT

system

PRE.PROC.digitisefilter

normaliseamplify

ANALYSIS

model.basedorAempirical

RESULT

predict.knowled.Identify/recognis.

FeedbackAloop

Statistical(Pattern(Recognition(Fundamentals

Data$setSample,$population

ClassificationClass$label,$statistical$classifier

GeneralizationHigh:dimensional$data,$use$only$subset$is$practical

FeaturesCompression,$feature$extraction

Statistical(Features(Histogram

Probability.density.function

Frequency.spectraPower.spectral.density.function.(time.and.space)

Misclassification.rateDecision.criteria,.optimum.thresholdCrossing.of.histograms

Classification.as.mappingInput.data.to.class.labelsParametric.and.nonAparametric.classifiers

NN.as.a.nonlinear.classifierWeights.as.parametersLearning:.optimum.choice.of.weights