CS 510: Lecture 8greenie/cs510/CS510-17-08.pdfDeep Learning Overview • Train networks with many...
Transcript of CS 510: Lecture 8greenie/cs510/CS510-17-08.pdfDeep Learning Overview • Train networks with many...
CS 510: Lecture 8Deep Learning, Fairness, and Bias
Next Week
• All Presentations, all the time
• Upload your presentation before class if using slides
• Sign up for a timeslot google doc, if you haven’t already done so
Artificial Neural Networks: History
• Belief that it was necessary to model underlying brain architecture for AI
• In contrast to encoded symbolic knowledge (best represented by expert systems)
• Hebb - learning is altering strength of synaptic connections
Neural Networks
• Attempt to build a computation system based on the parallel architecture of brains
• Characteristics:
• Many simple processing elements
• Many connections
• Simple messages
• Adaptive interaction
Brains
1011 neurons of > 20 types, 1014 synapses, 1ms–10ms cycle timeSignals are noisy “spike trains” of electrical potential
Axon
Cell body or Soma
Nucleus
Dendrite
Synapses
Axonal arborization
Axon from another cell
Synapse
Chapter 20, Section 5 3
Benefits of NN
• User friendly (well, reasonably)
• Non-linear
• Noise tolerant
• Many applications
• Credit fraud/assignment
• Robotic Control
“Neurons”
• Inputs (either from outside or other “neurons”)
• Weighted connections that correspond to synaptic efficiency
• Threshold values to weight the inputs
• Passed through activation function to determine output
Example Unit
• Binary input/output
• Rule
• 1 if w0*I0 + w1*I1 +wb > 0
• 0 if w0*I0 + w1*I1 +wb <= 0
Activation functions
(a) (b)
+1 +1
iniini
g(ini)g(ini)
(a) is a step function or threshold function
(b) is a sigmoid function 1/(1 + e−x)
Changing the bias weight W0,i moves the threshold location
Chapter 20, Section 5 5
Note similarity to logistic regression...
W1
W2
W3
f(x)
1.4
-2.5
-0.06
2.7
-8.6
0.002
f(x)
1.4
-2.5
-0.06
x = -0.06×2.7 + 2.5×8.6 + 1.4×0.002 = 21.34
How to Adapt?
• Perceptron Learning Rule
• change the weight by an amount proportional to the difference between the desired output and the actual output.
• As an equation: ΔWi = η * (D - Y)Ii, where D is desired output and Y is actual output
• Stop when converges
Limits of Perceptrons
• Minsky and Papert 1969
• Fails on “linearly inseparable” instances
• XOR
• linearly separable - pattern space can be separated by single hyperplane
Perceptrons vs Decision Trees
Multilayer Perceptrons (MLP)
Back Propagation
• Start with a set of known examples (supervised approach)
• Assign random initial weights
• Run examples through and calculate the mean-squared error
• Propagate the error by making small changes to the weights at each level
• Use chain rule to calculate the gradient efficiently
• Lather, rinse, repeat
Gradient Descent Algorithm
Have some function
Want
Outline:
• Start with some
• Keep changing to reduce
until we hopefully end up at a minimum
J(θ0,θ1)
The gradient of J (∇J) at a point can be thought of as a vector indicating which way is “uphill”
If J is an error function we want to move “downhill”- opposite to the gradient
Gradient descent algorithm
Have function JWant to produce vectors s.t. J(θ1)>J(θ2)>... start w/ θ0θi+1 = θi - ɑi∇J(θi)
ɑ(alpha) is the learning rate
Stochastic Gradient Descent
Update J every time you look at a training example
Some non-linear activation functions…
Most common activation function
A dataset Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Training the neural network Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Initialise with random weights
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Present a training pattern
1.4
2.7
1.9
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Feed it through to get output
1.4
2.7 0.8
1.9
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Compare with target output
1.4
2.7 0.8 0 1.9 error 0.8
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Adjust weights based on error
1.4
2.7 0.8 0 1.9 error 0.8
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Present a training pattern
6.4
2.8
1.7
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Feed it through to get output
6.4
2.8 0.9
1.7
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Compare with target output
6.4
2.8 0.9 1 1.7 error -0.1
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
Adjust weights based on error
6.4
2.8 0.9 1 1.7 error -0.1
Training data Fields class 1.4 2.7 1.9 0 3.8 3.4 3.2 0 6.4 2.8 1.7 1 4.1 0.1 0.2 0 etc …
And so on ….
6.4
2.8 0.9 1 1.7 error -0.1
Repeat this thousands, maybe millions of times – each time taking a random training instance, and making slight weight adjustments Algorithms for weight adjustment are designed to make changes that will reduce the error
The decision boundary perspective…Initial random weights
The decision boundary perspective…Present a training instance / adjust the weights
The decision boundary perspective…Present a training instance / adjust the weights
The decision boundary perspective…Present a training instance / adjust the weights
The decision boundary perspective…Present a training instance / adjust the weights
The decision boundary perspective…Eventually ….
The point I am trying to make• weight-learning algorithms for NNs are dumb
• they work by making thousands and thousands of tiny adjustments, each making the network do better at the most recent pattern, but perhaps a little worse on many others
• but, by dumb luck, eventually this tends to be good enough to learn effective classifiers for many real applications
Some other points
Detail of a standard NN weight learning algorithm – later
If f(x) is non-linear, a network with 1 hidden layer can, in theory, learn perfectly any classification problem. A set of weights exists that can produce the targets from the inputs. The problem is finding them.
Some other ‘by the way’ pointsIf f(x) is linear, the NN can only draw straight decision boundaries (even if there are many layers of units)
Some other ‘by the way’ pointsNNs use nonlinear f(x) so they can draw complex boundaries, but keep the data unchanged
Some other ‘by the way’ pointsNNs use nonlinear f(x) so they SVMs only draw straight lines, can draw complex boundaries, but they transform the data first but keep the data unchanged in a way that makes that OK
Neural network vocabulary
• Neuron = logistic regression or similar function
• Input layer = input training/test vector
• Bias unit = intercept term/always on feature
• Activation = response
• Activation function is a logistic (or similar “sigmoid” nonlinearity)
• Backpropagation = running stochastic gradient descent across a multilayer network
• Weight decay - regularization or Bayesian prior
Deep Learning
• Most current machine learning works well because of human-designed representations and input features
• Machine learning becomes just optimizing weights to best make a final prediction
• Representation learning attempts toautomatically learn good features or representations
• Deep learning algorithms attempt to learn multiple levels of representation of increasing complexity/abstraction
Deep Architecture
Deep Learning Overview
• Train networks with many layers (vs. shallow nets with just a couple of layers)
• Multiple layers work to build an improved feature space
• First layer learns 1st order features (e.g. edges…)
• 2nd layer learns higher order features (combinations of first layer features, combinations of edges, etc.)
• In current models layers often learn in an unsupervised mode and discover general features of the input space – serving multiple tasks related to the unsupervised instances (image recognition, etc.)
• Then final layer features are fed into supervised layer(s)
• And entire network is often subsequently tuned using supervised training of the entire net, using the initial weightings learned in the unsupervised phase
• Could also do fully supervised versions, etc. (early BP attempts)
Why Deep Learning?
Learning Representations
• Handcrafting features is time-consuming
• The features are often both over-specified and incomplete
• The work has to be done again for each task/domain/...
• We must move beyond handcrafted features and simple ML• Humans develop representations for learning and reasoning
• Our computers should do the same
The Curse of Dimensionality
Unsupervised Feature and Weight Learning
• Today, most practical, good NLP& ML methods require labeled training data (i.e., supervised learning)
• But almost all data is unlabeled
• Most information must be acquired unsupervised
• Fortunately, a good model of observed data can really help you learn classification decisions
Learning Multiple Levels of Representation
Successive Layers Learn Deeper Representations
edges
object parts(combination of edges)
object models
pixels
Impressive Results Especially on Large Datasets
• Object Recognition - better than anything out there
• Speech Recognition (google voice search)
• Many other perceptual tasks in vision and NLP
Why now?
• Bigger Data - deep learning works best
• Better Hardware - multicore CPUs and GPUs
• Better Algorithms - autoencoders, deep belief networks, etc
• Let us train multiple inner layers well
Breakthrough: Unsupervised Pre-training
Difficulties with Supervised Networks
• Early layers of MLP do not get trained well
• Diffusion of Gradient – error attenuates as it propagates to earlier layers
• Leads to very slow training
• Exacerbated since top couple layers can usually learn any task "pretty well" and thus the error to earlier layers drops quickly as the top layers "mostly" solve the task– lower layers never get the opportunity to use their capacity to improve results, they just do a random feature map
• Need a way for early layers to do effective work
• Often not enough labeled data available while lots of unlabeled data
• Can we use unsupervised/semi-supervised approaches to take advantage of the unlabeled data
• Deep networks tend to have more local minima problems than shallow networks during supervised training
Semi-supervised Learning
Semi-supervised Learning
Training Deep Networks
• Build a feature space
• Note that this is what we do with SVM kernels, or trained hidden layers in BP, etc., but now we will build the feature space using deep architectures
• Unsupervised training between layers can decompose the problem into distributed sub-problems (with higher levels of abstraction) to be further decomposed at subsequent layers
Greedy Layer-wise Training
• Train first layer using your data without the labels (unsupervised)
• Since there are no targets at this level, labels don't help. Could also use the more abundant unlabeled data which is not part of the training set (i.e. self-taught learning).
• Freeze the first layer parameters and start training the second layer using the output of the first layer as the unsupervised input to the second layer
• Repeat this for as many layers as desired
• This builds our set of robust features
• Use the outputs of the final layer as inputs to a supervised layer/model and train the last supervised layer(s) (leave early weights frozen)
• Unfreeze all weights and fine tune the full network by training with a supervised approach, given the pre-processed weight settings
Greedy Layer-wise Training
• Greedy layer-wise training avoids many of the problems of trying to train a deep net in a supervised fashion
• Each layer gets full learning focus in its turn since it is the only current "top" layer
• Can take advantage of the unlabeled data
• When you finally tune the entire network with supervised training the network weights have already been adjusted so that you are in a good error basin and just need fine tuning. This helps with problems of
• Ineffective early layer learning
• Deep network local minima
• We will discuss the two most common approaches
• Stacked Auto-Encoders
• Deep Belief Networks
The new way to train multi-layer NNs…
Train this layer first
The new way to train multi-layer NNs…
Train this layer first
then this layer
The new way to train multi-layer NNs…
Train this layer first
then this layer
then this layer
The new way to train multi-layer NNs…
Train this layer first
then this layer
then this layerthen this layer
The new way to train multi-layer NNs…
Train this layer first
then this layer
then this layerthen this layer
finally this layer
The new way to train multi-layer NNs…
EACH of the (non-output) layers is trained
to be an auto-encoderBasically, it is forced to learn good features that describe what comes from the previous layer
an auto-encoder is trained, with an absolutely standard weight-adjustment algorithm to reproduce the input
an auto-encoder is trained, with an absolutely standard weight-adjustment algorithm to reproduce the input
By making this happen with (many) fewer units than the inputs, this forces the ‘hidden layer’ units to become good
feature detectors
One Auto-encoder
73
Stacked Auto-encoders• Stack sparse auto-encoders on top of each
other, drop decode layer each time
74
Stacked auto-encoders
• Do supervised training on last layer • Then do supervised training on whole network
to fine tune the weights
75
Manifold Learning Hypothesis
76
Caveats
• Prevent the layers from just learning the identity (learn Features instead) –Undercomplete - middle layer smaller than input –Sparsity - penalize hidden unit activations
• Use regularization to keep most nodes at or near 0 –Denoising - Stochastically corrupt training instance,
but train encoder to decode uncorrupted instance –Contractive - force encoder to have small
derivatives (stay on manifold)77
Fairness and Learning
• Going to show video of Aylin • Link on course website to her talk (may be
easier for online students if there is feedback).
78