The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a...
Transcript of The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a...
![Page 1: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/1.jpg)
The Mathematical Foundations of Artificial Intelligence
Deep Conversations on Deep LearningA technical series hosted by IEEE Maine Section
W.D. Rawle, PhDSenior Member IEEEChair, IEEE Maine SectionSeptember 16, 2020
![Page 2: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/2.jpg)
Artificial Intelligence
2… Deep Conversations on Deep Learning Series
In computer science, artificial intelligence (AI), sometimes called machine intelligence, is intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans. Leading AI textbooks define the field as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals. Colloquially, the term "artificial intelligence" is often used to describe machines (or computers) that mimic "cognitive" functions that humans associate with the human mind, such as "learning" and "problem solving".
..
Russell, Stuart J.; Norvig, Peter (2009). Artificial Intelligence: A Modern Approach (3rd ed.). Upper Saddle River, New Jersey: Prentice Hall.
ISBN 978-0-13-604259-4
![Page 3: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/3.jpg)
3… Deep Conversations on Deep Learning Series
The Mathematical Foundationsof Artificial Intelligence
FUNCTIONAL ANALYSIS
NUMERICAL METHODS
PROBABILITYTHEORY
![Page 4: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/4.jpg)
4… Deep Conversations on Deep Learning Series
The Mathematical Foundations Of Artificial Intelligence
Functional Analysis: Establishes to domain of the model or “hypothesis” Defines operations within the domain
and transformations into adjacent domains Provides for measures of completeness:
orthonormal function sets, vector projection Simplifies to more tractable implementations:
linear algebra, matrix arithmetic, Fourier series.
![Page 5: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/5.jpg)
5… Deep Conversations on Deep Learning Series
The Mathematical Foundations Of Artificial Intelligence
Numerical Methods Solutions to multivariate classification problems
often require optimization routines: 1. Establishment of cost and gradient functions2. Numerical search strategies3. Linearization/determinism of stochastic process4. Application of heuristics and ontologies5. Numerical integration and differentiation required
for ill defined data or “complicated” regions
![Page 6: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/6.jpg)
6… Deep Conversations on Deep Learning Series
The Mathematical Foundations Of Artificial IntelligenceProbability Theory
Establishes performance bounds upon stochasticclassifiers: Bayesian networks, Particle Filters, Markov Chains, Maximum Likelihood, Parameter Estimation, Statistical Analysis of Physical Parameters
Accommodates stochastic processes and multi-variate data - employing measures such as Mahalanobis Distance and Mahalanobis-Bregman divergence
![Page 7: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/7.jpg)
7… Deep Conversations on Deep Learning Series
CNNs and Functional AnalysisCONVOLUTIONAL
NEURAL NETWORK
CL
AS
SIF
ICA
TIO
N
IMA
GE
IN
PU
TO
NE
FR
AM
E P
ER
CO
LO
R
SIZE/STRIDE
MAX POOL
Fooled by a little distortionMIT 6.S094 Lex Friedman Deep LearningLecture 1
CNN Approximations• Size/stride- convolution approx.• Max Pool – data loss• Training – insufficient samples
![Page 8: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/8.jpg)
8… Deep Conversations on Deep Learning Series
CNNs and Functional AnalysisQuestions??
• Is the image sufficiently sampled to capture “high frequency” effects- Nyquist criteria
• Does the discretization of the convolution function compromise the output
• How much data is lost when using max pool compression
• Is fidelity of training data sufficient• Would alternate approaches (DCT, for
example) provide sufficient compression and maintain fidelity
• What would be the difference in compute resource requirements
![Page 9: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/9.jpg)
9… Deep Conversations on Deep Learning Series
NNs and Numerical Methods
Linear Regression
hypothesis
Cost function
Numerical “gradient”
![Page 10: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/10.jpg)
10… Deep Conversations on Deep Learning Series
NNs and Numerical MethodsLogistic Regression
Activated hypothesis
Non linear activation
Cost function
Parameter Gradient
![Page 11: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/11.jpg)
11… Deep Conversations on Deep Learning Series
NNs and Numerical Methods
![Page 12: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/12.jpg)
12… Deep Conversations on Deep Learning Series
NNs and Numerical Methods
Forward Propagation
Back Propagation, minimize wrt θ, gradient derivatives
Given
![Page 13: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/13.jpg)
13… Deep Conversations on Deep Learning Series
NNs and Numerical MethodsSearching for the minima- Classic optimization theory• Conjugate gradient• Simplex• Direct search• Stochastic Gradient
Challenges• Well behaved – global minima• Oscillatory behavior• Regularization• Convergence Rate
![Page 14: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/14.jpg)
14
BBNs and Probability Theory
A simple example from Wikipedia
• Acyclic graph
Markov Model
14… Deep Conversations on Deep Learning Series
![Page 15: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/15.jpg)
15
BBNs and Probability Theory
Given
15… Deep Conversations on Deep Learning Series
![Page 16: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/16.jpg)
16
BBNs and Probability TheoryNaïve Bayes Probability Condition
Constructing a classifier from the probability model
16… Deep Conversations on Deep Learning Series
![Page 17: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/17.jpg)
17
Vapnik’s Learning Model
17… Deep Conversations on Deep Learning Series
1. A generator of random vectors � ∈ ��, drawn independently from a fixed but unknown probability distribution function � � .
2. A supervisor who returns an output value � to every input vector � according to a conditional distribution function � � �also fixed but unknown
3. A learning machine capable of implementing a set of functions � �, α , α ∈ Λ, where Λ is a set of parameters.
G S
LM
x y
LM observes pairs (x,y) (training set).Subsequently, the machine outputs ẏfor any given x.
ẏ
![Page 18: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/18.jpg)
18
The Nature of Statistical Learning
18… Deep Conversations on Deep Learning Series
• What are (necessary and sufficient) conditions for consistency of a learning process based on the ERM principle?
• How fast is the rate of convergence of the learning process?
• How can one control the rate of convergence (the generalization ability) of the learning process
• How can one construct algorithms that can control the generalization ability?
Empirical Risk Minimization Inductive Principle
![Page 19: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/19.jpg)
1919…
SVM Kernels
• homogeneous polynomial,• inhomogeneous polynomial, • Gaussian radical basis,• hyperbolic tangent
19… Deep Conversations on Deep Learning Series
The Nature of Statistical Learning
Support Vector Machines
![Page 20: The Mathematical Foundations of Artificial Intelligence...2020/09/16 · MAX POOL Fooled by a little distortion MIT 6.S094 Lex Friedman Deep Learning Lecture 1 CNN Approximations](https://reader036.fdocuments.us/reader036/viewer/2022081410/60955f228c6cb1452d7451ee/html5/thumbnails/20.jpg)
Thank you
20
W.D Rawle PhDSenior Member, IEEEChair, IEEE Maine [email protected]
Many thanks to Dr. Ali Abedi, NE Area Chair, IEEE Region 1 for providing Webex Resources