Post on 14-Oct-2020
Deep Learning Programming course!Lecture 1: Welcome toDeep Learning Programming
Dr. Hanhe LinUniversität Konstanz, 16.04.2018
1 Introduction
About me
- My name is Hanhe Lin. I got my PhD degree in University of Otago, New Zealand.- From October 2016, I am a postdoc researcher in workgroup of multimedia signal
processing (MMSP). Bachelor and Master are welcome!- Contact me:
- Office: Z704- Email: hanhe.lin@uni-konstanz.de- Homepage: LINK
2 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
About you
- What is your study background?- Have you learned some statistics/machine learning/computer vision courses before?- Any programming experience (C/C++, Java, R, Python...)?- Any special interest in Deep Learning?- . . .
3 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Course schedule
- One lecture and one exercise per week- Lecture
- Lecturer: Dr. Hanhe Lin- Time: Monday 13:30 – 15:00- Place: Z613
- Exercise- Tutor: Oliver Wiedemann- Assignment (programming exclusively) will be released on every Monday, you
must submit your result by Sunday- Time: Wednesday 13:30 – 15:00- Place: Z613
- I will attend an conference between 28/05 and 01/06, we should discuss an alternativedate for the missing lecture and exercise.
4 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Course content
- Lecture session:- Softmax regression- Support Vector Machine- Deep neural network- Convolutional neural network- Recurrent neural network- Hypterparameter tuning, regularization, optimization- Transfer learning- . . .
- Exercise session:- You will implement the learned model
5 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Course works and credits
- Credit will be given based on solving assignments and completing a final project. Youare required to use your laptop/desktop to solve assignments in exercise session.
- In the last few weeks, you are required to finish a project with a given dataset for aspecific target.
- You will be given training set and validation set to build and train a deep learningmodel
- You credits will be ranked by applied your model to a test set (not available toyou)
6 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Project plan
Phase 1: find your group members, up to 4 members in each groupPhase 2: framework design and implementation, present your workPhase 3: submit your train model and write technical report, one report per group, up to
4 pages by a given template
7 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Credits
- Exercise: 20%- Presentation: 20%- Technical report: 20%- Evaluation: 40%
8 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Presentation
- It will be arranged in exercise session in the last few weeks- Each group has 35 minutes to present their work and 10 minutes for Q&A
9 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Learning objective
You will:- have an insight of the fundamentals of Deep Learning- have ability to design your own deep learning algorithm to solve some specific
problems- know how to adopt the existing model to your work via transfer learning and fine-tuning
10 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Note
On Wednesday you will install Jupyter Notebook and Tensorflow, please bring your laptop.
11 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
1 Introduction
Any question?
12 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Introduction
Deep learning- a data-driven approach to Artificial Intelligence (AI)- a specific kind of machine learning- an interdisciplinary research, e.g., statistics, biology, ...- new capability for computers- ubiquitous
Examples: face detection and recognition, image analysis, self-driving car, . . .
13 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Machine Learning definition
- Field of study that gives computers the ability to learn without being explicitlyprogrammed (Arthur Samuel, 1959)
- Well-posed Learning Problem: A computer program is said to learn from experience Ewith respect to some task T and some performance measure P, if its performance onT, as measured by P, improves with experience E (Tom Mitchell, 1998)
14 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Question
Suppose your email program watches which emails you do or do not mark as spam, andbased on that learns how to better filter spam. What is its experience E, task T, andperformance P?
- Classifying emails as spam or not spam- Watching you label emails as spam or not spam- The number of emails correctly classified as spam/not spam
15 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Question
Suppose your email program watches which emails you do or do not mark as spam, andbased on that learns how to better filter spam. What is its experience E, task T, andperformance P?
- Classifying emails as spam or not spam T- Watching you label emails as spam or not spam E- The number of emails correctly classified as spam/not spam P
16 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Supervised and unsupervised learning
Supervised learning- Right answers, or ground-truth, are given- Two kinds in terms of outputs:
- Regression: predict continuous valued output- Classification: predict discrete valued output
Unsupervised learning- We cannot give right answer to all data as the data increase exponentially with the
development of technology- Unsupervised learning is proposed to handle the unlabeled data
17 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Question
You’re running a company, and you want to develop learning algorithms to address eachof two problems:
- Problem 1: you have a large inventory of identical items. You want to predict howmany of these items will sell over the next 3 months
- Problem 2: you’d like software to examine individual customer accounts, and for eachaccount decide if it has been hacked/compromised
18 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Question
You’re running a company, and you want to develop learning algorithms to address eachof two problems:
- Problem 1: you have a large inventory of identical items. You want to predict howmany of these items will sell over the next 3 months Regression
- Problem 2: you’d like software to examine individual customer accounts, and for eachaccount decide if it has been hacked/compromised Classification
19 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Question
Which would you address using an unsupervised learning algorithm?
- Given email labeled as spam/not spam, learn a spam filter- Given a set of news articles found on the web, group them into set of articles about the
same story- Given a database of customer data, automatically discover market segments and
group customers into different market segments- Given a dataset of patients diagnosed as either having diabetes or not, learn to
classify new patients as having diabetes or not
20 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Question
Which would you address using an unsupervised learning algorithm?
- Given email labeled as spam/not spam, learn a spam filter- Given a set of news articles found on the web, group them into set of articles about the
same story ✓- Given a database of customer data, automatically discover market segments and
group customers into different market segments ✓- Given a dataset of patients diagnosed as either having diabetes or not, learn to
classify new patients as having diabetes or not
21 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Why is deep learning taking off?
Neural network has a very long history in machine learning, it has attracted moreattentions recently due to:
- Increasing dataset size- Increasing computational power- Increasing performance
22 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Increasing dataset size
23 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Increasing computational power
24 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
2 What is Deep Learning
Increasing performance
25 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Binary classification
- Examples- Email: spam/not spam?- Disease: cancer/not cancer- Object: car/not car?
- y ∈ {0, 1}, where 1 is positive class, and 0 is negative class, e.g., spam – 1 vs notspam – 0
- Intuitively, negative class conveys absence of something
26 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Notations
- m: number of training example- x : “input” variables/features- y : “output” variables/features- (x (i), y (i)): i th training example
27 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Hypothesis
- if hθ(x) ≥ 0.5, predict y = 1- if hθ(x) < 0.5, predict y = 0- 1 ≥ hθ(x) ≥ 0
28 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Logistic regression model
- sigmoid/logistic function: σ(z) = 11+e−z
- hθ(x) = σ(θTx) = 1
1+e−θT x
, where θ = [θ0 θ1 . . . θn]
29 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Interpretation of hypothesis output
- Estimated probability that y = 1, given x , “parameterized” by θ:
hθ(x) = P (y = 1|x ; θ)
- Example: given tumor size, if we have hθ(x) = 0.7, which means: tell patient that 70%chance of tumor being malignant
- Since we only have two classes, we have:
P (y = 0|x ; θ) + P (y = 1|x ; θ) = 1⇒P (y = 0|x ; θ) = 1− P (y = 1|x ; θ)
30 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Logistic regression
hθ(x) = σ(θTx) =
1
1 + e−θT x
- When hθ(x) ≥ 0.5⇒ θTx ≥ 0, predict y = 1- When hθ(x) < 0.5⇒ θTx < 0, predict y = 0
31 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Loss function
- Loss function, cost function, error function are interchangeable.- Idea: choose θ so that hθ(x) is close to y for training examples (x, y)- The representation of loss function:
J(θ) =1
m
m∑i=1
Loss(hθ(x(i)), y (i))
32 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Loss function of logistic regression
-
Loss(hθ(x(i)), y (i)) =
{−log(hθ(x)) if y = 1−log(1− hθ(x)) if y = 0
- Intuition: if hθ(x) = 0, but y = 1, the learningalgorithm will be penalized by a very large cost
- We can compact two cases into one equation:
Loss(hθ(x(i)), y (i)) = −y log(hθ(x))−(1−y)log(1−hθ(x))
33 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Logistic regression summary
- Hypothesis: hθ(x) = 1
1+e−θT x
- Parameter: θ- Loss function: Loss(hθ(x (i)), y (i)) = −y log(hθ(x))− (1− y)log(1− hθ(x))- Goal: minθJ(θ), but how?
34 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Gradient descent
- Given loss function Loss(hθ(x (i)), y (i)) = −y log(hθ(x))− (1− y)log(1− hθ(x)), ourobjective is minθJ(θ).
- Repeat until converge {
θj := θj − α∂
∂θjJ(θ) j = 0, 1, 2, . . . , n
⇒θj := θj − α1
m
m∑i=1
(hθ(x(i))− y (i))x (i)j
}
35 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Advanced optimization
- Other advanced algorithms:- Conjugate gradient- Broyden–Fletcher–Goldfarb–Shanno (BFGS)- . . .
- Advantages:- No need to pick learning rate manually- Often faster than gradient descent
- Disadvantages:- More complex to implement
- We will discuss some approaches in our later course
36 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin
3 Logistic regression
Now you have learned your firstmachine learning model!
Questions?
37 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin