Post on 23-Apr-2020
Advanced Lecture on
Neural Information Processing Systems
(Lecture 03)
Ichiro Takeuchi
Nagoya Institute of Technology
Ichiro Takeuchi, Nagoya Institute of Technology 1/1
Nonlinear modeling
Consider training a model for relationship between elapsedtime after collision (x) and passenger’s head acceleration (y)
Ichiro Takeuchi, Nagoya Institute of Technology 2/1
Nonlinear modeling
Ichiro Takeuchi, Nagoya Institute of Technology 3/1
Linear modeling is not helpful here
Ichiro Takeuchi, Nagoya Institute of Technology 4/1
We want something like this
-150
-100
-50
0
50
100
0 10 20 30 40 50 60
Acc
eler
atio
n[G
]
Time[ms]
Ichiro Takeuchi, Nagoya Institute of Technology 5/1
Which nonlinear model should we use?
Consider a single input case x ∈ R, y ∈ R▶ y = w1 log x
▶ y = w1
√x+ w2 exp(−x2)
▶ y = w1 cos 2πx+ w2 sin 2πx2 + w3
1x
▶ y = log(w1 + w2x)
▶ y = w1+xexp(−w2x2)
▶ y = sin 2π(w1 + w2x) + cos 2π(w3 + w4x)
What’s the difference between the first and the latter 3models?
Ichiro Takeuchi, Nagoya Institute of Technology 6/1
Basis function approach
▶ For single input case, i.e., when x ∈ R, basis functionmodel is written as
y = f(x) = w0 + w1h1(x) + w2h2(x) + . . .+ wqhq(x),
where hk, k = 1, . . . , q is a basis function.
▶ How can we estimate the parameters w0, w1, . . . , wq byleast squares method?
minw0,w∈Rq
n∑i=1
(yi − (w0 +
d∑j=1
wjhj(x))
)2
Ichiro Takeuchi, Nagoya Institute of Technology 7/1
Basis function approach as linear models
▶ Original training set
Xn×1
=
x1
x1...xn
,y =
y1y2...yn
▶ Expanded training set
Xn×1
=
1 h1(x1) h2(x1) · · · hq(x1)1 h1(x2) h2(x2) · · · hq(x2)...
......
. . ....
1 h1(xn) h2(xn) · · · hq(xn)
,y =
y1y2...yn
Ichiro Takeuchi, Nagoya Institute of Technology 8/1
Basis function approach and linear model
▶ Basis function approach
y = f(x) = w0 · 1 + w1h1(x) + w2h2(x) + . . .+ wqhq(x)
▶ Linear regression with multiple inputs
y = f(x) = w0 · 1 + w1x1 + w2x2 + . . .+ wqxq
Ichiro Takeuchi, Nagoya Institute of Technology 9/1
Which basis functions should we use?
▶ Radial basis function
hk(x) = exp
(−(x− ck)
2
2σ2
)
0
0.2
0.4
0.6
0.8
1
0 20 40 60 80 100
Bas
is fu
nctio
n va
lues
hq(
x)
Input x
Ichiro Takeuchi, Nagoya Institute of Technology 10/1
How to determine q, {ck}qk=1, σ2 in RBF
▶ Approach 1▶ q ← n▶ ck ← xi, k = 1, . . . , q▶ s ← cross validation (explained later)
0
0.2
0.4
0.6
0.8
1
0 20 40 60 80 100
Bas
is fu
nctio
n va
lues
hq(
x)
Input x
Ichiro Takeuchi, Nagoya Institute of Technology 11/1
How to determine q, {ck}qk=1, σ2 in RBF
▶ Approach 2▶ q ← cross validation
▶ ck ←(kn
)thquantile of {xi}ni=1
▶ s ← cross validation
0
0.2
0.4
0.6
0.8
1
0 20 40 60 80 100
Bas
is fu
nctio
n va
lues
hq(
x)
Input x
Ichiro Takeuchi, Nagoya Institute of Technology 12/1
RBF Approach for Collision Data
▶ If we select good hyper-parameters (q, {ck}qk=1, s)
-150
-100
-50
0
50
100
0 10 20 30 40 50 60
Acc
eler
atio
n[G
]
Time[ms]
Ichiro Takeuchi, Nagoya Institute of Technology 13/1
Overfitting
▶ If we do not select good hyper-parameters (q, {ck}qk=1, s)
Ichiro Takeuchi, Nagoya Institute of Technology 14/1
Simulation Example for RBF
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
q = 1 q = 10
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
q = 20 q = 50
Ichiro Takeuchi, Nagoya Institute of Technology 15/1
Training Error and True Error
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 2 4 5 10 20 40 50
Err
or
# of basis "q"
Training ErrorTrue Error
Ichiro Takeuchi, Nagoya Institute of Technology 16/1
High dimensional problem
E.g. Gene expression microarray
▶ xij: activity of jth gene for ith patient
▶ yi: Effectiveness of a medicine
yi = f(xi) = w0 + w1xi1 + . . .+ w10000xi,10000
Ichiro Takeuchi, Nagoya Institute of Technology 17/1
How to avoid overfitting: Regularization
minw∈Rd
n∑i=1
(yi −w⊤xi
)subject to
d∑j=1
w2j ≤ s
Ichiro Takeuchi, Nagoya Institute of Technology 18/1
Ridge regression
w∗λ = arg min
w∈Rd
n∑i=1
(yi −w⊤xi)2 + λ
d∑j=1
w2j ,
where λ > 0 is the regularization parameter.
Ichiro Takeuchi, Nagoya Institute of Technology 19/1
Simulation Example for Ridge Regression
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
λ = 0 (q = 50) λ = 1.0 (q = 50)
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
TruthEstimated
λ = 10 (q = 50) λ = 100 (q = 50)
Ichiro Takeuchi, Nagoya Institute of Technology 20/1
Solving Ridge regression
▶ Training data
Xn×d
:=
x11 x12 · · · x1d
x21 x22 · · · x2d...
.... . .
...xn1 xn2 · · · xnd
=
x1
x2...xn
, yn×1
:=
y1y2...yn
▶ Solution
w∗λ = (X⊤X + λI)−1X⊤y
Ichiro Takeuchi, Nagoya Institute of Technology 21/1
Model selection
▶ Example: how to select the regularization parameter λ
▶ Training error cannot be used for model selection becauseit cannot detect over-training (as we will see).
Ichiro Takeuchi, Nagoya Institute of Technology 22/1
Training and validation data
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
Training dataValidation data
•: Training data, •: Validation data
Ichiro Takeuchi, Nagoya Institute of Technology 23/1
Training and validation data
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 2 4 5 10 20 40 50
Err
or
# of basis "q"
Training ErrorTrue Error
Validation Error
▶ Training error monotonically decreases
▶ Validation error can be used as a proxy of the true error
Ichiro Takeuchi, Nagoya Institute of Technology 24/1
Cross-validation
Training data Validation data
R1
R2
R3
R4
R5
▶ The model hyper-parameters (q, λ etc.) are selectedbased on the average validation error.
Ichiro Takeuchi, Nagoya Institute of Technology 25/1
Cross-validation example
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
Training DataValidation Data
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
Training DataValidation Data
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
Training DataValidation Data
Round 1 Round 2 Round 3
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
Training DataValidation Data
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-1 -0.5 0 0.5 1
Out
put y
Input x
Training DataValidation Data
Round 4 Round 5
Ichiro Takeuchi, Nagoya Institute of Technology 26/1
Leave-one-out cross-validation (LOOCV)
R1
R2
R n
R n-1
Training data Validation data
Ichiro Takeuchi, Nagoya Institute of Technology 27/1
Final exercise IGiven the data {(xi, yi)}ni=1, consider a constant model thatdoes not use the input x (not useful in practice)
f(x) = w0,
The parameter w0 is estimated by solving the followingminimization problem:
arg minw0∈R
n∑i=1
(yi − f(xi))2 = arg min
w0∈R
n∑i=1
(yi − w0)2
▶ First, show that the solution of the optimal solution ofthe above problem is the sample mean, i.e.,
arg minw0∈R
n∑i=1
(yi − w0)2 =
1
n
n∑i=1
yi
Ichiro Takeuchi, Nagoya Institute of Technology 28/1
Final exercise II
▶ Next, confirm that the training error and the LOOCVerror of the constant model are respectively written as
TrainEr :=n∑
i=1
(yi − arg min
w0∈R
n∑j=1
(yj − w0)2
)2
=n∑
i=1
(yi − y)2,
LoocvEr :=n∑
i=1
(yi − arg min
w0∈R
∑j =i
(yj − w0)2
)2
=n∑
i=1
(yi −
1
n− 1
∑j =i
yj
)2
.
Ichiro Takeuchi, Nagoya Institute of Technology 29/1
Final exercise III
▶ Finally, show that the relation of these two errors arewritten as
LoocvEr :=
(n
n− 1
)2
TrainEr.
Ichiro Takeuchi, Nagoya Institute of Technology 30/1