Concept learning, Regression Adapted from slides from Alpaydin’s book and slides by Professor...
-
Upload
ira-griffin -
Category
Documents
-
view
213 -
download
0
Transcript of Concept learning, Regression Adapted from slides from Alpaydin’s book and slides by Professor...
Concept learning, Regression
Adapted from slides from Alpaydin’s book and slides by Professor Doina Precup, Mcgill University
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)2
S, G, and the Version Space
most specific hypothesis, S
most general hypothesis, G
h H, between S and G isconsistent
and make up the version space
(Mitchell, 1997)
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)3
VC Dimension
N points can be labeled in 2N ways as +/– H shatters N if there
exists h H consistent for any of these: VC(H ) = N
An axis-aligned rectangle shatters 4 points only !
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)4
How many training examples N should we have, such that with probability at least 1 ‒ δ, h has error at most ε ?(Blumer et al., 1989)
Each strip is at most ε/4 Pr that we miss a strip 1‒ ε/4 Pr that N instances miss a strip (1 ‒ ε/4)N
Pr that N instances miss 4 strips 4(1 ‒ ε/4)N
4(1 ‒ ε/4)N ≤ δ and (1 ‒ x)≤exp( ‒ x) 4exp(‒ εN/4) ≤ δ and N ≥ (4/ε)log(4/δ)
Probably Approximately Correct (PAC) Learning
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)5
Use the simpler one because Simpler to use
(lower computational complexity)
Easier to train (lower space complexity)
Easier to explain (more interpretable)
Generalizes better (lower variance - Occam’s razor)
Noise and Model Complexity
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)6
Multiple Classes, Ci i=1,...,KNt
tt,r 1}{ xX
, if 0
if 1
ijr
jt
it
ti C
C
x
x
Train hypotheses hi(x), i =1,...,K:
, if 0
if 1
ijh
jt
it
ti C
C
x
xx
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)7
Regression
01 wxwxg
012
2 wxwxwxg
N
t
tt xgrN
gE1
21| X
N
t
tt wxwrN
w,wE1
2
0101
1|X
tt
t
N
ttt
xfr
r
rx 1,X
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)8
Model Selection & Generalization Learning is an ill-posed problem; data is not
sufficient to find a unique solution The need for inductive bias, assumptions about H Generalization: How well a model performs on
new data Overfitting: H more complex than C or f Underfitting: H less complex than C or f
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)9
Triple Trade-Off
There is a trade-off between three factors (Dietterich, 2003):
1. Complexity of H, c (H),2. Training set size, N, 3. Generalization error, E, on new data
As NE As c (H)first Eand then E
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)10
Cross-Validation
To estimate generalization error, we need data unseen during training. We split the data as Training set (50%) Validation set (25%) Test (publication) set (25%)
Resampling when there is few data
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)11
Dimensions of a Supervised Learner1. Model :
2. Loss function:
3. Optimization procedure:
|xg
t
tt g,rLE || xX
X|min arg
E*
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)12
Steps to solving a supervised learning problem1. Select the input-output pairs2. Decide how to encode the inputs and outputs –
This defines the instance space X, and the out put space Y.
3. Choose a class of hypotheses / representations: H4. Choose an error function to define the best
hypothesis5. Choose an algorithm for searching efficiently
through the space of hypotheses.
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)13
Example: What hypothesis class should we pick?
x y.86 2.49
.09 .83
-.85 -.25
.87 3.1
-.44 .87
-.43 .02
-1.1 -.12
.4 1.81
-.96 -.83
.17 .43
x
y y
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)14
Linear Hypothesis
Suppose y was a linear function of x: hw(x) = w0 + w1x (+ … )
wi are called parameters or weights.
To simplify notation we add an attribute x0 = 1to the other n attributes (also called the bias term).
where w and x are vectors of size n+1 How should we pick w ?
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)15
Error Minimization
We should make the predictions of hw close the true value y on the data we have.
We define an error function or a cost function. We will pick w such that the error function is
minimized.
How should we choose the error function?
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)16
Least Mean Squares (LMS)
Try to make hw(x) close to y on the examples in the training set. We define a sum-of-squares error function
We will choose w such as to minimize J(w) Compute w such that:
2
12
1)(
m
iii yxhJ ww
njwJw j
0,0