.ppt

15
Introduction to Machine Learning course 67577 fall 2007 Lecturer: Amnon Shashua Teaching Assistant: Yevgeny Seldin School of Computer Science and Engineering Hebrew University
  • date post

    13-Sep-2014
  • Category

    Documents

  • view

    401
  • download

    0

description

 

Transcript of .ppt

Page 1: .ppt

Introduction to Machine Learning

course 67577 fall 2007

Lecturer: Amnon ShashuaTeaching Assistant: Yevgeny Seldin

School of Computer Science and EngineeringHebrew University

Page 2: .ppt

What is Machine Learning?• Inference engine (computer program) that when given

sufficient data (examples) computes a function that matches as close as possible the process generating the data.

• Make accurate prediction based on observed data

• Algorithms to optimize a performance criterion based on observed data

• Learning to do better in the future based on what was experienced in the past

• Programming by examples: instead of writing a program to solve a task directly, machine learning seeks methods by which the computer will come up with its own program based on training examples.

Page 3: .ppt

Why Machine Learning?• Data-driven algorithms are able examine large amounts

of data. A human expert on the other hand is likely to be guided by subjective impressions or by examining a relatively small number of examples.

• Humans often have trouble expressing what they know but have no difficulty in labeling data

• Machine learning is effective in domains where declarative (rule based) knowledge is difficult to obtain yet generating training data is easy

Page 4: .ppt

Typical Examples• Visual recognition (say, detect faces in an image): the amount of

variability in appearance introduce challenges that are beyond the capacity of direct programming

• Spam filtering: data-driven programming can adapt to changing tactics by spammers

• Extract topics from documents: categorize news articles whether they are about politics, sports, science, etc.

• Natural language understanding: from spoken words to text; categorize the meaning of spoken sentences

• Optical character recognition (OCR)

• Medical diagnosis: from symptoms to diagnosis

• Credit card transaction fraud detection

• Wealth prediction

Page 5: .ppt

Fundamental Issues• Over-fitting: doing well on a training set does not guarantee

accuracy on new examples

• What is the resource we wish to optimize? For a given accuracy, use the smallest size training set

• Examples are drawn from some (fixed) distribution D over X x Y (instance space x output space). Does the learner actually need to recover D during the learning process?

• How does the learning process depend on the complexity of the family of learning functions (concept class C)? How does one define complexity of C?

• When the goal is to learn the joint distribution D then the problem is computationally unwieldy because the joint distribution table is exponentially large. What assumptions can be made to simplify the task?

Page 6: .ppt

Multiclass classification. K=2 is normally of most interest.

Supervised vs. Un-supervised

YXf :

kY ,...,1

Supervised Learning Models:

where X is the instance (data) space and Y is the output space

RY Regression. Predict the price of a used car given brand, year, mileage..Kinematics of a robot arm; navigate by determining steering angle from image input..

Un-supervised Learning Models:Find regularities in the input data assuming there is some structure in the input space

• Density estimation• Clustering (non-parametric density estimation): divide customers to groups which have similar attributes..• Latent class models: extract topics from documents• Compression: represent the input space with fewer parameters; projection to lower-dimensional spaces

Page 7: .ppt

Notations

),(),...,,( 11 mm yxyxZ

*,,}1,0{ XRXX nn

mxxS ,...,1

},,{,},1,1{ YRYY

Xx""),2.7,0,3.2,5.0(),0,0,1,1,1,0( textxxx

X is the instance space: space from which observations are drawn. Examples,

input instance, a single observation. Examples,

Y is the output space: set of possible outcomes that can be associated with a measurement. Examples,

An example is an instance-label pair (x,y). If |Y|=2 one typically uses {0,1} or {-1,1}.We say that an example (x,y) is positive if y=1 and otherwise we call it a negative exampleA training set Z consists of m instance-label pairs:In some cases we refer to the training set without labels:

Page 8: .ppt

Separating hyperplanes: a concept h(x) is specified by a vector and a scalar b such that:

Conjunction learning: a conjunction is a special case of a Boolean formula. A literalIs a variable or its negation and a term is a conjunction of literals, i.e.A target function is a term which consists of a subset of literals.In this case and

Each is called a concept or hypothesis or classifier. Example, if

Notations

||C

)( 321 xxx

YXhhC :|

Ch

},{,}1,0{ falsetrueYX n

}1,0{,}1,0{ YX n

ixxhihC )(:|

A concept (hypothesis) class C is a set (not necessarily finite) of functions of the form:

Other examples:

then C might be:

Decision trees: when then any boolean function can be described by a binary tree. Thus, C consists of decision trees ( )

nRX

otherwise

bxwxh

T

11

)(

nX 2 nC 3

nRw

Page 9: .ppt

The Formal Learning ModelProbably Approximate Correct (PAC)

• Distribution invariant: Learner does not need to estimate the joint distribution D over X x Y. Assumptions are that examples arrive i.i.d. and that D exists and is fixed.

• The training sample complexity (size of the training set Z) depends only the desired accuracy and confidence parameters - does not depend on D.

• Not all concept classes D are PAC-learnable. But some interesting classes are.

Page 10: .ppt

Unrealizable case: when and the training set isand D is over XxY

Realizable case: when a target concept is known to lie inside C.In this case, the training set is

sampled randomly and independently (i.i.d) according to some (unknown) Distribution D, i.e., S is distributed according to the product distribution

},...,{ 1 mxxS

Cxct )(

Cct

miiti xcxZ 1)(,

miii yxZ 1,

Ch )()(:)( xhxcxprobherr tD

dxxDxhxcindXx

t )())()((

falsextruex

xind01

)(

mDDD ...

Given a concept function

)(herr is the probability that an instance x sampled according to D will belabeled incorrectly by h(x)

PAC Model Definitions

Page 11: .ppt

given to the learner specifies desired accuracy, i.e.

Note: in realizable case because

)(min)( herrCoptCh

0)( Copt0)( tcerr

0 )()( Coptherr

01 given to the learner specifies desired confidence, i.e.

1])()([ Coptherrprob

The learner is allowed to deviate occasionally from the desired accuracybut only rarely so..

PAC Model Definitions

Page 12: .ppt

We will say that an algorithm L learns C if for every and for everyD over XxY, L generates a concept function such that theprobability that is at least

tc tc

tc tc

)()( Coptherr 1Ch

tc

PAC Model Definitions

Page 13: .ppt

from the set of all training examples to C with the following property:given any there is an integer such that if then, for any probability distribution D on XxY, if Z is atraining set of length m drawn randomly according to , thenwith probability of at least then hypothesis is such that

tc tc

tc tc

)()( Coptherr1

mD 1,0,

Formal Definition of PAC Learning

11),(:

m

miii CyxL

),(0 m0mm

CZLh )(

A learning algorithm L is a function:

We say that C is learnable (or PAC-learnable) if there is a learning algorithm for C

Page 14: .ppt

tc tc

tc tc

Formal Definition of PAC Learning

),(0 m does not depend on D, i.e., PAC model is distribution invariant

),(0 mThe class C determines the sample complexity. For “simple” classes would be small compared to more “complex” classes.

Notes:

Page 15: .ppt

tc tc

tc tc

Course Syllabus

C

m ln1C

C 1ln)(1ln1 Cvcdm

3 x PAC:

2 x Separating Hyperplanes:

Support Vector Machine, Kernels, LinearDiscriminant Analysis

3 x Unsupervised Learning:Dimensionality Reduction (PCA), Density Estimation,Non-parametric Clustering (spectral methods)

5 x Statistical Inference:Maximum Likelihood, Conditional Independence,Latent Class Models, Expectation-Maximization Algorithm,Graphical Models