An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine...

38
Introduction Machine learning The machine learning process An Introduction to Machine Learning Fabio Gonz´ alez, PhD MindLAB Research Group - Universidad Nacional de Colombia Introducci´ on a los Sistemas Inteligentes Fabio Gonz´ alez, PhD An Introduction to Machine Learning

Transcript of An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine...

Page 1: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

An Introduction to Machine Learning

Fabio Gonzalez, PhD

MindLAB Research Group - Universidad Nacional de Colombia

Introduccion a los Sistemas Inteligentes

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 2: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Outline

1 Introduction

2 Machine learningWhat’s machine learningHistorySupervised learningNon-supervised learning

3 The machine learning processModel learningModel evaluationFeature extractionModel application

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 3: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Observation and analysis

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 4: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Tycho Brahe

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 5: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Tycho Brahe

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 6: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Johannes Kepler

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 7: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Data and models

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 8: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Machine Learning

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 9: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Machine Learning with Images

Learning/ Model

Induction

Data Model

outperform the majority of state-of-the-art approaches in mitosisdetection.

The organization of the rest of this paper is as follows.In Sec. 2, we describe motivations of the proposal. In Sec. 3,we describe details of the new methodology. In Sec. 4, wepresent experimental results. Finally, in Sec. 5, we present ourconcluding remarks.

2 Motivation and Previous WorkRecently, the development of computerized systems for auto-mated mitosis detection has become an active area of researchwith the goal of developing decision support systems to be ableto relieve the workload of the pathologist. In a contest held inconjunction with the ICPR 2012 conference5,6 to identify the bestautomated mitosis detection algorithm, a variety of approachescompeted against each other. These approaches can be catego-rized as handcrafted feature based or feature learning based. Thecommonly used handcrafted features include various morpho-logical, shape, statistical, and textural features that attempt tomodel the appearance of the domain and, in particular, theappearance of the mitoses within the digitized images.7–10

Although domain inspired approaches (hand crafted) are use-ful in that they allow for explicit modeling of the kinds of fea-tures that pathologists look for when identifying mitoses, thereis another category of feature generation inspired by convolu-tional neural networks (CNN),11,12 CNN are multilayer neuralnetworks that learns a bank of convolutional filters at eachlayer.13,14 In contrast to handcrafted features, CNN is fully data-driven, therefore, it is more accurate in representing training

samples and is able to find feature patterns that handcraftedfeatures fail to describe. However, CNN is computationallydemanding and sensitive to the scalability of the training data.The winner14 of the ICPR contest used two 11 layers to achievean F-measure of 0.78. However, this approach is not feasible forclinical use since each layer of the CNN model comprised hun-dreds of neurons and required a large amount of time (severalweeks) for both training and testing.

Other methods achieved an F-measure of up to 0.71, basedprimarily on combining various handcrafted features. Althoughhandcrafted feature approaches are faster, drawbacks include(1) the fact that the identification of salient features is highlydependent on the evaluation dataset used and (2) the lack ofa principled approach for combining disparate features. Hence,it stands to reason that a combination of CNN and handcraftedfeatures will allow us to exploit the high accuracy of CNN whilealso reducing the computational burden (in terms of time) oftraining deep CNN models. By employing a light CNN model,the proposed approach is far less demanding computationally,and the cascaded strategy of combining handcrafted featuresand CNN-derived features enables the possibility of maximizingperformance by leveraging the disconnected feature sets.Previous work in this approach includes the Nippon ElectricCompany (NEC) team,13 where an attempt was made to stackthe CNN-learned features and handcrafted features yielded anF-measure of 0.659, suggesting that more intelligent combina-tions of CNN and handcraft features are required.

In this paper, we present a cascaded approach to combiningCNN and handcrafted features for mitosis detection. The work-flow of the new approach is depicted in Fig. 2. The first step is to

(a) (b) (c) (d) (e) (f)

Fig. 1 An illustration of the visual similarity between true mitotic processes and confounding false pos-itives. (a)–(c) True mitoses. (d)–(f) Confounding nonmitotic figures.

Fig. 2 Workflow of our methodology. Blue-ratio thresholding15 is first applied to segment mitosis can-didates. On each segmented blob, handcrafted features are extracted and classified via a random forestsclassifier. Meanwhile, on each segmented 80 × 80 patch, convolutional neural networks (CNN)11 aretrained with a fully connected regression model as part of the classification layer. For those candidatesthat are difficult to classify (ambiguous result from the CNN), we train a second-stage random forestsclassifier on the basis of combining CNN-derived and handcrafted features. Final decision is obtained viaa consensus of the predictions of the three classifiers.

Journal of Medical Imaging 034003-2 Oct–Dec 2014 • Vol. 1(3)

Wang et al.: Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features

Downloaded From: http://medicalimaging.spiedigitallibrary.org/ on 10/12/2014 Terms of Use: http://spiedl.org/terms

Prediction

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 10: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

The fourth paradigm

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 11: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

What’s machine learningHistorySupervised learningNon-supervised learning

Machine Learning

Construction and study of systems that can learn from data

Main problem: to find patterns, relationships, regularitiesamong data, which allow to build descriptive and predictivemodels.

Related fields:

StatisticsPattern recognition and computer visionData mining and knowledge discoveryData analytics

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 12: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

What’s machine learningHistorySupervised learningNon-supervised learning

Brief history

Fisher’s linear discriminant (Fisher, 1936)

Artificial neuron model (MCCulloch and Pitts, 1943)

Perceptron (Rosenblatt, 1957) (Minsky&Papert, 1969)

Probably approximately correct learning (Valiant, 1984)

Multilayer perceptron and back propagation (Rumelhart et al.,1986)

Decision trees (Quinlan, 1987)

Bayesian networks (Pearl, 1988)

Support vector machines (Cortes&Vapnik, 1995)

Efficient MLP learning, deep learning (Hinton et al., 2007)

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 13: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

What’s machine learningHistorySupervised learningNon-supervised learning

Machine Learning in the news

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 14: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

What’s machine learningHistorySupervised learningNon-supervised learning

Supervised learning

Fundamental problem:to find a function thatrelates a set of inputswith a set of outputs

Typical problems:

ClassificationRegression

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 15: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

What’s machine learningHistorySupervised learningNon-supervised learning

Supervised learning

Fundamental problem:to find a function thatrelates a set of inputswith a set of outputs

Typical problems:

ClassificationRegression

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 16: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

What’s machine learningHistorySupervised learningNon-supervised learning

Non-supervised learning

There are not labels for thetraining samples

Fundamental problem: to findthe subjacent structure of atraining data set

Typical problems: clustering,segmentation, dimensionalityreduction, latent topic analysis

Some samples may have labels,in that case it is calledsemi-supervised learning

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 17: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

What’s machine learningHistorySupervised learningNon-supervised learning

Non-supervised learning

There are not labels for thetraining samples

Fundamental problem: to findthe subjacent structure of atraining data set

Typical problems: clustering,segmentation, dimensionalityreduction, latent topic analysis

Some samples may have labels,in that case it is calledsemi-supervised learning

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 18: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

The machine Learning process

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 19: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Model learning

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 20: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Model induction from data

Learning is an ill-posed problem (more than one possiblesolution for the same particular problem, solutions aresensitive to small changes on the problem)

It is necessary to make additional assumptions about the kindof pattern that we want to learn

Hypothesis space: set of valid patterns that can be learnt bythe learning algorithm

Occam’s razor: ”All things being equal, the simplest solutiontends to be the best one.”

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 21: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Approaches to learning

Probabilistic:

Generative models: model P(Y ,X )Discriminative models: model P(Y |X )

Geometrical:

Manifold learning: model the geometry of the space where thedata livesMax margin learning: model the separation between the classes

Optimization:

Energy/loss/risk minimization

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 22: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Learning as optimization

General optimization problem:

minf ∈H

L(f ,D),

with H:hypothesis space, D:training data, L:loss/error

Example, logistic regression:

Hypothesis space:

y(x) = P(C+|x) = σ(wT x)

Cross-entropy error:

E (w) = − ln p(t|w) = −∑n=1

[tn ln yn + (1− tn) ln(1− yn)]

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 23: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Methods

Supervised generative:

Naıve BayesGraphical modelsMarkov random fieldsHidden markov models

Supervised discriminative:

Logistic regressionRidge regressionConditional random fields

Supervised geometrical

Max margin classification(SVM)k-nearest neighbors

Non-supervised generative:

Latent semantic analysisLatent Dirichlet allocationGaussian mixtures

non-supervised geometrical:

k-meansPCAManifold learning

Other

Neural networks (deeplearning)Decision tressAssociation rules

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 24: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Methods

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 25: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Strategies

Optimization (non-linear, convex, etc)

Stochastic gradient descent

Kernel methods

Maximum likelihood estimation

Maximum a posteriori estimation

Bayesian estimation (variational learning, Gaussian processes)

Expectation maximization

Maximum entropy models

Sampling (Markov Chain Monte Carlo, particle filtering)

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 26: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Evaluation

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 27: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Training error vsgeneralization error

Training error: ∑i=1

L(fw , Si )

Generalization error:E [(L(fw ,S)]

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 28: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Cross validation

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 29: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Overfitting and underfitting

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 30: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Regularization

Controls the complexity of a learned model

Usually, the regularization term corresponds to a norm of theparameter vector (L1 or L2 the most common)

In some cases, it is equivalent to the inclusion of a prior andfinding a MAP solution.

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 31: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Feature extraction

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 32: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Features

Features represent our prior knowledge of the problem

Depend on the type of data

Specialized features for practically any kind of data (images,video, sound, speech, text, web pages, etc)

Medical imaging:

Standard computer vision features (color, shape, texture,edges, local-global, etc)Specialized features tailored to the problem at hand

New trend: learning features from data

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 33: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Feature learning

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 34: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Unsupervised feature learning

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 35: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

AMIDA-MICCAI 2013 Challenge

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 36: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Model application

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 37: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

High-throughput data analytics

Large scale machine learning (big-data):

Large number of samplesLarge samples (whole-slide images, 4D high-resolutionvolumes)

Scalable learning algorithms (on-line learning)

Distributed computing architectures (Hadoop, Spark)

GPGPU computing and multicore architectures

Fabio Gonzalez, PhD An Introduction to Machine Learning

Page 38: An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

IntroductionMachine learning

The machine learning process

Model learningModel evaluationFeature extractionModel application

Questions?

[email protected]

http://www.mindlaboratory.org

Fabio Gonzalez, PhD An Introduction to Machine Learning