Introduction To Machine Learning

Chun Ming Chin

Microsoft Ventures - @MSFTVenturesChun Ming Chin - @chinchunming(Machine Learning Workshop by Microsoft Ventures)

Objectives

• Understand why is machine learning important• Learn how to apply machine learning in your use case• Adopt best practices in machine learning

Overview• Why Machine Learning• What is Machine Learning• Frame common tasks as machine learning problems

• Example 1: Mobile Optical Character Recognition on Asian text• Example 2: Predict Housing Rental Prices

• Accuracy Issues (i.e. Generalization)• Solutions to Generalization• Putting it all together: Machine learning in stock trading• Machine Learning Best Practices

Why Machine Learning?1. Increase barrier to entry when product/service quality is dependent on data

Product/ Service

Users

DataIncrease

quality/quantity

Why Machine Learning?2. Automate human operations to increase productivity and lower cost

• Example: Auto identify and ban bots that sign up on your website• Use Case: Consider rules based approach first. When tasks cannot be completed

with specific rules, then use ML.

_______@______.com

Why Machine Learning?3. Customize product/service to increase engagement and profits

• Examples: Customize sales page to increase conversion rates for online information product

ABCDEF

AB

CD

EF

Machine Learning Example

Chinese Traditional (Sophisticated and big)Japanese (Squiggly and cute) かき けこ さ し すせ

婆魔佛特級氣喜歡

Features in 2D space

ChineseJapanese

No.

of b

lack

pix

els i

n im

age

No. of straight lines in image

Goal: Get computer to classify input image as Chinese or Japanese.

• Features: Characteristics of the image/ measurements from datae.g. No. of black pixels/ orientation of strokes on images

Label 2Label 1

ML Terminology• Data point = Sample = Example

• Labels/ Classes/ Categories:• Discrete (e.g. Optical Character Recognition)• Continuous (e.g. Housing prices)

• Classification/decision boundary:• Separates regions of feature space• Hopefully helps separate different classes

Features in 2D space

Inde

x io

f dat

a po

int

Feature dimension j

𝑉𝑉 =

Label

Data Point

Decision boundary

What is Machine Learning?Unsupervised learning • Algos that operate on unlabelled examples

• Discover structure/ patterns in the data.

Supervised Learning• Algos trained on labelled examples• Predict an output for previously unseen

inputs.

Supervised Machine Learning (Classification)

Measurements (features) & associated class labels

Training Data Set

Training stage (Usually offline)

Trainingalgorithm 𝑓𝑓 𝑥𝑥

Structure + Parameters

Learned Model

Input Test Data Point

Measurements (features) only

𝑓𝑓 𝑥𝑥Predicted Class Label

Testing stage (Run time, online)

Mobile Optical Character Recognition of Asian Text

Input test images Image classification

Me Competition

Expense Middle

Classifier

𝑓𝑓 𝑥𝑥

Image Measurements:: Orientation of strokes in image/ spatial position of pixels

Use Case: Scale up a product concept with trade off on accuracy.

Supervised Machine Learning (Regression)

Measurements (features) & associated continuous labels

Training Data Set

Training stage

Trainingalgorithm 𝑓𝑓 𝑥𝑥

Structure + Parameters

Learned Model

Input Test Data Point

Measurements (features) only

𝑓𝑓 𝑥𝑥

Testing stage

Continuous value

Output

Example: Predict rental prices based on house area (Sq ft)

Training stage

1. Raw Input Data:

2. Use training algorithm from Python’s ML library.

3. Get resulting ML model 𝑓𝑓 𝑥𝑥

Use Case: Make predictions based on historical data

Case Study: Predict rental prices based on house area (Sq ft)

Rental Price ($)Feature: Area

Regressor

𝑓𝑓 𝑥𝑥Cheap

Expensive

Testing stage

1800 Sqft

Rent

al P

rice

($)

Area (Square Feet)

Optimization with Objective FunctionIteration 1

Iteration 2

Iteration 3

Iteration 4

Rent

al P

rice

($)

Area (Square Feet)

Generalization Issues

Legend

Test data point

Train data point

Generalization Issues

Under fitting:• Number of features used is too small• There are patterns in the data that algorithm is unable to fit

Over fitting:• Number of features used is too large• Fitting serious patterns in the training data set rather than capture true underlying

trends

Under fitting in Regression Over fitting in Regression

Under/Over Fitting in House Rental Prices Prediction

Under fitting in Classification Over fitting in Classification

Under/Over Fitting in Optical Character Recognition

Outlier

Generalization

Over fitUnder fit

Test Error

Bestgeneralization

No. of iterations

Erro

r

Training Error

Fixes for ML algorithmsSolutions to accuracy issues prioritized descending order of sensitivity to classification error:

1. Training data improvement• Get more training examples (Fixes over fitting)• Ensure training data is high quality (De-noise training data)

2. Modify objective function

3. Feature engineering • Increase/reduce number of features (Fixes under/over fitting)• Change features used (Fixes under fitting)

4. Optimization algorithm • Change the ML model used (SVMs , Decision trees, neural network, etc.)• Run optimization algo for more iterations to ensure it converges

Solution: Increase amount of training data

Test Error

Bestgeneralization

No. of iterations

Erro

r

Training Error

Test ErrorBest

generalization

No. of iterations

Erro

r

Training Error

Before After

Feature Engineering: Increase number of features • Combine features: 𝑥𝑥3 = 𝑥𝑥1x 𝑥𝑥2

• Convert continuous features into categorical features (i.e. Bucketize feature values)

• Create a new feature as an indicator for missing values in another feature and supply a default value to the missing feature value.

• To address non-linearly separable data, use non-linear features (e.g. For original feature x, add derived feature 𝑥𝑥2. But not advisable beyond degree 2).

Feature Engineering: Reduce number of features

Reduce no. of features and identify most important features • Data is more compact and dense• Can train and classify faster• Can improve accuracy

Filter out stroke orientationinformation along 0, 45, 90 and 135 degrees.

Extract Feature

Feature vector dimension:5 (No. of sub blocks per row)

x 5 (No. of sub blocks per column) x 2 (Calculate avg & var per sub block)x 4 (No. of orientations)= 200

Feature Engineering Example: Asian Optical Character Recognition• Use domain knowledge to choose features that distinguishes different classes well• Read academic papers to understand prior work in the field

Solution: Try different ML models (i.e. Optimization algorithms)

Decision Factors• Ease of training/ testing• Ease of debugging• Model size (Memory constraints)• Accuracy/ generalization potential• Data characteristics (e.g. For non-linearly separable data, use non-linear

models)

Support Vector Machines (SVM)Why:• Linear and non-linear classification• Best for binary (i.e. 2 class) classification

though can be used for multi-class scenario• Easy to train• Guaranteed global optimum• Scales well to high dimensional data

What:• Find decision boundary that maximizes

margin between classes. Boundary only determined by nearby data points

Support Vectors

Support Vector Machines (SVM)Choosing correct kernel function for non linear SVM (Original video from Udi Aharoni here: https://www.youtube.com/watch?v=3liCbRZPrZA)

1. Find non linear boundary that separates blue from red data points in 2D space

2. Map data points into 3D space using polynomial kernel.

3. Linearly separate data points in 3D space using a plane.

4. Map back to 2D space to determine non linear boundary.

Decision Trees Why:• Non linear classification & regression• Pros:

• Easy to understand and debug• Finds most important features in data• Requires little data preparation

• Cons:• Memory concerns limit accuracy of

decision trees. Deeper trees, higher test accuracy. But trees grow exponentially with depth

What: • Partition feature space into smaller pieces • Learn tree structure and split functions

Root Node

C = 11J = 1

𝑥𝑥1 > 140𝑥𝑥1 < 140

𝑥𝑥2 > 140𝑥𝑥2 < 140

C = 3J = 13

C = 2J = 1

C = 1J = 12

Split function

Decision Forests Why:• Solves instability in decision trees - Small variations in data can generate a different tree• Improves memory-accuracy tradeoff as trees can be parallelized.

What: • A collection (i.e. Ensemble) of trees• Aggregate predictions across all trees.

(Deep) Neural Networks

Why: • For non linear classification & regression• Pros:

• Fast in testing stage• Robust to noise

• Cons:• Slow in training stage• Only guarantees local optima

What• Sequence of non-linear combinations of extremely

simple computations with high connectivity

Input layer

Output layer

Hidden (Deep) Layers

Model selection: Use Cross ValidationIn

dex

iof d

ata

poin

t

Feature Dimension j Label

Validation Set

Training Set

Held Out Cross Validation1. Randomly split all data into 2 subsets:

• Training set (70%)• Validation set (30%)

2. Train machine learning model on training set.3. Pick model with lowest error on validation set.

K Fold Cross Validation1. Divide data into K pieces2. Train on K – 1 pieces3. Validate on remaining piece4. Average over K results to get generalization error of

model

91.66% 93.10%

21.40%

89.00% 89.00% 92.70%

75.00%

87.04%

0.02%

35.96% 35.96%

85.92%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

NearestNeighbor

Linear SVM Sigmoid SVM Polynomial SVM RBF SVM IntersectionSVM

Asian Optical Character Recognition Validation Results

With reduced feature dimension Without reduced feature dimension

Putting it all together: Machine Learning in Stock TradingUse case: Machines see patterns in big data faster and perform tasks faster than humans.

Case Study: Machine Learning in Stock Trading

• Goal: Create profitable trading strategy• Data: Use company (e.g. United Airlines) stock data from Wharton

Research Data Services (WRDS) https://wrdsweb.wharton.upenn.edu/wrds/

• Implementation: Predict closing price on next time step based on information from current time step

*Disclaimer: This may not make money

Case Study: Machine Learning in Stock TradingTraining stage

Trainingalgorithm

𝑓𝑓 𝑥𝑥Structure + Parameters

Learned Model

Case Study: Machine Learning in Stock Trading

BuyTrade price at next time step > Trade price of the current

time step (i.e. positive returns). Label = +1.

HoldTrade price at next time step = Trade price of current time

step. Label 0.

SellTrade price at next time step < Trade price of the current

time step (i.e. negative returns). Label = -1.

Measurements: Average trade price per sec, standard deviation of trade price per sec

Trade Price

Testing stage

Classifier

𝑓𝑓 𝑥𝑥

Training Data Improvements

o Terminology• Trade price: Price at which shares last traded hands• Bid price: Price a buyer is willing to pay• Bid size: No. of shares available at bid price• Offer price: Price a seller is looking to get• Offer size: No. of shares available at offer price

o Intuition… trade price next second should depend on the bid-offer information now.

o Add bid and offer price data (That is within 5% of trade price) to training data

Feature Engineering

Time = 9:30:03 am Time = 9:30:04 am Time = 9:30:05 am

United Airlines Inc. (Ticker Symbol UAL) Date: 2011 Dec 01

• Extract measurements from the distribution of the bid-offer curve at each second window of the bid-offer curve at each second window.

BidOffer

BidOffer

BidOffer

Machine Learning Best Practices

Combine human intuition/ wisdom with machine speed/ pattern recognition.

• Use domain knowledge to choose features that distinguishes different classes well

• Add specific rules to process input data before training stage/ output data after test stage. (In contrast with generalized rules from machine learning) Reduces training time when data is pre processed instead of letting ML model learn the patterns eventually.

Useful Machine Learning Tools

• Python installer for Windows with all necessary ML libraries (e.g., SciPy, NumPy, etc.) http://winpython.sourceforge.net/

• http://www.r-project.org/ R project for statistical computing

• http://prediction.io/ Create predictive features, such as personalization, recommendation and content discovery

• http://www.tableausoftware.com/ Enables you to visually analyze your data

Unsupervised Learning Application Scenarios

(e.g. MinHash Clustering, Matrix Factorization, Dimensionality Reduction)

1. Simplify your data so as to provide insights for 3rd party businesses.

2. Interpret data to test assumptions about your user’s behavior/ market

3. Cluster your customers into different groups with different needs so as to increase monetization.

4. Make more informed business decisions for your startup/ customers.

More Best Machine Learning Practices• Having simple models with clear explanations is better than complicated moel with unclear explanations for

debugging purposes.• To address non-linearly separable data: Use non linear features/ classifiers.• Don’t confuse cause and effect with noise from your data. Consider using certain statistical tests (e.g.

McNemmar’s statistics) to check whether your result is statistically significant.• If your training data is too small, it can cause over fitting problem.• Modularize machine learning model so that it is easy for new people to see conveniently and experiment

easily.• Have a common baseline when comparing improvements to your machine learned model. Common baselines

enable you to share resources when comparing between different techniques.• Modularize your code such that is easy for new people to experiment with different techniques (Parameter

sweeping etc.) quickly. Define inputs/ outputs/ training parameters clearly.• Compare to natural baselines. Guess global average for items ratings. Suggest globally popular items.• You can use UI/ UX to your advantage to find hacks around the balance issue of computational speed

(Latency) and memory capacity. • Incrementally update your ratings using Stochastic Gradient descent. (i.e. As I get new observations, I’ll

update for that user and item only). An alternative is weekly batch retraining.• The more expressive your model, the less expressive your features need to be. The less expressive your

model, the more expressive your features need to be.

• Think about scaling early:1. Sample a subset of ratings for each user so that you can handle the matrix

in memory.2. Use MinHash to cluster users (DDGR07)3. Distribute calculations with Map Reduce4. Distribute matrix operations with Map Reduce [GHNS11]5. Parallelize stochastic gradient descent [ZWSL10]6. Expectation-maximization for pLSI with MapReduce [DDGR07]

Note: Niche vs general – tf-idf. Both of us watching a niche movie should mean more than if I watch a popular movie. For practical constraints, people use item based similarity very often.

More Best Machine Learning Practices