Time Series Prediction and Support Vector Machines ICONS Presentation Spring 2006 N. Sapankevych 16...

Time Series Prediction andSupport Vector Machines

ICONS Presentation

Spring 2006

N. Sapankevych16 April 2006

N. Sapankevych - 4/16/062

Time Series Prediction and SVMs

Research Overview SVMs for Time Series Prediction Applications Q&A

AGENDA



Research focus on Support Vector Machines (SVMs) and their applications for time series prediction

– Extension of previous research work (and qualifiers) w/ MLPs and SVMs as classifiers

Goal to complete SVM time series prediction survey paper by end of Spring 2006 semester

– Work in progress (over 50 papers down-selected from several hundred published in dozens of different journals)

– Build on research work formulated in qualifying exams

Further goal to define thesis topic– Extend SVM time series prediction research– Potential collaboration w/ UCF (more on that later)

RESEARCH OVERVIEW



Quick SVM history and background– Invented by Vapnik early 1990’s

First instantiation actually demonstrated in COLT ‘92

– SVM another “class” of learning machine– Primary application for (mostly binary)

classification problems– Also, SVMs used for regression applications

(function estimation) – More recently, research on use in time series

prediction applications (focus of research) Note here SVR (Support Vector Regression) plays

significant part in this

TIME SERIES PREDICTION USING SVMs



Why use Artificial Neural Networks (ANNs) for time series prediction?

– Traditional methods assume “model” or “data generating process”

ARMA and its derivatives Kalman {other – IMM, MHT, Particle Filter, etc.} Stationary (and sometimes Gaussian) processes assumed

(drawback in some cases) Assumed model itself may be incorrect in some cases

– Neural Networks “let the data speak for itself” Model-free Can work in both linear or non-linear applications Performance tradeoff with computational complexity Found to perform well w/o all the data

TIME SERIES PREDICTION USING SVMs (con’t)



Many SVM advantages over other ANN-based learning machines such as MLPs (based on architecture)

– Guaranteed unique solution to cost function (quadratic programming)

Not necessarily so in MLPs

– Map non-linear functions to linear space using Kernel functions

Would need more layers/neurons as compared to MLPs

– Fewer free parameters for optimization than MLPs Selection of Kernel function and other constants somewhat of

an “art” for both linear and non-linear applications Selection of parameters may be just as hard, however




SVM basics:– For supervised learning and for a regression application,

define and minimize the “risk” functional

– Define the “loss” function (quadratic in this case)

– x is input vector (input)– y is output vector (response) is free variable– P(x,y) is NOT KNOWN– There are other loss functions

-insensitive loss function most popular (Vapnik)


),()),(,()( yxdPxfyLR

2)),(()),(,( xfyxfyL



More on loss functions: -insensitive loss function most popular (Vapnik –

linear model shown below)

– Note zero term for values within boundary– These loss functions have been shown (Vapnik,

others) to be robust for function estimation


otherwisexfy

xfyxfy

),(

),(,0),(



Empirical Risk Minimization (ERM) vs. Structural Risk Minimization (SRM)– ERM means minimize the following by finding :

– SRM means minimize the following by adjusting (C and as well):

– Why the extra term? If you don’t have much data, ERM may be inadequate

method by which to measure “goodness” of fit – add regularization term (also known as capacity control term)

– What does the extra term do? “Flattens” the fit Controls Capacity


L

iiemp xfy

LR

1

2)),((1

)(

2

1

2

2)),(( wxfy

L

CR

L

iireg



More on SRM and ERM:– One fundamental difference between SVMs and

MLPs is the regularization term– Regularization term assumes weights (w) are for

linear regression function

What if function to estimate is not linear?– Use Kernels: transform data to higher

dimensional space


bwxY

TangentHyperbolicXXK

RBFGaussianXXK

PolynomialXXK

iT

i

p

iT

00

2

22

tanh

)(2

1exp

1



SVM Architecture (from Scholkopf tutorial)




How do you solve for weights?– Solution turns out to be quadratic programming

problem– Use Lagrange multipliers to find optimal weights

Non-zero multipliers associated w/ “Support Vectors” {details found in my Qualifier #2 presentation –

classification example below}


X2

X1

Separating Hyperplane

Support Vectors



Much more to this effort (work in progress) and more questions to answer specifically for time series prediction:

– How to train SVM for this application? SMO Gradient Ascent Chunking {other} Computational complexity and data quantity issue for training How often do you need to retrain and when?

– Sparse data vs. performance Same “kind” of problem for Kalman Filter (time updates – covariance

growth)– How to pick a Kernel function?– How to pick a regularization function?– How to pick regularization constants?– How to manage error?

Stability?– Any “real time” applications?

Computational complexity may be prohibitive




Current research effort focused on exhaustive literature search for SVM Time Series Prediction and its applications– Note K.-R. Muller et. al. “Predicting Time Series

with Support Vector Machines” and Vapnik’s “Statistical Learning Theory” key publications in this area

– Other tutorials and resources available

SVM research relatively new endeavor (about 10+ years)– Published articles growing in number and by

application

APPLICATIONS



Largest quantity of published articles found to date is from financial market prediction applications

– Journal publications indicate SVMs perform well given non-linear nature of stock price prediction

– Other related financial applications such as credit rating forecasting

Several other applications– Utility forecasting (power load and consumption)– Manufacturing industry (machinery MTBF and reliability

forecasting)– Medical applications (drug effects prediction)– Environmental (rainfall, pollution)

Notably missing (as I have found so far)– Signal Processing– Electromechanical Control Systems– Remote Sensing applications (radar tracking, etc.) – Maybe SVMs not well suited for these real time applications?

APPLICATIONS (con’t)



Potential collaborative research opportunity and application to my current research

– Dr. W. Linwood Jones at UCF Central Florida Remote Sensing Laboratory (CFRSL) http://www.engr.ucf.edu/centers/cfrsl/Index.htm

– CFRSL focus on satellite-based remote sensing applications

SeaWinds microwave scatterometer measuring ocean surface wind vectors (QuickSCAT satellite)

Several algorithm development efforts (among other things)– Rainfall, windspeed, surface temperature, and other derived

remote sensing products

– Work in progress to establish cooperative research effort and (possibly) generate thesis topic

Possible research: multi-sensor applications using SVMs

APPLICATIONS (con’t)



Q&A

Q&A

Alexander– 7 months and getting bigger!

Time Series Prediction and Support Vector Machines ICONS Presentation Spring 2006 N. Sapankevych 16...

Documents

Transcript of Time Series Prediction and Support Vector Machines ICONS Presentation Spring 2006 N. Sapankevych 16...