20th July, 2006 Fareena Saqib NWF

Contents

Problem statement Motivation Project Aim Introduction Scope Literature Review Proposed Solution Methodology Project Modules Comparative Analysis Time Line Conclusion Research Accomplishments Future Recommendations


Problem Statement

Forecasting the performance of network using technique that better conserves the varying

patterns in the data using historical data collected by different

active monitoring tools

Content


Motivation GRID Management System

Allocation of task Parallel processing Storage

GRID System

Task Distributor

Remote

Network1

Remote

Network2

Remote network3

Remote

Network N

Content


Project Aim

The aim of project is to develop a module that forecasts the performance of different networks

based on historical data. So that efficiency of the system can be increased.

Content


Introduction

Forecasting techniques Holt Winters ARMA/ARIMA EWMA Regression Analysis

Why ARMA/ARIMA? Varying trends in network data

Content


ARMA/ARIMA Auto Regressive Integrated Moving Average

Box and Jenkins approach Merger of techniques

o Auto Regression (AR)o Moving Average (MA)

Benefits of AR

Benefits ofMA

ARIMAApproach

Better ResultsContent


ARMA/ARIMA Approach followed:

Box and Jenkins approach is followed

1. Identification of the model(Choosing tentative p,d,q)

1. Identification of the model(Choosing tentative p,d,q)

3. Forecasting 3. Forecasting

4. Diagnostic checking (are the estimated residuals white noise?)

4. Diagnostic checking (are the estimated residuals white noise?)

2. Parameter estimation ofthe chosen model

2. Parameter estimation ofthe chosen model

No

(Return to step 1)

Content


ARMA/ARIMA Identification

Through Correlogram Autocorrelation Function (ACF) Partial Auto Correlation Function (PACF)

Content


ARMA/ARIMA Estimation

Estimation of order Estimation of equation Estimation of coefficients Forecasting of data

Diagnostic Checking To check that model is fit to the data. Obtain residual Obtain ACF and PACF of residual

Content


Use of ARIMA Approach

Content


Use of ARMA/ARIMA

Sales of dates contains seasonal effect. Month of Ramadan

Sales of products Summer Winter Spring

USA economic forecasts Weather forecasts

Content


Network Weather Forecasting

Use of ARIMA in network forecast


ComputerScience

Field

EconomicsField

Statistics

EconometricsField

Network

GRID System

Content


Scope Study of different forecasting techniques

Pros and cons Selection of Technique Development of methodology Verification of the algorithm Modules:

o Data Processing moduleo Forecasting moduleo Visualization moduleo Testing Moduleo Comparative module o Development of user Interface

Content


Research Issues

Research Issues:

Development of algorithm using ARIMA approach Estimation of the coefficients. Diagnostic Checking tests.

Content


Literature Review Development of algorithm using ARIMA approach

Basic Econometrics by Damodar N.Gujaratio Basic concepts:o Time Series Analysis:o ARMA and ARIMA approach introduction.

Time Series Analysis Forecasting and Control by George E.P Box, Gwilym M.Jenkins,Gregory C.Reinsel:

o Study of ARMA/ARIMA in detail.o Box and Jenkins Approach

Basics of statisticso To understand and revise basic concepts of statistics involved in the

project.

Research Issues


Literature Review Estimation of the coefficients.

Estimation of coefficiento http://www.qmw.ac.uk/~ugte133/courses/tseries/8idntify.pdf

Non-linear approacheso http://www.ece.cmu.edu/~moura/papers/icassp88-ribeiro-ieeexplore.p

df Other approaches

o http://www.cs.cmu.edu/afs/cs/project/cmcl/archive/Remulac-papers/tech-report.pdf

Research Issues


Literature Review

Diagnostic Checking tests.

Basic Econometrics by Damodar N.Gujaratio Basic concepts:o Time Series Analysis:o ARMA and ARIMA approach introduction.

Basics of statisticso To understand and revise basic concepts of statistics

involved in the project.

Research Issues


Literature Review Algorithms Involved:

Data Processing

•Selection of Parameter

•Trim Operation •Regularization Algorithm •Moving Average for Interpolation

Forecasting

•Stationarity

•Order Estimation

•Coefficient Estimation

•Formulation of equation

Verification

•Calculation of Residuals

•Trend Analysis

•Portmanteau tests

Content


Proposed Solution

Content


ARIMA Modeling

Postulate General Class of Models

Use Model for Forecasting

Diagnostic Checking

Estimate Parameters in Tentatively

Entertained Model

Identify Model to be Tentatively

Entertained

yesNo

Content


Methodology

Read Data Files

Trim Operation

Data Files

RegularizationOperation

Interpolation Operation

Plot of Processed

Data

Content


MethodologyProcessed

Data

Read Processed

data in array

Trend Analysis

Order Estimation

Coefficients Estimation

Equation Formulation

Foecasting

Content


MethodologyForecasted

Data

Read Forecast

data

Calculate residuals

Trend Analysis

Portmanteau Test

Decision making based on results Content



EstimationOf order

EstimationOf order

Estimation ofCoefficient

Estimation ofCoefficient ForecastingForecasting

Data TrimData Trim User InterfaceUser Interface

correlogramcorrelogram

Estimatio

n

Access Data

Interpolation Interpolation Regularization

Regularization

CovarianceCovariance

IntegrationIntegration

Identification

Diagnostic Checking

Diagnostic Checking

Content


Architecture Diagram

Data files

Docs………………



ForecastingForecasting

User

Data Cleaning

Data Cleaning

EstimationEstimation

GUIGUI

Visualization through graphs

Visualization through graphs

Content


Use Case Diagram

Data Trim

Data Regularization

Correlogram

Data Interpolation

t-test

Integration

Coefficient estimation

ForecastingResidual Generation

Residual CorrelogramPortmanteauTest

Plot of data

Plot of forecast

Administrator

Plot of correlogram

Content


Project Module

Content


Data Processing Module

Content


Data Cleaning Module Data files

Define format of the data

Trim operation

Regularization operation

Interpolation operation

Content


Data Processing

: Administrator

:Input

:InputOutputFile

:TrimData

:Regularization

:Interpolation

1: SelectType(type)2: SelectFormat(format)

3: openFile5: Extract()

9: Regularize()

4: SelectFile()

6: Extract(data,type,format)7: getTrimmedData()

8: returnTrimData(data)

10: Regularize(Trimdata)11: getRegularizeData()

12: returnRegularizeData(data)

13: Interpolate(Regularizedata)14: getInterpolateData()

15: returnInterpolateData(interpolatedata)

Content


Identification Module

Content


Identification Module Calculated autocorrelations.

Number of lags

Trend analysis of autocorrelation coefficients Correlogram

Content


Identification Module

: Administrator

:AutoCorrelati

:InputOutputTrimFile

:Input

:Plot Graph

12: plot()

:Integration

2: openFile()4: selectParameter(param)

5: paramData()8: Correlogram()

1: lags(lags)

9: getAutoCorrelation coefficient()

10: returnAutoCorrelation(data)

3: selectFile()

11: plotCorrelations(data)

6: Integration(data)

7: IntegratedData(data)

Content


Estimation Module

Content


Estimation Module Estimation Issues:

Random variable generation with normal distribution.

Estimation of the Model Estimation of the order Estimation of coefficient Estimation of the equation Testing the t-test of the coefficient

Content


Order Estimation

: Administrator

:Input

:Integrate

:OrderEstimate

:t-test

1: selectData()4: orderEstimate()

2: integrateData()

3: returnIntegratedData(indata)5: order(indata)8: returnAnalysisData(data)

6: testCoefficient(standardarror)

7: returnSignificance()

Content


Forecasting Module

Content


Forecasting Module Processed data with equal time intervals.

Estimation of order.

Formulate Equation.

Estimation of coefficients.

Forecast parameter values.

Plot Graph of forecasted values.

Content


Forecasting

: Administrator

:Input

:InputOutputFile

:EstimateCoefficient

:Forecast

:Plot

:CalculateDates:Residual

Estimation

11: forecast(forecastData)

1: orderAR(ar)2: orderMA(ma)

3: forcast()13: plotForecast()

4: forecast(data)

14: PlotForecast(dates,forecastdata)

7: Coefficient(ar,ma,data,rr)

8: returnCoefficient(c)

12: writeData(dates,forecastData)

9: futureDates(t)

10: returnDates(dates)

5: ResidualEstimate(ar,ma,data)

6: residuals(rr)

Content


Residual test Module

Content


Residual Module Graph of the residuals to check white noise

Check if the forecasting is valid or not.

Correlogram of residuals.

Portmanteau tests To test the Q-values

Content


Residual test

: Administrator

:Input

:ResidualPlot

:ResidualCorrelogram

:Portmanteau test

:AutoCorrelation

1: plotResidual()3: enterLags(l)

2: plot(residual)

4: rCorrelogram(rresidual)

7: returnCovarience(cov)

8: performTest(correlation)

9: sendAnalysis(analysis)

5: getCorrelation()

6: returnCorrelation(correlation)

Content


Visualization Module

Content


Visualization Module Visualization of steps carried by algorithm.

Visualization of the tool developed.

Plots of data at different processing stages.

Content


Demo: Snapshot of tool

Content


Results

Content


Content


Correlogram

Content


ARMA/ARIMA results

200

300

400

500

600

700

800

900

1000

1100

25 50 75 100 125 150 175 200

THMAX THMAXF

Content


Comparison Module

Content


Data plot of xtr

Content


Comparison of Results

0

50

100

150

200

250

300

350

400

450

500

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97

HW

ARIMA

Content


Time Line

Content


ConclusionThe basic requirement of Network Weather Forecasting has been achieved by resorting to two prong efforts.

Firstly the technique of ARMA/ARIMA was followed and secondly an Algorithm was developed, both of which converged in dynamic Network data forecasting.

The methodology adopted was:

Available data on the subject was gathered and processed to be used as an input to the forecasting module.

After studying ARMA/ARIMA and ascertaining its suitability, algorithm was developed.

Based on the adopted approach and developed algorithm, experiments on forecasting were conducted employing the duly processed data.

The results obtained through different experiments were computed, compared and characteristics were plotted to come out with a fair idea of the final accomplishment.

Analysis of the results, their comparisons and other details were carried out before preparation of the report.

Documentation was undertaken to compile the project report.

Content


Research Accomplishments

Developed an algorithm for network weather forecasting.

A research paper on using ARIMA approach in network weather forecasting.

A Journal on results of different forecasting techniques on network data and their comparative analysis.

Content


Future Recommendations The forecasting carried out is based on a single approach which could

be explored for new dimensions.

The forecasting was carried out by employing three different tools of data collection which could be expanded to more numbers of tools in the future.

Efforts maybe initiated to apply new techniques like neural networks and others which are bound to come up in the fast developing field of information technology.

Forecasting of data should be made universal and the present form of retaining it on a single machine could be transformed as a web based tool serving all surfers of the web.

Content


Demo

Content


Thank You!


Appendix


Trim

: Administrator

:Input :InputOutputFile

:TrimData :TrimAbing :TrimIperf :TrimThrouhlay

1: SelectType(type)

2: SelectFormat(format)

3: openFile

4: SelectFile()

5: Extract(data,type,format)6: Extract(data,type,format)

7: [type=abing]Trim(data)

8: [type=iperf]Trim(data)

9: [type=throulay]Trim(data)


Regularization : Administrator

:Input :InputOutputTrimFile

:RegularizeData

:RegularizeAbing

:RegularizeIperf :RegularizeThrouhlay

1: SelectType(type)

2: openFile

4: Regularize(data,type)

3: SelectFile()

5: Regularize(data,type)6: [type=abing]Regularize(data)

7: [type=iperf]Regularize(data)

8: [type=throulay]Regularize(data)


Interpolation

: Administrator

:Input :InputOutputRegularizeFile

:RegularizeData

:InterpolateAbing

:InterpolateIperf :InterpolateThrouhlay

1: SelectType(type)

2: openFile

4: Interpolate(data,type)

3: SelectFile()

5: Interpolate(data,type)6: [type=abing]Interpolate(data)

7: [type=iperf]Interpolate(data)

8: [type=throulay]Interpolate(data)


Parameter estimation : Administrator

:Input :ParamGenerator

:InputOutputTrimFile

:DataFormatter

1: selectTool()

4: openFile()

6: selectParameter(param)

7: paramData()

2: generateParameterList

3: parameterList(param)

5: selectFile()

8: ExtractParam(param)

9: dataforForecast(data)


Identification Module : Administrator

:AutoCorrelation

:Plot Graph:Integration:InputOutputTrimFile

:Input:Input

1: lags(lags)

3: selectFile()2: openFile()


5: paramData()


9: getAutoCorrelation coefficient()8: Correlogram()


12: plot()




Correlogram

: Administrator

:AutoCorrelation

:Plot Graph:Integration:InputOutputTrimFile

:Input:Input

1: lags(lags)

3: selectFile()2: openFile()


5: paramData()


9: getAutoCorrelation coefficient()8: Correlogram()


12: plot()




Order Estimation

: Administrator:Input :Integrate :OrderEstimate :t-test

1: selectData()

4: orderEstimate()

2: integrateData()

5: order(indata)

3: returnIntegratedData(indata)



8: returnAnalysisData(data)


Forecast : Administrator


:Plot:ResidualEstimation

:CalculateDates


:Forecast

1: orderAR(ar)

2: orderMA(ma)

3: forcast()4: forecast(data)





9: futureDates(t)



6: residuals(rr)


13: plotForecast()


Residual tests : Administrator


:Plot:ResidualEstimation

:CalculateDates


:Forecast

1: orderAR(ar)

2: orderMA(ma)

3: forcast()4: forecast(data)





9: futureDates(t)



6: residuals(rr)


13: plotForecast()


Trim

: Administrator

:Input

:InputOutputFile

:TrimData

:TrimAbing

:TrimIperf

:TrimThrouhlay


3: openFile5: Extract(data,type,format)

4: SelectFile()

6: Extract(data,type,format)

7: [type=abing]Trim(data)

8: [type=iperf]Trim(data)9: [type=throulay]Trim(data)


Regularization

: Administrator

:Input :InputOutputTrimFile

:RegularizeData

:RegularizeAbing

:RegularizeIperf

:RegularizeThrouhlay

1: SelectType(type)2: openFile


3: SelectFile()


6: [type=abing]Regularize(data)

7: [type=iperf]Regularize(data)

8: [type=throulay]Regularize(data)


Interpolate

: Administrator

:Input

:InputOutputRegularizeFile

:RegularizeData

:InterpolateAbing

:InterpolateIperf

:InterpolateThrouhlay

1: SelectType(type)2: openFile


3: SelectFile()


6: [type=abing]Interpolate(data)

7: [type=iperf]Interpolate(data)

8: [type=throulay]Interpolate(data)


Data Processing

: Administrator

:Input

:InputOutputFile

:TrimData

:Regularization

:Interpolation


3: openFile5: Extract()

9: Regularize()

4: SelectFile()

6: Extract(data,type,format)7: getTrimmedData()

8: returnTrimData(data)

10: Regularize(Trimdata)11: getRegularizeData()

12: returnRegularizeData(data)

13: Interpolate(Regularizedata)14: getInterpolateData()

15: returnInterpolateData(interpolatedata)


Parameter selection

: Administrator

:Input

:ParamGenerator

:InputOutput

:DataFormatter

1: selectTool()4: openFile()

6: selectParameter(param)7: paramData()

2: generateParameterList

3: parameterList(param)

5: selectFile()

8: ExtractParam(param)

9: dataforForecast(data)


Order estimation

: Administrator

:Input

:Integrate

:OrderEstimate

:t-test

1: selectData()4: orderEstimate()

2: integrateData()

3: returnIntegratedData(indata)5: order(indata)8: returnAnalysisData(data)




Forecasting

: Administrator

:Input

:InputOutputFile


:Forecast

:Plot

:CalculateDates:Residual

Estimation


1: orderAR(ar)2: orderMA(ma)

3: forcast()13: plotForecast()

4: forecast(data)





9: futureDates(t)



6: residuals(rr)


Residual test

: Administrator

:Input

:ResidualPlot

:ResidualCorrelogram

:Portmanteau test

:AutoCorrelation

1: plotResidual()3: enterLags(l)

2: plot(residual)

4: rCorrelogram(rresidual)

7: returnCovarience(cov)

8: performTest(correlation)

9: sendAnalysis(analysis)

5: getCorrelation()

6: returnCorrelation(correlation)


Thank you!!!

Network Weather Forecasting MAGGIE (NWF)

Documents

Transcript of Network Weather Forecasting MAGGIE (NWF)