Economics of Computations and Job-Specific Service Level Agreements Bin Li. and Dr. Lee Gillam....

Economics of Computations and Job-Specific Service

Level Agreements

Bin Li. and Dr. Lee Gillam.

Department of Computing, FEPS

Outline

• Part I

• Service Level Agreement• SLA definition• SLA type

• Non-Negotiable: AWS, GAE Examples• Negotiable: job-specific, task oriented

• Simple service brokerage use case• Aim: to build job-specific comparison service for computational market• SLA frameworks

• Proposed SLA structure: based on WS-agreement standard• Service level characteristics

• Availability, performance, autonomic, security • Potentials in computational market• Motivations and literatures

• Level of service is formally defined between service provider and service consumer

• Legal service contract: rights and liabilities.

• Provider: reputation

• Consumer: trust basis• What the services will deliver?• How the services are used?• Choose which provider?

• Legal agreement document • Services description • Requirements• Charges• Legal issues (rights and liabilities)• Penalty / Compensation

Service Level Agreements (SLAs)

3

• SLA Type

• Non-Negotiable: • Pre-defined, abstract and obscure, • In favor of provider, general provider liabilities are

documented to satisfy the most common requirements of consumer,

• Least rights for the consumer,• Common in Cloud service

• Involved penalty: usage credits or stop using service•

• Negotiable:

• Legal document, long and boring with lots of legal terms, difficult to understand

SLAs cont.

(standard edition agreement)… ANY USE (of APP service) THEREOF SHALL BE AT CUSTOMER'S OWN RISK. GOOGLE AND ITS LICENSORS MAKE NO WARRANTY OF ANY KIND … NON-INFRINGEMENT. GOOGLE ASSUMES NO RESPONSIBILITY FOR THE PROPER USE OF THE SERVICE. … GOOGLE MAKES NO REPRESENTATION THAT GOOGLE (OR ANY THIRD PARTY) WILL ISSUE UPDATES OR ENHANCEMENTS TO THE SERVICE. GOOGLE DOES NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE SERVICE WILL BE UNINTERRUPTED OR ERROR FREE.

Google Apps (SLAs)

(SLA)During the Term of the applicable Google Apps Agreement, the Google Apps Covered Services web interface will be operational and available to Customer at least 99.9% of the time in any calendar month (the "Google Apps SLA"). If Google does not meet the Google Apps SLA, and if Customer meets its obligations under this Google Apps SLA, Customer will be eligible to receive the Service Credits (not money back but 3 to 15 days longer service) ...

Customer must notify Google within thirty days from the time Customer becomes eligible to receive a Service Credit. Failure to comply with this requirement will forfeit Customer’s right to receive a Service Credit.

5

Amazon Web Service (SLAs)(EC2)AWS will use commercially reasonable efforts to make Amazon EC2 available with an Annual Uptime Percentage (defined below) of at least 99.95% during the Service Year. In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit.

(S3)AWS will use commercially reasonable efforts to make Amazon S3 available with a Monthly Uptime Percentage (defined below) of at least 99.9% during any monthly billing cycle (the “Service Commitment”). In the event Amazon S3 does not meet the Service Commitment, you will be eligible to receive a Service Credit (10% to 25% of you monthly billing).

“The test for commercially reasonable efforts is less stringent than that imposed by the ‘best efforts’ clauses contained in some agreements.” -- http://definitions.uslegal.com/c/commercially-reasonable-efforts/

To receive a Service Credit, you must submit a request (i) include your account number … (ii) include … the dates and times of each incident of Region Unavailable that you claim to have experienced including instance ids of the instances that were running and affected during the time of each incident; (iii) include your server request logs … (iv) … within thirty (30) business days …

99.95% availability = 0.178days/year down = 4.3 hours/year down

SLAs cont. ---- Job-specific SLA

• SLA Type• Negotiable:

• End-User with critical data or applications requirements,• Representing more flexible user requirements• Job/Application-specific, task-oriented

• Handel manually: inefficient• Dynamic SLAs

• Job-/Application-Specific SLA• Can be applied to both types of SLAs

• Describe services of particular submitted task

• Server management: automatically and dynamically (autonomic) create SLAs while the user demand

changes, per-job SLA , different from ITIL (a general continual SLA)

• Concept and practice of SLA brings the notion of risk management into computational market

• System performance monitoring: system availability, forecasting

• Ensure QoS: Act as a contract between providers and users, negotiate with brokers.

• Clarifies the business nature and parties’ obligations

7

Simple Service Brokerage Use Case

Use case cont.

9

Use case cont.

• Aim: provide the same kind of comparison service for compute resources (Cloud service).

• Goods: compute service which is job-specific .

• Retailers: resource providers (Amazon, Rackspace, Microsoft, Google).

• Invoice: SLA.

• Other factors: • Availability (risk or availability confidence),• Insurance, • Price • Penalty• etc.

• Key: machine readable (automation, autonomic and efficiency)

Objective

11

Structure Xml example

TWO FRAMEWORKS:• Web Service Agreement (WS-Agreement. OGF):

GRAAP, part of Service-Oriented Architecture (SOA), XML

syntax, Machine readable.

• Web Service Level Agreement (WSLA): IBM

• Cloud computing use case group

WS-Agreement:• SDTs: identify the work to be done

• the required platform; • the software involved; • the set of expected arguments;• input/output resources;• etc.

• GTs: provide assurance between provider and consumer on quality of service (QoS)

• price of the service;• insurance price;• the probability of failure;• the penalty for failure;• the starting time• the probability of completion;• etc.

SLA Frameworks &WS-Agreement Structure

Cloud Service Levels (Characteristics)

• Availability: (how often the service can be accessed over a time horizon)• Numbers of “Nines”:

• S3: monthly: 99.9% availability = outrage 43.2 minutes/month• Who should define unavailability?

• EC2: “Unavailable” means that all of your running instances have no external connectivity during a five minute period and you are unable to launch replacement instances.

• Job-specific: future resource availability • Reliability: how well consumer trust the provider

• Related to availability, but slightly different; consumer opinion • Combine cloud offerings: great power and flexibilities but less reliability• Confidence Level: How confidence the provider itself with its availability “nines”?• Job-specific: probability of (job) completion

• Performance:• Throughput: how quick the service respond;• Load balancing: how the overload is avoid;• Elasticity: ability of growing infinitely with limitations;• Linearity: the system performance as workload increases;• Agility: how quick when respond to scaling up or down;• Data durability: the likelihood of data loss;• etc.

• Autonomic: monitoring, automation and dynamic, machine readable.

• Security: privacy, data encryption, legal issues13

Grid, Utility, Cloud…… Computing

Potential computational market


Computational MarketComputational Market

Biggest structure change in IT since 1960s.TechMarketView: by 2012, uk software market 15% will be delivered by Cloud. (22% are applications)


16



Economics Issues Economics Issues

Service Level AgreementsService Level Agreements

......................................absent: Pricing, Liability, etc.

Risk AssessmentRisk Assessment


Time series AnalysisTime series Analysis







Resource MonitoringResource Monitoring ........Analysis Analogy ........

Derivatives Risk AnaDerivatives Risk Ana

Financial DerivativesFinancial Derivatives

Financial Risk Management MeasuresFinancial Risk Management Measures

Financial MarketFinancial Market

....... ...........................

.................. ..............

................................. ...............................Resource PoFResource PoF

Firms PoDFirms PoD

............

..

............

.........


18

•Financial Grids:• Macleod G., Donachy P., Harmer T.J., Perrot R. H., Conlon B., Press J., Lungu F., “Implied Volatility Grid: Grid

Based Integration to Provide On Demand Financial Risk Analysis”, Belfast e-Science Centre, Queen’s University of Belfast, 2005.

• Donachy P., Stødle D., “Risk Grid - Grid Based Integration of Real-Time Value-at-Risk (VaR) Services”, EPSRC UK e-Science All Hands Meeting, 2003.

• Germano G., Engel M., “City@home: Monte Carlo derivative pricing distributed on networked computers”, Grid Technology for Financial Modelling and Simulation, 2006.

• Schumacher J., Jaekel U., and Zimmermann F., “Grid Services for Derivatives Pricing”, Grid Technology for Financial Modelling and Simulation, 2006.

•Computational economics:• Gray, J. (2003): Distributed Computing Economics. Microsoft Research Technical Report: MSRTR-2003-24 (also

presented in Microsoft VC Summit 2004, Silicon Valey, April 2004)• Chetty, M. and Buyya., R. (2002). Weaving electrical and computational grids: How analogous are they?

Computing in Science and Engineering, to appear, May/June 2002.• Kenyon, C. and Cheliotis, G. (2002). Architecture requirements for commercializing grid resources. In 11th

IEEE International Symposium on High Performance Distributed Computing (HPDC'02).• Kenyon, C. and Cheliotis, G. (2003), Grid Resource Commercialization: Economic Engineering and Delivery

Scenarios. Grid Resource Management: State of the Art and Research Issues.• Kerstin, V., Karim, D., Iain, G. and James, P. (2007), AssessGrid, Economic Issues Underlying Risk

Awareness in Grids, LNCS, Springer Berlin / Heidelberg• Birkenheuer, G., Hovestadt, M., Voss, K., Kao, O., Djemame, K., Gourlay, I., Padgett,J .: Introducing Risk

Management into the Grid. Proc. 2nd IEEE Intl. Conf. on e-Science and Grid Computing, Amsterdam, The Netherlands (2006)

Background and literature:

Financial Market Computational Market

ResourcesEquities, Commodities, Currencies...

Financial derivatives

Computers, workstations, Network speed, clusters…

computational power

Capacity characteristicStorable (Stock) /

Non-storable(futures, forwards)

Non-storable

AnalysisUnderlying prices changes time

seriesResource usage time series

Time horizonHolding period

(Hourly, daily, weekly, yearly) Hourly, daily, weekly, yearly

Portfolio Many Resources (assets) Many Computer resources

Confidence Confidence Level / percentile Confidence of resources availability

Result The expected worst loss Optimize the resource use

Risk Market lossesResource Portfolio probability of

Failure

Default Company probability of default Resource probability of failure

Comparison

20

Summary

• Part I

• Service Level Agreement• SLA definition• SLA type

• Non-Negotiable: AWS, GAE Examples• Negotiable: job-specific, task oriented

• Simple service brokerage use case• Aim: to build job-specific comparison service for computational market• SLA frameworks

• Proposed SLA structure: based on WS-agreement standard• Service level characteristics

• Availability, performance, autonomic, security • Potentials in computational market• Motivations and literatures

Outline• Part II

• Analogy: Financial market• Financial market vs. computational market• Financial risk management, portfolio theory • Value-at-Risk (option free portfolio)• Credit Risk

• CDS, CDO• Default probability (Moody’s KMV)

• Asset market value and volatility• Distance of Default• Probability of Default

• Constructing Job-specific SLA• Building probability of failure• Building probability of completion• Building job-specific charges

• Managing multiple Job-specific SLAs (providers) • Conclusion and Future Work

Thank you for your attention

Questions

Computational Economics and Job-Specific Service

Level Agreements

Bin Li. and Dr. Lee Gillam.

Department of Computing, FEPS

• Aim: provide the same kind of comparison service for compute resources (Cloud service).

• Goods: compute service which is job-specific .

• Retailers: resource providers (Amazon, Rackspace, Microsoft, Google).

• Invoice: SLA.

• Other factors: • Availability (risk or availability confidence),• Insurance, • Price • Penalty• etc.

• Key: machine readable (automation, autonomic and efficiency)

Objective

25

Structure Xml example

TWO FRAMEWORKS:• Web Service Agreement (WS-Agreement. OGF):

GRAAP, part of Service-Oriented Architecture (SOA),

XML syntax, Machine readable.

• Web Service Level Agreement (WSLA): IBM

• Cloud computing use case group

SDTs: identify the work to be done• the required platform; • the software involved; • the set of expected arguments;• input/output resources;• etc.

GTs: provide assurance between provider and requester on quality of service (QoS)

• price of the service;• insurance price;• the probability of failure;• the penalty for failure;• the starting time• the probability of completion;• etc.

SLA Frameworks &WS-Agreement Structure

Cloud Service Levels (Characteristics)

• Availability: (how often the service can be accessed over a time horizon)• Numbers of “Nines”:

• 99.95% availability = outrage 4.3 hours/year• Who should define unavailability?

• EC2: “Unavailable” means that all of your running instances have no external connectivity during a five minute period and you are unable to launch replacement instances.

• Job-specific: future resource availability • Reliability: how well consumer trust the provider

• Related to availability, but slightly different; consumer opinion • Combine cloud offerings: great power and flexibilities but less reliability• Confidence Level: How confidence the provider itself with its availability “nines”?• Job-specific: probability of (job) completion

• Performance:• Throughput: how quick the service respond;• Load balancing: how the overload is avoid;• Elasticity: ability of growing infinitely with limitations;• Linearity: the system performance as workload increases;• Agility: how quick when respond to scaling up or down;• Data durability: the likelihood of data loss;• etc.

• Autonomic: monitoring, automation and dynamic, machine readable.

• Security: privacy, data encryption, legal issues

Outline• Part II

• Analogy: Financial market• Financial market vs. computational market• Financial risk management, portfolio diversification • Value-at-Risk (option free portfolio)• Credit Risk

• CDS, CDO• Credit rating: default probability (Moody’s KMV)

• Asset market value and volatility• Distance of Default• Probability of Default

• Constructing Job-specific SLA• Building probability of failure• Building probability of completion• Building job-specific pricing

• Managing multiple Job-specific SLAs (providers) • Conclusion

Time series AnalysisTime series Analysis







Resource MonitoringResource Monitoring ........Analysis Analogy ........

Derivatives Risk AnaDerivatives Risk Ana

Financial DerivativesFinancial Derivatives

Financial Risk Management MeasuresFinancial Risk Management Measures

Financial MarketFinancial Market

....... ...........................

.................. ..............

................................. ...............................Resource PoFResource PoF

Firms PoDFirms PoD

............

..

............

.........


Grid for Financial Risk Analysis • Risk Fact:

• Risk is an integral part of the real world in general, and the financial world in particular.

• Market• Grid infrastructures in Bank of America and HSBC: 3000 to 6000 processors• Computational services market: Customers willing to pay for use of computer

systems instead of purchasing and maintaining hardware and software. • Grid / Cloud: HP, Amazon, Sun, IBM etc.

• Financial Risk Management:• Monitory based, losses or profits.• Risk can only be reduced (Mitigated) but never eliminated.• Fundamental risk management theory: Portfolio (diversification).

• To ensure market event has reduced impact on the whole portfolio• Depends on the correlation or covariance of the return and other assets.• Diversified portfolio: standard deviation of each asset; correlation among

assets• Useful analysis measurements (models): Mean-Variance; Correlation; The sensitivities

(The Greeks); Value-at-Risk

Value-at-Risk (VaR)• Defined by Philippe Jorion, Value at Risk theory “summarizes the worst maximum potential

loss in value of a portfolio of financial instruments over a certain target horizon with a given level of confidence”.

• 3 Components: • Confidence Level (Quantiles), • Holding Period (Time Horizon) • Monetary Base.

Value-at-Risk (VaR)

Monte Carlo Simulation using Condor DAG

Value-at-Risk (VaR)

Methods Comparison

VaR Monte Carlo Simulation Evaluation

Single Financial Instrument MSC Speedup

Option-free Financial Portfolio MSC Speedup

Credit Risk

• Associated with the risk that a reference entity or an obligor who fails to meet its repayment in due time.

• Repayment: principles and debts

• The credit risk = firm default risk

• There are two main determinants of credit risk:

• Loss Given Default (LGD).• (Distance to Default) or Probability of Default(PD), that is, the

probability that the debtor does not pay.• accounting-based models • market-based models (Moody’s KMV)

•

Moody’s KMV-Merton

•

Moody’s KMV-Merton Cont.

Moody’s KMV-Merton Cont.

•

1 Yr

Distribution of asset value at horizon

AssetValue

Today

EDF

Time

Value

Default PointDistance-to-Default

Asset Volatility

Moody’s KMV

Possible asset value path

Liabilities Value (M)

Equity Value (M)

Equity Volatility

Interest Rate

Asset Value (M)

Asset Volatility DD PD

SUN 2004 5311.5 6491 0.5531 0.01877 11700.5477 0.307669673 1.774783 0.037967

2005 6589 6438 0.4634 0.03619 12791.99922 0.233437137 2.077272 0.018888

2006 6141 6674 0.3223 0.04932 12519.47206 0.171815304 2.965301 0.001512

IBM 2004 57246.5 27864 0.2306 0.01877 84046.00467 0.076451448 4.170844 1.52E-05

2005 59617 29747 0.1472 0.03619 87245.03471 0.050189199 6.309559 1.40E-10

2006 53901 33089 0.1787 0.04932 84406.12033 0.070073267 5.157586 1.25E-07

Some results of company PD

The Bridge

Risk analysis

Complex financial products and markets

Service-based

Financial Grids

Computational Economics

compute Resources

Risk-balanced portfolio

Develop possible formulation

provide

construct

• Grid based financial risk analysis applications (Financial Grids):- Great demands on available resources;

- Assume availability at any given time.

• Aim:- Ability to predict (risks of resource availability for) the predictability (risks on

historical use portfolio).

• Major impetus for work-Uncertainty: availability of computation Resource

-Predict future resource availability: computation Resource Monitoring

The Bridge

Building probability of failure

Closest work: Kerstin et al: risk-aware Grid architecture.Kerstin, V., Karim, D., Iain, G. and James, P., “AssessGrid, Economic Issues Underlying Risk Awareness in Grids”, LNCS, Springer Berlin / Heidelberg, 2007

• Specific financial analysis for creating computation economy over queuing-based systems.

Computation Economy as a commodity market;

Due considerations:

1. For trading and hedging of risk, options, futures and structured products.

2. Collecting data: historical computation resource use -> predict future resource use for such class of applicatioons.

3. Construction of portfolios of computer resources (Extension of financial models (CDOs) offers potential for a future market in computation economics) .

• Diversify the risk (resource probability of failure) within the overall portfolio.

CPU usage (Real Time, year data)

CPU usage (Changes, year data)

CPU usage (Changes, MC simulated, normal)

Predict Future Resource Availability

• Grid Resource Historical Usage Analyzing:

• Data source: UK’s National Grid Service (NGS)

• Monitoring system: Ganglia

• Grid middleware: Globus

• Data dimensions: 37 system metrics in XML, including

use of network bandwidth, temperature and CPU use• Minimum capture interval: 15 seconds

• Measurements:• Distribution analysis• Skewness, Kurtosis analysis

• Prediction:• Simulation under Normal distribution assumption• Simulation under Laplace distribution assumption

Building job-specific chargesPrice Comparison Service: Ami: computation resource price benchmark.

Amazon Web Service: success Cloud business model;computation resource cost in real market.

(Price obtained in Dec, 2009)

AWS Linux Instances Monthly Charge

Moderate I/O base instance

price

434Mb Ubunt

u image costs

Large Small VaR (Small)

Amazon EC2 VM per Instance instance-hour (or partial hour) $0.44 $0.11 $0.11

EC2 Bandwidth

In $0.10 $0.10 $0.01

Out $0.17 $0.17 $0.01

S3 use

Outbound data transfer (per month)

$0.17 $0.17 $0.01

other $0.30 $0.30 $0.30TAXES 15%

Total Cost (incl. VAT) $1.36 $0.98 $0.51

• Some REAL Reliability: Of 64 instances in 10 experiments, only 7 completed (1 failing node in other 3)

VaR (640,000

simulation)AWS Condor

Eucalyptus

Overall submission (seconds)

90 95 228

Cost ($) 0.510.48

(90/95 *0.51)

0.20 (90/228*

0.51)

Price benchmark

Building probability of completion

Foster’s Hypothesis

http://ianfoster.typepad.com/.a/6a00d8341c6ed053ef0120a51f9fd2970c-pi

106s

234s

76s

• Is a Cloud better than a Supercomputer? Grid/HPC: shorter application runtime and less distributionsCloud: longer application runtime and larger distributions

ready and relatively easy to use.

Performance

Financial CDO

• Future commercialized computational market: multiple

providers (SLAs)

• Collateralized Debt Obligations (CDOs)

• Structured transaction

• Generic CDO:

• Special Purpose Vehicle (SPV)

• Underlying assets

• Collateral Management

• Tranche Management

• Risk-identified chunks: Tranches (in the order that secured to be

get paid. Eg. AAA; AA; BBB; BB and equity)

• Premium: basis points for each tranche

CDO Components

Managing multiple SLAs

Constructing Resource CDOProcesses:

• sort resources among the system into different classes according to the historical information. • make different basis points with premium to guarantee various performances.• top class resource should have highest premium to insure the most availability and performance.

resources CDO

Managing multiple SLAs (Autonomic SLAs)

• Dynamically alter themselves as the resource status changes.

• Strongly connected to the resource CDO, therefore the monitoring system.

• Also considers the situation while the job in tranches fails.

• The more expensive and lower risk submission is always guaranteed completion.

• Protects the processes in the more senior tranches.

• Protecting the brokers.

• Multiple providers? Future grid and Cloud computing will benefit.

• Analogy:• Financial price changes – Computation resource usage changes

• Financial risk management – Risk assessment in computation market

• Financial derivatives – Service level agreement

• Firm probability of default – machine probability of failure

• Build Computation Economy:

• Key: Binding autonomic SLA with Risk analysis

• Aim: Computation price comparison

• Measuring risk: Predict the predictability (future resource availability)

• Risk mitigate: Resource CDOs

• Initial steps: predict future resource availability (probability of failure);

building probability of completion;

build job-specific service price benchmark;

construct resource CDO;

Conclusion

Further references

Li, B., Gillam, L., and O'Loughlin, J. (2010) Towards Application-Specific Service Level Agreements: Experiments in Clouds and Grids, In Antonopoulos and Gillam (Eds.), Cloud Computing: Principles, Systems and Applications. Springer-Verlag.

Li, B. and Gillam, L. (2009), Grid Service Level Agreements using Financial Risk Analysis Techniques, In Antonopoulos, Exarchakos, Li and Liotta (Eds.), Handbook of Research on P2P and Grid Systems for Service-Oriented Computing: Models, Methodologies and Applications. IGI Global.

http://binlialfie.appspot.com/publications.html

Thank you for your attention

Questions

Economics of Computations and Job-Specific Service Level Agreements Bin Li. and Dr. Lee Gillam....

Documents

Transcript of Economics of Computations and Job-Specific Service Level Agreements Bin Li. and Dr. Lee Gillam....