Design and Evaluation of a Model for Multi-tiered Internet Applications

Bhuvan UrgaonkarInternship project talk – Services Management Middleware

Dept, IBMAug. 20, 2004

Internet Data Centers

Internet applications run on data centers Server farms

Provide computational and storage resources

Service-level agreements Response time

guarantees Problem: Need good application models to

determine right resource allocations

Multi-tiered Applications

Internet applications: multiple tiers Example: 3 tiers: HTTP, J2EE app

server, database Replicable components

Example: clustered HTTP, J2EE server

requests

http J2EE

database Load balancing gateway

Existing Application Models

Several models for single-tier apps Queuing models for web servers: Chase et

al (USITS 03), Chandra et al (IWQoS 2003) PODC 2004: G/G/1 based model

Model only one (bottleneck) tier Ranjan et al (IWQoS 2002), Villela et al

(IWQoS 2004)

Black-box Approach Black-box approach

Treat application as a black-box Measure response time from outside Increase allocation if response time > SLA

Use a model to decide how much to allocate Strawman #1: black-box for multi-tier apps Problems:

Unclear which tier needs more capacity Bottleneck tier may not be replicable

Black-box Approach

Extension of Single-tier Model

Strawman #2: use single-tier provisioning independently at each tier

Example: Breakdown resp time into per-tier delays Use G/G/1 model for each tier

Problems: How to breakdown resp time? G/G/1 based model found to be very

conservative! Wasted capacity

Talk Outline

Motivation Multi-tier Application Model Preliminary Evaluation Ongoing Work / Discussion Summary

Key Insights/Observations

Tier i requests service from tier (i+1) Scheduling: PS closest tractable among

policies Session-based workloads:

A session consists of a succession of requests Think times are user dependent

Quantity of interest: Per-request resp time

requests

http J2EE

database Load balancing gateway

Queuing-theoretic Model Example: 3-tier application

Natural model: Network of queues shown below

. . . Capturing session-based workload

Infinite server system Closed-queuing system

X_1 X_2 X_3Z

Mean-value Analysis MVA algorithm

Inputs: Avg service times, visit ratios Computes avg. delays and resp. time

. . .

E[X_1] E[X_2] E[X_3]E[Z]

V_0=1 V_1 V_2 V_3Visitratios

MVA Algorithm

for m = 1 to M do Qm = 0for n = 1 to N do begin

for m = 1 to M doRm = Dm for the infinite server

Rm = Dm*(1+Qm) for other servers

X = n / Σ Rm ……… throughput

for m = 1 to M do Qm = X*Rm …… Little’s Lawend

Key relation: Am(N) = Qm(N-1) Am : # customers an arriving customer finds in queue m Qm : # customers in queue m

MVA Algorithm: Discussion Can handle any service time distrib

if scheduling discipline is PS Extension to multiple classes exists

Need to measure service times and visit ratios on a per-class basis

Gives only averages, not distribs Each queue is really only modeling

a single resource

Finding Model Parameters

Visit ratios Easy to obtain from various logs E.g. Apache-tomcat-mysql

V_apache = 2 V_mysql = avg # queries per servlet V_tomcat = V_mysql + 1

Finding Model Parameters Service times

Apache and Tomcat can be made to log time spent at and beyond them

X_apache (T_apache–T_tomcat)/2

X_tomat (T_tomcat–V_mysql*X_mysql)/(V_mysql+1)

X_mysql avg. query exec. time

What we haven’t captured …

Inter-tier load balancers Resources held at tier i while awaiting

response from tier (i+1) Increased service times at high loads

E.g. context switches, protocol processing, contention for locks

Tails of response times Multiple resources Load imbalances due to session affinity

Test Applications

RUBiS (eBay like auction app) BrowseItems, PutBid, AuthorizeBid, PutComment,

RegisterUser, SellItem, SearchItem, StoreBid, …

RUBBoS (slashdot like b-board app)

AcceptStory, BrowseStories, ModerateComment, PostComment, RegisterUser, RegisterStory, ViewStory, …

Experimental Setup

Rubis (e-auctions), Rubbos (b-board) Apache, Tomcat, Mysql

Apache mod_jk redirector Tiers 1 and 2 are replicable Java client

Average think time = 1 sec One thread per session

Apache+mod_jk Tomcat MysqlClient

RUBiS: Response Times

Model works well in a restricted region Tomcat had a connection limit of 75

rubis

0

5000

10000

15000

20000

25000

30000

0 100 200 300 400 500

Num sessions

Avg

resp

tim

e (m

sec)

Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

rubis

0

1000

2000

3000

4000

5000

0 20 40 60 80 100

Num sessionsAv

g re

sp ti

me

(mse

c)

Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

RUBiS: CPU Utilizations

rubis

0

1000

2000

3000

4000

5000

0 20 40 60 80 100

Num sessions

Avg

resp

tim

e (m

sec)

Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

rubis - cpu utils

0

20

40

60

80

100

0 20 40 60 80 100

Num sessionsAv

g cp

u ut

il

Apache

Tomcat

Mysql

Client

App tier is the bottleneck

RUBiS: Processes and Conns.

rubis - num processes

020406080

100120140160180200

0 20 40 60 80 100

Num sessions

Num

pro

cess

es

Apache

Tomcat

Mysql

Client

rubis - num tcp conns

0

50

100

150

200

250

300

0 20 40 60 80 100

Num sessions

Num

tcp co

nns

Apache

Tomcat

Mysql

Client

RUBBoS: Response Times

Again, works well in a restricted region

rubbos

0

2000

4000

6000

8000

10000

12000

0 100 200 300 400 500

Num sessions

Avg r

esp t

ime (

mse

c)

Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

rubbos

0

300

600

900

1200

1500

0 20 40 60 80 100

Num sessionsAv

g res

p tim

e (m

sec)

Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

RUBiS: DB-intensive workload

Replaced SELECT with SELECT SQL_NOCACHE

rubis - resp times

0

5000

10000

15000

0 20 40 60 80 100

Num sessions

Avg r

esp t

ime (

mse

c)

Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

Database tier is bottleneck

rubis - cpu utils

0

20

40

60

80

100

0 20 40 60 80 100

Num sessions

CPU

util

Apache

Tomcat

Mysql

Client

Query Caching at the Database

Able to capture effect of query caching at DB Interesting to do: Caching at app tier

Reduced visit ratio at database

rubis - resp times

0

5000

10000

15000

0 20 40 60 80 100

Num sessions

Avg

resp

tim

e (m

sec)

Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

rubis - caching at database

0

3000

6000

9000

12000

15000

0 20 40 60 80 100

Num sessions

Avg

resp

tim

e (m

sec) Obs at apache

Obs at tomcat

Pred at apache

Pred at tomcat

Multiple Classes of Sessions

Class 1 : App server intensive Class 2 : Database intensive

rubis - 10 sess of Class 1

0

5000

10000

15000

0 20 40 60 80 100

Num sess of Class 2

Avg

resp

tim

e (m

sec) Obs Class 1

Obs Class 2

Pred Class 1

Pred Class 2

rubis - 10 sess pf Class 2

0

500

1000

1500

2000

0 10 20 30 40 50

Num sess of Class 1

Avg

resp

tim

e (m

sec) Obs Class 1

Obs Class 2Pred Class 1Pred Class 2

Talk Outline

Motivation Multi-tier Application Model Preliminary Evaluation Ongoing Work / Discussion Summary

Multiple Servers at a Tier

Apache+mod_jk

Tomcat

MysqlClient

Apache+mod_jk

Tomcat

MysqlClient

Load Imbalance

rubis - 2 app servers

0

500

1000

1500

2000

2500

1 2 5 10 20 50 80 100

Num sessions

Av

g r

es

p t

ime

(m

se

c)

Obs at apache

Obs at tomcat1

Obs at tomcat2

Ongoing: Introduce a skew factor for adjusting the visit ratios to the servers in the replicated tier

Session affinity Variable session

requirements

Applying the Model to more Apps/Implementations

EJB based implementations of RUBiS and RUBBoS

TPC-W Workloads that stress resources other

than CPU More??

Misc. Issues/Discussion Investigate utility of the model in

Capacity planning Dynamic provisioning Admission control

Measurements How many observations to gather?

Handling incr. svc times at higher loads Context switching, locks, protocol processing, …

Response time tails

Summary

Network of queues based model Experimental evaluation for 2 apps

Model works well in limited operating regions Simple enhancements to handle multiple

classes, multiple servers, load imbalance More work needed on several aspects

updating service times, how many observations, resp time tails, more apps and workloads

Acknowledgements

Mike Spreitzer

Asser Tantawi

Giovanni Pacifici

Thank you!

Extension of Single-tier Model