CS 672 – Introduction to Performance Evaluation and ...menasce/cs672/slides/CS672-intro.pdf · CS...
Transcript of CS 672 – Introduction to Performance Evaluation and ...menasce/cs672/slides/CS672-intro.pdf · CS...
1
1
CS 672 – Introduction to Performance Evaluation and
Capacity Planning
Dr. Daniel A. Menascéhttp://www.cs.gmu.edu/faculty/menasce.html
Department of Computer ScienceGeorge Mason University
1999−2000 D. A. Menascé. All Rights Reserved.
2
Examples of Performance Problems
• AOL• Market crash of October 1997: investors
lost millions of dollars!• Holiday season of 1998: e-commerce sites
were crowded!• Online trading surge:
http://www.nytimes.com/aponline/f/AP-Internet-Trading-Snarls.html
1999−2000 D. A. Menascé. All Rights Reserved.
2
3
AOL Demand Increase and Capacity Planning Problem
Avg . Sess ion Dura t i on (m in )
0
5
10
15
20
25
30
35
D e c -95
Sep -96
Jan-97
A v g .S e s s i o nDuration( m i n )
P o l y .(Avg .S e s s i o nDuration(min))
Dai ly Sessions (Mi l l ions)
0
2
4
6
8
10
12
Dec-95 Sep-96 J a n - 9 7
DailySess ions(mill ions)
Linear (Dai lySess ions(mil l ions))
Sessions(millions)
Linear (DailySessions(millions))
Peak No. Users
-
50,000
100,000
150,000
200,000
250,000
300,000
Dec-95
Sep-96
Jan-97
PeakSimultaneous Users
Linear (PeakSimultaneous Users)
1999−2000 D. A. Menascé. All Rights Reserved.
4
Performance Problems Threaten E-commerce Success
• “... nascent attempt to reclaim its role as a leading source of information faltered Tuesday when its Web site crashed after being flooded by visitors. ... the overwhelming response caught the company unprepared.”
latimes.com October 20, 1999.
1999−2000 Menascé. All Rights Reserved.
3
5
• The site went down around 4 p.m. PT yesterday while experiencing record traffic, according to a company spokeswoman. The site posted this message: "Please check back soon. Due to the high number of visitors currently using our site, we're unable to start your session at this time."
... and other e-commerce sites have experienced similar intermittent outages as the volume of visitors flooding their sites steadily increases.
CNET News.com October 22, 1999, 11:30 a.m. PT
Performance Problems Threaten E-commerce Success
1999−2000 Menascé. All Rights Reserved.
6
E-commerce Growth(source: Giga Information Group)
94,960
207,572
349,242
438,879
43,926
72,730
97,529
21,227
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
500,000
1999 2000 2001 2002
Ann
ual V
olum
e (in
$m
illio
n)
Business to Business Business to Consumer
1999−2000 Menascé. All Rights Reserved.
4
7
Top e-tailers of 2001(*)
Rank Site
Unique Audience (thousands)
1 amazon.com 31,4872 Columbia House 8,6833 ToysRUs 7,5644 Barnes & Noble 6,2125 apple.com 6,1346 bestbuy.com 5,9937 dell.com 5,7138 walmart.com 5,6529 hp.com 5,223
10 sears.com 4,91911 target.com 4,68912 JCPenney.com 4,65413 cdnow.com 4,27514 hallmark.com 3,97415 compaq.com 3,657
(*) excludes travel sites,financial services, auctions,fnline subscriptions andsoftware OEM.
Source: Nielsen/NetRatings.
2003 Menascé. All Rights Reserved.
8
Nielsen/NetRatings: 2001 Holiday E-commerce Index (*)
Category Week ending Week ending Percent28-Oct 2-Dec Change
Toys & Games 2,761,504 6,156,039 122.90%Consumer Electronics 2,928,326 5,779,183 97.40%Home & Garden 868,266 1,582,941 82.30%Shopping Aggregators 20,055,366 36,286,128 80.90%Apparel 3,469,005 5,785,191 66.80%Virtual Dept. Stores 21,748,016 35,057,201 61.20%Value-Oriented Sites 6,773,770 9,137,098 34.90%Specialty Gifts 594,747 797,077 34.00%Computer Hardware 9,631,755 11,398,165 18.30%Books/Music/Video 4,931,930 5,071,701 2.80%Total 73,762,686 117,050,723 58.70%
2003 Menascé. All Rights Reserved.
(*) 5 representatives in each area. Number of visits.
5
9
Business in the Internet Age(Business Week, June 22, 1998)
Type of Business 1997 2001 (forecast)Business to Business 8.000 183.000Travel 0.654 7.400Financial Services 1.200 5.000PC Hardware & Software 0.863 3.800Entertainment 0.298 2.700Ticket Event Sales 0.079 2.000Books & Music 0.156 1.100Apparel & Footware 0.092 0.514Total 11.342 205.514
1999−2000 D. A. Menascé. All Rights Reserved.
10
How often do you use the Web for shopping for personal reasons?
GVU's 10 th WWW User Survey (October 1998)
3%
27%
31%
17%
9%
13%
0%
5%
10%
15%
20%
25%
30%
35%
Never <1/mo 1-2/mo 3-5/mo 6-9/mo >10/mo
1999 Menascé. All Rights Reserved.
6
11
Caution Signs Along the Road(Gross & Sager, Business Week, June 22, 1998, p. 166.)
There will be jolts and delays alongthe way for electronic commerce:congestion is the most
obvious challenge.
1999−2000 D. A. Menascé. All Rights Reserved.
12
Dissatisfying Experiences in E-commerce GVU's 10th WWW User Survey (October 1998)
4%
23%
14%13%
18%
22%
7%
0%
5%
10%
15%
20%
25%
Hasn't
Happe
ned
Confu
sing S
ite
Poor
page
desig
n
Obnox
ious S
ite
Downlo
ad to
o slow
Couldn
't find
Other R
easo
ns
1999 Menascé. All Rights Reserved.
7
13
What are people saying about Web performance…
• “Tripod’s Web site is our business. If it’s not fast and reliable, there goes our business.”,
Don Zereski, Tripod’s vice-president of Technology (Internet World)
• “Computer shuts down Amazon.com book sales. The site went down at about 10 a.m. and stayed out of service until 10 p.m.”
The Seattle Times, 01/08/98
1999−2000 D. A. Menascé. All Rights Reserved.
14
What are people saying about Web performance…
• “Sites have been concentrating on the right content. Now, more of them -- specially e-commerce sites -- realize that performance is crucial in attracting and retaining online customers.”
Gene Shklar, Keynote, The New York Times, 8/8/98
1999−2000 D. A. Menascé. All Rights Reserved.
8
15
What are people saying about Web performance…
• “Capacity is King.”Mike Krupit, Vice President of Technology, CDnow,
06/01/98
• “Being able to manage hit storms on commerce sites requires more than just buying more plumbing.”
Harry Fenik, vice president of technology, Zona Research, LANTimes, 6/22/98
1999−2000 D. A. Menascé. All Rights Reserved.
16
What are people saying about Internet performance…
• “The capacity crunch is real and will continue for quite some time.”
Mike O’Dell, Chief Scientist, UUNET
1999−2000 D. A. Menascé. All Rights Reserved.
9
17
Impacts of Bad Performance
• Bad performance: response time above 8 seconds (eight-second rule).
• $43.5 billion lost each year in e-commerce due to bad performance (Zona Research, April 1999).
• Holiday Season of 1998: over 1/3 of customers gave up due to slowness, 44% turned to conventional stores, 14% moved to another site.
1999−2000 Menascé. All Rights Reserved.
18
Performance Problems in E-commerce tend to get worse!
1999 Menascé, Almeida, Fonseca & Mendes. All Rights Reserved.
• Proliferation of mobile devices• Easier to use interfaces (VUI, novel
browsing paradigms)• Increasing load placed by software agents• Impacts of authentication and payment
protocols (e.g., SSL, TLS, and SET) on e-commerce site performance.
10
19
Important Questions
• What is performance? What are important performance metrics?
• What is capacity?• What is capacity planning?• How important are capacity planning and
performance evaluation?
1999−2000 D. A. Menascé. All Rights Reserved.
20
Virtual Car Dealer Example
Internet
clients
Webserver
Database:• vehicle description• price• picture
virtual car dealership
1999−2000 D. A. Menascé. All Rights Reserved.
11
21
Virtual Car Dealer Example• Web server request types:
– retrieve document and images– search the DB according to make, model, price
range, and distance of dealer from buyer.– purchase request.
• Critical request: search• 5% of searches generate a car sale• Average sale generates $18,000 in revenues.
1999−2000 D. A. Menascé. All Rights Reserved.
22
Virtual Car Dealer Example• Service Level Considerations:
Response time(in sec)
Outcome
4 < r ≤ 6 60% lost
r > 6 95% lost
1999−2000 D. A. Menascé. All Rights Reserved.
12
23
Virtual Car Dealer Example• Management questions:
– will the Web server support the load increase while preserving the response time below 4 sec?
– if not, at which point will its capacity be saturated?
– how much money could be lost daily if the Web server saturates when the load increases?
1999−2000 D. A. Menascé. All Rights Reserved.
24
Virtual Car Dealer Results
2003 D. A. Menascé. All Rights Reserved.
Current Current+10% Current+20% Current+30%Searches/day 92,448 101,693 110,938 120,182Response time (sec) 2.9 3.8 5.7 11.3Sales lost (%) 0 0 60 95Sales per day 4,622 5,085 2,219 300Daily revenue (in $1,000) 1,664 1,831 1,997 2,163Potential daily revenue (in $1,000) 1,664 1,831 799 108Lost daily revenue (in $1,000) 1,198 2,055
13
25
Important Concepts in Example• Workload characterization?
• Workload Growth Forecast?
• Performance metrics?
• Desired Service Levels?
1999−2000 D. A. Menascé. All Rights Reserved.
26
Important Concepts in Example• Workload characterization?
– retrieve, search, and buy transactions– critical transaction: search– arrival rate: 92,448 transactions/day
1999−2000 D. A. Menascé. All Rights Reserved.
14
27
Important Concepts in Example• Workload growth forecast?
– 10%, 20%, 30% increase in the workload intensity.
– no change expected in workload nature.
1999−2000 D. A. Menascé. All Rights Reserved.
28
Important Concepts in Example• Performance metrics of interest?
– response time for search transactions.– How can be the response time measured?
1999−2000 D. A. Menascé. All Rights Reserved.
15
29
Important Concepts in Example• Performance metrics of interest?
– Response time for search transactions.– How can the response time be measured?
• at the server. What are the components?
• at the browser. Then, Internet and client LAN delays are part of response time.
• Measuring end-user response time for e-commerce sites: www.keynote.com
1999−2000 D. A. Menascé. All Rights Reserved.
30
Important Concepts in Example• Service levels?
– response time for search transactions < 4 sec.
1999−2000 D. A. Menascé. All Rights Reserved.
16
31
Intranet Performance Example• Major airplane manufacturer with 60,000
employees is implementing an intranet to support:– corporate training,– help desk support,– dissemination of internal corporate news, and– handling of personnel forms and memos.
1999−2000 D. A. Menascé. All Rights Reserved.
32
Intranet Performance Example• Help desk application:
– dedicated Web server with a FAQs DB about common hardware/software problems and solutions,
– submission of problem descriptions via forms,– requests for status on previous claims.
1999−2000 D. A. Menascé. All Rights Reserved.
17
33
Intranet Performance Example• 10% of employees submit requests to the help
desk application every day• 70% of these requests fall in the 10:00AM
12:00PM period and from 2:00PM to 4:00PM.• arrival rate =
(60,000 * 0.1 * 0.7) / (4 * 3,600) = 0.29 request/sec.
1999−2000 D. A. Menascé. All Rights Reserved.
34
Intranet Performance Example• The OS on all clients will be changed:
– likely to create big surge in number of requests to the help desk application.
• Management question:– how will the help desk application response time
vary with the arrival rate?
1999−2000 D. A. Menascé. All Rights Reserved.
18
35
Intranet Performance Result
0
10
20
30
40
50
60
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Arrival Rate (tps)
Res
po
nse
Tim
e (s
ec) Connections start to
be refused.
current load.
1999−2000 D. A. Menascé. All Rights Reserved.
36
Important Concepts in Example• Workload characterization?
• Workload Growth Forecast?
• Performance metrics?
• Desired Service Levels?
1999−2000 D. A. Menascé. All Rights Reserved.
19
37
Important Concepts in Example• Workload characterization?
– help desk application: searches to a FAQs DB, submission of problems via forms, requests for status on previous claims
– critical transaction: help desk support requests– arrival rate: 0.29 tps during peak period.
1999−2000 D. A. Menascé. All Rights Reserved.
38
Important Concepts in Example• Workload growth forecast?
– surge in requests to help desk due to new OS installed on client machines.
1999−2000 D. A. Menascé. All Rights Reserved.
20
39
Important Concepts in Example• Performance metrics of interest?
– response time for help desk requests.– connection rejection probability
1999−2000 D. A. Menascé. All Rights Reserved.
40
Important Concepts in Example• Service levels?
– response time for search transactions < 12 sec.
1999−2000 D. A. Menascé. All Rights Reserved.
21
41
Capacity Planning Input and Output Variables
Capacity Planning
WorkloadEvolution
SystemParameters
DesiredService Levels
SaturationPoints
Cost-effectiveAlternatives
1999−2000 D. A. Menascé. All Rights Reserved.
42
Capacity Planning Input and Output Variables
Capacity Planning
WorkloadEvolution
SystemParameters
DesiredService Levels
SaturationPoints
Cost-effectiveAlternatives
•intensity•nature
1999−2000 D. A. Menascé. All Rights Reserved.
22
43
Capacity Planning Input and Output Variables
Capacity Planning
WorkloadEvolution
SystemParameters
DesiredService Levels
SaturationPoints
Cost-effectiveAlternatives
•intensity•nature
• processors, disks, networks• max. no. connections
1999−2000 D. A. Menascé. All Rights Reserved.
44
Capacity Planning Input and Output Variables
Capacity Planning
WorkloadEvolution
SystemParameters
DesiredService Levels
SaturationPoints
Cost-effectiveAlternatives
•intensity•nature
• processors, disks, networks• max. no. connections
• RT <= 2 sec• tput > 10 tps
when are service levels violated? 1999−2000 D. A. Menascé. All Rights Reserved.
23
45
Capacity Planning Input and Output Variables
Capacity Planning
WorkloadEvolution
SystemParameters
DesiredService Levels
SaturationPoints
Cost-effectiveAlternatives
•intensity•nature
• processors, disks, networks• max. no. connections
• RT <= 2 sec• tput > 10 tps
when are service levels violated? e.g., tps/$ 1999−2000 D. A. Menascé. All Rights Reserved.
46
Capacity Planning Definition
Capacity Planning is the process of predicting when the service levels will be violated as a function of the workload evolution, as well as the determination of the most cost-effective way of delaying system saturation.
1999−2000 D. A. Menascé. All Rights Reserved.
24
47
Capacity Planning Concept
0
1
23
4
5
67
8
9
50 60 70 80 100
no. of client workstations
responsetime(sec)
servicelevel
1999−2000 D. A. Menascé. All Rights Reserved.
48
Capacity Planning Concept
0
1
23
4
5
67
8
9
50 60 70 80 100
no. of client workstations
responsetime(sec)
servicelevel
cpuIO
network
1999−2000 D. A. Menascé. All Rights Reserved.
25
49
Typical Capacity Planning Questions
• Situation: migrating from a mainframe based to a C/S system.
• Questions:– how many clients will the new system support
with acceptable response time?– How many servers and how should they be
configured to handle the load?– Should I use a two-tier or a three-tier
architecture?
1999−2000 D. A. Menascé. All Rights Reserved.
50
Two-tier C/S architecture
GUIAppl.OS OS
DBMSExec SQL request
Result of SQL request
DBclient DB server
1999−2000 D. A. Menascé. All Rights Reserved.
26
51
Three-tier C/S architecture
GUI +validation
OS OS
DBMS
Exec SQLrequest
Result of SQL request
DB
OS
clientApplication
ServerDB Server
Appl.
Exec function
Function result
1999−2000 D. A. Menascé. All Rights Reserved.
52
Typical Capacity Planning Questions
• Situation: migrating from a mainframe based to a C/S system.
• Questions:– What should be the configuration of the
application servers?– Should the DB be replicated and how?– Which DB replication strategy should be used?
1999−2000 D. A. Menascé. All Rights Reserved.
27
53
Typical Capacity Planning Questions
• Situation: redesigning an e-commerce Web site and adding a lot of multimedia content.
• Questions:– Will the bandwidth of the link to the ISP
support more multimedia content?– Will the IO bandwidth be enough to provide
adequate response time to multimedia requests?– What is the most appropriate configuration for
the Web site?
1999−2000 D. A. Menascé. All Rights Reserved.
54
Typical Capacity Planning Questions
• Situation: redesigning an e-commerce Web site and adding a lot of multimedia content.
• Questions:– Should I use many small boxes or a few large
boxes to support the load?– What type of load balancing scheme should be
used?
1999−2000 D. A. Menascé. All Rights Reserved.
28
55
Importance of Performance Evaluation and Capacity Planning
• Risk of financial losses• External image of the company• User dissatisfaction• Productivity decrease• Procurement cycle and budgetary
constraints.
1999−2000 D. A. Menascé. All Rights Reserved.
56
Common Mistakes in Capacity Planning
• Performance varies linearly with the workload!
• Just throw more iron and the problem is solved.
• Incorrect data gathering procedures: garbage in garbage out.
1999−2000 D. A. Menascé. All Rights Reserved.