Earthquake Risks and Effective of Earthquake Load and Wind Load on Behavior Of
Effective Use of Load Testing Tools to Tune Web Applications
-
Upload
softwarecentral -
Category
Documents
-
view
827 -
download
9
description
Transcript of Effective Use of Load Testing Tools to Tune Web Applications
04/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
11
Effective Use of Load Testing Tools Effective Use of Load Testing Tools to Tune Web Applications to Tune Web Applications
Tom Igielski, CSTE, CPC, CITom Igielski, CSTE, CPC, CI
The Thomas Consulting Group, Inc.The Thomas Consulting Group, Inc.
[email protected]@thomasconsultgrp.com
January 8January 8thth, 2009, 2009
2204/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Discussion TopicsDiscussion Topics
Web Terminology and ArchitecturesWeb Terminology and Architectures
Common Problem AreasCommon Problem Areas
Suggested Testing ApproachesSuggested Testing Approaches
How Does Architecture Affect ScalabilityHow Does Architecture Affect Scalability
Tuning Your ApplicationTuning Your Application
3304/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Application ScalabilityApplication ScalabilityWeb applications are considered “scalable” if a graph depicting
Web page response times compared to the number of concurrent users accessing the system increases in a linear fashion
Users
Res
pon
seT
ime
Expected
Acceptable
Crash
Ideal
4404/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
BottleneckBottleneck
A limiting resource that affects the system under testA limiting resource that affects the system under test
Typically the cause of slow or unacceptable Typically the cause of slow or unacceptable performanceperformance
Affects performance and scalability of a systemAffects performance and scalability of a system
Only one in existence at any point in timeOnly one in existence at any point in time
5504/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Typical Web Problem Areas
ApplicationServer
WebServer
DatabaseServer
Firewall
HubHub Router
ISP1
Router
Internet bottlenecks
Application server issues
End-user Connection issues
LoadBalancer Web
Server
Load balancerproblems
Faulty web server within a cluster
Server HW, device and OS issues
ISP2
ISP performance and peering
Network vs. Server issues
ExternalDatafeed
Content issues
Errors
6604/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Candidate Tuning AreasCandidate Tuning Areas
25%
20%
23%
27%
5%
Network
Web Server
A[pplication
Database
Other
7704/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Why Do Bottlenecks Exist?Why Do Bottlenecks Exist?ObviousObvious Inadequate HardwareInadequate Hardware Inadequate Network CapacityInadequate Network Capacity Poor Implementation / Tuning of DatabasesPoor Implementation / Tuning of Databases
Not So ObviousNot So Obvious Poor Architecture and System DesignPoor Architecture and System Design Limited Expertise in System Design / DevelopmentLimited Expertise in System Design / Development Changing RequirementsChanging Requirements Changing Project PlansChanging Project Plans Not Thinking About “Load” or “Scalability” or “Performance” Not Thinking About “Load” or “Scalability” or “Performance”
During Des & DevDuring Des & Dev
8804/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Suggested Testing ApproachSuggested Testing Approach
Plan the Testing ActivitiesPlan the Testing Activities Develop a StrategyDevelop a Strategy Clearly State Your ObjectivesClearly State Your Objectives Define the Scope of the Performance TestDefine the Scope of the Performance Test Identify the Performance RequirementsIdentify the Performance Requirements Develop Necessary Test CasesDevelop Necessary Test Cases Line Up the Necessary Testing ResourcesLine Up the Necessary Testing Resources Define Needed Test Data and Process to MaintainDefine Needed Test Data and Process to Maintain Identify any Necessary ToolsIdentify any Necessary Tools
9904/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Suggested Testing ApproachSuggested Testing Approach
Build the Test TeamBuild the Test Team QA / Testing PersonnelQA / Testing Personnel Database AdministratorsDatabase Administrators Network EngineersNetwork Engineers Application ArchitectsApplication Architects DevelopersDevelopers AnalystsAnalysts SME’sSME’s
101004/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Suggested Testing ApproachSuggested Testing Approach
Develop the Test ScriptsDevelop the Test Scripts Select Busiest PagesSelect Busiest Pages
Always Include the Home PageAlways Include the Home Page Test Most Common Business Test Most Common Business
TransactionsTransactions Test Most Critical Business FunctionsTest Most Critical Business Functions Test Pages that are SuspectTest Pages that are Suspect
111104/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Suggested Testing ApproachSuggested Testing Approach
Establish the Test EnvironmentEstablish the Test Environment Procure the Load Test Tool(s)Procure the Load Test Tool(s)
# Vusers Needed# Vusers Needed
Protocols to SupportProtocols to Support
Monitors NecessaryMonitors Necessary Procure / Configure Test ServersProcure / Configure Test Servers Establish Necessary DatabasesEstablish Necessary Databases Create / Maintain Test DataCreate / Maintain Test Data
121204/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Running Load TestsRunning Load Tests
Start load testing software and allow the system to Start load testing software and allow the system to stabilize before adding many usersstabilize before adding many users
Start ramping up users until you see some system Start ramping up users until you see some system slowdown appearslowdown appear
Check system components for problemsCheck system components for problems Could be servers, application, networkCould be servers, application, network
131304/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Ideal Business ScenarioIdeal Business Scenario
12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12
Customer Search250 users, 50 / hr / user
Customer Referrals100 users, 5 / hr / user
Credit Checks500 users, 5 / hr / user
Recommend Product300 users, 5 / hr / user
Loan Approval150 users, 2 / hr / user
BusinessProcess
am pm
Peak Load = BPMix for Scenario
141404/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Hits / Sec vs. # VUsers
0
10
20
30
40
50
60
70
80
0:00
1:04
2:08
3:12
4:16
5:20
6:24
7:28
8:32
9:36
10:4
011
:44
12:4
813
:52
14:5
616
:00
17:0
418
:08
Relative Time
Val
ues Hits / Sec
# Vusers
151504/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Load Test CasesLoad Test Cases
Generally 3 CategoriesGenerally 3 Categories End-to-endEnd-to-end ModularModular Single or low number of pagesSingle or low number of pages
Modular tests of single / low number of pages work best for isolating Modular tests of single / low number of pages work best for isolating bottlenecksbottlenecks
Create short tests that focus narrowly on problem pages or functionsCreate short tests that focus narrowly on problem pages or functions
Run scripts one at a time, with either fixed delays or no delaysRun scripts one at a time, with either fixed delays or no delays
161604/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Successful Load TestsSuccessful Load Tests
MetricsMetrics Hits / SecHits / Sec # Virtual Users# Virtual Users Transaction Response TimeTransaction Response Time Transactions / SecondTransactions / Second Pages / SecondPages / Second % CPU Utilization% CPU Utilization
171704/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Desired Metric: Hits / SecondDesired Metric: Hits / Second
The number of requests made to the web server per secondThe number of requests made to the web server per second
How many should the application support?How many should the application support?
Determine peak based on the assumption that a % of activity in a Determine peak based on the assumption that a % of activity in a day comes within a window of a few hoursday comes within a window of a few hours
Let H = peak hits per second
Let h = # hits received over a one month period
Let a = % of activity that comes during peak time
Let t = peak time in hours
then
H = h * a / (days * t * minutes * seconds)
H = h * a / (108,000 * t)
181804/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Desired Metric: # Virtual Users
How many users should the application support?Determine the peak Virtual Users
Peak hits/second + page view times
Let U = peak virtual users
Let H = peak hits per second
Let p = average number of hits / page
Let v = average time a user views a page
U = (H / p) * v
191904/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Metrics - ExampleMetrics - Example
h = 150,000,000 hits per month
a = 10% of traffic occurs during peak time
t = peak time is 2 hours
p = a page consists of 6 hits
v = the average view time is 30 seconds
H = (h X a) / (108,000 * t)H = (150,000,000 * .1) / (108,000 X 2)H = 48
U = (H / p) * vU = (48 / 6) * 30U = 8 * 30U = 240
Desired Metric – 48 Hits / Sec or 240 Virtual Users
202004/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
V1280 Average Transaction Response Time
0.5
1
1.5
2
2.5
3
0:00
1:04
2:08
3:12
4:16
5:20
6:24
7:28
8:32
9:36
10:4
0
11:4
412
:48
13:5
2
14:5
6
16:0
0
17:0
4
18:0
8
19:1
2
Relative Time After Ramp Up
Res
po
nse
Tim
e (S
eco
nd
s)
20 Vusers 40 Vusers 80 Vusers
How # VUsers Affect Response Time
212104/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Performance RequirementsPerformance Requirements
Number of “registered” usersNumber of “registered” usersNumber of simultaneous usersNumber of simultaneous usersData growth rateData growth rateTransaction RatesTransaction Rates Pages / SecPages / Sec Hits / SecHits / Sec Business Functions / Unit of TimeBusiness Functions / Unit of Time
Response time limitsResponse time limitsHow users will use the systemHow users will use the system Business scenariosBusiness scenarios
Impending events that will drive useImpending events that will drive use
222204/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Architecture and TuningArchitecture and Tuning
ServersServers System Utilization System Utilization
(CPU)(CPU) Memory UtilizationMemory Utilization ThroughputThroughput Disk I/ODisk I/O Current ConnectionsCurrent Connections QueueQueue Queue Wait TimeQueue Wait Time
DatabasesDatabases Table ScansTable Scans Table LocksTable Locks Cache HitsCache Hits IndexesIndexes
232304/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Architecture and TuningArchitecture and Tuning
Load BalancersLoad Balancers Total Concurrent Total Concurrent
ConnectionsConnections System Utilization System Utilization
(CPU)(CPU) ThroughputThroughput Load DistributionLoad Distribution Load Balancing Load Balancing
MechanismMechanism
FirewallsFirewalls Total Concurrent Total Concurrent
ConnectionsConnections Total Current SSL Total Current SSL
ConnectionsConnections System Utilization System Utilization
(CPU)(CPU) ThroughputThroughput
242404/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Example ProblemsExample Problems
High server CPU usage but goals not metHigh server CPU usage but goals not met Have servers been tuned?Have servers been tuned? Has application code been optimized?Has application code been optimized? SSL overhead?SSL overhead? Additional HardwareAdditional Hardware
Low server memory availabilityLow server memory availability Sufficient memory for applicationSufficient memory for application Memory leak?Memory leak?
252504/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
%CPU Utilization vs. Hits / SecTest Environment
0
10
20
30
40
50
60
70
80
90
0:00
1:04
2:08
3:12
4:16
5:20
6:24
7:28
8:32
9:36
10:40
11:44
12:48
13:52
14:56
16:00
17:04
18:08
19:1
220:
1621:
2022
:24
Relative Time
%C
PU
/ H
its
pe
r S
ec
%CPU
Hits/Sec
262604/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Comparison of Hits / Sec and % CPU Utilization
0
20
40
60
80
100
120
0 10 20 30 40 50 60
Hits / Sec
% C
PU
1 Instance
2 Instances
272704/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Comparison of %CPU vs. Hits / Sec
0
20
40
60
80
100
120
0 2 4 6 8 10 12 14 16 18 20
Hits / Sec
% C
PU 1 Instance
2 Instances
3 Instances
282804/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Comparison of Hits / Sec vs. CPU UtilizationRunning 1 and 2 Instances
0
10
20
30
40
50
60
70
80
90
100
0:00
0:16
0:32
0:48
1:04
1:20
1:36
1:52
2:08
2:24
2:40
2:56
3:12
3:28
3:44
4:00
4:16
4:32
4:48
5:04
5:20
5:36
5:52
6:08
6:24
6:40
6:56
7:12
7:28
Relative Time
% C
PU
/ H
its
pe
r S
ec
%CPU 1 Instance
%CPU 2 Instances
Hits / Sec 1 Instance
Hits / Sec 2 Instances
292904/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Example ProblemsExample Problems
Low activity on systems but increased transaction Low activity on systems but increased transaction response timeresponse time Look at requests queuedLook at requests queued Too few connections availableToo few connections available
Application server activity Application server activity ↓↓ but response time and but response time and timeouts timeouts ↑ as ↑ as observed by clientobserved by client Check server configurationCheck server configuration Look at execute threads availableLook at execute threads available Network problemsNetwork problems
303004/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Example ProblemsExample Problems
Response times “spike” during the runResponse times “spike” during the run Set up application servers in “cluster” modeSet up application servers in “cluster” mode
Eliminates problems with garbage collectionEliminates problems with garbage collection Garbage CollectionGarbage Collection
Heap size large, incorrect garbage collection Heap size large, incorrect garbage collection settingssettings
Disk I/O and CPU high on Application ServersDisk I/O and CPU high on Application Servers Application or server logging levels too highApplication or server logging levels too high
313104/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Affects on Response timeAffects on Response time
Affect of Garbage Collection on Response Time
0
0.5
1
1.5
2
2.5
0:00
0:30
1:00
1:30
2:00
2:30
3:00
3:30
4:00
4:30
5:00
5:30
6:00
6:30
7:00
7:30
8:00
8:30
9:00
9:30
10:0
010
:30
11:0
011
:30
12:0
0
Relative Time
Re
sp
on
se
Tim
e
Response Time
323204/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Example ProblemsExample Problems
Poor Java server performancePoor Java server performance Optimize the JVMOptimize the JVM Check JVM switches for startingCheck JVM switches for starting
Memory consumptionMemory consumptionGarbage collectionGarbage collectionThread countsThread counts
Consider Using Multiple InstancesConsider Using Multiple Instances
333304/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
343404/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
353504/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
V1280 CPU %
17
19
21
23
25
27
29
31
0:00
1:04
2:08
3:12
4:16
5:20
6:24
7:28
8:32
9:36
10:4
0
11:4
412
:48
13:5
2
14:5
6
16:0
0
17:0
4
18:0
8
19:1
2
Relative Time After Ramp Up
CP
U %
90 Threads 60 Threads 30 Threads
Comparing Thread Counts
363604/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
V1280 Average Transaction Response Time
1.2
1.3
1.4
1.5
1.6
1.7
1.8
0:00
1:04
2:08
3:12
4:16
5:20
6:24
7:28
8:32
9:36
10:4
0
11:4
412
:48
13:5
2
14:5
6
16:0
0
17:0
4
18:0
8
19:1
2
Relative Time After Ramp Up
Res
po
nse
Tim
e (S
eco
nd
s)
90 Threads 60 Threads 30 Threads
Comparing Thread Counts
373704/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Example ProblemsExample Problems
Web servers being used unequally with Web servers being used unequally with load balancer in placeload balancer in place Problem with the load balancing Problem with the load balancing
algorithm being usedalgorithm being used Servers sending “false” busy stats back Servers sending “false” busy stats back
to load balancerto load balancer ““Run away” processes on select serversRun away” processes on select servers
383804/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Comparison of Response Times
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0:16
0:48
1:20
1:52
2:24
2:56
3:28
4:00
4:32
5:04
5:36
6:08
6:40
7:12
7:44
8:16
8:48
9:20
9:52
10:2
410
:56
11:2
812
:00
Relative Time
Re
sp
on
se
Tim
e
Instance 1
Instance 2
Load Balancing Problems
393904/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Example ProblemsExample Problems
Slow database query responseSlow database query response Use profiling tool to identify slow queriesUse profiling tool to identify slow queries Optimize queriesOptimize queries
Eliminate unnecessary sortingEliminate unnecessary sortingEliminate unnecessary triggersEliminate unnecessary triggersMinimize number of temp tablesMinimize number of temp tablesUse stored procedures where possibleUse stored procedures where possible
Create necessary indexesCreate necessary indexes
404004/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Database ProblemsDatabase Problems
Impacts on Database
0
5
10
15
20
25
30
Tim
e0:
160:
481:
201:
522:
242:
563:
284:
004:
325:
045:
366:
086:
407:
127:
448:
168:
489:
209:
52
Relative Time
Hit
s /
Se
c Baseline
Test1
Test2
414104/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
Monitoring ActivityMonitoring Activity
424204/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
When Things Go WellWhen Things Go Well
Hits Per Second / CPU %
0
10
20
30
40
50
60
70
0:00
0:48
1:36
2:24
3:12
4:00
4:48
5:36
6:24
7:12
8:00
8:48
9:36
10:2
4
11:1
212
:00
12:4
8
13:3
6
14:2
4
Relative Time
Hit
s P
er
Se
co
nd
/ C
PU
%
Hits Per Second - 403.0.21 CPU % - 403.0.21
Hits Per Sec - 403.0.24 CPU % - 403.0.24
434304/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
When Things Go WellWhen Things Go Well
Average Transaction Response Time
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0:00
0:48
1:36
2:24
3:12
4:00
4:48
5:36
6:24
7:12
8:00
8:48
9:36
10:2
4
11:1
212
:00
12:4
8
13:3
6
14:2
4
Relative Time
Ave
rag
e Tr
ansa
ctio
n R
esp
on
se
Tim
e (S
eco
nd
s)
403.0.21 403.0.24
444404/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
SummarySummary
Web architectures are complex and provide Web architectures are complex and provide many areas that can be “tuned”many areas that can be “tuned”
Develop a specific process for Load TestingDevelop a specific process for Load Testing
Take a surgical approach to address problems – Take a surgical approach to address problems – one at a timeone at a time
454504/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.
QuestionsQuestions??