BBV14 Plakat Performance Testing Def NEU€¦ · BENCHMARK TESTING LOAD TESTING PERFORMANCE TESTING...
1
BENCHMARK TESTING LOAD TESTING PERFORMANCE TESTING STRESS TESTING VOLUME TESTING UNIT & ALGORITHM EFFICIENCY (1) A standard against which measurements or comparisons can be made. (2) A test that is used to compare components or systems with each other or to a standard as in (1). E.g. very roughly „The new release must be at least as efficient as the previous release.“ Typical Metric: Comparison to the competitor, comparison to the predecessor. Testing where the system is subjected to large volumes of data. Typical Metric: amount of registered users, database migration/upgrade/size, response time in dependency to data amount. The process of testing to determine the resource-utilization of a software product. Typical Metric: cycles of search algorithm, re-entry of data regions, memory allocation, etc. The process of testing to determine the performance of a software product as the round trip time, system response time, etc. E.g. within unit testing this can be the run time of an algorithm. Typical Metric: delay, run time measurements, response time, network latency. A type of effiency testing conducted to evaluate the behavior of a component or system with increasing load, e.g. numbers of parallel users and/or numbers of transactions, to determine what load can be handled by the component or system. This kind of testing is used to measure the system capability and identify the system bottlenecks based on load specific metrics. Typical Metric: transactions per second, usage attempts per hour, failure rate. A type of efficiency testing conducted to evaluate a system or component at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources such as access to memory or servers. Typical Metric: administration and maintenance attempts must be possible under over load situations. RESOURCES TIME [email protected] www.bbv.ch FINAL REPORTING INITIALIZE Requirements User Stories Business Cases Non-functional Requirements • Expected Load • Required Performance • Reaction Time • Overload Control • User Expectations IDENTIFY GOALS Non-functional Requirements (NFR) Risks Benchmarking PLAN TESTS "Efficiency Tests" Testing Stages IMPLEMENT Load Model Scaling Test Data Tools & Scripts Test Harness Efficiency Acceptance Stress Load Performance Volume Re-Entry Algorithm Efficiency PROD SYSTEM End-to-End SYSTEM Integration INTEGRATION UNIT Test Planning • Cost-Benefit Analysis • Test Scope • Timeline • Amount • Resources • Efficiency Profile (e.g. Load Model) • Definition of Acceptance Criteria EXECUTE & EVALUATE FINALIZE Feedback • Quick Feedback to Stakeholders • Detailed Evaluation • Process Improvement • Product Improvement Outcome • Final Reporting • Backup • Retrospective Time Transactions per Sec. Time Transactions per Sec. PRODUCT IMPROVEMENT PROCESS IMPROVEMENT Network Engineer Application Developer Application Developer PERFORMANCE TUNING AREAS B1 B2 B3 B4 B5 B6 B7 B8 B9 Network Component tuning, Increase Internet bandwidth Configuration (methods, e.g. Round Robin) Caching configuration / Kernel tuning OS-Update / Firmware update Configuration / Communication protocol / Fixing memory leaks / Content optimization / Concurrency / Dedicated system Indexing / Clustering / Hashing Reduce monitoring level / Change monitoring software / Drop monitoring Interface tuning, Subsystem tuning Increase I/O, Optimize file cache Load Balancer Reverse Proxy Operation System & HW Resources Application: Config/Programming Database Monitoring Interface Storage Network Engineer Possible Bottlenecks Measures Role in Technical Team Network Engineer System Engineer System Engineer Application Developer DB Administrator All Application Developer System Engineer Stakeholder Stakeholders require concise information that clearly highlights key points. Intuitive visual representations, consolidated data and summarized information are given preference, as the information is usually required less frequently than by other team members. Project Team Project team members – project manager, development lead, and the test manager – have the same needs as stakeholders but require data more frequently and in greater detail. Technical Team Technical team members are mainly interested in a continuous flow of information related to monitored data, observations and test results. CHECKLIST NON-FUNCTIONAL REQUIREMENTS • Expected user characteristics, determine expected load Number of users Number of (business) transactions per time Sizing and memory requirements • Definition of load scenarios / perturbance scenarios / load peaks • Performance Response times of functions: max. and average Response times on load: max. and average Execution times of business cases: max. and average Execution times of business cases on load: max. and average • System availability Robustness Behavior on overload / overload protection Restart time on load • Consideration of legal and contractual restraints • Considerate performance expectations and performance needs of users Legal Disclaimer: While we have made every attempt to ensure that the information in this publication has been obtained from reliable sources, bbv Software Services (bbv) is not responsible for any errors or omissions, or for the results obtained from the use of this information. All information is provided with no guarantee of completeness or accuracy, and without warranty of any kind. In no event will bbv or its employees therefore be liable to you or anyone else for any decision made or action taken in reliance on the information in this publication. The information in this publication should not be used as a substitute for consultation with professional bbv advisors. Before making any decision or taking any action, you should consult a bbv professional. The names of actual companies and products (e.g. ScrumAlliance, bbv Software Services) mentioned in this publication may be the trademarks of their respective owners. ©2014 bbv Software Services www.zenwis.ch PERFORMANCE TESTING PERFORMANCE TESTING BUSINESS VIEW OPERATIONAL VIEW BUSINESS VIEW OPERATIONAL VIEW PROCESS & PLANNING VIEW
Transcript of BBV14 Plakat Performance Testing Def NEU€¦ · BENCHMARK TESTING LOAD TESTING PERFORMANCE TESTING...
BENCHMARK TESTING
LOAD TESTING
PERFORMANCE TESTING
STRESS TESTING
VOLUME TESTING
UNIT & ALGORITHMEFFICIENCY
(1) A standard against which measurements or comparisons can be made.(2) A test that is used to compare components or systems with each other or to a standard as in (1).E.g. very roughly „The new release must be at least as efficient as the previous release.“Typical Metric: Comparison to the competitor, comparison to the predecessor.
Testing where the system is subjected to large volumes of data.Typical Metric: amount of registered users, database migration/upgrade/size, response time in dependency to data amount.
The process of testing to determine the resource-utilization of a software product.Typical Metric: cycles of search algorithm, re-entry of data regions, memory allocation, etc.
The process of testing to determine the performance of a software product as the round trip time, system response time, etc.E.g. within unit testing this can be the run time of an algorithm.Typical Metric: delay, run time measurements, response time, network latency.
A type of effiency testing conducted to evaluate the behavior of a component or system with increasing load, e.g. numbers of parallel users and/or numbers of transactions, to determine what load can be handled by the component or system. This kind of testing is used to measure the system capability and identify the system bottlenecks based on load specific metrics. Typical Metric: transactions per second, usage attempts per hour, failure rate.
A type of efficiency testing conducted to evaluate a system or component at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources such as access to memory or servers. Typical Metric: administration and maintenance attempts must be possible under over load situations.
RESO
UR
CES
TIM
E
[email protected] www.bbv.ch
FINAL REPORTING
INITIALIZE
RequirementsUser Stories
Business Cases
Non-functional Requirements• Expected Load• Required Performance• Reaction Time• Overload Control• User Expectations
IDENTIFY GOALS
Non-functional Requirements
(NFR)
Risks
Benchmarking
PLAN TESTS"Efficiency Tests" Testing Stages
IMPLEMENT
Load Model
Scaling
Test Data
Tools & Scripts
Test Harness
EfficiencyAcceptance
Stress
Load
Performance
Volume
Re-Entry
AlgorithmEfficiency
PROD
SYSTEMEnd-to-End
SYSTEMIntegration
INTEGRATION
UNIT
Test Planning• Cost-Benefit Analysis• Test Scope• Timeline• Amount• Resources• Efficiency Profile (e.g. Load Model)• Definition of Acceptance Criteria
EXECUTE & EVALUATE FINALIZE
Feedback• Quick Feedback to Stakeholders• Detailed Evaluation• Process Improvement• Product Improvement
Outcome• Final Reporting• Backup• Retrospective
Time
Tran
sact
ion
s p
er S
ec.
Time
Tran
sact
ion
s p
er S
ec.
PRODUCT IMPROVEMENT
PROCESS IMPROVEMENT
Network Engineer
ApplicationDeveloper
ApplicationDeveloper
PERFORMANCE TUNING AREAS
B1
B2
B3
B4
B5
B6
B7
B8
B9
Network Component tuning, Increase Internet bandwidth
Configuration (methods, e.g. Round Robin)
Caching configuration / Kernel tuning
OS-Update / Firmware update
Configuration / Communication protocol / Fixingmemory leaks / Content optimization /Concurrency / Dedicated system
Indexing / Clustering / Hashing
Reduce monitoring level / Change monitoringsoftware / Drop monitoring
Interface tuning, Subsystem tuning
Increase I/O, Optimize file cache
Load Balancer
Reverse Proxy
Operation System & HW Resources
Application: Config/Programming
Database
Monitoring
Interface
Storage
Network Engineer
Possible Bottlenecks Measures Role in Technical Team
Network Engineer
System Engineer
System Engineer
Application Developer
DB Administrator
All
Application Developer
System Engineer
StakeholderStakeholders require concise information that clearly highlights key points. Intuitive visual representations, consolidated data and summarized information are given preference, as the information is usually required less frequently than by other team members.
Project TeamProject team members – project manager, development lead, and the test manager – have the same needs as stakeholders but require data more frequently and in greater detail.
Technical TeamTechnical team members are mainly interested in a continuous flow of information related to monitored data, observations and test results.
CHECKLIS
T
NON-FUNC
TIONAL R
EQUIREME
NTS
• Expected
user char
acteristic
s, determi
ne expecte
d load
Number of
users
Number of
(business)
transacti
ons per ti
me
Sizing and
memory re
quirements
• Definiti
on of load
scenarios
/ perturba
nce scenar
ios / load
peaks
• Performa
nce
Response t
imes of fu
nctions: m
ax. and av
erage
Response t
imes on lo
ad: max. a
nd average
Execution
times of b
usiness ca
ses: max.
and averag
e
Execution
times of b
usiness ca
ses on loa
d: max. an
d average
• System a
vailabilit
y
Robustness
Behavior o
n overload
/ overload
protectio
n
Restart ti
me on load
• Consider
ation of l
egal and c
ontractual
restraint
s
• Consider
ate perfor
mance expe
ctations a
nd
perfo
rmance nee
ds of user
s
Legal Disclaimer: While we have made every attempt to ensure that the information in this publication has been obtained from reliable sources, bbv Software Services (bbv) is not responsible for any errors or omissions, or for the results obtained from the use of this information. All information is provided with no guarantee of completeness or accuracy, and without warranty of any kind. In no event will bbv or its employees therefore be liable to you or anyone else for any decision made or action taken in reliance on the information in this publication. The information in this publication should not be used as a substitute for consultation with professional bbv advisors. Before making any decision or taking any action, you should consult a bbv professional. The names of actual companies and products (e.g. ScrumAlliance, bbv Software Services) mentioned in this publication may be the trademarks of their respective owners. ©2014 bbv Software Services
ww
w.z
enw
is.c
h
PER
FO
RM
AN
CE T
EST
ING PERFORMANCE TESTING
BUSINESS VIEW
OPERATIONAL VIEW
BUSINESS VIEW
OPERATIONAL VIEW
PROCESS & PLANNING VIEW
