Testing in the Cloud
An investigation of the cloud platform for performance testing
Richard BishopSenior Performance Consultant - Intechnica
Richard Bishop
Senior Performance Consultant at Intechnica 18 years working in IT 13 years as a performance tester
• Acuma (consultancy), Cisco, BT, Vodafone, HBoS, Barclays• HBoS, LBG• Independent contractor• Intechnica
Involved in Mercury User Group Now “Chapter leader” and director for Vivit, HP Software User Group
Contents
Introduction Cloud introduction Cloud hype Common concerns
Cloud Testing Cloud-based testing Pros and cons Choosing a test tool Testing in and from the cloud
Cloud Introduction
“Cloud computing is a model for enabling convenient, on-demand
network access to a shared pool of configurable computing
resources that can be rapidly provisioned and released
with minimal management effort or service provider interaction.”
Source: US National Institute of Standards and Technology (NIST)
Cloud does NOT equal simplification...
The Hype Cycle
As defined by Gartner, the Hype Cycle describes how new technology goes through a cycle of hype and expectations
before its true potential and benefits are eventually realised.
The Hype Cycle
The Hype cycle in detail...
The Hype Cycle - 2011
Cloud types
Cloud is a very broad term and it makes sense to subdivide it further.
In traditional IT, you manage entire stack. Apps, Data, Middleware, OS, Physical hardware etc…
Cloud types
IaaSResponsibility for raw block storage, networking and hardware is outsourced.
Crossover point is at OS level.Utility computing model, pay for what you use.
Cloud types
PaaSResponsibility for support and upgrades of operating
system is also passed over to third party.Similar “pay for what you use” computing model.
Cloud types
SaaSComplete application, data, responsibility for management, maintenance etc.
are passed to a third party.
Cloud types
Common concerns
All “eggs in one basket”
Unproven technology
Support / Learning curve
“Lock-in” (e.g. Azure / AWS proprietary databases)
Costs difficult to quantify and control
Data security and loss of control
Performance, performance, performance……..
vs.
Cloud-based testing
Where does each test tool fit in the cloud?
Cloud-based testing
Cloud test pitfalls
http://www.infoworld.com/d/cloud-computing/face-the-facts-cloud-performance-isnt-always-stable-170066
“Cloud computing performance varies more than you might think….”“….. the price of consistency likely exceeds what you want to spend”
http://www.twitter.com/DavidLinthicum
Cloud test pitfalls
“Cloud computing performance varies more than you might think….”“….. the price of consistency likely exceeds what you want to spend”
This is true for testers as well as developers – we need to work around cost uncertainty & variable performance of cloud platforms.
Measure the performance and consider provisioning some load from conventional sources too!
Realism vs. Repeatability
Realism Repeatability
But how will itreally perform?
Did my change make a difference?
As testers, need to balance realism and repeatability.
Different types of tests require different approaches
Our approach to cloud-testing
Avoid pitfalls• Repeat tests to obtain average results• Avoid peak load times for cloud environments• Re-validate tests within physical test environments• Use diagnostics software to drill down into application performance
…..better application understanding is vital
Intechnica’s tool, based on Facilita Forecast, for flexible load & stress testing from the cloud.
Our approach to cloud-testing
Use benefits of the cloud• Cloud-based load generators• Cloud-based test environments• Cloud-based test environment management
Intechnica’s tool for provisioning cloud-based environments with simplified interface & startup/shutdown scheduler – Available now
Test tools sample checklist
Platform support Ease of installation Ease of scripting Ease of use Data management Cost Stability Scalability Repeatability Market penetration
(availability of skilled testers) Support
Real-time monitoring Real-time load adjustment Dynamic load-adjustment SUT monitoring
o e.g. UNIX / Windows support
Integration with other toolso e.g. Wily, dynaTrace, HP
diagnostics Limits on load (licence / physical) Scheduled tests IP spoofing Rendezvous points Bandwidth throttling
The Cloud Testing Future
Traditional computing models will increasingly be replaced by cloud
Migration can’t be done without careful planning
100% cloud isn’t desirable for many sectors• Mixed cloud based and hosted load generation• Cloud based and traditional test environments• Hybrid approach is more likely
Doesn’t have to be “all or nothing”
Cloud works • Only if you plan and develop performance tests to exploit its benefits• Key thing is to balance realism and cost
Web: www.intechnica.co.ukEmail: [email protected]: 0845 680 9679Fax: 0845 2991647Address: Fourways House, 4th Floor, 57 Hilton Street, Manchester, M1 2EJ
Contact
[email protected]://www.intechnica.co.uk/@intechnica
Top Related