SaaS Testing Overview - Foundation
-
Upload
ram-garg -
Category
Technology
-
view
15.310 -
download
0
description
Transcript of SaaS Testing Overview - Foundation
SaaS Testing Overview
Software Performance Assurance Confidential
Ram Garg
CEO
Software Performance Assurance (SPA)
Software Performance Assurance Confidential
Outline
SaaS Introduction What is it?
Why SaaS?
SaaS Challenges Key Challenges
SaaS Testing How to approach SaaS testing?
How SaaS Testing is different?
SaaS Test Tools
Software Performance Assurance Confidential
SaaS Introduction
What is SaaS?
Software Performance Assurance Confidential
SaaS: What is it?
Software as a Service (SaaS)
Platform as a Service (PaaS)
Infrastructure as a Service (IaaS)
Software-as-a-Service (SaaS) is the application layer of the cloud computing model.
Application
Platform
Infrastructure
Software Performance Assurance Confidential
SaaS Definition
“Software as a service (SaaS), sometimes
referred to as "software on demand," is
software that is 100% deployed over the
internet … . With SaaS, a provider licenses
an application to customers as a service
on demand, through a subscription or a
"pay-as-you-go" model.”
– Wikipedia definition
Software Performance Assurance Confidential
Architecture: Three Key Differentiators
Key Architecture
Differentiators
Scalable
Multi-tenant Efficient
Configurable
Software Performance Assurance Confidential
SaaS Introduction
Why SaaS?
Upgrades
Platform
Installation
Infrastructure
management Responsibility
Software Ownership
Software Performance Assurance Confidential
The Software Business Model Changes …
non -SaaS
Customer
Customer
Client - Server
Larger, less frequent
Multi-version
SaaS
Provider
Provider
Central
Shorter, frequent
Single-version
Maintenance
Backward compatibility
Multi-tenancy
Configurability
Requirement
Software Performance Assurance Confidential
Changing the Business Model cont …
non -SaaS
Low
No
No
Customer
SaaS
High, Critical
Yes
Yes
Provider
Profits
Sales focus / Success
Cost, Allocation
Software Performance Assurance Confidential
Changing the Business Model cont …
non -SaaS
High, CapEx
Close the deal,
License Revenue
Initial sale + maintenance
SaaS
Low, OpEx
Provide value,
Lack of Churn
Ongoing Subscription
Ecosystem
Starting point to every service within
cloud computing ecosystem
Highest on the maturity level and
growth curve amongst XaaS
Highest revenue generating service
within cloud model for next several
years
SaaS (Software)
PaaS (Platform)
IaaS (Infrastructure)
Software Performance Assurance Confidential
SaaS Catalysts
Intrigue of access to unlimited Computing and storage resources, made available and paid for only as needed (Pay as you go)
Decreasing cost of bandwidth
Customer frustration with traditional software buying cycle (license, maintenance contract, upgrades)
Perceived increase in control over the vendor relationship
Early SaaS success stories – Salesforce.com, WebEx, etc. Demonstrate broader value proposition
You focus on your application and not on infrastructure
Software Performance Assurance Confidential
SaaS Introduction
SaaS Early Adoption
Software Performance Assurance Confidential
SaaS Adoption: Verticals
In-Stat research on
vertical market adoption
shows the healthcare
market is currently most
satisfied with hosted
applications, with
education identified as an
important potential target
market also.
AMI-Partners„ report -
"Software-as-Service Hot
Spots” : Education,
Healthcare, and Financial"
Adoption rate of
SaaS in SMB is
higher than large
enterprises
All research points out that SaaS has been adopted heavily in :
HR applications (ERP, Salesforce automation, eRecruiting)
Online Collaboration (Web conferencing)
CRM
Healthcare
Software Performance Assurance Confidential
SaaS Adoption Factors(source: Forrester)
Availability & Reliability (71%) Fast Deployment (45%)
Reduced dependencies on internal IT (24%)
Flexible consumption (35%)
Simplified pricing model (32%)
Software Performance Assurance Confidential
The SaaS Market is and will
Continue to Grow…
Software Performance Assurance Confidential
Today and tomorrow …
Software Performance Assurance Confidential
Today and tomorrow …
Software Performance Assurance Confidential
Today and tomorrow …
Software Performance Assurance Confidential
SaaS Challenges
Challenges
Software Performance Assurance Confidential
Primary Concerns with the SaaS Model
Security concerns
Concerns over compliance with standards
Concerns over integration with legacy applications
Concerns over reliability and robustness of service
Not comfortable with putting mission-critical application on shared infrastructure and have access over internet
SaaS can not be configured to meet my corporation‟s all specific needs
Software Performance Assurance Confidential
Numerous Challenges
Security Performance Availability SLA
Data Integrity PrivacyCompliance
with Standards
Regulatory
Integration Multi-tenancyLack of
confidence
Software Performance Assurance Confidential
SaaS Testing
Software Performance Assurance Confidential
SaaS Testing
Why SaaS Testing?
Software Performance Assurance Confidential
Why SaaS Testing?The average enterprise IT
department devotes up to 50
percent of its entire technology
infrastructure to development
and test, with up to 90 percent
of the available test
infrastructure remaining idle.
IBM Research
Those using cloud facilities to
run applications, 49.8% said
they were doing so
experimentally or for
prototyping; 28.6% for non-
critical business applications
and 21.6% for business
critical applications
Evans Data Survey
The average online
business had 8 days of
downtime a year due to
performance issues
leading to lost revenues
Aberdeen Group
A delay in website page
loading by even one second
can impact conversions by
7% and customer satisfaction
by 16%.
Aberdeen Group
Cloud
Testing
Trend leads to trend …
Overall growth in Cloud Computing (investment,
revenue, ROI, market cap, etc.) is leading to a
marked increase in adoption of SaaS Testing
Software Performance Assurance Confidential
SaaS Testing
What is SaaS Testing?
Software Performance Assurance Confidential
Classifying SaaS Testing
“Testing The Cloud”
Testing applications which are in
cloud or migrating to cloud.
Test the performance, reliability &
security of applications
“Testing Using Cloud”
Testing applications using the
cloud-based hardware, software,
infrastructure and computing
resources.
Tests performed like performance,
load & stress, security and
compatibility
“Testing The SaaS”
Testing applications which are in
cloud or migrating to cloud.
Test the performance, reliability &
security of applications
“Testing SaaS Using Cloud”
Testing applications using the
cloud-based hardware, software,
infrastructure and computing
resources.
Tests performed like performance,
load & stress, security and
compatibility
Testing the SaaS
Testing SaaS Using Cloud
Cloud / SaaS
Testing
Software Performance Assurance Confidential
How SaaS Testing is Different
Easier
• No client-server installation
• No multiplatform support
• No multiple version upgrades
• No backward compatibility
Challenges
• Frequent live upgrades
• Too many pages
• Multiple privileges
• Immature SaaS-specific testing tools
• Access partitions critical
Software Performance Assurance Confidential
SaaS Testing
Approach
Software Performance Assurance Confidential
SaaS Testing Preparation
Assess Testing Needs
Understand Usage
Patterns
Prepare Test Plan and Test
Suite
Prepare Test Data
Prepare Test
Environment
Software Performance Assurance Confidential
SaaS Testing Areas
Secured SaaS
Application
Network testing
Infrastructure Testing
Application testing
Ensure SaaS testing does not damage the shared infrastructure
Software Performance Assurance Confidential
SaaS Functional Testing
• Perform Manual and Exploratory test (similar to traditional applications)
• Check Browser compatibility
• Perform regression
• SaaS regression must cover significant more functionality as single defect will impact all the customers
• Functional and regression test automation.
• This is important to comprehensive testing cost-effectively as above point indicates that, in case of SaaS, regression should cover more functionality.
• Test SaaS on cloud-based staging environment
• Testing on in-house environment and then directly deploying on cloud-based production environment may result in chaos
• Test in a multi-user environment where different users are performing different activities.
Software Performance Assurance Confidential
SaaS Security Testing
• Test application security• Cross Site scripting, SQL Injection, HTTP header injection
• Test network security• Secured communication, Backup and storage policies, Controlled
access to sites and servers
• Test for possible attacks
• Test roles and privileges in multi-tenant environment• Many roles and privilege levels
• Test all the upgrades to detect broken security, Ensure security patches are not missed
• Test data security / integrity among multiple tenants
Software Performance Assurance Confidential
SaaS Security Testing cont …
• Identify scenarios where application might be vulnerable
• Test PCI compliance
• Test cookie setting, SSL configuration, information leak, etc.
• Log errors, warnings and suspicious requests
• Tests tampering with the licensing mechanism; gaining unauthorized access to application data, editing registry entities or configuration files
Software Performance Assurance Confidential
SaaS Performance Testing
• Test performance extensively first time and for each upgrade
• SaaS is revenue generating and business impacting applications. Single performance issue will impact all your customers
• Test performance extensively as SaaS need to handle large number of users and we don‟t have luxury of re-booting or going down every once in a while
• Test load characteristics, ramp up rate, protocols, load origination,
• Establish ability to test performance of your SaaS isolated fashion so that network related issues can be identified and passed to network infrastructure and internet providers
• Test key metrics and SLAs
• Test performance for mobile users
Software Performance Assurance Confidential
In-Service Upgrade Testing
• Ability to upgrade (full release, patches) system with zero-to-minimum down time is essential
• Global users make this complicated as somewhere in the world someone might be accessing your application
• Thoroughly test in staging environment before perform upgrade ass this is one of the most complex features
Remote Accessibility and Usage
• Global access with acceptable response time
• 24 hours availability
Software Performance Assurance Confidential
Disaster Recovery
• Shutdown to bring up application quickly: server configuration; install and deploy software ; restore data
• Roll back to last known stable data
Localization
• Make sure it supports international characters as SaaS application can be accessed from anywhere in the world
• Optimize by supporting target market language
Software Performance Assurance Confidential
SaaS Testing
Test Tools
Performance
Functional
Software Performance Assurance Confidential
Traditional and Cloud-based Test Tools
Traditional Test Tools
QTP, WinRunner
Rational Robot
Rational Functional Tester
SilkTest
Selenium, STAFF, Fitness, Watir
LoadRunner, Rational Robot,
Rational Performance Tester
Silk Performer
Jmeter, Grinder, WebLoad,
OpenSTA
Cloud-based Test Tools
Gomez
PushToTest,
SOASTA
CloudTesting
uTest
Many others
Software Performance Assurance Confidential
Traditional Vs. Cloud-based Tools
Testing SaaS Using Traditional Tools
Many providers who provide SaaS testing using traditional
testing tools
Many open source matured tools
Many options for professional services by independent
service providers
Professional service options allow cheaper cost
Testing SaaS Using Cloud-based Tools
A few providers who provide SaaS testing using their own
cloud-based test tools
Limited open source test tools options
Captive professional service by test tools provider which is not even core for their business
Captive profession service is sold at very high rate
Software Performance Assurance Confidential
Conclusion
Software Performance Assurance Confidential
Conclusion
Customers buy each month – keep them happy
SaaS is only a delivering model, customers will not compromise on functionality, stability, security, and performance. They will not switch to SaaS it if not better than traditional systems.
Save time on support, maintenance, and upgrade, etc.
You need to pay more attention to stuff like accessibility, security, deployments. The way you earn reputation
Don‟t buy cloud-based test solution just because you are testing SaaS. In most cases, traditional test solutions work just fine and save your 30% –50% cost.