Saas Testing Techniques_ValueLabs
-
Upload
gautham-sreeramozu -
Category
Documents
-
view
139 -
download
0
Transcript of Saas Testing Techniques_ValueLabs
05/07/12 11:15 AM
Cloud TestingTo ensure a successful cloud computing strategy, you must be able to:
Manage performance and availability across the entire cloud service delivery chain
Monitor cloud application performance from the end-user perspective
Test your cloud applications prior to deployment
Monitor your cloud applications after they go into production
The performance and availability of cloud applications can have a dramatic impact on user adoption and
revenue. Monitoring and testing the performance of those applications requires uninterrupted visibility
across the entire application delivery chain i.e. from your data center, through the Internet and cloud
service providers to your end user’s own device and browser
Note: Traditional data center monitoring tools simply won’t work in the cloud. You need to monitor and
test your cloud applications from the only perspective that really matters: your end users.
Cloud Testing has four major objectives:
To assure the quality of cloud-based applications deployed in a cloud, including their functional
services, business processes, and system performance as well as scalability based on a set of
application-based system requirements in a cloud
To validate software as a service (SaaS) in a cloud environment, including software performance,
scalability, security and measurement based on certain economic scales and pre-defined SLAs.
To check the provided automatic cloud-based functional services, for example auto-provisioned
functions
To test cloud compatibility and inter-operation capability between SaaS and applications in a
cloud infrastructure, for example, checking the APIs of SaaS and their cloud connectivity to others
i.e. another SaaS with in the same/different cloud or end user interface .
Below Table shows the detailed tasks and comparative view among different parties:
Test Type Testing focuses Cloud/SaaS-Oriented
Testing inside a Cloud
Online Application-
Based
Testing on a Cloud
Cloud-Based Application
Testing over Clouds
Service
Function
Testing
GUI-based and API
based
service functions
Testing SaaS/Cloud
based
service functions
inside a cloud
Testing online-
based
application service
functions on a
cloud
Testing cloud-based
application service
functions over a cloud
infrastructure
Integration
Testing
SaaS interactions
and
Cloud connections
Vendor-specific
component and
service
integration inside a
private/public cloud
Integration
between online
clients and back-
end
servers on a cloud
- End-to-end application
integration over clouds
- Integration with legacy
systems over clouds
API and
Connectivity
Testing
API interfaces
and connectivity
protocols (HTTPS,
REST, SOAP, RMI)
SaaS/Cloud API &
connectivity testing in
a cloud
Testing user-
centered
service APIs and
connectivity on a
cloud
Testing application service
APIs and connectivity over
Clouds
Performance
& Scalability
Testing
Performance and
scalability based on
a
SLA
SaaS/Cloud
performance
and scalability testing
in a cloud based on
the
given SLA
User-oriented
application
performance and
scalability
testing on a cloud
End-to-end system-level
performance and
scalability
inside/on/over cloud based
on a given SLA
Security SaaS/Application SaaS/Cloud security User-oriented System-level end-to-end
Testing data,
processes,
functions, and
user privacy
features and user
privacy
in a cloud
security and
privacy on a cloud
security over clouds
Interoperability
&
Compatibility
Testing
Validate different
client interfaces
and
technologies and
diverse
compatibilities on
different platforms
and
browsers
Testing Cloud/
SaaS compatibility,
connectivity protocols
and UI/client
technologies inside a
cloud
Testing user-
centered
interoperability,
compatibility of
platforms/
OS/browsers, and
client
technologies on a
cloud
Testing application
compatibility, end-to-end
interoperability and
application connectivity to
legacy systems over
clouds
Regression
Testing
Changed &
impacted
SaaS/Cloud service
features and
related APIs/
connectivity
Cloud/SaaS-oriented
regression testing
inside
a cloud
User-centered re-
validation
on a cloud
End-to-end application
system regression over
clouds
1.3 Cloud Testing VS. Conventional Software Testing:
Internet-Based Software Testing
(i.e. Distributed/Web-Based System
Infrastructure)
Cloud-Based Software Testing
Primary
Testing
Objectives
- Assure the quality of system
functions and performance based on
the given specifications
- Check usability, compatibility,
interoperability.
- Assure the quality of functions and
performance of SaaS , Clouds,
and applications by leveraging a cloud
environment
- Assure the quality of cloud elasticity &
scalability based a SLA
Testing as a
service
- In-house internal software testing as
engineering tasks
Real-time on-demand testing service offered
by a third-party
- Online testing service based on a pre-
defined SLA
Testing and
Execution
Time
- Offline test execution in a test lab.
- Testing a product before its delivery
- On-demand test execution by third-parties;
- Online test execution in a public cloud;
- Offline test execution in a private cloud
Testing
Environment
- A pre-fixed and configured test
environment in a test lab. with
purchased hardware and/or software
- An open public test environment with diverse
computing resources
- A scalable private test environment in a test
lab.
Testing
Costs
- Required hardware costs and
software (license) costs
- Engineering costs in a test process
- Based on a pre-defined service-level-
agreement (SLA)
- TaaS and Cloud testing service costs (pay-
as-you-test)
- Engineering costs in SaaS/Cloud/application
vendors
Test
Simulation
- Simulated online user access
- Simulated online traffic data
- Virtual/online user access simulation
- Virtual/online traffic data simulation
Function
Validation
- Validating component functions and
system functions
as well as service features
- SaaS/Cloud service functions, end-to-end
application functions
- Leveraged functions with legacy systems
Integration
Testing
- Function-based integration
- Component-based integration
- SaaS-based integration in a cloud
- SaaS integration between clouds
- Architecture-based integration
- Interface/connection integration
- Application-oriented end-to-end integration
over clouds
- Enterprise-oriented application integration
between SaaS/Cloud
and with legacy systems
Security
Testing
Aim to the following targets:
Function-based security
features
User privacy
Client/server access security
Process access security
Data/message integrity
Aim to the following targets:
SaaS/Cloud security features,
including monitor and measurement
User privacy in diverse web clients
End-to-end application security over
clouds
SaaS/Cloud API and connectivity
security
Security testing with virtual/real-time
tests in a vendor’s cloud
Scalability
&
Performance
Testing
- Performed a fixed test environment
- Apply simulated user
access, ,messages, and test
data
- Online monitor and evaluation
- Performed in a scalable test environment
based on a SLA
- Apply both virtual and real-time online test
data
- Online monitor, validation, and measurement
Characteristics of SaaSSoftware as a Service (SaaS) is defined as software that is deployed over the internet… With SaaS, a
provider licenses an application to customers either as a service on demand, through a subscription, in a
“pay-as-you-go” model, or (increasingly) at no charge when there is opportunity to generate revenue
from streams other than the user, such as from advertisement or user list sales. SaaS applications are
designed for end-users, delivered over the web.
Following are the characteristics of SaaS:
Web access to commercial software
Software is managed from a central location
Software delivered in a “one to many” model
Users not required to handle software upgrades and patches
Application Programming Interfaces (APIs) allow integration between different pieces of software
SaaS Attributes:
Integration with External Applications: Simple Object Access Protocol (SOAP)-based Service
Oriented Architecture (SOA), Extract Transform Load (ETL) and On Line Analytical Processing (OLAP)
Application Programming Interfaces (APIs)
Manageability: Multi-tenant architecture to support clients from a single instance in order to reduce
the costs of infrastructure, hosting and management
Performance: Distributed data caching and code optimization tools for improving performance and
response time
Scalability: Meta-database and load balancing for scalability
Security: Multi-tiered, multi-layered, role-based security model. Typically improves due to centralization
of data, increased security-focused resources, etc., but raises concerns about loss of control over certain
sensitive data. Security is often as good as or better than traditional systems, in part because providers
are able to devote resources to solving security issues that many customers cannot afford. Providers
typically log accesses, but accessing the audit logs themselves can be difficult or impossible
Time-to-Market: Distributed Agile methodology and platform (GlobalLogic Velocity) to accelerate time-
to-market and provide shorter release cycles
Usability: AJAX-based APIs to provide interactive, professional-looking Graphical User Interfaces
(GUIs) supported by a dedicated team of usability experts
Compatibility: Portability experts to provide consistent support across a variety of browser platforms
Availability: 24/7 in-house support services to ensure uptime and continuous availability
Expertise on Open Source: Use of tools to reduce total cost of ownership
Reliability: Improves through the use of multiple redundant sites, which makes it suitable for business
continuity and disaster recovery. Nonetheless, most major cloud computing services have suffered
outages and IT and business managers are able to do little when they are affected.
Sustainability: Comes about through improved resource utilization, more efficient systems, and
carbon neutrality. Nonetheless, computers and associated infrastructure are major consumers of energy
Maintainability: Usually this includes System/Integration testing, Performance testing, and User
Acceptance testing cycles. The client must be confident the new version of the software works in their
environment AND with all of the interfacing applications.
The process is significantly streamlined with SaaS. The client is relieved of the burden of testing the new
software release in their environment, as the SaaS provider handles this for them.
Note: If your implementation of a vendor's SaaS application is integrated with one or more external
application (be they on-premise or SaaS), you must work closely with the vendor to ensure that no APIs
upon which your integrations depend are being deprecated as part of this release. If you are dependent
upon deprecated APIs, you must re-write your interfaces to the new API or your intra-SaaS application
business process will fail.
Interoperability: Cloud computing architectures are a heterogeneous blend of technologies and
platforms. The various software applications residing in the cloud do not exist in isolation. They must be
able to communicate and exchange information transparently, irrespective of the technologies used to
implement them. Thus, interoperability among the cloud SaaS is a relevant and significant issue in cloud
computing. Interoperability between SaaS is possible using Web standards and middleware (possibly
hosted in Cloud)
Adaptability: The entire way the software runs can be tailored for individual organizations
and to let any company define the hierarchies specific to them, and yet the overall software
works out of a single code base
Customizable: In Software-as-a-Service (SaaS) delivery model a vendor maintains a single
application instance, which is used by multiple tenants. However, due to changing business requirements
tenants expect customizations. Providing such customizations is trivial to retain tenants but a challenge to
the vendor due to multi-tenancy.
The table below lists all the Attributes of SaaS Application:
User experience
Usability
o Responsiveness
o Efficiency
o Performance
Data
State
State full
Stateless
Stability
o Personalizable
User interface
o Graphical
o Interactive
o Distributed
o Textural
o None
Interaction model
o Device
o SaaS
o Online
Application constraints
Database constraints
Persistence
Online/Offline
Structure
Unstructured
Indexed
Searchable
Transaction management
Security
Emergency hot fix or breach
management
Security procedures
Trust relationship with platform
Applications security model
Data flow
Malicious code
Access controls
Maintainability
Available skill sets
Language support (dev)
Application standards
Technology implementation
Application-code complexity and
volume
Configuration management
Operational management
Remote access
Identity
Cryptography
Auditing
Authentication/Authorization model
Flexible
Technology
Affordability
Resource cost
Development
Available skills
Software enhancements cost
Licensing
Postproduction hardware
Decommissioning
Initial hardware
Scalability
Replication
Caching
Pooling
Software load balancing
Scale out
Scale up
Hardware load balancing
Conformability
Auditable
Regulatory
Standards
Availability
Technology/Configuration/
Implementation to support availability
Uptime requirement
Portability Reliability
Cross-platform
Within platform
Configuration management
Startup and automatic recovery
System performance
Recovery procedures and methods
Load balancing
Fault tolerance
Distributability
Local
Geo-distributed
Interoperability
Communications and data usage
Integration impacts
Architecture compatibility
Ease integration (APIs)
Extensibility
Meta-model
Configurable
Reusability
Distributable and reusable
Modularity
Hierarchy
Code abstraction
How we test our SaaS QA Platform?
The cloud is defined by its service model, deployment model and usage: Mostly
cloud applications are based on SaaS. They are Software as a Service solutions that run completely on
cloud infrastructure and platforms. Hence testing of SaaS applications is completely different from testing
the traditional applications. They need to be tested on three levels: namely the infrastructure, the platform
and the application itself. The usage of standard services of applications also means a change for system
testing.
In principle, it’s not different from testing any other application; it requires merging different techniques
used in daily basis.
Cloud/SaaS-oriented testing: This type of testing activities usually is performed
inside a cloud by engineers of cloud/SaaS vendors. The primary objective is to assure the
quality of the provided service functions offered in a cloud (or a SaaS program). These
engineers must go through unit testing, integration, system function validation, regression
testing and cross platform (compatibility) testing, as well as performance and scalability
evaluation. Since clouds and SaaS usually provide certain service APIs and connectivity
interfaces to their customers, it is required task for engineers to validate these APIs and
connectivity in a cloud environment. In addition, testing cloud-based or SaaS-based security
services and functional features must be tested. Furthermore, performance testing and
scalability evaluation in a cloud is very important and critical to cloud/SaaS vendors because
this assures the quality of cloud elasticity to support SaaS and cloud services inside a cloud.
Testing Categories: Following are list of testing techniques that can be used to test SAAS
platform at different phases:
Test Category Testing techniques
Business Testing Manual/Automation functional Testing
Exploratory Testing
End to End business workflow testing
Manual/Automated regression testing
Data integration and data migration testing
Checklist validation
Security Testing Application Security Testing
Network Security testing
User Access and Roles testing
Data security integrity testing
Compliance testing
Identity Federation mechanism testing
Performance Testing Scalability testing
Volume Testing
Availability testing
Reliability testing
Load testing for single instance
Load testing in a instance loaded environment
Compatibility Testing Multi-browser and OS compatibility
Localization testing
Accessibility testing from remote locations
Internalization testing
Interface backward compatibility testing
Live Testing Disaster recovery testing
Statefull scenario testing
Live upgrade testing
Saas Attribute Testing Multi-tenancy isolation testing
Api Integration testing
Billing mechanism testing
Functional Testing Checklist:
For any application we make sure that the functionality works as expected. This is the standard functional
testing to validate if the app is doing what it is supposed to do.
Conduct rigorous Manual tests as per defined test plans, keeping the end user in mind
Conduct Exploratory tests based on existing or new test cases
Conduct Browser compatibility testing to check performance of the application on different web
browsers
Conduct Regression testing on every release, minor upgrade, an integration or data migration.
Automate Functional and Regression tests
Conduct tests in target environment – whether it is your data center or the Cloud.
Conduct reliability testing to find the total defects of the application and thus reduce the number of
failures, during real time deployment.
Multi-platform support/Compatibility Testing:We can use combination different browsers versions and operating systems to perform cross-platform testing.
We have to use browsers and operating systems majorly used by the end users to find issues that end user
may encounter with his/her browser/operating system combinations.
Load over different clouds:Application / system stability is a major factor as the user count is expected to be in multiples of hundreds.
SaaS based application needs to handle large amounts of users and we don’t have the luxury of re-booting or
going down once in a while. Conduct load testing under normal as well as peak load conditions in multiple
environments i.e. to determine the limits
Stress over different clouds:Due to the cloud characteristics, it is imperative to identify issues as system is tested to breaking points
maximum expected capacity or often beyond to 2x, 3x,nx expected usage. Pushing systems to maximum load
capacity and beyond i.e. Exceed Break points
Capacity Testing:Being hosted in a cloud environment it is prudent to determine maximum capacity for current or future
hardware, bandwidth or other needs or to validate that installed hardware and network will support expected
usage scenarios i.e. Plan for the Future
Conduct scalability tests to determine the capacity of the application to scale up or down as per
requirements
Availability Testing:Conduct availability testing for a planned period of time and 24/7
Volume Testing:Conduct volume testing for your data
Performance/Latency over different clouds:Measure response times and isolate issues related to specific steps or actions while system is subjected
to increasing load from different locations and multi user operations. Measuring response time variance
over load and time
Reliability/Soak Testing: Measuring performance degradation over longer periods at varying load levels i.e. Reliability over
time
Remote Access and Usage:We make sure that all users regardless if they come from the US, Holland, India, Argentina, or
Australia can work with the system with good response times.There are many emulators that can help
test this in your lab
Failover Testing/Disaster recovery & rollback procedures:Another testing task coming to us from the IT side of the house. Here we run 2 main scenarios:
o System down that needs to be brought up quickly (with the same machines or with new ones
that require installation and configuration)
o Rollbacks to the last known stable version, including data.
o Verify Redundancy
Application/Infrastructure security testing:The goal of this is to test the underlying infrastructure and security of the app:
Test the security of the SaaS application for typical web application security issues such as HTTP
header injection, Cross Site scripting (XSS), SQL Injection etc.
Test security of the network where SaaS application is being deployed
Test possible scenarios of security attacks/threats
Test the application with respect to access privileges with the corresponding job roles (especially in a
multi-tenant environment)
Test the security, integrity & accessibility of test data (especially in a multi-tenant environment )
Determine situations that could make the SaaS application vulnerable
Test compliance with Payment Card Industry Data Security Standard (PCI Compliance)
Maintain logs of security warnings, errors and requests from unreliable sources
Security Management: Dedicated or shared firewall, firewall management; VPNs; intrusion detection &
IDS management; systems software hardening; security audit / vulnerability scanning and notification.
Vulnerability AssessmentsWhat are your weaknesses? Our vulnerability scans are based on
a variety of compliance regulations such as:
Payment Card Industry (PCI) Data Security Standard
Health Insurance Portability and Accountability Act (HIPAA)
Sarbanes-Oxley (SOX)
Gramm-Leach-Bliley Act (GLBA)
Federal Information Security Management Act (FISMA)
Statement on Auditing Standards Number 70 (SAS70)
Managed Firewalls: Test with different types of firewall devices such as From Cisco to Checkpoint to Sonic
wall and implement the best practices
Patch Management: Each customer environment is different and we stay aware of how change will affect
it. We work with end users to establish a patching strategy that meets end user needs. Every patch is
analyzed. A risk assessment is made to be sure that your environment is not only safer by applying the patch
but won't be adversely affected.
Intrusion Detection Systems (IDS): We use both Network Intrusion Detection Systems (NIDS) and Host-
based Intrusion Detection Systems (HIDS) to ensure that the “bad guys” stay out. With systems powered by
Cisco, Checkpoint and OSSEC we perform log analysis, file integrity checking, policy monitoring, root kit
detection, real-time alerting and active response.
Security Policies:We know that security is a top priority. Whether it's user admin or system builds, our
documented procedures are built on years of experience and industry best practices for security and
compliance.
Incident Response:If you're learning the hard way at your own or a third-party location and experience an
attack, the vendor team can leap into action to control and repair the damage. We understand how to contain
the breach, develop an action plan to systematically verify integrity of your network and all your devices, then
recommend and help implement solutions to protect from future attacks.
GRC (Governance Risk Compliance) testing: Devise a unique comprehensive testing strategy for
compliance with standards like PCI and government regulations
Scalability over different clouds: This assures the quality of cloud elasticity to support SaaS and
cloud services inside a cloud.
Operational Testing:This area is intended for the operations team whose objective is to make sure the apps are working fine, and
take care of customer service & billing. Usually, there are tools that are built as part of the product which help
the operational team members to monitor, track and analyze for issues. The areas to look for:
Application, Services, App Server, Platforms (OS), Databases and Data Center Level
Logs/Alerts/Warnings/Errors for functionality and performance.
Billing and Customer Support Tools, especially for integration
Integration and API Testing:Success of SaaS apps lie in how well you have thought of scenarios where third party developers can build
their own apps using your APIs, and add value to your product. So testing all the APIs for functionality,
security, usability, performance and completeness of documentation is critical to make them successful:
SaaS based application interactions and cloud connections with different client interfaces, database
servers
SaaS based integration in a Cloud
SaaS integration between Clouds
Application oriented end to end integration over clouds
Enterprise oriented application integration between Saas/Cloud and with Legacy systems
Usability over different clouds: Test the Responsiveness, efficiency, Performance and
Personalizable
Cloud based Unit testing: Unit testing on different clouds
Cloud-based application integration: Application integration testing on different clouds
End-to-end system function testing: System testing on different Clouds
Cloud based System Integration testing: SaaS usually provide certain service APIs and
connectivity interfaces to their customers, it is required task for engineers to validate these APIs and
connectivity in a cloud environment.
Saas based application interactions and cloud connections with different API interfaces and
connectivity protocols(HTTPS, REST, SOAP and RMI).
Live updates and deployments: Here is something we think about in regular applications:
How do you deploy the system while it is still running, and if needed how do you minimize down-time for
the users? This is something that is developed and handled by our IT department, but since the delivery
and update of the product is part of the overall user experience we test it constantly.
Internationalization/I18N: Since our platform is used by people around the world we make sure
that we support the use International Characters.
Use Case Scenarios: Ethernet Fabric by validating QoS of high density 10/40/100Gb Ethernet top of rack, end of row
and WAN optimization switches
Storage Networks by ensuring QoS of Fibre Channel and Fiber Channel over Ethernet storage
networking devices and storage systems
Application Networking by testing QoS of Firewalls, IPS, WAN accelerators, Proxy Servers, SSL
VPNs, etc
Cloud Virtualization by benchmarking QoS of virtual switching and virtual appliances from blade
servers to any point in the Ethernet and Storage fabric
Exchange documents, such as purchase orders, with any business partner over the Web with
B2B integration technologies, while eliminating the costs of proprietary EDI solutions.
Automate any mission-critical process such as Order-to-Cash, giving you more visibility into your
business, whether you represent marketing, sales, IT or support.
Interoperability Between Local and Global ADC Functions:Cloud balancing is based on making routing decisions based on a combination of local and global variables.
This requires interoperability between local and global ADC functions. Standards-based APIs may eventually
emerge that will facilitate the cross-vendor exchange of cloud balancing variables. In the mean time, in those
situations in which multiple ADC vendors are involved, IT organizations will need to take advantage of the
APIs supported by each vendor in order to achieve an integrated set of variables to use to make routing
decisions. Another option that IT organizations have is to adopt a single vendor strategy for both local and
global ADC functions. The feasibility of implementing a single vendor strategy across the enterprise and one
or more IaaS providers is enhanced if the ADC is available in a virtual appliance form factor.
Focuses on different client interfaces and connecting to legacy systems
Synchronizing Data between Cloud Sites:In order for an application to be executed at the data center that is selected by the cloud balancing
system, the target server instance must have access to the relevant data. In some cases, the data can be
accessed from a single central repository. In other cases, the data needs to co-located with the
application. The co-location of data can be achieved by migrating the data to the appropriate data center,
a task that typically requires highly effective optimization techniques. In addition, if the data is replicated
for simultaneous use at multiple cloud locations, the data needs to be synchronized via active-active
storage replication, which is highly sensitive to WAN latency.
Challenges:Most organizations report impediments to SaaS testing like – short notice periods for QA notification, frequent
testing of live upgrades, short validation cycle times, impact on multiple subscriber organizations, privacy
violations, errors due to rapid addition of new features, time taken for data migration, concerns over data
security & integrity etc. cloud the obvious benefits of SaaS testing.
1. Handling Changes through Frequent Releases: Every time the Application is upgraded,
the users have to understand the impact of the change, validate it against the existing system &
ensure that the impact on the existing features of the application is minimal. Managing and executing
all these activities within a short time span (1-2 weeks) is challenging. When SaaS upgrades involve
interface upgrading, compatibility and integration issues across old and new interfaces crop up for the
subscribers. Live upgrades being simulated or tested on the SaaS application impedes the activity of
the existing users.
2. Security Testing: Maintaining data security, accessibility & integrity on a single SaaS application
across multiple tenants. To understand individual privacy requirements, privilege levels, behavioral
patterns and provide adequate privacy to the data can be a daunting task. Cloud computing security
challenges fall into three broad categories:
Data Protection: Securing your data both at rest and in transit
User Authentication: Limiting access to data and monitoring who accesses the data
Disaster and Data Breach Contingency Planning
3. Integration Challenges: When subscribers integrate their internal enterprise applications with
SaaS, inbound and outbound data integration validations from client networks to the SaaS providers
is needed. In such cases it is very difficult to conduct thorough validation simultaneously ensuring
100% data security and privacy
4. Data Migration Issues: Data migration across different SaaS applications or from other
applications to SaaS can be challenging in terms of time taken for understanding the requirements
and the exhaustive integration validation processes
5. Licensing: The SaaS app licensing may vary by functionality, usage (such as volume of
transactions or amount of specific data) or # of named/concurrent users. All this needs to be tested
across every release.
6. Performance testing: Successfully modeling the most-used business transactions, application
usage and user mix may require greater diligence than an on-premise application.
Risks: Accountability and Data Risk
User Identity Federation
Regularity Compliance
Business continuity and Resiliency
User Privacy & Secondary Usage of Data
Service & Data Integration
Multi-tenancy & Physical Security
Incidence Analysis & Forensics
Infrastructure Security
Non-production Environment Exposure
1. Accountability: In traditional data center, the owning organization(End user) is accountable for
security at all layers i.e. Application/ Database/Computing/Network/Storage layers. You can outsource
hosted services but you cannot outsource accountability.
In a cloud, who is accountable for security at these layers?
Data can be stored anywhere at different geographical locations:
How sensitive is the data? (Informal blogs, public network sharing posts, public news, New group
messages, Health Records, Criminal Records, Credit History and Payroll)
Who owns the data?
Is data encrypted single Vs multiple keys
Data Mitigation:
Logical isolation of the data of multiple consumers
Provider fully destroys deleted data
Multiple encryption keys
2. User Identity Federation:
Security Risks
Managing Identities across multiple providers
Less control over user lifecycle (off-boarding)
User experience
Mitigations
Federated Identity
Auth for backend integrations
Tighter user provisioning controls
3. Regulatory Compliance:
Data that is perceived to be secure in one country may not be perceived secure in another
country/region. European Union (EU) has very strict privacy laws and hence data stored in US may
not comply with those EU laws (US Patriot Act allows federal agencies limitless powers to access any
corporate data etc)
Lack of transparency in the underlying implementations makes it difficult for data owners to
demonstrate compliance (SOX/HIPAA etc.)
Lack of consistent standards and requirements for global regulatory compliance –data governance
can no longer be viewed from a point-to-point data flow perspective but rather a multi-point to multi-
point.
Mitigations
Apply risk management framework, case-by-case basis
Define data protection requirements and SLAs
Provider / Consumer agreement to a pre-defined RACI model
4. Business Continuity and Resiliency:
Lack of know-how and capabilities need
Cloud provider may be acquired by a consumer’s competitor
Monetary losses due to an outage
Mitigations
Contract defines Recovery Time Objectives and monetary penalty for downtime
Cloud provider’s Business Continuity program certified to standard such as BS 25999
5. User Privacy & Secondary Usage of Data:
Users Providers
Privacy of my data:
Address, Email, (Personally Identifiable
Information)
Health, personal financial info
Keep Revenue Up/ Cost Down:
Push out the liabilities to user via Privacy
and Acceptable Use Policy
Build Additional Services on users behavior
Personal Details (email, IMs,….) (targeted advertisements) e.g. Google
Email, banner adv.
Do minimal to achieve compliance
Keep their social applications more open
(increased adoption)
User personal data mined or used (sold) without consent-Targeted Advertisements, third
parties
User Privacy data transferred across jurisdictional borders
No opt out features for user (user can not delete data)
Lack of individual control on ensuring appropriate usage, sharing and protection of their
personal information.
Law Obligation for providers
Key escrows to law agencies
Subpoena
Mitigations
Policy Enactment
o Privacy and Acceptable Usage
o Consent (Opt In / Opt Out)
o Policy on Secondary Usage
De-identification of personal Information
Encrypted storage
Terms of Service with providers
o Responsibility on compliance
o Geographical affinity
6. Service and Data Integration:
Data traverses through the internet between end users and cloud data centers.
How secure the integrations are?
Mitigations
Encryption keys single Vs multiple
Secured protocols
7. Multi-tenancy and Physical Security:
Security Risks
Inadequate Logical Separations
Co-mingled Tenant Data
Malicious or Ignorant Tenants
Cross-Tenant Attacks
Side channel Attacks
Scanning other tenants
DoS
Shared Service-single point of failures
Uncoordinated Change Controls and
Misconfigs
Performance Risks
WordpressOutage June 2010
100sof tenants (CNN,..) down in multi-
tenant environment.
Uncoordinated Change in database
Mitigations
Architecting for Multi-Tenancy
Data Encryption (per tenant key management)
Controlled and coordinated Change Management
Transparency/Audit-ability of Administrative Access
Regular Third Party Assessments
Virtual Private Cloud (VPC)
8. Incidence Analysis & Forensic Support:
Complex integration and dynamics in cloud computing present significant challenges to timely
diagnosis and resolution of incidents such as:
Malware detection and
Immediate intrusion response to mitigate the impact.
Implications to Traditional Forensics? (Seizing equipment and analysis on media/data recovered)
International differences in relevant regulations …
Mitigations
Comprehensive logging
Without compromising Performance
Dedicated Forensic VMImages
Infrastructure Security:Malicious parties are actively scanning the internet for Vulnerable Applications or Services such as:
Active Unused Ports
Default Passwords
Default Configurations
Data
Mitigations
Segregation of duties and role based administrative privileges
Third party audits and app vulnerability assessments
Tiered architecture with appropriate security controls between them
Hardening(Networks, OS, Apps)
9. Non-production Environment Exposure:
Non-Production Environments are for design, development, and test activities internally within an
organization:
Typical non-prod environment use generic authentication credentials
Security flaws
Data copied to non-prod from its production equivalent
High risk of an unauthorized user getting access to the non production environment
Mitigations
Use multi layers of authentication
Non-prod data is not identical to production
Don’t use cloud for developing a highly sensitive app in the cloud
Over Coming Challenges of Saas Testing:
Challenges Mitigation PlanTesting frequent SaaS upgrades – Short notice
period
(1-2 weeks) for a QA notification to validate the
application
The use of automation tools for building regression suites
brings in business value and helps quickly validate the
impact of upgrades
Business knowledge for effective testing of
configurable and non-configurable components
Gain comprehensive and competent knowledge on the
configurable and non-configurable components of SaaS
applications
Any non-configurable upgrade/change to the application
will need to be assessed thoroughly since this will have an
impact on all SaaS subscribers
Though the configurable upgrade/change would not
impact every client, it is advised to validate the impact of
these changes as well
Validating interface compatibility The backward compatibility of a SaaS interface needs to
be validated to ensure that the organizations do not have
to make any changes at their end, and can continue using
SaaS applications as before
Compliance with government regulations and
other standards
Devise a unique comprehensive testing strategy for
compliance with standards like PCI and government
regulations
Data security and privacy Validation of strong encryptions is needed to ensure data
security
Data security and privacy would need to be thoroughly
validated amongst multiple tenant scenarios to ensure
that there are no loop holes
Testing access controls, multi-privileges for securityPerform access control and multi-privilege tests with users
that have varied roles, different privileges and are
executing unique activities (simulating real life usage
scenarios)
Data integration - inbound & outbound Test data transfers between an organization’s network
and SaaS applications.
- Also, measure, compare and validate the performance of
data migrations between SaaS applications and an
organization’s network
Simulating live upgrade testing Live upgrade tests should be carried out in cloud based
pre-production environments
Use automation tools to simulate the scenario of multiple
concurrent users logged on to a current SaaS version.
Conduct live upgrades in cloud based environments
Use automation tools to validate the accuracy of the
upgrade
Optimization of testing that is common to the
impacted core and non-core areas of SaaS when
getting customized
Create a test strategy to test the core product of SaaS
Create a standard suite of automated test cases to
validate the core SaaS product
Create a map/grid of the core and the non-core areas of
the SaaS application that are most likely to be impacted
during customization
Run a regression suite selecting the tests associated with
the impacted areas
Data migration from the existing system to SaaS
application
Identify the different data sources in the existing system
that need to be migrated to the SaaS application.
Select tools that will help in the data migration and in the
post migration validation
Frequent releases of feature rich SaaS
applications increases the time taken for testing,
owing to the significant number of pages to be
Create an automated test library for SaaS applications
that help reduce the associated testing effort that comes
with each frequent release
covered
Rapid addition of new features to the core SaaS
product to meet new customer demands and to
stay competitive. However, every change is a
potential security bug/ performance issue
Formulate a comprehensive strategy for testing the SaaS
applications with test tools that cover functional,
performance and security requirements
Maintain a test repository of results, performance
benchmarks and access privilege grids, which would
facilitate faster validation
Execute comprehensive tests with automated tools that
cover the functional and nonfunctional requirements.
Conduct a continual impact analysis of requirements and
regularly update the test library to help minimize risks.
Implementation of Saas Testing:Now, let’s take a look at the SaaS testing process itself. SaaS testing begins with assessing the
functional and non-functional requirements for the SaaS application, including business, operational
and non-functional needs. Once this is done, the focus then moves into understanding the usage
pattern of the application. This particular set factors in the variations due to geographies, peak periods
and network latencies across regions.
A test plan would need to be developed to include all components of the SaaS application. The plan
would also have details on how these components would be tested and the resources needed to carry
out the same. Once the test plan is approved, the QA team would prepare test cases, test suites and
eventually get the test data ready. The QA environment is then validated for its preparedness for SaaS
Testing. After the assessments confirm the preparedness, test data is populated in the QA
environment through data migration from the existing system.
Then the test team focuses on the automated test suite generation for functional and non-functional
validations. This would be followed by test execution, reporting, publishing and finally culminate with
the issuance of the SaaS readiness certification. See figure 3 (The SaaS Testing Process) to get details
on all steps and processes required for ensuring a systematic and successful SaaS Testing.
Assess the functional & non-functional test requirements
Understand the usage patterns
Test strategy & plan
Prepare test case & suite
Prepare test environment
Populate test data
Generate automated test suite for functional & non-functional test requirements
Execute SaaS testing, report & publish
SaaS Certification
Benefits of Saas TestingThere are multiple benefits that SaaS testing delivers to organizations:
Reduces effort required and go-to-market time associated in procurement, upgrades, renewals,
contracts, maintenance and deployment
Lowers costs associated with test tools, test environments, maintenance and upgrades.
Helps focus on the SaaS application configuration rather than on provisioning for the
application and associated infrastructure requirements
Significantly reduces CAPEX associated with setting up of environment for SaaS application,
helping convert the same into OPEX
Reduces shelf ware risk of SaaS application and testing tools associated with the validation of
the application
Testing costs are reduced by almost one third as the need to test client server installations,
multi-platform backend support, multiple versions of upgrades and backward compatibility is
completely eliminated
Using SaaS testing tools are not system or machine dependent. For example, any local
machine connected to a cloud network can be used for performance testing of the SaaS
application This helps save effort and overhead expenses associated with the installation,
configuration and maintenance of additional machines for enabling SaaS testing tools
Conclusion:
SaaS testing focuses on ensuring high quality across the application, its cloud characteristics and SaaS
attributes. It also includes testing for security, privacy, accessibility and standards compliance as well.
A thorough understanding of the SaaS application, the customer specific implementation, components
that are configurable and non-configurable and how any change or upgrade would impact the
application is absolutely needed to ensure a successful SaaS application testing. The automated
validation of the functional and non-functional requirements of the SaaS application helps shorten the
release cycle of frequent SaaS application upgrades and releases. The data integration/ migration
pertaining to SaaS applications would also need thorough validation. The key to successful SaaS
testing is putting together the right test strategy, automating the tests for functional and non-
functional requirements and leveraging best practices that would help maximize the investments in
SaaS and in turn help the organization achieve the intended business outcome.