1

Performance & Security Testing – Critical today in the Indian Testing Industry

2

Agenda

» Performance - Issues & Costs

» Leveraging Performance Management

» What AppLabs Offers

Performance -Issues & Costs

4

Corporate Performance

Corporate performance is about Effort spent vs. Value generated

Performance services are about Optimizing effort and maximizing value

Performance concerns are about TechnologyBusiness Users

Performance is propelled by StabilityScalability Speed

Performance management is central to corporate performanceBottom-line is

5

Factors influencing Security

6

Should Performance be an Issue?

Dipping sales

Diminishing subscriptions

Dipping advertisement revenues

Direct costs of fixing errors

Compensatory claims

Outsourcing costs

Performance failures reduce revenues

Performance failures increases expenses

Squeezing ROI

Anything that adversely impacts ROI is an issue – Performance is an issue

7

Performance Failure Risks

Performance issues

Infrastructure/ bandwidth bottlenecks

Crashes & breaches

High response time

Under/Over utilization of resources

Downtimes

Post production problem resolution

Resolution time

Cost implications

Influence on end-User

Extremely high Very high High

8

Security Failure Risks

Security issues

Information Loss or Theft

Breaches

Un authorized access

Compliance

Downtimes

Post production problem resolution

Resolution time

Cost implications

Influence on end-User

Extremely high Very high High

9

Performance failures – How real are the risks?

10

Case of a Public Sector Agency

» The Identity and Passport Service (IPS), a UK government agency went live with its electronic passport application system (EPA2)

» Problem: Performance issues jammed the system building up application backlogs

» Reason: Low weight age placed on the system performance while going live

» Costs:

» IPS had written off £5.5m of software development costs

» Outsourcing service costs shot up from £48.9m in 2005-06 to £86.6m in 2006-07

» Post debacle measures: IPS is doubling the time for User Acceptance Testing

Source: Compterworld

11

Case of an ISV

» The Independent Software Vendor (ISV) deployed an application to support end-user technology for a telecommunications service provider

» Problem: Server crash, service request refusals, connection drops and slow response of application during peak hours

» Reasons: Low weightage placed on the system performance while going live, improper server configuration, underutilized server resources

» Costs:

» Business implications – loss of sales, subscriptions and revenues

» Outsourcing service costs for performance testing

» Post debacle measures:

» Performance testing was outsourced

» Root-cause analysis to address connection drop issues

» Resolution of the server level performance of the application

» Monitoring for CPU, Memory, Network, and Database performance

12

User’s View on Response Times

Response time User’s view

<1 Second User feels that the system is reacting instantaneously

< 2 Seconds User experiences a slight delay but is still focused on the website

< 5 to 6 Seconds

(Static Web sites) Maximum time a user focuses on a web site, reaches the distract zone

<6 to 8 Seconds

(Dynamic Web sites)

Maximum time a user focuses on a web site, reaches the distract zone

> 10 Seconds User is most likely to be distracted from the website and looses interest

13

» Privilege escalation

» Successfully exploiting this vulnerability could result in an agent obtaining access to another agents customers’ details

» Authentication

» Attackers can access the application by guessing or brute force if there is no proper authentication management built in

» SSO (Single Sign On) applications pose high risks

» Injection flaws » Each form used for the application may/may not be validated for expected input

» Improper input validation can result in attacks

» Cross Site Scripting, SQL injection, HTTP response splitting and Cross Site Request Forgery

» Obsolete/unnecessary services

» Can be exploited easily

» Configuration Management

» Needs to reviewed to ensure that there is no risk

Security Risks

Common security risks in web applications

14

What Impedes Performance Management

» Perception that a powerful processor can resolve client server performance issues

» Absence of scientific approach - No real-time metrics to measure performance

» Performance testing still an outsider in SDLC and is always a last–minute activity

» Managing performance testing with inferior tools and lack of resource allocation

» Apathy towards training manpower to manage performance testing

» Lack of exposure to best of breed services

Lack of holistic view to Performance Management

15

What Impedes Security Management

» Perception that the network is safe with sufficient controls like firewall, antivirus, antispam, IDS/IPS

» Absence of scientific approach - No real-time metrics to measure security

» Security testing still an outsider in SDLC and is always a last–minute activity

» Managing security testing with inferior tools and lack of resource allocation

» Apathy towards training manpower to manage security testing

» Lack of exposure to best of breed services

Lack of holistic view to Security Management

17

Drivers for Performance Management

Business Drivers

» Rising usage of internet

» Business growth increasing in relatively shorter periods of time

» Explosion of E-Business applications across industries

» Stock trading

» Insurance services

» Ticket reservations

» Retail purchases

» Unexpected spikes in load

» Better performance for better ROI

Business Drivers

» Rising usage of internet

» Business growth increasing in relatively shorter periods of time

» Explosion of E-Business applications across industries

» Stock trading

» Insurance services

» Ticket reservations

» Retail purchases

» Unexpected spikes in load

» Better performance for better ROI

Industry is now proactive for Performance Management in IT

Driving forces for

PerformanceManagement

Technology Drivers

» Complexity of Internet applications

» Scalability of applications

» Cross platform compatibility issues

» Globalization of applications

» SOA based implementations

18

Application Security Drivers1.

Inte

rnet

Em

erge

nce

2.

Evolu

tion

of n

ew

tech

nolo

gies

1.

Use of Fram

eworks

2.D

iversity in Application types

1. Outsourcing2. Skill set

19

Drivers for the Indian Market

» Active internet users in India reaches 32 million and is growing at 35-40%

» PC-based internet access and mobile-based internet access are driving the

growth of E-Commerce industry

» Online payment systems are getting safer leading to online shopping from

smaller cities to rise

» Web based software systems are getting popular and pervasive across

verticals

» Online sales peak during the festive seasons and occasions like valentines day,

new year, friendship day etc. Younger generations prefer buying and sending

gifts online

Exponential growth in E-Commerce makes performance management a key winning factor

Source:Internet and Mobile Association of India (IAMAI)

20

Digital Downloads

» Rise in mobile subscribers & digital downloads

» New download facilities - ring tones, games, music etc.

Travel industry

» Increase in the number of travelers & travels per traveler

» Annual growth in the number of travelers is expected to increase five-fold, from 300,000 to 1.5 million

eTailing

» Expected to rise by 30%

» Physical cost elimination is giving buyers & sellers best deals

» Competition is forcing down the value of online products while the number of online transactions is continuously rising

Online Classifieds

» Users have access to large databases

» Rise in sales of exclusive videos, research data, reports

Indian E-Commerce Market

Source:Internet and Mobile Association of India (IAMAI)

Indian B2C E-Commerce industry 2007-2008 (estimate) - INR 9210

cr.

Indian B2C E-Commerce industry is expected to

grow at 30%

21

Industry Trends & Facts

Key survey findings:

Source: CSI Survey 2007; The 12th Annual Computer Crime and Security Survey

» The average annual loss reported in 2007 shot up to $350,424 from

$168,000 the previous year (2006).

» 18% of the respondents suffered a “targeted attack”; defined as a malware

attack aimed exclusively at their organization

» Financial fraud overtook virus attacks as the source of the greatest

financial losses.

» Virus attacks moved to second place: first time in the last seven years

22

Security Attacks – Industry Data

Financial Fraud has overtaken Virus – This has happened for the first time

23

Financial services, though certainly keepers of great monetaryassets, are also typically well protected in comparison to other industries; they account for 14 percent of breaches.

The type of asset compromised most frequently is without doubt online data. Compromises to online data repositorieswere seen in more cases than all other asset classes combined by a ratio of nearly five to one.

Security Breaches – Industry Data

24

Why Application Security Defects Matter

» Frequent

• 3 out of 4 business websites are vulnerable to attack (Gartner)

» Pervasive

• 75% of hacks occur at the Application level (Gartner)

» Undetected

• QA testing tools not designed to detect security defects in applications

• Manual patching - reactive, never ending, time consuming and expensive

» Dangerous

• When exploited, security defects destroy company value and customer trust

32% Hijack Session/

Identity Theft

11% e-Shoplifting

21% Full Control and Access to

Information

2% Delete Web Site

27% Privacy Breach

7% Modify Information

25

Bad Business

• On average, there are 5 to 15 defects in every 1,000 lines of code US Dept. of Defense and the Software Engineering Institute

Slow Business

• It takes 75 minutes on average to track down one defect. Fixing one of these defects takes 2 to 9 hours each

5 Year Pentagon Study

• Researching each of the 4,200 vulnerabilities published by CERT last year for 10 minutes would have required 1 staffer to research for 17.5 full workweeks or 700 hours

Intel White paper, CERT, ICSA Labs

Loss of Business

• A company with 1,000 servers can spend $300,000 to test & deploy a patch; most companies deploy several patches a week

Gartner Group

Impact of Security Defects

26

Industry Views

Gartner predictions

» SOA implementation to increase application failure due to unplanned downtimes to 60% by 2010

» SOA applications require performance management for end-to-end performance and capacity planning for demand fluctuations

Forrester views

» It costs less to correct capacity and performance issues before deployment

» Production-like environment for performance testing is critical to address the threat to the production environment

Aberdeen best-in-class performance criteria include

» Reduced time-to-information and time-to-decision/ action

» Customer satisfaction relating to speed, accuracy, data access and availability for end-users

» Customer retention

Testing Market

» Global software testing market is $13 billion (IDC)

» Outsourced testing services market is approximately $6.1 billion (Dataquest)

» Global market opportunity for Indian software testing companies to reach $8 billion by 2010 (Gartner)

» The requirement for software testing professionals in India is estimated to reach 2 lakh by 2010 (CIOL)

Leveraging Performance Management

28

Performance is Integral for Quality

Locates and fixes errors in an operational

program.

Product meets specifications and fulfills user’s objectives.

Product has access to required software or data

Product performs its intended function with required precision.

Product performs with optimum use of resources

Product performs its intended function for

intended number of users

Product couples well with another system(s) Correctness

Reliability

Efficiency

Integrity

Maintainability

Testability

Interoperability

Performance management ensures quality product

29

When to start Security Testing

Security should be a process that should be implemented throughout the software development life cycle

30

Application Readiness for Security Testing

Targeted features implemented

Functionality Testing is complete

Environment replicates production

Hardening is complete

31

When to Start Performance Testing

Timely performance

testing reduces the

cost and effort

Pre-production Stage

Test Driven Approach

Post-Production Monitoring

» Scalability issues of the core system

» Optimization of code & configurations

» Hardware sizing & benchmark for target loads

» Infrastructure/ network/ hardware level changes

» Improvement in response times

» Peace of mind

» Further optimization of code & configurations

» Application level bottlenecks

» Configuration changes

Benefits increase as performance testing spreads across the stages

Co

st/

Eff

ort

Benefits

32

Moving Ahead with Performance

» Product functionality will soon be a worn-out strategy for differentiation

» Performance cannot be ensured in isolation

» Performance will be the buzz word to gain competitive edge

» Resolving enterprise level performance issues demands clear communication within IT groups – managers, developers, testers, operations team, database administrators and network administrators

» Communication should be on test agendas; performance data, performance results, application upgrades become critical

» Performance management optimizes resources

» Automation aids in faster problems identification and resolution

» Tight IT budgets may not warrant new hardware purchases

» Performance cannot be taken as a final step in SDLC

» Performance criteria will be part of design principles even prior to development

» Scalability testing should cover each component and at every application layer before final integration

33

Moving Ahead with Performance

» Thinking ahead helps

» End-user experience is

not all

» Faster response time is

essential but there’s

more to it

» Know your historic traffic numbers and stress the application 3-4 times that load/ future expected load

» Stress test for peak loads – new product launch, promotional offers, seasonal offers etc.

» Checking further into the system for critical performance bottlenecks may avoid server crashes only few days after production

» Data integrity is a must. Delivering irrelevant/ erroneous data is more damaging than slow response times

» In an agile test approach where testing moves into the SDLC, changes become timely and affordable

» Economize testing especially scalability, by sharing test resources

» It is about time and cost

34

Third-Party Solution Providers

» Better equipped with the hardware and software resources required for peak load testing

» Interpreting performance data is not easy nor is the prescription of solutions for the same

» Testing can be done remotely at low-costs

» Have the expertise to test with an internationalization perspective as companies need to provide global access to their web applications

» Load generation from across the globe

» Risk reduction by simulating concurrent internet users as realistically as possible

» External solution providers complement the internal performance management effectively

The question is not whether performance is tested, but how well it is being done.

AppLabs Performance Services

36

AppLabs Offerings

Performance Testing Tools Expertise

Technology Expertise

Testing Expertise

» Databases: Oracle, SQL Server, Sybase, PostGreSOL

» OS: Solaris, OS/400, AIX, Linux, NT, Windows, HP VMS, HP Unix, MVS

» Operations Support: BEA WebLogic, IBM WebSphere, IIS, Apache Tomcat, iPlanet

» J2EE, .NET technologies

» Web, client-server

» Streaming media

» Hardware devices

» Wireless apps

» AppMeter (AppLabs)

» WebLoad (RadView)

» LoadRunner (HP/Mercury)

» Silk Performer (Borland)

» QA load (Compuware)

AppLabs CoE

» Core testing

» Performance

» Security

» Tools & Automation

» Consulting

37

Case Study – Retail

» Business Challenge» To ascertain the impact of a security breach on their application,

Pantaloon engaged AppLabs to carry out Web Application Penetration Testing on the servers exposed to the Internet.

» Solution» Comprehensive application security checks were conducted to

establish the applications susceptibility to hack attacks. » This phase complied with Open Web Application Security Project

standards and vulnerabilities identified through research by AppLabs’ Security Center of Excellence.

» These tests were run using a combination of automated and manual test tools.

» The engagement concluded with the delivery of a comprehensive assessment report with severity ratings for the vulnerabilities, alongside detailed descriptions and recommendations on how to address them.

37

38

Case Study – Retail

» Key Benefits

» The intense level of security testing performed aided the client in maintaining information integrity and confidentiality of such sensitive information;

» The test results supported the client in understanding business and technical risks to help fortify the security policy;

» In working with an independent testing organization, the client’s customers would be more confident that the web site is secure and online transactions are safe – providing it with a differentiator in the market;

» Implementation of the prioritized action plan which detailed the timelines to fix the different severity level vulnerabilities has enhanced the overall security posture of the application.

38

39

Case study – Financial Services

Client: Leading service provider for financial services, and has launched a trading application

Business challenge:

» Testing the trading application for stability with a load of 10,000 concurrent users

» Ensure the site responds to all users with minimal response time and optimal server utilization

Solution:

» Tested for receiving ticker responses from ticker plant servers

» Constructed scripts using ideal mix of loads

» Highlighted under-configuration

» Identified uneven resource utilization across web servers

» Identified load balancer issues - load was not split across the servers leading to high response

times at 5000 user load

» Ensure scalability of the application for 10,000 concurrent users

Benefits:

» Load balancer issues resolved by ideal server sizing for 10,000 users

» Resolved server (IIS & SQL) related scalability and sustainability issues

» Benchmark results for “Orders per second” achievable for 10,000 concurrent users was provided

40

Case study – Consumer Product

Client: India's largest consumer products company, migrating its consumer products warehousing

management (MFG PRO) to SAP

Business challenge:

» Check the sustenance of hardware capacity with the load of 60 depots configurations from the

existing 2 depots

» Identify scalability of the server capacity for database and application servers of SAP for 60

depots

Solution:

» Monitored the SAP implementation - server setup for performance; capacity utilization for the

current loads (existing 2 warehouses)

» Developed a formula based on number of users from each depot, connection speed from VSAT to

dial-up, load generated from the existing depots in terms of throughput and overhead

» Provided the formula for visualized utilization in case of adding 58 more depots

Benefits:

» Hardware resizing for the SAP implementation roll out in 60 warehouses

» Capacity formula & capacity report based on the analysis of the current implementation