Performance Testing: Putting Cloud Customers Back in the Driver’s Seat

Performance Testing:

Putting Cloud Customers Back in the Driver’s Seat

Imad Mouline, CTO, Gomez Twitter: @imadmouline

Agenda

A brief overview of the performance problem in the cloud

Real-world data: how have cloud providers performed so far?

Optimization opportunities in the cloud

Takeaways

Q&A

Typical Web Application Delivery

The Web Application Delivery Chain

MajorISP

Local ISP

Mobile Carrier

Internet

Content DeliveryNetworks

3rd Party/Cloud Services

Browsers and devices Users

Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

Traditional zone of control

MajorISP

Local ISP

Mobile Carrier

Internet



Browsers and devices

Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

Moving Web Applications to the Cloud: Benefits


Users


Elastic and scalableFocus on my business

while someone manages infrastructure

Cloud

MajorISP

Local ISP

Mobile Carrier

Internet



Browsers and devices UsersUsers

Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

The Problem: The Cloud Creates Performance Concerns



Cloud

Cloud is opaque• Loss of visibility and control• Traditional tools don’t apply

The Answer: Adopt an “Outside-In” User Point of View


Users

MajorISP

Local ISP

Mobile Carrier

Internet




Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

Users


Full understanding of performance from user perspective

“Outside-in” customer point of view

MajorISP

Local ISP

Mobile Carrier

Internet




Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

My users

Other users

Other users

Other users

The Problem: The Cloud Creates Performance Concerns


Cloud is shared• Others can affect my performance

My app

Otherapp

Otherapp

Otherapp

MajorISP

Local ISP

Mobile Carrier

Internet




Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

My users

Other users

Other users

Other users

The Answer: Collective Intelligence


Cloud is shared

Multiple contributors help diagnose issues for everyone

My app

Otherapp

Otherapp

Otherapp

“Outside-in” customer point of view

Web Applications in the Cloud: Need to Be Able to Pinpoint Problems


MajorISP

Local ISP

Mobile Carrier

Internet




Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

My users

Other users

Other users

Other usersIs it my

cloud provider

problem?

Is it an ISP or

Internet problem?

Is it a service

provider problem?

Is it a browser or

device problem?

Introducing… CloudSleuth

A Year In The Cloud

An end-user perspective on cloud performance

What we measured

Cloned reference web application deployed across various IaaS and PaaS providers

Added various services such as CDN when available

Structured application to highlight performance issues

How we measured

Various locations around the world

Backbone and Last Mile locations

Every 15 minutes, 24/7, for almost 1 year

Used a strict definition of availability.

Not all clouds perform the same way

Average Response Time of Reference Application, as measured from US backbone locations

Taking the long view – response time

3/25/0

9

4/4/0

9

4/14/0

9

4/24/0

9

5/4/0

9

5/14/0

9

5/24/0

9

6/3/0

9

6/13/0

9

6/23/0

9

7/3/0

9

7/13/0

9

7/23/0

9

8/2/0

9

8/12/0

9

8/22/0

9

9/1/0

9

9/11/0

9

9/21/0

9

10/1/0

9

10/11/0

9

10/21/0

9

10/31/0

9

11/10/0

9

11/20/0

9

11/30/0

9

12/10/0

9

12/20/0

9

12/30/0

9

1/9/1

0

1/19/1

0

1/29/1

0

2/8/1

0

2/18/1

0

2/28/1

0

3/10/1

00

1

2

3

4

5

6

7

8

9

10

Average Response Time for reference transaction on Amazon S3, as measured from US locations

© 2

010

Gom

ez –

All

Rig

hts

Res

erve

d

Taking the long view - availability

99

99.1

99.2

99.3

99.4

99.5

99.6

99.7

99.8

99.9

100

100.1

Weekly availability for reference transaction on Amazon S3, as measured from US locations

Geographic Latency – GoGrid

Geographic Latency – S3

Going International?

It’s not all about network latency

Enough scary news…… let’s see something good for a change

Geographic Latency – S3

Added services help… a lot!

Unique opportunities for optimization

Better performance, for little work, at no extra cost?

Best Practice: Define your goals and build a plan

Align goals across your organization Why are we moving to the cloud?

Common goals include:

1. Additional Capacity – How much capacity do we need during normal and peak times?

2. Improved End-User Experience – What performance goals are we trying to deliver against?

3. Greater Elasticity – How quickly can the provider we select ramp up to meet our needs?

4. Flexible Bursting – How fast do we need to be able to access additional capacity?

If only there was a button to push!

Best Practice: Keep your end-users in mind

How was that experience for your customers?

Test your cloud applications the same way your customers use it:

What they do? Customers care about completing tasks

NOT whether the homepage is available

Where they do it from? Your customers don’t live in the cloud, test from their perspective

When they do it? Test at peak and normal traffic levels, to find all the problems

What expectations do customers have? Is 4 seconds fast enough or does it have to be quicker?

Geographic disparities

4 sec’s

22 sec’s

Find configuration, capacity, and performance problems before end-users with pre-production testing and production monitoring

Best Practice: Performance testing cloud capabilities

Evaluate vendors based on your goals…

CapacityTest vendors to 15-20% past estimated capacity goals

ElasticityBaseline end-user performance before & after testing

Test during pre-deployment and in production

Ramp elasticity testing to peak levels

BurstabilityIsolate the cloud elements from other infrastructure to test

Test the “failover process”

Best Practice: Set SLAs to match your goals

Set SLAs based on your goals…

End-user availability and response times

Capacity & elasticity objectives

Burstability goalsSet SLAs based on how you are using the cloud…

SaaS PaaS IaaS• Transactional Success Rate• Fast end-user experience across key geographies

• Availability and performance of key components, services, and API’s

• Hardware & OS availability / performance• Network connectivity

• Capacity and elasticity requirements

Putting together a cloud management strategy

Key Steps Vendor Evaluation

Deployment Testing

Production Monitoring

Service Level Management

Validate the burstability & elasticity of cloud solutionsIdentify application performance bottlenecksMeasure performance under normal and peak conditionsBaseline and monitor service level objectivesMeasure how end-user performance impacts behaviorMonitor performance & availability from the end-users’ perspective

[email protected]

Twitter: @imadmouline

mailto:[email protected]

Performance Testing: Putting Cloud Customers Back in the Driver’s Seat

Technology

Transcript of Performance Testing: Putting Cloud Customers Back in the Driver’s Seat