Performance Testing@

With 100,000 concurrent users

WHO ARE WE

• AWS for most client projects and our own• Enterprise/Solution Architecture +

Development• Performance Engineering, Profiling,

Optimization• "DevOps"• AWS for most client projects and our own

WHAT WE WANTED

Performance test applications with 100,000 concurrent users

(Websocket + HTTP)

• Easy to use• Traceable history of tests• Good fit with modern DevOps

processes• Low maintenance and cost

99 Problems

Quickly max out a single test node(or even 10 of them)

Performance tests should be available to whole team(not just the guru)

Shouldn't have to write another App(or pay A LOT for one)

REQUIREMNTS

SOLUTION

(one of many)

STEP 1

Build App(Compile, Test, Package)

STEP 2

Build your environment (Puppet, Chef, etc)Deploy new application version

STEP 3

Create load test slave instances

STEP 4

Start running the test plan against new environment

STEP 5

• Test slaves return results to Jenkins.

• Then clean up• Shut down instances• Snapshot new AMI

PIPELINE

Build app -> Create test env + Deploy -> Test new environment

LESSONS LEARNED 1/2

• Performance problems found and fixed Garbage Collection, Serialization and encoding

• Very helpful when sizing infrastructure Webservice tier network was bottleneck Different instance sizes picked

• Scaling the test clients can be hard

LESSONS LEARNED 2/2

• Performance testing via ELB• Ramp up slowly• Make sure client see DNS

changes• ELB's not great for Websocket

• Linux tuning• Ephemeral port range• Buffers everywhere• File handles, flood protection, etc• It does a pretty good job by

default

Results & Deep Insight

Deep Insight with dynaTrace APM beyond simple JMeter results

Clearly understand the difference between:• Load generation• Real User Monitoring • Always-on deep transaction tracing

TEST AUTOMATION / CI

• Unit Tests are fully integrated as part of CI• Each build also measures performance for each testcase

BONUSES

Easily re-used for different types of build/testing

• BYO load test tool• Spot instances are cheaper• Not just for load testing• Test Chef/Puppet scripts• OS updates

• Browser based (Webdriver, PhantomJS, etc)

• Auto-scale testing• Snapshot AMI after successful tests

Questions ?

Thanks !

Twitter: msquareau

http://m-square.com.au