Download - Full Stack Load Testing

Full Stack Load Testing

Boston Web PerformanceAugust 27th, 2014

What do you mean, “testing”?

Unit

Functional

IntegrationPerformance

Security

Usability

YOUR CODE

YOUR CODE

ELB

RDS Stripe

Web Cache

Memcache

•Production-level load•Production-level servers•Production-level cost

•Production-level load•Production-level servers•Production-level cost•Production-level bugs

DO IT LIVEYeah, well, that’s just like, your opinion, man

SCRIPT IT

…cURL?

$ for i in `seq 1 1000`; do curl http://example.com/login &; done

Apache Bench$ ab -n 100 -c 10 http://www.yahoo.com/Concurrency Level: 10Time taken for tests: 1.889 secondsComplete requests: 100Failed requests: 0Write errors: 0Total transferred: 1003100 bytesHTML transferred: 949000 bytesRequests per second: 52.94 [#/sec] (mean)Time per request: 188.883 [ms] (mean)Time per request: 18.888 [ms] (mean, across all concurrent requests)Transfer rate: 518.62 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median maxConnect: 57 59 1.7 59 64Processing: 117 126 7.5 124 162Waiting: 57 62 7.0 60 98Total: 175 186 8.0 184 224

Percentage of the requests served within a certain time (ms) 50% 184 66% 186 75% 187 80% 188 90% 192 95% 203 98% 216 99% 224 100% 224 (longest request)

Selenium

“More realistic”

“Netflow is a feature that was introduced on Cisco routers that give the ability to collect IP network traffic as it enters or exits an interface.”

- Wikipedia

Netflow?

Source IP Source Port Destination IP Destination Port Protocol

“Sooo…. what do you do?”

• Layer 3 (Standard)– IP, port, protocol

• Layer 7 (FlowView)– Application names

tcpreplayroot@pw29:~# tcpreplay -i eth7 -tK --loop 50000 --netmap --unique-ip smallFlows.pcapSwitching network driver for eth7 to netmap bypass mode... done!File Cache is enabledActual: 713050000 packets (460826550000 bytes) sent in 385.07 seconds.Rated: 1194660947.8 Bps, 9557.28 Mbps, 1848532.79 ppsFlows: 60450000 flows, 156712.44 fps, 712150000 flow packets, 900000 non-flowStatistics for network device: eth7 Attempted packets: 713050000 Successful packets: 713050000 Failed packets: 0 Truncated packets: 0 Retried packets (ENOBUFS): 0 Retried packets (EAGAIN): 0Switching network driver for eth7 to normal mode... done!

So…?

Lesson #1: Control the load directly

Lesson #2: Use Real Traffic

Lesson #3: Push on the sticky points

Thanks!

[email protected]@_tr

appneta.com <-- pretty graphs! we’re hiring!