T14 Test Techniques

10/16/2014 1:30:00 PM

Speed Up Testing with Monitoring Tools

Presented by:

Jim Hirschauer

AppDynamics

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Jim Hirschauer

AppDynamics Technology evangelist for AppDynamics, Jim Hirschauer has an extensive background in highly available, business critical, large enterprise IT operations environments. Jim has been interested in application performance testing and monitoring since his days as a systems administrator in a retail bank. His passion for performance analysis led him down a path to design, implement, and manage the cloud computing monitoring architecture for a top-ten investment bank. During his tenure at the bank, Jim created new processes and procedures that reduced the time spent in the test and QA cycle while simultaneously increasing overall code release quality and dramatically improving end user experience.

Awesome Ops-ing!!

Speed up Testing Using Monitoring Tools"

Abstract"•  In many organizations, the software development lifecycle is a pretty

tedious process. However, by using the right tools and process, you can accelerate development and release higher quality code than ever before. Proper monitoring tools and methodologies are the cornerstone of rapid software delivery. These tools provide immediate feedback with actionable information so that problems can be addressed as they are detected instead of waiting until the end of a testing cycle. Earlier detection of problems combined with tests that are a better representation of production workloads are the keys to releasing better code, faster. Jim Hirschauer shows how to use monitoring software to make a major impact during development, test, and production. Examine typical use cases for server monitoring, log monitoring, and application performance monitoring, and learn about open source testing tools including Siege, Multi-Mechanize, and Bees with Machine Guns. Understand how to use each of these tools in development, test, and production as well as creating a feedback loop that drives continuous improvement."

10/1/14% 2%@HirschOnAPM%

Jim Hirschauer"Technology Evangelist with AppDynamics"

" @HirschOnAPM"

"Aerospace Engineer, Guitarist, Dad,

Snowboarder, Beer and Scotch Drinker, Mediocre Sailor, Better Pilot"

10/1/14% 3%@HirschOnAPM%

Rumored to have designed monitoring architectures for top Financial Services

institutions."

Also Rumored to have introduced new test and QA processes at these very

same institutions.""

These allegations can neither be confirmed nor denied."

10/1/14% 4%@HirschOnAPM%

1. Ops bring valuable skills to the SDLC""

2.  “Early Ops” helps production Ops"

"3. Ops speed up the SDLC"

10/1/14% 5%@HirschOnAPM%

Why does performance

matter?"

10/1/14% 6%@HirschOnAPM%

Performance !=!

Revenue"

10/1/14% 7%@HirschOnAPM%

Microsoft found that Bing searches that were 2 seconds

slower resulted in a 4.3% drop in revenue per user"

10/1/14% 8%@HirschOnAPM%

When Mozilla shaved 2.2 seconds off their landing page, Firefox

downloads increased 15.4% !(60 million more downloads) !

"

10/1/14% 9%@HirschOnAPM%

Making Barack Obama�s website 60%

faster increased donation conversions by 14%"

10/1/14% 10%@HirschOnAPM%

Amazon and Walmart increased revenue 1% for every

100ms of improvement"

10/1/14% 11%@HirschOnAPM%

What was Amazon’s revenue last year?"

10/1/14% 12%@HirschOnAPM%

$75 BB!!

X 1%!!

= $750 MM"10/1/14% 13%@HirschOnAPM%

Performance directly impacts the bottom

line"

10/1/14% 14%@HirschOnAPM%

Getting woken up in the middle of the night sucks!!!"

10/1/14% 15%@HirschOnAPM%

Brand/Reputation Damage"

10/1/14% @HirschOnAPM% 16%

The SDLC Visualized"

10/1/14% @HirschOnAPM% 17%

*Blatantly%stolen%from%my%friend%and%colleague%DusGn%WhiIle%

10/1/14% 18%@HirschOnAPM%

HOW DO YOU DEAL WITH TEST WINDOW OBLITERATION?"

TESTING!BEFORE!TEST"

10/1/14% @HirschOnAPM% 20%

Basic testing earlier using FOSS"

10/1/14% @HirschOnAPM% 21%

Siege"

10/1/14% 22%@HirschOnAPM%

siege -c 10 -b -t 10S http://dustinwhittle.com/%

10/1/14% 23%@HirschOnAPM%

** SIEGE 2.72** Preparing 10 concurrent users for battle.The server is now under siege...Lifting the server siege... done.Transactions: 263 hitsAvailability: 100.00 %Elapsed time: 9.36 secsData transferred: 0.35 MBResponse time: 0.35 secsTransaction rate: 28.10 trans/secThroughput: 0.04 MB/secConcurrency: 9.82Successful transactions: 263Failed transactions: 0Longest transaction: 0.54Shortest transaction: 0.19%

10/1/14% 24%@HirschOnAPM%

Crawl the entire app to discover all urls"

10/1/14% 25%@HirschOnAPM%

sproxy -o ./urls.txt %

10/1/14% 26%@HirschOnAPM%

SPROXY v1.02 listening on port 9001"...appending HTTP requests to: ./urls.txt"...default connection timeout: 120 seconds%

10/1/14% 27%@HirschOnAPM%

wget -r -o verbose.txt -l 0 -t 1 --spider -w 1 -e robots=on"

-e "http_proxy = http://127.0.0.1:9001"""http://dustinwhittle.com/"%

sort -u -o urls.txt urls.txt%

10/1/14% 28%@HirschOnAPM%

Benchmark traffic across all unique urls

with siege"

10/1/14% 29%@HirschOnAPM%

siege -v -c 50 -i -t 3M -f urls.txt -d 10%

10/1/14% 30%@HirschOnAPM%

10/1/14% 31%@HirschOnAPM%

Multi-Mechanize is an open source framework for performance and load

testing"

10/1/14% 32%@HirschOnAPM%

Multi-Mechanize Benefits"

"Run many test scripts"

"Control workload pattern (ramping)"

"Visual output"

10/1/14% 34%@HirschOnAPM%

What about when you need more than one test machine?"

10/1/14% 35%@HirschOnAPM%

Bees with Machine Guns"10/1/14% 36%@HirschOnAPM%

A utility for arming (creating) !many bees (micro EC2

instances)! to attack (load test)!

targets (web applications)"

10/1/14% 37%@HirschOnAPM%

pip install beeswithmachineguns"

10/1/14% 38%@HirschOnAPM%

# ~/.boto !!![Credentials] !!aws_access_key_id=xxx !aws_secret_access_key=xxx !!![Boto] !!ec2_region_name = us-west-2 !ec2_region_endpoint = ec2.us-west-2.amazonaws.com"

10/1/14% 39%@HirschOnAPM%

bees up -s 2 -g default -z us-west-2b -i ami-bc05898c -k appdynamics-dustinwhittle-aws-us-west-2 -l ec2-user"

10/1/14% 40%@HirschOnAPM%

Connecting to the hive. Attempting to call up 2 bees. Waiting for bees to load their machine guns... . . . . Bee i-3828400c is ready for the attack. Bee i-3928400d is ready for the attack. The swarm has assembled 2 bees."

10/1/14% 41%@HirschOnAPM%

bees report"

10/1/14% 42%@HirschOnAPM%

Read 2 bees from the roster.Bee i-3828400c: running @ 54.212.22.176Bee i-3928400d: running @ 50.112.6.191%

10/1/14% 43%@HirschOnAPM%

bees attack -n 1000 -c 50 -u

http://dustinwhittle.com/"

10/1/14% 44%@HirschOnAPM%

Read 2 bees from the roster.Connecting to the hive.Assembling bees.Each of 2 bees will fire 50000 rounds, 125 at a time.Stinging URL so it will be cached for the attack.Organizing the swarm.Bee 0 is joining the swarm.Bee 1 is joining the swarm.Bee 0 is firing his machine gun. Bang bang!Bee 1 is firing his machine gun. Bang bang!Bee 1 is out of ammo.Bee 0 is out of ammo.Offensive complete. Complete requests: 100000 Requests per second:1067.110000 [#/sec] (mean) Time per request: 278.348000 [ms] (mean) 50% response time: 47.500000 [ms] (mean) 90% response time: 114.000000 [ms] (mean)Mission Assessment: Target crushed bee offensive.The swarm is awaiting new orders.%

10/1/14% 45%@HirschOnAPM%

bees down"

10/1/14% 46%@HirschOnAPM%

BWMG - Disclaimer"

10/1/14% 47%@HirschOnAPM%

A SHORT BACKGROUND STORY"

10/1/14% 48%@HirschOnAPM%

Creating new words using “ing”"

The present participle refers to things that are still happening. To make the present participle, the ending -ing is added to the infinitive ('I am asking her a question')."Source: http://www.oxforddictionaries.com/us/words/verb-tenses-adding-ed-and-ing"

10/1/14% 49%@HirschOnAPM%

Sportsing!!!"

10/1/14% 50%@HirschOnAPM%

AND NOW BACK TO OUR REGULARLY SCHEDULED PRESENTATION"

10/1/14% 51%@HirschOnAPM%

Ops Offer-ings "

•  Distributed troubleshooting experience"

•  Extensive infrastructure knowledge"

•  Monitoring tool expertise"

•  General IT Awesome-ing!!!"

10/1/14% 53%@HirschOnAPM%

Process ING"

10/1/14% 54%@HirschOnAPM%

Instrument-ing"

To equip with instruments especially for measuring and recording data."Source: http://www.merriam-webster.com/dictionary/instrument"

"

10/1/14% 55%@HirschOnAPM%

Oracle%

Purchase%Search%Flight%Flight%Status%

Login%

Network%

CDN%Browser(s)%

NaGve%Mobile%

App%

Tomcat%Service% Weblogic%Service% .NET%Service%

Cassandra%

ESB/MQ%

Apache%

JBoss%Service%

OS#

End#User# Applica1on#

Network#

Instrument Everything?"

10/1/14% @HirschOnAPM% 56%

Log#Files#

Database#

10/1/14% 57%@HirschOnAPM%

percepGon%

aIenGon%

observaGon%

NoGce[ing%

Architecture problems?"

10/1/14% @HirschOnAPM% 58% 58%

Resource contention?"

10/1/14% @HirschOnAPM% 59% 59%

Infrastructure problems?"

Network problems?"

Code problems?"

10/1/14% @HirschOnAPM% 62% 62%

Poor performing queries?"

10/1/14% @HirschOnAPM% 63% 63%

Grok-ing"

Grok means to understand so thoroughly that the observer becomes a part of the observed—to merge, blend, intermarry, lose identity in group experience. It means almost everything that we mean by religion, philosophy, and science—and it means as little to us (because of our Earthling assumptions) as color means to a blind man."Source: http://en.wikipedia.org/wiki/Grok"

10/1/14% @HirschOnAPM% 64%

Source:%hIp://ihumanable.com/blog/2010/01/11/grok/%

10/1/14% 65%@HirschOnAPM%

10/1/14% 66%@HirschOnAPM%

Know your applications!"

10/1/14% 67%@HirschOnAPM%

Log Analytics Tools"

FOSS:"•  Logstash"•  Greylog2"•  Elastic Search + Flume""Paid:"•  Splunk"•  Loggly"•  SumoLogic"•  AppDynamics (beta)"

10/1/14% 68%@HirschOnAPM%

Infrastructure Monitoring Tools"

FOSS:"•  Nagios"•  Groundworks"•  Ganglia"

Paid:"•  Zenoss"•  Groundworks"•  HP Patrol"

10/1/14% 69%@HirschOnAPM%

Network Monitoring Tools"

FOSS:"•  OpenNMS"•  OP5"•  NetXMS""Paid:"•  PRTG"•  ExtraHop"•  Boundary"•  Solarwinds"

10/1/14% 70%@HirschOnAPM%

Application Monitoring Tools"

FOSS:"•  Helios"•  Hyperic"•  Jconsole""Paid:"•  AppDynamics (Free Lite Version)"•  CA Wily"•  IBM SmartCloud APM"

10/1/14% 71%@HirschOnAPM%

THE RESULT"

10/1/14% 72%@HirschOnAPM%

• 2-5X faster testing"• Fewer Day 1 – Week 1 issues"• Better production performance"• Prioritization of development effort"• Continuous improvement cycle"

10/1/14% 73%@HirschOnAPM%

When you get back to work (dev)…"

•  Contact Ops team and ask about tools"•  Contact QA and ask about production

workload details"•  Invite monitoring ops to participate in next

load test"

•  Walk monitoring ops through application functionality before next test"

•  Actively participate in next load test"

When you get back to work (ops)…"

•  Find a problematic application"•  Determine if production workload details are

available"•  Contact Dev team and discuss assisting during

next load test"•  Contact QA and ask about production workload

details"•  Enable and/or access appropriate monitoring in

load test environment"•  Actively participate in next load test"

If you do these things, you’ll be on the path to…"

10/1/14% 76%@HirschOnAPM%

Thank You"

Shameless Plug: Free Monitoring"http://www.appdynamics.com"

10/1/14% 77%@HirschOnAPM%