AppSphere 15 - AppDynamics: Beyond APM - Building an Operations Center

AppDynamics: Beyond APM – Building an Operations Center Paul Pantazis - Operations Manager Shane Emery - Systems Administrator

Introduction: Speaker

•  18 years of IT OPS experience •  Started as a DEV but saw the light ( before there was a

term for DEVOPS) Copyright © 2015 AppDynamics. All rights reserved. 2

PAUL PANTAZIS Manager, IT Ops, Entertainment Partners

Our situation

•  Typical (and in some cases “not so typical”) performance issues. •  The idea of slow also changed depending on the individual you

spoke to ( perception vs. reality) •  DEV à OPS ( some mumbling about the effect of a hypervisor

on “nice Java apps” ..) •  OPS à DEV what a mess as you can imagine.. •  Either way customers usually pay the price •  We had to do something…

Copyright © 2015 AppDynamics. All rights reserved. 3

Our stack - Incoming Acronyms

•  Java, .NET, C •  Jboss, Mule ESB, MQ, Citrix, IIS, Tomcat, NGINX •  Multiple Relational backend technologies: SQL Server, IBM DB2 etc. •  Java Swing, Spring, EJB’s, Adobe Air, Angular JS •  Linux, Windows •  Multi-tiered/”distributed” apps (multiple intersystem integrations ) •  AWS, Azure, On-Prem


What we needed was ..

•  Ability to triage and pinpoint bottlenecks (High Level ) •  Ability to “dig-deep” really identify Root Cause and facilitate

remediation's from there – Also do it fast. •  Stop finger pointing and actually solve problems •  Incorporate groups in the problem solving process typically

not able to contribute (more information on the escalations)


What we needed was ..

•  Track and Troubleshoot specific Transactions end to end •  Alerting ( not the usual OPS “stuff” ):

•  Generate response time, load, % normal, slow, stall •  Error reports for monthly products ops review meetings •  Dig into the database •  Daily transaction analysis to gage general health of the app,

especially after major releases or when users complain of slow system performance


How we ended up here..

•  Started a RFP to look at Market leaders in the space •  Needed something that was easy to setup and install •  Needed immediate “fix” •  Needed a vendor who will strategically partner with us •  Needed a tool that did not have a basic level steep learning curve •  Needed a platform that was intuitive to both DEV/OPS and Business folks alike

(in other words views of the data can be abstracted and represent at the various levels of understanding)


Typical Operations Center/IT interaction over an issue

•  E-mail chain 200 messages deep

•  Well dressed folks on cell

phones pacing nervously asking “How much longer?”

•  End result -----à


Typical DEV/IT interaction in the app_dynamics world


Because we are working on solving the problem not pointing fingers …


Setup at a Glance… •  On Premise! •  VM Controllers with OS/DB backups

–  Application Environment -> v4.1.5 •  1 Controller -> 900+ Agents, 64GB Ram, 16 CPU, 2TB Storage •  1 EUM -> 8GB Ram, 4 CPU, 100GB Storage •  1 Analytics -> POC, 16GB Ram, 4 CPU, 100GB Storage •  1 APP4DB -> 10 Collectors, 4GB Ram, 2 CPU, 100GB Storage

–  Instrumenting .Net, Java, PHP


The End Result…

•  “Measure twice; cut once” •  The same is true with our AppDynamics experience

–  By measuring the performance of our applications in development, make adjustments, and then continue to do so in production…

–  AppDynamics directly impacted customer satisfaction in a positive way

•  AppDynamics helped us measure performance, analyze results, and immediately act to keep performance high

•  Fast and available Applications kept customers happy!


Operations Center

“A facility dedicated to monitoring infrastructure and applications availability as well as facilitating the response and coordination to address operational issues negatively

affecting information technology services”


From APM to NOC

•  Early integration and collaboration of traditional OPS Teams with DevOps Silos (just kidding) is undoubtedly critical.

•  Equally, if not more important, is the operational lifecycle, upkeep and maintenance of that very SDLC output (your apps..)

•  Its not “pretty work” but its vital


Of course I am Talking Operations Center


Disclosure


•  We are not naïve about the complexity of building a Operations center and fully understand the often sizable investments and complicated requirements that come with that …

•  However, we also question the need for such complex solutions for the average IT Shop, especially when there are already investments in-house that can get us “most of the way there”

Key Elements of a Operations Center

•  Incident Detection •  MIM (Major Incident Management) •  Triage •  Runbook automation


Elements of Operations Center

Detecting Incidents: one key element of a NOC is to detect incidents as early as possible. In the word of

app_D that means dashboards, auto-baselining and alerting



“ We are using app_D to notify NOC staff of an issue very quickly, while it is in a "yellow" state, before it becomes red/

critical and impacts end-users. ”


Elements of Operations Center (application Dashboard)


Elements of Operations Center (Health Rule Violation Details)


Elements of Operations Center (Garbage Collection Details)


Elements of Operations Center (CPU & Garbage Collection Correlation)



Triage: when an incident occurs, the NOC also needs to be

very efficient at triaging the issue to find root cause, and remediate those problems as quickly as possible.



AppDynamics helped provide deep visibility, down to the line of code or SQL query, to quickly pinpoint where there issue is,

so the appropriate teams can be engaged for resolution.

This eliminated the need for an "all hands on deck" approach, where all resources are engaged even though the problem is

not in their area.


Elements of Operations Center (Application Flow Map)


Elements of Operations Center (Slow Transaction Snapshots)


Elements of Operations Center (Specific Snapshot Flow Map)


Elements of Operations Center (Specific Snapshot Call Graph)


Elements of Operations Center (Specific Snapshot Hotspots)



•  Runbook automation: most NOC's maintain a runbook, which provides guidance to the team on how to handle specific types of issues.



AppDynamics, has helped us begin to start automating many

of the runbook operations in our stack, thereby freeing up capacity of more senior NOC resources for other critical

tasks.


Elements of Operations Center (Health Rule)


Elements of Operations Center (Action)


Elements of Operations Center (Action to Run Script)


Elements of Operations Center (Policy to link Health Rule to Action)


Additional Supporting Elements

Different Types of Dashboards …


End User Experience Monitoring


Got Heat Maps?


Got Ticket System?


Got Pager System?


Got TV?


Challenges

•  Monitoring endpoints –  Many applications can share the same endpoint –  Example: Single Credentials

•  Databases –  Applications architected for multiple database technologies

•  Customization –  Every business is unique, even though you will get useful and intuitive

data out of the gate •  Sharing with other technologies, ex solarwinds

–  Minimizing overhead –  Overlapping


Future Ventures…

•  Enhance Operations Center –  Not just CPU/DISK alerting, Take Action? –  More interactions with IT Operations?

•  End User Experience Monitoring –  Instrument from the outside looking in –  Implementing more applications

•  Server Monitor Pro –  Use bundled packages for legacy systems

•  C program Instrumentation –  Beta


Thank You

AppSphere 15 - AppDynamics: Beyond APM - Building an Operations Center

Documents

Transcript of AppSphere 15 - AppDynamics: Beyond APM - Building an Operations Center