5 Steps for Identifying Deficiencies and Fixing Problems FAST

Confidential, Dynatrace LLC 1

5 Steps for Identifying Deficiencies and Fixing Problems FAST

Gary CarrSoftware ArchitectAmerican Fidelity

Asad AliProduct Specialist

Dynatrace

1. Know what’s important to ALL key stakeholders• Who identifies deficiencies?• Who discovers problems?• Who wants resolution?

2. Create specific criteria for PoC• Review current systems, know your gaps• Stay open minded

3. Work on Deployment4. Always look for the metrics5. Share the results

Confidential, Dynatrace LLC

The 5 Steps


Nine times on Forbes “Top 100 Places to Work”

Focus on 24x7 service to Customers, Colleagues,

Community

Technology supports relationships, rather than replacing

Partner support is key to growth

1.9 million policies, 1,800 employees

Who is American Fidelity?


In-house tools provided the basics

Before Dynatrace

• API, ESB, F5 URL calls• DNS related

problems• No visibility into

exceptions


• Two kinds of problems• Visible, but lots of time and resources• Invisible

Before Dynatrace

“It was like being in a dark room”Traci Tabyanan, Applications Manager, American Fidelity


• Business• More accurately measure

user satisfaction• Provide excellent customer

experience• Release applications more

often• Reduce technical debt

Reasons for Enterprise-ready APM

“We’re heading toward being a technology company that supplies insurance”Diana Bittle, VP of Software Solutions, American Fidelity

43k Users


Reasons for Enterprise-ready APM• Technical

• Reproduce production problems accurately in dev

• Understand how 3rd party tools operate

• Excessive database calls• Visual topology of complex delivery

chain• Between tiers and technology

platforms• Inconsistent performance between

test environments for same build• Code-level diagnostics


• Use of guesswork to identify root cause• Lack of visibility into the end user response time• Visibility gaps between application tiers• Need to reproduce problems to identify root cause• Necessity for war rooms for major problems

How to identify deficiencies in your APM


Don’t be like Michael Scott…

“the office” is a trademark of NBC


Prepare for Successful PoC• Focused on enterprise wide applications• Invited developers, testers, DevOps, security, and project managers• Shared applications and issues with APM experts• Set up before PoC began with specialists from our company and Dynatrace• Very specific requirements for specific tests

• Found a need in our company and used the PoC to solve• Visibility from individual’s browser to deepest calls in the system• In-depth analysis of resource consumption per transaction• Dashboards specific to each stakeholder

• Showed enterprise level• AND provided easy drill-down


• Define concrete and specific PoC success criteria• Use most significant application(s)• Stress test your environment with the APM tool• Invite all stakeholders to the PoC• Look for value DURING• Ensure apples-to-apples

Lessons Learned from 500+


• Problem #1: Most of the execution time for main app was spent in .NET remoting and ADO• + ID where most of

time was spent• Focus on reducing

technical debt in those areas

AF’s PoC: Key Findings


• Problem #2: Able to isolate remoting issue to one particular server• Been looking for a month• Saw because capture all

transaction, including exceptions• Seamless fit: infrastructure

and processes• Helped id specific code

changes

AF’s PoC: Key Findings


• Simple view of complex delivery chain

• Automatically created

• Auto-discovery removes guesswork

PoC: Key Findings

Dynatrace

Art!


• Database connectivity issues• Configuration settings • Remoting calls => $40M savings• Architecture anti-patterns• Memory leaks• N+1 query patterns

Other Key Findings


• Involve DevOps early!• Share successes• Early example:

• 2-day MIRT right after deployment, so were not using Dynatrace much yet…

• Let’s check one thing easier than thought

• Solution in 20 minutes• Benefit grows exponentially

• Now Dynatrace involved early in MIRT

Deployment Success


Metrics help build better apps, faster

More enrollments. Yeah!



More processing. Boo!

• From 20 minutes to 2 hours

• Errors and unexpected timeouts

• Tried increasing planned timeouts



• Quickly saw exact problem• Database query

taking 7 minutes Called 443 times

• Down to 56 calls• Straightforward

code change


Gasps + Clapping = Performance HeroEven as enrollments continue to grow, processing time remains low


Take-aways from a developer• Look at System Profiles> YourApp> Diagnose

Events> Exceptions• Customize your timeframe (i.e. 7PM – 9PM)• Use PurePath™ = “Intense Tracing”

• Look for method names• Look for long execution times• Keep drilling down• Sampling would not have found this issue

• Don’t be afraid to ask for help


• Coincidences bring nightmares• Sometimes the state of the system is the problem• Gap-free APM provides

• All the data leading up to issue, from every tier• Plus inter-tier stitching• Prioritize and reduce technical debt

• Faster fixes• “Customer cannot log in, it happened at 8:45 a.m.”• Just grab all the data from 8:44 a.m. to 8:46 a.m.

It’s all about the visibility


• Ability to compare slow transactions with fast transactions• Reduces the need to

reproduce problems• Easily help identify root cause

of pool exhaustion issues• Enables conversion of

application data to business data• Eliminates guesswork

Benefits of gap-free data


User Satisfaction• Now developers have a

closer connection to customers• APM shows impact of coding

methods• Developers can see real-time

user satisfaction


• Promotes DevOps culture• Reduces Ops anxiety over newer

code release• Enables frequent code releases• Make Dev an equal partner in

problem resolution• Promotes innovation

Dev and Prod synch serves customers


• War room calls directly show the power• Town Hall meetings: App Development, Business, Marketing,

and more• TV and tablets with HTML5• Results• Challenges!

How AF spreads the use of APM


• Promote key improvements to upper management

• Use APM tool to make the developers the “hero”

• Translate APM success to business $$$

• Display APM dashboards in public areas

• Use APM tool to reduce schism between Dev and Ops

Insights from other customers on spreading APM


• Continue to improve relationship with the business• Using dashboards• See their applications

• Dynatrace Community• Sharing information

• Training• User Groups and Conferences• New release strategy

Extra notes


•Define concrete PoC criteria•Use significant application(s) for PoC• Look for value DURING• Share success of APM results• Promote DevOps culture•Make Dev an equal partner in problem resolution

Summary


1. Know what’s important to ALL key stakeholders• Who identifies deficiencies?• Who discovers problems?• Who wants resolution?

2. Create specific criteria for PoC• Review current systems, know your gaps• Stay open minded

3. Work on Deployment4. Always look for the metrics5. Share the results

In Review: The 5 Steps

5 Steps for Identifying Deficiencies and Fixing Problems FAST

Technology

Transcript of 5 Steps for Identifying Deficiencies and Fixing Problems FAST