Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along...
Transcript of Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along...
![Page 1: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/1.jpg)
1 #Dynatrace
Wolfgang Gottesheim @gottesheim
Performance Challenges along the Continuous Delivery Pipeline
![Page 2: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/2.jpg)
2 #Dynatrace
Example of a “Bad” Web Deployment 282! Objects
on that page 9.68MB Page Size
8.8s Page
Load Time Most objects are images
delivered from your main
domain
Very long Connect time
(1.8s) to your CDN
![Page 3: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/3.jpg)
3 #Dynatrace
Application problems in the wild 526s to render a financial transaction report
1 SQL running
210s!
Debug Logging with
log4j on outdated log4j
library (sync issue)
![Page 4: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/4.jpg)
4 #Dynatrace
But now we‘re doing
Continuous Delivery
![Page 5: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/5.jpg)
5 #Dynatrace
700 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds
![Page 6: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/6.jpg)
6 #Dynatrace
It‘s not about the pipeline… (only)
![Page 7: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/7.jpg)
7 #Dynatrace
Commit Stage
Acceptance Testing
Release
Developers
Capacity Testing
UAT
![Page 8: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/8.jpg)
8 #Dynatrace
Metrics
to the rescue!
![Page 9: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/9.jpg)
9 #Dynatrace
We want to replace magic with data!
Commit Stage
Acceptance Testing
Release Capacity Testing
UAT
![Page 10: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/10.jpg)
10 #Dynatrace
4 examples
![Page 11: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/11.jpg)
11 #Dynatrace
#1
![Page 12: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/12.jpg)
12 #Dynatrace
Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:
230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
![Page 13: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/13.jpg)
13 #Dynatrace
8MB background image for STPCon
![Page 14: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/14.jpg)
14 #Dynatrace
Key Metrics
# Resources
Size of Resources
Page Size
![Page 15: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/15.jpg)
15 #Dynatrace
#2
![Page 16: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/16.jpg)
16 #Dynatrace
#1: Loading too much data
24889! Calls to the
Database API!
High CPU and High Memory
Usage to keep all data in Memory
![Page 17: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/17.jpg)
17 #Dynatrace
#2: On individual connections
Classical N+1 Query Problem
Individual SQL really <1ms
12444! individual connections (and no pooling…)
![Page 18: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/18.jpg)
18 #Dynatrace
#3: Homegrown Caching Framework
Lots of time spent in Hashtable.get
Called from their Entity Objects
![Page 19: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/19.jpg)
19 #Dynatrace
Key Metrics
# Functional Errors # of SQL Calls # of same SQL Execs (1+N) # of Connections Rows/Data Transferred
![Page 20: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/20.jpg)
20 #Dynatrace
#3
Simply splitting web service calls is not a migration to a microservice architecture
![Page 21: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/21.jpg)
21 #Dynatrace
3136! Calls to H2 mostly executed on async background
threads
40! internal Web Service Calls that do all these DB Updates
21671! Calls to Oracle
DB Exceptions on both Databases
DB Exceptions on both Databases
33! Different connections used
![Page 22: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/22.jpg)
22 #Dynatrace
Key Metrics
# Service Calls # of Threads Timing (execution, wait) # SQL executions # of SAME SQLs
![Page 23: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/23.jpg)
23 #Dynatrace
#4
![Page 24: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/24.jpg)
24 #Dynatrace
Is this a successful new Build?
![Page 25: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/25.jpg)
25 #Dynatrace
Look at Resource Usage: CPU, Memory, …
![Page 26: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/26.jpg)
26 #Dynatrace
Memory? Look at Heap Generations
![Page 27: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/27.jpg)
27 #Dynatrace
Root Cause: Dependency Injection
![Page 28: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/28.jpg)
28 #Dynatrace
Prevent: Monitor Memory Metrics for every Build
![Page 29: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/29.jpg)
29 #Dynatrace
Key Metrics
# of Objects per Generation
# of GC Runs
Total Impact of GC
![Page 30: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/30.jpg)
30 #Dynatrace
1. # Resources
2. Size of Resources
3. Page Size
4. # Functional Errors
5. # SQL Executions
6. # of SAME SQLs
Where to start?
![Page 31: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/31.jpg)
31 #Dynatrace
P A M
I
ick gree easure ntegrate
Allow me to introduce…
![Page 32: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/32.jpg)
32 #Dynatrace AND MANY MORE Kieker
![Page 33: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/33.jpg)
33 #Dynatrace
We successfully replaced magic with data!
Commit Stage
Acceptance Testing
Release Capacity Testing
UAT
![Page 34: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/34.jpg)
34 #Dynatrace
Just one more thing…
![Page 35: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/35.jpg)
35 #Dynatrace
It‘s also about the pipeline…
Define pipeline metrics!
![Page 36: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/36.jpg)
36 #Dynatrace
Connect your Tests with Quality
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test Framework Results Architectural Data
We identified a regresesion
Problem solved
Exceptions probably reason for
failed tests Problem fixed but now we have an
architectural regression
Problem fixed but now we have an
architectural regression Now we have the functional and
architectural confidence
Let’s look behind the
scenes
![Page 37: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/37.jpg)
37 #Dynatrace
#1: Analyzing each Test
#2: Metrics for each Test
#3: Detecting Regression based on Measure
![Page 38: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/38.jpg)
38 #Dynatrace
Quality-Metrics based
Build Status
![Page 39: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/39.jpg)
39 #Dynatrace
Pull data into Jenkins, Bamboo ...
![Page 40: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/40.jpg)
40 #Dynatrace
Make Quality a first-class citizen
„Too hard“
„we‘ll get round to this later“
„not cool enough“
![Page 41: Performance Challenges along the Continuous Delivery Pipeline€¦ · Performance Challenges along the ... Example of a “Bad” Web Deployment 282! Objects 9.68MBPage Size on that](https://reader034.fdocuments.us/reader034/viewer/2022042305/5ed0acefd84e6310ae13914f/html5/thumbnails/41.jpg)
41 @Dynatrace
Wolfgang Gottesheim Free Tools: http://bit.ly/dttrial
Twitter @gottesheim