Troubleshooting ASP.NET and IIS Scalability Hotspots
-
Upload
dynatrace -
Category
Technology
-
view
523 -
download
5
Transcript of Troubleshooting ASP.NET and IIS Scalability Hotspots
![Page 1: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/1.jpg)
Please Standby: Webcast to Begin Shortly
Unable to listen via your speakers? Dial in: Toll Free: (877) 309-2074 Access Code: 119-063-034
International: Click “telephone” in audio section,then click “additional numbers”
Troubleshooting ASP.NET and IIS Scalability Hotspots
![Page 2: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/2.jpg)
Confidential, Dynatrace, LLC
Asad AliDirector, Product SpecialistDynatrace
Troubleshooting ASP.NET and IIS Scalability Hotspots
![Page 3: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/3.jpg)
• Real Use Cases• IIS/ASP.net Metrics• High Latency Application Example• Synchronization Issues• Sharepoint Example
Agenda
![Page 4: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/4.jpg)
Confidential, Dynatrace LLC
Real Use Cases
![Page 5: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/5.jpg)
Online Room Service
22% spent in IIS
![Page 6: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/6.jpg)
Online Room Service
Tip: Elapsed Time tells us WHEN a Method was executed!
![Page 7: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/7.jpg)
![Page 8: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/8.jpg)
Online Banking: Slow Balance Check1.69m (=101s!) To
Check Balance!
87% spent in IIS
![Page 9: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/9.jpg)
Time really spent in IIS
Tip: Elapsed Time tells us WHEN a Method was executed!
Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET
Tip: Thread# gives us insight on Thread Queues / Switches
![Page 10: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/10.jpg)
Confidential, Dynatrace LLC
• Modules Installed
• IIS/ASP.net Metrics
What To Look
![Page 11: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/11.jpg)
Confidential, Dynatrace LLC
IIS Modules
![Page 12: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/12.jpg)
IIS/ASP.net Metrics
![Page 13: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/13.jpg)
Confidential, Dynatrace LLC
Arrival Rate• Rate at which requests are arriving in the queue
• Perfmon Counter: Http Service Request Queues\ArrivalRate
• Should be >= W3WP_W3SVC\Requests / sec
IIS/ASP.net Metrics
![Page 14: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/14.jpg)
Confidential, Dynatrace LLC
MaxQueueItemAge• Incoming requests are waiting at this long to be processed
• Perfmon Counter: Http Service Request Queues\MaxQueueItemAge
IIS/ASP.net Metrics
![Page 15: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/15.jpg)
Confidential, Dynatrace LLC
CLR ThreadPool Queue•Requests are queued if all the CLR threads are busy
• Perfmon Counter: ASP.NET v4.0.30319/Requests Queued
• Symptom: ASP.net returns 503 Service Unavailable
IIS/ASP.net Metrics
![Page 16: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/16.jpg)
Confidential, Dynatrace LLC
Number of Free DB Connections
• Number of free ADO.net connections available
• Perfmon Counter: .Net Data Provider for SqlServer/NumberOfFreeConnections
• Alert: If value == 0
ADO.Net Metrics
![Page 17: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/17.jpg)
Confidential, Dynatrace LLC
Number of Active DB Connections
•Number of Active ADO.net connections
•Perfmon Counter: .Net Data Provider for SqlServer/NumberOfActiveConnections
ADO.Net Metrics
![Page 18: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/18.jpg)
Confidential, Dynatrace LLC
High Latency Application Example
![Page 19: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/19.jpg)
![Page 20: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/20.jpg)
Confidential, Dynatrace LLC
Too Many Queries crash
Undersized Pools
![Page 21: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/21.jpg)
Online Banking: Slow Balance Check1.69m (=101s!) To
Check Balance!
87% spent in IIS 600! SQL Executions
![Page 22: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/22.jpg)
SQL ExecutionsFinding: EVERY SQL
statement is executed on ITS OWN Connection!
Tip: Look at “GetConnection”
![Page 23: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/23.jpg)
Lesson Learned
ASP.NET Worker Thread Pool Sizing!
DB Connection PoolsMore Efficient SQL
![Page 24: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/24.jpg)
SQL ExecutionsFinding: EVERY SQL
statement is executed on ITS OWN Connection!
Tip: Look at “GetConnection”
![Page 25: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/25.jpg)
Confidential, Dynatrace LLC
SQL Executions
#1: Same SQL is executed 67! times
#2: NO PREPARATION because everything
executed on new Connection
![Page 26: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/26.jpg)
Confidential, Dynatrace LLC
• Idle vs. Busy Threads• # SQLs / Request• # GetConnection
Tips To Apply
![Page 27: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/27.jpg)
Confidential, Dynatrace LLC
SynchronizationIssues
![Page 28: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/28.jpg)
Confidential, Dynatrace LLC
Synchronization Issues
![Page 29: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/29.jpg)
Load vs. Memory vs Response Time vs. Threads
Confidential, Dynatrace LLC
Increasing MemoryAvg Response Time: Increases over Time
Max Response Time correlates with # Threads
Max Response Time correlates with # Threads
Load: System can’t handle constant load over time
# of Worker Threads increase with constant load
Layer Breakdown: Performance Problem is in their own code (Sync Time)
![Page 30: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/30.jpg)
Confidential, Dynatrace LLC
96% Synchronization TimeMonitor.Enter: By far biggest
performance contributor
Called from GetStatus and _GetDebugStatus. Why also from Debug method? Bug?
![Page 31: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/31.jpg)
Confidential, Dynatrace LLC
Excessive file attribute access second hotspot
Can FileExist check be cached to reduce
File Access?
File Access next hotspots
![Page 32: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/32.jpg)
Confidential, Dynatrace LLC
• # of Threads vs # Incoming Requests
• Time Spent in Sync• Time Spent in I/O
Tips To Apply
![Page 33: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/33.jpg)
Confidential, Dynatrace LLC
Sharepoint Example
![Page 34: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/34.jpg)
Confidential, Dynatrace LLC
Frustrated user report bad response times
Frustrated User
Slow Page Load caused by Browser JS Time
Slow Page Load caused by Server-Side Processing
![Page 35: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/35.jpg)
Confidential, Dynatrace LLC
Really slow page6.8s to deliver Default.aspx page
Involved Web Parts
Most of the Time spentIn waiting
![Page 36: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/36.jpg)
Confidential, Dynatrace LLC
WebPart uses multiple parallel threads
Async Threads are busy with I/O
![Page 37: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/37.jpg)
Confidential, Dynatrace LLC
First remote call is very slow
Web Service call by ContentEditorWebPart
HttpWebRequests uses ServicePoint internally
First Web Serivce Requests takes 5.8s to return
![Page 38: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/38.jpg)
Confidential, Dynatrace LLC
Thread limit lets all other threads wait!
We have 10 parallel calls in our background threads
The other background threads spend their time“waiting” in the ServicePoint
![Page 39: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/39.jpg)
Confidential, Dynatrace LLC
Solution: Change Defaults
![Page 40: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/40.jpg)
Confidential, Dynatrace LLC
• Review installed IIS modules• Pay attention to IIS/ASP.net metrics• Optimize thread synchronization• Reduce I/O • Analyze SQL statements• Use DB Connection Pool Optimally
Tips To Apply
![Page 41: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/41.jpg)
Confidential, Dynatrace LLC
Free Trial: bit.ly/dttrial
![Page 42: Troubleshooting ASP.NET and IIS Scalability Hotspots](https://reader035.fdocuments.us/reader035/viewer/2022062400/58ae28271a28ab557e8b45d1/html5/thumbnails/42.jpg)
Thank you!
Time for Q&A