Application Profiling for Memory and Performance
-
Upload
pradeepfn -
Category
Technology
-
view
1.056 -
download
0
description
Transcript of Application Profiling for Memory and Performance
![Page 1: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/1.jpg)
Application Profiling for
Memory & PerformanceSrinath Perera
WSO2 IncPradeep Fernando
WSO2 IncDinuka Malalanayke
WSO2 Inc
![Page 2: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/2.jpg)
Understanding Server Performance
![Page 3: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/3.jpg)
When Concurrency Increases?
• Throughput increases, but where will it stop?• When there are contention for the server• Contention for what?
o CPUo Memoryo Disk - I/Oo Network -I/O
• If your server running on full capacity, there should be at least one resource under contention.
![Page 4: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/4.jpg)
How to measure Contention?
• CPU - CPU or load average • Memory - GC logs, monitor memory, sometime swapping • Disk I/O - IOPS per seconds, load average • Network I/O - network utilization, load average • What is load average? length of processor queue of the OS• How much load average is too much?
o Load average = number of cores (machine is fully used)o Load average = 2* number of cores (machine is loaded)
![Page 5: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/5.jpg)
If you cannot find contention
• Either load is too small • You have too much locks. Look for lock profile • Server might be processing with too less threads (IO and
CPU thread switches)
![Page 6: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/6.jpg)
Performance Tuning
• Increase throughput/ latency or both • Know that latency vs. throughput is often a tradeoff• Run the system and see how it does, make sure you are
putting in enough load• Verify where is the bottleneck? is it CPU, memory, network,
or disk • Start tuning
![Page 7: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/7.jpg)
Tuning
• When you tune, bottlenecks will shift, you will have to shift your focus
• Can you tune settings (Application, JVM, OS) to improve and shift the bottleneck? play with parameters .. we call this parameter sweep
• Then focus on the code, look for right profileo CPU profile o Memory/ Allocation profile o Network profile o IO profile o Database profile
![Page 8: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/8.jpg)
Programmer Nightmares
![Page 9: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/9.jpg)
Permgen Errors
CPU spin
DeadLocks
Unacceptable LatencyNullPointer
ArrayIndexOutOfBoundsException
ClassCastException
Programmer Nightmares
[image] http://www1.free-clipart.net/gallery2/clipart/Holidays/Halloween/Ghost_-_Boo.jpg
OutofMemoryException
![Page 10: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/10.jpg)
Introducing: Profiling...
• Best thing would be to write bug free, high performant code• Normally development happens in iterative manner - get
something to work and improve on that.• Basic profiling can help you to eliminate most of the memory
and performance issues.
![Page 11: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/11.jpg)
Right Tools...
• There are many commercial and open source tools• We are going to use,
o Jprofiler by ejTechnologies CPU, Memory, Allocation, Network, DB
o Eclipse Memory Analyzer tool Find memory leaks
o Standard JDK tooling - Jconsole Basic stats
[image] http://writingsongs.com/pictures/toolbox.jpg
![Page 12: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/12.jpg)
Environment Setup
With WSO2 Carbon
![Page 13: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/13.jpg)
Setting up JProfiler
• Profiling modes local/remote• Applying common filters for WSO2 products• Different views..
![Page 14: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/14.jpg)
JProfiler Contd.
CPU view - Call trees, Hot spotsMemory view - Allocation Hot spots, Recorded objects, Heap walkerVM telemetry view - Memory, Threads status graphsProbes - JDBC , Sockets
![Page 15: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/15.jpg)
JConsole
• Connect to the running instance local/remote• Different memory banks• Getting most out of Mbeans.• Getting a memory dump and analyzing with Eclipse MAT.• Remote JMX URL gets printed in the console during server
startup: JMX Service URL : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
username : admin password : admin
![Page 16: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/16.jpg)
JConsole
![Page 17: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/17.jpg)
Memory Analyzer Tool
• Analyze heap dumps.• Identifying OOM issues are trivial with basic leak report.
![Page 18: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/18.jpg)
Usual Suspects
Some of the common issues...
![Page 19: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/19.jpg)
Permgen Errors
• Permgen space relates to the 'Programme code' part, a.k.a - Loaded classes.
• ClassLoader leaks• Typically happens in Container environments• Can uncover by doing deploy/redeploy cycles.• If one of the application objects get referenced by an object,
outside the application, permgen error waiting to happen.
![Page 20: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/20.jpg)
Permgen Errors
Container
URLClassLoader Application
Application ObjectsContainer provided
Objects
![Page 21: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/21.jpg)
Memory Leaks - OOM
• Languages like C, gave the control of memory to the programmer.
• Java is a managed memory, language.• Still the JVM can't free up the memory, if the application is
holding on to the objects.• Unnecessary accumulation of objects,
o Object creation for each and every connectiono Slow output rate in the input/output system
o Collecting clusterwide messages without an upper bound.
![Page 22: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/22.jpg)
DeadLocks
• Re-designing the lock acquiring sequence• Using LockManagers that keeps track of lock acquisition
Resource A
Resource B
![Page 23: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/23.jpg)
Liveness issues...
• Starvation - A thread can starve for CPU cycles, if it is blocked by a lock-wait.
• Poor Responsiveness • LiveLock scenarios
![Page 24: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/24.jpg)
Programming Best Practices
• Think about memory aspects/ data retrieval process while writing code.
• Optimize the critical Path (look at CPU profile).• Make use of Standard libs as much as possible.• Profile your apps before releasing them to production.
![Page 25: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/25.jpg)
WSO2 make use of profiling ?
![Page 26: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/26.jpg)
Identity Server - Losing weight
• Carbon server optimized to run in constrained memory environment.
• No front end components - 'createWorker' ant task.• Selective Admin service loading. -Doptimized=true• Removed unwanted functionality based on the use-case,
(p2 - provisioning, etc)
![Page 27: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/27.jpg)
Raspberry-Pi Clusters
• Application server cluster running on R-Pis.
• Powering the WSO2Con 2013 mobile app
• 512MB of RAM and ARM processor.
![Page 28: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/28.jpg)
Demo
CPU spin and Memory leak scenario
![Page 29: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/29.jpg)
Questions
![Page 30: Application Profiling for Memory and Performance](https://reader033.fdocuments.us/reader033/viewer/2022061223/54c6427d4a7959ba0b8b4625/html5/thumbnails/30.jpg)
Thank You