Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City...
-
Upload
winfred-henry -
Category
Documents
-
view
217 -
download
2
Transcript of Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City...
![Page 1: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/1.jpg)
Heap Growth Detection in C++GrowthTracker
1
![Page 2: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/2.jpg)
Heap Growth Detection in C++Motivation
• Scalable City needs to run continuously– Many months without intervention/access– Had slow growth of memory
• Caused crash after several weeks
– Available analysis tools reported no leaks!• Software frees all memory correctly!• Had different kind of undetected memory issue
2
![Page 3: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/3.jpg)
What is a Memory Leak?
3
![Page 4: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/4.jpg)
What is a Memory Leak?
• Has become a broad term for memory mismanagement• Definition depends on the programming language
– Does it have Garbage Collection?
4
![Page 5: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/5.jpg)
What is a Memory Leak?
• C/C++ Programmers generally subscribe to traditional definition:
Def. 1: A memory leak occurs iff all references to a region of memory are lost before the memory is freed.
5
![Page 6: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/6.jpg)
What is a Memory Leak?
• Java/C# Programmers generally have a different definition:
Def. 2: A memory leak occurs when a program’s memory is unexpectedly growing.
– Using Def. 1, Java/C# don’t have memory leaks• Garbage Collector frees memory that is no longer referenced
6
![Page 7: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/7.jpg)
What is a Memory Leak?
7
Def .2(unbounded growth)
Def .1(lost refs)
C/C++ Java/C#
Def .2(unbounded growth)
Def .1(lost refs)
![Page 8: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/8.jpg)
What is a Memory Leak?
• We need new terminology!– Java/C# programmers
• Concerned with unbounded memory growth, not leaks– References exist to all memory growth
• Have clearer understanding of their problem
– C/C++ programmers• Multiple definitions causes confusion• False sense of security when tools report no memory
leaks– Unbounded memory growth may still exist!
• A term for unbounded memory growth not caused by memory leaks, would clearly parse the 2 problems
8
![Page 9: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/9.jpg)
New “Leak” Terminology
• Properties of new term– Unbounded Heap Growth– Reference(s) retained– Manifests as dynamically growing data structure– Will eventually kill the process– Doesn’t fit leaky pipe metaphor
• Memory Tumor– Structure of cells that exhibits unbounded growth
9
![Page 10: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/10.jpg)
Memory Tumor Example
void main(){
queue q;
while( key != ESC ) // exits when ESC key pressed
q.push(0);
}
• q will grow constantly while the program runs• there are no memory leaks
– if the user hits ESC, all memory is freed when q goes out of scope
10
![Page 11: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/11.jpg)
Leak vs. Tumor
11
X
X
X
X
X
Leak Tumor
heap heap
![Page 12: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/12.jpg)
Separation of Concerns
12
Memory Leak(Def 1: lost refs)
C/C++/Java/C#
Memory Tumor
![Page 13: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/13.jpg)
Current C++ Tumor Detection Tools
• SWAT and Hound– Only ones we’re aware of– Both use Staleness detection
• Misses some tumors by design• Memory can be accessed, but not needed
– Investigate at allocation level• Don’t need to modify source code
– Not open source or commercially available
13
![Page 14: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/14.jpg)
Memory Tumor Detection Theory
• Tumors – Data Structures that grow without bound
• Healthy data structures– Will grow– Maximum size stabilizes
14
![Page 15: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/15.jpg)
Memory Tumor Detection Challenges
• Detect all growth that doesn’t stabilize – Don’t dismiss non-stale growth
• Tests must exhibit the growth that exists in a program’s implementation
• Support Multithreaded programs
15
![Page 16: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/16.jpg)
Memory Tumor Detection Approach
• Growth Tracker Tool– Container Tracking
• Keep references to all data structures in memory
– Growth Tracking• Track data structure size changes over time• Identify those with unbounded growth
• Automated Test– Cyclically execute all code paths (user created)
16
![Page 17: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/17.jpg)
Growth Tracker Tool
• Container tracking– CAT (Central Aggregate Tracker)
• Maintains references to all aggregates in the system– Create wrappers for each aggregate type in
system• Templated constructors, multiple inheritance• Add to CAT on construction, remove on destruction
– Namespace replacement to enable wrappers• Find and replace to apply new namespace• Wrappers disabled with compile time flag• Example: trak::std::vector<int>
17
![Page 18: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/18.jpg)
Growth Tracker Tool
• Growth Tracking– Take periodic samples of the CAT– Exponentially increasing interval sizes
• Reduces false positives & negatives over time– Report growing aggregates at each sample
18
![Page 19: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/19.jpg)
Growth Tracking Heuristic
• Take periodic samples of the CAT• Two Interval Analysis
– 1st interval establishes aggregate age, gives time to stabilize– 2nd interval proves stability, non-tumors shouldn’t grow– 2nd interval becomes the 1st for next more accurate test
• Exponentially increasing interval sizes– Reduces false positives & negatives over time
• Monitor size maximums– Reduces size fluctuation false positives
• At each interval report all aggregates that:– Increased their size maximum– Have existed for two full intervals
• Prioritize results by size & reporting frequency
19
![Page 20: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/20.jpg)
Growth Tracking Heuristic
• Two interval analysis
20
time
memory
1 2 3 4
Not reported(growth stabilized)
A Data Structure Memory Footprint
![Page 21: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/21.jpg)
Diagnosing Unbounded Heap Growth in C++ Detection Approach
• Two interval analysis
21
time
memory
1 2 3 4
Reported as tumor(false positive)
Not reported(growth stabilized)
![Page 22: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/22.jpg)
Growth Tracking Heuristic
• Take periodic samples of the CAT• Two Interval Analysis
– 1st interval establishes aggregate age, gives time to stabilize– 2nd interval proves stability, non-tumors shouldn’t grow– 2nd interval becomes the 1st for next more accurate test
• Exponentially increasing interval sizes– Reduces false positives & negatives over time
• Monitor size maximums– Reduces size fluctuation false positives
• At each interval report all aggregates that:– Increased their size maximum– Have existed for two full intervals
• Prioritize results by size & reporting frequency
22
![Page 23: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/23.jpg)
Diagnosing Unbounded Heap Growth in C++ Detection Approach
• Growth Tracking– Exponentially increasing interval size
23
time
memory
1 2 3 4
In this example: constant intervals would not report growth half the time
![Page 24: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/24.jpg)
Growth Tracking Heuristic
• Take periodic samples of the CAT• Two Interval Analysis
– 1st interval establishes aggregate age, gives time to stabilize– 2nd interval proves stability, non-tumors shouldn’t grow– 2nd interval becomes the 1st for next more accurate test
• Exponentially increasing interval sizes– Reduces false positives & negatives over time
• Monitor size maximums– Reduces size fluctuation false positives
• At each interval report all aggregates that:– Increased their size maximum– Have existed for two full intervals
• Prioritize results by size & reporting frequency
24
![Page 25: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/25.jpg)
Diagnosing Unbounded Heap Growth in C++ Detection Approach
• Growth Tracking– Max size variable
25
1 2 3
time
memory
ceiling
Growth would be reported without
max size
![Page 26: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/26.jpg)
Growth Tracking Heuristic
• Take periodic samples of the CAT• Two Interval Analysis
– 1st interval establishes aggregate age, gives time to stabilize– 2nd interval proves stability, non-tumors shouldn’t grow– 2nd interval becomes the 1st for next more accurate test
• Exponentially increasing interval sizes– Reduces false positives & negatives over time
• Monitor size maximums– Reduces size fluctuation false positives
• At each interval report all aggregates that:– Increased their size maximum– Have existed for two full intervals
• Prioritize results by size & reporting frequency
26
![Page 27: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/27.jpg)
Growth Tracker Targets
• Multi-threaded Applications– Initial CAT implementation works
• Requires locking for each aggregate constructor• Potential to diminish multi-threaded performance• Good starting point
– Need new CAT implementation• Eliminate Locks
– Multiple bucket approach – Map aggregate construction from different threads to buckets
• Design can accelerate sampling process as well
27
![Page 28: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/28.jpg)
Growth Tracker Drawbacks
– Source code modification• Tracking requires compilation with our wrappers• Allows consideration of Objects not just allocations.
– Limited information about identified tumors• Full type string & allocation number• Code location possible
– requires stack tracing (slower)
– Reliance on the user• Must identify custom data structures• Must run feature complete and cyclic test
28
![Page 29: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/29.jpg)
Growth Tracker Drawbacks
– Multi-threaded potential slow down– Persistent buckets
• Example: Linear hash table with std::vector buckets
• More useful to include child bucket sizes in parent’s output and stop reporting individual children
– Multiple instances of same tumor reported• Parent report including children would resolve
29
![Page 30: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/30.jpg)
Growth Tracker Results
– Scalable City• Identified tumor• Eliminated memory growth
– Ogre3D Rendering Engine• Identified 2 tumors• Our fix integrated into their code base
– Bullet Physics Engine• Tests revealed no tumors in Core• 1 tumor found in demo framework
30
![Page 31: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/31.jpg)
Growth Tracker Results
– Google Chrome / Chromium• Identified 21 tumors• Fixed the fastest growing tumor ourselves
– WebKit (Safari Browser, etc.)• Identified 2 tumors• Submitted fix to code base
31
![Page 32: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/32.jpg)
Growth Tracker Paper
• Recently accepted for publication• IEEE International Conference on
Software Testing, Verification and Validation (ICST 2013)
32
![Page 33: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/33.jpg)
Growth Tracker Proposed Work
– Resolve Multithreaded locking limitations• Solution designed, needs implementation
– Reduce tracking of temporaries• Detect stack-based data structures• Multi-layer CAT to separate entries by age
– Will reduce overhead of CAT insertion/removal
33
![Page 34: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/34.jpg)
Growth Tracker Proposed Work
• Automation– Reduce reliance on the user– Detect custom data structures
• Automatically create wrappers when possible
– Improvements to code transformation process• After initial code conversion, detect when wrapper
is forgotten.
34
![Page 35: Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cfa5503460f949cc3b2/html5/thumbnails/35.jpg)
Growth Tracker Proposed Work
• Prioritize tracking parent data structures– Would address persistent bucket problem– Would reduce reports of multiple instances of
same tumor– Must identify relationships between data
structures.
35