Dealing With Leaks

Dealing with Leaks

What is a resource leak ?

• Repeated allocation of a resource without timely deallocation

Probable resources:• memory

virtual memory, non-paged pool memory

• address space• OS objects / handles

files, threads, events, semaphores, sockets, windows, timers, etc.

• disk space• application level / non OS recourses e.g. database records

Unreferenced resources

int global = 20;

void foo(){

int b = 22;HANDLE h = CreateEvent(....);

int * int_ptr = int(1000); int * array_ptr = new int[1024];}

Unreferenced heap memoryProcess / User memory

static storage (fixed size)

stackgrows ondem and

stack (one per thread)

? ar? n mod

1024 e lements(4096 bytes)

0xDE ADBEAF

0x17E D54230xD EA DBEAF

0x1 7E D5423

heapgrows ondem and

Kerne l / O S memory

0x80101234

0x80101DC0 event ob ject0x801013580x38C

int global = 20;

void foo(){ int b = 22; HANDLE h = CreateEvent(...); int * int_ptr = int(1000); int * array_ptr = new int[1024];

execution is here}

int main(){ int a = 21; foo();

return EXIT_SUCCESS;}

Unreferenced heap memoryProcess / User memory

static storage (fixed size)

? ar? n mod

1024 e lements(4096 bytes)

0xD EA DBEAF

0x1 7E D5423

heapgrows ondem and

Kerne l / O S memory

0x80101234

0x80101DC0 event ob ject0x801013580x38C

int global = 20;

void foo(){ int b = 22; HANDLE h = CreateEvent(...); int * int_ptr = int(1000); int * array_ptr = new int[1024];

int main(){ int a = 21; foo(); execution is here return EXIT_SUCCESS;}

Unbounded resource consumptionstd::map<unsigned, transaction_t> transactions;

void do_transaction(P1_t p1, P2_t p2)

transactions[id] = transaction_t(p1,p2);

C++, avoiding the problem

• encapsulate ownership within objects or smart pointersbstr_t CComPtr shared_ptr auto_ptr scoped_ptr auto_handle std::string, avoid BSTR, char*

• only use raw pointers to implement tightly encapsulated objects,e.g. when implementing shared_ptr

• never use raw pointers to transfer ownership• never use raw pointers with business logic

exception: indication optional objects - use of NULL• Use a garbage collector• Problem:

No language enforcement -- trusts the programmer

VB, avoiding the problem

• Stay within Visual Basic 5/6 environment (avoid C / OS APIs).

• Release objects (with events) in “unload” methods

set some_object = nothing

• Forms reference objects (with events) and the objects can reference the form. This is a cyclic reference.

• VB7 is implemented on .NET only and garbage collected.

Java, .NET, avoiding the problem

• Stay within JVM or .NET VM (avoid C / OS APIs).You have no destructors, and finalizers are restrictive.

• Cyclic references -- no problem they are garbage collected.

• Don’t get lax, watch out for unbounded resource consumption.

Garbage collection - the solution

Process / U se r memory

static sto rage (fixed size)

managedheap

0x17E D 5423

0x17E D5423

heapgrows ondem and

17ED78AC

0x17E D78A C

garbage

• pause the entire program

• find every object that can be reached from the static storage or thread stacks

• mark those objects with an “age”

• delete all older objects, in any order you like

Garbage collection - the solution

Process / U se r memory

static sto rage (fixed size)

managedheap

0x17E D 5423

0x17E D5423

17ED78AC

0x17E D78A C

• pause the entire program

• find every object that can be reached from the static storage or thread stacks

• mark those objects with an “age”

• delete all older objects, in any order you like

Garbage collection - the penalties• Pause the entire program!

When?For how long?

• Find EVERY reachable object?Page in every object from disk!Must run in physical memory to be efficient.Working set will be same as used memory!

• Delete all the garbage, in what order?Any order!So how do I test deterministically?So I cannot use finalizers?

• These may or may not be acceptable compromises• Remember it only solves half the problem!

Workarounds• Watch dogs

Who watches the watch dog?What can the watch dog do? Bark!

• COM+Memory and lifetime limits can be configured for COM servers

• Heap limitsMaximum heap size can be configured

• WRITE GOOD CODE!• FIND THE LEAKS!

Available tools

• Bounds checker• Purify• Heap Agent• Great Circle• Lint• Parasoft• C++ debug heap

Typical restrictions

• Crashes itself on large scale programs• Crashes large scale programs• Only reports unreferenced memory not unbounded allocations• Only reports malloc, free etc• Does not report OS API leaks like SysAllocString• Only supports C++• Requires recompilation• Reports many false positives• Requires many hours to run• Grinds target program to a halt

perfmon

Identifying the problem

• Monitor resourcesperfmon “process private bytes”

“process handle count”“process thread count”

taskmgrVM Size, Handle count, Thread count, GDI Objects

• Exercise the program• Coverage test -- does it leak?• Use case tests -- when does it leak (most)?

Quantifying the problem

time and/or number of use cases

custom eracceptab lerestart tim e

sim ilar to MT BF

m ax acceptab lereso urce usage

• Must stay out of the red

• Must consistently reach the green, without crashing

• The restart period might be... 1 hour 1 working day 1 working week 1 year 1000 transactions never!

• Max resources might be... 100 KB 2 GB

Needle in a haystack• 3 processes• 250 component DLLs• 3500 source files• 1,500,000 lines of source code• 100 KB of leaks per transaction• exercising 15% of the code

great only 225,000 lines of source to check!• 10,000 allocations per transaction• caused by probably no more than half a dozen lines of code

I have hidden a few leaks in the 1,500,000 lines of D3000 code.We are going to find them NOW!

Ignore one time initialization

time and/or number of use cases

custom eracceptab le

restart tim esim ilar to M T BF

m ax acceptab leresource usage

without one-timein itia lization leaks

problem isthe slope

with one-timein itia lization leaks

• Ignore the first few transactions

• They might cause one-time initialization leaks, but who cares?

Cycle and observe everything

• Choose a small prime number

• Perform this number of transactions or cycles

• Track all allocations and deallocations

cycles / transactions

leaks percycle

cycles toignore

cycles tomonitor (7) cycles to

continue

Cycle again observing deallocationsstrtype a = b;

strtype & strtype::operator=(const strtype & rhs){ if (m_len < rhs.m_len) { char * temp = new char[rhs.m_len]; delete m_p; m_p = temp; } m_len = rhs.m_len; memmove(m_p, rhs.m_p, m_len); return *this;}

• Allocations made in the last cycle, may not be deleted until the next

• Perform additional cycles, only tracking deallocations

Leak Browser learns

• Now generate a report of the leak• Leak Browser has just “learnt”, what you are interesting in• Next time Leak Browser will examine these few areas of code in

greater detail (requiring more CPU)

• Clear the leaks• Repeat 7 monitoring cycles• Repeat some additional cycles• Regenerate the report

Not everything is a leak

• Background tasks may provide “background noise”for example, incoming news or price updates

• Caches may delay some deallocations• Queues and logs may include records of the last 1000

transactions

Capabilities

• Resource profiling - finding the bloat• Fast over 10,000 allocations per second can be tracked• Tracks about 300 key APIs, not just malloc & free• Finds both unreferenced leaks and unbounded leaks• Does not require code recompilation• Is not intrusive with large applications• Remote installation and debugging• It works!

How does it work?• Traps key APIs• API is redirected

into Leak Browser stub

• Stack trace is recorded with allocation details

• In-memory database is maintained

• If no PDB further instrumentation can be done

Core OS

C/C++ runtime

W in32

Your code

Challenges• Must be thread safe but...

– Cannot hold a thread lock and call the OS– Cannot allocate memory– Restrictions on using thread local storage

(it may not be allocated yet, or it may have been deallocated)

• Has only micro-seconds to trace the stack– Standard Microsoft algorithm takes seconds– Cannot afford to call the OS

• Must cope with optimized code– for stack tracing, and instrumentation– must modify hundreds or thousands of functions without side effects

Limitations

• Need to be able to cycle the application• No true garbage detection (yet)• .NET interop only (so far)• No JVM support - none planned• Slow display -- too much HTML• Occasional crashes, but more reliable than the rest

Summary

• Use garbage collection where appropriate• Use best practice for the programming language of choice• Monitor resource usage during development• Are you within your customer acceptable restart period?• If not

– Leak Browserwww.leakbrowser.com

Dealing With Leaks

Documents

Transcript of Dealing With Leaks

Dealing With Conflicts

CS 326A: Motion Planning Kynodynamic Planning + Dealing with Moving Obstacles + Dealing with Uncertainty + Dealing with Real-Time Issues.

Dealing with distractions

Dealing With Competition

Finding & Dealing with Mould in your Homeqamtraining.net/docs/english/Finding and Dealing with...When water leaks or spills occur indoors - act quickly. If wet or damp materials or

Dealing with Difficu..

Dealing with children

Dealing with water leaks - Crawley GOV

Dealing with infidelity

Dealing with civil contingencies: emergency planning …researchbriefings.files.parliament.uk/documents/CBP-8016/CBP-8016.pdf · Dealing with civil contingencies: ... 4 Dealing with

Dealing with envy

Dealing Data Leaks: Creating Your Data Breach Response Plan

Dealing with Crawlers

Dealing With Problems

Dealing with distraction

Dealing with Depression in Rural Australia Dealing with Depression in Rural Australia Dealing with Depression in Rural Australia.

Dealing With Difficult People - Workbook With Difficult...“When dealing with people, ... your work. The Volatile Volumizer 3. ... Dealing With Difficult People Workbook

Dealing with competition1

Quantifying Leaks with Acoustic Loggers

Recognizing Leaks and Problems...Recognizing Leaks and Problems CAA Seminar –Increasing Your Property Value March 23, 2004. ... handling, using, distributing, or dealing in any material