Demand Technology Software, Inc.

Memory LeaksDemand Technology

1020 Eighth Avenue South, Suite 6, Naples, FL 34102phone: (941) 261-8945 fax: (941) 261-5456

e-mail: markf@demandtech.comhttp://www.demandtech.com

2Demand Technology, Inc. Memory Leaks

Outline Topics

Virtual Memory concepts

Virtual Memory constraints in 32-bit Windows

Leaking processes

Page faults/sec

Vir t ualaddress

space

V ir t ualaddress

space

P hysical M em ory

Page Tables

3Demand Technology, Inc. Memory Leaks

Virtual Memory Manager

4 GB virtual per process address space Lower 2 GB - Private

Page 0 reserved Code pages Heap

Upper 2 GB - System System code Shared dlls System cache

User

System

0000 0000

7fff ffff16

ffff ffff16

4Demand Technology, Inc. Memory Leaks

UserUser

Virtual Memory Manager

All address spaces share the virtual addresses from the same upper 2 GB of the System area

All virtual addresses in the lower 2 GB User area are unique

Shared memory in the System range can be used for IPC

User

System

0000 0000

7fff ffff16

ffff ffff16Shared

5Demand Technology, Inc. Memory Leaks

Virtual Memory Manager

Boot.ini /3 GB switch Virtual storage

constraint relief for some applications

SQL Server Exchange 2000 Etc.

0000 0000

c000 0000 16

ffff ffff 16

User

System

Shared

6Demand Technology, Inc. Memory Leaks

AWE

Windows 2000 - Address Windowing Extensions Address RAM > 4 GB

limit -Intel Pentium III, IV

requires PSE36 Driver Real addresses only Not backed on paging

files– PTEs remain 32-bit 0000 0000

c000 0000 16

ffff ffff 16

User

System

Shared

PSE36

0f ffff ffff 16

AW

E

7Demand Technology, Inc. Memory Leaks

Win64 Virtual Memory

Architectural component 64 bit 32 bit

Virtual Memory 16TB 4GB

Paging File Size 512 TB 64 GB

Hyperspace 8 GB 4 MB

Paged Pool 128 GB 192 MB

Non-paged Pool 128 GB 128 MB

System cache 1 TB 1 GB

System PTE (page table entries) 128 GB 660 MB

8Demand Technology, Inc. Memory Leaks

Virtual Memory Manager

Real Memory allocation Counters Available Bytes Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes

Instantaneous Counters (all reported as Bytes)

Cache Bytes is actually the pageable System Working Set = Pool Paged Resident Bytes + System Cache Resident Bytes

+ System Code Resident Bytes + System Driver Resident

Bytes

9Demand Technology, Inc. Memory Leaks

Virtual Memory Manager

Memory allocation Counters Available Bytes Available KBytes Available MBytes

Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes

10Demand Technology, Inc. Memory Leaks

What is the difference between Pool Paged Bytes and Pool Nonpaged Bytes?

Which applications would use which VM pool?

11Demand Technology, Inc. Memory Leaks

Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F

PTEs/ cache extension

System data cache

Paged system pool

80000000

C0000000

Virtual Memory map

12Demand Technology, Inc. Memory Leaks

Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F

PTEs/ cache extension

System data cache

Paged system pool

80000000

C0000000

Virtual Memory map

Paged Memory mapped

files Shared DLLs

File Server IIS (html, jpg, gif)

Page tables

nonPaged: I/O buffers used by

device drivers TCP Session data

13Demand Technology, Inc. Memory Leaks

Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F

PTEs/ cache extension

System data cache

Paged system pool

80000000

C0000000

Virtual Memory map

Virtual memory map is static, created once during OS initialization

When any one area is full, memory allocations to that pool will fail

One reserved area of virtual memory can fill up faster than the others;

14Demand Technology, Inc. Memory Leaks

Nonpaged pool size (ref. Q126402):

MinimumNonPagedPoolSize = 256KMinAdditionNonPagedPoolPerMb = 32KDefaultMaximumNonPagedPool = 1 MBMaxAdditionNonPagedPoolPerMb = 400KPTE_PER_PAGE = 1024PAGE_SIZE=4096

NonPagedPoolSize = MinimumNonPagedPoolSize + ((Physical MB - 4) * MinAdditionNonPagedPoolPerMB)

If NonPagedPoolSize > 128 MB THEN NonPagedPoolSize = 128 MB

15Demand Technology, Inc. Memory Leaks

Paged pool size:

Size = (2 * MaximumNonPagedPoolSize) / PAGESIZE

Size = (Size + (PTE_PER_PAGE - 1)) / PTE_PER_PAGE

PagedPoolSize = Size * PAGESIZE * PTE_PER_PAGE

If PagedPoolSize > 192 MB THEN PagePoolSize = 192 MB

Note: PagedPoolSize sizeof (RAM) for systems < 192 MB

16Demand Technology, Inc. Memory Leaks

Default Paged and Nonpaged pool size can be overridden:

17Demand Technology, Inc. Memory Leaks

Virtual Memory constraints

Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM) 4 GB 2 GB private area is not enough virtual memory for some

applications e.g., SQL Server, Exchange database (store.exe) Due to fragmentation, it is typically not possible to allocate all 2 GB

2 GB system area is not enough virtual memory for some applications

File cache for a conventional IIS-managed web site with many static .htm, jpg, gif, etc., files to retrieve

18Demand Technology, Inc. Memory Leaks

Virtual Memory constraints

Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM) 4 GB Ample RAM exists, but it is not possible for your

applications to access it due to virtual memory addressing limitations

Large number of Available Bytes

19Demand Technology, Inc. Memory Leaks

Virtual Memory constraints

So try the /3 GB switch 1 GB system area is not enough virtual memory for some

applications Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool

– Where Session data from TCP connections is stored

Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS)

During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time

20Demand Technology, Inc. Memory Leaks

Exchange 2000 memory tuning Exchange default memory allocation

parameters are “self-tuning,” but may not be optimal on servers with > 1 GB RAM Adjust HKLM\SYSTEM\CurrentControlSet\Services\

SMTPSVC\Queuing MsgHandleThreshold & MsgHandleAsyncThreshold

HKLM\SYSTEM\CurrentControlSet\Services\Inetinfo\Parameters

FileCacheMaxHandles

21Demand Technology, Inc. Memory Leaks

Exchange 2000 memory tuning Store DB cache

Store cache normally expands to fill RAM But, maximum cache = 900 MB This value can be adjusted using ADSI Edit

tool: msExchESEParamCacheSizeMax msExchESEParamCacheSizeMin

Also, consider adjusting msExchESEParamLogBuffers attribute for active, back-end servers

22Demand Technology, Inc. Memory Leaks

Detecting memory leaks Processes that allocate virtual memory, but

later “forget” to free it. MS says leaks won’t happen in .Net

managed code due to automatic garbage collection

But, meanwhile…, Where to look depends on whether process

or system addresses are being allocated: Per Process: Virtual Bytes, Private Bytes, Pool Paged

Bytes, Handle Count System level: Memory Pool Paged Bytes, Pool Nonpaged

Bytes and the Objects Object

23Demand Technology, Inc. Memory Leaks

Detecting memory leaks

Look for a steady increase or a sharp spike in process Virtual Bytes, or the System’s Pool Paged Bytes. If RAM is not full, the leak may also be manifest

in the Memory allocation counters and result in increased paging, if RAM fills up.

For example:

24Demand Technology, Inc. Memory Leaks

Detecting memory leaks (1)Memory Leak example

0

50

100

150

200

250

300

350

400

Time of Day

MB

A vailable Bytes N on-paged Pool BytesPool Paged Resident Bytes System Cache Resident BytesSystem Code Resident Bytes System Dr iver Resident Bytes

25Demand Technology, Inc. Memory Leaks

Detecting memory leaks (2)

26Demand Technology, Inc. Memory Leaks

Detecting memory leaks (2)

27Demand Technology, Inc. Memory Leaks

Detecting memory leaks (2)

28Demand Technology, Inc. Memory Leaks

Detecting memory leaks (2)

29Demand Technology, Inc. Memory Leaks

Lab exercise. Open

SUPPORT.200205080300.EXCHG2K.sum.smf, SUPPORT.200205090300.EXCHG2K.sum.smf, and SUPPORT.200205100300.EXCHG2K.sum.smf and investigate virtual memory usage.

Can you find the application that is leaking?

30Demand Technology, Inc. Memory Leaks

Detecting Memory Leaks (3)

31Demand Technology, Inc. Memory Leaks

Detecting Memory Leaks (3)

32Demand Technology, Inc. Memory Leaks

Detecting Memory Leaks (3)

33Demand Technology, Inc. Memory Leaks

Detecting Memory Leaks (3)

34Demand Technology, Inc. Memory Leaks

Page replacement

Theoretically, Page Faults/sec

Cache faults/sec + Transition faults/sec + Demand Zero Faults/sec + Pages Read/sec

But empirically…

35Demand Technology, Inc. Memory Leaks

Page Faults/ sec

0

300

600

900

1200

1500

Time of Day

Pag

es/s

ec

0

300

600

900

1200

1500

cache faults/ secpage reads/ sect ransit ion faults/ secdemand zero faults/ secpage faults/ sec

36Demand Technology, Inc. Memory Leaks

Page replacement

Empirically, Page Faults/sec

Transition faults/sec + Demand Zero Faults/sec + Pages Read/sec

37Demand Technology, Inc. Memory Leaks

Questions

?