Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ......
Transcript of Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ......
![Page 1: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/1.jpg)
Asymmetric NUMA:Multiple-memory management for the rest of us
Paul [email protected]
Magnus [email protected]
Renesas Technology
ELC-Europe 2007
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 1 / 30
![Page 2: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/2.jpg)
Outline
Introduction to NUMAUniprocessor and BeyondFrom UMA to NUMASymmetric and Asymmetric NUMANUMA for Embedded Systems
Linux Kernel Memory ManagementSingle Node BasicsMultiple NodesAsymmetric NUMA
Application Node ControlMemory Policiestmpfs and cpusets
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 2 / 30
![Page 3: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/3.jpg)
Introduction to NUMA
Outline
Introduction to NUMAUniprocessor and BeyondFrom UMA to NUMASymmetric and Asymmetric NUMANUMA for Embedded Systems
Linux Kernel Memory ManagementSingle Node BasicsMultiple NodesAsymmetric NUMA
Application Node ControlMemory Policiestmpfs and cpusets
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 3 / 30
![Page 4: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/4.jpg)
Introduction to NUMA Uniprocessor and Beyond
Uniprocessor and Beyond
Uniprocessor System (UP):I A single processor has all memory bus bandwidth to itself.
Symmetric Multi-Processor System (SMP):I Multiple processors share memory bus bandwidth.
Memory bus bandwidth becomes a bottleneck for large systems.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 4 / 30
![Page 5: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/5.jpg)
Introduction to NUMA Uniprocessor and Beyond
Uniprocessor and Beyond
Uniprocessor System (UP):I A single processor has all memory bus bandwidth to itself.
Symmetric Multi-Processor System (SMP):I Multiple processors share memory bus bandwidth.
Memory bus bandwidth becomes a bottleneck for large systems.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 4 / 30
![Page 6: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/6.jpg)
Introduction to NUMA From UMA to NUMA
From UMA to NUMA
Uniform Memory Access (UMA)
I Uniprocessor or Symmetric Multi-Processor configurations.I Memory is accessed through a high-speed local bus.
Memory access speed is constant for all processors and memories.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 5 / 30
![Page 7: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/7.jpg)
Introduction to NUMA From UMA to NUMA
From UMA to NUMA
Uniform Memory Access (UMA)
I Uniprocessor or Symmetric Multi-Processor configurations.I Memory is accessed through a high-speed local bus.
Memory access speed is constant for all processors and memories.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 5 / 30
![Page 8: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/8.jpg)
Introduction to NUMA From UMA to NUMA
Non-Uniform Memory Access (NUMA):
I A NUMA system is divided into multiple nodes.I Each node is equipped with zero or more processors.I Local memory may exist on the same node as the processor.I Remote memory access is provided through interconnects.
With NUMA the access speed for memory varies with the distancebetween processors and memory regions.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 6 / 30
![Page 9: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/9.jpg)
Introduction to NUMA From UMA to NUMA
Non-Uniform Memory Access (NUMA):
I A NUMA system is divided into multiple nodes.I Each node is equipped with zero or more processors.I Local memory may exist on the same node as the processor.I Remote memory access is provided through interconnects.
With NUMA the access speed for memory varies with the distancebetween processors and memory regions.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 6 / 30
![Page 10: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/10.jpg)
Introduction to NUMA Symmetric and Asymmetric NUMA
Symmetric and Asymmetric NUMA
Symmetric NUMA:I Equal amounts of memory is assigned to each node.
Asymmetric NUMA:I The amount of memory for each node may vary greatly.I Nodes may be equipped with memory but without processors.I Memoryless nodes - processor-only configurations.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 7 / 30
![Page 11: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/11.jpg)
Introduction to NUMA Symmetric and Asymmetric NUMA
Symmetric and Asymmetric NUMA
Symmetric NUMA:I Equal amounts of memory is assigned to each node.
Asymmetric NUMA:I The amount of memory for each node may vary greatly.I Nodes may be equipped with memory but without processors.I Memoryless nodes - processor-only configurations.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 7 / 30
![Page 12: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/12.jpg)
Introduction to NUMA Symmetric and Asymmetric NUMA
Symmetric and Asymmetric NUMA
Symmetric NUMA:I Equal amounts of memory is assigned to each node.
Asymmetric NUMA:I The amount of memory for each node may vary greatly.I Nodes may be equipped with memory but without processors.I Memoryless nodes - processor-only configurations.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 7 / 30
![Page 13: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/13.jpg)
Introduction to NUMA NUMA for Embedded Systems
NUMA for Embedded Systems
The existing NUMA interfaces in Linux are. . .I architecture-independent.I well-established.
. . . and they allow us to:I Select node memory to back file data with.I Select node for process memory ranges.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 8 / 30
![Page 14: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/14.jpg)
Introduction to NUMA NUMA for Embedded Systems
NUMA for Embedded Systems
The existing NUMA interfaces in Linux are. . .I architecture-independent.I well-established.
. . . and they allow us to:I Select node memory to back file data with.I Select node for process memory ranges.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 8 / 30
![Page 15: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/15.jpg)
Introduction to NUMA NUMA for Embedded Systems
NUMA for Embedded Systems
“...superh is starting to use NUMA now, due tovarying access times of various sorts of memory.one can envisage other embedded setups doing that”
- Andrew Morton, on linux-mm
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 9 / 30
![Page 16: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/16.jpg)
Linux Kernel Memory Management
Outline
Introduction to NUMAUniprocessor and BeyondFrom UMA to NUMASymmetric and Asymmetric NUMANUMA for Embedded Systems
Linux Kernel Memory ManagementSingle Node BasicsMultiple NodesAsymmetric NUMA
Application Node ControlMemory Policiestmpfs and cpusets
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 10 / 30
![Page 17: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/17.jpg)
Linux Kernel Memory Management Single Node Basics
Physical Memory
struct page mem_map[]
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 11 / 30
![Page 18: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/18.jpg)
Linux Kernel Memory Management Single Node Basics
Physical Memory
PFN - Page Frame Number
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 12 / 30
![Page 19: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/19.jpg)
Linux Kernel Memory Management Single Node Basics
Memory Models
One contiguous range of memory: CONFIG_FLATMEM
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 13 / 30
![Page 20: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/20.jpg)
Linux Kernel Memory Management Single Node Basics
Memory Models
More than one contiguous range of memory:CONFIG_SPARSEMEM or CONFIG_DISCONTIGMEM
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 14 / 30
![Page 21: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/21.jpg)
Linux Kernel Memory Management Single Node Basics
Physically Contiguous Allocators
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 15 / 30
![Page 22: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/22.jpg)
Linux Kernel Memory Management Single Node Basics
Physical Page Allocator
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 16 / 30
![Page 23: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/23.jpg)
Linux Kernel Memory Management Single Node Basics
Physical Page Allocator
Binary Buddy Allocator Algorithm
struct page *alloc_pages(gfp_t, unsigned int)void __free_pages(struct page *, unsigned int)
Order N allocations:
I Order 0 -> 1 page.I Order 1 -> 2 pages.I Order 2 -> 4 pages.I . . .I Order N -> 2 ^N pages.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 17 / 30
![Page 24: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/24.jpg)
Linux Kernel Memory Management Single Node Basics
SLAB Allocator
A caching object-based allocator.
Create and destroy a cache of objects:
struct kmem_cache *kmem_cache_create(...)void kmem_cache_destroy(struct kmem_cache *)
Allocate and free objects using the cache:
void *kmem_cache_alloc(struct kmem_cache *, gfp_t))void kmem_cache_free(struct kmem_cache *, void *)
/proc/slabinfo provides statistics.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 18 / 30
![Page 25: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/25.jpg)
Linux Kernel Memory Management Single Node Basics
SLAB Allocator
A caching object-based allocator.
Create and destroy a cache of objects:
struct kmem_cache *kmem_cache_create(...)void kmem_cache_destroy(struct kmem_cache *)
Allocate and free objects using the cache:
void *kmem_cache_alloc(struct kmem_cache *, gfp_t))void kmem_cache_free(struct kmem_cache *, void *)
/proc/slabinfo provides statistics.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 18 / 30
![Page 26: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/26.jpg)
Linux Kernel Memory Management Single Node Basics
SLAB Allocators (2.6.23)SLAB (1996)
I linux/mm/slab.c, ~4500 lines, CONFIG_SLABI Default allocator, per-cpu and per-node data.
SLOB (2003)
I linux/mm/slob.c, ~600 lines, CONFIG_SLOBI Simple and small, but with single free list.
SLUB (2007)
I linux/mm/slub.c, ~4100 lines, CONFIG_SLUBI Unqueued allocator, minimizes cache line usage.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 19 / 30
![Page 27: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/27.jpg)
Linux Kernel Memory Management Single Node Basics
kmalloc and kfree
A thin multi-purpose layer on top of the SLAB allocator.
void *kmalloc(size_t, gfp_t)void kfree(const void *)
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 20 / 30
![Page 28: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/28.jpg)
Linux Kernel Memory Management Multiple Nodes
Multiple Nodes (CONFIG_NUMA=y)
CONFIG_SPARSEMEM or CONFIG_DISCONTIGMEM required.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 21 / 30
![Page 29: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/29.jpg)
Linux Kernel Memory Management Asymmetric NUMA
Asymmetric NUMAPhysical Page Allocator
I Kernel allocates from all nodes during initialization.I 16MB cut-off added.
I Kernel defaults to node-local allocations during run time.I Use node 0 for System RAM.
SLAB Allocators
I SLABI SLUB
I Requires patches to exclude small nodes.I SLOB
I Low overhead, preferred SLAB Allocator for Asymmetric NUMA.I Primitive locking - possible performance issues for SMP systems.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 22 / 30
![Page 30: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/30.jpg)
Linux Kernel Memory Management Asymmetric NUMA
Asymmetric NUMA
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 23 / 30
![Page 31: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/31.jpg)
Linux Kernel Memory Management Asymmetric NUMA
Asymmetric NUMA - Overhead
/ # cat /sys/devices/system/node/node1/meminfo
Node 1 MemTotal: 128 kBNode 1 MemFree: 72 kBNode 1 MemUsed: 56 kBNode 1 Active: 0 kB...
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 24 / 30
![Page 32: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/32.jpg)
Application Node Control
Outline
Introduction to NUMAUniprocessor and BeyondFrom UMA to NUMASymmetric and Asymmetric NUMANUMA for Embedded Systems
Linux Kernel Memory ManagementSingle Node BasicsMultiple NodesAsymmetric NUMA
Application Node ControlMemory Policiestmpfs and cpusets
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 25 / 30
![Page 33: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/33.jpg)
Application Node Control Memory Policies
What are Memory Policies?
Memory Policies control the behavior of the memory allocator.
They allow us to adjust. . .I Per-process memory allocation policy.I Memory allocation policies for ranges of process memory.I Memory Policies for file systems such as tmpfs.
Using Memory Policies we can. . .I Select which nodes to allocate from.I Chose between optimizing for latency or bandwidth.I Allow or disallow fallback allocation from other nodes.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 26 / 30
![Page 34: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/34.jpg)
Application Node Control Memory Policies
What are Memory Policies?
Memory Policies control the behavior of the memory allocator.
They allow us to adjust. . .I Per-process memory allocation policy.I Memory allocation policies for ranges of process memory.I Memory Policies for file systems such as tmpfs.
Using Memory Policies we can. . .I Select which nodes to allocate from.I Chose between optimizing for latency or bandwidth.I Allow or disallow fallback allocation from other nodes.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 26 / 30
![Page 35: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/35.jpg)
Application Node Control Memory Policies
What are Memory Policies?
Memory Policies control the behavior of the memory allocator.
They allow us to adjust. . .I Per-process memory allocation policy.I Memory allocation policies for ranges of process memory.I Memory Policies for file systems such as tmpfs.
Using Memory Policies we can. . .I Select which nodes to allocate from.I Chose between optimizing for latency or bandwidth.I Allow or disallow fallback allocation from other nodes.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 26 / 30
![Page 36: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/36.jpg)
Application Node Control Memory Policies
Memory Policies - ModesMPOL_DEFAULT
I Prioritize local node over remote ones.
MPOL_BIND
I Allocate from specified nodes only, one by one.
MPOL_INTERLEAVE
I Spread out allocations over specified nodes.
MPOL_PREFERRED
I Allocate from specified nodes, one by one.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 27 / 30
![Page 37: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/37.jpg)
Application Node Control Memory Policies
Memory Policies
Per-process control of memory allocations:
int set_mempolicy(int mode, unsigned long *nodemask,unsigned long maxnode)
Control ranges of process memory:
int mbind(void *start, unsigned long len, int mode,unsigned long *nodemask, unsigned long maxnode,unsigned flags)
man 2 set_mempolicyman 2 mbindman 3 numa - libnuma by Andi Kleen
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 28 / 30
![Page 38: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/38.jpg)
Application Node Control Memory Policies
Memory Policies
Per-process control of memory allocations:
int set_mempolicy(int mode, unsigned long *nodemask,unsigned long maxnode)
Control ranges of process memory:
int mbind(void *start, unsigned long len, int mode,unsigned long *nodemask, unsigned long maxnode,unsigned flags)
man 2 set_mempolicyman 2 mbindman 3 numa - libnuma by Andi Kleen
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 28 / 30
![Page 39: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/39.jpg)
Application Node Control tmpfs and cpusets
tmpfs and cpusets
tmpfs is a file system which keeps all files in memory. Mount optionscan be used to select memory policy.
mount -t tmpfs -o mpol=bind:0,2 tmpfs /mytmpfs
See linux/Documentation/filesystems/tmpfs.txt for more information.
cpusets is kernel mechanism that assigns processes to a subset of allavailable processors and memory nodes.
More information available in linux/Documentation/cpusets.txt
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 29 / 30
![Page 40: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/40.jpg)
Application Node Control tmpfs and cpusets
tmpfs and cpusets
tmpfs is a file system which keeps all files in memory. Mount optionscan be used to select memory policy.
mount -t tmpfs -o mpol=bind:0,2 tmpfs /mytmpfs
See linux/Documentation/filesystems/tmpfs.txt for more information.
cpusets is kernel mechanism that assigns processes to a subset of allavailable processors and memory nodes.
More information available in linux/Documentation/cpusets.txt
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 29 / 30
![Page 41: Asymmetric NUMA: - Multiple-memory … NUMA: Multiple-memory management for the rest of us ... Outline Introduction to NUMA ... Asymmetric NUMA ELC-Europe 2007 18 / 30.](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adfd01e7f8b9a1c248c7b95/html5/thumbnails/41.jpg)
Summary
Summary
I Asymmetric NUMA brings NUMA to the embedded space.I Well-established interfaces outweights the added overhead.
Magnus Damm ([email protected]) Asymmetric NUMA ELC-Europe 2007 30 / 30