Kernel Synchronization
Examples From the Linux Kernel
Michael E. Locasto
BIG PICTURE: HOW CAN THE KERNEL CORRECTLY SERVICE REQUESTS?
kernel control flow is a complicated, asynchronous interleaving
Main Ideas / Concepts
Atomic operations in x86
Kernel locking / synchronization primitives
Kernel preemption
Read-Copy-Update
The “big kernel lock”
Kernel Preemption
Kernel preemption is a concept in which the kernel can preempt other running kernel control paths (be they on behalf of a user or another kernel thread)
Acquiring a spinlock automatically disables kernel preemption (as we will see in the code)
Synchronization Primitives
Atomic operationsDisable interrupts (cli/sti modify IF of eflags)Lock memory bus (x86 lock prefix)Spin locksSemaphoresSequence LocksRead-copy-update (RCU) (lock free)
Barriers
Barriers are serializing operations; they “gather” and make operations sequential.
Memory barrier:x86 in/out on I/O portsx86 lock prefixx86 writes to CReg, SReg/eflags, DReg
x86 instr meaning
lfence read barrier
sfence write barrier
mfence r/w barrier
Barrier Implementation
Motivating Example: Using Semaphores in the Kernel
what are: down_read, up_read, and mmap_sem
START WITH THE DATA STRUCTURE: MM->MMAP_SEM
Let’s start with the data structure and see where that leads…
current->mm->mmap_sem
struct mm_struct: include/linux/mm_types.h
PRIMITIVE ONE: ATOMIC TYPE AND OPERATIONS
On x86, these operations are atomic
simple asm instructions that involve 0 or 1 aligned memory access
read-modify-update in 1 clock cycle (e.g., inc, dec)
anything prefixed by the IA-32 ‘lock’ prefix
atomic_t: include/linux/types.h
Example: Reference Counters
Refcounts: atomic_t; associated with resources, but keeps count of kernel control paths accessing the resource
PRIMITIVE TWO: SPINLOCKS
/include/linux/spinlock_types.h
typedef struct spinlock{struct raw_spinlock rlock;
} spinlock_t;
typedef struct raw_spinlock{arch_spinlock_t raw_lock;
} raw_spinlock_t;
arch/x86/include/asm/spinlock_types.h#L10
slock=1 (unlocked), slock=0 (locked)
spinlock API (partial)
/include/linux/spinlock.h
/kernel/spinlock.c
include/linux/spinlock_api_smp.h
Linux Tracks Lock Dependencies @ Runtime
PRIMITIVE THREE: SEMAPHORESHere we mainly consider Read/Write Semaphores
Important Caveats about Kernel Semaphores
Semaphores are *not* like spinlocks in the sense that the invoking process is put to sleep rather than busy waits.
As a result, kernel semaphores should only be used by functions that can safely sleep (i.e., not interrupt handlers)
might_sleep() leads (eventually) to:
rwsem_wake
__rwsem_do_wake
On our way out, allow a writer at the front of the waiting queue to proceed.
Then allow unlimited numbers of readers to access the critical region.
Advanced Techniques
Sequence LocksA solution to the multiple
readers-writer problem in that a writer is permitted to advance even if readers are in the critical section.
Readers must check both an entry and exit flag to see if data has been modified underneath them.
Read-Copy-Update (RCU)Designed to protect data structures
accessed by multiple CPUs; allows many readers and writers.
Basic idea is simple (and in the name). Readers access data structure via a pointer; writers initially act as readers & create a copy to modify. “Writing” is just a matter of updating the pointer.
RCUOnly for kernel control paths; disables preemption.
Used to protect data structures accessed through a pointer
by adding a layer of indirection, we can reduce wholesale writes/updates to a single atomic write/update
Heavy restrictions: RCU tasks cannot sleep
readers do little work
writers act as readers, make a copy, then update copy. Finally, they rewrite the pointer.
cleanup is correspondingly complicated.
RCU EXAMPLE: GETPPID(2)http://lxr.linux.no/#linux+v2.6.35.14/kernel/timer.c#L1354
EXERCISE: TIME PERFORMANCE COST OF SYNCHRONIZATION
Does synchronization impose a significant cost?(test at user level)
CODE: AUTOMATICALLY DRAWING RESOURCE GRAPHS
Top Related