CLIP OS: Building a defense-in-depth OSaround Linux kernel security improvements

Timothée Ravier, Mickaël Salaün

Agence nationale de la sécurité des systèmes d’information (ANSSI)

September 26, 2018

About the ANSSI

I Agence nationale de la sécurité des systèmes d’information

I French authority in the area of cyberdefence, network and informationsecurity

I We are not an intelligence agency

ANSSI CLIP OS: Building a defense-in-depth OS 2/27

Overview

CLIP OS ?

I Linux distribution developed by the ANSSI

I Initially only available internally

I Now open source, mostly under the LGPL v2.1+I Code and issue tracker hosted on GitHub:

I Version 4: available as reference and for upstream patch contribution1

I Version 5: currently developed version, alpha status2

1https://github.com/CLIPOS-Archive2https://github.com/CLIPOS

ANSSI CLIP OS: Building a defense-in-depth OS 4/27

Hardened OS

I Hardened Linux kernel and userspace

I Confined services

I "Unprivileged" admin, audit and update roles:⇒ the root account is not usable

I Automatic updates using A/B partition model (similar to Android 7+)

ANSSI CLIP OS: Building a defense-in-depth OS 5/27

Multilevel security OS

I Provide two isolated user environments: low and highI Interactions follow the Bell-LaPadula model:

I Write up: upload documents from low to highI Read down: high has read only access to untrusted USB devicesI Trusted write down: encrypt documents from high to write them in low

I Level high can only access network through a VPN

I Per level user device assignment

ANSSI CLIP OS: Building a defense-in-depth OS 6/27

Multilevel from the end user point of view

ANSSI CLIP OS: Building a defense-in-depth OS 7/27

Admin panel: devices assignment per level

ANSSI CLIP OS: Building a defense-in-depth OS 8/27

Differences with Qubes OS

CLIP OS development began 5 years earlier than Qubes OS

Goal of CLIP OSI We target non-expert usersI Bell-LaPadula model with two levelsI We favor a defense-in-depth approach

Technical point of viewI Hypervisor vs. supervisor isolationI Limited access right, even for the administrator

ANSSI CLIP OS: Building a defense-in-depth OS 9/27

Differences with Qubes OS

CLIP OS development began 5 years earlier than Qubes OS

Goal of CLIP OSI We target non-expert usersI Bell-LaPadula model with two levelsI We favor a defense-in-depth approach

Technical point of viewI Hypervisor vs. supervisor isolationI Limited access right, even for the administrator

ANSSI CLIP OS: Building a defense-in-depth OS 9/27

Differences with Qubes OS

CLIP OS development began 5 years earlier than Qubes OS

Goal of CLIP OSI We target non-expert usersI Bell-LaPadula model with two levelsI We favor a defense-in-depth approach

Technical point of viewI Hypervisor vs. supervisor isolationI Limited access right, even for the administrator

ANSSI CLIP OS: Building a defense-in-depth OS 9/27

Architecture

ANSSI CLIP OS: Building a defense-in-depth OS 10/27

CLIP OS 4

Hardening mechanisms

Gentoo HardenedI Hardened toolchainI Flexible patching

Linux-VServerI Linux namespaces with additional constraintsI Unique container and network IDs: XIDs and NIDs

grsecurity/PaXI Kernel self-protection (e.g., memory protection, CFI)I Multiple userspace hardening features (e.g., chroot, TPE)

CLIP LSMI Complement the Linux permission modelI Leverage Linux-VServer and grsecurity/PaX

ANSSI CLIP OS: Building a defense-in-depth OS 12/27

Hardening mechanisms

Gentoo HardenedI Hardened toolchainI Flexible patching

Linux-VServerI Linux namespaces with additional constraintsI Unique container and network IDs: XIDs and NIDs

grsecurity/PaXI Kernel self-protection (e.g., memory protection, CFI)I Multiple userspace hardening features (e.g., chroot, TPE)

CLIP LSMI Complement the Linux permission modelI Leverage Linux-VServer and grsecurity/PaX

ANSSI CLIP OS: Building a defense-in-depth OS 12/27

Hardening mechanisms

Gentoo HardenedI Hardened toolchainI Flexible patching

Linux-VServerI Linux namespaces with additional constraintsI Unique container and network IDs: XIDs and NIDs

grsecurity/PaXI Kernel self-protection (e.g., memory protection, CFI)I Multiple userspace hardening features (e.g., chroot, TPE)

CLIP LSMI Complement the Linux permission modelI Leverage Linux-VServer and grsecurity/PaX

ANSSI CLIP OS: Building a defense-in-depth OS 12/27

Hardening mechanisms

Gentoo HardenedI Hardened toolchainI Flexible patching

Linux-VServerI Linux namespaces with additional constraintsI Unique container and network IDs: XIDs and NIDs

grsecurity/PaXI Kernel self-protection (e.g., memory protection, CFI)I Multiple userspace hardening features (e.g., chroot, TPE)

CLIP LSMI Complement the Linux permission modelI Leverage Linux-VServer and grsecurity/PaX

ANSSI CLIP OS: Building a defense-in-depth OS 12/27

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevelisolation

Memory (PaX)Deny writable memory to be executable, throughout the system lifetime

DevctlEnforce and extend W ⊕ X from devices to mount points

Mount pointsEnforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC flagEnforce and extend W ⊕ X from mount points to scripts (via interpreters)

ANSSI CLIP OS: Building a defense-in-depth OS 13/27

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevelisolation

Memory (PaX)Deny writable memory to be executable, throughout the system lifetime

DevctlEnforce and extend W ⊕ X from devices to mount points

Mount pointsEnforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC flagEnforce and extend W ⊕ X from mount points to scripts (via interpreters)

ANSSI CLIP OS: Building a defense-in-depth OS 13/27

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevelisolation

Memory (PaX)Deny writable memory to be executable, throughout the system lifetime

DevctlEnforce and extend W ⊕ X from devices to mount points

Mount pointsEnforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC flagEnforce and extend W ⊕ X from mount points to scripts (via interpreters)

ANSSI CLIP OS: Building a defense-in-depth OS 13/27

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevelisolation

Memory (PaX)Deny writable memory to be executable, throughout the system lifetime

DevctlEnforce and extend W ⊕ X from devices to mount points

Mount pointsEnforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC flagEnforce and extend W ⊕ X from mount points to scripts (via interpreters)

ANSSI CLIP OS: Building a defense-in-depth OS 13/27

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevelisolation

Memory (PaX)Deny writable memory to be executable, throughout the system lifetime

DevctlEnforce and extend W ⊕ X from devices to mount points

Mount pointsEnforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC flagEnforce and extend W ⊕ X from mount points to scripts (via interpreters)

ANSSI CLIP OS: Building a defense-in-depth OS 13/27

O_MAYEXEC

ANSSI CLIP OS: Building a defense-in-depth OS 14/27

Partitioning

Hardened containersI Leverage Linux-VServer admin and watch (audit) conceptsI New capability bounding sets: for root and per containerI Hardened chroot

Container content and interactionI Tailored filesystem layouts per serviceI Container management with vsctl and clip-libvserver (self-jailing)

ANSSI CLIP OS: Building a defense-in-depth OS 15/27

Partitioning

Hardened containersI Leverage Linux-VServer admin and watch (audit) conceptsI New capability bounding sets: for root and per containerI Hardened chroot

Container content and interactionI Tailored filesystem layouts per serviceI Container management with vsctl and clip-libvserver (self-jailing)

ANSSI CLIP OS: Building a defense-in-depth OS 15/27

Veriexec and permissions (CLIP-LSM)

GoalI Split Linux capabilities (e.g., Fuse, unshare)I Add new permissions (e.g., network, XFRM)I Can be tied to an XIDI Does not use xattr (thus independent from the filesystem)

Configuration example: /etc/verictl.d/chromium

/usr /.../ chrome - sandbox 1002 eSETUID | SETGID | SYS_CHROOT SETUID | SETGID | SYS_CHROOT -cUP sha256 45 bcbd1 ...

ANSSI CLIP OS: Building a defense-in-depth OS 16/27

Veriexec and permissions (CLIP-LSM)

GoalI Split Linux capabilities (e.g., Fuse, unshare)I Add new permissions (e.g., network, XFRM)I Can be tied to an XIDI Does not use xattr (thus independent from the filesystem)

Configuration example: /etc/verictl.d/chromium

/usr /.../ chrome - sandbox 1002 eSETUID | SETGID | SYS_CHROOT SETUID | SETGID | SYS_CHROOT -cUP sha256 45 bcbd1 ...

ANSSI CLIP OS: Building a defense-in-depth OS 16/27

Veriexec example

ANSSI CLIP OS: Building a defense-in-depth OS 17/27

CLIP OS 5

General Linux kernel hardening

I Strict whitelist of kernel options, but easily composable setsI Paranoid command line

I iommu=force, pti=on, spectre_v2=on, etc.

I Strict sysctl defaultsI kernel.kptr_restrict, kernel.yama.ptrace_scope, etc.

ANSSI CLIP OS: Building a defense-in-depth OS 19/27

Enabling Linux kernel hardening

GoalsI Protecting the kernel from itself and from userspaceI Include additional features for userspaceI Being able to test kernel and userspace coordinated changes

Security may come firstI We can handle minor compatibility breakage in our userspaceI Will accept changes that upstream may reject

Interaction with upstream & KSPPI Include in-progress or ready-for-upstream patchesI Integrate and validate patches in a single treeI Maintain hardening patches for latest stable kernel

ANSSI CLIP OS: Building a defense-in-depth OS 20/27

Enabling Linux kernel hardening

GoalsI Protecting the kernel from itself and from userspaceI Include additional features for userspaceI Being able to test kernel and userspace coordinated changes

Security may come firstI We can handle minor compatibility breakage in our userspaceI Will accept changes that upstream may reject

Interaction with upstream & KSPPI Include in-progress or ready-for-upstream patchesI Integrate and validate patches in a single treeI Maintain hardening patches for latest stable kernel

ANSSI CLIP OS: Building a defense-in-depth OS 20/27

Enabling Linux kernel hardening

GoalsI Protecting the kernel from itself and from userspaceI Include additional features for userspaceI Being able to test kernel and userspace coordinated changes

Security may come firstI We can handle minor compatibility breakage in our userspaceI Will accept changes that upstream may reject

Interaction with upstream & KSPPI Include in-progress or ready-for-upstream patchesI Integrate and validate patches in a single treeI Maintain hardening patches for latest stable kernel

ANSSI CLIP OS: Building a defense-in-depth OS 20/27

Patch series: linux-hardened

FeaturesI Memory hardening improvements, including:

I better userspace ASLRI slab allocators hardening (mostly SLUB)I simpler page sanitizing

I Various restrictions: TIOCSTI ioctl, perf subsystem, device timing sidechannels, etc.

I Miscellaneous additions: more BUG_ONs, more __ro_after_init, etc.

I Development status: In progressI CLIP OS status: MergedI Upstream status: Most changes unlikely to be merged upstream

ANSSI CLIP OS: Building a defense-in-depth OS 21/27

Patch series: linux-hardened

FeaturesI Memory hardening improvements, including:

I better userspace ASLRI slab allocators hardening (mostly SLUB)I simpler page sanitizing

I Various restrictions: TIOCSTI ioctl, perf subsystem, device timing sidechannels, etc.

I Miscellaneous additions: more BUG_ONs, more __ro_after_init, etc.

I Development status: In progressI CLIP OS status: MergedI Upstream status: Most changes unlikely to be merged upstream

ANSSI CLIP OS: Building a defense-in-depth OS 21/27

Upstream contribution integration: Lockdown

FeaturesI Reduce options for root to run untrusted code in kernel context

I Development status: Feature completeI CLIP OS status: MergedI Upstream status: Ready for upstream integration

ANSSI CLIP OS: Building a defense-in-depth OS 22/27

Upstream contribution integration: Lockdown

FeaturesI Reduce options for root to run untrusted code in kernel context

I Development status: Feature completeI CLIP OS status: MergedI Upstream status: Ready for upstream integration

ANSSI CLIP OS: Building a defense-in-depth OS 22/27

Upstream contribution integration: STACKLEAK

FeaturesI Reduce information leaks and block attacks using uninitialized kernel

stack variables:I Erase the stack before returning from system calls

I Improve runtime detection of kernel stack overflows (e.g. Stack Clash):I Instrument calls to alloca()

CLIP OS specific changesI Kept alloca()-related changes (dropped for upstream in v15)

I Development status: Feature completeI CLIP OS status: MergedI Upstream status: Ready for upstream integration

ANSSI CLIP OS: Building a defense-in-depth OS 23/27

Upstream contribution integration: STACKLEAK

FeaturesI Reduce information leaks and block attacks using uninitialized kernel

stack variables:I Erase the stack before returning from system calls

I Improve runtime detection of kernel stack overflows (e.g. Stack Clash):I Instrument calls to alloca()

CLIP OS specific changesI Kept alloca()-related changes (dropped for upstream in v15)

I Development status: Feature completeI CLIP OS status: MergedI Upstream status: Ready for upstream integration

ANSSI CLIP OS: Building a defense-in-depth OS 23/27

Upstream contribution integration: STACKLEAK

FeaturesI Reduce information leaks and block attacks using uninitialized kernel

stack variables:I Erase the stack before returning from system calls

I Improve runtime detection of kernel stack overflows (e.g. Stack Clash):I Instrument calls to alloca()

CLIP OS specific changesI Kept alloca()-related changes (dropped for upstream in v15)

I Development status: Feature completeI CLIP OS status: MergedI Upstream status: Ready for upstream integration

ANSSI CLIP OS: Building a defense-in-depth OS 23/27

Upstream contribution: Landlock

FeaturesI Enables seccomp-bpf -like self-sandboxing for unprivileged processesI Stackable LSMI Powered by eBPFI Dynamic filesystem access control using whitelists & blacklistsI See landlock.io

I Development status: Initial feature set readyI CLIP OS status: PlannedI Upstream status: Work in progress

ANSSI CLIP OS: Building a defense-in-depth OS 24/27

Upstream contribution: Landlock

FeaturesI Enables seccomp-bpf -like self-sandboxing for unprivileged processesI Stackable LSMI Powered by eBPFI Dynamic filesystem access control using whitelists & blacklistsI See landlock.io

I Development status: Initial feature set readyI CLIP OS status: PlannedI Upstream status: Work in progress

ANSSI CLIP OS: Building a defense-in-depth OS 24/27

Upstream contribution: VServer-like LSM

FeaturesI Adds a single kernel enforced indentifier for confined environmentsI Similar in principle to VServer XID or to "Container IDs"I Inspired by the VServer patchI Integrated as a stackable LSM

I Development status: Early development stageI CLIP OS status: Planned

ANSSI CLIP OS: Building a defense-in-depth OS 25/27

Upstream contribution: VServer-like LSM

FeaturesI Adds a single kernel enforced indentifier for confined environmentsI Similar in principle to VServer XID or to "Container IDs"I Inspired by the VServer patchI Integrated as a stackable LSM

I Development status: Early development stageI CLIP OS status: Planned

ANSSI CLIP OS: Building a defense-in-depth OS 25/27

Conclusion

Take awayI Hardened Linux distro and kernelI Coordinated userspace and kernelspaceI Support multilevel security

Ongoing projectI Contributions welcomeI Browse the doc and the sources to find more interesting features:

docs.clip-os.org

ANSSI CLIP OS: Building a defense-in-depth OS 26/27

Conclusion

Take awayI Hardened Linux distro and kernelI Coordinated userspace and kernelspaceI Support multilevel security

Ongoing projectI Contributions welcomeI Browse the doc and the sources to find more interesting features:

docs.clip-os.org

ANSSI CLIP OS: Building a defense-in-depth OS 26/27

Thanks!

� clip-os.org B clipos@ssi.gouv.fr

� v4: github.com/CLIPOS-Archive

� v5: github.com/CLIPOS

We’re hiring! (but not directly for CLIP OS)

Linux system security experthttps://www.ssi.gouv.fr/emploi/expert-en-securite-des-systemes-linux/

Boot chain and root partition integrity protection

1 UEFI Secure Boot support:I Custom keys (i.e. not signed by Microsoft)I Requires enrollment in hardware

2 Minimal bootloader (gummiboot/systemd-boot)3 EFI bundle:

I Linux kernelI initramfsI kernel command line

4 DM-Verity partition:

I DM-Verity root hash set in kernel command lineI Forward error correction support (FEC)I Read only uncompressed SquashFS root

filesystem

ANSSI CLIP OS: Building a defense-in-depth OS 28/27

Boot chain and root partition integrity protection

1 UEFI Secure Boot support:I Custom keys (i.e. not signed by Microsoft)I Requires enrollment in hardware

2 Minimal bootloader (gummiboot/systemd-boot)

3 EFI bundle:

I Linux kernelI initramfsI kernel command line

4 DM-Verity partition:

I DM-Verity root hash set in kernel command lineI Forward error correction support (FEC)I Read only uncompressed SquashFS root

filesystem

ANSSI CLIP OS: Building a defense-in-depth OS 28/27

Boot chain and root partition integrity protection

1 UEFI Secure Boot support:I Custom keys (i.e. not signed by Microsoft)I Requires enrollment in hardware

2 Minimal bootloader (gummiboot/systemd-boot)3 EFI bundle:

I Linux kernelI initramfsI kernel command line

4 DM-Verity partition:

I DM-Verity root hash set in kernel command lineI Forward error correction support (FEC)I Read only uncompressed SquashFS root

filesystem

ANSSI CLIP OS: Building a defense-in-depth OS 28/27

Boot chain and root partition integrity protection

1 UEFI Secure Boot support:I Custom keys (i.e. not signed by Microsoft)I Requires enrollment in hardware

2 Minimal bootloader (gummiboot/systemd-boot)3 EFI bundle:

I Linux kernelI initramfsI kernel command line

4 DM-Verity partition:I DM-Verity root hash set in kernel command lineI Forward error correction support (FEC)I Read only uncompressed SquashFS root

filesystem

ANSSI CLIP OS: Building a defense-in-depth OS 28/27