LCU14 BURLINGAME

Behan Webster, LCU14

LCU14-209: LLVMLinux

● About the LLVMLinux project● State of Linux kernel being compiled with clang● Details of specific known problems/patches still being upstreamed● How can you help?● Contact info

Linaro Overview

● Fully build the Linux kernel for multiple architectures, using the Clang/LLVM toolchain

● Discover LLVM/Kernel issues early and find fixes quickly across both communities

● Upstream patches to the Linux Kernel and LLVM projects

● Bring together like-minded developers● Enable the kernel community to do more in depth

analysis of the kernel code

The LLVMLinux Project Goals

● Fetches, patches, builds, tests: clang, kernel, qemu, etc○ git clone http://git.linuxfoundation.org/llvmlinux.git○ cd llvmlinux/target/vexpress (or vexpress64)○ make

LLVMLinux Build/Test System

● A mainline kernel tree with all LLVMLinux patches applied on top is now available:○ git://git.linuxfoundation.org/llvmlinux/kernel.git

● Dated llvmlinux branches○ remotes/origin/llvmlinux-2014.09.16

● The master branch is rebased regularly

Patched Mainline Kernel Tree

● LLVM/clang:○ All LLVMLinux patches for LLVM are Upstream○ Newer LLVM patches to support the Linux kernel are mostly

being added by upstream maintainers● Linux Kernel:

○ Roughly 49 kernel patches for various arches○ LLVMLinux branch in linux-next

LLVMLinux Project Status

● Patches still working their way upstream

Remaining LLVMLinux Kernel Patches

Architecture Number of Patches

Patches Submitted

Patches Accepted

all 23 17 2

arm 12 11 1 (+7)

aarch64 11 8 7

x86_64 3 1 1

TOTAL 49 37 11

● Total patches currently required for a single architecture

LLVMLinux Kernel Patches

Architecture Number of Patches

Patches Submitted

Patches Accepted

arm (+all) 35 (12+23) 28 (11+17) 3 (1+2)

aarch64 (+all) 34 (11+23) 25 (8+17) 9 (7+2)

x86_64 (+all) 26 (3+23) 18 (1+17) 3 (1+2)

● Basic Kbuild support for clang and x86 has been upstreamed● Specific support for ARM and aarch64 ready to be upstreamed

Kbuild

● Renato Golin, Vicinius Tinti, Saleem Abdulrasool and Stepan Dyatkovskiy are working on fixing IA issues in clang to support the Linux ARM kernel code (and ultimately AARCH64)

● David Woodhouse has added .code16 support for X86 ASM● For now we disable the IA and use GNU as instead

Integrated Assembly Status

● gcc passes -march to GNU as● clang doesn't... (Bug submitted PR)● Probably should be fixed in clang● Work around patch for now

-CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto

+CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto -Wa,-march=armv8-a+crypto

Different option passing

● GNU89/GNU90 (used by gcc)○ Function will be inlined where it is used○ No function definition is emitted○ A non-inlined function may also be provided

● GNU99/C99 (used by clang)○ Function will be inlined where it is used○ An external function is emitted○ No other function of the same name may be provided.

● Solution? Use “static inline” instead.● Only still an issue for ARM support for ftrace (submitted)

extern inline: Different for gnu89 and gnu99

● gcc is less picky about placement of __attribute__(())● clang requires it at the end of the type or variable

-struct __read_mostly va_alignment va_align = {

+struct va_alignment __read_mostly va_align = {

● (This particular patch was just accepted)

Attribute Order

● (Named registers for X86 kernel have been removed from the mainline kernel by Andi Kleen)

● ARM and AARCH64 still like using named registers● Clang now supports using a globally named register for the stack

pointer (Thanks Renato!)● For ARM/AARCH64 move to using a global in asm/thread_info.h

register unsigned long current_stack_pointer asm ("sp");

● Patches for AARCH64 now accepted, acked for ARM

Named Registers

● One of the uses of Named Registers in the ARM code is due to a deficiency in gcc

● The new code which works with gcc fails in clang● Solution, provide routines for both, and choose at compile time

● Gcc:asm("mrc p15, 0, %0, c13, c0, 4" : "=r" (off) : "Q" (*sp));

● Clang:asm("mrc p15, 0, %0, c13, c0, 4" : "=r" (off) : : "memory");

ARM percpu patch

● The following error is generated with clang for AArch64:error: invalid operand in inline asm: 'prfm pldl1keep, ${0:a}'

● Per comments by Tim Northover on the LLVM Bug database:It's rather unclear how it's better than "prfm pstl1keep, [%0]" though. Not all instructions can make use of any offset, so wouldn't we have to be conservative and always map it to "[xN]"?

● When %a0 is changed to [%x0] it uncovered a GCC bug: https://bugs.linaro.org/show_bug.cgi?id=635

● Changing the "p" to "r" resolves the issue for both clang and GCC.- asm volatile("prfm pldl1keep, %a0\n" : : "p" (ptr));+ asm volatile("prfm pldl1keep, [%x0]\n" : : "r" (ptr));

Missing “%a” for inline ASM

● By default clang merges globals with internal linkage into one: MergedGlobals

● Allows globals to be addressed using offsets from a base pointer● Can reduce the number of registers used● Modpost script in the Linux kernel uses symbol names to look for

section mismatches (e.g. regular code calling init code)● MergedGlobals breaks modpost (false positive section

mismatches)● Current solution: use -mno-global-merge to stop global merging● Updates to modpost may allow this optimization to be enabled

again in the future

Section Mismatch Issues (MergedGlobals)

● Clang emits code which uses the “aeabi” ARM calls which are implemented in compiler-rt (equivalient to libgcc)

● Compiler-rt doesn't easily cross compile yet...

void __aeabi_memcpy(void *dest, const void *src, size_t n)

void __aeabi_memmove(void *dest, const void *src, size_t n)

void __aeabi_memset(void *s, size_t n, int c)

● Still needed for ARM● No longer required for AARCH64

ARM eabi support

● VLAIS isn't supported by Clang (undocumented gcc extension) char vla[n]; /* Supported, C99/C11 */

struct {

char flexible_member[]; /* Supported, C99/C11 */

} struct_with_flexible_member;

struct {

char vlais[n]; /* Explicitly not allowed by C99/C11 */

} variable_length_array_in_struct;

● VLAIS is used in the Linux kernel in a number of places, spreading mostly through reusing patterns from data structures found in crypto

Variable Length Arrays In Structs

- struct {

- struct shash_desc shash;

- char ctx[crypto_shash_descsize(hash)];

- } desc;

+ char desc[sizeof(struct shash_desc)

+

+ crypto_shash_descsize(hash)] CRYPTO_MINALIGN_ATTR;

+ struct shash_desc *shash = (struct shash_desc *)desc;

unsigned int i;

- desc.shash.tfm = hash;

+ shash->tfm = hash;

VLAIS Removal Example (from crypto/hmac.c)

#define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)

#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN

#define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN)))

struct shash_desc {

struct crypto_shash *tfm;

u32 flags;

void *__ctx[] CRYPTO_MINALIGN_ATTR;

};

VLAIS Removal Example (the missing pieces)

● USB Gadget patch is in mainline● Mac80211 patch is in mainline● Netfilter patch is in mainline● apparmor patch accepted● Bluetooth patch accepted● Only the VLAIS patches for crypto are left:

(btrfs, dm-crypt, hmac, libcrc32c, testmgr, etc)

● However, we recently found a few previously unknown instances of VLAIS in raid10 and exofs...

Status of VLAIS in the Linux Kernel

static int _sp2d_alloc(unsigned pages_in_unit, unsigned group_width,

unsigned parity, struct __stripe_pages_2d **psp2d)

[...]

unsigned data_devs = group_width - parity;

struct _alloc_all_bytes {

struct __alloc_stripe_pages_2d {

struct __stripe_pages_2d sp2d;

struct __1_page_stripe _1p_stripes[pages_in_unit];

} __asp2d;

struct __alloc_1p_arrays {

struct page *pages[group_width];

struct page *scribble[group_width];

char page_is_read[data_devs];

} __a1pa[pages_in_unit];

} *_aab;

Huge use of VLAIS in fs/exofs/ore_raid.c

● Don’t use the non-C99 practices we showed in previous slides● Make it known you want to be able to use Clang to compile the

kernel (tell your Linaro representative!)● Test LLVMLinux patches● Report bugs to the LLVMLinux mailing list● Help get LLVMLinux patches upstream● Work on unsupported features and Bugs

○ http://llvm.linuxfoundation.org/index.php/Broken_kernel_options● Submit new targets and arch support● Patches welcome

How Can You Help?

Embrace theDragon.

He's cuddly.Thank you

http://llvm.linuxfoundation.org

● Project wiki page○ http://llvm.linuxfoundation.org

● Project Mailing List○ http://lists.linuxfoundation.org/mailman/listinfo/llvmlinux○ http://lists.linuxfoundation.org/pipermail/llvmlinux/

● IRC Channel○ #llvmlinux on OFTC○ http://buildbot.llvm.linuxfoundation.org/irclogs/OFTC/%23llvmlinux/

● LLVMLinux Community on Google Plus

Contribute to the LLVMLinux Project

More about Linaro Connect: connect.linaro.org Linaro members: www.linaro.org/membersMore about Linaro: www.linaro.org/about/