bgclang – LLVM/Clang for the BG/Q

Goals of This Presentation:

✔ To explain what LLVM is, what Clang is, and how they can be used.

✔ To explain the components of bgclang and how they can be used.

✔ To explain how to install bgclang.

✔ To discuss future plans for bgclang.

bgclang is both a research platformand a production-quality compiler.

What is LLVM:

LLVM is not a “low-level virtual machine”!

LLVM is a multi-architecture infrastructure for constructing compilers and other toolchain components.

LLVM IRArchitecture-independent

canonicalization.(e.g. LICM)

Architecture-awareoptimization.

(e.g. Vectorization)

Architecture-awarelegalization.

Instructionselection

Instruction-leveloptimization

Scheduling andregister allocation

Assembly printing orbinary generation or

JIT execution.

What is Clang:

Clang is a C++ frontend for LLVM...

C++ Source(C++14, C11, etc.)

Parsing andsemantic analysis

LLVM IR

Code generation

Static analysis

bgclang Components:

LLVM

QPX patches(+ SLEEF integration,

prefetching, etc.)

BG/Q Address Sanitizer patches QPX intrinsics patches

Clangcompiler-rt

(address sanitizer runtime)

A lot of upstream work has comeout of this project...

libc++C++11 STL(patched)

clang-omp patches

SLEEFVector math-functions library

(patched)

libiomp5OpenMP runtime

(patched)

GNU binutils(patched)

BG/Q MPI wrapper scripts

Installing bgclang:

1) Go to http://trac.alcf.anl.gov/projects/llvm-bgq/ and download the rpms2) Convince rpm that you do have a /bin/sh installed:

rpm -Uhv --dbpath /tmp/bgclang/rpm --prefix /tmp/bgclang vpkg-bin-sh-1-1.ppc64.rpm3) Install the “stage 1” Clang/LLVM and libc++:

rpm -Uhv --dbpath /tmp/bgclang/rpm --prefix /tmp/bgclang bgclang-stage1-3.4-1.ppc64.rpm \ bgclang-stage1-libcxx-3.4-2.ppc64.rpm

4) Install everything else: rpm -Uhv --dbpath /tmp/bgclang/rpm --prefix /tmp/bgclang \ bgclang-binutils-r209570-20140527-1-1.ppc64.rpm \ bgclang-r209570-20140527-1-1.ppc64.rpm \ bgclang-compiler-rt-r209570-20140527-1-1.ppc64.rpm \ bgclang-libcxx-r209570-20140527-1-1.ppc64.rpm \ bgclang-libomp-r209570-20140527-1-1.ppc64.rpm \ bgclang-sleef-r209570-20140527-1-1.ppc64.rpm

But don't actually install into /tmp:pick somewhere else!

(especially if you want to usedynamic linking, because your /tmp

won't be mounted on the compute nodes)

Installing bgclang (cont.):

Now you'll have something like this:

$ ls -l /tmp/bgclang | awk '{print $1, $9, $10, $11}' lrwxrwxrwx bin -> current/bin lrwxrwxrwx compiler-rt -> current/compiler-rt lrwxrwxrwx current -> r189357-20130827 lrwxrwxrwx docs -> current/docs lrwxrwxrwx include -> current/include lrwxrwxrwx lib -> current/lib lrwxrwxrwx libc++ -> current/libc++ lrwxrwxrwx libstdc++fixup -> current/libstdc++fixup lrwxrwxrwx mpi -> current/mpi lrwxrwxrwx omp -> current/omp drwxr-xr-x r209570-20140527 drwxr-xr-x rpm lrwxrwxrwx scan-build -> current/scan-build lrwxrwxrwx scan-view -> current/scan-view lrwxrwxrwx share -> current/share lrwxrwxrwx sleef -> current/sleef drwxr-xr-x stage1 lrwxrwxrwx wbin -> current/wbin

When you upgrade, you'll need to update the“current” symlink yourself.

Each new build goes into a directory like this...

MPI wrapper scripts.

Recommendation:add this to your PATH environmental variable.

bgclang ALCF softenv keys

On ALCF systems (Vesta, Mira, Cetus), bgclang is installed for you; you can use the softenv keys:

+mpiwrapper-bgclang bgclang wrappers and toolchain +mpiwrapper-bgclang.legacy bgclang.legacy wrappers and toolchain

Add these to your .soft file to use them by default, or run: soft add +mpiwrapper-bgclang

If you have a single-threaded application,remember to use the “legacy” MPI

for better performance.

bgclang (MPI) wrappers

MPI wrappers:

mpicc - The MPI C99 compiler mpic++ and mpicxx - The MPI C++03 compiler mpic++11 and mpicxx11 - The MPI C++11 compiler

Non-MPI wrappers (in the wbin directory):

bgclang (or powerpc64-bgq-linux-clang) - The C99 compiler bgclang++ (or powerpc64-bgq-linux-clang++) - The C++03 compiler bgclang++11 (or powerpc64-bgq-linux-clang++11) - The C++11 compiler

The C++11 wrappers don't just add -std=c++11(or -std=gnu++11, as they actually do), but also

switch your STL implementation from the system'slibstdc++ to libc++ to provide a fully-conforming C++11 environment.

General Usage

Use Clang “just like” gcc: Clang tries to keep its options compatible with gcc (including for warnings).

Here are some notable optimization options:

● -ffast-math – Allow non-IEEE-conforming transformations● -fopenmp – Enable OpenMP support● -ffp-contract=off – Don't form FMAs by default (the bgclang default is “fast”)● -fno-strict-aliasing – Don't assume ISO strict type-based aliasing rules● -fno-vectorize – Turn off loop vectorization● -fno-slp-vectorize – Turn off SLP vectorization● -mno-qpx – Turn off all QPX vector support

Like bgxlc and gcc, bgclang defaults to static linking (except when address sanitizer is enabled).

In order to use dynamic linking:

1) Compile all files with -fPIC2) Link shared libraries (.so files) using -shared3) Link executables using -dynamic

If you don't (or use -shared instead), you'll get this error:“Application executable ELF header contains invalid value,

errno 8 Exec format error"

Autovectorization, software prefetching, etc.

a[i] = b[i] + 1.0;Uses QPX and dcbt,

essentially perfect (at least for one thread).

Some are better than bgxlc,some are worse...

General Usage (cont.)

Language levels:

● bgclang – Defaults to “gnu99” (c99 + GNU extensions)

● bgclang++ - Defaults to C++03 + GNU extensions

● bgclang++11 – Defaults to C++11 + GNU extensions (with libc++)

The BG/Q SPI/CNK headers won't compilein strict C99 mode, so don't try and use them

with -std=c99.

If you're getting linking errors regarding inlinefunctions, try passing: -fgnu89-inline

(unlike gcc, Clang defaults to the C99 inline rules)

Because bgclang++11 uses libc++ as its STL, which is not ABI-compatbilewith the libstdc++ used by gcc and bgclang++, you will get linking errorsif you try to link together code compiled with bgclang++11 and bgclang++

if the functions have STL types in function signatures.

OpenMP

Enable OpenMP support with: -fopenmp

✔ Full OpenMP 3.1 support with some OpenMP 4 features

✔ Derived from Intel's clang-omp project and uses Intel's libiomp5 runtime library (patched).

✔ The runtime library has not yet been optimized for the BG/Q

#pragma omp simdfor (int i = 0; i < n; ++i) { ...}

OpenMP 4 SIMD directives are included!

Vector intrinsics and math functions

✔ bgclang supports the vector4double type and IBM-style QPX intrinsics (vec_add), etc.

✔ You can use IBM's SIMD MASS library from bgclang

✔ bgclang comes with its own SIMD math library (derived from Naoki Shibata's SLEEF library):

#include <qpxmath.h>

// the following functions are available (the functions with the _u1 suffix have no more than 1 ulp error):

vector4double xldexp(vector4double x, const int *q);void xilogb(vector4double d, int *l);

vector4double xsin(vector4double d);vector4double xcos(vector4double d);void xsincos(vector4double d, vector4double *ds, vector4double *dc);vector4double xtan(vector4double d);vector4double xasin(vector4double s);...vector4double xsin_u1(vector4double d);vector4double xcos_u1(vector4double d);...

There are 35 double-precision functions,and also single-precision functions such as:

vector4double xsinf(vector4double d);

Bgclang's autovectorizer can automatically generatecalls to these functions!

Link-time optimization

Code optimizations done as part of the linking process: uses a linker plugin that runs LLVM IR optimizations and code generation while you link: the “compile” is faster, the “link” is slower!

bgclang -flto a.c -c -o a.o

bgclang-ar q a.a a.o

bgclang b.c -c -o b.o

bgclang -flto a.a b.o -o main

You must use bgclang-ar to make archives,and bgclang-nm to list the symbols in the objects and archives.

Address Sanitizer

Address sanitizer is an instrumentation-based memory debugging tool to catch:

✔ Out-of-bounds memory accesses (buffer overruns)

✔ Use-after-free errors

Add: -fsanitize=address when both compiling and linking. This will cause you to use dynamic linking!

Memory space in c1 mode:[0x5000001600000, 0x5000001700000] - L2 Interface[0x5000001500000, 0x5000001600000] - L1P Interface[0x4064880000000, 0x40648c0000000] - L2 Atomic Operations[0x03fdc0000000, 0x03fe00000000] - Memory-Mapped I/O[0x007000000000, 0x007400000000] - Process Windows[0x003245000000, 0x003245000000] - Shared Memory[0x003003000000, 0x003003100000] - Dynamic Loader (ld64.so.1)[0x001c07000000, 0x002000000000] - Heap, Stack, Dynamic Libs[0x000001000000, ...] - App. Text and Constant Data

ASAN will map:[0x001e037ff000, 0x001e23620000]500 MB shadow region.

On most systems, we can allocate unreserved pages (“overcommit”), but we can't under CNK...

Address Sanitizer (cont.)

Because the instrumentation is more complicated, and because the A2 lacks out-of-order dispatch, etc., the overhead is greater than on a server-grade x86_64 chipset.

Nevertheless, it is still possible to debug using realistic problem sizes at scale.

Future Work

● Polyhedral optimizations

● Profile-guided optimization (with customization to work well with MPI, etc.)

● Better optimizations, etc. (all of the regular compiler improvements one would like to see)

● C++14, 17, and maybe even some Fortran

Farther in the future: Combining static analysis, profiling and optimizer feedback to suggest useful changes (data-structure changes, pragmas, etc.) to the user.

Links

● https://www.alcf.anl.gov/user-guides/bgclang-compiler

● http://trac.alcf.anl.gov/projects/llvm-bgq

● http://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss

And http://llvm.org along with its various project sites!

All these links are available from here.

Acknowledgments

➔ Jeff Hammond, Kalyan Kumaran, Mike Papka, Katherine Riley, William Scullin, and everyone else at ALCF who has supported the development of bgclang

➔ Ramesh Balakrishnan, Michael Blocksome, Thomas Gooding, Michael Kruse, Jack Poulson, Erik Schnetter, and the many others who provided early and continuing testing

➔ Chandler Carruth, Tobi Grosser, Chris Lattner, Jakob Olesen, Nadav Rotem, Bill Schmidt, Arnold Schwaighofer, Andy Trick, and all other members of the LLVM community who assisted, directly or indirectly, with this work

➔ ALCF is supported by DOE/SC under contract DE-AC02-06CH11357

In short, a lot of people have contributedto making this possible!