Fuzzing:The New Unit Testing

C++ Russia 2017, Moscow, Feb 25Dmitry Vyukov, dvyukov@, Google

Agenda● What is fuzzing● Coverage-guided fuzzing● Small tutorial● How to write effective fuzzers● Fuzzing@Google

What is Fuzzing?wikipedia.org/wiki/Fuzz_testing:

Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program.

Who cares?- We are not testing/checking anything!- Random data will not trigger any bugs!

Fuzzing can find lots of bugs- With the help of sanitizers:

- Use-after-free, buffer overflows- Uses of uninitialized memory- Memory leaks- Data races, deadlocks- Int/float overflows, bitwise shifts by invalid amount (other UB)

- Plain crashes: - NULL dereferences, uncaught exceptions, div-by-zero

- Resource usage bugs:- Memory exhaustion, hangs or infinite loops, infinite recursion (stack overflows)

- Logical bugs (lots of, see below)

Data is not necessary "white noise"- There is number of tricks to generate "not so random" data- May or may not require some human help- If used correctly achieves very impressive code coverage

What can be fuzzed?Anything that consumes complex inputs:

● Parsers of any kind (xml, json, asn.1, pdf, truetype, ...) ● Media codecs (audio, video, raster & vector images, etc)● Network protocols (HTTP, RPC, SMTP, MIME...)● Crypto (boringssl, openssl)● Compression (zip, gzip, bzip2, brotli, ...)● Formatted output (sprintf, template engines)● Compilers and interpreters (Javascript, PHP, Perl, Python, Go, Clang, ...)● Regular expression matchers (PCRE, RE2, libc’s regcomp)● Text/UTF processing (icu)● Databases (SQLite)● Browsers, text editors/processors (Chrome, vim, OpenOffice)● OS Kernels (Linux), drivers, supervisors and VMs

Must have for everything that consumes untrusted inputs, open to internet or otherwise security sensitive.

Types of Fuzzers- Grammar-based generation

- Generate random inputs according to grammar rules- Peach, packetdrill, csmith, gosmith, syzkaller

- Blind mutation- Requires a corpus of representative inputs, apply random mutations to them- ZZUF, Radamsa

- Grammar reverse-engineering- Learn grammar from existing inputs using algorithmic approach of machine learning- Sequitur algorithm, go-fuzz

- Symbolic execution + SAT solver- Synthesize inputs with maximum coverage using black magic- KLEE

- Coverage-guided fuzzers- Genetic algorithm that strives to maximize code coverage- libFuzzer, AFL, honggfuzz, syzkaller

- Hybrid

Coverage-guided fuzzingBuild the program with code coverage instrumentation;

Collect initial corpus of inputs (optional);

while (true) {

Choose a random input from corpus and mutate it;

Run the target program on the input, collect code coverage;

If the input gives new coverage, add mutation back to the corpus;

}

Coverage-guiding in actionif input[0] == '{' {

if input[1] == 'i' && input[2] == 'f' {if input[3] == '(' {

input[input[4]] = input[5]; // potential OOB write}

}}

Requires "{if(" input to crash, ~2^32 guesses to crack when blind.

Coverage-guiding:Guess "{" in ~2^8, add to corpus.Guess "{i" in ~2^8, add to corpus.Guess "{if" in ~2^8, add to corpus.Guess "{if(" in ~2^8, add to corpus.Total: ~2^10 guesses.See: AFL: Pulling JPEGs out of thin air

Mutations● erase/insert/change/shuffle bit/byte/bytes● crossover/splice 2 inputs● insert token from a dictionary● insert magic numbers (2^10±1, 2^16±1, 2^31±1, 2^32±1)● change an ASCII integer (e.g. "123" => "2465357635")● ...

Coverage flavoursBasic blocks:

... (A)if (...) { ... (B)}... (C)

-fsanitize-coverage=bb

Edges:

... (A)if (...) { ... (B)}... (C)

-fsanitize-coverage=trace-pc-guard

Gives better feedback signal.

Counters:

for (...) { ... (hit N times)}

-fsanitize-coverage=8bit-counters

Gives better feedback signalfor loops and recursion.

Cracking hashesWhat about more complex cases?

if (*(uint32_t*)input == crc32(input+4, size-4)) {...}

if (*(uint64_t*)input == 0xBCEBC041BADBALL) {...}

Cracking hashesIntercept comparison operations:

● compiler intercepts int comparisons (-fsanitize-coverage=trace-cmp)● runtime intercepts strcmp/memcmp and friends

Several possibilities:

● extract int/string literals and insert them into inputs● find one comparison operand in the input and replace with the other operand● use PC^POPCNT(op1^op2) as "coverage" signal (Hamming distance)

Dictionaries● User-provided

○ e.g. for HTTP: "HTTP/1.1", "Host", "Accept-Encoding"

● Automatically extracted from program○ memcpy(input, "HTTP/1.1", 8)

Tutorial"...one of the most highly regarded and expertly designed C++ library projects in the world"

boost.regex

(latest version 1.63, in boost since 1.18)

Tutorial: fuzzing functionAs simple as:

int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) { try { std::string str((char*)Data, Size); boost::regex e( str); boost::match_results<std::string::const_iterator> what; boost::regex_match(str, what, e, boost::match_default); } catch (const std::exception&) {} return 0;}

Tutorial: building (the hard part)1. Build boost with coverage and AddressSanitizer:

./b2 cxxflags="-fsanitize-coverage=trace-pc-guard -fsanitize=address" toolset=clang install

2. Build fuzzer with coverage, AddressSanitizer and libFuzzer:

clang++ fuzzer.cc -fsanitize-coverage=trace-pc-guard -fsanitize=address libFuzzer.a

The rest is at tutorial.libfuzzer.info

Demo

30 minutes, 13 bugs (ticket/12818):

AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:132 in re_skip_past_nullAddressSanitizer: heap-buffer-overflow basic_regex_parser.hpp:2599 in parse_perl_extensionAddressSanitizer: heap-buffer-overflow perl_matcher.hpp:221 in re_is_set_memberAddressSanitizer: heap-buffer-overflow perl_matcher.hpp:166 in re_is_set_memberAddressSanitizer: heap-buffer-overflow interceptors.inc:278 in strlenAddressSanitizer: stack-overflow basic_regex_creator.hpp:1054 in create_startmapAddressSanitizer: SEGV on unknown address 0x0000000016e0MemorySanitizer: use-of-uninitialized-value perl_matcher.hpp:166 in re_is_set_memberbasic_regex_parser.hpp:904: runtime error: shift exponent 325804978 is too large for 32-bit type 'unsigned int'basic_regex_parser.hpp:2599: runtime error: load of value 56794092, which is not a valid value for type 'syntax_element_type'a.out: perl_matcher_common.hpp:606: Assertion `r.first != r.second' failedDirect leak of 4096 byte(s) in 1 object(s) allocated in get_mem_block regex.cpp:204ALARM: working on the last Unit for 17 seconds

Will find more when these are fixed!

Results

Finding logical bugsNot only security/stability

- But we don't know the right result!- Use your imagination!

Finding logical bugs● sanity checks on results

○ uncompressesed image decoder: 100 byte input -> 100 MB output?○ function returns both error and object, or no error and no object○ know that some substring must present in output, but it is not○ encrypt, check that decryption with wrong key fails

● sometimes we do know the right result○ any sorting: check that each element is present, check that it's not descending○ building a trie: check size, all elements are present

● asserts○ assert(a == b)

Finding logical bugsRound-trip:

● encode-decode● serialize-deserialize● compress-decompress● encrypt-decrypt● assemble-disassemble

Checks:● decode-encode: check that encode don't fail● decode-encode-decode: check that second decode don't fail● decode-encode-decode: check that decode results are equal● encode-decode-encode: check that encode results are equal

Very powerful technique.

Finding logical bugsComparing two (or more) implementations gives phenomenal results:● check that output is equal● or at least check that ok/fail result is the same

○ e.g. gcc and clang both accept or reject the code

But I don't want to write the second impl!● there can be several libraries implementing the same (libxmlFoo vs libxmlBar)● implementation in a different language (re2 vs Go's regexp)● compare "fast but complex" with "slow but dumb" (sometimes easy to write)● compare different functions (marshalBinary vs marshalText)

Quick Quiz: how to fuzz clang-format?clang-format: shuffles whitespaces in a source file.

Let's imaging destiny of mankind depends on correctness of clang-format!

How would you fuzz test it?

Quick Quiz: how to fuzz clang-format?● run with asan/msan/ubsan● format twice, compare results (e.g. relies on unordered_map order)● format, then format result (must be idempotent)● strip all whitespaces, compare before/after● check violations of max line length● compile before/after (formatting breaks/unbreaks code)

Regression testingNormally you run fuzzer for a long time.

But any guided fuzzer accumulates corpus of inputs with max coverage.

And that's perfect for regression testing! Just run it once on every change!

Fuzzing@Google Why?- faster and faster development- more and more code- correctness is important- stability is still important- security is super important- want to move fast, but keep development costs under control

Traditional testing is not enough anymore!

Fuzzing@Google How?- Developers can write "fuzz tests"

- picked up by automatic large-scale fuzzing system- but also work as regression unit tests

- OSS-Fuzz: continuous fuzzing for OSS- 50+ projects, 190 fuzzers- libFuzzer, radamsa, AFL (coming)- 5000 cores

- ClusterFuzz: automated fuzzing for Chromium- 350 fuzzers- libFuzzer, radamsa, AFL, custom fuzzers- 12000 cores- Automatically files bugs and verifies fixes

- syzkaller: continuous fuzzing of Linux kernel- several upstream branches + android/chromeos- 100+ VMs + physical devices

Fuzzing@Google

~15'000 bugs

GLIBC MUSL LIBC pugixml PCRE ICU Freetype ffmpeg Harfbuzz SQLite Python OpenSSL BoringSSL libxml2 BPF Capstone file Radare2 gRPC WOFF2 LLVM Tensorflow libav FreeType2 Foxit libtiff Go Linux libexif libFLAC Little CMS Adobe Reader Adobe Flash Player Adobe DNG SDK ESET NOD32 ClamAV BitDefender poppler ghostscript dcraw qcms libwebp libwebm libvpx gipfeli libots Snapseed Dart VM IJG libjpeg-turbo libpng mozjpeg PHP Firefox Internet Explorer Apple Safari LibreOffice GnuTLS GnuPG OpenSSH bash tcpdump JavaScriptCore pdfium libmatroska libarchive wireshark ImageMagick lcms libbpg lame libsndfile less lesspipe strings dpkg rcs systemd-resolved libyaml Info-Zip unzip libtasn pfctl mandoc IDA Pro clamav nasm ctags mutt procmail fontconfig pdksh wavpack redis cmsgpack taglib privoxy perl libxmpradare2 SleuthKit X.Org exifprobe jhead Xerces-C metacam exiv btrfs Knot DNS curl wpa_supplicant dnsmasq imlib2 libraw libwmf uudecode MuPDF libbson libsass boost

Sales pitch● Fuzzing is complimentary to any other testing technique● Fuzzing is mandatory for anything security-related● Fuzzing finds LOTS of bugs● Fuzzing is easy to use

Call to action:

● choose 1 library that uses complex inputs (important or you suspect for bugs)● write a fuzzer● run locally with ASAN

Thanks!

Q&Ahttp://tutorial.libfuzzer.info

Dmitry Vyukov, dvyukov@

Referenceshttps://en.wikipedia.org/wiki/Fuzz_testinghttps://github.com/google/sanitizershttp://llvm.org/docs/LibFuzzer.htmlhttp://tutorial.libfuzzer.infohttp://lcamtuf.coredump.cx/afl/https://github.com/google/oss-fuzzhttps://github.com/google/syzkallerhttps://github.com/dvyukov/go-fuzz