Trust boundaries...

Mateusz Kocielskim.kocielski@logicaltrust.net

LogicalTrust

ConfidenceKraków, Poland, May 2015

$ whoami

I pentester at LogicalTrust as $DAILYJOBI open source committer:

I PHP - bug fixingI NetBSD - libsaslc(3) & random things...

I security:I PHP - CVE-2010-1868, CVE-2010-1917, CVE-2010-4150,

CVE-2010-4156, CVE-2011-1938, ...I stunnel - CVE-2013-1762I OpenSSH - CVE-2011-0539I Apache - CVE-2014-0117, CVE-2014-0226I FreeBSD - CVE-2015-1414I ...

Trust boundaries?

Our goal is to investigate how many assumptions are hidden in beliefthat some piece of code is secure.

We want to harbour a seed of doubt in those, who are too confidentand encourage those, who cannot sleep at night.

I want to share with you some of my findings...

What is trust? - Wikitionary

trust (source: http://en.wiktionary.org/wiki/trust):

I 1. Confidence in or reliance on some person or quality.

I 4. That which is committed or entrusted; something received inconfidence; a charge.

I 5. That upon which confidence is reposed; ground of reliance; hope

I 7. The condition or obligation of one to whom anything is confided;responsible charge or office.

I ...

What is trust?

Source: http://pl.wiktionary.org/wiki/Plik:Male˙House˙Sparrow˙%28Passer˙domesticus%29˙feeding˙from˙hand.jpg

In software we trust?

By saying ”I trust this software”, we’d like to think that:I it doesn’t hurt us ­ lack of bugs (at least those connected

somehow with security).I it does what we think it does (and nothing else) ­ lack of

backdoors etc.

We all have our own definition of trusted software, moreover thisdefinition varies on many factors like requirements etc. For thispresentation we can assume that everyone in this room are (at

least) a bit paranoid.

A short survey...

Can we trust following software:I LinuxI WindowsI MS OfficeI ApacheI OpenBSDI OpenSSLI OpenSSHI working as ”cloud” (innovative clusters placed in cloud operating

with synergy crap) - GMail, Dropbox, Google Drive, ...

Keeping data in cloud

Keeping data in cloud

Source: http://download.fsfe.org/advocacy/stickers/thereisnocloud/thereisnocloud-v2-74x74.pdf, CC-By-Sa MarkusMeier

Simple observations

It’s easier to trust software:I small (e.g. bc vs. Excel)I open source

I ”given enough eyeballs, all bugs are shallow” - Linus’s LawI we can verify code ourselfI who is paranoid enough not to trust binary sets of your favourite

distro? :)I Reproducible builds can solve that problem

I with good reputation (e.g. OpenBSD vs. Windows)

...but let’s talk about the facts

Here is the plan:I take a piece of software that we believe is secureI run it and expose it to the internetI verify what we meant by claiming it is secure?I think about the consequences

So which software to choose?...

Who’ll be our hero?!

...by acclamation we’re choosing:

openssh

Why OpenSSH?

I small - version 6.8 has around 90K LOCI good design (priv. separation etc.)I well written (security in mind...)

I ”I am always looking for bugs in OpenSSH as it is written in clearto read source code and has very strong security.” - Kingcope(source: http://kingcope.wordpress.com/2013/09/13/opensslopenssh-ecdsa-authentication-code-inconsistent-return-values-no-vulnerability/)

I Everybody uses it:I zmap (source: https://zmap.io/paper.pdf): (...) port 22 hit rate:

0.57% (...)

I generally recognized as a safe and robust software

OpenSSH on the screen

Source: http://nmap.org/movies/matrix/trinity-nmapscreen-hd-crop-1200x728.jpg

...so yesterday...

Trust boundaries

OpenSSH

libopenbsd-compat

openssl

libc

libz

kernel CPU/hw physics ???

Simple observations pt. 2

I layer n (lower) is broken → layer n + 1 (higher) is brokenI even (relatively) simple software has complex foundations

Source: http://vignette1.wikia.nocookie.net/uncyclopedia/images/c/c1/CaptobviousChooseOption.gif/revision/latest?cb=20070106161415

Trust boundaries

OpenSSH

libopenbsd-compat

openssl

libc

libz

kernel CPU/hw physics ???

OpenSSH - potential stack overflowkey.c:static intcert_parse(Buffer *b, Key *key, const u_char *blob, u_int blen){

u_char *principals, *critical, *exts, *sig_key, *sig;u_int signed_len, plen, clen, sklen, slen, kidlen, elen;Buffer tmp;char *principal;int ret = -1;int v00 = key->type == KEY_DSA_CERT_V00 ||key->type == KEY_RSA_CERT_V00;

[...]if ((key->cert->signature_key = key_from_blob(sig_key,sklen)) == NULL) {

[...]Key *key_from_blob(const u_char *blob, u_int blen){[...]

if (key_is_cert(key) && cert_parse(&b, key, blob, blen) == -1) {error("key_from_blob: can’t parse cert data");goto badkey;

}[...]

Fixed a year or two ago, left here for historical reasons... key from blob canbe called remotely using pubkey authentication. If you’re interested then takea look to previous versions of auth2-pubkey.c.

OpenSSH - potential stack overflow

(Un)fortunately certificate is handled by the Buffer structure, whichmaximum length is bounded:buffer.c:#define BUFFER_MAX_CHUNK 0x100000[...]void *buffer_append_space(Buffer *buffer, u_int len){

u_int newlen;void *p;

if (len > BUFFER_MAX_CHUNK)fatal("buffer_append_space: len %u not supported", len);

No cookies this time, but maybe somewhere in space there are systems (orconfigurations) which are exploitable. (Bounded stack + somethingimportant near to it).

OpenSSH - CVE-2011-0539

”OpenSSH does not properly initialise a nonce field with random datawhen generating legacy certificates (”-t” command line option ofssh-keygen). This can result in certain stack memory being used asnonce, which can lead to the disclosure of potentially sensitiveinformation.” - source: http://secunia.com/advisories/43181

key.c patch:/* -v01 certs put nonce first */

+ arc4random_buf(&nonce, sizeof(nonce));if (!key_cert_is_legacy(k)) {- arc4random_buf(&nonce, sizeof(nonce));buffer_put_string(&k->cert->certblob, nonce, sizeof(nonce));}

Trust boundaries

OpenSSH

libopenbsd-compat

openssl

libc

libz

kernel CPU/hw physics ???

Libraries - libopenssh-compat

I OpenBSD specific functions for !OpenBSD platformsI heavily relies on the OpenSSL (e.g. rng implementation)I let’s trust it! :)

Libraries - OpenSSL

I uber complex library which implements various crypto stuffI around 450 K LOC of hard-core CI there are rumours, that some people learnt C by writing this

library....I among developers this library has rather a bad reputation

I Why no SSL? -https://www.varnish-cache.org/docs/trunk/phk/ssl.html

I OpenSSL is written by monkeys -http://www.peereboom.us/assl/assl/html/openssl.html

I lots of projects rely on OpenSSL (700+ ports in the FreeBSDports tree)

I we would rather be sorry if someone found a bug in this code...

Libraries - OpenSSL - top comments

grep -Ri xxx .:crypto/asn1/a_strex.c: fld_len = 0; /* XXX: what should this be? */apps/passwd.c: /* XXX: really we should know how to print a size_t, not cast it */ssl/t1_enc.c: /* I should fix this up TLS TLS TLS TLS TLS XXXXXXXX */ssl/d1_pkt.c: /* XXX: check what the second ’&& type’ is about */include/openssl/pem.h: /* XXX(ben): don#t think this is used!... */crypto/asn1/asn1_mac.h: /* BIG UGLY WARNING! This is so damn ugly I wanna puke. Unfortunately,

some macros that use ASN1_const_CTX still insist on writing in the inputstream. ARGH! ARGH! ARGH! Let’s get rid of this macro package.... */

crypto/objects/obj_dat.c: ad.obj=(ASN1_OBJECT *)a; /* XXX: ugly but harmless */crypto/objects/obj_lib.c: return((ASN1_OBJECT *)o); /* XXX: ugh! Why? What kind of

duplication is this??? */crypto/engine/eng_cryptodev.c: /* XXXX just disable all digests for now, because it sucks. */

More stuff @ https://twitter.com/OpenSSLFact

Libraries - LibreSSL

I fork done by OpenBSD guys right after the famousCVE-2014-0160

I ”libressl is a version of the tls/crypto stack forked from openssl in2014, with goals of modernizing the codebase, improving security,and applying best practice development processes.” -http://www.libressl.org

I The first 30 days, and where we go from here -http://www.openbsd.org/papers/bsdcan14-libressl/

I More than 30 Days Later -http://www.openbsd.org/papers/eurobsdcon2014-libressl.html

Libraries - LibreSSL - favicon.ico

Source: http://www.libressl.org/favicon.ico

Libraries - LibreSSL - BN

Code is complex, so let’s pick some small sublibrary - BigNumberimplementation:

I part of the lib which is used in many other parts of libraryI should be rather easy to implement (is it? :))

Libraries - LibreSSL - fun fact

LibreSSL uses (sometimes) three zero representations: 0, −0 and 0..0:I those ”zeros” are values returned by BN functions...I once 0 = 0..0 = −0, sometimes not...

...inconsistency can’t hurt us..., can it?

Libraries - LibreSSL - off-by-one #1Let’s take BIGNUM −0 and apply it to the BN bn2hexcrypto/bn/bn print.c:

char *BN_bn2hex(const BIGNUM *a){char *buf;char *p;

buf=(char *)OPENSSL_malloc(a->top*BN_BYTES*2+2);[...]p=buf;if (a->neg) *(p++)=’-’;if (BN_is_zero(a)) *(p++)=’0’;for (i=a->top-1; i >=0; i--)[...]*p=’\0’;

How to get −0 in LibreSSL? It is a task for the listener

Libraries - LibreSSL - off-by-one #2

Let’s call BN rand(BN, 1, 1, 0)- bnrand crypto/bn/bn rand.c:

static intbnrand(int pseudorand, BIGNUM *rnd, int bits, int top, int bottom){unsigned char *buf = NULL;[...]bytes = (bits + 7) / 8;bit = (bits - 1) % 8;buf = OPENSSL_malloc(bytes);if (top != -1) {if (top) {if (bit == 0) {buf[0] = 1;buf[1] |= 0x80;

[...]

Libraries - libzzlib - easy, small library for data compression/decompressioninflate.c:int ZEXPORT inflateInit_(strm, version, stream_size)z_streamp strm;const char *version;int stream_size;{return inflateInit2_(strm, DEF_WBITS, version, stream_size);

}[...]int ZEXPORT inflateInit2_(strm, windowBits, version, stream_size)z_streamp strm;int windowBits;const char *version;int stream_size;{[...]struct inflate_state FAR *state;[...]if (strm->zalloc == (alloc_func)0) {[...]}state = (struct inflate_state FAR *)ZALLOC(strm, 1, sizeof(struct inflate_state));[...]strm->state = (struct internal_state FAR *)state;state->window = Z_NULL;ret = inflateReset2(strm, windowBits);[...]

}

Developer can install own memory allocation procedure

Libraries - libz cont.

inflate.c:int ZEXPORT inflateReset2(strm, windowBits)z_streamp strm;int windowBits;{int wrap;struct inflate_state FAR *state;if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR;state = (struct inflate_state FAR *)strm->state;if (windowBits < 0) {wrap = 0;windowBits = -windowBits;

} else {wrap = (windowBits >> 4) + 1;

}/* set number of window bits, free window if different */if (windowBits && (windowBits < 8 || windowBits > 15))return Z_STREAM_ERROR;

if (state->window != Z_NULL && state->wbits != (unsigned)windowBits) {ZFREE(strm, state->window);state->window = Z_NULL;

}[...]

OpenSSH is not affected, but how about other popular software? YES IT IS!

Libraries - libc - dobule free - getaddrinfo IDN

$ traceroute $(printf "\302a")*** glibc detected *** traceroute: munmap_chunk(): invalid pointer: 0x00007fff1b43a547 ***======= Backtrace: =========/lib64/libc.so.6(cfree+0x166)[0x32244758c6]/lib64/libc.so.6[0x32244bc116]/lib64/libc.so.6(getaddrinfo+0x21a)[0x32244be94a]traceroute[0x402926]traceroute[0x4029f1]traceroute[0x406281]traceroute[0x403546]/lib64/libc.so.6(__libc_start_main+0xf4)[0x322441d9f4]traceroute[0x401619]======= Memory map: ========00400000-00409000 r-xp 00000000 68:06 7103807 /bin/traceroute00608000-00609000 rw-p 00008000 68:06 7103807 /bin/traceroute00609000-0060a000 rw-p 00609000 00:00 000808000-00809000 rw-p 00008000 68:06 7103807 /bin/traceroute00ff7000-01018000 rw-p 00ff7000 00:00 0 [heap]3224000000-322401c000 r-xp 00000000 68:06 7332914 /lib64/ld-2.5.so

http://www.openwall.com/lists/oss-security/2015/01/27/11

Trust boundaries

OpenSSH

libopenbsd-compat

openssl

libc

libz

kernel CPU/hw physics ???

Kernel

I Operating system kernel has usually millions code linesI Next to each other:

I network protocolsI filesystemsI sound systemI ...programming ZOO

Kernel - FreeBSD - IGMP - CVE-2015-1414netinet/igmp.c:case IGMP_VERSION_3: {

struct igmpv3 *igmpv3;uint16_t igmpv3len;uint16_t srclen;int nsrc;[...]igmpv3 = (struct igmpv3 *)igmp;/* Validate length based on source count. */nsrc = ntohs(igmpv3->igmp_numsrc);srclen = sizeof(struct in_addr) * nsrc;if (nsrc * sizeof(in_addr_t) > srclen) {

IGMPSTAT_INC(igps_rcv_tooshort);return;

}[...]igmpv3len = iphlen + IGMP_V3_QUERY_MINLEN +srclen;

if ((m->m_flags & M_EXT ||m->m_len < igmpv3len) &&(m = m_pullup(m, igmpv3len)) == NULL) {IGMPSTAT_INC(igps_rcv_tooshort);return;

}igmpv3 = (struct igmpv3 *)(mtod(m, uint8_t *)+ iphlen);

if (igmp_input_v3_query(ifp, ip, igmpv3) != 0) {[...]

Kernel - NetBSD - libprop - #1

common/lib/libprop/*:

buf = malloc(pref->pref_len + 1, M_TEMP, M_WAITOK | M_CANFAIL);if (buf == NULL)return (ENOMEM);

error = copyin(pref->pref_plist, buf, pref->pref_len);

It’s not exploitable on x86/amd64, but may hurt sparc64.

Kernel - NetBSD - libprop - #2common/lib/libprop/*:

#define _PROP_EOF(c) ((c) == ’\0’)#define _PROP_ISSPACE(c) \((c) == ’ ’ || (c) == ’\t’ || (c) == ’\n’ || (c) == ’\r’ || \_PROP_EOF(c))

struct _prop_object_internalize_context *_prop_object_internalize_context_alloc(const char *xml){[...]/** Skip any whitespace and XML preamble stuff that we don’t* know about / care about.*/for (;;) {while (_PROP_ISSPACE(*xml))xml++;

Trust boundaries

OpenSSH

libopenbsd-compat

openssl

libc

libz

kernel CPU/hw physics ???

CPU/HWI Bugs in the CPUs

I Kris Kaspersky - Remote Code Execution Through Intel CPUBugs - HITB 2009

I Pentium F00F bugI Pentiun FDIV bug

I Bugs in the processor’s microcode - XEON example:I (...) Erratum AAK167/BT248: ”If a logical processor has EPT

(Extended Page Tables) enabled, is using 32-bit PAE paging, andaccesses the virtual-APIC page then a complex sequence ofinternal processor micro-architectural events may cause anincorrect address translation or machine check on either logicalprocessor. (...) -http://lists.debian.org/debian-user/2013/09/msg00126.html

I Bugs in (random) devices (incl. its firmware)I have you ever wondered what’s inside the firmware?

I Backdoors in devicesI Intel, NSA & RDRAND ... :)

Trust boundaries

OpenSSH

libopenbsd-compat

openssl

libc

libz

kernel CPU/hw physics ???

WARNING

Dragons live here!For the moment I’ll talk about things that:

I I don’t understand

I relate to philosophy rather than thug life...

Physics

I Have you ever heard about bugs in physics?I What will happen if our openssh will run in space?

I Have we tested it in such environment?I Are the physics’ laws consistent?I What if somebody can remotely change CPU/memory state?

I Haha, I hope you didn’t believe that?I Can we defend somehow?

I encoding correction (i.e. ECC)I mirroring hardware

I OpenSSH will not suffer...But if we write a code to manage spacerobot, then it’s not funny anymore...

Trust boundaries

OpenSSH

libopenbsd-compat

openssl

libc

libz

kernel CPU/hw physics ???

???

I In God We Trust... Everything Else We Test...I ...maybe some three letter agency can manipulate the physics’

laws... :)

Fallen actors

I Protocols: ARP, IP, TCP, SSH, ...I Algorithms: DH, RSA, DSA, EC, ...

Fallen actors - protocols

I There are bugs in protocols:I SSL - CRIME, BEAST...I TCP - SMURF, SYNI ...

I We can formally proof that there are no bugs in the protocol...I ...but in order to do so we need a model which is usually

simplification of realityI ”the absence of proof is not the proof of absence”

Forgotten elements - algorithms

I We can use math tools to verify that RSA does the jobI We can do it modulo some model

I Do you know that RSA relies on Factorization /∈ PI Do you know that we don’t know if P 6= NP?I Do you know that three letters agencies spent lots on money on

breaking crypto?I Finally, do you know that basing on ZF or ZFC we can’t be sure

what we do? :)I Assuming that we verified our algorithm, there’s still long road:

I we have to choose language (with formal semantics)I code our algorithm in chosen languageI verify that we coded exactly what we meant (tests, formal proofs)I having verified code we should recall the forgotten actor...

Forgotten compiler

I Attentive listener recalls that: ”OpenSSH is written in C, butprocessor is fed with machine code...”

I How do we know that we do not introduce any bugs incompilation process?

I Finding and Understanding Bugs in C Compilers(http://www.cs.utah.edu/˜regehr/papers/pldi11-preprint.pdf):”Compilers should be correct. To improve the quality of C compilers, wecreated Csmith, a randomized test-case generation tool, and spentthree years using it to find compiler bugs. During this period wereported more than 325 previously unknown bugs to compilerdevelopers. Every compiler we tested was found to crash and also tosilently generate wrong code when presented with valid input. (...)”

CompCert

CompCert - verified compiler from C90 subset to PPC, ARM, x86I Verified = result code is 100% consistent with C90 semanticsI Useful when compiling critical code for embedded solutionsI Its translation process is INSANELY complex:

Project’s homepage: http://compcert.inria.fr/

Conclusions

I Trust in even the simplest piece of code implies belief in hiddenassumptions which we usually don’t even realize

I We’re forced to believe that each ”layer” works wellI For some reason three letters agencies are spending millions on

breaking thingsI Crypto is working, washing machines do the jobI Some of the instances of OpenSSH which I run are reachable from

the Internet, and usually I sleep well...

I Sometimes it’s good to turn on paranoia when thinking about ourcomputers

I ”Better paranoid than sorry” - comment from the OpenSSHproject

I Read the code or not be surprisedI We’ll never get rid of bugs in software :)

Credits

Large parts of this presentation were done in cooperation withMarek Kroemeke and Filip Palian, THANKS!

Some reading material

I http://c2.com/cgi/wiki?ProofsCantProveTheAbsenceOfBugsI http://cm.bell-labs.com/who/ken/trust.htmlI http://en.wikipedia.org/wiki/Argument˙from˙ignoranceI http://reproducible.alioth.debian.org/presentations/

2014-02-01-FOSDEM14.pdfI https://wiki.freebsd.org/ReproducibleBuildsI https://wiki.debian.org/ReproducibleBuildsI https://fosdem.org/2015/schedule/event/stretching˙out˙for˙

trustworthy˙reproducible˙builds/attachments/paper/668/export/events/attachments/stretching˙out˙for˙trustworthy˙reproducible˙builds/paper/668/2015˙01˙31˙FOSDEM15.pdf

I http://www.cl.cam.ac.uk/research/security/capsicum/

Time for questions (and maybe answers)

Q&A