The Science of Cyber: Quantifying software risk.plusweb.org/Portals/0/CHAPTER/Mudge Zatko -...

A 501(c)3 Corporation

The Science of Cyber: Quantifying software risk.

[email protected]

0

The problem:

The information security field suffers from a lack of hard data, especially when it comes to measuring risk in software and systems.

1

Vulnerability Title Fix Avail? Date Added

XXXXXXXXXXXX XXXXXXXXXXXX Local Privilege Escalation Vulnerability No 8/25/2010

XXXXXXXXXXXX XXXXXXXXXXXX Denial of Service Vulnerability Yes 8/24/2010

XXXXXXXXXXXX XXXXXXXXXXXX Buffer Overflow Vulnerability No 8/20/2010

XXXXXXXXXXXX XXXXXXXXXXXX Sanitization Bypass Weakness No 8/18/2010

XXXXXXXXXXXX XXXXXXXXXXXX Security Bypass Vulnerability No 8/17/2010

XXXXXXXXXXXX XXXXXXXXXXXX Multiple Security Vulnerabilities Yes 8/16/2010

XXXXXXXXXXXX XXXXXXXXXXXX Remote Code Execution Vulnerability No 8/16/2010

XXXXXXXXXXXX XXXXXXXXXXXX Use-After-Free Memory Corruption Vulnerability No 8/12/2010

XXXXXXXXXXXX XXXXXXXXXXXX Remote Code Execution Vulnerability No 8/10/2010

XXXXXXXXXXXX XXXXXXXXXXXX Multiple Buffer Overflow Vulnerabilities No 8/10/2010

XXXXXXXXXXXX XXXXXXXXXXXX Stack Buffer Overflow Vulnerability Yes 8/09/2010

XXXXXXXXXXXX XXXXXXXXXXXX Security-Bypass Vulnerability No 8/06/2010

XXXXXXXXXXXX XXXXXXXXXXXX Multiple Security Vulnerabilities No 8/05/2010

XXXXXXXXXXXX XXXXXXXXXXXX Buffer Overflow Vulnerability No 7/29/2010

XXXXXXXXXXXX XXXXXXXXXXXX Remote Privilege Escalation Vulnerability No 7/28/2010

XXXXXXXXXXXX XXXXXXXXXXXX Cross Site Request Forgery Vulnerability No 7/26/2010

XXXXXXXXXXXX XXXXXXXXXXXX Multiple Denial Of Service Vulnerabilities No 7/22/2010

Additional security layers often create vulnerabilities…

Awaiting Vendor Reply/Confirmation Awaiting CC/S/A use validationVendor Replied – Fix in developmentColor Code Key:

6 of the vulnerabilities are in security

software

vulnerability watchlist

2

– Certifications and evaluations• Laborious and time consuming• Evaluate processes and specifications rather than

the resulting product• Manually intensive after the fact audits by

“experts”

– Industry and Marketing labels• Secured by SSL, Safe and Secure!• Misuse of certification and evaluation labels• Not comparative for similar products• Producers of products incentivized to not exceed

bare minimum

– Source code review• Requires access to source code• NDAs and incentive structure work against

consumers• Misses compiler, linker, loader, and processor

specifics

– Legislation • Impede reverse engineering and disclosure• Remove liability for vendors• Head-in-sand approach is tactical, not strategic…

Industry’s approach, and why it is insufficient…

Source: www.usb-k.comSource: csrc.nist.gov

Source: dhs.arkansas.gov

Source: centuria.com

Source: www.ul.com

Source: dmca.com

Source: www.cisco.com

Source: wipo.int

3

Comparative security of applications in common Operating Systems

This approach supports assertions like…

On average software A is X% less/more likely to have a new cyber loss than software B.

4

5% 50% 95%

Relative hardening line

Soft

er T

arge

tsH

arder Targets

Score

2016 Q4 – 2017 Q1

5

5% 50% 95%


Soft

er T

arge

tsH

arder Targets

Score

Office 2011 Avg (16.5)

Office 2011

AutoUpdate (7)

Office 2011 Main

Applications (45.6)

2016 Q4 – 2017 Q1

6

5% 50% 95%


Soft

er T

arge

tsH

arder Targets

Score

Office 2011 Avg (16.5)

Office 2011

AutoUpdate (7)

Office 2011 Main

Applications (45.6)

Office 2016 Main

Applications (57.5)

Office 2016

AutoUpdate (64)

Office 2016 Avg (77)

2016 Q4 – 2017 Q1

7

5% 50% 95%


Soft

er T

arge

tsH

arder Targets

Score

2016 Q4 – 2017 Q1

Chrome (75)Safari (59)32b Firefox (35)

8

Chrome (95)64b Firefox

32b Firefox (51)

Edge (101) 10

Application CITL Ranking 2017 Cash Value for Exploiting*

Microsoft Edge 1 $80,000.00

Google Chrome 2 $80,000.00

Apple Safari 3 $50,000.00

Mozilla Firefox 4 $30,000.00

Market value for new exploits: SW

* Trend Micro Pwn20wn CanSecWest 2017

Big Takeaway:

The market price for a new exploit against a target application is a measure of the level of effort an attacker must spend to create said exploit.

11

Histograms of 3 common OSes

Windows 10 (Windows Server 2016)

Scores:5th percentile : 54 (approx.)50th percentile : 72 (approx.)95th percentile : 102 (approx.)

OSX El Capitan


Linux Ubuntu 16.0 LTS


12

Market value for new exploits: OS escalation

OS CITL Ranking 2017 Cash Value for Exploiting*

Windows 10 1 $30,000.00

Mac OS X 2 $20,000.00

Ubuntu (Linux) 3 $15,000.00 * Trend Micro Pwn20wn CanSecWest 2017

13

Windows 10 (Windows Server 2016)


OSX El Capitan


Linux Ubuntu 16.0 LTS


A quick peek at the Internet of Things landscape…

14

Our physical systems are vulnerable to cyber attacks…

Small group of academics took control of a car using Bluetooth and OnStar. They were able to disable the brakes, control the

accelerator, and turn on the interior microphone.[1]

Chinese cyber attack: “Highly sophisticated and targeted attack” on Google corporate infrastructure (known as Aurora)

False speedometer readingNote that the car is in park…[1] K. Koscher, et al. "Experimental Security Analysis of a Modern Automobile," in Proceedings of

the IEEE Symposium on Security and Privacy, Oakland, CA, May 16-19, 2010.

15

Default Linux, Hardened Linux, and IoT as viewed by basic safety features only (Application Armoring)

Linux – Ubuntu 16 (LTS distribution)

10k+ binaries

large attack surface

Plenty moderately soft and weak targets, some hardened targets

(The majority of cloud instances are behind this release…)

Custom Hardened Gentoo Linux

6k+ binaries

“Smaller” attack surface

Majority of targets are hard (via App Armoring only)

No code modified

One time NRE re-compilation of system and apps (less than ~$500)

LG Smart TV (LG UHD 4K HDR Smart LED TV 55uh8500)

2k+ binaries (1/3 of the “smaller” hardened Linux)

Perfect opportunity to make a smaller and harder target… TRIVIALLY!

Vast majority of targets extremely soft/fragile.

No good reason for this… (This is the common build for *lots* of IoT) 16

What are we measuring?

17

http://www.consumer.ftc.gov/articles/0072-shopping-home-appliances-use-energyguide-label

https://en.wikipedia.org/wiki/Monroney_sticker

https://en.wikipedia.org/wiki/Nutrition_facts_label#United_States

Monroney StickerNLEA

EnergyGuide

18

Static Analysis (Binaries)

Application Measurements

Complexity Measures:• Code size• Branch density• Stack adjusts• Cyclomatic complexity• …

Application Armoring• Compiler

• Stack guards• Function fortification• CFI/CPI• …

• Linker• ASLR• Segment and Section ordering• …

• Loader• Section, Segment execution chars• Allocations and access• Code signing / verification• …

Compartmentalization• Sandboxes, virtual machines, containers

Developer Hygiene• ~500 POSIX and ANSI functions

• Ick, Bad, Risky, Good• Consistency• Frequency/count• …

19

Static Analysis (Binaries)

Application Measurements

Complexity Measures:• Code size• Branch density• Stack adjusts• Cyclomatic complexity• …

Application Armoring• Compiler

• Stack guards• Function fortification• CFI/CPI• …

• Linker• ASLR• Segment and Section ordering• …

• Loader• Section, Segment execution chars• Allocations and access• Code signing / verification• …

Compartmentalization• Sandboxes, virtual machines, containers

Developer Hygiene• ~500 POSIX and ANSI functions

• Ick, Bad, Risky, Good• Consistency• Frequency/count• …

Car Analogy

How many moving parts, how spaghetti is the electrical wiring?

Seat BeltsAir BagsAnti-Lock BrakesSafety Glass…

Crumple zones

Craftsmanship, accuracy, competence…

20

Dynamic Analysis Components

a

a

a

Exploitability

Disruptability

Runtime Complexity

Illegal Instruction, Invalid Memory Reference, Bus Error

Highly Exploitable

Exponential – highly vulnerable

Highly Unstable

Depending on your environment, disruptability can be more important than exploitability.

Exploitability• Crashes resulting in RCE

Disruptability• Abnormal termination• Hangs

Runtime Complexity• Worst case algorithmic

complexity

• Braking distance• Skid pad

performance• Side and front

impact crash testing

• Rollover• Etc.

Car Analogy

21

CITL Dynamic Analysis (Quick Aside)

Quick Stats: First 285 targets dynamically stressed

94 of 285 crashed (33%)Total unique crashes: 6499

Of those automatically analyzed1237 crashes were “exploitable”

3698 were “unknown”

This is equivalent to having 1/3 of all cars on a {dyno,skid pad,smog tester} burst into flames and explode!

Wait until we hook the applications up to an actual crash inducing harness…Source: www.pcb.its.dot.gov22

Static Analysis

Dynamic Analysis

Analytics

Report Generation

Predictive Model

f(x)

f(x) =

Update predictive model.Store static and dynamic results.

Predict dynamic results from model.Store static and predicted results.

Results DB

if

if

Target is in Dynamic DB

Target is NOT in Dynamic DB

CITL Measurement, Analysis, and Modeling Stack

Dynamic DB

Static DB

Re-run predictive analyses as model updates.

23

The value of coarse and fine grained analysis,as extracted from CITL detailed data sets.

The following slides contain exemplars using only the Application Armoring, and Function Hygiene subsections of CITL data.

24

Coarse Grained View: Application Armoring Focus3 OS X browsers

As an attacker, which browser is hardest to 0day in this view? 25

Fine Grained View:

A surprise discovery you could only find by looking at the binaries…

26

Q3 2016

27

Something unexpected lived in the bottom 5th

percentile across all platforms…

Source code analysis alone misses it, only evident via binary analysis.

Q3 2016

28

the story of Anaconda...

29

BAD

BAD

(Really BAD)

(0,0) = GOOD

Fixed addresses and symbols per file

30

BAD

BAD

(Really BAD)

(0,0) = GOOD

Fixed addresses and symbols per file

31

The story ofSource: wikipedia

32

33


What it is: A DARPA funded freemium pre-packaged roll-up of Python and R interpreters and big-data analytics libraries and packages.

(Pytnon interpreters, NumPy, Pandas PyData, OpenSSL, libcurl, libxml, qmake, …).


Advertised Customers:

BoeingNielsen

Thomas ReutersBarclays

Kaiser Permanente,DisneyNIST

JP MorganL3 Communications

RaytheonApp NexusMicrosoft

HPCapital One

IBMPhilips,

FICOSurvey Monkey

MaycesCTC

AmazonCISCO

Met OfficeBridgestone

DuPontAkuna Capital

GeicoNOAA

The Weather ChannelLos Alamos National Labs

DARPALinkedInInQTel

SiemensBank of America

Citadel



34



BoeingNielsen





HPCapital One

IBMPhilips,

FICOSurvey Monkey

MaycesCTC

AmazonCISCO


DuPontAkuna Capital

GeicoNOAA


DARPALinkedInInQTel


Citadel



Bottom scores on ALL operating systems: Windows, Linux, OSX – WHY?!

35

36



BoeingNielsen





HPCapital One

IBMPhilips,

FICOSurvey Monkey

MaycesCTC

AmazonCISCO


DuPontAkuna Capital

GeicoNOAA


DARPALinkedInInQTel


Citadel




600+ Binaries

Linux: * 90% writeable GOT

* < 3% of files are (partially) fortified* < 15% have stackguards

OSX: * lacking ASLR* missing stackguards* missing fortification

Windows:* lacking CFI* non-safe SHE* no high-entropy VA for libs* …

37

The story of

Disclaimer: I use Anaconda because I run a 2 person shop. I accept the extra risk for convenience. Boeing, Disney, etc. should likely not be accepting this much needless risk…

Source: wikipedia


BoeingNielsen





HPCapital One

IBMPhilips,

FICOSurvey Monkey

MaycesCTC

AmazonCISCO


DuPontAkuna Capital

GeicoNOAA


DARPALinkedInInQTel


Citadel




GCC 4.1.2 from 2008 July 4RedHat 4.1.2-55 (which was released in 2005)GCC 4.4.7 from 2012 March 13RedHat 4.4.7-1 (which was released in 2005)

Apparently they recompile modern source using antiquated dev-environments:

Loses almost a DECADE of development environment security improvements! (Think 2017 Volvo built on 1970’s plant/assembly line)

600+ Binaries

Linux: * 90% writeable GOT

* < 3% of files are (partially) fortified* < 15% have stackguards

OSX: * lacking ASLR* missing stackguards* missing fortification

Windows:* lacking CFI* non-safe SHE* no high-entropy VA for libs* …

Disclaimer and context:

• This is not a pass|fail evaluation. Risk is quantified and is comparable between applications within a platform. Platforms will be evaluated similarly.

• The approach uses binaries only. No source code or cooperation from the vendor is needed. Targets are all types of binary applications (including OS Kernels, firmware, etc.).

• We look for overall vulnerability classes and trends, we do not look for specific instances.

• CITL is a 501(c)3 Corporation.

• Outside of our scope: configuration, past history (legacy vulns), interpreted scripts, and corporate policies. We predict the likelihood of future problems.

38

https://V50.io

https://www.thedigitalstandard.org/

http://www.consumerreports.org/

https://en.wikipedia.org/wiki/File:DARPA_Logo.jpg

https://en.wikipedia.org/wiki/File:Stripe_logo,_revised_2016.png

39

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force.

FAR Clause 252.235-7010 – Acknowledgement of Support and Disclaimer:

This material is based upon work supported by the United States Air Force under Contract No. FA8750-15-C-0282.

40

[email protected]

If you have commercial use or custom data access inquiries,that’s someone else:

[email protected]://www.v50.io/contact/

41

mailto:[email protected]

The Science of Cyber: Quantifying software risk.plusweb.org/Portals/0/CHAPTER/Mudge Zatko -...

Documents

Transcript of The Science of Cyber: Quantifying software risk.plusweb.org/Portals/0/CHAPTER/Mudge Zatko -...