Technical Report - Expected Exploitability: Predicting the Development ofFunctional Vulnerability Exploits

Octavian Suciu, Connor Nelson†, Zhuoer Lyu†, Tiffany Bao†, Tudor Dumitras,University of Maryland, College Park

†Arizona State University

AbstractAssessing the exploitability of software vulnerabilities at

the time of disclosure is difficult and error-prone, as featuresextracted via technical analysis by existing metrics are poorpredictors for exploit development. Moreover, exploitabilityassessments suffer from a class bias because “not exploitable”labels could be inaccurate.

To overcome these challenges, we propose a new metric,called Expected Exploitability (EE), which reflects, over time,the likelihood that functional exploits will be developed. Keyto our solution is a time-varying view of exploitability, adeparture from existing metrics. This allows us to learn EEusing data-driven techniques from artifacts published afterdisclosure, such as technical write-ups and proof-of-conceptexploits, for which we design novel feature sets.

This view also allows us to investigate the effect of thelabel biases on the classifiers. We characterize the noise-generating process for exploit prediction, showing that ourproblem is subject to the most challenging type of label noise,and propose techniques to learn EE in the presence of noise.

On a dataset of 103,137 vulnerabilities, we show that EEincreases precision from 49% to 86% over existing metrics,including two state-of-the-art exploit classifiers, while itsprecision substantially improves over time. We also highlightthe practical utility of EE for predicting imminent exploits andprioritizing critical vulnerabilities.

We develop EE into an online platform which is publiclyavailable at https://exploitability.app/.

1 Introduction

Weaponized exploits have a disproportionate impact on se-curity, as highlighted in 2017 by the WannaCry [67] andNotPetya [15] worms that infected millions of computersworldwide. Their notorious success was in part due to the useof weaponized exploits. The cyber-insurance industry regardssuch contagious malware, which propagates automaticallyby exploiting software vulnerabilities, as the leading risk forincurring large losses from cyber attacks [58]. At the sametime, the rising bar for developing weaponized exploits [66]pushed black-hat developers to focus on exploiting only 5%of the known vulnerabilities [32]. To prioritize mitigation

efforts in the industry, to make optimal decisions in the gov-ernment’s Vulnerabilities Equities Process [29], and to gain adeeper understanding of the research opportunities to preventexploitation, we must evaluate each vulnerability’s ease ofexploitation.

Despite significant advances in defenses [66], exploitabilityassessments remain elusive because we do not know whichvulnerability features predict exploit development. For exam-ple, expert recommendations for prioritizing patches [52, 53]initially omitted CVE-2017-0144, the vulnerability later ex-ploited by WannaCry and NotPetya. While one can proveexploitability by developing an exploit, it is challenging toestablish non-exploitability, as this requires reasoning aboutstate machines with an unknown state space and emergentinstruction semantics [18]. This results in a class bias ofexploitability assessments, as we cannot be certain that a “notexploitable” label is accurate.

We address these two challenges through a metric calledExpected Exploitability (EE). Instead of deterministically la-beling a vulnerability as “exploitable” or “not exploitable”,our metric continuously estimates over time the likelihoodthat a functional exploit will be developed, based on historicalpatterns for similar vulnerabilities. Functional exploits gobeyond proof-of-concepts (POCs) to achieve the full secu-rity impact prescribed by the vulnerability. While functionalexploits are readily available for real-world attacks, we aimto predict their development and not their use in the wild,which depends on many other factors besides exploitabil-ity [32, 60, 77].

Key to our solution is a time-varying view of exploitability,a departure from the existing vulnerability scoring systemssuch as CVSS [47], which are not designed to take into ac-count new information (e.g., new exploitation techniques,leaks of weaponized exploits) that becomes available after thescores are initially computed [20]. By systematically com-paring a range of prior and novel features, we observe thatartifacts published after vulnerability disclosure can be goodpredictors for the development of exploits, but their timelinessand predictive utility varies. This highlights limitations ofprior features and a qualitative distinction between predict-ing functional exploits and related tasks. For example, priorwork uses the existence of public PoCs as an exploit predic-

arX

iv:2

102.

0786

9v2

[cs

.CR

] 3

Feb

202

2

tor [32, 33, 68]. However, PoCs are designed to trigger thevulnerability by crashing or hanging the target application andoften are not directly weaponizable; we observe that this leadsto many false positives for predicting functional exploits. Incontrast, we discover that certain PoC characteristics, such asthe code complexity, are good predictors, because triggeringa vulnerability is a necessary step for every exploit, makingthese features causally connected to the difficulty of creatingfunctional exploits. We design techniques to extract featuresat scale, from PoC code written in 11 programming languages,which complement and improve upon the precision of previ-ously proposed feature categories. We then learn EE from theuseful features using data-driven methods, which have beensuccessful in predicting other incidents, e.g., vulnerabilitiesthat are exploited in the wild [32, 60, 77], data breaches [37]or website compromises [64].

However, learning to predict exploitability could be de-railed by a biased ground truth. Although prior work hadacknowledged this challenge for over a decade [9, 60], noattempts were made to address it. This problem, known in themachine-learning literature as label noise, can significantlydegrade the performance of a classifier. The time-varyingview of exploitability allows us to uncover the root causes oflabel noise: exploits could be published only after the datacollection period ended, which in practice translates to wrongnegative labels. This insight allows us to characterize thenoise-generating process for exploit prediction and propose atechnique to mitigate the impact of noise when learning EE.

In our experiments on 103,137 vulnerabilities, EE signif-icantly outperforms static exploitability metrics and priorstate-of-the art exploit predictors, increasing the precisionfrom 49% to 86% one month after disclosure. Using ourlabel noise mitigation technique, the classifier performanceis minimally affected even if evidence about 20% of exploitsis missing. Furthermore, by introducing a metric to capturevulnerability prioritization efforts, we show that EE requiresonly 10 days from disclosure to approach its peak perfor-mance. We show EE has practical utility, by providing timelypredictions for imminent exploits, even when public PoCs areunavailable. Moreover, when employed on scoring 15 criti-cal vulnerabilities, EE places them above 96% of non-criticalones, compared to only 49% for existing metrics.

In summary, our contributions are as follows:• We propose a time-varying view of exploitability based

on which we design Expected Exploitability (EE), a met-ric to learn and continuously estimate the likelihood offunctional exploits over time.

• We characterize the noise-generating process systemati-cally affecting exploit prediction, and propose a domain-specific technique to learn EE in the presence of labelnoise.

• We explore the timeliness and predictive utility of vari-ous artifacts, proposing new and complementary featuresfrom PoCs, and developing scalable feature extractors

for them.• We perform three case studies to investigate the practical

utility of EE, showing that it can qualitatively improveprioritization strategies based on exploitability.

EE is available as an online platform athttps://exploitability.app/ and described in the ArtifactAppendix B.

2 Problem Overview

We define exploitability as the likelihood that a functionalexploit, which fully achieves the mandated security impact,will be developed for a vulnerability. Exploitability reflectsthe technical difficulty of exploit development, and it does notcapture the feasibility of lunching exploits against targets inthe wild [32,60,77], which is influenced by additional factors(e.g., patching delays, network defenses, attacker choices).

While an exploit represents conclusive proof that a vul-nerability is exploitable if it can be generated, proving non-exploitability is significantly more challenging [18]. Instead,mitigation efforts are often guided by vulnerability scoringsystems, which aim to capture exploitation difficulty, such as:

1. NVD CVSS [47], a mature scoring system with its Ex-ploitability metrics intended to reflect the ease and tech-nical means by which the vulnerability can be exploited.The score encodes various vulnerability characteristics,such as the required access control, complexity of theattack vector and privilege levels, into a numeric valuebetween 0 and 4 (0 and 10 for CVSSv2), with 4 reflectingthe highest exploitability.

2. Microsoft Exploitability Index [21], a vendor-specificscore assigned by experts using one of four values tocommunicate to Microsoft customers the likelihood of avulnerability being exploited [20].

3. RedHat Severity [56], similarly encoding the difficultyof exploiting the vulnerability by complementing CVSSwith expert assessments based on vulnerability charac-teristics specific to the RedHat products.

The estimates provided by these metrics are often inaccu-rate, as highlighted by prior work [2, 3, 20, 57, 61] and by ouranalysis in Section 5. For example, CVE-2018-8174, an ex-ploitable Internet Explorer vulnerability, received a CVSS ex-ploitability score of 1.6, placing it below 91% of vulnerabilityscores. Similarly, CVE-2018-8440, an exploited vulnerabilityaffecting Windows 7 through 10 was assigned score of 1.8.

To understand why these metrics are poor at reflectingexploitability, we highlight the typical timeline of a vulner-ability in Figure 1. The exploitability metrics depend on atechnical analysis which is performed before the vulnerabilityis disclosed publicly, and which considers the vulnerabilitystatically and in isolation.

However, we observe that public disclosure is followedby the publication of various vulnerability artifacts such as

2

20 40 60 80 1100 2700

ExploitabilityMetrics

Write-ups

PoC Info

PoC Code

Exploits

Discovery Disclosure Publication Delay (days)

Technical analysis

Exploit development

Wit

PC If

PC C

d

VD A

l

Tt

Tweets

////

// ////

Figure 1: Vulnerability timeline highlighting publication de-lay for different artifacts and the CVSS Exploitability metric.The box plot delimits the 25th, 50th and 75th percentiles, andthe whiskers mark 1.5 times the interquartile range.

write-ups and PoCs containing code and additional technicalinformation about the vulnerability, and social media discus-sions around them. These artifacts often provide meaningfulinformation about the likelihood of exploits. For CVE-2018-8174 it was reported that the publication of technical write-upswas a direct cause for exploit development in exploit kits [8],while a PoC for CVE-2018-8440 has been determined totrigger exploitation in the wild within two days [80]. The ex-amples highlight that existing metrics fail to take into accountuseful exploit information available only after disclosure andthey do not update over time.

Figure 1 plots the publication delay distribution for differ-ent artifacts released after disclosure, according to our dataanalysis described in Section 5. Data shows not only that theseartifacts become available soon after disclosure,1 providingopportunities for timely assessments, but also that static ex-ploitability metrics, such as CVSS, are often not available atthe time of disclosure.Expected Exploitability. The problems mentioned abovesuggest that the evolution of exploitability over time can bedescribed by a stochastic process. At a given point in time,exploitability is a random variable E encoding the probabilityof observing an exploit. E assigns a probability 0.0 to thesubset of vulnerabilities that are provably unexploitable, and1.0 to vulnerabilities with known exploits. Nevertheless, thetrue distribution E generating E is not available at scale, andinstead we need to rely on a noisy version Etrain, as we willdiscuss in Section 3. This implies that in practice E has tobe approximated from the available data, by computing thelikelihood of exploits, which estimates the expected value ofexploitability. We call this measure Expected Exploitability(EE). EE can be learned from historical data using supervisedmachine learning and can be used to assess the likelihood ofexploits for new vulnerabilities before functional exploits are

1Interestingly, some social media posts appear before disclosure. These"leaks from coordinated disclosure" were investigated by prior work [60].

developed or discovered.

3 Challenges

We recognize three challenges in utilizing supervised tech-niques for learning, evaluating and using EE.Extracting features from PoCs. Prior work investigated theexistence of PoCs as predictors for exploits, repeatedly show-ing that they lead to a poor precision [2, 33, 68]. However,PoCs are designed to trigger the vulnerability, a step alsorequired in a functional exploit. As a result, the structureand complexity of the PoC code can reflect exploitation dif-ficulty directly: a complex PoC implies that the functionalexploit will also be complex. To fully leverage the predictivepower of PoCs, we need to capture these characteristics. Wenote that while public PoCs have a lower coverage comparedto other artifact types, they are broadly available privatelybecause they are often mandated when vulnerabilities arereported [27].

Extracting features using NLP techniques from prior ex-ploit prediction work [9,32,60] is not sufficient, because codesemantics differs from that of natural language. Moreover,PoCs are written in different programming languages andare often malformed programs [2, 45], combining code withfree-form text, which limits the applicability of existing pro-gram analysis techniques. PoC feature extraction thereforerequires text and code separation, and robust techniques toobtain useful code representations.Understanding and mitigating label noise. Prior workfound that the labels available for training have biases [9, 60],but to our knowledge no prior attempts were made to linkthis issue to the problem of label noise. The literature dis-tinguishes two models of non-random label noise, accordingto the generating distribution: class-dependent and feature-dependent [25]. The former assumes a uniform label flippingprobability among all instances of a class, while the latterassumes that noise probability also depends on individualfeatures of instances. If Etrain is affected by label noise, thetest time performance of the classifier could suffer.

By viewing exploitability as time-varying, it becomes im-mediately clear that exploit evidence datasets are prone toclass-dependent noise. This is because exploits might notyet be developed or be kept secret. Therefore, a subset ofvulnerabilities believed not to be exploited are in fact wronglylabeled at any given point in time.

In addition, prior work noticed that individual vendors pro-viding exploit evidence have uneven coverage of the vulnera-bility space (e.g., an exploit dataset from Symantec would notcontain Linux exploits because the platform is not coveredby the vendor) [60], suggesting that noise probability mightbe dependent on certain features. The problem of feature-dependent noise is much less studied [50], and discoveringthe characteristics of such noise on real-world applications is

3

considered an open problem in machine learning [25].Exploit prediction therefore requires an empirical under-

standing of both the type and effects of label noise, as well asthe design of learning techniques to address it.Evaluating the impact of time-varying exploitability.While some post-disclosure artifacts are likely to improveclassification, publication delay might affect their utility astimely predictions. Our EE evaluation therefore needs to usemetrics which highlight potential trade-offs between timeli-ness and performance. Moreover, the evaluation needs to testwhether our classifier can capitalize on artifacts with highpredictive power available before functional exploits are dis-covered, and whether EE can capture the imminence of certainexploits. Finally, we need to demonstrate the practical util-ity of EE over existing static metrics, in real-world scenariosinvolving vulnerability prioritization.Goals and Non-Goals. Our goal is to estimate EE for a broadrange of vulnerabilities, by addressing the challenges listedabove. Moreover, we aim to provide estimates that are bothaccurate and robust: they should predict the development offunctional exploits better than the existing scoring systemsand despite inaccuracies in the ground truth. The closestwork to our goal is DarkEmbed [68], which uses naturallanguage models trained on underground forum discussionsto predict the availability of exploits. In contrast, we aim topredict functional exploits from public information, a moredifficult task as we lack direct evidence of black-hat exploitdevelopment.

We do not aim to generate functional exploits automati-cally. We also do not aim to analyze the code of existingexploits in order to assess how close they are to becomingfunctional. Instead, we aim to quantify the exploitability ofknown vulnerabilities objectively, by predicting whether func-tional exploits will be developed for them.While we aim tomake our exploit predictor robust to systematic label noise,we do not attempt to improve the model’s robustness to ad-versarial examples [60], as techniques for achieving this havebeen widely studied elsewhere [12]. Finally, we do not aim topredict which vulnerabilities are likely to be exploited in thewild, in real-world attacks, because this likelihood is influ-enced by additional factors (e.g., attacker choices, patchingdelays) that are out of scope for this paper.

4 Data Collection

In this section we describe the methods used to collect the dataused in our paper, as well as the techniques for discoveringvarious timestamps in the lifecycle of vulnerabilities.

4.1 Gathering Technical InformationWe use the CVEIDs to identify vulnerabilities, because it isone of the most prevalent and cross-referenced public vulner-

ability identification systems. Our collection contains vulner-abilities published between January 1999 and March 2020.Public Vulnerability Information. We begin by collectinginformation about the vulnerabilities targeted by the PoCsfrom the National Vulnerability Database (NVD) [48]. NVDadds vulnerability information gathered by analysts, includingtextual descriptions of the issue, product and vulnerabilitytype information, as well as the CVSS score. Nevertheless,NVD only contains high-level descriptions. In order to build amore complete coverage of the technical information availablefor each vulnerability, we search for external references inseveral public sources. We use the Bugtraq [10] and IBM X-Force Exchange [30], vulnerability databases which provideadditional textual descriptions for the vulnerabilities. We alsouse Vulners [74], a database that collects in real time textualinformation from vendor advisories, security bulletins, third-party bug trackers and security databases. We filter out thereports that mention more than one CVEID, as it would bechallenging to determine which particular one is discussed.In total, our collection contains 278,297 documents from76 sources, referencing 102,936 vulnerabilities. We referto these documents as write-ups, which, together with theNVD textual information and vulnerability details, provide abroader picture of the technical information publicly availablefor vulnerabilities.Proof of Concepts (PoCs). We collect a dataset of publicPoCs by scraping ExploitDB [22], Bugtraq [10] and Vul-ners [74], three popular vulnerability databases that containexploits aggregated from multiple sources. Because there issubstantial overlap across these sources, but the formatting ofthe PoCs might differ slightly, we remove duplicates using acontent hash that is invariant to such minor whitespace differ-ences. We preserve only the 48,709 PoCs that are linked toCVEIDs, which correspond to 21,849 distinct vulnerabilities.Social Media Discussions. We also collect social media dis-cussions about vulnerabilities from Twitter, by gatheringtweets mentioning CVE-IDs between January 2014 and De-cember 2019.We collected 1.4 million tweets for 52,551 vul-nerabilities by continuously monitored the Twitter FilteredStream API [72], using the approach from our prior work [60].While the Twitter API does not sample returned tweets, shortoffline periods for our platform caused some posts to be lost.By a conservative estimate using the lost tweets which werelater retweeted, our collection contains over 98% of all publictweets about these vulnerabilities.Exploitation Evidence Ground Truth. Because we are notaware of any comprehensive dataset of evidence about devel-oped exploits, we aggregate evidence from multiple publicsources.

We begin from the Temporal CVSS score, which tracksthe status of exploits and the confidence in these reports. TheExploit Code Maturity component has four possible values:"Unproven", "Proof-of-Concept", "Functional" and "High".

4

The first two values indicate that the exploit is not practi-cal or not functional, while the last two values indicate theexistence of autonomous or functional exploits that work inmost situations. Because the Temporal score is not updatedin NVD, we collect it from two reputable sources: IBM X-Force Exchange [30] threat sharing platform and the TenableNessus [70] vulnerability scanner. Both scores are used asinputs to proprietary severity assessment solutions: the formeris used by IBM in one of their cloud offerings [75], whilethe latter is used by Tenable as input to commercial vulnera-bility prioritization solutions [71]. We use the labels "Func-tional" and "High" as evidence of exploitation, as definedby the official CVSS Specification [47], obtaining 28,009exploited vulnerabilities. We further collect evidence of 2,547exploited vulnerabilities available in three commercial ex-ploitation tools: Metasploit [55], Canvas [31] and D2 [16].We also scrape the Bugtraq [10] exploit pages, and createNLP rules to extract evidence for 1,569 functional exploits.Examples of indicative phrases are: "A commercial exploitis available.", "A functional exploit was demonstrated by re-searchers.".

We also collect exploitation evidence that results from ex-ploitation in the wild. Starting from a dataset collected fromSymantec in our prior work [60], we update it by scrapingSymantec’s Attack Signatures [4] and Threat Explorer [65].We then aggregate labels extracted using NLP rules (matchinge.g., "... was seen in the wild.") from scrapes of Bugtraq [10],Tenable [69], Skybox [63] and AlienVault OTX [49]. Inaddition, we use the Contagio dump [43] which contains acurated list of exploits used by exploit kits. These sourceswere reported by prior work as reliable for information aboutexploits in the wild [32,46,60]. Overall, 4,084 vulnerabilitiesare marked as exploited in the wild.

While exact development time for most exploits is not avail-able, we drop evidence if we cannot confirm they were pub-lished within one year after vulnerability disclosure, simulat-ing a historical setting. Our ground truth, consisting of 32,093vulnerabilities known to have functional exploits, thereforereflects a lower bound for the number of exploits available,which translates to class-dependent label noise in classifica-tion, issue that we evaluate in Section 7.

4.2 Estimating Lifecycle Timestamps

Vulnerabilities are often published in NVD at a later date thantheir public disclosure [7, 36]. We estimate the public disclo-sure dates for the vulnerabilities in our dataset by selecting theminimum date among all write-ups in our collection and thepublication date in NVD, in line with prior research [36, 62].This represents the earliest date when expected exploitabilitycan be evaluated. We validate our estimates for the disclo-sure dates by comparing them to two independent prior esti-mates [36, 62], on the 67% of vulnerabilities which are alsofound in the other datasets. We find that the median date dif-

ference between the two estimates is 0 days, and our estimatesare an average of 8.5 days earlier than prior assessments. Sim-ilarly, we estimate the time when PoCs are published as theminimum date among all sources that shared them, and weconfirm the accuracy of these dates by verifying the commithistory in exploit databases that use version control.

To assess whether EE can provide timely warnings, weneed the dates for the emergence of functional exploits andattacks in the wild. Because the sources of exploit evidencedo not share the dates when exploits were developed, weestimate these dates from ancillary data. For the exploit toolk-its, we collect the earliest date when exploits are reportedin the Metasploit and Canvas platforms. For exploits in thewild, we use the dates of first recorded attacks, from priorwork [60]. Across all exploited vulnerabilities, we also crawlVirusTotal [73], a popular threat sharing platform, for thetimestamps when exploit files were first submitted. Finally,we estimate exploit availability as the earliest date amongthe different sources, excluding vulnerabilities with zero-dayexploits. Overall, we discovered this date for 10% (3,119)of the exploits.These estimates could result in label noise,because exploits might sometimes be available earlier, e.g.,PoCs that are easy to weaponize. In Section 7.3 we measurethe impact of such label noise on the EE performance.

4.3 DatasetsWe create three datasets that we use throughout the paperto evaluate EE. DS1 contains all 103,137 vulnerabilities inour collection that have at least one artifact published withinone year after disclosure. We use this to evaluate the time-liness of various artifacts, compare the performance of EEwith existing baselines, and measure the predictive powerof different categories of features. The second dataset, DS2,contains 21,849 vulnerabilities that have artifacts across alldifferent categories within one year. This is used to comparethe predictive power of various feature categories, observetheir improved utility over time, and to test their robustness tolabel noise. The third one, DS3 contains 924 out of the 3,119vulnerabilities for which we estimated the exploit emergencedate, and which are disclosed during our classifier deploymentperiod described in Section 6.3. These are used to evaluatethe ability of EE to distinguish imminent exploit.

5 Empirical Observations

We start our analysis with three empirical observations onDS1, which guide the design of our system for computing EE.Existing scores are poor predictors. First, we investigatethe effectiveness of three vulnerability scoring systems, de-scribed in Section 2, for predicting exploitability. Becausethese scores are widely used, we will utilize them as base-lines for our prediction performance; our goal for EE is toimprove this performance substantially. As the three scores

5

0 1 2 3 4

CVSSv3 Exploitability

0

0.2

0.4

0.6

0.8

1

Metr

ic V

alu

e

P

R

P

R

3 2 1

MS Exploitability Index

0

0.2

0.4

0.6

0.8

1

P

R

3 2 1 0

RedHat Severity

0

0.2

0.4

0.6

0.8

1

Figure 2: Performance of existing severity scores at capturingexploitability. We report both precision (P) and recall (R).The numerical score values are ordered by increasing severity.

0 2 4 6 8 10

CVSSv2 Exploitability

0

0.2

0.4

0.6

0.8

1

Metr

ic V

alu

e

Precision (P) E

Recall (R) E

Figure 3: Performance of CVSSv2 at capturing exploitability.We report both precision (P) and recall (R).

do not change over time, we utilize a threshold-based deci-sion rule, which predicts that all vulnerabilities with scoresgreater or equal than the threshold are exploitable. By varyingthe threshold across the entire score range, and using all thevulnerabilities in our dataset, we evaluate their precision (P):the fraction of predicted vulnerabilities that have functionalexploits within one year from disclosure, and recall (R): thefraction of exploited vulnerabilities that are identified withinone year.

Figure 2 reports these performance metrics. It is possibleto obtain R = 1 by marking all vulnerabilities as exploitable,but this affects P because many predictions would be falsepositives. For this reason, for all the scores, R decreases as weraise the severity threshold for prediction. However, obtaininga high P is more difficult. For CVSSv3 Exploitability, P doesnot exceed 0.19, regardless of the detection threshold, andsome vulnerabilities do not have scores assigned to them.CVSSv2 also exhibits a very poor precision, as illustrated inFigure 3.

When evaluating the Microsoft Exploitability Index on the1,100 vulnerabilities for Microsoft products in our datasetdisclosed since the score inception in 2008, we observe thatthe maximum precision achievable is 0.45. The recall isalso lower because the score is only computed on a subset ofvulnerabilities [20].

On the 3,030 vulnerabilities affecting RedHat products, weobserve a similar trend for the proprietary severity metric,where precision does not exceed 0.45.

These results suggest that the three existing scores predict

NVD Publication

CVSS Publication

Write-Ups

Tweets

PoC Info

PoC Code

0 0.5 1 1.5 2 2.5

Date First Artifact - Date Disclosure [log10(# days)]

0

0.2

0.4

0.6

0.8

1

CD

F

(a)

NVD Publication

CVSS Publication

Write-Ups

Tweets

PoC Info

PoC Code

-4 -3 -2 -1 0 1 2

Date First Artifact - Date Exploit Available [log10(# days)]

0

0.2

0.4

0.6

0.8

1

CD

F

(b)

Figure 4: (a) Number of days after disclosure when vulnera-bility artifacts are first published. (b) Difference between theavailability of exploits and availability of other artifacts. Theday differences are in logarithmic scale.

exploitability with > 50% false positives. This is compoundedby the facts that (1) some scores are not computed for all vul-nerabilities, owing to the manual effort required, which intro-duces false negative predictions; (2) the scores do not change,even if new information becomes available; and (3) not all thescores are available at the time of disclosure, meaning thatthe recall observed operationally soon after disclosure will belower, as highlighted in the next section.Artifacts provide early prediction opportunities. To assessthe opportunities for early prediction, we look at the publica-tion timing for certain artifacts from the vulnerability lifecycle.In Figure 4(a), we plot, across all vulnerabilities, the earli-est point in time after disclosure when the first write-ups arepublished, they are added to NVD, their CVSS and technicalanalysis are published in NVD, their first PoCs are releasedand when they are first mentioned on Twitter. The publica-tion delay distribution for all collected artifacts is available inFigure 1.

Write-ups are the most widely available ones at the timeof disclosure, suggesting that vendors prefer to disclose vul-nerabilities through either advisories or third-party databases.However, many PoCs are also published early: 71% of vul-nerabilities have a PoC on the day of disclosure. In contrast,only 26% of vulnerabilities in our dataset are added to NVDon the day of disclosure, and surprisingly, only 9% of theCVSS scores are published at disclosure. This result suggeststhat timely exploitability assessments require looking beyondNVD, using additional sources of technical vulnerability in-formation, such as the write-ups and PoCs. This observationdrives our feature engineering from Section 6.1.

Figure 4(b) highlights the day difference between the dateswhen the exploits become available and the availability ofthe artifacts from public vulnerability disclosure. For morethan 92% of vulnerabilities, write-ups are available beforethe exploits become available. We also find that the 62%of PoCs are available before this date, while 64% of CVSS

6

assessments are added to NVD before. Overall, the avail-ability of exploits is highly correlated with the emergenceof other artifacts, indicating an opportunity to infer the exis-tence of functional exploits as soon as, or before, they becomeavailable.Exploit prediction is subject to feature-dependent labelnoise. Good predictions also require a judicious solution tothe label noise challenge discussed in Section 3. The time-varying view of exploitability revealed that our problem issubject to class-dependent noise. However, because we ag-gregate evidence about exploits from multiple sources, theirindividual biases could also affect our ground truth. To testfor such individual biases, we investigate the dependence be-tween all sources of exploit evidence and various vulnerabilitycharacteristics. For each source and feature pair, we perform aChi-squared test for independence, aiming to observe whetherwe can reject the null hypothesis H0 that the presence of anexploit within the source is independent of the presence of thefeature for the vulnerabilities. Table 1 lists the results for all12 sources of ground truth, across the most prevalent vulnera-bility types and affected products in our dataset. We utilizethe Bonferroni correction for multiple tests [19] and a 0.01significance level. We observe that the null hypothesis canbe rejected for at least 4 features for each source, indicatingthat all the sources for ground truth include biases causedby individual vulnerability features. These biases could bereflected in the aggregate ground truth, suggesting that exploitprediction is subject to class- and feature-dependent labelnoise.

6 Computing Expected Exploitability

In this section we describe the system for computing EE,starting from the design and implementation of our featureextractor, and presenting the classifier choice.

6.1 Feature EngineeringEE uses features extracted from all vulnerability and PoCartifacts in our datasets, which are summarized in Table 2.Novel: PoC Code. Intuitively, one of the leading indicatorsfor the complexity of functional exploits is the complexity ofPoCs. This is because if triggering the vulnerability requiresa complex PoC, an exploit would also have to be complex.Conversely, complex PoCs could already implement func-tionality beneficial towards the development of functionalexploits. We use this intuition to extract features that reflectthe complexity of PoC code, by means of intermediate rep-resentations that can capture it. We transform the code intoAbstract Syntax Trees (ASTs), a low-overhead representationwhich encodes structural characteristics of the code. Fromthe ASTs we extract complexity features such as statisticsof the node types, structural features of the tree, as well as

statistics of control statements within the program and therelationship between them. Additionally, we extract featuresfor the function calls within the PoCs towards external libraryfunctions, which in some cases may be the means throughwhich the exploit interacts with the vulnerability and therebyreflect the relationship between the PoC and its vulnerability.Therefore, the library functions themselves, as well as thepatterns in calls to these functions, can reveal informationabout the complexity of the vulnerability, which might in turnexpress the difficulty of creating a functional exploit. Wealso extract the cyclomatic complexity from the AST [34],a software engineering metric which encodes the number ofindependent code paths in the program. Finally, we encodefeatures of the PoC programming language, in the form ofstatistics over the file size and the distribution of languagereserved keywords.

We also observe that the lexical characteristics of the PoCcode provide insights into the complexity of the PoC. Forexample, a variable named shellcode in a PoC might sug-gest that the exploit is in an advanced stage of development.In order to capture such characteristics, we extract the codetokens from the entire program, capturing literals, identifiersand reserved keywords, in a set of binary unigram features.Such specific information allows us to capture the stylisticcharacteristics of the exploit, the names of the library callsused, as well as more latent indicators, such as artifacts in-dicating exploit authorship [11], which might provide utilitytowards predicting exploitability. Before training the classi-fier, we filter out lexicon features that appear in less than 10training-time PoCs, which helps prevent overfitting.Novel: PoC Info. Because a large fraction of PoCs containonly textual descriptors for triggering the vulnerabilities with-out actual code, we also extract features that aim to encodethe technical information conveyed by the authors in the non-code PoCs, as well as comments in code PoCs. We encodethese features as binary unigrams. Unigrams provide a clearbaseline for the performance achievable using NLP. Nev-ertheless, in Section 7.2 we investigate the performanceof EE with embeddings, showing that there are additionalchallenges in designing semantic NLP features for exploitprediction, which we leave for future work.Prior Work: Vulnerability Info and Write-ups. To capturethe technical information shared through natural languagein artifacts, we extract unigram features from all the write-ups discussing each vulnerability and the NVD descriptionsof the vulnerability. Finally, we extract the structured datawithin NVD that encodes vulnerability characteristics: themost prevalent list of products affected by the vulnerability,the vulnerability types (CWEID [14]), and all the CVSS BaseScore sub-components, using one-hot encoding.Prior Work: In-the-Wild Predictors. To compare the effec-tiveness of various feature sets, we also extract 2 categoriesproposed in prior predictors of exploitation in the wild. The

7

Functional Exploits Exploits in the Wild

Tenable X-Force Metasploit Canvas Bugtraq D2 Symantec Contagio Alienvault Bugtraq Skybox Tenable

CWE-79 X X X X X X X X X X X X(0.006)

CWE-94 X X X X X X X X X X X X (1.000)

CWE-89 X X X X X X (1.000) X X X X X X (0.284)

CWE-119 X X X X X X X X X X X X(0.001)

CWE-20 X X X X X X X X (1.000) X X(0.002) X X (1.000)

CWE-22 X X (0.211) X X (1.000) X (1.000) X X (1.000) X (1.000) X (0.852) X (1.000) X (1.000) X (1.000)

Windows X X X X X X (0.012) X X X X X X

Linux X X X X X X (1.000) X X X X X X

Table 1: Evidence of feature-dependent label noise. A Xindicates that we can reject the null hypothesis H0 that evidence ofexploits within a source is independent of the feature. Cells with no p-value are < 0.001.

11

Code / Text Separation

PoCs

NLPProgram Analysis

PoC Code Features

NVDWrite-ups

Write-Up Features

NVD Info Features

PoC Info Features

Figure 5: Diagram of the EE feature extraction system.

Exploit Prediction Scoring System (EPSS) [33] proposes 53features manually selected by experts as good indicators forexploitation in the wild. This set of handcrafted features con-tains tags reflecting vulnerability types, products and vendors,as well as binary indicators of whether PoC or weaponizedexploit code has been published for a vulnerability. Second,from our collection of tweets, we extract social media featuresintroduced in prior work [60], which reflect the textual de-scription of the discourse on Twitter, as well as characteristicsof the user base and tweeting volume for each vulnerability.Unlike the original work, we do not perform feature selectionon the unigram features from tweets because we want to di-rectly compare the utility of Twitter discussions to these fromother artifacts. None of the two categories will be used in thefinal EE model because of their limited predictive utility.

6.2 Feature Extraction System

Below we describe the components of our feature extractionsystem, illustrated in Figure 5, and discuss how we addressthe challenges identified in Section 3.Code/Text Separation. Only 64% of the PoCs in our datasetcontain any file extension that would allow us to identify theprogramming language. Moreover, 5% of them have conflict-ing information from different sources, and we observe thatmany PoCs are first posted online as freeform text withoutexplicit language information. Therefore, a central challenge

is to accurately identify their programming languages andwhether they contain any code. We use GitHub Linguist [26],to extract the most likely programming languages used ineach PoC. Linguist combines heuristics with a Bayesian clas-sifier to identify the most prevalent language within a file.Nevertheless, the model obtains an accuracy of 0.2 on classi-fying the PoCs, due to the prevalence of natural language textin PoCs. After modifying the heuristics and retraining theclassifier on 42,195 PoCs from ExploitDB that contain fileextensions, we boost the accuracy to 0.95. The main cause oferrors is text files with code file extensions, yet these errorshave limited impact because of the NLP features extractedfrom files.

Table 3 lists the number of PoCs in our dataset for eachidentified language label (the None label represents the caseswhich our classifier could not identify any language, includingless prevalent programming languages not in our label set).We observe that 58% of PoCs are identified as text, whilethe remaining ones are written in a variety of programminglanguages. Based on this separation, we develop regularexpressions to extract the comments from all code files. Afterseparating the comments, we process them along with the textfiles using NLP, to obtain PoC Info features, while the PoCCode features are obtained using NLP and program analysis.Code Features. Performing program analysis on the PoCsposes a challenge because many of them do not have a validsyntax or have missing dependencies that hinders compilationor interpretation [2, 45]. We are not aware of any unifiedand robust solution to simultaneously obtain ASTs from codewritten in different languages. We address this challenge byemploying heuristics to correct malformed PoCs and parsingthem into intermediate representations using techniques thatprovide robustness to errors.

Based on Table 3, we observe that some languages arelikely to have a more significant impact on the prediction per-formance, based on prevalence and frequency of functionalexploits among the targeted vulnerabilities. Given this obser-vation, we focus our implementation on Ruby, C/C++, Perl

8

Type Description #

PoC Code (Novel)Length # characters, loc, sloc 33

Language Programming language label 11

Keywords count Count for reserved keywords 820

Tokens Unigrams from code 92,485

#_nodes # nodes in the AST tree 4

#_internal_nodes # of internal AST tree nodes 4

#_leaf_nodes # of leaves of AST tree 4

#_identifiers # of distinct identifiers 4

#_ext_fun # of external functions called 4

#_ext_fun_calls # of calls to external functions 4

#_udf # user-defined functions 4

#_udf_calls # calls to user-defined functions 4

#_operators # operators used 4

cyclomatic compl cyclomatic complexity 4

nodes_count_* # of AST nodes for each node type 316

ctrl_nodes_count_* # of AST nodes for each control statement type 29

literal_types_count_* # of AST nodes for each literal type 6

nodes_depth_* Stats depth in tree for each AST node type 916

branching_factor Stats # of children across AST 12

branching_factor_ctrl Stats # of children within the Control AST 12

nodes_depth_ctrl_* Stats depth in tree for each Control AST node type 116

operator_count_* Usage count for each operator 135

#_params_udf Stats # of parameters for user-defined functions 12

PoC Info (Novel)PoC unigrams PoCs text and comments 289,755

Write-ups (Prior Work)Write-up unigrams Write-ups text 488,490

Vulnerability Info (Prior Work)NVD unigrams NVD descriptions 103,793

CVSS CVSSv2 & CVSSv3 components 40

CWE Weakness type 154

CPE Name of affected product 10

In-the-Wild Predictors (Prior Work)EPSS Handcrafted 53

Social Media Twitter content and statistics 898,795

Table 2: Description of features used. Unigram features arecounted before frequency-based pruning.

and Python. Note that this choice does not impact the ex-traction of lexical features from code PoCs written in otherlanguages.

For C/C++ we use the Joern fuzzy parser for program anal-ysis, previously proposed for bug discovery [79]. The toolprovides robustness to parsing errors through the use of islandgrammars and allows us to successfully parse 98% of thefiles.

On Perl, by modifying the existing Compiler::Parser [42]tool to improve its robustness, and employing heuristics tocorrect malformed PoC files, we improve the parsing successrate from 37% to 83%.

For Python, we implement a feature extractor based onthe ast parsing library [54], achieving a success rate of 67%.We observe that this lower parsing success rate is due tothe reliance of the language on strict indentation, which is

Language # PoCs # CVEs (% exploited)

Text 27743 14325 (47%)

Ruby 4848 1988 (92%)

C 4512 2034 (30%)

Perl 3110 1827 (54%)

Python 2590 1476 (49%)

JavaScript 1806 1056 (59%)

PHP 1040 708 (55%)

HTML 1031 686 (56%)

Shell 619 304 (29%)

VisualBasic 397 215 (41%)

None 367 325 (43%)

C++ 314 196 (34%)

Java 119 59 (32%)

Table 3: Breakdown of the PoCs in our dataset according toprogramming language.

often distorted or completely lost when code gets distributedthrough Webpages.

Ruby provides an interesting case study because, despitebeing the most prevalent language among PoCs, it is alsothe most indicative of exploitation. We observe that thisis because our dataset contains functional exploits from theMetasploit framework, which are written in Ruby. We extractAST features for the language using the Ripper library [59].Our implementation is able to successfully parse 96% of thefiles.

Overall, we successfully parse 13,704 PoCs associated with78% of the CVEs that have PoCs with code. Each vulnerabil-ity aggregates only the code complexity features of the mostcomplex PoC (in source lines of code) across each of the fourlanguages, while the remaining code features are collectedfrom all PoCs available.Unigram Features. We extract the textual features usinga standard NLP pipeline which involves tokenizing thetext from the PoCs or vulnerability reports, removing non-alphanumeric characters, filtering out English stopwords andrepresenting them as unigrams. For each vulnerability, thePoC unigrams are aggregated across all PoCs, and separatelyacross all write-ups collected within the observation period.When training a classifier, we discard unigrams which occurless than 100 times across the training set, because they areunlikely to generalize over time and we did not observe anynoticeable performance boost when including them.

6.3 Exploit Predictor DesignThe predictor concatenates all the extracted features, and usesthe ground truth about exploit evidence, to train a classifier

9

which outputs the EE score. The classifier uses a feedforwardneural network having 2 hidden layers of size 500 and 100respectively, with ReLU activation functions. This choice wasdictated by two main characteristics of our domain: featuredimensionality and concept drift. First, as we have manypotentially useful features, but with limited coverage, linearmodels, such as SVM, which tend to emphasize few importantfeatures [40], would perform worse. Second, deep learningmodels are believed to be more robust to concept drift and theshifting utility of features [51], which is a prevalent issue inthe exploit prediction task [60]. The architecture was chosenempirically by measuring performance for various settings.Classifier training. To address the second challenge iden-tified in Section 3, we incorporate noise robustness in oursystem by exploring several loss functions for the classifier.Our design choices are driven by two main requirements: (i)providing robustness to both class- and feature- dependentnoise, and (ii) providing minimal performance degradationwhen noise specification is not available.

BCE: The binary cross-entropy is the standard, noise-agnostic loss for training binary classifiers. For a set of Nexamples xi with labels yi ∈ {0,1}, the loss is computed as:

LBCE =− 1N

N

∑i=1

[yilog(pθ(xi))+(1− yi)log(1− pθ(xi))]

where pθ(xi) corresponds to the output probability predictedby our classifier. BCE does not explicitly address requirement(i), but can be used to benchmark noise-aware losses that aimto address (ii).

LR: The Label Regularization, initially proposed as a semi-supervised loss to learn from unlabeled data [39], has beenshown to address class-dependent label noise in malwareclassification [17] using a logistic regression classifier.

LLR =− 1N

N

∑i=1

[yilog(pθ(xi))]−λKL(p||pθ)

The loss function complements the log-likelihood loss overthe positive examples with a label regularizer, which is theKL divergence between a noise prior p and the classifier’soutput distribution over the negative examples pθ:

pθ =1N

N

∑i=1

[(1− yi)log(1− pθ(xi))]

Intuitively, the label regularizer aims to push the classifier pre-dictions on the noisy class towards the expected noise priorp, while the λ hyper-parameter controls the regularizationstrength. We use this loss to observe the extent to whichexisting noise correction approaches for related security tasksapply to our problem. However, this function was not de-signed to address (ii) and, as our results will reveal, yieldspoor performance in our problem.

FC: The Forward Correction loss has been shown to sig-nificantly improve robustness to class-dependent label noisein various computer vision tasks [50]. The loss requires apre-defined noise transition matrix T ∈ [0,1]2x2, where each

0 0.2 0.4 0.6 0.8 1

pθ(x)

0

1

2

3

4

5

6

Loss

(pθ(x

i),0

)

prior=0.0

prior=0.1

prior=0.3

prior=0.5

prior=0.7

prior=0.9

(a)

0 0.2 0.4 0.6 0.8 1

pθ(x)

0

1

2

3

4

5

6

Loss

(pθ(x

i),1

)

prior=0.0

prior=0.1

prior=0.3

prior=0.5

prior=0.7

prior=0.9

(b)

Figure 6: Value of the FC loss function of the output pθ(xi),for different levels of prior p, when y = 0 (a) and y = 1 (b)

element represents the probability of observing a noisy labely j for a true label yi: Ti j = p(y j|yi). For an instance xi, the log-likelihood is then defined as lc(xi) =−log(T0c(1− pθ(xi))+T1c pθ(xi)) for each class c ∈ {0,1}. In our case, because weassume that the probability of falsely labeling non-exploitedvulnerabilities as exploited is negligible, the noise matrix can

be defined as: T =

(1 0p 1− p

), and the loss reduces to:

LFC =− 1N

N

∑i=1

[yilog((1− p)pθ(xi))+

+(1− yi)log(1− (1− p)pθ(xi))]

On the negative class, the loss reduces the penalty for con-fident positive predictions, allowing the classifier to output ahigher score for predictions which might have noisy labels.This prevents the classifier from fitting of instances with po-tentially noisy labels.

Figure 6 plots the value of the loss function on a singleexample, for both classes and across the range of priors p. Onthe negative class, the loss reduces the penalty for confidentpositive predictions, allowing the classifier to output a higherscore for predictions which might have noisy labels. Thisprevents the classifier from fitting of instances with potentiallynoisy labels. FC partially addresses requirement (i), beingexplicitly designed only for class-dependent noise. However,unlike LR, it naturally addresses (ii) because it is equivalentto BCE if p = 0.

FFC: To fully address (i), we modify FC to account forfeature-dependent noise, a loss function we denote FeatureForward Correction (FFC). We observe that for exploit pre-diction, feature-dependent noise occurs within the same labelflipping template as class-dependent noise. We use this ob-servation to expand the noise transition matrix with instancespecific priors: Ti j(x) = p(y j|x,yi). In this case the transitionmatrix becomes:

T (x) =

(1 0

p(x) 1− p(x)

)Assuming that we only possess priors for instances that have

10

certain features f , the instance prior can be encoded as alookup-table:

p(x,y) =

{p f if y = 0 and x has f

0 otherwiseWhile feature-dependent noise might cause the classifier tolearn a spurious correlation between certain features and thewrong negative label, this formulation mitigates the issue byreducing the loss only on the instances that possess thesefeatures. In Section 7 we show that feature-specific priorestimates are achievable from a small set of instances, and usethis observation to compare the utility of class- and feature-specific noise priors in addressing label noise. When trainingthe classifier, we discovered optimal performance when usingan ADAM optimizer for 20 epochs and a batch size of 128,using a learning rate of 5e-6.Classifier deployment. We evaluate the historic performanceof our classifier by partitioning the dataset into temporal splits,assuming that the classifier is re-trained periodically, on allthe historical data available at that time. At the time theclassifier is trained, we do not include the vulnerabilities dis-closed within the last year because the positive labels fromexploitation evidence might not be available until later on. Wediscovered that the classifier needs to be retrained every sixmonths, as less frequent retraining would affect performancedue to a larger time delay between the disclosure of trainingand testing instances. During testing, the system operates ina streaming environment in which it continuously collectsthe data published about vulnerabilities then recomputes theirfeature vectors over time and predicts their updated EE score.The prediction for each test-time instance is performed withthe most recently trained classifier. To observe how our clas-sifier performs over time, we train the classifier using thevarious loss functions and test its performance on all vulner-abilities disclosed between January 2010, when 65% of ourdataset was available for training, and March 2020.

7 Evaluation

We evaluate our approach of predicting expected exploitabil-ity by testing EE on real-world vulnerabilities and answeringthe following questions, which are designed to address thethird challenge identified in Section 3: How effective is EE ataddressing label noise? How well does EE perform comparedto baselines? How well do various artifacts predict exploitabil-ity? How does EE performance evolve over time? Can EEanticipate imminent exploits? Does EE have practicality forvulnerability prioritization?

7.1 Feature-dependent Noise RemediationTo observe the potential effect of feature-dependent labelnoise on our classifier, we simulate a worst-case scenario inwhich our training-time ground truth is missing all the ex-

Feature % Noise Actual Prior Est. Prior # Inst to Est.CWE-79 14% 0.93 0.90 29CWE-94 7% 0.36 0.20 5CWE-89 20% 0.95 0.95 22

CWE-119 14% 0.44 0.57 51CWE-20 6% 0.39 0.58 26CWE-22 8% 0.39 0.80 15Windows 8% 0.35 0.87 15

Linux 5% 0.32 0.50 4

Table 4: Noise simulation setup. We report the % of negativeinstances that are noisy, the actual and estimated noise prior,and the # of instances used to estimate the prior.

BCE LR FC FFC

Feature P AUC P AUC P AUC P AUC

CWE-79 0.58 0.80 0.67 0.79 0.58 0.81 0.75 0.87

CWE-94 0.81 0.89 0.71 0.81 0.81 0.89 0.82 0.89

CWE-89 0.61 0.82 0.57 0.74 0.61 0.82 0.81 0.89

CWE-119 0.78 0.88 0.75 0.83 0.78 0.87 0.81 0.89

CWE-20 0.81 0.89 0.72 0.82 0.80 0.88 0.82 0.90

CWE-22 0.81 0.89 0.69 0.80 0.81 0.89 0.83 0.90

Windows 0.80 0.88 0.71 0.81 0.80 0.88 0.83 0.90

Linux 0.81 0.89 0.71 0.81 0.81 0.89 0.82 0.90

Table 5: Noise simulation results. We report the precision ata 0.8 recall (P) and the precision-recall AUC. The pristineBCE classifier performance is 0.83 and 0.90 respectively.

ploits for certain features. The simulation involves trainingthe classifier on dataset DS2, on a ground truth where all thevulnerabilities with a specific feature f are considered notexploited. At testing time, we evaluate the classifier on theoriginal ground truth labels. Table 4 describes the setup forour experiments. We investigate 8 vulnerability features, partof the Vulnerability Info category, that we analyzed in Sec-tion 5: the six most prevalent vulnerability types, reflectedthrough the CWE-IDs, as well as the two most popular prod-ucts: linux and windows. Mislabeling instances with thesefeatures results in a wide range of noise: between 5-20% ofnegative labels become noisy during training.

All techniques require priors about the probability of noise.The LR and FC approaches require a prior p over the entirenegative class. To evaluate an upper bound of their capabil-ities, we assume perfect prior and set p to match the frac-tion of training-time instances that are mislabeled. The FFCapproach assumes knowledge of the noisy feature f . Thisassumption is realistic, as it is often possible to enumerate thefeatures that are most likely noisy (e.g. prior work identifiedlinux as a noise-inducing feature due to the fact that thevendor collecting exploit evidence does not have a productfor the platform [60]). Besides, FFC requires estimates of thefeature-specific priors p f . We assume an operational scenariowere p f is estimated once, by manually labeling a subset of

11

instances collected after training. We use the vulnerabilitiesdisclosed in the first 6 months after training for estimatingp f and require that these vulnerabilities are correctly labeled.Table 4 shows the actual and the estimated priors p f , as wellas the number of instances used for the estimation. We ob-serve that the number of instances required for estimationis small, ranging from 5 to 51 across all features f , whichdemonstrates that setting feature-based priors is feasible inpractice. Nevertheless, we observe that the estimated priorsare not always accurate approximations of the actual ones,which might negatively impact FFC’s ability to address theeffect of noise.

In Table 5 we list the results of our experiment. For eachclassifier, we report the precision achievable at a recall of 0.8,as well as the precision-recall AUC. Our first observationis that the performance of the vanilla BCE classifier is notequally affected by noise across different features. Interest-ingly, we observe that the performance drop does not appearto be linearly dependent on the amount of noise: both CWE-79and CWE-119 result in 14% of the instances being poisoned,yet only the former inflicts a substantial performance dropon the classifier. Overall, we observe that the majority ofthe features do not result in significant performance drops,suggesting that BCE offers a certain amount of built-in robust-ness to feature-dependent noise, possibly due to redundanciesin the feature space which cancel out the effect of the noise.

For LR, after performing a grid search for the optimalλ parameter which we set to 1, we were unable to matchthe BCE performance on the pristine classifier. Indeed, weobserve that the loss is unable to correct the effect of noiseon any of the features, suggesting that it is not a suitablechoice for our classifier as it does not address any of the tworequirements of our classifier.

On features where BCE is not substantially affected bynoise, we observe that FC performs similarly well.However,on CWE-79 and CWE-89, the two features which inflict themost performance drop, FC is not able to correct the noiseeven with perfect priors, highlighting the inability of the ex-isting technique to capture feature-dependent noise. In con-trast, we observe that FFC provides a significant performanceimprovement. Even for the feature inducing the most degra-dation, CWE-79, the FFC AUC is restored within 0.03 pointsof the pristine classifier, although suffering a slight precisiondrop. On most features, FCC approaches the performance ofthe pristine classifier, in spite of being based on inaccurateprior estimates.

Our result highlights the overall benefits of identifyingpotential sources of feature-dependent noise, as well as theneed for noise correction techniques tailored to our problem.In the remainder of this section, we will use the FFC withp f = 0 (which is equivalent to BCE), in order to observe howthe classifier performs in absence of any noise priors.

EE (AUC = 0.73)

Baseline: EPSS (AUC = 0.45)

Baseline: Social Media Clf (AUC = 0.59)

Baseline: MS Exploitability

Baseline: RedHat Severity

Baseline: CVSSv3 Exploitability

0 0.2 0.4 0.6 0.8 1

Recall

0

0.2

0.4

0.6

0.8

1

Precision

(a)

NVD Info (AUC = 0.62)

Handcrafted (EPSS) (AUC = 0.47)

Social Media (AUC = 0.56)

Write-Ups (AUC = 0.72)

PoC Info (AUC = 0.87)

PoC Code (AUC = 0.86)

0 0.2 0.4 0.6 0.8 1

Recall

0

0.2

0.4

0.6

0.8

1

Precision

(b)

Figure 7: Performance, evaluated 30 days after disclosure,of (a) EE compared to baselines, (b) individual feature cate-gories. We report the Area under the Curve (AUC) and listthe corresponding TPR/FPR curves in Appendix A.1.

7.2 Effectiveness of Exploitability Prediction

Next, we evaluate the effectiveness of our system compared tothe three static metrics described in Section 5, as well as twostate-of-the-art classifiers from prior work. These two predic-tors, EPSS [33], and the Social Media Classifier (SMC) [60],were proposed for exploits in the wild and we re-implementand re-train them for our task. EPSS trains an ElasticNetregression model on the set of 53 hand-crafted features ex-tracted from vulnerability descriptors. SMC combines thesocial media features with vulnerability information featuresfrom NVD to learn a linear SVM classifier. For both baselines,we perform hyper-parameter tunning and report the highestperformance across all experiments, obtained using λ= 0.001for EPSS and C = 0.0001 for SMC. SMC is trained startingfrom 2015, as our tweets collection does not begin earlier.

In Figure 7a we plot the precision-recall trade-off of theclassifiers trained on dataset DS1, evaluated 30 days after thedisclosure of test-time instances. We observe that none ofthe static exploitability metrics exceed 0.5 precision, whileEE significantly outperforms all the baselines. The perfor-mance gap is especially apparent for the 60% of exploitedvulnerabilities, where EE achieves 86% precision, whereasthe SMC, the second-best performing classifier, obtains only49%. We also observe that for around 10% of vulnerabilities,the artifacts available within 30 days have limited predictiveutility, which affects the performance of these classifiers.EE uses the most informative features. To understand whyEE is able to outperform these baselines, in Figure 7b we plotthe performance of EE trained and evaluated on individualcategories of features (i.e., only considering instances whichhave artifacts within these categories). We observe that thehandcrafted features are the worst performing category, per-haps due to the fact that the 53 features are not sufficient tocapture the large diversity of vulnerabilities in our dataset.These features encode the existence of public PoCs, which

12

Component: NVD & Write-Ups (AUC = 0.85)

Component: Poc Info & Code (AUC = 0.88)

All: EE (AUC = 0.92)

0 0.2 0.4 0.6 0.8 1

Recall

0.5

0.6

0.7

0.8

0.9

1

Precision

(a)

ExploitedNon-Exploited

0 5 10 15 20 25 30

EE Prediction Delay

0

0.1

0.2

0.3

0.4

0.5

Mea

n ε

(b)

Figure 8: (a) Performance of EE compared to constituentsubsets of features. (b) P evaluated at different points in time.

is often used by practitioners as a heuristic rule for deter-mining which vulnerabilities must be patched urgently. Ourresults suggest that this heuristic provides a weak signal forthe emergence of functional exploits, in line with prior workpredicting exploits [2, 33, 68], which concluded that PoCs"are not a reliable source of information for exploits in thewild" [2]. Nevertheless, we can achieve a much higher preci-sion at predicting exploitability by extracting deeper featuresfrom the PoCs. The PoC Code features provide a 0.93 preci-sion for half of the exploited vulnerabilities, outperformingall other categories. This suggests that code complexity canbe a good indicator for the likelihood of functional exploits,although not on all instances, as indicated by the sharp dropin precision beyond the 0.5 recall. A major reason for thisdrop is the existence of post-exploit mitigation techniques:even if a PoC is complex and contains advanced functionality,defenses might impede successful exploitation beyond denialof service. This highlights how our feature extractor is ableto represent PoC descriptions and code characteristics whichreflect exploitation efforts. Both the PoC and Write-up fea-tures, which EE capitalizes on, perform significantly betterthan other categories.

Surprisingly, we observe that social media features, arenot as useful for predicting functional exploits as they are forexploits in the wild [60], a finding reinforced by the resultsof the experiments conducted below, which show that theydo not improve upon other categories. This is because tweetstend to only summarize and repeat information from write-ups, and often do not contain sufficient technical informationto predict exploit development. Besides, they often incur anadditional publication delay over the original write-ups theyquote. Overall, our evaluation highlights a qualitative distinc-tion between the problem of predicting functional exploitsand that of predicting exploits in the wild.EE improves when combining artifacts. Next, we look atthe interaction among features on dataset DS2. In Figure 8awe compare the performance of EE trained on all feature sets,with that trained on PoCs and vulnerability features alone.We observe that PoC features outperform these from vul-

nerabilities, while their combination results in a significantperformance improvement. The result highlights the two cat-egories complement each other and confirm that PoC featuresprovide additional utility for predicting exploitability. Onthe other hand, as described below, we observe no addedbenefit when incorporating social media features into EE. Wetherefore exclude them from our final EE feature set.EE performance improves over time. In order to evaluatethe benefits of time-varying exploitability, the precision-recallcurves are not sufficient, because they only capture a snapshotof the scores in time. In practice, the EE score would becompared to that of other vulnerabilities disclosed within ashort time, based on their most recent scores. Therefore, weintroduce a metric P to compute the performance of EE interms of the expected probability of error over time.

For a given vulnerability i, its score EEi(z) computed ondate z and its label Di (Di = 1 if i is exploited and 0 other-wise), the error P EE(z, i,S) w.r.t. a set of vulnerabilities S iscomputed as:

P EE(z, i,S) =

||{D j=0∧EE j(z)≥EEi(z)| j∈S}||

||S|| if Di = 1||{D j=1∧EE j(z)≤EEi(z)| j∈S}||

||S|| if Di = 0

If i is exploited, the metric reflects the number of vulnerabil-ities in S which are not exploited but are scored higher thani on date z. Conversely, if i is not exploited, P computesthe fraction of exploited vulnerabilities in S which are scoredlower than it. The metric captures the amount of effort spentprioritizing vulnerabilities with no known exploits. For bothcases, a perfect score would be 0.0.

For each vulnerability, we set S to include all other vulner-abilities disclosed within t days after its disclosure. Figure 8bplots the mean P over the entire dataset, when varying tbetween 0 and 30, for both exploited and non-exploited vul-nerabilities. We observe that on the day of disclosure, EEalready provides a high performance for exploited vulnera-bilities: on average, only 10% of the non-exploited vulnera-bilities disclosed on the same day will be scored higher thanan exploited one. However, the score tends to overestimatethe exploitability of non-exploited vulnerabilities, resultingin many false positives. This is in line with prior observa-tions that static exploitability estimates available at disclosurehave low precision [60]. By following the two curves alongthe X-axis, we observe the benefits of time-varying features.Over time, the errors made on non-exploited vulnerabilitiesdecrease substantially: while such a vulnerability is expectedto be ranked above 44% exploited ones on the day of dis-closure, it will be placed above 14% such vulnerabilities 10days later. The plot also shows that this sharp performanceboost for the non-exploited vulnerabilities incurs a smallerincrease in error rates for the exploited class. We do not ob-serve great performance improvements after 10 days fromdisclosure. Overall, we observe that time-varying exploitabil-ity contributes to a substantial decrease in the number of false

13

NVD (AUC = 0.72)

Component: Poc Info & Code (AUC = 0.86)

NVD & Write-Ups (AUC = 0.82)

NVD & Write-Ups & PoCs (EE) (AUC = 0.89)

EE + Social Media (AUC = 0.89)

0 0.2 0.4 0.6 0.8 1

Recall

0

0.2

0.4

0.6

0.8

1

Precision

(a)

NVD (AUC = 0.76)

Component: Poc Info & Code (AUC = 0.87)

NVD & Write-Ups (AUC = 0.84)

NVD & Write-Ups & PoCs (EE) (AUC = 0.90)

EE + Social Media (AUC = 0.90)

0 0.2 0.4 0.6 0.8 1

False Positive Rate

0

0.2

0.4

0.6

0.8

1

Tru

e P

ositi

ve

Ra

te

(b)

Figure 9: Performance of the classifier when adding SocialMedia features.

Unigrams (EE) (AUC = 0.92)

TF-IDF (AUC = 0.90)

Embeddings (AUC = 0.82)

0 0.2 0.4 0.6 0.8 1

Recall

0.5

0.6

0.7

0.8

0.9

1

Precision

(a)

Unigrams (EE) (AUC = 0.91)

TF-IDF (AUC = 0.89)

Embeddings (AUC = 0.80)

0 0.2 0.4 0.6 0.8 1

False Positive Rate

0

0.2

0.4

0.6

0.8

1

Tru

e P

ositi

ve

Ra

te

(b)

Figure 10: Performance of the classifier when consideringadditional NLP features.

positives, therefore improving the precision our estimates. Tocomplement our evaluation, the precision-recall trade-offs atvarious points in time is reported in Appendix A.1.Social Media features do not improve EE. In Figure 9 weinvestigate the effect of adding Social Media features on EE.The results evaluate the classifier trained on DS2, over thetime period spanning our tweets collection. We observe that,unlike the addition of PoC features to these extracted fromvulnerability artifacts, these features do not improve the per-formance of the classifier. This is because tweets generallyreplicate and summarize the information already contained inthe technical write-ups that they link to. Because these fea-tures convey little extra technical information beyond otherartifacts, potentially also incurring an additional publicationdelay, we do not incorporate these features in the final featureset of EE.Effect of higher-level NLP features on EE. We investigatetwo alternative representations for natural language features:TF-IDF and paragraph embeddings. TF-IDF is a commondata mining metric used to encode the importance of individ-ual terms within a document, by means of their frequencywithin a document, scaled by their inverse prevalence acrossthe dataset. Paragraph embeddings, which were also used

by DarkEmbed [68] to represent vulnerability-related postsfrom underground forums, encode the word features into afixed-size vector space. In line with prior work, we use theDoc2Vec model [35] to learn the embeddings on the docu-ment from the training set. We use separate models on theNVD descriptions, Write-ups, PoC Info and the commentsfrom the PoC Code artifacts. We perform grid search for thehyper-parameters of the model, and report the performanceof the best-performing models. The 200-dimensional vectorsare obtained from the distributed bag of words (D-BOW) al-gorithm trained over 50 epochs, using a windows size of 4,a sampling threshold of 0.001, using the sum of the contextwords, and a frequency threshold of 2.

In Figure 10 we compare the effect of the alternative NLPfeatures on EE. First, we observe that TF-IDF does not im-prove the performance over unigrams. This suggests that ourclassifier does not require term frequency to learn the vulner-ability characteristics reflected through artifacts, which seemsto even hurt performance slightly. This can be explained intu-itively, as different artifacts frequently reuse the same jargonfor the same vulnerability, but the number of distinct artifactsis not necessarily correlated with exploitability. However, theTF-IDF classifier might over-emphasize the numerical valueof these features, rather than learning their presence.

Surprisingly, the embedding features result in a significantperformance drop, in spite of our hyper-parameter tuning at-tempts. We observe that the various natural language artifactsin our corpus are long and verbose, resulting in a large numberof tokens that need to be aggregated into a single embeddingvector. Due to this aggregation and feature compression, thedistinguishing words which indicate exploitability might notremain sufficiently expressive within the final embedding vec-tor that our classifier uses as inputs. While our results donot align with the DarkEmbed work finding that paragraphembeddings outperform simpler features, we note that Dark-Embed is primarily using posts from underground forums,which the authors report are shorter than public write-ups.Overall, our result reveals that creating higher level, semantic,NLP features for exploit prediction is a challenging problem,and requires solutions beyond using off-the-shelf tools. Weleave this problem to future work.EE is stable over time. In order to observe how EE is in-fluenced by the publication of various artifacts, we look atthe changes in the score of the classifier. Figure 11 plots,for the entire test set, the distribution of score changes intwo cases: at the time of disclosure compared to an instancewith no features, and from the second to the 30th day afterdisclosure, on days where there were artifacts published foran instance. We observe that at the time of disclosure, theclassifier changes drastically, shifting the instance towardseither 0.0 or 1.0, while the large magnitude of the shifts in-dicate a high confidence. However, we observe that artifactspublished on subsequent days have a much different effect.In 79% of cases, published artifacts have almost no effect on

14

At disclosure

Days 2-30 after

-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0

EE score change

0

0.2

0.4

0.6

0.8

PD

F

Figure 11: The distribution of EE score changes, at the timeof disclosure and on all events within 30 days after disclosure.

EE w/o 10% PoCs (AUC = 0.85)EE w/o 30% PoCs (AUC = 0.86)EE w/o 50% PoCs (AUC = 0.87)EE w/o 70% PoCs (AUC = 0.88)EE w/o 90% PoCs (AUC = 0.89)EE w all PoCs (AUC = 0.90)

0 0.2 0.4 0.6 0.8 1

Recall

0.5

0.6

0.7

0.8

0.9

1

Precision

(a)

EE w/o 10% PoCs (AUC = 0.82)EE w/o 30% PoCs (AUC = 0.84)EE w/o 50% PoCs (AUC = 0.86)EE w/o 70% PoCs (AUC = 0.87)EE w/o 90% PoCs (AUC = 0.88)EE w all PoCs (AUC = 0.89)

0 0.2 0.4 0.6 0.8 1

False Positive Rate

0

0.2

0.4

0.6

0.8

1

True

Pos

itive

Rat

e

(b)

Figure 12: Performance of our classifier when a fraction ofthe PoCs is missing.

changing the classification score, while the remaining 21%of events are the primary drivers of score changes. The twoobservations allow us to conclude that artifacts published atthe time of disclosure contain some of the most informativefeatures, and that EE is stable over time, its evolution beingdetermined by few consequential artifacts.EE is robust to missing exploits. In order to observe how EEperforms when some of the PoCs are missing, we simulatea scenario in which a varying fraction of them are not seenat test-time for vulnerabilities in DS1. The results are plottedin Figure 12, and highlight that, even if a significant fractionof PoCs is missing, our classifier is able to utilize the othertypes of artifacts to maintain a high performance.

7.3 Case StudiesIn this section we investigate the practical utility of EE throughthree case studies.EE for critical vulnerabilities. To understand how well EEdistinguishes important vulnerabilities, we measure its per-formance on a list of recent ones flagged for prioritized re-mediation by FireEye [23]. The list was published on De-

PCV SS P EPSS P EE (δ) P EE (δ+10) P EE (δ+30) P EE (2020-12-07)

Mean 0.51 0.36 0.31 0.25 0.22 0.04Std 0.24 0.28 0.33 0.25 0.27 0.11

Median 0.35 0.40 0.22 0.12 0.10 0.00

Table 6: Performance of EE and baselines at prioritizing crit-ical vulnerabilities. P captures the fraction of recent non-exploited vulnerabilities scored higher than critical ones.

PCV SS P EPSS P EE (δ) P EE (δ+10) P EE (δ+30) P EE (2020-12-07)

Mean 0.51 0.42 0.34 0.34 0.23 0.11Std 0.39 0.32 0.40 0.40 0.30 0.26

Median 0.43 0.35 0.00 0.04 0.14 0.00

Table 7: Performance of EE and baselines at prioritizing crit-ical vulnerabilities. P captures the fraction of recent non-exploited vulnerabilities for the same products and scoredhigher than critical ones.

cember 8, 2020, after the corresponding functional exploitswere stolen [24]. Our dataset contains 15 of the 16 criticalvulnerabilities.

We measure how well our classifier prioritizes these vul-nerabilities compared to static baselines, using the P prioriti-zation metric defined in the previous section, which computesthe fraction of non-exploited vulnerabilities from a set S thatare scored higher than the critical ones. For each of the 15 vul-nerabilities, we set S to contain all others disclosed within 30days from it, which represent the most frequent alternativesfor prioritization decisions. Table 6 compares the statisticsfor the baselines, and for P EE computed on the date criti-cal vulnerabilities were disclosed δ, 10 and 30 days later,as well as one day before the prioritization recommendationwas published. CVSS scores are published a median of 18days after disclosure, and we observe that EE already outper-forms static baselines based only on the features availableat disclosure, while time-varying features improve perfor-mance significantly. Overall, one day before the prioritizationrecommendation is issued, our classifier scores the criticalvulnerabilities below only 4% of these with no known exploit.Table 7 shows the performance statistics of our classifier when|S| contains only vulnerabilities published within 30 days ofthe critical ones and that affect the same products as the criti-cal ones. The result further highlights the utility of EE, as itsranking outperforms baselines and prioritizes the most criticalvulnerabilities for a particular product.

Table 8 lists the 15 out of 16 critical vulnerabilities in ourdataset flagged by FireEye. The table lists the estimated dis-closure date, the number of days after disclosure when CVSSwas published, and when exploitation evidence emerged. Ta-ble 9 contains the per-vulnerability performance of our classi-fier for all 15 vulnerabilities when |S| contains vulnerabilitiespublished within 30 days of the critical ones. Below, we man-ually analyze some of the 15 vulnerabilities in more detailsby combining EE and P .

15

EE at disclosureEE after 10 daysEE after 20 daysCVSS ExploitabilityEPSS

5 10 15 20 25 30

Exploit Publ. Delay (# days after disclosure)

0.4

0.5

0.6

0.7

0.8

0.9

1

Tim

e-va

ryin

g A

UC

Sco

re

(a)

EE w 0% PoCs functionalEE w 10% PoCs functionalEE w 30% PoCs functionalEE w 70% PoCs functionalEE w 90% PoCs functional

5 10 15 20 25 30

Exploit Publ. Delay (# days after disclosure)

0.4

0.5

0.6

0.7

0.8

0.9

1

Tim

e-va

ryin

g A

UC

Sco

re

(b)

Figure 13: Time-varying AUC when distinguishing exploitspublished within t days from disclosure (a) for EE and base-lines, (b) simulating earlier exploit availability.

CVE-2019-0604: Table 9 shows the performance of ourclassifier on CVE-2019-0604, which improves when moreinformation becomes publicly available. At the disclosuretime, there is only one available write-up which yields a lowEE because it contains little descriptive features. 23 days later,when NVD descriptions become available, EE decreases evenfurther. However, two technical write-ups on days 87 and 352result in sharp increases of EE, from 0.03 to 0.22 and to 0.78respectively. This is because they contain detailed technicalanalyses of the vulnerability, which our classifier interpretsas an increased exploitation likelihood.

CVE-2019-8394: P fluctuates between 0.82 and 0.24 onCVE-2019-8394. At disclosure time, this vulnerability gath-ers only one write-up, and our classifier outputs a low EE.From disclosure time to day 10, there are two small changesin EE, but at day 10, when NVD information is available,there is a sharp decrease on EE from 0.12 to 0.04. From day10 to day 365, EE does not change anymore due to no moreinformation added. The decrease of EE at day 10 explainsthe sharp jump between P EE(0) and P EE(10) but not thefluctuations after P EE(10). This is caused by the EE of othervulnerabilities disclosed around the same period, which ourclassifier ranks higher than CVE-2019-8394.

CVE-2020-10189 and CVE-2019-0708: These two vul-nerabilities receive high EE throughout the entire observa-tion period, due to detailed technical information availableat disclosure, which allows our classifier to make confidentpredictions. CVE-2019-0708 gathers 35 write-ups in total,and 4 of them are available at disclosure. Though CVE-2020-10189 only gathers 4 write-ups in total, 3 of them are availablewithin 1 day of disclosure and contained informative features.These two examples show that our classifier benefits from anabundance of informative features published early on, and thisinformation contribute to confident predictions that remainstable over time.

Our results indicate that EE is a valuable input to patchingprioritization frameworks, because it outperforms existingmetrics and improves over time.

CVE-ID Disclosure CVSS Delay Exploit Delay

2019-11510 2019-04-24 15 125

2018-13379 2019-03-24 73 146

2018-15961 2018-09-11 66 93

2019-0604 2019-02-12 23 86

2019-0708 2019-05-14 2 8

2019-11580 2019-05-06 28 ?

2019-19781 2019-12-13 18 29

2020-10189 2020-03-05 1 5

2014-1812 2014-05-13 1 1

2019-3398 2019-03-31 22 19

2020-0688 2020-02-11 2 16

2016-0167 2016-04-12 2 ?

2017-11774 2017-10-10 24 ?

2018-8581 2018-11-13 34 ?

2019-8394 2019-02-12 10 412

Table 8: List of exploited CVE-IDs in our dataset recentlyflagged for prioritized remediation. Vulnerabilities whereexploit dates are unknown are marked with ’?’.

EE for emergency response. Next, we investigate how wellour classifier can predict exploits published shortly after dis-closure. To this end, we look at the 924 vulnerabilities in DS3for which we obtained exploit publication estimates. To testwhether the vulnerabilities in DS3 are a representative sampleof all other exploits, we perform a two-sample test, underthe null hypothesis that vulnerabilities in DS3 and exploitedvulnerabilities in DS2 which are not in DS3 are drawn fromthe same distribution. Because instances are multivariate andour classifier learns feature representations for these vulnera-bilities, we use a technique designed for this scenario, calledClassifier Two-Sample Tests(C2ST) [38]. C2ST repeatedlytrains classifiers to distinguish between instances in the twosamples and, using a Kolmogorov-Smirnoff test, comparesthe probabilities assigned to instances from the two, in orderto determine whether any statistically significant differencecan be established between them. We apply C2ST on thefeatures learned by our classifier (the last hidden layer whichcontains 100 dimensions), and we were unable to reject thenull hypothesis that the two samples are drawn from the samedistribution, at p=0.01. Based on this result, we concludethat DS3 is a representative sample of all other exploits inour dataset. This means that, when considering the featuresevaluated in our paper, we find no evidence of biases in DS3.

We measure the performance of EE at predicting vulnerabil-ities exploited within t days from disclosure. For a given vul-nerability i and EEi(z) computed on date z, we can computethe time-varying sensitivity Se=P(EEi(z)> c|Di(t)= 1) and

16

CVE-ID PCV SS P EPSS P EE(0) P EE(10) P EE(30) P EE(2020-12-07)

2014-1812 0.81 0.48 0.00 0.01 0.03 0.03

2016-0167 0.97 0.15 0.79 0.50 0.13 0.04

2017-11774 0.61 0.12 0.99 0.13 0.23 0.08

2018-13379 0.28 0.42 0.00 0.06 0.06 0.00

2018-15961 0.25 0.55 0.39 0.46 0.41 0.00

2018-8581 0.64 0.30 0.42 0.29 0.26 0.01

2019-0604 0.34 0.54 0.73 0.62 0.80 0.01

2019-0708 0.30 0.07 0.00 0.00 0.00 0.00

2019-11510 0.34 0.85 0.45 0.41 0.61 0.00

2019-11580 0.32 0.89 0.04 0.06 0.01 0.02

2019-19781 0.36 0.01 0.09 0.13 0.00 0.00

2019-3398 0.82 0.40 0.67 0.30 0.10 0.00

2019-8394 0.69 0.07 0.22 0.82 0.76 0.48

2020-0688 0.77 0.62 0.00 0.00 0.00 0.00

2020-10189 0.24 0.01 0.00 0.00 0.00 0.00

Table 9: The performance of baselines and EE at prioritizing critical vulnerabilities.

specificity Sp = P(EEi(z) ≤ c|Di(t) = 0) [28], where Di(t)indicates whether the vulnerability was already exploited bytime t. By varying the detection threshold c, we obtain thetime-varying AUC of the classifier which reflects how wellthe classifier separates exploits happening within t days fromthese happening later on. In Figure 13a we plot the AUC forour classifier evaluated on the day of disclosure δ, as well as10 and 20 days later, for the exploits published within 30 days.While the CVSS Exploitability remains below 0.5, EE(δ)constantly achieves an AUC above 0.68.This suggests thatthe classifiers implicitly learns to assign higher scores to vul-nerabilities that are exploited sooner than to these exploitedlater. For EE(δ+10) and EE(δ+20), in addition to similartrends over time, we also observe the benefits of additionalfeatures collected in the days after disclosure, which shift theoverall prediction performance upward.

We further consider the possibility that the timestamps inDS3 may be affected by label noise. We evaluate the potentialimpact of this noise with an approach similar to the one inSection 7.1. We simulate scenarios where we assume that apercentage of PoCs are already functional, which means thattheir later exploit-availability dates in DS3 are incorrect. Forthose vulnerabilities, we update the exploit availability dateto reflect the publication date of these PoCs. This providesa conservative estimate, because the mislabeled PoCs couldbe in an advanced stage of development, but not yet fullyfunctional, and the exploit-availability dates could also be settoo early. We simulate percentages of late timestamps rangingfrom 10–90%. Figure 13b plots the performance of EE(δ) inthis scenario, averaged over 5 repetitions. We observe that

Player 1 Player 2Loss if attacked l1(t) = 5000,∀t l2(t) = 500,∀t

Patching rate h1(t) = 1−0.8t ,∀t h2(t) = 1−0.8t ,∀t

Table 10: Cyber-warfare game simulation parameters.

even if 70% of PoCs are considered functional, the classi-fier outperforms the baselines and maintains an AUC above0.58, Interestingly, performance drops after disclosure andis affected the most on predicting exploits published within12 days. Therefore, the classifier based on disclosure-timeartifacts learns features of easily exploitable vulnerabilities,which get published immediately, but does not fully capturethe risk of functional PoC that are published early. We miti-gate this effect by updating EE with new artifacts daily, afterdisclosure. Overall, the result suggests that EE may be use-ful in emergency response scenarios, where it is critical tourgently patch the vulnerabilities that are about to receivefunctional exploits.EE for vulnerability mitigation. To investigate the practicalutility of EE, we conduct a case study of vulnerability miti-gation strategies. One example of vulnerability mitigation iscyber warfare, where nations acquire exploits and make de-cisions based on new vulnerabilities. Existing cyber-warfareresearch such as that by Bao et al. [5] rely on knowledge ofexploitability for game strategies. For these models, it is there-fore crucial that exploitability estimates are timely and accu-rate, because inaccuracies could lead to sub-optimal strategies.Because these requirements match our design decisions for

17

learning EE, we evaluate its effectiveness in the context of acyber-game. We simulate the case of CVE-2017-0144, thevulnerability targeted by the EternalBlue exploit [44]. Thegame has two players, where Player 1, a government, pos-sesses an exploit that gets stolen, and Player 2, an evil hackerwho might know about it could purchase it or re-create it. Weset the game parameters aligned with the real-world circum-stances for the EternalBlue vulnerability, shown in Table 10.In this setup, Player 1’s loss of being attacked is significantlygreater than Player 2’s, because a government needs to takeinto account the loss for a large population, as opposed tothat for a small group or an individual. Both players beginpatching once the vulnerability is disclosed, at round 0. Thepatching rates, which are the cumulative proportion of vul-nerable resources being patched over time, are equal for bothplayers and follow the pattern measured in prior work [7].We also assume that the exploit becomes available at t = 31,as this corresponds to the delay after which EternalBlue waspublished.

Our experiment assumes that Player 1 uses the cyber-warfare model [5] to compute whether they should attackPlayer 2 after vulnerability disclosure. The calculation re-quires Player 2’s exploitability, which we assign using twoapproaches: The CVSS Exploitability score normalized to 1(which yields a value of 0.55), and the time-varying EE. Ourclassifier outputs an exploitability of 0.94 on the day of disclo-sure, and updates it to 0.97 three days later, only to maintainit constant afterwards. We compute the optimal strategy forthe two approaches, and we compare them using the resultingutility for Player 1.

Figure 14 shows that the strategy associated with EE ispreferable over the CVSS one. Although Player 1 will in-evitably lose in the game (because they have a much largervulnerable population), EE improves Player 1’s utility by 10%.Interestingly, we find that EE also changes Player 1’s strat-egy to towards a more aggressive one. This is because EEis updated when more information emerges, which in turnincreases the expected exploitability assumed for Player 2.When Player 2 is unlikely to have a working exploit, Player1 would not attack because that may leak information onhow to weaponize the vulnerability, and Player 2 may con-vert the received exploit to an inverse attack [6]. As Player2’s exploitability increases, Player 1 will switch to attack-ing because it is likely that Player 2 already possesses anexploit. Therefore, an increasing exploitability pushes Player1 towards a more aggressive strategy.

8 Related Work

Predicting exploits in the wild. Most of the prior exploitprediction work has been towards the tangential task of pre-dicting exploits in the wild. This has been investigated in ourprior study [60] and Chen et al. [13] by monitoring Twitter

A

A

A

A

AA

AA

AAAAAAAAAAAAAAAAAAAAAAAAA

AA

S

S

S

S

S

SS

SS

SSSSSSSSSSSSSSSSSSSSSS

EE

CVSS Exploitability

A = ATTACK

S = STOCKPILE

0 5 10 15 20 25 30 35

Game Round

-20000

-15000

-10000

-5000

Pla

ye

r 1

's u

tility

Figure 14: Result of our cyber-warfare game simulation. Theutility of the player is improved by 2,000 points when usingEE. The player actions are also different, the CVSS playeronly attacking when the exploit is leaked, in round 31.

for vulnerability discussions, and Xiao et al. [77] by usingpost-disclosure field data about exploitation. Jacobs et al. [32]used vulnerability prevalence data to improve prediction. Ja-cobs et al. [33] proposed EPSS, a scoring system for exploits.Allodi [1] calculated the likelihood of observing exploits inthe wild after they are traded in underground forums.Vulnerability Exploitability. Allodi and Massacci [2] inves-tigated the utility of the CVSS scores for capturing the likeli-hood of attacks in the wild. Prior work by Bozorgi et al. [9]formulated exploitability estimation as the problem of predict-ing the existence of PoCs based on vulnerability characteris-tics. Allodi and Massaci [3] concluded that the publication ofa PoC in ExploitDB is not a good indicator for exploits in thewild. Our work shows that, while their presence might notbe a sufficiently accurate indicator, the features within thesePoCs are useful for predicting functional exploits. DarkEm-bed [68] uses natural language models trained on private datafrom underground forum discussions to predict the availabil-ity of exploits, but such artifacts are generally published witha delay [1]. Instead, EE uses only publicly available artifactsfor predicting exploits soon after disclosure; we were unableto obtain these artifacts for comparison upon contacting theauthors.PoCs. PoCs were also investigated in measurements on vul-nerability lifecycles. Shahzad et al [62] performed a measure-ment of the vulnerability lifecycle, discovering that PoCs aregenerally released at the time of disclosure. Mu. et al [45]manually curated and utilized PoCs to trigger the vulnerabil-ities. FUZE [76] used PoCs to aid exploit generation for 5functional exploits.Label Noise. The problem of label noise has been studiedextensively in machine learning [25], primarily focusing onrandom and class-dependent noise. Limited work focuses onfeature-dependent noise, which requires either strong theoret-ical guarantees about the sample space [41] or depends on alarge dataset of clean labels to learn noise probabilities [78].

18

In security, the closest to our work is the study by Deloachet al [17] which models noise in malware classification asclass-dependent.

9 Conclusions

By investigating exploitability as a time-varying process, wediscover that it can be learned using supervised classifica-tion techniques and updated continuously. We discover threechallenges associated with exploitability prediction. First, itis prone to feature-dependent label noise, a type consideredby the machine learning community as the most challenging.Second, it needs new categories of features, as it differs qual-itatively from the related task of predicting exploits in thewild. Third, it requires new metrics for performance evalua-tion, designed to capture practical vulnerability prioritizationconsiderations.

We design the EE metric, which, on a dataset of 103,137vulnerabilities, improves precision from 49% to 86% overstate-of-the art predictors. EE can learn to mitigate feature-dependent label noise, capitalizes on highly predictive fea-tures that we extract from PoCs and write-ups, improves overtime, and has practical utility in predicting imminent exploitsand prioritizing critical vulnerabilities.Acknowledgments. We thank Vulners and Frank Li for theirdata. We also thank the anonymous reviewers, YigitcanKaya and Ben Edwards for feedback. This material was sup-ported by a grant from the Department of Defense, the ArmyResearch Office (W911NF-17-1-0370), the National Sci-ence Foundation (CNS-2000792), and based upon work sup-ported by the Defense Advanced Research Projects Agency(DARPA) under Agreement No. HR00112190093. Approvedfor public release; distribution is unlimited.

References

[1] L. Allodi. Economic factors of vulnerability trade andexploitation. In Proceedings of the 2017 ACM SIGSACConference on Computer and Communications Security,pages 1483–1499. ACM, 2017.

[2] L. Allodi and F. Massacci. A preliminary analysisof vulnerability scores for attacks in wild. In CCSBADGERS Workshop, Raleigh, NC, Oct 2012.

[3] L. Allodi and F. Massacci. Comparing vulnerabilityseverity and exploits using case-control studies. ACMTransactions on Information and System Security (TIS-SEC), 17(1):1, 2014.

[4] Symantec attack signatures. https://www.symantec.com/security_response/attacksignatures/.

[5] T. Bao, Y. Shoshitaishvili, R. Wang, C. Kruegel, G. Vi-gna, and D. Brumley. How shall we play a game?: a

game-theoretical model for cyber-warfare games. In2017 IEEE 30th Computer Security Foundations Sym-posium (CSF), pages 7–21. IEEE, 2017.

[6] T. Bao, R. Wang, Y. Shoshitaishvili, and D. Brumley.Your exploit is mine: Automatic shellcode transplant forremote exploits. In 2017 IEEE Symposium on Securityand Privacy (SP), pages 824–839. IEEE, 2017.

[7] L. Bilge and T. Dumitras, . Before we knew it: anempirical study of zero-day attacks in the real world.In ACM Conference on Computer and CommunicationsSecurity, pages 833–844, 2012.

[8] BleepingComputer. Ie zero-day adopted by rig exploitkit after publication of poc code. https://www.bleepingcomputer.com/news/security/ie-zero-day-adopted-by-rig-exploit-kit-after-publication-of-poc-code/, 2018.

[9] M. Bozorgi, L. K. Saul, S. Savage, and G. M. Voelker.Beyond heuristics: learning to classify vulnerabilitiesand predict exploits. In KDD, Washington, DC, Jul2010.

[10] Bugtraq. Securityfocus. https://www.securityfocus.com/, 2019.

[11] A. Caliskan-Islam, R. Harang, A. Liu, A. Narayanan,C. Voss, F. Yamaguchi, and R. Greenstadt. De-anonymizing programmers via code stylometry. In24th USENIX Security Symposium (USENIX Security15), pages 255–270, 2015.

[12] A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay,and D. Mukhopadhyay. Adversarial attacks and de-fences: A survey. arXiv preprint arXiv:1810.00069,2018.

[13] H. Chen, R. Liu, N. Park, and V. Subrahmanian. Usingtwitter to predict when vulnerabilities will be exploited.In Proceedings of the 25th ACM SIGKDD InternationalConference on Knowledge Discovery & Data Mining,pages 3143–3152, 2019.

[14] T. M. Corporation. Common weaknesses enumeration.https://cwe.mitre.org.

[15] NotPetya Technical Analysis. https://www.crowdstrike.com/blog/petrwrap-ransomware-technical-analysis-triple-threat-file-encryption-mft-encryption-credential-theft/.

[16] D2 Security. D2 exploitation pack. https://www.d2sec.com/pack.html, 2019.

[17] J. DeLoach, D. Caragea, and X. Ou. Android malwaredetection with weak ground truth data. In 2016 IEEE

19

International Conference on Big Data (Big Data), pages3457–3464. IEEE, 2016.

[18] T. F. Dullien. Weird machines, exploitability, and prov-able unexploitability. IEEE Transactions on EmergingTopics in Computing, 2017.

[19] O. J. Dunn. Multiple comparisons among means. Jour-nal of the American statistical association, 56(293):52–64, 1961.

[20] C. Eiram. Exploitability/priority index rating systems(approaches, value, and limitations), 2013.

[21] Microsoft exploitability index. Microsoft, 30 March2020. https://www.microsoft.com/en-us/msrc/exploitability-index.

[22] ExploitDB. The exploit database. https://www.exploit-db.com/, 2019.

[23] FireEye. Fireeye red team tools. https://github.com/fireeye/red_team_tool_countermeasures/blob/master/CVEs_red_team_tools.md.

[24] FireEye. Unauthorized access of fireeye red team tools.https://www.fireeye.com/blog/threat-research/2020/12/unauthorized-access-of-fireeye-red-team-tools.html.

[25] B. Frénay and M. Verleysen. Classification in the pres-ence of label noise: a survey. IEEE transactions onneural networks and learning systems, 25(5):845–869,2013.

[26] GitHub. linguist. https://github.com/github/linguist, 2020.

[27] Hackerone - disclosure guidelines. HackerOne, 30March 2009. https://www.hackerone.com/disclosure-guidelines.

[28] P. J. Heagerty, T. Lumley, and M. S. Pepe. Time-dependent roc curves for censored survival data and adiagnostic marker. Biometrics, 56(2):337–344, 2000.

[29] T. W. House. Vulnerabilities equities policy and processfor the united states government, 2017.

[30] IBM. Ibm x-force exchange. https://exchange.xforce.ibmcloud.com/.

[31] Immunity Inc. Canvas. https://www.immunityinc.com/products/canvas/, 2019.

[32] J. Jacobs, S. Romanosky, I. Adjerid, and W. Baker. Im-proving vulnerability remediation through better exploitprediction. In The 2019 Workshop on the Economics ofInformation Security (WEIS), Jun 2019.

[33] J. Jacobs, S. Romanosky, B. Edwards, I. Adjerid, andM. Roytman. Exploit prediction scoring system (epss).Digital Threats: Research and Practice, 2(3), July 2021.

[34] D. Landman, A. Serebrenik, E. Bouwers, and J. J. Vinju.Empirical analysis of the relationship between cc andsloc in a large corpus of java methods and c functions.Journal of Software: Evolution and Process, 28(7):589–618, 2016.

[35] Q. Le and T. Mikolov. Distributed representations ofsentences and documents. In International conferenceon machine learning, pages 1188–1196. PMLR, 2014.

[36] F. Li and V. Paxson. A large-scale empirical studyof security patches. In Proceedings of the 2017 ACMSIGSAC Conference on Computer and CommunicationsSecurity, pages 2201–2215, 2017.

[37] Y. Liu, A. Sarabi, J. Zhang, P. Naghizadeh, M. Karir,M. Bailey, and M. Liu. Cloudy with a chance of breach:Forecasting cyber security incidents. In 24th USENIXSecurity Symposium (USENIX Security 15), pages 1009–1024, 2015.

[38] D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545, 2016.

[39] G. S. Mann and A. McCallum. Simple, robust, scalablesemi-supervised learning via expectation regularization.In Proceedings of the 24th international conference onMachine learning, pages 593–600, 2007.

[40] M. Melis, D. Maiorca, B. Biggio, G. Giacinto, andF. Roli. Explaining black-box android malware de-tection. In 2018 26th European Signal ProcessingConference (EUSIPCO), pages 524–528. IEEE, 2018.

[41] A. K. Menon, B. Van Rooyen, and N. Natarajan. Learn-ing from binary labels with instance-dependent corrup-tion. arXiv preprint arXiv:1605.00751, 2016.

[42] MetaCPAN. Compiler::parser. https://metacpan.org/pod/Compiler::Parser, 2020.

[43] Mila Parkour. Contagio dump. http://contagiodump.blogspot.com/2010/06/overview-of-exploit-packs-update.html, 2019.

[44] MITRE CVE. Cve-2017-0144. https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-0144,2019.

[45] D. Mu, A. Cuevas, L. Yang, H. Hu, X. Xing, B. Mao,and G. Wang. Understanding the reproducibilityof crowd-reported security vulnerabilities. In 27thUSENIX Security Symposium (USENIX Security 18),pages 919–936, Baltimore, MD, Aug. 2018. USENIXAssociation.

20

[46] A. Nappa, R. Johnson, L. Bilge, J. Caballero, andT. Dumitras, . The attack of the clones: A study ofthe impact of shared code on vulnerability patching. InS&P, 2015.

[47] A complete guide to the common vulnerability scoringsystem. https://www.first.org/cvss/v3.0/specification-document.

[48] National vulnerability database. http://nvd.nist.gov/.

[49] Alienvault otx. AlienVault, 30 March 2009. https://otx.alienvault.com/.

[50] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, andL. Qu. Making deep neural networks robust to labelnoise: A loss correction approach. In Proceedings ofthe IEEE Conference on Computer Vision and PatternRecognition, pages 1944–1952, 2017.

[51] F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, andL. Cavallaro. {TESSERACT}: Eliminating experimen-tal bias in malware classification across space and time.In 28th {USENIX} Security Symposium ({USENIX} Se-curity 19), pages 729–746, 2019.

[52] Massive microsoft patch tuesday security update formarch. Qualys, 30 March 2017. https://blog.qualys.com/laws-of-vulnerabilities/2017/03/14/massive-security-update-from-microsoft-for-march.

[53] Microsoft resumes security updates with ’largest’ patchtuesday release. Redmont Mag, 30 March 2017. https://redmondmag.com/articles/2017/03/14/march-2017-security-updates.aspx.

[54] Python. ast. https://docs.python.org/2/library/ast.html, 2020.

[55] Rapid7. The metasploit framework. https://www.metasploit.com/, 2019.

[56] Severity ratings. RedHat, 30 March 2009. https://access.redhat.com/security/updates/classification/.

[57] Microsoft correctly predicts reliable exploits just 27Reuters, 30 March 2009. https://www.reuters.com/article/urnidgns852573c400693880002576630073ead6/microsoft-correctly-predicts-reliable-exploits-just-27-of-the-time-idUS186777206820091104.

[58] First probabilistic cyber risk model launched by RMS.https://www.artemis.bm/news/first-probabilistic-cyber-risk-model-launched-by-rms/.

[59] Ruby. Ripper. https://ruby-doc.org/stdlib-2.5.1/libdoc/ripper/rdoc/Ripper.html, 2020.

[60] C. Sabottke, O. Suciu, and T. Dumitras, . Vulnerabilitydisclosure in the age of social media: Exploiting Twitterfor predicting real-world exploits. In USENIX SecuritySymposium, Washington, DC, Aug 2015.

[61] R. B. Security. Cvssv3: New system, new problems(file-based attacks). https://www.riskbasedsecurity.com/2017/01/16/cvssv3-new-system-new-problems-file-based-attacks/.

[62] M. Shahzad, M. Z. Shafiq, and A. X. Liu. A large scaleexploratory analysis of software vulnerability life cycles.In Proceedings of the 2012 International Conference onSoftware Engineering, 2012.

[63] SkyBox. Vulnerability center. https://www.vulnerabilitycenter.com/#home, 2019.

[64] K. Soska and N. Christin. Automatically detectingvulnerable websites before they turn malicious. InProceedings of the 23rd USENIX Security Symposium,San Diego, CA, USA, August 20-22, 2014., pages 625–640, 2014.

[65] Symantec Corporation. Symantec threat explorer. https://www.symantec.com/security-center/a-z,2019.

[66] L. Szekeres, M. Payer, T. Wei, and D. Song. Sok:Eternal war in memory. In 2013 IEEE Symposium onSecurity and Privacy, pages 48–62. IEEE, 2013.

[67] Player 3 has entered the game: say hello to ’wannacry’.https://blog.talosintelligence.com/2017/05/wannacry.html.

[68] N. Tavabi, P. Goyal, M. Almukaynizi, P. Shakarian, andK. Lerman. Darkembed: Exploit prediction with neurallanguage models. In AAAI, 2018.

[69] Tenable. Tenable research advisories. https://www.tenable.com/security/research, 2019.

[70] Tenable Network Security. Nessus vulnerability scanner.http://www.tenable.com/products/nessus.

[71] Severity vs. vpr. Tenable, 30 March 2019. https://docs.tenable.com/tenablesc/Content/RiskMetrics.htm.

[72] Twitter. Filtered stream. https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/introduction.

[73] Virustotal. Virus total. www.virustotal.com.

21

[74] Vulners. Vulners vulnerability database. https://vulners.com/.

[75] Y. Watanabe. Assessing security risk of your containerswith vulnerability advisor. IBM, 30 March 2019. https://medium.com/ibm-cloud/assessing-security-risk-of-your-containers-with-vulnerability-advisor-f6e45fff82ef.

[76] W. Wu, Y. Chen, J. Xu, X. Xing, X. Gong, and W. Zou.{FUZE}: Towards facilitating exploit generation forkernel use-after-free vulnerabilities. In 27th USENIXSecurity Symposium (USENIX Security 18), pages 781–797, 2018.

[77] C. Xiao, A. Sarabi, Y. Liu, B. Li, M. Liu, and T. Dumi-tras. From patching delays to infection symptoms: usingrisk profiles for an early discovery of vulnerabilities ex-ploited in the wild. In 27th USENIX Security Symposium(USENIX Security 18), pages 903–918, 2018.

[78] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learn-ing from massive noisy labeled data for image classi-fication. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 2691–2699, 2015.

[79] F. Yamaguchi, N. Golde, D. Arp, and K. Rieck. Model-ing and discovering vulnerabilities with code propertygraphs. In 2014 IEEE Symposium on Security andPrivacy, pages 590–604. IEEE, 2014.

[80] ZDNet. Recent windows alpc zero-day has beenexploited in the wild for almost a week. https://www.zdnet.com/article/recent-windows-alpc-zero-day-has-been-exploited-in-the-wild-for-almost-a-week/, 2018.

A Appendix

EE (AUC = 0.84)

Baseline: EPSS (AUC = 0.71)

Baseline: Social Media Clf (AUC = 0.85)

0 0.2 0.4 0.6 0.8 1

False Positive Rate

0

0.2

0.4

0.6

0.8

1

Tru

e P

ositi

ve

Ra

te

(a)

NVD Info (AUC = 0.79)

Handcrafted (EPSS) (AUC = 0.76)

Social Media (AUC = 0.83)

Write-Ups (AUC = 0.85)

PoC Info (AUC = 0.86)

PoC Code (AUC = 0.82)

0 0.2 0.4 0.6 0.8 1

False Positive Rate

0

0.2

0.4

0.6

0.8

1

Tru

e P

ositi

ve

Ra

te

(b)

Figure 15: ROC curves for the corresponding precision-recallcurves in Figure 7.

Component: NVD & Write-Ups (AUC = 0.84)

Component: Poc Info & Code (AUC = 0.87)

All: EE (AUC = 0.91)

0 0.2 0.4 0.6 0.8 1

False Positive Rate

0

0.2

0.4

0.6

0.8

1

Tru

e P

ositi

ve

Ra

te

Figure 16: ROC curve for the corresponding precision-recallcurves in Figure 8a.

EE at disclosure (AUC = 0.87)

EE after 10 days (AUC = 0.89)

EE after 30 days (AUC = 0.90)

EE after 365 days (AUC = 0.92)

0 0.2 0.4 0.6 0.8 1

Recall

0.5

0.6

0.7

0.8

0.9

1

Precision

(a)

EE at disclosure (AUC = 0.85)

EE after 10 days (AUC = 0.88)

EE after 30 days (AUC = 0.88)

EE after 365 days (AUC = 0.91)

0 0.2 0.4 0.6 0.8 1

False Positive Rate

0

0.2

0.4

0.6

0.8

1

Tru

e P

ositi

ve

Ra

te

(b)

Figure 17: Performance of EE evaluated at different points intime.

A.1 Evaluation

Additional ROC Curves. Figures 15 and 16 highlight thetrade-offs between true positives and false positives in classi-fication.EE performance improves over time. To observe how ourclassifier performs over time, in Figure 17 we plot the perfor-mance when EE is computed at disclosure, then 10, 30 and365 days later. We observe that the highest performance boosthappens within the first 10 days after disclosure, where theAUC increases from 0.87 to 0.89. Overall, we observe thatthe performance gains are not as large later on: the AUC at30 days being within 0.02 points of that at 365 days. Thissuggests that the artifacts published within the first days af-ter disclosure have the highest predictive utility, and that thepredictions made by EE close to disclosure can be trusted todeliver a high performance.

B Artifact Appendix

B.1 AbstractWe developed a Web platform and an API client that allowusers to retrieve the Expected Exploitability (EE) scores pre-dicted by our system. The system gets updated daily with thenewest scores.

We implemented an API client in python, distributed via

22

Jupyter notebooks in a Docker container, which allows usersto interact with the API and download the EE scores to re-produce the main result from the paper, in Figure 7a andFigure 15a, or explore the performance of the latest modeland compare it to the performance of the models from thepaper.

B.2 Artifact check-list (meta-information)• Program: Docker (tested on v20.10.8)• Run-time environment: UNIX-like system• Metrics: Precision, Recall, Precision-Recall AUC, TPR, FPR,

AUC.• Output: Plots from the paper, reproducing and expanding

Figure 7a and Figure 15a.• Experiments: Running Jupyter notebooks.• How much disk space required (approximately)?: 4GB• How much time is needed to prepare workflow (approxi-

mately)?: 15 min• How much time is needed to complete experiments (ap-

proximately)?: 30 min• Publicly available?: The website and client code are publicly

available. The API requires an API token which we provideupon request.

B.3 Description

B.3.1 Web Platform

The Web platform exposes the scores of the most recent model, andoffers two tools for practitioners to integrate EE in vulnerability orrisk management workflows.

The Vulnerability Explorer tool allows users to search and inves-tigate basic characteristics of any vulnerability on our platform, thehistorical scores for that vulnerability as well as a sample of theartifacts used in computing its EE. One use-case for this tool is theinvestigation of critical vulnerabilities, as discussed in Section 7.3 -EE for critical vulnerabilities.

The Score Comparison tool allows users to compare the scoresacross subsets of vulnerabilities of interest. Vulnerabilities can befiltered based on the publication date, type, targeted product oraffected vendor. The results are displayed in a tabular form, whereusers can rank vulnerabilities according to various criteria of interest(e.g., the latest or maximum EE score, the score percentile amongselected vulnerabilities, whether an exploit was observed etc.). Oneuse-case for the tool is the discovery of critical vulnerabilities thatneed to be prioritized soon or for which exploitation is imminent, asdiscussed in Section 7.3 - EE for emergency response.

B.3.2 API Client

The API allows clients to download historical scores for a givenvulnerability (using the /scores/cveid endpoint), or all the pre-diction scores for a particular model on a particular date (using the/scores/daily endpoint). The API documentation describes theendpoints and the parameters required for each call, and providesexample code for clients to interact with the API.

B.4 How to accessThe Web platform is available at https://exploitability.app/. The APIand the client code are available at https://api.exploitability.app/.

B.5 InstallationThe practitioner tools are available on the Web platform. To use theAPI, clone the code repository, point a terminal to that folder andrun bash docker/run.sh. This will create the Docker containerand spawn a Jupyter server. Use the URL displayed in the consoleto open a browser session within that container.

B.6 Evaluation and expected resultsTo reproduce the results from the paper, open and run the fol-lowing notebook: reproducibility_plot_performance.ipynbIn an API key is provided, this will download the requiredscores used in the paper, cache them in various files inscores_reproducibility_download/, and use these files to com-pute the performance of EE and baselines. The output consists of 2figures, which correspond to Figure 15a and Figure 7a in our paper.

To evaluate the latest model, open and run the followingnotebook: latest_plot_performance.ipynb In an API key isprovided, the notebook will download all the scores producedby our latest model on 2021-10-10, cache them into a file inscores_latest_download/, and use this file to compute the perfor-mance of EE. The output consists of 2 figures which are comparableto Figure 15a and Figure 7a in our paper.

As of December 2021, we observe that the performance of ourlatest model predicting EE, computed before 2021-10-10, is veryclose to the performance reported in the paper (0.69 PR AUC / 0.96AUC for latest model vs 0.73 PR AUC / 0.84 AUC in the paper),demonstrating that our online predictor is functional and in line withthe claims from the paper. The performance of the latest model onother dates can be computed by changing the SCORES_DATE variableand re-running the notebook.

B.7 Experiment customizationWhen running latest_plot_performance.ipynb, users can cus-tomize the SCORES_DATE variable in the notebook to observe theperformance of our model on different dates.

23