BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP...
Transcript of BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP...
![Page 1: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/1.jpg)
BLAG: Improving the Accuracy of Blacklists
Sivaram Ramanathan1, Jelena Mirkovic1 and Minlan Yu2
1 University of Southern California/Information Sciences Institute2 Harvard University
![Page 2: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/2.jpg)
IP Blacklists
• IP Blacklists contain a list of known malicious IP addresses.• IP Blacklists are commonly used to
aid more sophisticated defenses such as spam filters, IDS, etc.
• IP blacklists can be used as an emergency response under a novel or large volumetric attack• Easy to implement as only IP
addresses are checked and can be done at line rate.
2
![Page 3: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/3.jpg)
Problems with IP Blacklists
3
• Focus only on specific attack types with limited vantage points.
Problems
Fragmentedinformation
![Page 4: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/4.jpg)
Problems with IP Blacklists
4
• Focus only on specific attack types with limited vantage points.• Historical blacklist data can capture reoffending malicious addresses.
Problems
Fragmentedinformation
Snapshots intime
![Page 5: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/5.jpg)
Problems with IP Blacklists
5
• Focus only on specific attack types with limited vantage points.• Historical blacklist data can capture reoffending malicious addresses.• Addresses are added only after a malicious event is observed.
Problems
Fragmentedinformation
Snapshots intime Reactive
![Page 6: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/6.jpg)
Problems with IP Blacklists
6
• Focus only on specific attack types with limited vantage points• Historical blacklist data can capture reoffending malicious addresses• Addresses are added only after a malicious event is observed
Problems
Fragementedinformation
Snapshots intime Reactive
Can we aggregate blacklists in a smart way to address these problems?
![Page 7: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/7.jpg)
Fragmented Information
Blacklists miss many attacks1,2 and may monitor only specific a type of attack.
7
[1] Kührer, Marc, Christian Rossow, and Thorsten Holz. "Paint it black: Evaluating the effectiveness of malware blacklists." International Workshop on Recent Advances in Intrusion Detection. Springer, Cham, 2014.[2] Pitsillidis, Andreas, et al. "Taster's choice: a comparative analysis of spam feeds." Proceedings of the 2012 Internet Measurement Conference. ACM, 2012.
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
- offenders in one given attack
![Page 8: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/8.jpg)
Fragmented Information
8
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
- offenders in one given attack
Blacklists miss many attacks1,2 and may monitor only specific a type of attack.[1] Kührer, Marc, Christian Rossow, and Thorsten Holz. "Paint it black: Evaluating the effectiveness of malware blacklists." International Workshop on Recent Advances in Intrusion Detection. Springer, Cham, 2014.[2] Pitsillidis, Andreas, et al. "Taster's choice: a comparative analysis of spam feeds." Proceedings of the 2012 Internet Measurement Conference. ACM, 2012.
![Page 9: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/9.jpg)
Fragmented Information
9
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
- offenders in one given attack
Compromised machines are constantly re-used for initiating different types of attacks over time.
![Page 10: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/10.jpg)
Fragmented Information
10
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
Compromised machines are constantly re-used for initiating different types of attacks over time. A Possible solution: Combining different types of blacklists can improve attack coverage.
- offenders in one given attack
![Page 11: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/11.jpg)
Snapshots in Time
11
1 Day 1 Month 3 Months 6 Months
- offenders in one given attack
Historical blacklist data (union of all offenders over time) can further be useful to improve offender detection.
![Page 12: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/12.jpg)
Snapshots in Time
12
1 Day 1 Month 3 Months 6 Months
- offenders in one given attack
Historical blacklist data (union of all offenders over time) can further be useful to improve offender detection.
![Page 13: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/13.jpg)
Snapshots in Time
13
1 Day 1 Month 3 Months 6 Months
- offenders in one given attack
Historical blacklist data (union of all offenders over time) can further be useful to improve offender detection.
![Page 14: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/14.jpg)
Careful Aggregation
14
Blacklists accuracy varies spatially• Blacklists are maintained by individuals or organizations that use
proprietary algorithms to include or exclude an address.• Blacklists could list some legitimate addresses
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
- offenders in one given attack
- legitimate clients of a given network during the same attack
![Page 15: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/15.jpg)
Careful Aggregation
15
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
- offenders in one given attack
- legitimate clients of a given network during the same attack
Combining blacklists can potentially amplify the number of misclassifications.
![Page 16: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/16.jpg)
Careful Aggregation
16
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
Historic
al
Historic
al
Historic
al
- offenders in one given attack
- legitimate clients of a given network during the same attack
Combining blacklists can further potentially amplify the number of misclassifications.
![Page 17: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/17.jpg)
Many misclassifications across different testing scenarios!
Careful Aggregation
Combining historical blacklists can further potentially amplify the number of false positives
17
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
Historic
al
Historic
al
Historic
al
- offenders in one given attack
- legitimate clients of a given network during the same attack
Goal: Aggregate historical blacklists and reduce misclassifications.
![Page 18: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/18.jpg)
Blacklists are Reactive
18
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
- offenders in one given attack
Addresses are usually listed after an attack takes place, cannot be used for prevention.
![Page 19: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/19.jpg)
Blacklists are Reactive
19
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
- offenders in one given attack
Addresses are usually listed after an attack takes place, cannot be used for prevention.Possible solution: we could list groups of addresses in the same subnet (IP prefixes), hoping to capture future attackers - expansion1.
[1] Zhang, Jing, et al. "On the Mismanagement and Maliciousness of Networks." NDSS. 2014.
![Page 20: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/20.jpg)
Careful Expansion
20
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
Historic
al
Historic
al
Historic
al
- offenders in one given attack
- legitimate clients of a given network during the same attack
Expansion can further amplify misclassifications!
![Page 21: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/21.jpg)
Historic
al
SpamBlacklist
DDoS Blacklist
Malware Blacklist
Combined Blacklist
Expansion can further amplify misclassificationsWe need a better technique to combine blacklists efficiently and select some addresses to be expanded into prefixes.
Careful Expansion
21
Historic
al
Historic
al
- offenders in one given attack
- legitimate clients of a given network during the same attack
Goal: Expand some addresses into prefixes that do not cause more misclassifications.
![Page 22: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/22.jpg)
Outline
• Introduction• Quantifying problems faced by blacklists• BLAG• Datasets• Evaluation• Summary
22
![Page 23: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/23.jpg)
How BLAG Works
23
Aggregation
....
![Page 24: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/24.jpg)
How BLAG Works
24
Aggregation
157 Blacklists
....
![Page 25: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/25.jpg)
How BLAG Works
25
Aggregation Estimatemisclassification
157 Blacklists
....
![Page 26: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/26.jpg)
How BLAG Works
26
Aggregation Estimatemisclassification
Sample inbound trafficfor a network
157 Blacklists
....
![Page 27: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/27.jpg)
How BLAG Works
27
Aggregation Estimatemisclassification
Sample inbound trafficfor a network
RecommendationSystem
157 Blacklists
....
![Page 28: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/28.jpg)
How BLAG Works
28
Aggregation Estimatemisclassification
SelectiveExpansion
Sample inbound trafficfor a network
RecommendationSystem
157 Blacklists
....
![Page 29: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/29.jpg)
Aggregation of Blacklists
• Historical blacklist data can be useful.• However, including addresses reported way back in the past can
increase the misclassifications.• PRESTA1 showed that recently listed addresses have a higher
tendency to be malicious than older ones.• BLAG uses the same metric as that of PRESTA to assign a relevance
score, based on when the address was listed in a blacklist• Recently listed addresses have a higher score.
29[1] West, Andrew G., et al. "Spam mitigation using spatio-temporal reputations from blacklist history." Proceedings of the 26th Annual Computer Security Applications Conference. ACM, 2010.
![Page 30: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/30.jpg)
Aggregation of Blacklists: Relevance Scores
• For address a listed in blacklist b,
𝑟",$ = 2'()*+'
,
30
![Page 31: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/31.jpg)
Aggregation of Blacklists: Relevance Scores
• For address a listed in blacklist b,
𝑟",$ = 2'()*+'
,
Where,• t is the current time
31
![Page 32: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/32.jpg)
Aggregation of Blacklists: Relevance Scores
• For address a listed in blacklist b,
𝑟",$ = 2'()*+'
,
Where,• t is the current time• tout is the last time when an address a was listed in blacklist b
32
![Page 33: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/33.jpg)
Aggregation of Blacklists: Relevance Scores
• For address a listed in blacklist b,
𝑟",$ = 2'()*+'
,
Where,• t is the current time• tout is the last time when an address a was listed in blacklist b• l is constant, which ensures that the score decays over time
33
![Page 34: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/34.jpg)
• For address a listed in blacklist b,
𝑟",$ = 2,
'+'()*
Where,• t is the current time• tout is the last time when address a was listed in blacklist b• l is constant, which ensures that the score decays exponentially over
time
Aggregation of Blacklists: Relevance Scores
34
A high relevance score means that an IP has been recently listed and has a higher tendency of being malicious.
![Page 35: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/35.jpg)
Estimate Misclassifications–Recommendation System
35
• Commonly found in popular services like Netflix, Amazon, and YouTube to improve user retention and increase revenue. • Recommend new items to users based on their or similar users’
previous ratings of similar items.
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
![Page 36: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/36.jpg)
Estimate Misclassifications–Recommendation System
36
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
![Page 37: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/37.jpg)
Estimate Misclassifications–Recommendation System
37
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
Likes green books.
![Page 38: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/38.jpg)
Estimate Misclassifications–Recommendation System
38
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
Likes green books.
Dislikes yellow books.
![Page 39: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/39.jpg)
Estimate Misclassifications–Recommendation System
39
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
?
![Page 40: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/40.jpg)
Estimate Misclassifications–Recommendation System
40
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
0.99 0.97 0.8 0.92
0.8 0.85 0.99 0.59
0.7 0.6 0.6 0.66
0.66 0.79 0.5 0.6
0.77 0.85 0.4 0.79
0.8
0.99
0.29
0.55
0.72
0.8 0.59 0.6 0.7 0.99
1
1
0.8
0.99
0.99
1
![Page 41: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/41.jpg)
Estimate Misclassifications–Recommendation System
41
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
0.99 0.97 0.8 0.92
0.8 0.85 0.99 0.59
0.7 0.6 0.6 0.66
0.66 0.79 0.5 0.6
0.77 0.85 0.4 0.79
0.8
0.99
0.29
0.55
0.72
0.8 0.59 0.6 0.7 0.99
1
1
0.8
0.99
0.99
1
![Page 42: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/42.jpg)
Estimate Misclassifications–Recommendation System
42
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
0.99 0.97 0.8 0.92
0.8 0.85 0.99 0.59
0.7 0.6 0.6 0.66
0.66 0.79 0.5 0.6
0.77 0.85 0.4 0.79
0.8
0.99
0.29
0.55
0.72
0.8 0.59 0.6 0.7 0.99
1
1
0.8
0.99
0.99
1
![Page 43: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/43.jpg)
Estimate Misclassifications–Recommendation System
43
1 0.8
0.8 1 0.6
0.6
0.8
0.4 0.8
0.8
1
0.8 0.6 1
0.8
1
1
0.99 0.97 0.8 0.92
0.8 0.85 0.99 0.59
0.7 0.6 0.6 0.66
0.66 0.79 0.5 0.6
0.77 0.85 0.4 0.79
0.8
0.99
0.29
0.55
0.72
0.8 0.59 0.6 0.7 0.99
1
1
0.8
0.99
0.99
1
![Page 44: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/44.jpg)
Estimate Misclassifications
44
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
.. .. 0.8 .. ..
• BLAG arranges IP addresses and blacklists in a matrix, where rows are addresses and columns are blacklists.• If an address a is listed in blacklist b, BLAG assigns the relevance score
ra,b to the cell.
![Page 45: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/45.jpg)
Estimate Misclassifications
45
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
.. .. 0.8 .. ..
BLAG uses legitimate traffic traces of a network to introduce a new blacklist called the Misclassification Blacklist (MB), which consists only of misclassifications.
![Page 46: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/46.jpg)
Estimate Misclassifications
46
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
1
1
1
.. .. 0.8 .. ..
For every known misclassification from the training data, BLAG allocates a score of 1.
![Page 47: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/47.jpg)
Estimate Misclassifications
47
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?
Goal: Find the relevance scores for remaining addresses in MB.
![Page 48: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/48.jpg)
Estimate Misclassifications
48
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?IP1
IP2
Goal: Find the relevance scores for remaining addresses in MB.
![Page 49: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/49.jpg)
Estimate Misclassifications
49
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?IP1
IP2
Goal: Find the relevance scores for remaining addresses in MB.
![Page 50: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/50.jpg)
Estimate Misclassifications
50
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?IP1
IP2
Goal: Find the relevance scores for remaining addresses in MB.
IP1
IP2
![Page 51: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/51.jpg)
Estimate Misclassifications
51
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ? Likely to be a misclassification!IP1
IP2
Goal: Find the relevance scores for remaining addresses in MB.
IP1
IP2
![Page 52: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/52.jpg)
Estimate Misclassifications
52
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
169.231.140.68
193.1.64.5
193.1.64.8
216.59.0.8243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem Prune
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?
Master blacklistcandidates
Using a defined threshold customized for every network (0.7 in this case), BLAG prune out addresses that are potentially misclassified.
![Page 53: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/53.jpg)
Why Recommendation System?
• Given the incomplete view of the address space, there are many addresses that cannot be determined to be a misclassification (or not).• Several latent factors influence an address to be a misclassification.• Proprietary algorithms historical data or overall reputation of the
blacklist• The recommendation system helps us identify other addresses:• Which “behave” similar to our known misclassifications.• They are listed on same or similar blacklists as our known
misclassifications, with similar scores.
53
![Page 54: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/54.jpg)
Selective Expansion
54
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
169.231.140.68
193.1.64.5
193.1.64.8
216.59.0.8243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem Prune
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?
Master blacklistcandidates
Check1
OK
OK
OK
OK
Check 1: If a prefix has any known misclassification, it is excluded from expansion.
![Page 55: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/55.jpg)
Selective Expansion
55
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
169.231.140.68
193.1.64.5
193.1.64.8
216.59.0.8243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem Prune
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?
Master blacklistcandidates
Check1
OK
OK
OK
Check2
!
OK
OK
OK OK
Check 2: If a prefix has any likely misclassification, it is excluded from expansion.
![Page 56: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/56.jpg)
Selective Expansion
56
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
169.231.140.68
193.1.64.5
193.1.64.8
216.59.0.8243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem Prune
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?
Master blacklistcandidates
Check1
OK
OK
OK
Check2
!
OK
OK
OK OK
Check 2: If a prefix has any likely misclassification, it is excluded from expansion.
![Page 57: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/57.jpg)
Selective Expansion
57
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
169.231.140.68
193.1.64.5
193.1.64.8
216.59.0.8243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem Prune
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?
Master blacklistcandidates
Check1
OK
OK
OK
Check2
!
OK
OK
OK OK
Check 2: If a prefix has any likely misclassification, it is excluded from expansion.
IP1
IP3
![Page 58: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/58.jpg)
Selective Expansion
58
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m
Blacklis
t 3
..
169.231.140.68
193.1.64.5
193.1.64.8
216.59.0.8243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
169.231.140.68
193.1.64.8
216.59.16.171
Blacklis
t 1
Blacklis
t 2
Blacklis
t m-1
Blacklis
t 3
..
243.13.0.23
MB
169.231.140.10
243.13.222.203
193.1.64.5
216.59.0.8
Recommendationsystem Prune
193.1.64.0/24
216.59.0.0/24
169.231.140.68Selectiveexpansion
0.28 0.11 .. ..
.. 0.46 .. ..
0.72 0.23 .. ..
.. .. 0.32 ..
.. 0.58 .. ..
0.15
..
0.25
0.95
0.87
.. .. .. .. ..
.. 0.79 0.87 .. 0.81
0.22
0.4
0.12
0.91
0.6
0.92
0.99
.. .. 0.78 .. .. 0.75
0.3 0.1 .. ..
.. 0.5 .. ..
0.7 0.5 .. ..
.. .. 0.04 ..
.. 0.7 .. ..
0.1
..
0.1
0.9
0.9
.. .. .. .. ..
.. 0.7 1 .. 0.9
?
?
?
1
?
1
1
.. .. 0.8 .. .. ?
Master blacklistcandidates
BLAG masterblacklist
Check1
OK
OK
OK
Check2
!
OK
OK
OK OK
BLAG expands addresses to their /24 prefix only when both conditions are satisfied.
![Page 59: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/59.jpg)
Outline
• Introduction• Quantifying problems faced by blacklists• BLAG• Datasets• Evaluation• Summary
59
![Page 60: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/60.jpg)
Monitored BlacklistsBlacklistdataset
Malware Reputation Spam Attack
57 blacklists
Emergingthreats
Malware bytes
Malware domainlist
Cisco talos
Binary defensesystems
32 blacklists
Alienvault Spamhaus
Nixspam
Cleantalk
39 blacklists
Snort labs
DShield
Maxmind
29 blacklists
• 157 blacklists monitored from Jan 2016 to Dec 2017 roughly categorized into four attack variants.• Collected over 176 million IP addresses during this period.
60
![Page 61: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/61.jpg)
Ground Truth for Evaluating Blacklists
• Three types of ground truth, each with its corresponding legitimate and attack dataset.• The legitimate portion is to
validate the false detections of blacklists. • The attack portion is to validate
the accurate detections of blacklists.
Legit emailsfrom IRB study
(6K)
Spam mails fromMailinator
(39K)
Legit requests touniversity server
(45K)
Mirai malwareinfected hosts
(390K)
Legit requestssent to B-root
(14K)
Attackers to B-root (5.5M)
Ground truth
Email DDoSUniv DDoSDNS
61
![Page 62: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/62.jpg)
Email Dataset
62
Training
June 1, 2016
June 7, 2016
![Page 63: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/63.jpg)
Email Dataset
63
Training
Knownmisclassifications
June 1, 2016
June 7, 2016
![Page 64: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/64.jpg)
Email Dataset
64
Training Validation
Knownmisclassifications
June 1, 2016
June 7, 2016
June 14, 2016
![Page 65: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/65.jpg)
Email Dataset
65
Training Validation
Knownmisclassifications Estimate threshold
June 1, 2016
June 7, 2016
June 14, 2016
![Page 66: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/66.jpg)
Email Dataset
66
Training Validation Testing
Knownmisclassifications Estimate threshold
June 1, 2016
June 7, 2016
June 14, 2016
June 30, 2016
![Page 67: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/67.jpg)
Email Dataset
67
Training Validation Testing
Known
misclassificationsEstimate threshold
June 1, 2
016
June 7, 2
016
June 14, 2
016
June 30, 2
016
Ham emails (IRB study)
3KHam emails (IRB study)
2KHam emails (IRB study)
4K
![Page 68: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/68.jpg)
Email Dataset
68
Training Validation Testing
Known
misclassificationsEstimate threshold
June 1, 2
016
June 7, 2
016
June 14, 2
016
June 30, 2
016
Ham emails (IRB study)
3KHam emails (IRB study)
2KHam emails (IRB study)
4K
Spam emails (Mailinator)
13KSpam emails (Mailinator)
26K
![Page 69: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/69.jpg)
Outline
• Introduction• Quantifying problems faced by blacklists• BLAG• Datasets• Evaluation• Summary
69
![Page 70: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/70.jpg)
Evaluation
• Accuracy of BLAG: Compare the performance of BLAG with competing approaches• Best: The best-performing blacklist on a given ground truth dataset (hindsight)
at the given time (of the ground truth dataset).• Historical: All addresses listed in all blacklists up until ground truth dataset.• PRESTA+L: Blacklisting approach taken by PRESTA algorithm that uses spatial
properties of blacklisted addresses to generate a new blacklist.• Metrics: • Specificity - the percentage of legitimate addresses that were not false
positives.• Recall - the percentage of offenders that were detected.
70
![Page 71: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/71.jpg)
BLAG is Accurate
Best blacklists have high specificity (>99%) but poor recall(< 4%) indicating that even the best blacklist is not enough to capture all attackers. 71
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Specifcity
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Recall
![Page 72: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/72.jpg)
BLAG is Accurate
72
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Specifcity
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Recall
Historical blacklists improve recall to 18% but with a drop in specificity by 12%, indicating that naïve combination of all blacklists has potential to capture attackers, but lowers specificity.
![Page 73: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/73.jpg)
BLAG is Accurate
73
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Specifcity
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Recall
BLAG with expansion further improves recall, with only a slight drop in specificity and has better specificity than historical blacklists.
![Page 74: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/74.jpg)
BLAG is Accurate
PRESTA+L has been tuned to have same recall as BLAG, but the specificity is lower than BLAG (82% vs 95%)
74
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Specifcity
0
20
40
60
80
100
Best Historical PRESTA+L BLAG
(%)
Recall
![Page 75: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/75.jpg)
Other evaluations
• Evaluated BLAG on two other datasets: DDoSUniv and DDoSDNS.• Other expansion techniques -- expand using BGP prefixes or by
autonomous systems.• Impact of • Number of blacklists• Size of misclassification blacklists
• Contribution of recommendation system in aggregation and expansion phase.• Parameter tuning techniques.
75
![Page 76: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/76.jpg)
Public datasets
• All monitored blacklists are available at:
https://steel.isi.edu/Projects/BLAG/
• Includes scripts to deploy BLAG in your network.
76
![Page 77: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/77.jpg)
Outline
• Introduction• Quantifying problems faced by blacklists• BLAG• Datasets• Evaluation• Summary
77
![Page 78: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/78.jpg)
Summary
• Blacklists have poor attack detection.• Combining blacklists from different sources improves attack detection,
but also increases misclassifications.• BLAG (Blacklist aggregator)
• Assigns relevance scores to addresses belonging to blacklists• Predicts addresses that are likely to be misclassifications using a
recommendation system • Expands selective addresses into prefixes for better attack detection
• BLAG has better performance than competing approaches such as PRESTA
78
![Page 79: BLAG: Improving the Accuracy of Blacklistsminlanyu.seas.harvard.edu/talk/ndss20.pdfProblems with IP Blacklists 5 •Focus only on specific attack types with limited vantage points.](https://reader033.fdocuments.us/reader033/viewer/2022050505/5f97380cc435ee4e9b39b8be/html5/thumbnails/79.jpg)
Thank You! Questions?
All monitored blacklists are available at:
https://steel.isi.edu/Projects/BLAG/
79