Phinding Phish: An Evaluation of Anti-Phishing Toolbars
description
Transcript of Phinding Phish: An Evaluation of Anti-Phishing Toolbars
CMU Usable Privacy and SecurityLaboratory
http://cups.cs.cmu.edu/
Phinding Phish: An Evaluation of Anti-Phishing Toolbars
Yue Zhang, Serge Egelman, Lorrie Cranor, and Jason
Hong
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Anti-Phishing Tools 84 Listed on download.com (Sept. ‘06)
Included in many browsers
Poor usability•Many users don’t see indicators•Many choose to ignore them•But usability is being addressed
Are they accurate?
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Tools Tested CallingID
Cloudmark
EarthLink
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Tools Tested eBay
Firefox
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Tools Tested IE7
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Tools Tested Netcraft
Netscape
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Tools Tested SpoofGuard
TrustWatch
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Source of Phish High volume of fresh phish
• Sites taken down after a day on average• Fresh phish yield blacklist update information
Can’t use toolbar blacklists
We experimented with several sources• APWG - high volume but many duplicates and legitimate URLs included
• Phishtank.org - lower volume but easier to extract phish
• Assorted other phish archives - often low volume or not fresh enough
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Phishing Feeds Anti-Phishing Working Group•[email protected]•ISPs, individuals, etc.•>2,000 messages/day•Filtering out URLs from messages
PhishTank•http://www.phishtank.org/•Submitted by public•~48 messages/day•Manually verify URLs
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testbed for Anti-Phishing Toolbars Automated testing Aggregate performance statistics Key design issue:•Different browsers•Different toolbars•Different indicator types
Solution: Image analysis•Compare screenshots with known states
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Phish!!
Warning!!
Image-Based Comparisons Two examples: TrustWatch and Google TrustWatch:
Google:
ScreenShot
ScreenShot
VerifiedNot verified
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testbed System Architecture
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testbed System ArchitectureRetrieve Potential Phishing SitesRetrieve Potential Phishing Sites
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testbed System Architecture
Send URL to WorkersSend URL to Workers
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testbed System Architecture
Worker Evaluates Worker Evaluates Potential Phishing SitePotential Phishing Site
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testbed System Architecture
Task Manager Task Manager Aggregates ResultsAggregates Results
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Experiment Methodology Catch Rate: Given a set of phishing URLs, what percentage of
them are correctly labeled as phish by the tool - count block and warning only - taken down sites removed
False Positives: Given a set of legitimate URLs, what percentage of
them are incorrectly labeled as phish by the tool - count block and warning only - taken down sites removed
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Experiment 1 PhishTank feed used Equipment:
• 1 Notebook as Task Manager• 2 Notebooks as Workers
10 Tools Examined:• CloudMark• Earthlink• eBay• IE7• Google/Firefox• McAfee• Netcraft• Netscape• SpoofGuard• TrustWatch
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Experiment 1 100 phishing URLs
• PhishTank feed• Manually verified• Re-examined at 1, 2, 12, 24 hour intervals• Examined blacklist update rate (except w/SpoofGuard)
• Examined take-down rate
514 legitimate URLs• 416 from 3Sharp report• 35 from bank log-in pages• 35 from top pages by Alexa• 30 random pages
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Experiment 2 APWG phishing feed 9 of the same toolbars tested + CallingID
Same testing environment
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Results of Experiment 1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 12 24
cloudmark
earthlink
eBay
firefox w/google
ie7
mcafee
netcraft
netscape
spoofguard
trustwatch
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Results of Experiment 2
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 12 24
cloudmark
earthlink
eBay
firefox w/google
ie7
callingID
netcraft
netscape
spoofguard
trustwatch
firefox
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
False Positives
Toolbar False Positive
SpoofGuard 218 (42%)
CallingID 10 (2%)
Cloudmark 5 (1%)
EarthLink 5 (1%)
Not a big problem for most of the toolbars
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Overall findings No toolbar caught 100%
Good performers:•SpoofGuard (>90%)
Though 42% false positives•IE7 (70%-80%)•Netcraft (60%-80%)•Firefox (50%-80%)
Most performed poorly:•Netscape (10%-30%)•CallingID (20%-40%)
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
More findings Performance varied with feed
• Better with Phishtank: Cloudmark, Earthlink, Firefox, Netcraft
• Better with APWG: eBay, IE7, Netscape
• Almost the same: Spoofguard, Trustwatch
Different increases over time• More increases on APWG• Reflects the “freshness” of URLs
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
CDN Attack Many tools use blacklists
Many examine IP addresses (location, etc.)
Proxies distort URLs• Used Coral CDN• Append .nyud.net:8090 to URLs• Uses PlanetLab
Works on:• Cloudmark• Google• TrustWatch• Netcraft• Netscape
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Page Load Attack Some wait for page to be fully loaded•SpoofGuard•eBay
Insert a web bug taking infinite load time•5 lines of PHP•1x1 GIF•Infinite loop spitting out data very slowly
Tool stays in previous state
Unable to indicate anything
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Conclusion Tool Performance
• No toolbars are perfect• No single toolbar will outperform others• Heuristics have false positives
Whitelists? Hybrid approach?
Testing Methodology• Get fresher URLs• Test other than default settings
User interfaces• Usability is important
Traffic light? Pop up message? Re-direct page?
CMU Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/