Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry...

79
Splunk for Web Application Security Kyle Barry

Transcript of Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry...

Page 1: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Splunk for Web Application Security

Kyle Barry

Page 2: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 3: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Kyle BarrySecurity Engineering Manager

@allofmywats

Page 4: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 5: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

The world’s handmade marketplace

Page 6: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

30 million members

Page 7: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

$895 million in GMS

Page 8: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

30% of transactions are international

Page 9: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

1.4 billion page views a month

Page 11: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Vulnerabilities

Page 12: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Vulnerabilities

• Cross Site Scripting (XSS)

• Persistent vs. Non-persistent

Page 13: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Vulnerabilities

• Cross Site Scripting (XSS)

• Persistent vs. Non-persistent

• SQL Injection (SQLi)

Page 14: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Vulnerabilities

• Cross Site Scripting (XSS)

• Persistent vs. Non-persistent

• SQL Injection (SQLi)

• Authentication Bypass

Page 15: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Fraud

Page 16: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Fraud

• Account Takeover

Page 17: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Fraud

• Account Takeover

• Phishing

Page 18: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Fraud

• Account Takeover

• Phishing

• Scams

Page 19: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Getting it Done

Page 20: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

40+ Daily Deploys

Page 21: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

5 Minutes to Production

Page 22: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 23: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Config Flags

Page 24: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Continuous Integration

Page 25: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Splunk?

Page 26: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Yes!

Page 27: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Phishing Attackkaroart.etsy.com

Page 28: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 29: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

An Attack in 4 Parts

Page 30: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Step 1: Alerting

Page 31: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Alerting

• Cron Job?

Page 32: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Alerting

• Cron Job?

• Offline Processing?

Page 33: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Alerting

• Cron Job?

• Offline Processing?

• Real Time?

Page 34: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Alerting

• Cron Job?

• Offline Processing?

• Real Time?

• Splunk

Page 35: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

High Number of Logins from a Single IP

Page 36: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Alerting

• Info Log

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info][login] status="success" user="kbarry" ip="123.321.123.321"

• Splunk Search

source="/data/current/web/info.log" action="login" | transaction ip | sort -eventcount | table ip eventcount

Page 37: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Step 2: Damage Control

Page 38: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Damage Control

• Find Affected Accounts

• The Old Way*

awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ } END { for (i in ipcount) { printf "%15s - %d\n", i, ipcount[i] } }' logfile.log

• Splunk Search

source="/data/current/web/info.log" action="login" | transaction ip | sort -eventcount | table ip eventcount

* : The old way courtesy of http://serverfault.com/questions/11028

Page 39: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Step 3: Look for Trends

Page 40: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 41: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Step 4: Clean Up

Page 42: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Clean Up

• Find Affected Accounts

• The Old Way*

awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ } END { for (i in ipcount) { printf "%15s - %d\n", i, ipcount[i] } }' logfile.log

• Splunk Search

source="/data/current/web/info.log" action="login" | transaction ip | sort -eventcount | table ip eventcount

* : The old way courtesy of http://serverfault.com/questions/11028

Page 43: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Putting it Together

Page 44: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Security Mechanisms

Page 45: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Incident Response

Page 46: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Ad-hoc analysis of a large dataset

Driven by an event or incident

Needs to be fast

Page 47: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Reactive Security

Page 48: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Real-time event monitoring and alerting

Events that trigger immediate response

You always query the same data and you do it often

Page 49: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Proactive Security

Page 50: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Things we do now to protect us later

Actions taken to prevent future compromise

Page 51: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Incident Response

wildlifeprints.etsy.com

Page 52: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Phishing Attack

Page 53: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Scanners

Page 54: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Reactive Security

Page 55: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Alerting

• Failed Logins (Info Log)

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info][login] failed login user:"kbarry" ip:"123.321.123.321"

• Splunk Search

source="/data/current/web/info.log" log_namespace="login" | transaction user | sort -eventcount | table user eventcount

source="/data/current/web/info.log" log_namespace="login" | transaction ip | sort -eventcount | table ip eventcount

Page 56: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Alerting

• XSS Alert

source="/data/syslog/current/web/access.log" | regex request_uri="(%3Cscript%3E|alert\(|onerror)"

• SQLi Alert

source="/data/syslog/current/web/info.log" "database syntax error at"

Page 57: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 58: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 59: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Proactive Security

wildlifeprints.etsy.com

Page 60: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Goal

Full-site SSL for all Etsy sellers

Page 61: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 62: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Full Site SSL

• Which pages are important?

source="/data/syslog/current/web/access.log" is_seller=1 | transaction request_uri | table request_uri is_ssl

• How much traffic?

[insert query here]

Page 63: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 64: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Security Dashboards

Page 65: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 66: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 67: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million
Page 68: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Splunk Friendly Application

Page 69: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Make Logging Easy

Page 70: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

<?

$data = array('user' => 'kbarry', 'ip' => 123.321.123.321);

Logger::info('failed captcha', $data,'login');

?>

Page 71: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Make Splunking Easy

Page 72: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Access Log[Wed Apr 22 16:37:41 2013] "POST /signin HTTP/1.1" 200 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11" MF9JqDVpY93VOMreyvI2UC24wRjT

Info Log

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info] [login] failed captcha user:"kbarry" ip:"123.321.123.321"

Page 73: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Global Request ID

Page 74: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Access Log[Wed Apr 22 16:37:41 2013] "POST /signin HTTP/1.1" 200 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11" MF9JqDVpY93VOMreyvI2UC24wRjT

Info Log

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info] [login] failed captcha user:"kbarry" ip:"123.321.123.321"

Page 75: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Recap

Page 76: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Recap

• The More You Know (TM)

Page 77: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Recap

• The More You Know (TM)

• Security is a Real Time Problem

Page 78: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million

Recap

• The More You Know (TM)

• Security is a Real Time Problem

• Instrument your Application Appropriately

Page 79: Splunk for Web Application Security...Splunk for Web Application Security Kyle Barry Kyle Barry Security Engineering Manager @allofmywats The world’s handmade marketplace 30 million