How the Crowd Outperforms Traditional Security Testing

Sr. Security Engineer@leifdreizler

Your Elastic Security Team.

So What Does Bugcrowd Actually Do?• Incorporate up to 17,000 freelance security

researchers as part of a public or private engagement

• Run a crowd sourced pen test • Manage an ongoing bug bounty program

What’s a bug bounty program?

A Brief History of Bug Bounty Programs

These and other companies trust Bugcrowd

Things We’ll Cover

• How to incorporate Crowdsourced Security into DevOpsSec • Accelerating your RO(security)I • What’s in it for me (as a security person)? • Bug bounty fun facts, pitfalls, and war stories

introduce crowd sourcing

Bug Bounty Programs Responsible Disclosure

Crowdsourced Penetration Test

…because people are the new automation

[REDACTED] eCommerce provider

• Long time customer of [EXPENSIVE WEB APP SCANNER] getting “clean results”

• A Researcher gained super admin access through a chained attack within 24 hours of launch

• They thought they were doing a great job at writing secure code…

assume it’s broken

Instructure received 5-10x the number of unique vulnerabilities compared to previous pen tests

Case Study (Company A)

• Gone through previous security testing and remediation with a reputable webapp pentesting vendor

• Expecting low priority results

• 6 P1s

• 4 P2s

• ~30 P3/P4

Case Study (Company B)

• Building a new application

• Had internal security testing built into the SDLC

• 5 P1s

• 16 P2s

• ~30 P3/P4

Lots of bugs == great dev training

Software is always going to have bugs

[REDACTED] Financial Services

• Extortion attempt from Eastern Europe

• Resolved by creating a “one man bug bounty” (we didn’t tell him he was the only one though…)

• Bug received in 15 mins

History

0

125

250

375

500

1995 2000 2005 2010 2015

Adoption of bug bounty and vulnerability disclosure programs.

Bug bounties are awesome…

Minimize Investment

Maximize Quality

Accelerate RO(security)I

Makes a Statement

It’s not just about being cost-effective,

or loud…

It’s about leveling the playing field…

…but bug bounties are hard.

Plan ahead

The mistake *everyone* makes:

VULNERABILITY DATA

PEOPLE

[REDACTED] Digital Advertising

• Engaged Bugcrowd to help them assess the state of the code

• So many valid vulnerabilities submitted they shut down the bounty in 24 hours

• Thrilled with the results!

The Golden Rule:

Touch the code ==

Pay the bug

Align expectations before you engage

Bug bounties create controlled incidents…

[REDACTED] Online Marketplace

• The DevOps and Security teams watched vulns being submitted in real time

• Non-security minded people learned a lot from the process

• Great insight into how ‘good guys that think like bad guys’ work

Mozilla

Thanks to @mwcoates http://www.slideshare.net/michael_coates/bug-bounty-programs-for-the-web

Clearing their assurance debt

Boogeymanbelief

DevOpsSec feeling confident?

Try a Gamified Pentest

1. Create a pool that benefits your engineering team (team drinks, party, event, whatever)

2. Replace an existing pentest w/ a time-boxed bug bounty program

3. Pay out from the reward pool

4. What ever the hackers don’t get, DevOpsSec gets to keep.

Great things happen when you tighten the security feedback loop between your engineers, and what they consider to be

the outside world

Bugcrowd Stats• 28% US based, 28% based in India

• 90 countries have contributed

• Great Britain has low submission numbers, but high average priority

• 37k Total Submissions/6.3k Valid and Unique (17%)

• 16% of Valid Submissions are P1 or P2

• 54% of Paid Programs have at least one P1 or P2

• 93% of those Programs have 2+

• 18% XSS, 10% Logic Flaws, 9% CSRF, 6% Info Disclosure, 2% SQLi

Content Security Policy

Content Security Policy

• Designed to prevent XSS attacks

• unsafe-inline, unsafe-eval, script-src

• report-uri, and report-only mode

• http://c0nrad.io/blog/csp.html

• https://blog.matatall.com/

Highlights from the 2014 Facebook Report• Started in 2011

• Currently $500 minimum, no defined maximum

• 17,011 Submissions

• 61 Eligible bugs were high severity

• 123 Countries (65 Rewarded)

• $1.3 million paid to 321 researchers

Countries with High # of Valid SubsValid Bugs Average $

RewardIndia 196 $1,343Egypt 81 $1,220USA 61 $2,470UK 28 $2,768

Philippines 27 $1,093src: https://www.facebook.com/notes/facebook-bug-bounty/2014-highlights-bounties-get-better-than-ever/1026610350686524

Highlights from the 2014 Github Report

• First year of the program

• $200 - $5,000 (recently doubled upper end)

• 1,920 Submissions

• 73 Unique Vulnerabilities (57 medium/high)

• 33 Unique Researchers earned a total of $50,100 for the med/high vulnerabilities

src: https://github.com/blog/1951-github-security-bug-bounty-program-turns-one

Highlights from the 2014 Google Report

• Started in 2010

• Paid over 200 researchers over $1.5 mil

• $150k highest single payout

• Over 500 unique and valid bugs

• Over half of the bugs in Chrome were reported and fixed in beta or dev builds

src: http://googleonlinesecurity.blogspot.com/2015/01/security-reward-programs-year-in-review.html

Looking Forward with Microsoft in 2015• Started in 2013

• Recently added Azure and raised max payout for “Online Services Bounty Program” to 15k

• Added Project Spartan

• “Mitigation Bypass” bounty and “Bonus bounty for Defense” focus on novel methods to bypass active mitigations (e.g. ASLR and DEP)

• Pay up to $100k for exploit + $50k for defense

src: http://blogs.technet.com/b/msrc/archive/2015/04/22/microsoft-bounty-programs-expansion-azure-and-project-spartan.aspx

Conclusion• Bug bounties are cost effective, and highly marketable, but that’s not

the full story…

• …they create controlled incidents that can powerfully impact the security awareness of your builders.

• Allow people that have historically been ‘builders’ to see how ‘breakers’ think

• Get DevOps to believe in and defeat the boogeyman

The premier platform for crowdsourced security testing.

We’re hiring!

jobs@bugcrowd.com