Blending the Automated and the Manual:

Making Application Vulnerability

Management Your Ally DevOpsConnect | San Francisco | 2015

Who We Are

! Kris Curylo | Application Security Manager at Ally Financial

! Dan Cornell | CTO at Denim Group

Introduction

! Application security programs and in particular, application testing has traditionally been a fairly slow and manual process.

! Development teams are moving faster through the implementation of DevOps processes.

! We need to keep up.

Why I’m The One Talking To You

!  I have spent the past 2 years building the application security program at Ally Financial

!  I inherited a pile of tools and a few (unclear) requirements

!  I was told to “Make it work, make it work better, make it provide value rather than just check the box”

! Oh, and make sure you do it with existing resources and budget.

!  I’m guessing if you’re here, you’re probably in a similar position.

How I Got Started

! Take Inventory (of EVERYTHING)

!  Applications

!  Processes

!  Tools

!  Requirements

!  Complaints

! Organize

! Plan

Pain Points

! Too many “things” ! Too many tools

! Too many processes

! Too many interfaces for data

! Too many report formats

! Redundant decisioning

! This all leads to the biggest complaints: ! Everything takes too long and is inconsistent

Automate and Consolidate

! Need fewer manual processes

!  Managing requirements

!  Running scans

!  Handling data

!  But…can’t have no manual processes

! Need better view into data

!  Single TODO list of vulnerabilities to address

!  Slice and dice

Great…What Do I Do Now?

!  I used SharePoint:

! Created my own application inventory

! Created test tracking process

! Automated “compliance calculation”

! Exposed it to stakeholders

! This reduced complexity and allowed stakeholders to make informed decisions and prioritize security requirements with other business objectives.

What About Vulnerability Management?

!  We use lots of vendors & tools: !  HP WebInspect (DAST)

!  Veracode (SAST)

!  Trustwave/Cenzic Hailstorm (DAST)

!  BurpSuite (DAST)

!  OWASP Zap (DAST)

!  HP Quality Center (Defect tracking)

!  Leads to passing reports around or sending people to various interfaces

Communication Patterns

! “Here’s a 300 page PDF with a color graph on the front page”

! “Here’s another, different, 300 page PDF with a different color graph on the front page”

Automate and Consolidate

Security Services Request

Security Orchestration

Manual Assessment

3rd Party Manual Assessment

Testing Tools & Services

AppSec False Positive Analysis

Defect Tracker

Reporting & Metrics

Developer Remediation IDE

ThreadFix Background

!  Application vulnerability management platform

!  ThreadFix allows teams to:

!  Create a consolidated view of your applications and vulnerabilities

!  Prioritize application risk decisions based on data

!  Translate vulnerabilities to developers in the tools they are already using

!  Extensive REST API for automation

!  Allow application security teams to focus on high-value activities

!  Open Source ThreadFix Community Edition: !  https://github.com/denimgroup/threadfix

!  http://www.threadfix.org/

APIs Are the “Key”

! Today, we specifically require any new tool or process to integrate with ThreadFix to be considered for use in the program

! We have worked through every testing tool we have to identify APIs and individually review them for adding automation to the process.

No API? No Problem...

!  ThreadFix's RESTful API allows us to write our own automation

!  Using SharePoint and standard naming conventions to upload test results via workflow

!  Create cron jobs to batch upload

Automate and Consolidate – Next Steps

Security Services Request

Security Orchestration

Manual Assessment

3rd Party Manual Assessment

Testing Tools & Services

AppSec False Positive Analysis

Defect Tracker

Reporting & Metrics

Developer Remediation IDE

Web Application Firewall

Training Plans

Build Servers

Attack Surface Seeding

Can’t Escape the Manual

!  External test results from manual efforts are now tracked along side our own test results !  For ASPs and external vendors, we can require

them to submit their own test results to us !  Standardized submissions have allowed us to gain

better insight to 3rd party security posture

Bring Everything Together

! Using ThreadFix, we: ! Give our management, development and support

teams one interface

! Expose the data that matters to the proper people

! Retain proper tracking of vulnerability meta data and decisioning

! Reduce overall complexity while increasing value and agility (pun intended...)

! Pull results from testing tools as they become available

Speak to the Developers (In Their Own Language)

!  HP Quality Center APIs allow us to push defects directly into the defect tracker from ThreadFix

!  ThreadFix then pulls info back when the developers update the defect records

!  Eclipse API shows results in the IDE along side the code

Unplanned Advantages

!  With all data residing in one spot, we can identify trends !  What training should we offer to developers?

!  When training was conducted, did it help?

!  Are certain teams, languages, business units better or worse at specific things?

!  Do we have an opportunity to develop a pattern to address certain flaws?

!  Most complete view of application security posture we have ever had to enable better decision making of risk and priorities

We Found Lots of Places to Introduce Automation:

! Static testing execution

! Dynamic testing execution

! Results review

! Result tracking

! Compliance tracking

! Metrics

Advice From the Field

!  Don’t let perfect be the enemy of good

!  Small victories and incremental progress will keep your efforts in front of management and dev teams

!  Tackle a crowd pleaser early on

!  If you address the loudest critic quick, you will gain credibility and will be more apt to get help implementing automation

!  Build it and they will come

! Get one build server integrated

! Get one application team using ThreadFix alone for all decisioning.

! Get one team to publish defects into your bug tracker through ThreadFix

Lessons Learned

The Good

!  Developers want to write good code. They will use the tools made available if they are not too intrusive

!  Building in automation allows us identify trends and systemic opportunity for improvement regardless of developer participation

!  There are more opportunities for automation than expected

Lessons Learned

The Bad !  Retrofitting an existing program is painful !  No matter how much you automate, it will never

be enough !  We learned some scary things about our

environment. !  Expect to be overwhelmed.

Where We Go Next

Push automation further:

!  Integrate further with build servers

!  �Virtual Patching� via WAF rules

!  Automate �sanity check� scans through attack surface mapping and API to dynamic tools

!  Targeted training based on flaws present in applications

Questions / Contact

Kris Curylo

! Kristopher.Curylo@ally.com

Dan Cornell

! dan@denimgroup.com

! @danielcornell