Blending Automated and Manual Testing

Making Application Vulnerability Management Pay Dividends

My Background

• Dan Cornell, founder and CTO of Denim Group

• Software developer by background (Java, .NET, etc)

• OWASP San Antonio@danielcornell

My Background

• Steve Springett, Application Security Architect for Axway

• Software developer by background

• Leader of OWASP Dependency-Track

• Contributor to OWASP Dependency-Check@stevespringett

Goal: Continuous Security

• Prerequisites– Standardization– Continuous Integration– Continuous Delivery

• Compliments– Continuous Acceptance

Standardization

• All projects use same build system• All projects built the same way• Automated onboarding for new projects• Per-project build expertise not required

MetricsArtifacts

Continuous Integration

Continuous Integration Factory

Source Code (SCM)

Deliverables

Continuous Delivery

Continuous Delivery Factory

Artifacts

Security Metrics

Continuous Security

Continuous Security Factory

Source Code (SCM) Deliverables

Automated Security Metrics

• Static Analysis Findings• Dynamic Analysis Findings• Component Analysis Findings• Attack Surface Analysis Findings

Continuous Security Pipe

Jenkins CI ThreadFix Defect TrackerSCM

False Positive

TargetApplication

12

ThreadFixAccelerate Software Remediation

ThreadFix is a software vulnerability aggregation and management system that helps organizations aggregate vulnerability data, generate virtual patches, and interact with software defect tracking systems.

ThreadFix

• Open Source (MPL) application vulnerability management platform

• Create a consolidated view of your applications and vulnerabilities

• Prioritize application risk decisions based on data

• Translate vulnerabilities to developers in the tools they are already using

ThreadFix Community Edition• Main ThreadFix website: www.threadfix.org

– General information, downloads

• ThreadFix GitHub site: www.github.com/denimgroup/threadfix – Code, issue tracking

• ThreadFix GitHub wiki: https://github.com/denimgroup/threadfix/wiki – Project documentation

• ThreadFix Google Group: https://groups.google.com/forum/?fromgroups#!forum/threadfix – Community support, general discussion

Vulnerability AggregationAutomated

Automated Manual

Access to Vulnerability Data

• Tradeoffs– The more places the vulnerability data lives, the

more likely a compromise– Withholding information from people who need it

makes remediation more challenging

Managing All Vulnerability Data

• Manual activities– Penetration Testing– Code Reviews

• 3rd Party Data Sources– Customer-performed Testing– External auditor-performed Results

SSVL and Manual Results

• SSVL Data Format:– https://github.com/owasp/ssvl

• SSVL Conversion Tool:– https://github.com/denimgroup/threadfix/wiki/SSVL-Converter

RESTful API to Vulnerability Data

CustomR&D Monitoring

Dashboard

CustomDashboards

Key Performance Indicators

• Don’t go overboard – Use only what is needed• Progress and velocity• Per team comparison• Min/max/avg time to close per severity• By CWE

Lessons Learned

• Always automate static analysis• Always automate attack surface analysis• Always automate component analysis• Always automate dynamic analysis• Always perform manual dynamic analysis• Use native tools & workflow for static analysis

Lessons Learned

• Provide as much visibility as possible– Varying degrees of detail– Multiple delivery vehicles

• Set clear pass/fail criteria for Security Bars– Provide custom dashboard to provide status and

advanced warning

Additional Advice

• Automation is not better than manual– It’s faster and more efficient– Both are necessary

• Don’t forget manual assessments– Threat Modeling– Secure Design/Architecture and Code Review– Penetration Testing

Finally

• Vulnerabilities in CI / CD / CS Infrastructure– Threat Model– Secure Architecture Review– Patch Management – Configuration Management– Key Management– Always use TLS

Q & A