Blending Automated and Manual Testing
-
Upload
denim-group -
Category
Technology
-
view
961 -
download
0
Transcript of Blending Automated and Manual Testing
![Page 1: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/1.jpg)
Blending Automated and Manual Testing
Making Application Vulnerability Management Pay Dividends
![Page 2: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/2.jpg)
My Background• Dan Cornell, founder and CTO of Denim
Group
• Software developer by background (Java, .NET, etc)
• OWASP San Antonio@danielcornell
![Page 3: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/3.jpg)
My Background• Steve Springett, Application Security
Architect for Axway
• Software developer by background
• Leader of OWASP Dependency-‐Track
• Contributor to OWASP Dependency-‐Check@stevespringett
![Page 4: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/4.jpg)
Goal: Continuous Security
• Prerequisites– Standardization– Continuous Integration– Continuous Delivery
• Compliments– Continuous Acceptance
![Page 5: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/5.jpg)
Standardization
• All projects use same build system• All projects built the same way• Automated onboarding for new projects• Per-‐project build expertise not required
![Page 6: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/6.jpg)
MetricsArtifacts
Continuous Integration
Continuous Integration Factory
Source Code (SCM)
![Page 7: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/7.jpg)
Deliverables
Continuous Delivery
Continuous Delivery Factory
Artifacts
![Page 8: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/8.jpg)
Security Metrics
Continuous Security
Continuous Security Factory
Source Code (SCM) Deliverables
![Page 9: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/9.jpg)
Automated Security Metrics
• Static Analysis Findings• Dynamic Analysis Findings• Component Analysis Findings• Attack Surface Analysis Findings
![Page 10: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/10.jpg)
Continuous Security Pipe
Jenkins CI ThreadFix Defect TrackerSCM
False Positive
![Page 11: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/11.jpg)
TargetApplication
![Page 12: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/12.jpg)
12
ThreadFixAccelerate Software Remediation
ThreadFix is a software vulnerability aggregation and management system that helps organizations aggregate vulnerability data, generate virtual patches, and interact with software defect tracking systems.
![Page 13: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/13.jpg)
ThreadFix
• Open Source (MPL) application vulnerability management platform
• Create a consolidated view of your applications and vulnerabilities
• Prioritize application risk decisions based on data
• Translate vulnerabilities to developers in the tools they are already using
![Page 14: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/14.jpg)
ThreadFix Community Edition• Main ThreadFixwebsite: www.threadfix.org
– General information, downloads
• ThreadFix GitHub site: www.github.com/denimgroup/threadfix– Code, issue tracking
• ThreadFix GitHubwiki: https://github.com/denimgroup/threadfix/wiki– Project documentation
• ThreadFix Google Group: https://groups.google.com/forum/?fromgroups#!forum/threadfix– Community support, general discussion
![Page 15: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/15.jpg)
Vulnerability AggregationAutomated
Automated Manual
![Page 16: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/16.jpg)
Access to Vulnerability Data
• Tradeoffs– The more places the vulnerability data lives, the more likely a compromise
–Withholding information from people who need it makes remediation more challenging
![Page 17: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/17.jpg)
Managing All Vulnerability Data
• Manual activities– Penetration Testing– Code Reviews
• 3rd Party Data Sources– Customer-‐performed Testing– External auditor-‐performed Results
![Page 18: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/18.jpg)
SSVL and Manual Results
• SSVL Data Format:– https://github.com/owasp/ssvl
• SSVL Conversion Tool:– https://github.com/denimgroup/threadfix/wiki/SSVL-‐Converter
![Page 19: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/19.jpg)
RESTful API to Vulnerability Data
CustomR&D Monitoring
Dashboard
CustomDashboards
![Page 20: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/20.jpg)
Key Performance Indicators
• Don’t go overboard – Use only what is needed• Progress and velocity• Per team comparison• Min/max/avg time to close per severity• By CWE
![Page 21: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/21.jpg)
Lessons Learned
• Always automate static analysis• Always automate attack surface analysis• Always automate component analysis• Always automate dynamic analysis• Always perform manual dynamic analysis• Use native tools & workflow for static analysis
![Page 22: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/22.jpg)
Lessons Learned
• Provide as much visibility as possible– Varying degrees of detail– Multiple delivery vehicles
• Set clear pass/fail criteria for Security Bars– Provide custom dashboard to provide status and advanced warning
![Page 23: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/23.jpg)
Additional Advice
• Automation is not better than manual– It’s faster and more efficient– Both are necessary
• Don’t forget manual assessments– Threat Modeling– Secure Design/Architecture and Code Review– Penetration Testing
![Page 24: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/24.jpg)
Finally
• Vulnerabilities in CI / CD / CS Infrastructure– Threat Model– Secure Architecture Review– Patch Management – Configuration Management– Key Management– Always use TLS
![Page 25: Blending Automated and Manual Testing](https://reader031.fdocuments.us/reader031/viewer/2022030311/58eebde51a28abf3468b4577/html5/thumbnails/25.jpg)
Q & A