DevOps at Capital OneFocusing on Pipeline and Measurement

@TopoPal

@TopoPal

Capital One

Millions of accounts One of the largest Digital Banks #1 Information Week’s Elite 100 ~ 20 years old

@TopoPal

Different DNA Build our own software Build on public cloud MicroServices Open Source DevOpsSec and Continuous Delivery

@TopoPal

• Enterprise Architecture• DevOpsSec Strategy Owner• DevOps Evangelist

• Shared Technology Group• Product Manager of Continuous

Delivery Tools Platform• DevOps Evangelist• Core Contributor and Community

Manager of Hygieia

Personal Journey

@TopoPal

@TopoPal

• Waterfall• Manual Build• Manual Deployment• Manual Test• Data Center• Closed Source First

• Agile• Automated Build• Automated

Deployment• Automated Test• Public Cloud• Open Source First

Agile & DevOps Transformation Journey

@TopoPal

Mostly Out-Sourced Mostly In-Sourced

Agile & DevOps Transformation Journey

Vertical Silos Product Team Dev, Ops, QA, RM Engineers

@TopoPal

DOES 2014Building out Automation steps

DOES 2015 Scaling DevOps, Open Source, Cloud,

Innovation

DOES 2016Measure, Improve, Mature

@TopoPal

Typical DevOps Success Story

Code Commit Random 100s /day

Deployment

Prod

Manual Automated

Integration Monthly 15 mins

QA, Perf Monthly 4 / dayMonthly/Quarterly Once / sprint

Testing Manual Automated

@TopoPal

2016What’s in your pipeline?

@TopoPal

http://www.devopsdays.org

@TopoPal

Deliver High Quality Working Software Faster

@TopoPal

Deliver High Quality Working Software Faster• No security flaws

• No legal flaws • Minimum defects• All levels of testing done • Code reviewed and source controlled• Testing of application, configuration, scripts

etc.

• Across LOBs, Shared Services and 3rd Parties

• Tested end-to-end• All dependencies are satisfied

• How fast? ASAP?

@TopoPalhttps://upload.wikimedia.org/wikipedia/commons/c/c8/Can_We_Do_it_Better_or_Faster...We_Want_Your_Ideas_-_NARA_-_534240.jpg

@TopoPal

@TopoPal

Feb 8, 1700 — March 17, 1782

Daniel J. Bernoulli

@TopoPal

Constrict flow, Increase Speed, Lessen Pressure

https://www.khanacademy.org/science/physics/fluids/fluid-dynamics/a/what-is-volume-flow-rate

@TopoPal

Commit

Deploy

@TopoPal

http://www.netuba.org/

@TopoPal

https://en.wikipedia.org/wiki/Oil_refinery

@TopoPal

https://commons.wikimedia.org/wiki/File:US_Navy_060906-N-8257O-026_Damage_Controlman_1st_Class_Petty_Officer_Derrick_Harney_assists_his_students_in_repairing_a_broken_pipeline_during_the_hands_on_patch_training_portion_of_the_Damage_Control_Wet_Trainer.jpg

@TopoPal

• Design• Measure• Improve

Pipeline

@TopoPal

Pipeline Design

@TopoPal

Pipeline must have 16 gates Source code version controlOptimum branching strategy

Static analysis> 80% Code coverage

Vulnerability scanOpen source scan

Artifact version controlAuto provision

Immutable serversIntegration testing Performance testing

Build, Deploy, Testing automated for every commitAutomated Change Order

Zero downtime releaseFeature Toggle

@TopoPal

Pipeline Measurement

@TopoPal

https://devops-research.com/

@TopoPal

https://devops-research.com/ https://github.com/capitalone/Hygieia

@TopoPal

Increase Speed = Reduce Wait Time

@TopoPal

Opportunities

• Branching Strategy• Process

@TopoPal

Pipeline Improvement

Improve Branching

@TopoPal

Branching• We recommend “Trunk based”

development.• Other option:

@TopoPal

Pipeline Improvement

Improve Process

• Automate Release Process• Revisit Audit & Compliance

@TopoPal

Risks are real• Intentional damage• Unintentional damage• Untested code in production

But….There is a better way

@TopoPal

Hypothesis

• DevOpsSec & CI/CD provide better controls

• A model with ~30 practices can satisfy audit and compliance

• If everything is source code, no one needs access to production

• For emergency, “Break Glass”

@TopoPal

Result

Production Release 1+ / dayOnce / sprint

# of Applications with Release Automation: 20+Max. # of Releases in 1 day for 1 Application: 34

With “Segregation of Duties”

@TopoPal

Goal

Release Automation without

classic “Segregation of Duties”

@TopoPal

Coming Soon to Open Source

• A secure & compliant pipeline model• A forked and enhanced version of

“LGTM”

@TopoPal

@TopoPal

Thank You!