The Human Side of DevSecOps

download The Human Side of DevSecOps

of 22

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of The Human Side of DevSecOps

The Human Side of DevSecOps

The Human Side of DevSecOps


2016 VERACODE INC.#@tojarrettOver 20 years in software development and managementAt Veracode since 2008Grammy award winnerBacon number of 3About Tim Jarrett



This talk assumes automation.

2016 VERACODE INC.#As noted up front, this is a talk about people and organizational factors. If you think that DevOps begins and ends at the pipeline and the technologies that plug into it, you may want to wait for the recap on Twitter. And this talk is definitely not for you if your job description for your AppSec team includes configuring, tuning, and running code and web app scanners by hand.

For those of you who have had one or more transformation efforts fail due to change management failures, staff shortages, or other organizational issuesand for those who have one or more automated AppSec tools and are wondering how to make them successfulthis talk is for you.3

DevOps: transformation or tragedy?

2016 VERACODE INC.#I think none of us would be at this webinar if we didnt think there is transformational power in DevOps. But for a lot of teams facing the DevOps transformation, the potential comes with a lot of anxiety.

After all, were talking about collapsing silos. In the real world, when that happens, people die.

This goes double for integrating Security into DevOps. But given that DevOps has already absorbed development, operations, QA and release engineering, why should the security transition be harder?4

h/t @petecheslock, DevOpsDays Austin


# The reason is culture clash. More than any other organization, Security has been publicly skeptical of the benefits of DevOps. Theyve seen a lot of development processes come and go, and insecure software keeps getting built.

h/t @petecheslock, DevOpsDays Austin5

Culture clash revisited

2016 VERACODE INC.#Lets put a finer point on the skepticism. Security has traditionally been a discipline of controls and gates. But theres no room for this sort of manual, expert-driven check in DevOps, just as theres no room for weeks of manual quality assurance testing if you want to ship code several times a day.

That means developers have to get security conscious, and security folks have to stop looking down their noses at DevOps and figure out how to help it move faster, not stand in the way.6

Credit: Gene Kim, IT Revolution


# As with a lot of other things, Gene Kims Three Ways help us to see the problem:

We need to see software delivery as an end to end system that includes security as part of the problem.

Security needs to help create some of the feedback loops from production (and earlier stages of the process), and needs to help Dev interpret and act on them.

And security needs to participate in the culture of continual experimentation and learning.7

Why desiloing Security is hard

SourceCory Scott, LinkedIn Director Information Security, Information Security Talent Pool Research, BlackHat CISO Summit 2015.

2016 VERACODE INC.#A big part of the challenge of meeting the need for security to participate in DevOps is math: specifically, for every four people employed in infosec, there are three additional job openings.

You cant hire an infosec team big enough to define security requirements, perform secure code reviews, interpret security testing results, and handle security alerts for all your applications in your entire portfolio if youre operating at DevOps speed.8

Consider the theory

2016 VERACODE INC.#Theory of Constraints: in an end to end flow, identify the constraint, exploit,subordinate, elevate, then repeatWhat is the constraint in DevSecOps?Define requirements --> Develop code --> Review code --> Build code --> Test code --> Handle bugs(Almost) all have some touch between security and developmentFirst pass: Security is the constraintExploit constraint maximize throughput with processes like security reviews etc.Subordinate constraint implicitly done either by minimizing the amount of security changes that happen or adding surge capacityElevate constraint Optimize handoffs, change the process


Consider the theory

Development work productsSecurityRelease velocity starved

2016 VERACODE INC.#So to put it another way, once youve optimized your Dev and Ops processes, Security becomes the constraint.


Theory of constraints for security in software developmentRemove low value work from security team, shift upstream where possible

Minimize changes requiring security review


2016 VERACODE INC.#IdentifyExploit Subordinate

(This theory, by the way, is why I say that this talk isnt for you if you are still running your scanners by hand. If youre doing that, you havent even begun to exploit the constraint of your security team yet.)

Elevate? How do we do that? The definition is In this step, more substantive changes are implemented to break the constraint. These changes may necessitate a significant investment of time and/or money and may involve targeted review of lost productive time, tactical actions, design and/or component upgrades, and supplementing the constraint with additional throughput.

But how does that work if you dont have enough people in the first place?

You do it by making more people who can do the work that security is doing11

Enter Security Champions!

Security Champions to the rescue

2016 VERACODE INC.#Enter Security Champions members of the DevOps team who can perform lower-skill, high frequency jobs to take load off the security team and help DevSecOps to scale. You can think of this role as essentially a new guild, like release engineers or folks who work on unit tests.

Some examples of things security champions can take on (well talk about a few of these):Peer reviewsSecurity groomingCode reviews (very specific topics based on certain security controls)Data validationEncodingParameterizationLoggingError handlingProduct security incident response new CVEs based on impact and severityProvide details of components, how to tell if youre exposed, etcKnown vulns e.g. Shellshock/Heartbleed/Struts-Shock triage impact, provide remediation plan

This sounds great, but how will we make this happen?


Pick the right peopleStart strongEmpower, within limits

2016 VERACODE INC.#So how do we do this practically? Theres a lot to talk about here, but I want to focus on how you get started-- how to pick the right people for the guild, how do you get the program started, and how do you set up the working relationship between the security champion and the central security team.

Well be drawing examples from Veracodes own experience standing up a security champion function.13

How to pick the right peopleJust developersBrand new(Too) JuniorAlready in a scrum role

2016 VERACODE INC.#Lets assume youve made the case to management. The first step is drafting your team. Lets talk about how not to pick the right people

-- You shouldnt assume that only developers are going to be good security champions. A strong QA resource may be a great candidate. ---- They should just have clear expectations on the time commitment.---- And make sure to loop the managers into the call for volunteers.---- And dont stop with just one per team you want to make sure youre not introducing a new single point of failure-- You should also have clear requirements. These will probably be specific to your culture and process, but some likely common requirements include:---- Not new to company (ramping on day 2 day)---- Not too junior (needed an influencer)---- Not in an existing scrum role (PO, SM)


Start strongStart with formal training in security fundamentalsReinforce with eLearningUse CTFs and other opportunities to learn in the wildSet guidelines for common activities

2016 VERACODE INC.#Now that you have the team in place, you need to give them a certain baseline knowledge. How you do this will differ according to what your security team looks like, but some of the basics are:

1. security fundamentals (2 day instructor led training)CIA, etcTrust no oneSecurity controls must be server-sideDeny by defaultUse white listsDefense in depthThreat modeling

2. Reinforce with eLearning to provide ongoing training and access

3. Ongoing education with CTF exercisesUse stuff thats readily available


Empower, within limitsSecurity grooming within guidelinesSecurity review guidelinesKnow when, and how, to escalate

2016 VERACODE INC.#Set guidelines for common activities Product specific grooming guidelinesUpdate based on findingsSelf-service review guidelines (what to teach vs. what can be done by anyone) automate if possible (pro tip)Does it need a reviewDoes it have to be a security team personCrypto?Can it be security championDictates basic acceptance criteria


2016 VERACODE INC.#All of these things are great activities. To be really successful, you need to measure what youre doing and how youre making progress.17

Measuring and managingBaseline security maturityCode review certificationsIndividual and team goals

2016 VERACODE INC.#Product security maturity modelBaseline and regular updatesOpen Samm and BSIMM useless built our ownNeeded a vision of an ideal security programCode review certificationeLearning, classroom training, code review testFirst year mentored transition failureSC do first reviewExpert validates resultsAbsence of findings was a problemFormal exercise w