Implementing DevOps in Highly Regulated Environments (HRE)
Transcript of Implementing DevOps in Highly Regulated Environments (HRE)
![Page 1: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/1.jpg)
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution
Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution
Implementing DevOps in Highly Regulated Environments (HRE)Joseph Yankel
Team Lead
Secure Lifecycle Solutions
CERT | SEI | CMU
![Page 2: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/2.jpg)
Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution
Topics
BackgroundWhat is an HRE and why is it different?Common PitfallsHRE Assessment Approach and PlanRequirements Analysis and evaluation
People, Process, Platform Moving Forward
![Page 3: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/3.jpg)
3Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Background
• The Software Engineering Institute (SEI) is a Federally Funded Research and Development Center (FFRDC)
• Research and practice in software development, acquisition, and maintenance practices
• Assisted numerous government organizations in modernizing their software development practices in the spirit of DevOps principles.
• Application security is the principle quality attribute of the software they produce.
![Page 4: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/4.jpg)
4Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps and How it started
DevOps is a set of principles and practices emphasizing collaboration and communication between software development teams and IT operations staff along with acquirers, suppliers and other stakeholders in the life cycle of a software system [1]
[1] IEEE P2675 DevOps Standard for Building Reliable and Secure Systems Including Application Build, Package and Deployment
• Patrick Debois “Agile infrastructure and operations: how infra-gile are you?”, Agile 2008 Conference
• John Allspaw “ 10+Deploys per Day: Dev and Ops Cooperation”, Velocity 2009
• DevOps Days, October 30th 2009, #DevOps term born
![Page 5: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/5.jpg)
5Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Follow Agile methodologies and use shiny and new technology• Scrum, Kanban, and other modern
development approaches• Self-directed, self-managed, self-organized• Each developer may have their own
development environment
Dev wants to deliver new software features faster.Ops wants to maintain stability, operations up-time.
• Operations - Support applications and services - Manage infrastructure- Provide Service Strategy, Design- Secure Systems
Dev Ops
![Page 6: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/6.jpg)
6Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps has four Fundamental Principles
Collaboration: Between project team roles
Infrastructure as Code: All assets are versioned, scripted, and shared
Automation: Deployment, testing, provisioning, any manual process
Monitoring: Any metric in the development or operational spaces that can inform priorities, direction, and policy
![Page 7: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/7.jpg)
7Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Ask yourself:
• Do your Devs know exactly what actual production looks like?• Does Ops know how Devs package a build? • Is it consistent?• Can both Dev and Ops collaborate on server configuration and apply it
automatically to both development and production environments?• Do business analysts know the cost of feature addition or modification?• Can project managers measure project status at any point in time?• Can the customer measure project status at any point in time?
Without a Collaborative Culture, You Don’t Have DevOps
![Page 8: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/8.jpg)
8Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
• Blame-Free Culture• No Hiding of Problems• Culture of shared responsibility• Collective decision and continuous learning
• Cross-Silo Goals• Incentivize Collaboration• Reduce “Not My Job”• Increase Sense of Purpose
• Optimize Ease-of-Use• Tools: Chat, ChatOps, Wiki• Integrated Pipelines
Enabling Effective Collaboration
![Page 9: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/9.jpg)
9Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
A Common Question
How can I implement a Secure DevOps process and platform in my team / directorate / project / organization / unit?
How to assess the current state?Where are the productivity bottlenecks? Whom to train on what?What and how to measure?How to monitor?
![Page 10: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/10.jpg)
10Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Current State of practice
• With Surveys; • DevOps State of report last couple years
• 2014, 2015, 2016 and 2017
• Lead to research for Performance and ROI; • Dora (DevOps Research and Assessment);• Performance Matrix against industry practices
• Maturity Assessment; Ranger4• Not Started, Starting, Fundamental, Managed and Optimizing DevOps
• And others like • Tool approach based assessment• IBM, CA Technologies, ThoughtWorks or similar
![Page 11: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/11.jpg)
11Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Topics
BackgroundWhat is an HRE and why is it different?Common PitfallsHRE Assessment Approach and PlanRequirements Analysis and evaluation
People, Process, Platform Moving Forward
![Page 12: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/12.jpg)
12Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
What is HRE?
• Highly-Regulated Environment (DoD, Health, Finance, etc…)• Air-gapped computer system• Isolated working Groups• Strong physical Security• Segregation of Duties• Information classification
• Inability to speak, share/collaborate on artifacts• Level of Security and Risk management• Limitation of Continuous Deployment• Physical System Integration• Strong Audit Trail on each level of development activities
![Page 13: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/13.jpg)
13Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
A closer look at HRE..
![Page 14: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/14.jpg)
14Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
How we can assess these kinds of environments and then deploy Secure DevOps processes and
techniques?
![Page 15: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/15.jpg)
15Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Topics
BackgroundWhat is an HRE and why is it different?Common PitfallsHRE Assessment Approach and PlanRequirements Analysis and evaluation
People, Process, Platform Moving Forward
![Page 16: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/16.jpg)
16Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Common Pitfalls
HELP!
![Page 17: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/17.jpg)
17Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
What can go wrong? (Organizational Culture)
DevOps is • A FAD• Only about tooling• A Product• Only about Dev and Ops• The same for all organizations• Only continuous integration/deployment• New organizational unit
![Page 18: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/18.jpg)
18Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Topics
BackgroundWhat is an HRE and why is it different?Common PitfallsHRE Assessment Approach and PlanRequirements Analysis and evaluation
People, Process, Platform Moving Forward
![Page 19: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/19.jpg)
19Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps in HRE Assessment; Overview
Interview with functional leads from key areas related to Application Development.Review of:• Validation of statements (e.g., through
observations of the work environment or shadowing)
• Demonstrations of any software tools used for automation of software development and deployment
• Cultural perspective related to development evolution and Security team
• Legal, Risk Management and all stakeholders
![Page 20: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/20.jpg)
20Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment; Plan
1. Agree on definitions(DevOps, DevSecOps) and process2. Identify stakeholders3. Perform interviews on each team4. Identify and analyze technical tool stack5. Collect key metrics and establish measurement6. Identify gap areas and develop a roadmap7. Select a suitable project to implement: Build , Learn, Evaluate
![Page 21: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/21.jpg)
21Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
• Scheduling interviews with teams• Conduct anonymous surveys• Analyze outcomes• Provide feedback to the teams• Brief the executive team
DevOps on HRE Assessment; Process
![Page 22: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/22.jpg)
22Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Identify Stakeholders
![Page 23: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/23.jpg)
23Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
![Page 24: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/24.jpg)
24Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
![Page 25: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/25.jpg)
25Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
![Page 26: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/26.jpg)
26Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
![Page 27: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/27.jpg)
27Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
![Page 28: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/28.jpg)
28Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Business Analyst/ PM• Requirements development & management• Acquisition & contracting process• Risk management process• Compliances requirements• Project Planning and tracking
![Page 29: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/29.jpg)
29Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Developer• Development methodology
• agile, waterfall, SAFe, EP, Lean, or cowboy coding• Development environments• Task assignment/management / completion• Collaboration with other (internal/external) teams
![Page 30: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/30.jpg)
30Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Quality Assurance Team• Software testing methodologies• Software {quality} assurance• Compliances verification• Audit requirements• Feedback to dev team
![Page 31: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/31.jpg)
31Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Deployment/Release Mgr.• Software configuration management• Integration process• Software verification and validation process• Software review and audit process• Securing the deployment pipeline
![Page 32: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/32.jpg)
32Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: IT Operations (not Ops)• Software operational process• Team engagement• Policy knowledge management• Assets management• IT governance• Service management• Audit and monitoring
![Page 33: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/33.jpg)
33Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Information Security• Management and auditing supply chain• Security controls• Security polices (compliance requirements)• Application security testing• Product security management (PSIRT)• Security awareness training and knowledge management
![Page 34: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/34.jpg)
34Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Technology Stack• Development language and tools• IT solution stack• Enterprise support services• Legacy systems• Application development support tools• Software reuse process• Accreditation and approval process
![Page 35: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/35.jpg)
35Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Metrics and Measurement• Software metrics• Quality metrics• Checkpoint diagnostic
• Qualitative process baseline• Quantitative performance baseline• Benchmark performance comparison
• Define end-goal as developing a Secure Software: • What that means to all stakeholders
![Page 36: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/36.jpg)
36Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Identify Suitable Project (Rollout Plan)Select {new or existing} project as pilot•Most stakeholders involvement•Minimize risk to business•Ability learn/develop/ implement security in the process•Scalable to the organization
![Page 37: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/37.jpg)
37Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Topics
BackgroundWhat is an HRE and why is it different?Common PitfallsHRE Assessment Approach and PlanRequirements Analysis and evaluation
People, Process, Platform Moving Forward
![Page 38: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/38.jpg)
38Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Feedback to the team
- Collaborate all team leads - Share identified requirements
- Categorize and prioritize the requirements
- Collectively develop implementation plan:
People + Process + Platform = Plan
![Page 39: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/39.jpg)
39Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: People
Heavy collaboration between all stakeholders• Secure Design / Architecture decisions• Secure Environment / Network configuration• Secure Deployment planning• Secure Code Review
Constantly available open communication channels:• Dev and OpSec together in all project decision meeting• Chat/e-mail/Wiki services available to all team members
![Page 40: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/40.jpg)
40Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Process
Establish a process to enable people to succeed using the platform to develop Secure applicationSuch that;
• Constant communication and visible to all• Ensures that tasks are testable and repeatable• Frees up human experts to do challenging,
creative work• Allows tasks to be performed with minimal effort
or cost • Creates confidence in task success, after past
repetitions• Faster deployment , frequent quality release
![Page 41: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/41.jpg)
41Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps on HRE Assessment: Platform
Where people use process to build secure software
• Automated environment creation and provisioning
• Automated infrastructure testing• Parity between Development, QA, Staging,
and Production environments• Sharing and versioning of environmental
configurations• Collaborative environment between all
stakeholders
![Page 42: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/42.jpg)
42Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Topics
BackgroundWhat is an HRE and why is it different?Common PitfallsHRE Assessment Approach and PlanRequirements Analysis and evaluation
People, Process, Platform Moving Forward
![Page 43: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/43.jpg)
43Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
DevOps + Environment Parity
![Page 44: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/44.jpg)
44Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Adding Security
![Page 45: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/45.jpg)
45Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
SLS team GitHub Projects
• Once Click DevOps deploymenthttps://github.com/SLS-ALL/devops-voltron
• Sample app with DevOps Process https://github.com/SLS-ALL/flask_api_sample
• Tagged checkpoints• v0.1.0: base Flask project• v0.2.0: Vagrant development configuration• v0.3.0: Test environment and Fabric deployment• v0.4.0: Upstart services, external configuration files• v0.5.0: Production environment
• On YouTube:https://www.youtube.com/watch?v=5nQlJ-FWA5A
![Page 46: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/46.jpg)
46Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
For more information…
SEI DevOps Bloghttps://insights.sei.cmu.edu/devops
![Page 47: Implementing DevOps in Highly Regulated Environments (HRE)](https://reader030.fdocuments.us/reader030/viewer/2022020916/61aac21cb1ef735ed168d7f4/html5/thumbnails/47.jpg)
47Implementing DevOps in Highly Regulated Environments © 2018 Carnegie Mellon University
Contact InformationJoseph Yankel Team Lead Secure Lifecycle Solutions [email protected]
@securelifecycle
Web Resources (CERT/SEI)
http://www.cert.org/
http://www.sei.cmu.edu/