2 When everybody cares about the product, but CI/CD is … · 2019-06-08 · Real World CI/CD...

Institute of Software Technology (ISTE)

Reliable Software Systems Group (RSS)

2

Thomas F. Düllmann

When everybody cares about the product, but CI/CD is neglected:

Assessing and Improving Dependability and Security of

CI/CD Infrastructures

June 4th, 2019

SecSE 2019 (Oxford, GB)

Motivation

[1]

[2]

[3]

[4]

[1] http://yesofcorsa.com/wp-content/uploads/2017/12/Autobahn-Wallpaper-HQ.jpg

[2] https://aisrtlnext-a.akamaihd.net/masters/936779/1032x581/a20-bei-tribsees-gesperrt-autobahn-bricht-einfach-weg.jpg

[3] http://www.ln-online.de/var/storage/images/oz/nachrichten/mv-aktuell/a20-bei-tribsees-schwerer-unfall-auf-kaputter-ostsee-autobahn-brandenburger-

kracht-mit-auto-in-pkw-vier-verletzte/717008474-2-ger-DE/Brandenburger-uebersieht-Rostocker-Pkw-vor-A20-Baustelle-Vier-Verletzte_big_teaser_article.jpg

[4] https://twitter.com/azolyak/status/986629551189995522

2019-06-04T. F. Düllmann: When everybody cares about the product, but CI/CD is neglected: Assessing and Improving Dependability and

Security of CI/CD Infrastructures

2

Foundations

»Dependability..



3

... is the ability to avoid service

failures that are more

frequent and more severe

than is acceptable.«

»Security..

Avizienis et al. Basic Concepts and Taxonomy of Dependable

and Sec. Comp. IEEE Trans. Dependable Sec. Comput., 2004

ISO/IEC. 2016. ISO/IEC 27000: Information technology —

Security techniques — Information security management systems

... ensures the confidentiality,

availability, and integrity of

information.«

Source: GitLab presentation

Problem: CD pipelines are neglected

business-critical infrastructures

4T. F. Düllmann: When everybody cares about the product, but CI/CD is neglected: Assessing and Improving Dependability and


2019-06-04

ApproachesAbstraction of Pipelines, Identification of Vulnerabilities,

Conversion of Pipelines to Workflows, Analysis of real-

world Pipeline data

Vision:Improve dependability and security of

CD pipelines by using DevOps Practices

RCoSE, ICSE 2018

QUDOS, ICSA 2019

Roadmap and Agenda



2019-06-04

CI/CD pipelines

Modeling Threats CriticalityDevOps practices

1

• Industry case studies

• Vulnerabilities

• Threat analysis

• Impact of manual changes

5• DevOps methods

• Evaluation

4

• BPMN tooling

• Simulation

• Comparison with Reality

3

• Real World Pipelines

• Properties/Metrics

• Formal Modeling

and Simulation

2

• Feature discovery (IaC/BPMN)

• DSL design

• DSL IaC/BPMN

• Questions

• Real-world CD pipelines?

• Important security aspects?

• Vulnerabilities of CD pipelines?

• Method

• 2 Projects using CD pipelines

• Survey

• Abstracted CD pipeline

• STRIDE threat analysis

• Results

• Focus on T, I, D: 21 STRIDE scenarios in total

• Identified 22 confirmed vulnerabilities (11 per project)

T. F. Düllmann: When everybody cares about the product, but CI/CD is neglected: Assessing and Improving Dependability and


Case Study

62019-06-04

CI/CD in Relation to Workflows/BPMN



7

The DSL StalkCD

• abstracts a CD process from a Jenkinsfile

• adds information relevant for visualization

• bridges the functional gap between

Jenkinsfile and BPMN

• forms basis for portability

• can be extended to be used with other CI/CD tools

• allows vendor-agnostic pipeline representation

Jenkinsfile BPMN

Tra

nsf

orm

atio

n

So

ftw

are

StalkCD File (YAML)

StalkCDData Model

pa

rse

ge

ne

rate

tra

nsla

te

wri

te

Real World CI/CD Pipelines

• Metrics and evolutionary data about real-world pipelines

• Draw conclusions about infrastructure

• Baseline for usage in formal models (e.g., Petri Nets)



8

TravisTorrent Joined DatasetGitHubTorrent

Git

Custom Extraction

Raw Logs

Summary



2019-06-04

CI/CD pipelines

Modeling Threats CriticalityDevOps practices

1

• Industry case studies

• Vulnerabilities

• Threat analysis

• Impact of manual changes

5• DevOps methods

• Evaluation

4

• BPMN tooling

• Simulation

• Comparison with Reality

3

• Real World Pipelines

• Properties/Metrics

• Formal Modeling

and Simulation

2

• Feature discovery (IaC/BPMN)

• DSL design (StalkCD)

• DSL IaC/BPMN

Conclusion / Future Work

• Acquire information, data, and metrics about real CI/CD infrastructures

• Become tool-agnostic and use tools from other domains

• Explore possible approaches from formal methods and evaluate their usefulness

• Use DevOps approaches not only for the product, but also the infrastructure



10

Backup Slides



11

2019-01-22Thomas F. Düllmann (SPEC DevOps Performance), Subgroup Proposal: Performance of Continuous Delivery Infrastructure

Real World

Infrastructure

as Code (IaC)

StalkCD

DSL

Simulation

Evaluation Comparison

KPIs

Optimization Compensation

Overview

12

2019-01-22Thomas F. Düllmann (SPEC DevOps Performance), Subgroup Proposal: Performance of Continuous Delivery Infrastructure

Real World

Infrastructure

as Code (IaC)

DSL

Simulation

Evaluation Comparison

KPIs

Optimization Compensation

Overview

13

ModelingPerformance

Metrics

Case Studies

Pipeline

Abstracted CD Pipeline

14

Developer

9. retrieve artifacts

Artifact repository6. store artifacts

4. retrieve sources

CI/CD server2. notify

Repository1. push

build test deploy

Deployment server

8. trigger

Library store5. get libraries

Entity

Activity

Event

STRIDE Example



15

Occurrence 1. Push

Threat type S T R I D E

Threat Commit arbitrary code; manipulate or

remove pipeline scripts

Effect Malicious code; no delivery

Vulnerability • None or few access restrictions

• No review of code changes

• No testing of pipeline scripts

• Focus on T, I, and D

• 21 scenarios in total

Investigation of the Use of DevOps Practices in CD Pipelines

DevOps Practices

• Canary Releasing

• A/B Testing

• Monitoring

• Fault Injection

16

Chaos Engineering

• Chaos Monkeys (Netflix)



2019-06-04

Related Work

• Security Tactics by Ullah et al. [18]

• Integrating Security in Agile Development Processes and CD Pipelines by Stazic [16]

• Security Hardening of CD Pipelines by Bass et al. [2]

• Securing Artifacts in a CD Pipeline by Kuusela [8]

• Security Risk Analysis of Public CI Services by Gruhn et al. [4]

• Threat Modeling Process for an Exemplary Software Supply Chain by Lipke [10]



2019-06-04

References[1] Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl E. Landwehr. 2004. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Trans. Dependable Sec. Comput. 1, 1

(2004), 11–33.

[2] Len Bass, Ralph Holz, Paul Rimba, An Binh Tran, and Liming Zhu. 2015. Securing a deployment pipeline. In Proc. IEEE/ACM 3rd International Workshop on Release Engineering (RELENG). IEEE, 4–7.

[3] Leonard J. Bass, Ingo M. Weber, and Liming Zhu. 2015. DevOps — A Software Architect’s Perspective. Addison-Wesley.

[4] Volker Gruhn, Christoph Hannebauer, and Christian John. 2013. Security of public continuous integration services. In Proc. 9th International Symposium on Open Collaboration (OpenSym). 15:1–15:10.

[5] Jez Humble. 2017. Continuous Delivery Sounds Great, but Will It Work Here? Queue 15, 6, Article 70 (Dec. 2017), 20 pages.

[6] Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Pearson Education.

[7] ISO/IEC. 2016. ISO/IEC 27000: Information technology — Security techniques — Information security management systems — Overview and vocabulary. (02 2016).

[8] Juha Kuusela. 2017. Security testing in continuous integration processes. Master’s thesis. Aalto University, School of Science, Finland.

[9] Hanno Langweg and Einar Snekkenes. 2004. A classification of malicious software attacks. In Proc. International Conference on Performance, Computing, and Communications (IPCCC). IEEE, 827–832.

[10] Simon Lipke. 2017. Building a Secure Software Supply Chain using Docker. Master’s thesis. Hochschule der Medien, Stuttgart, Germany.

[11] Kief Morris. 2016. Infrastructure as code: managing servers in the cloud. O’Reilly Media, Inc.

[12] Michael Nygard. 2007. Release It!: Design and Deploy Production-Ready Software.

[13] Open Web Application Security Project. 2017. OWASP Top 10 - 2017. (2017).

[14] Casey Rosenthal, Lorin Hochstein, Aaron Blohowiak, Nora Jones, and Ali Basiri. 2017. Chaos Engineering: Building Confidence in System Behavior through Experiments (1st ed.). O’Reilly.

[15] Adam Shostack. 2014. Threat modeling: Designing for security. John Wiley & Sons.

[16] Damir Stažić. 2017. Security DevOps: Konzeption einer Umgebung zur Integration von Sicherheitstests in agile Softwareentwicklungsprozesse. Master’s thesis. Reutlingen University.

[17] Matthias Tichy, Michael Goedicke, Jan Bosch, and Brian Fitzgerald. 2017. Rapid Continuous Software Engineering. Journal of Systems and Software 133 (2017),159.

[18] Faheem Ullah, Adam Johannes Raft, Mojtaba Shahin, Mansooreh Zahedi, and Muhammad Ali Babar. 2017. Security Support in Continuous Deployment Pipeline. In Proc. 12th International Conference on

Evaluation of Novel Approaches to Software Engineering (ENASE). 57–68.

[19] Johannes Wettinger. 2017. Gathering solutions and providing APIs for their orchestration to implement continuous software delivery. Ph.D. Dissertation. University of Stuttgart, Germany.



2019-06-04

Survey questions

Backup slides



192019-06-04

1. In your opinion which security objectives should be pursued to CD pipelines?

Please do not focus on a specific used pipeline. Think in general.

2. In your opinion which security attribute is the most important one in respect to

CD pipelines (artifacts, files, scripts, connections, ...)? Order the following security attributes (confidentiality,

integrity, availability, authorization, authentication, nonrepudiation) according to their importance. The attribute on top is

the most important one for you.

3. In your opinion what are possible attack scenarios for the pipeline you use? Against which attacks would you like to

protect your pipeline?

4. Which security objectives are pursued in your project in respect to CD pipelines? Which are implemented?

5. How many years of experience in software development do you approximately have?

6. Which tools do you know and/or use? Response options: (DevOps tools) Jenkins; Kubernetes; TeamCity;

Spinnaker; Travis; GoCD; Concourse CI; JFrog Artifactory; (static analysis tools) PMD; Checkstyle; FindBugs;

FindBugs Security; (security tools) OWASP ZAP; BDD Security;

JFrog Xray; Security Monkey; Black Duck; Snyk

Survey questions

Backup slides



202019-06-04

7. In which role do you interact with your CD pipeline? Response options: user (committing code to the project, usage of

the CD pipeline); installation and operation of the pipeline; configuration of the pipeline; other

8. In your opinion how important is the topic security vulnerabilities in CD pipelines?

Response options: 1; 2; 3; 4; 5 (1: not important, 5: very important)

9. How often do you deal with security in your development process?

Response options: Never; only occasionally; quite often; most of the time; no answer

10. In the next step think about the security of the [...] CD pipeline. In your opinion how secure is this pipeline?

Response opinion: 1; 2; 3; 4; 5 (1: means CD pipeline is insecure, 5: means CD pipeline is secure (pipeline has no

vulnerabilities))

Case Study: Survey

21

3; 16%

3; 16%

1; 5%11; 58%

1; 5%

User, installation, operation

User, configuration

Scrum Master

User,

installation,

operation,

configuration

User only

(Committing code,

usage of the UI’s of CD

pipeline components)

How often do you deal with security in your development process?

In which roles do you interact with your (projects) CD pipeline?T. F. Düllmann: When everybody cares about the product, but CI/CD is neglected: Assessing and Improving Dependability and


2019-06-04

Survey results

Backup slides



222019-06-04

Survey results

Backup slides



232019-06-04



242019-06-04

Survey results

Backup slides

Survey results – security objectives

Backup slides



252019-06-04

• No pipeline modification through users who have no access rights

• No triggering of the pipeline through unauthorized persons

• Securing source code, logs and artifacts

• Securing environment properties such as login data

• Securing credentials (encrypt all sensitive data)

• Build steps should not be manipulated

• No vulnerabilities in dependencies

• Reduce human errors (storing password)

• Secure transmission over Hypertext Transfer Protocol Secure (HTTPS) or Secure

Shell (SSH)

• Use 4-eye-principle

• Check access rights of the components of the CD pipeline

Survey results – attack scenarios

Backup slides



262019-06-04

Integrity

• An attacker or a third person who has unauthorized access manipulates the configuration of the pipeline. The

manipulation can affect every specific pipeline file like the Jenkinsfile, Dockerfile or on any other configuration like

the CD server configuration or any component of the server.

• Manipulation of artifacts, logs or deployment scripts.

• Injection of malicious code, files which can include worms or viruses into the CD pipeline. These files can be injected

through back doors or leaks in the application. It is possible that such malicious code is deployed.

• In many cases, the used pipeline tools have vulnerabilities and open new doors for potential attackers.

Availability

• DoS attacks - effectively shut down the server.

• An unavailable pipeline would prevent the delivery of the software

• Attacks which manage to change something on the pipeline can damage the environment in which the pipeline is

running.

Confidentiality

• Execute a MITM attack.

• Cross build injection attack.

• An attacker can gain sensitive data such as credentials if used plugins, libraries or pipeline components have

vulnerabilities.

Survey results – security objectives

Backup slides



272019-06-04

• Requiring authentication and authorization

• Securing credentials and hide critical data.

• Review the process

• No information should be included in the source code of applications

• Implemented access control (not all team members have administrator rights)

• Keep the pipeline components and software up to date

STRIDE vulnerabilities

Backup slides



282019-06-04

• Internal employees (human errors)

• Unencrypted connections between CD pipeline components

• Insecure environment of the CD pipeline components

• None or few access restrictions

• Use of vulnerable versions of the CD pipeline components

• Vulnerable CD pipeline configurations

• Vulnerable code commits, CD pipeline scripts, Docker images/containers, artifacts

• No review of changes on the CD pipeline

Tools

Backup slides



292019-06-04

Tools

Backup slides



302019-06-04

2 When everybody cares about the product, but CI/CD is … · 2019-06-08 · Real World CI/CD...

Documents

Transcript of 2 When everybody cares about the product, but CI/CD is … · 2019-06-08 · Real World CI/CD...