Securing Application Deployments in CI/CD Environments (Updated slides:
-
Upload
binu-ramakrishnan -
Category
Technology
-
view
687 -
download
0
Transcript of Securing Application Deployments in CI/CD Environments (Updated slides:
Securing Application Deployments in CI/CD Environments
Hello!I am Binu RamakrishnanPrincipal Security Engineer at Yahoo
You can find me at:@securitysauce
Overview◎CI/CD platform overview.
◎High level objectives.
◎Threat modeling.
◎Security patterns and best practices.
CI/CD
CI/CD Deployments◎Single-tenant◎Multi-tenant
Single-tenantDedicated build environments.
Multi-tenantShared build environment.
Driven by economies of scale.
CI/CD in Nutshell
Enterprise CI/CD
High Level Security Objectives
Verifiable Chain of Trust
No Elevated Privileges for Build Jobs
Threat Modeling
CI/CD Admin Exclusive access to CI/CD
platform
Actors
Developer User/customer of the
platform.
CI/CD Platform
DeveloperLimited admin access to
platform.
Trust Boundaries.
https://www.flickr.com/photos/nasamarshall/14596371842
Trust Boundaries
Web interface◎ UI
◎ REST APIs
Application
Trust Boundaries
Isolation based on Docker containers or VMs.
Host
Trust Boundaries
Segmentation
Network
Attack SurfaceEntry Points
Exit Points
Entry Points
◎ Build web interface.
◎ Commit and build notification handlers.
◎ Source and package dependencies.
Internal Entry Points
◎ Build slave endpoint (to launch builds; SSH).
◎ Other control plane endpoints/REST APIs.
Exit Points
◎ Deployable artifacts.
◎ Build notifications - eg emails, IRC and Hipchat
messages.
◎ Build console logs.
◎ Git repo update with build status and badges.
Threats
Common Threats
◎ User account compromise & insider threats.
○User and platform admins.
◎ Network Intrusion.
○CI/CD internal and external endpoints.
Build Slave CompromiseLarge attack surface, spread across multiple networks (iphone, Android, Server app etc.)
How?◎ A network level compromise, exploiting a vulnerability in build slave.
◎ Jobs break out of build container.Impact◎ Access to production servers.
◎ Listen to the network, spoof identity and access unauthorized data.
Shared/Non-ephemeral Keys◎ Locally stored long-lived keys to access protected
services.Two forms:
◉ Shared key
◉ Per-job keys (eg. OAuth tokens)
SSH over Unrestricted ShellAllows arbitrary commands to get executed on a remote host.
Building External CodeAn attacker can take this path to get into internal networks, either by adding backdoors or exploiting known vulnerabilities with open source software.
Baseline security controls that are expected to be there in 2016
Existing Security Controls◎ Authenticated endpoints.
◎ Web application security practices - CSRF, HTTPS,
HTTPOnly cookies, XSS protection etc.
◎ Source control: granular authorization for repos.
◎ Use KMS to manage secrets; no hardcoded secrets.
Security Patterns for Risk Reduction
Ephemeral Keys
https://www.flickr.com/photos/articnomad/241620406
Stateless Auth Architecture
https://www.flickr.com/photos/18946008@N06/14551311971/
◎ Events:
○ Commit trigger.
○ Manual trigger from build UI.
○ Automated/cron job.
○ Trigger a downstream job.
◎ Upstream service stores downstream service credentials (OAuth, Shared Keys etc.).
◎ Equal trust on all components in the pipeline
Audit Trails
https://www.flickr.com/photos/adriensifre/8403355648
Container, ToolChainHardening
http://www.publicdomainfiles.com/show_file.php?id=13493588417214
Network Segmentation
https://www.flickr.com/photos/bretagne-balades/15355029654
Do not pull PII or other sensitive info to build machine.
Minimal Builds
Few more...SSH: Use Restricted ShellsHeadless SSH access for automated deployment should use a restricted shell.
Roll Keys PeriodicallyEstablish a process to periodically roll trust anchor keys (and do it periodically).
Restrict Job Console LogsRestrict build job console logs only to authorized users
Enable 2FAAdmins must follow good security hygiene and use 2FA to access platform application and hosts
Prune Admin Access ListKeep admin list small for build systems and Git repo access
Vulnerability Patch MgmtMaintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure
Acknowledgements◎ Christopher Harrell
◎ St John Johnson
◎ Mike Shema
◎ Jeremiah Wuenschel
Let’s recap major conceptsEphemeral KeysEphemeral keys are the future. Service providers should start supporting ephemeral keys for authorization
Stateless Auth ArchitectureAugment the trust dependency of 1:1 relationship between the pipeline components with workflow job tokens.
Audit LogsA verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD.
Minimal BuildsAvoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum
Network SegmentationNetwork level Isolation of CI/CD machines from other machines
ToolChain HardeningBuild tools and Docker containers must be adequately hardened.
Thanks!Any questions?You can find me at:@securitysauce
Appendix-A
Use◎ Source code◎ Pre-built packages
Risks◎ Targeted back doors◎ Vulnerabilities
Hard Problem: Securing Supply Chain
This threat is more to do with applications than the CI/CD platform itself.
Open source components constitute a large part of modern Internet based applications.
Appendix-B
Credits
Special thanks to all the people who made and released these awesome resources for free:◎ Presentation template by SlidesCarnival◎ Photographs by Unsplash & Death to the Stock Photo (
license)