Pluggable Infrastructure with CI/CD and Docker.

Bob Killen

Systems Programmer/Analyst Senior

Department of Pathology

University of Michigan Health System

Jeffrey Sica

Business Systems Analyst Lead

Department of Pathology

University of Michigan Health System

DisclaimerAll opinions spoken by Bob or Jeff are their own and not that of the

University of Michigan or its affiliates.

A bit of backgroundCentral IT vs Pathology Informatics

HIPAA Concerns

Our original infrastructure

● Before Docker, there were VMs

● Started on Kubernetes

● Migrated to Individual Hosts

Challenges we faced● Managing multiple instances of the same infrastructure

○ Public applications must be firewalled from Internal applications (HIPAA)

● Rapid prototyping and demoing of applications

○ Victims of our own success

● Large infrastructure, small staff

○ 40+ independant in house developed applications or services

○ 1 Designer, 1.5 Developers, 1.5 Sysadmins

Workflow - Outside of Host Cluster

Commit and

Code Push

Source Control server notifies

Build Server of new commit.

Build Server schedules

build at new commit.

If build successful; tag image and

push to registry. Otherwise notify

group.

After successful build pushed to the registry,

trigger cluster update. Notify if problem arises

during update.Build

Workflow - In Cluster

Cluster is notified

of new task.

Cluster determines hosts with

appropriate resource offerings.

Hosts pull containers

associated with task.

Once containers pass healthcheck,

front end is notified of host:port

Front End updates configuration

and reloads with new services.

Service is now accessible to public network

The Pillars of a Container Centric Infrastructure

The path we have chosen.

Gitlab Jenkins DTR Mesos

Docker

Gitlab● Open source

● Close-enough mirror to the familiar (GitHub)

● Low barrier to entry for non-developers

● Existing integration into other “pillars” (Jenkins)

● Now comes in a handy dandy container

○ “ docker pull gitlab/gitlab-ce ”

○ https://hub.docker.com/r/gitlab/gitlab-ce/

Jenkins● Open Source

● Time tested solution

● Plug-ins galore

○ Docker

○ Mesos Cluster

○ Gitlab Webhooks

● No real constraints on architecture

○ No plugin? Use Bash!

● Blank slate for any project

● Windows Support

Docker Trusted Registry● Security Requirements

○ 100% on premise.

○ LDAP backed.

○ RBAC - v0 api available, use and integration capability growing.

■ https://docs.docker.com/docker-trusted-registry/api/

○ One-click upgrade.

○ User friendly UI.

● Support Offerings

○ 24/7 support REQUIRED for use in a clinical care setting.

○ Guided install with staff on-site where needed

Mesos● Generic abstraction of resources. Useful to combine varied host systems (both

physical and virtual).

● Caters well to an on prem solution.

● Slightly more mature in the cluster solution ecosystem.

○ Powered by Mesos Users - http://mesos.apache.org/documentation/latest/powered-by-mesos/

● Large framework ecosystem; with lots of active development.

Mesos (Cont.)

● https://open.mesosphere.com/frameworks/

Just to name a

few….

Mesos (Cont…future)

Mesos Windows Epic: https://issues.apache.org/jira/browse/MESOS-3094

Docker + Windows + Mesos = Match made in the clouds

Screencast Demo: https://www.youtube.com/watch?v=dC2vitIh84M

Docker (CE)● Support Offerings

○ Touching on this again...24/7 support is REQUIRED when used in a clinical setting.

● Certified stable releases suitable for production use.

● Predictable release schedule for upgrade planning.

● Orchestration components are swappable...docker engine is not, and is the base

requirement for everything.

Final Form: Pluggable Infrastructure Pt. 1

● Application container function should be independent of infrastructure design.

○ Container init script + environment or cluster seed script.

■ container init script variables are static

■ seed script populates container init script with cluster specific information.

■ Build server can inject seed script for environment (local dev vs cluster deployment)

Example Init Script$VAR1 and $VAR2 can be set at

container creation or ‘seeded’ via a

script sourced as

$ENVIRONMENT_INIT. The

supplied script could hit a variety

of endpoints for configuration

information (consul, etcd,

registrator, etc).

If nothing is provided, it assumes

the default value.

This allows for init script to remain

static, with only the sourced seed

script changing per environment.

Final Form: Pluggable Infrastructure Pt. 2● Infrastructure components in source control; and if at all possible built as

containers themselves.

○ Apply standard ‘infrastructure as code’ methods.

■ Version Everything

■ Design tests and integrate with CI/CD

Final Form: Pluggable Infrastructure Pt. 2 (Cont.)● Components that may modify host configuration require some level of clean-up. e.

g. anything that may require iptables.

○ trap SIGINT, SIGKILL etc. then perform action.

○ Does require some form of ‘init’ or process management. - supervisord, tini, monit etc

● Use of VRRP / Keepalived and deploying in pairs when services may be bound to

a single point of failure (no DNS load balancing etc).

○ Provides incredibly simple IP failover.

○ Prerequisites for use:

■ sysctl - net.ipv4.ip_nonlocal_bind=1

■ container run with host networking and cap NET_ADMIN

○ Added bonus of making it EXTREMELY easy to upgrade and switch services.

■ If upgrading: push new version to backup, then push to master.

Keepalived + HAproxy Demo

Example Keepalived ConfigurationMaster Backup

Description

● state - MASTER or BACKUP - Defines server role in the group.

● interface - The interface that keepalived will use for VRRP traffic and

monitor by default.

● vrrp_unicast_bind - The ip in which vrrp should bind to.

● vrrp_unicast_peer - The bind ip of the other node in the group.

● virtual_router_id - Unique number between 1-255 that identifies the vrrp

group. Multiple instances of vrrp can run at the same time on the same servers.

● priority - In scenarios with more than 2 nodes, priority will dictate the

failover order with highest priority becoming the master.

● advert_int - Healthcheck interval (seconds)

● authentication - PASS (cleartext password) or AH (IPSEC

Authorization Header). Password and method used by the nodes

to validate each other.

● virtual_ipaddress - Virtual IP Address to be generated and interface it

should be bound to. IPs in this block WILL be tracked by keepalived. Used for

private network interface tracking.

● virtual_ipaddress_excluded - Virtual IP Address to be generated and

interface it should be bound to. IPs in this block will NOT be tracked, but

automatically failed over if an event would trigger a failure of IP(s) in the

virtual_ipaddress block. Used for public facing IPs.

● track_interface - Interface adapters that should have their link state

monitored for failover trigger.

Future State Challenges● Better Secret Management

○ Consul / Vault injection at runtime and seeded via ENVIRONMENT_INIT.

○ Mount secure volume

● Profiling containers for lxc capabilities + apparmor profiles

○ https://github.com/jfrazelle/bane

● Improve logging with logstash framework

○ https://github.com/mesos/logstash

● Notary integration into build pipeline.

Questions?Bob Killen

rkillen@med.umich.edu

@mrbobbytables

github.com/mrbobbytables

Jeffrey Sica

jsica@med.umich.edu

@jeefy

github.com/jeefy