Package All the things
From #packagingsucks to #packaginglove

Kris Buytaert@krisbuytaert

Kris Buytaert

I used to be a Dev,

Then Became an Op

OpenMosix Release Manager & Packager

Chief Trolling Officer and Open Source Consultant @inuits.eu

Everything is an effing DNS Problem

Building Clouds since before the bookstore

Organising too many confs , #devopsdays, #loadays, ...

Evangelizing devops

Why talk about Packaging ?

devops =~ clams

Culture

(Lean)

Automate all the things ... Build Automation

Package all the things

Test Automation

IAC

Monitoring , Metrics ...

Sharing

Let's talk about Packaging

Who are you ?

Do you package ?Packaging software in a distro ?

Packaging languages ?

Packaging in an enterprise

#packagingsucks

#packagingsucks

Really ..Missing upstream

Ancient upstream

Unneeded dependencies

Broken upstream

Distro Policies

Maximum RPM ?

Packaging is needed

Dependencies, tooling, repositories

Package all the things

Why ops like to package

Packages give you features

Consistency, security, dependencies

Uniquely identify where files come from

Package or cfg-mgmt

Source repo not always available

Firewall / Cloud etc ..

Weird deployment locations , no easy access

Little overhead when you automate

CONFIG does not belong in a package

In Continuous Delivery

Unmodified , Tested artifacts go trough a pipeline. application code,Infra codemetadatatests

We need to package these so they become immutable

Deploy once vs More

Scaleout

Deploy fresh platform

No manual action to get the artifact on the platform

Languages and Packaging

CPAN,RubyGems,PIP,...

Every single language tries to package their libraries,Ops people hate those packagesAnd repackage them

Devs vs ops care about

Getting the latest versions working on their dev setup

Get that from the internet fast

A reproducable

Shippable

Artifact

Now we go to lunch while Maven downloads the InternetSome java deveveloper on the internet

Why download the internet ?

--latest

No strict versioning

Dependencies

(rubygems, php composer, .... )

Where do you version ?

'build' on production ?

It works on my machine !

What ops do ..

Build a full stack in a chroot

Deploy ruby in jruby

Software Collections

Loose more hair

Enter Omnibus

Easily create full-stack installers for your project across a variety of platforms.

Doesn't break ruby versions in /opt/

Distros and Packaging

Launch all deamons

Because in a clustered setup I want the daemon running only on the node my CRM tells it to.

Because I first want to reconfigure innodb etc before you launch it

Mostly aimed at single server setups :(

Packaging Upstream Software

Packaging Drupal 6

Packaging Drupal 7

In broken locations , a website with it's content does NOT belong in /etc/drupal6

Some Distros

Package unexisting releases

Change version numbers

Make developers hate those distros

The distro that once shipped an 2.6 Kernel version of openMosix knows I`m talking about them.

Contributing to Distros ?

Distribution packaging policies are not designed for people who package softwareJohn Vincent, on his blog in 2013

Is it a distro's job to ship all webapps in the world ?

Not all packages are equal

fpm

Yes the F stands for Effing

#packagingsucks

Missing upstream

Ancient upstream

Unneeded dependencies

Broken upstream

Distro Policies

Maximum RPM ?

Packaging is needed

Dependencies, tooling, repositories

Anger driven development

fpm

fpm

fpm -t rpm -s dir -n hornetq -v 2.2.5 hornetq Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz+ exit 0Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ /usr/lib/rpm/brp-compress+ /usr/lib/rpm/brp-strip+ /usr/lib/rpm/brp-strip-static-archive+ /usr/lib/rpm/brp-strip-comment-noteProcessing files: hornetq-2.2.5-1.x86_64Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILDWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpmWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpmExecuting(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Created /usr/local/hornetq-2.2.5.x86_64.rpm

fpm in action

https://github.com/KrisBuytaert/build-gems

Fork, pull

Jenkins pulls , builds , pushes to repo

So we 'solved' packaging, now how to ship packages ?

It works, it doesn't work

Imagine a working puppet setup

At different customers

Upstream Vendor releases new software

Half of the customers call crying that their platform is broken

A broken mcollective

Customers with self managed (package) repos are happy

Customers using upstream repos are in pain

Repository Management

Repository Management

PulpPro : MirroringLove

Con : Mongo, Stability, .deb, forge

PRM

Yum Repo Server by IS24

Version vs Latest

Version your repos ?ensure => latests

Latest your environments ?

Strict versioning in config ?Ensure => '0.98.4'Use Hiera

But it's 2014, and we have :

Packages just got bigger

If all you know is docker, every whale looks like a private cloud

Image Build by devs,
maintained by nobody

In Continuous Delivery

Immutable , Tested artifacts that go trough a pipeline.

Golden Image ,What about environmental changes

This is what we created cfgmgmt tools for !

Make sure you use docker the right way !

#devopsdays 2010 Open Space Conclusions

Always package software YOU deployExceptions: code that changes faster than you can package it. (Very rare)

Do NOT package Config FILES ,Use a cfgmgmt tool for this

Languages are still reinventing the wheel :(

Contact

Kris Buytaert Kris.Buytaert@inuits.be

Further Reading@krisbuytaert http://www.krisbuytaert.be/blog/http://www.inuits.be/

Inuits

Duboistraat 502060 AntwerpenBelgium891.514.231

+32 475 961221