Building a GNU/Linux distributionwith DevOps in mind

Daniel Dehennin

Ple de Comptences Logiciels Libres

OpenNebulaConf 2016cc by-sa 4.0

1 / 25

http://pcll.ac-dijon.fr/pcllhttp://2016.opennebulaconf.com/http://creativecommons.org/licenses/by-sa/4.0

Ple de Comptences Logiciels Libresfoss and agility in french Minister of National Education

Original mission EOLE GNU/Linux meta-distributionCeCILL / GPL software licensingAgile consulting for other development teams

2 / 25

https://en.wikipedia.org/wiki/Free_and_open-source_softwarehttps://en.wikipedia.org/wiki/Agile_software_developmenthttps://en.wikipedia.org/wiki/Ministry_of_National_Education_(France)https://fr.wikipedia.org/wiki/EOLE_(Linux)http://www.cecill.infohttps://www.gnu.org/licenses/#GPL

Why did we get to OpenNebula?bare metal elastic limit is too low

Testing our OS was done on physical desktop computersSome lucky developers could have at most 2 VMs on theirworkstation

.

...... EOLE development needed elasticity

3 / 25

Looking for virtualisation infrastructuremany choices: too big, not enough flexible or immature

2012: two new quite powerful workstations testing party

Proxmox needed a reboot to add a new networkArchipel barely emergedGaneti was promisingOpenStack was already too much

.

...... Started with OpenNebula 3.8

4 / 25

First useslocal workflow on central servers

Each user was responsible for its own infrastructureTeam documentation with conventions

.

...... Everybody was admin

.

...... Load on higher skilled users

5 / 25

Emerged use cases and needs

Short lifetime VMsFactorise VM templates and imagesReduce usage cost for usersReproducible environments for QAUse production like environments for dev

6 / 25

Complete virtual infrastructure per user

7 / 25

Complete virtual infrastructure per userworkaround #2125

VLAN isolated networksStandard network namesOne set of networks per userOne user = one gateway

.

...... Sharing VM templates requires avoiding UNAME on networks

8 / 25

http://dev.opennebula.org/issues/2125

Automatic user environment generationmanual creation of 21 times 25 networks is not an option

custom contextualisation of the gateway (per user IP)generate/update virtual networks

9 / 25

Pilot OpenNebula from Jenkinscontinuous integration of OS

Check installation from ISOCheck daily upgrade to find broken packagesCheck default configurationsCheck user data import

.

...... Produce VM images and templates at each step

10 / 25

Pilot OpenNebula from Jenkinscontinuous integration of OS

11 / 25

Pilot OpenNebula from Jenkinssome numbers

120 VMs per night 90k VMs since 2014 3TB of qcow2 images (25TB virtual)

12 / 25

Jenkins jobs produce ready to use VMsreduce environment setup time

13 / 25

Development is so simpleuntil an IA will make our work

Preparing the coding session

...1 Start an infrastructure

...2 Create a branch of your repository

...3 Clone the repository on the VM

14 / 25

Development is so simpleuntil an IA will make our work

Hack until it works

...1 Code locally on your workstation

...2 Pull the changes on the VM

...3 make install

...4 Test and cycle to ...1 until it works

15 / 25

Development is so simpleuntil an IA will make our work

Test like a user

...1 Cleanup your local branch

...2 Merge, push and make a package

...3 Cleanup the VMDELETE-RECREATE (< 5.0)Revert to initial disk snapshot ( 5.0)

...4 Upgrade OS new package

...5 Test....... Jenkins jobs will install the new packages during next night

16 / 25

QA campaignAutomation is not the panacea

Squash-TMTargeted environmentsCritical features

17 / 25

Physical limitationstest bed was burning

40k VMs

The two dedicated workstations was fine for testingWorkload was memory bounded bumped to 2x32GB

.

...... NFS access on workgroup NAS was too slow

18 / 25

New hypervisor nodesblade runners

First VDCTwo servers48 cores384GB RAM

Second VDCOne server8 cores42GB RAM

19 / 25

Storage on SANthe corosync/pacemaker/cLVM/GFS2 sandwich

20 / 25

Storage on SANEverybody has a gun and too many want to use it

.

...... corosync/pacemaker can be challenging

.

...... Hot/cold storage I/Os not stripped on all LUNs

21 / 25

The future

Replace home made orchestration code with SaltStackHost other teamsCeph storageDocker uses

22 / 25

The killer features of OpenNebulayou cant make us use something else without them

Low load on the teamWe can adapt to our use cases and contribute

Features 3/17Bugs 4/46

.

...... Muchas gracias OpenNebula Systems

23 / 25

http://dev.opennebula.org/projects/opennebula/issues?utf8=%E2%9C%93&set_filter=1&f[]=author_id&op[author_id]=%3D&v[author_id][]=1416http://opennebula.systems

ThanksMany thanks to the FOSS community for all the great software. So fewthings would exists without them.

This talk was realised with the help of the following libre software:

Composition system LATEX TeX LiveThe most powerful text editor available today GNU/EmacsThe Awesome window managerThe Universal Operating System Debian GNU/Linux

24 / 25

https://en.wikipedia.org/wiki/TeX_Livehttps://www.gnu.org/software/emacshttp://awesome.naquadah.org/http://www.debian.org/http://www.gnu.orghttp://www.kernel.orghttp://www.debian.org/https://www.gnu.org/software/emacshttp://awesome.naquadah.org/https://fr.wikipedia.org/wiki/TeX_Live

Licence

The slides are licensed under Creative Commons by-sa 4.0

Attribution Share alike

You can obtain a copy of the license.by Internet........http://creativecommons.org/licenses/by-nc-sa/4.0

.by snail mail..

......

Creative Commons444 Castro Street, Suite 900 Mountain View,California, 94041, USA.

25 / 25

http://creativecommons.org/licenses/by-sa/4.0http://creativecommons.org/licenses/by-nc-sa/4.0

Ple de Comptences Logiciels LibresWhy did we get to OpenNebula?Test bed infrastructureLooking for virtualisation infrastructureFirst usesEmerged use cases and needsComplete virtual infrastructure per user

AutomationGenerate user environmentPilot OpenNebula from JenkinsJenkins jobs produce ready to use VMs

Development and test is so simplePreparing the coding sessionHack until it worksTest like a user

QAProduction infrastructurePhysical limitationsNew hypervisor nodesStorage on SAN

The futureThe killer feature of OpenNebulaAppendixThanksLicence