June 20–23, 2016 | Berlin, Germany

OPNFV on ARM Hardware Freedom of Choice Has Arrived

Florin Dumitrașcu, ENEA

Armband Project Scope

Add support for ARM Architecture-based servers in OPNFV

Drive population of ARM-based servers in OPNFV Labs

Work with upstream community to release OPNFV on multi-architectures, on any relevant silicon required by the industry

What we do …

3OPNFV on ARM, OPNFV Summit 2016

The Team for Brahmaputra on ARM

4OPNFV on ARM, OPNFV Summit 2016

Armband Project Numbers

Commits per repository Authors per repository

Main committers to date: ENEA, Cavium, ARM

Source: http://projects.bitergia.com/opnfv/browser/scm-repos.html?page=1

5OPNFV on ARM, OPNFV Summit 2016

Armband Project Facts

o Development

• Git: https://git.opnfv.org/cgit/armband/

• Gerrit: https://gerrit.opnfv.org/gerrit/#/admin/projects/armband

• Jira: https://jira.opnfv.org/projects/ARMBAND/issues/

o Testing

• Dashboard: https://testresults.opnfv.org/dashboard/# (Fuel, arm-pod1)

o Continuous Integration / Continuous Deployment

• Jenkins: https://build.opnfv.org/ci/view/armband

o Artifacts for Brahmaputra

• ISO, documentation: http://artifacts.opnfv.org/armband.html

o Communication

• Wiki: https://wiki.opnfv.org/display/armband

• IRC Channel: #opnfv-armband @ Freenode

6OPNFV on ARM, OPNFV Summit 2016

ENEA Pharos Lab (Kista, Sweden)

• Applied Micro X-Gene 2, ARMv8-64 8 cores @ 2.4GHz• 32GB RAM , 1x128GB SSD, 2x1TB HDD• 1x10Gbps SFP+ NICs, 2x1Gbps NICs (RJ45)• AMD Opteron A1100, ARMv8-64 Cortex A57, 8 cores• 2 x 16GB RAM (32 GB total), 1 TB HDD• 2 x 10Gbps NIC

https://wiki.opnfv.org/display/pharos/Enea-pharos-lab

• Cavium Networks CN8890-CRB 1S ThunderX• ARMv8.1-64 48 cores @ 1.8GHz• 8 x 16GB RAM (128GB total), 1 x 500GB HDD• 1 x 40Gbps QSFP+ NIC, 2 x 10Gbps SFP+ NICs, 1 x 1Gpbs NIC

(RJ45, IPMI interface)• Cavium Networks CN8890-CRB 1S ThunderX• ARMv8.1-64 48 cores @ 1.8GHz• 8 x 16GB RAM (128GB total), 1 x 500GB HDD• 1 x 40Gbps QSFP+ NIC, 2 x 10Gbps SFP+ NICs, 1 x 1Gpbs NIC

(RJ45, IPMI interface)

7OPNFV on ARM, OPNFV Summit 2016

Armband Workflow

clone repository

make submodules-init

make patches-import

modify and commit

make build

make patches-export

make submodules-clean

(create submodule)

1

2

3

4

5

6

7

8

$ git clone ssh://<Linux Foundation User>@gerrit.opnfv.org:29418/armband

$ git submodule -b stable/8.0 add --name fuel-web \https://github.com/openstack/fuel-web.git upstream/fuel-web

Apply patches from patches/<sub-project>/* to respective submodules in/upstream/<sub-project>. Branch “armband-workbench” created in submodules.

All the sub-projects are registered as git submodules.Call make submodules-init to initialize them.

Modify sub-projects for whatever you need. Commit your changes when you want them taken into account in the build.

Build to validate changes.

Each commit on “armband-workbench” branch of each sub-project will be exported to the patches /<sub-project>/ via “git format-patch”. Only commit patches !

Clean workbench branches.

8OPNFV on ARM, OPNFV Summit 2016

Armband Fuel Installer

o Patching upstream repositories for ARM • OpenStack Fuel• Fuel@OPNFV• Fuel plugins

o Using Git Submodules• submodules defined in .gitmodules file• url – upstream repository • branch – lock on upstream branch• upstream – path for local workspace (clone)

o Pros• Not an explicit upstream fork• Organize patches • Patch files before building• Gerrit automatically pulls upstream changes into

submodules (requires patch rebasing)

o Cons• Overhead for simple patches• Cannot pin Fuel to a tag like Fuel@OPNFV does

Source: https://git.opnfv.org/cgit/armband/tree/.gitmodules

9OPNFV on ARM, OPNFV Summit 2016

/armband

Armband Directory Structure

/armband/patches

/armband/upstream Source: https://git.opnfv.org/cgit/armband/tree

/armband/patches/fuel-library

10OPNFV on ARM, OPNFV Summit 2016

aptly create mirror

Armband Package Repositories (1)

Local mirror of fuel-infra repository

Upstream fuel-infrarepository

http://mirror.fuel-infra.org/mos-repos/ubuntu/

aptly update mirror

Local fuel-infrapackage repository

aptly repo importaptly repo create

Snapshot of localfuel-infra pkg repository

aptly snapshot create

Published local fuel-infra package repository

(internal use)

http://<local-ip>/repo/mirantis/mos-repos/ubuntu/

aptly publish snapshot

Published snapshots archive

(internal use)

http://<local-ip>/repo/archive

Contains both binary-amd64 and binary-arm64 packages. However, in this repository, only non-native packages (“Architecture: all”) are contained for binary-arm64 (e.g. “nova-api”)

11OPNFV on ARM, OPNFV Summit 2016

Native fuel-infra pkg built on arm64, e.g. “ceph”

aptly repo add

gbp-buildpackage(build/patch/version bump)

Native arm64 local fuel-infra packages repository

Snapshot of arm64 localfuel-infra pkg repository

aptly snapshot create

Armband arm64 ports packagesE.g. “kernel, qemu, grub-efi-arm64”

aptly repo add

gbp-buildpackage(arm64 specific ports)

Armband arm64 local packages repository

Snapshot of Armband arm64 local pkg repository

aptly snapshot createPublished local fuel-

infra package repository(internal use)

binary-arm64 / Architecture: arm64(e.g. “ceph, mongodb, galera”)

Armband Package Repositories (2)

binary-amd64 / Architecture: all, amd64binary-arm64 / Architecture: all(e.g. “nova-api”)

binary-arm64 / Architecture: arm64(e.g. “kernel, qemu, grub-efi-arm64”)

Published Armband package repository

(public)

aptly snapshot merge

12OPNFV on ARM, OPNFV Summit 2016

Armband Ubuntu Trusty Ports

Component ARM64 Notes

ARM64 Kernel 4.2.0 re-packaged from Yarmouth repositories.One kernel that currently supports ALL our arm64 targets: Cavium, APM, AMD.

GRUB-EFI-ARM64 Bootstrapping nodes from Fuel's Cobbler requires an arm64 netloader, compatible with syslinux like config (e.g. using MAC address as config name).

DMIDECODE dmidecode in Trusty does not support all arm64 targets properly, repackage with fixes from upstream repository.

OHAI Add arm64 CPU details detection. Add ethtool support for reading L1 info.

QEMU Bump QEMU to 2.5 for multiple arm64 fixes and newly added support.

UDEV Provide a predictable eth naming scheme for arm64 (aarch64) boards withon-SoC ethernet devices, like APM Mustang, AMD Softiron.

LIBGUESTFS Update to libguestfs 1.32.2 and repackage for Trusty. Fixes lots of dependecies for arm64.

LSHW Fix Ubuntu Trusty bug on arm64: "AArch64: slow cpuinfo due to redundant loop“.

OPNFV-QUAGGA Build opnfv-quagga required by ODL plugin on Ubuntu Trusty arm64 build.

13OPNFV on ARM, OPNFV Summit 2016

Fuel-infra ports for arm64 architecture

Component ARM64 Notes

MYSQL Version bump to MySQL 5.6.28 and add wsrep patch 25.13 for Galera support.

GALERA Version bump to Galera 3-25.3.14.

LIBVIRT Version bump to libvirt 1.3.2.Rebase libvirt required by Qemu 2.5 on Ubuntu Trusty arm64 build GIC support.Force GIC version detection by default on ARMv7/8.

CEPH Add arm64 architecture to debian/control (for ceph-fuse[-dbg]).

CIRROS Upstream package available for arm64. Add it to Armband fuel-infra repository.

MONGODB Enable arm64 Java Script (V8) support on arm64.

14OPNFV on ARM, OPNFV Summit 2016

Armband Functest

o Deployment scenarios• Currently running os-odl_l2-nofeature-ha scenario only.• Work in progress to activate os-nosdn-nofeature-ha, os-odl_l3-nofeature-ha.

o Tempest• Using Cirros TestVM for arm64.• Live instance snapshot and VM live migration require kernel and qemu

patching for GICv3 on ThunderX targets (affects 15 test cases).

o Rally• Fix Rally to allow customization of RAM size for instance flavors.

These were hardcoded to 64M, 128M causing out of memory on arm64 in some scenarios (https://review.openstack.org/#/c/315171/).

o vIMS• Live test working with vIMS instantiated manually.• Work in progress to port Cloudify on arm64 CentOS for vIMS orchestration.

o Yardstick• Work in progress to port test images and tools on arm64.

15OPNFV on ARM, OPNFV Summit 2016

On the way to Colorado Release

o Migrate Armband Fuel installer from Brahmaputra (Liberty) to Colorado (Mitaka)• Rebase Fuel components in Armband git repository.

• Rebase OPNFV components in Armband git repository.

• Update Armband package repository.

o Validate and extend where needed OpenStack Mitaka functionality on arm64• Fix bugs and workarounds reported in Armband release notes aligned to Brahmaputra 3.0.

• Cherry-pick upstream fixes (if made available) for problems reported in same notes.http://artifacts.opnfv.org/armband/brahmaputra/docs/release-notes.html

o Yardstick support• Modify the script to build composite x86 + arm64 image with all packages needed by Yardstick.

• Validate ARM compatibility – tools, etc.

o Full orchestration of Clearwater vIMS with Cloudify for Functest• Build CentOS7 arm64 image with everything required by Cloudify.

• Integrate in Functest.

16OPNFV on ARM, OPNFV Summit 2016

Upstream challenges

o Unify x86 and arm64 packages in a single mirror

o Fuel plugin builder currently using lots of hardcoding

o Common Fuel build (Fuel master) for x86 and arm64 environment deploys

17OPNFV on ARM, OPNFV Summit 2016

Questions

OPNFV on ARM, OPNFV Summit 2016 18

Thank you !