Light Weight Container for Embedded Systems - iot.bzh · redpesk: CI system suite from IoT.bzh for...

redpakEmbedded Ultra Light Weight

Managed Containers

AGL AMM July 2020video: https://vimeo.com/435689572 Fulup Ar Foll

CEO IoT.bzh

https://vimeo.com/435689572

(AGL AMM July-2020)REDPAK - Ultra Light Weight Embedded Containers 2

➢ 30 Linux embedded engineers➢ Profitable since creation in 2015➢ Goal: deploy Linux in cars, boats...

● https://iot.bzh/en/● http://github.com/iotbzh● https://github.com/redpesk

LORIENT

About IoT.bzh

https://iot.bzh/en/

http://github.com/iotbzh

https://github.com/redpesk


From Hypervisor to Light Weight container


Linux Isolation Classes● Hypervisors (Xen, L4RE, Integrity, VmWare, …)

– Share/Isolate hardware resources

– Each instance duplicates a private operating system

– Huge flexibility: Linux, Windows, Android, RTOS, ...● IT Containers (Docker, LXC, NSpawn, …)

– Share kernel

– Each instance duplicates a private rootfs

– Very scalable, infrastructure elasticity, data-center scalability, …

– Usually run as ‘privileged’● Application Containers (Snap, Flatpak, AppImage, Electron, …)

– Share desktop resources (user data)

– Mostly designed to address Linux desktop fragmentation

– Usually run as ‘unprivileged’


Containers/Hypervisor for embedded systems

Image wikipedia

● Containers rely on Linux kernel capabilities✔ ++ Linux kernel is extremely robust (can run for years without a failing) ✔ ++ Linux kernel security model is widely used and proven✔ ++ Linux kernel is and will remain maintain for free for years✔ ++ Linux kernel has one and only one official tree✔ -- Linux kernel is big (Millions of lines, hundred MB of RAM)

● Embedded Hypervisor✔ ++ Small on size, easier to certify (earlier to address safety certification constrains)✔ ++ Real RtOS capabilities (hard real-time, short boot time, direct hardware access, ...)✔ -- Fragmented market : Zephyr, L4RE, Integrity, ... (no consistency, no portability, risk on long term maintenance)✔ -- Advance features useless for embedded systems (VM start/stop, CPU mitigation, Live migration, Hardware

abstraction, …)✔ -- Add a lot of complexity (two BSPs: hardware/hypervisor + mix(hypervisor+hardware)/linux)✔ -- Not design for cybersecurity


Linux Containers & Safety

● One Linux kernel is enough– Multiplying Linux kernels doesn’t help (like two engines for a 4x4 car?)– Linux kernel is rock solid (when it fails, it’s a bug and not a feature)

– Targeting embedded apps containers provide everything we need:● rootless/unprivileged like redPak for classical embedded apps/services● rooted/privileged like LXC for full subsystem as Android

● Embedded systems may still require RTOS capabilities:– Smaller and easier to certify– Fast boot– Hard real-time Constrains

Image wikipedia


Linux Containers & RTOS cohabitation● Builtin SoC virtualization (Renesas RCAR)

– Boot R7 with RTOS

– Boot Linux

– Use hardware mailboxes● Class-2 hypervisor (Jailhouse)

– Boot Linux

– Split hardware

– Boot hypervisor

– Use hypervisor mailboxes● Class-1 Hypervisor (L4RE)

– Boot hypervisor

– Split hardware

– Start RtOS activities

– Boot Linux

– Use VirtIO

Renesas R-Car M3 SoC

Aut

oSA

R C

lass

ic

R7

R-CAR M3 SoC

A53

Cor

e S

ervi

ces

Linux Kernel

Isolation / Container (Namespace, Cgroups, ...)

Light Rootless Containers (i.e. Redpack)

Ro

ote

d C

onta

ine

r(i.

e.

An

dro

id)

A53 A53 A53 A57 A57

AGL Legacy

Ap

p-1

Ap

p-2

Ap

p-3

App

-4

App

-5


redpak motivations● Provide application isolation

– Restricted filesystem visibility– Resources access/usage (API, CPU, RAM, Network, …)– Built-in security model with MAC (Mandatory Access Control)

● Maximize resource sharing & minimize system overload– No duplication of root-fs– Reuse shared libraries between instances– Restrict RAM, Disk, CPU containerization cost– Boost container startup time

● Prevent “diplomatic suitcase” container model– Strict enforcement on installed packages & dependencies– Keep the system auditable– White box container model


redpak structuring choice

● Standard package organization– No specific development tools (standard RPM, build-system, CI, ...)

– No Constraints/Limits on: sharing, hierarchy, privileges, ...

– Traditional package management (DNF, RPM repositories, Update, …)

● Enforce a static resource sharing mode– A Node only uses resources from its hierarchy (libraries, data, services, ...)

– Sibling’s nodes may only share resources through parent’s nodes

– System is predictable and fully manageable from CI

– Atomic by-Node update model


redpak structure disk / runtime

Projects- Shared- Restricted- Private

Profiles- Shared- Restricted- Private

Core OS- Shared- Restricted- Private

Platforms- Shared- Restricted- Private

Applications

redNodesHierarchy

On disk / At Runtime

Fake ROOT

/bin + /lib (RO CoreOS mount from usr/lib64+bin)

/usr (FAKE)

● lib64/qt (RO from QT-5.12 node)● /bin (RO from xxx node)

/etc (FAKE)

● /passwd (fake file build at runtime)● /resolv.conf (symlink from coreos)

/var (tmpfs)

home/app (RW mount from project node)

RedNodeRuntime Flat Representation


redpak Resources Sharing Model

● Nodes may expose resources at any level of their family tree● Nodes share resources through a common ancestor● Common Linux ACLs apply (MAC, DAC, Cgroups)● CoreOs ‘Root’ not used (new fake ‘/’ for each application)● CoreOS ‘/usr’ is typically mounted in read-only● Special directories (/etc, /var, /tmp, …) recreated on demands● Share/Unshare of namespace configurable (net, pid, ...)● Every ‘RedProcess’ runs within a private namespace


redpak Use cases

● Security enforcement– In addition to other kernel/application security mechanisms, containers can be used to re-

enforce isolation of applications/domains● Quality Assurance

– Reducing the surface of applications leads to reduced number of test to achieve QA.● Version transitioning

– Run legacy applications that require different versions of the “same” library● Modular Over-The-Air Software Update (SOTA)

– Per-node SOTA update

– RPM hierarchical dependency check

– Transaction or snapshot version rollback


/CoreOS

/Platform LIMITED CPU/RAM

/Profile NO NETWORK

/Project NO BINARIES

/Application: RESTRICTED

Simplify QA & Reduce attack surface

● The profile NO BINARIES removes accesses to:– /bin– /usr/bin– /lib/libdl.so

● Restricted applications can only run their own code


Multiple Frameworks & Platforms

/CoreOS (Redpesk - LTS-8)

Adaptive AutoSAR/Platform (AGL)

/Profile-HTML5 /project-AutoSAR

Au

toS

AR

App

s

/Project

Ele

ctro

n A

pps

/Profile-QT

/Project

QT

AG

L A

pps

Legato

/legacy-project

Lega

cy A

pps

Proprietary Framework

/custom-project

Pro

prie

tary

Ap

ps


Multiple Flavor/Version of componant

/CoreOS (Redpesk - LTS-8)

/Platform (AGL - GG)

/Profile-HTML5

LIBS QT 5.12

QT

New Apps

/Profile-Qt-Legacy

/Project-C

LIBS QT 5.10

Old Apps

LIBS QT 5.12

New Apps

/Profile-Qt-New

/Project-B

LIBS QT 5.12

New AppsNew Apps

/Project-B

Electron Apps

Chromium SDK


Node Atomic Software Update

● Per-node update● Direct update with DNF/RPM● Update as binary tree (ostree)● Hierarchical per node RPM data base

● Update Rollback● DNF transaction● LVM snapshot● Full node reinstallation

● Standard I/O organisation● Compatible with Mender.io or other SOTA tools

Imag

e w

ikip

edia


Container Latency OverheadContainer Type x86_64: Intel Minnow (ms) aarch64: Renesas M3 (ms)

Bwrap (run only) 37 50

redpak (run only) 48 59

redpak (config+run) 110 75

LXC (not rootless) 172 104

Runc (rootless) 332 212

Nspawn (not rootless) 1099 386

Benchmark notes: ● I/Os are slower on Minnow than on RCAR/M3: this explains config overhead.● nspawn spinlock ~1s on every tested Intel boards (probably a bug on init config)● adding an extra level to redpak hierarchy adds 1-2ms per level to config build time● bwrap has no config: redpak startup time changes if config needs to be rebuilt or not


Bench/PoC outcome

● Linux kernel widely address embedded application constrains● Overhead startup time and memory footprint remains acceptable for most usecases ● Scalable containers capabilities (un/privileges, root/rootless, configuration share/unshare)

● Native support for resource management (cgroup), Cybersecurity (MAC, DAC, ...)

● Kernel community proved long term maintenance

● Realtime & Certification still require a smaller footprint ● SoC (CPU, GPU, RAM, GPIO, ...) native isolation (i.e Renesas R-CAR)● Detach hardware from Linux (i.e. Jailhouse)● Split hardware with a minimalist level-1 hypervisor (i.e. L4RE)


redpak / redpesk

● redpesk: CI system suite from IoT.bzh for AGL LTS– Handles projects and teams

– Builds RPMs and publish binary feeds

– Run the tests and notify QA reports

– Creates updates

● redpak: IoT.bzh package management tool– Takes RedPesk RPM repositories as input

– Installs/updates RPM packages on the target

– Creates the “share/unshare” namespace to isolate services and applications

Both systems are designed to collaborate. Nevertheless, they may also operate independently.


redpak Roadmap

● POC completed ● Source https://github.com/redpesk/red-pak● Demo https://vimeo.com/412131602

● Summer-2020● Remove Python dependencies (Current Stable LibDNF)● Rebuilt Red-DNF on top to libdnf-5 (FCS version end-2020)● Finish Red-RPM cleanup to remove DNF dependencies

● Q3-2020●

Complete Integration with redpesk – Configuration management– On the air update

● Q1-2021● FSC version fully integrated with redpesk

https://github.com/redpesk/red-pak



Links

● redpak:

– Intro: https://vimeo.com/435689572

– Demo: https://vimeo.com/412131602

– Sources: https://github.com/redpesk/red-pak

● redpesk:

– Website: https://redpesk.bzh/

– Documentation: https://docs.redpesk.bzh/

– Sources: https://github.com/redpesk● IoT.bzh:

– Website: https://iot.bzh/

– Publications: https://iot.bzh/en/publications

– Videos: https://vimeo.com/search?q=redpesk● Citroen Sahara 2CV 4X4

– Paper by Kurt Ernst [here]



https://github.com/redpesk/red-pak

https://redpesk.bzh/

https://docs.redpesk.bzh/

https://github.com/redpesk

https://iot.bzh/

https://iot.bzh/en/publications

https://vimeo.com/search?q=redpesk

https://www.hemmings.com/stories/2016/06/02/is-citroens-2cv-4x4-sahara-the-most-innovative-off-roader-ever-built

Light Weight Container for Embedded Systems - iot.bzh · redpesk: CI system suite from IoT.bzh for...

Documents

Transcript of Light Weight Container for Embedded Systems - iot.bzh · redpesk: CI system suite from IoT.bzh for...