Ottimizzare i tempi di boot di Linux

Agenda

Overview

Case study: Raspberry Pi

Kernel optimizations

rootfs optimizations

Q/A

Why a faster boot?

Consumer electronics products require very fast boot times (digital camera, TV, mobile phone, etc.)

Fast recovery upon system failures (crashes)

You can show to your friends your new awesome board booting in a couple of seconds ;-)

Overview of boot phases

Firmware (boot-loader)Hardware probing

Hardware initialization

Kernel load and decompression

Kernel executionCore init (start_kernel)

Driver init (initcalls)

User-space init/sbin/init

RC scripts

Application start (first user impression)

Generic concepts

Identify boot time functionalities

Measure boot time of each functionality

Remove unnecessary functionalities

Optimize required functionsDefer loading of less-important features (modularization - LKM)

Re-order initialization / parallelizationAsynchronous initialization

Generic concepts

Identify boot time functionalities

Measure boot time of each functionality

Remove unnecessary functionalities

Optimize required functionsDefer loading of less-important features (modularization - LKM)

Re-order initialization / parallelizationAsynchronous initialization

Premature optimization is the root of all evil. -Donald Knuth

Case study

Raspberry Pi

Boot time functionalityReponsive ssh login

Tools and techniques used to improve boot time

Focusing on cold-boot optimizations

Boot steps

RPi boot steps:1st-stage boot loader from SoC BCM2835 ROM

bootcode.bin: 2nd-stage boot loader from external SD card, starts the GPU

start.elf: GPU firmware, starts the CPU

kernel.img: Linux

rootfs: external SD card

application: ssh (dropbear)

Measure boot-time functionality (kernel)

CONFIG_PRINTK_TIMEConfigure it in Kernel hacking section

Adds timing informations to kernel messages (dmesg)

# dmesg...[ 0.022030] initcall bcm_mbox_init+0x0/0x38 returned 0 after 0 usecs[ 0.022054] calling bcm_power_init+0x0/0xa4 @ 1[ 0.022065] bcm_power: Broadcom power driver[ 0.022080] bcm_power_open() -> 0[ 0.022090] bcm_power_request(0, 8)[ 0.522766] bcm_mailbox_read -> 00000080, 0[ 0.522783] bcm_power_request -> 0[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

Kernel initcall tracer

Introduced in Linux 2.6.28

Add initcall_debug to the kernel boot options

Allows to record the timings of each initcall in dmesg

# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 | tail -5[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

Kernel initcall tracer (example)

dmesg > dmesg.log

cat dmesg.log | perl scripts/bootgraph.pl > /boot/dmesg.svg

Raspbian

http://www.raspbian.org

General-purpose distro based on Debian

Optimized for the Raspberry Picompilation settings adjusted to produce hard-float code

2013-05-25-wheezy-raspbian.img

Raspbian: boot time

# dmesg...[ 3.107163] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24[ 5.664780] EXT4-fs (mmcblk0p2): recovery complete[ 5.677326] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)[ 5.689427] VFS: Mounted root (ext4 filesystem) on device 179:2.[ 5.700523] devtmpfs: mounted[ 5.706028] Freeing init memory: 128K[ 6.257687] init start[ 7.337703] udevd[154]: starting version 175[ 8.511002] Registered led device: led0[ 16.041293] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 16.522927] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 25.553157] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1[ 28.672178] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:507900k SS[ 30.436046] init done

- Shell after ~6.3sec

- SSH after ~30.5sec

sysv-rc-conf

Run-level configuration for SysV like init scriptsdisable npt

disable plymouth

disable rsync

disable x11-common and lightdm

disable alsa-utils

disable swap (dphys-swapfile)

disable checkfs

disable ntp

Static IP vs DHCP

Assign a static IP to the board (save the time to get an IP via DHCP)

Disable CONFIG_IP_PNP in the kernel .configUsed to mount rootfs via NFS

Raspbian: boot time after simple optimizations

# dmesg...[ 3.132731] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24[ 5.730131] EXT4-fs (mmcblk0p2): recovery complete[ 5.742783] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)[ 5.754881] VFS: Mounted root (ext4 filesystem) on device 179:2.[ 5.765958] devtmpfs: mounted[ 5.771460] Freeing init memory: 128K[ 6.310549] init start[ 7.389321] udevd[154]: starting version 175[ 8.536932] calling leds_init+0x0/0x60 [led_class] @ 229[ 8.537065] initcall leds_init+0x0/0x60 [led_class] returned 0 after 82 usecs[ 8.543802] calling gpio_led_driver_init+0x0/0xc [leds_gpio] @ 229[ 8.544067] Registered led device: led0[ 8.545242] initcall gpio_led_driver_init+0x0/0xc [leds_gpio] returned 0 after 1344 usecs[ 14.724169] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 15.202445] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 24.493346] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1[ 24.991030] init done

About 25 sec to login via ssh, instead of 30 sec (16% faster)

Build a minimal distro from scratch: requirements

build toolchain (gcc, glibc, binutils)git://github.com/raspberrypi/tools.git

HINT: use hard-float toolchain (arm-bcm2708hardfp-linux-gnueabi)

Linux (kernel)git://github.com/raspberrypi/linux.git

busybox (rootfs)git://busybox.net/busybox.git

dropbear (ssh server)https://matt.ucc.asn.au/dropbear/

Kernel initcalls (before optimization)

# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8...[ 0.810927] initcall iscsi_transport_init+0x0/0x154 returned 0 after 446 usecs[ 1.225990] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 451 usecs[ 1.232472] initcall pm_qos_power_init+0x0/0xac returned 0 after 713 usecs[ 1.230686] initcall sysctl_ipv4_init+0x0/0x98 returned 0 after 755 usecs[ 0.810406] initcall vchiq_init+0x0/0x1dc returned 0 after 1565 usecs[ 1.228797] initcall sdhci_drv_init+0x0/0xc returned 0 after 1829 usecs[ 0.560116] initcall inet_init+0x0/0x260 returned 0 after 2739 usecs[ 0.808747] initcall loop_init+0x0/0x11c returned 0 after 4852 usecs[ 0.803722] initcall brd_init+0x0/0x1c0 returned 0 after 8230 usecs[ 0.540586] initcall genhd_device_init+0x0/0x84 returned 0 after 9765 usecs[ 0.556774] initcall chr_dev_init+0x0/0xd8 returned 0 after 12234 usecs[ 0.538982] initcall param_sysfs_init+0x0/0x1d4 returned 0 after 19531 usecs[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

Kernel optimizations
(initcalls)

disable kprobes (CONFIG_KPROBES): ~95ms

reduce the number of PTYs: ~90msCONFIG_LEGACY_PTY_COUNT=256 => CONFIG_LEGACY_PTY_COUNT=1

disable framebuffer: ~33ms

remove loop device: ~5ms

Kernel initcalls (after optimization)

# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8...[ 0.570870] initcall tun_init+0x0/0x8c returned 0 after 259 usecs[ 0.568035] initcall des_generic_mod_init+0x0/0x10 returned 0 after 341 usecs[ 0.975476] initcall bcm2835_cpufreq_module_init+0x0/0xc returned 0 after 358 usecs[ 0.569678] initcall pty_init+0x0/0x194 returned 0 after 375 usecs[ 0.561542] initcall vc_mem_init+0x0/0x1b4 returned 0 after 393 usecs[ 0.974729] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 393 usecs[ 1.018751] initcall deferred_probe_initcall+0x0/0x6c returned 0 after 395 usecs[ 0.561078] initcall bcm2708_gpio_init+0x0/0xc returned 0 after 405 usecs[ 1.018028] initcall pm_qos_power_init+0x0/0x60 returned 0 after 506 usecs[ 0.559402] initcall inet_init+0x0/0x260 returned 0 after 1540 usecs[ 1.021006] initcall net_secret_init+0x0/0x1c returned 0 after 2145 usecs[ 1.024343] initcall initialize_hashrnd+0x0/0x1c returned 0 after 3052 usecs[ 0.557293] initcall chr_dev_init+0x0/0xd8 returned 0 after 10978 usecs[ 0.541305] initcall param_sysfs_init+0x0/0x19c returned 0 after 19531 usecs[ 1.016084] initcall sdhci_drv_init+0x0/0xc returned 0 after 39250 usecs[ 0.973739] initcall dwc_otg_driver_init+0x0/0xc4 returned 0 after 392536 usecs[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

Kernel optimizations
(other optimizations)

preset loops_per_jiffyAt each boot the kernel calibrates a delay loop (used later by the udelay() function)

1 jiffy = time between 2 timer interrupts

CONFIG_HZ=100 => 250ms!!!

loop

disable console outputremove console=xxx and add quiet to the kernel boot parameters

use LZO kernel decompression (CONFIG_KERNEL_LZO)LZO is a compression algorithm that is much faster than gzip, at the cost of a slightly degrade compression ratio (+10%)

reduce kernel size...A smaller kernel is faster to load, less code also means smaller working set (good for caches)

Kernel image (before => after)

rootfs

Generated using busybox and dropbear

Hints about busybox (reduce fork()s):CONFIG_FEATURE_SH_STANDALONE: use applets instead of fork/exec/wait external binaries

CONFIG_FEATURE_SH_NOFORK: call _main() directly without spawning another task

Build everything with -Os

Use mklibs to strip system libraries (rootfs is mounted to /mnt)mklibs -v -D -d lib2 -L /mnt/lib --ldlib lib/ld-linux.so.3 --target=arm-bcm2708-linux-gnueabi /mnt/bin/* /mnt/sbin/* /mnt/usr/sbin/* /mnt/usr/bin/*; rm -rf lib; mv lib2 lib

After all these stops total rootfs size is ~3.2MB

filesystem optimizations

Split the filesystem into read-only portion and read/write portionRead-only filesystem mounts faster

Use Squashfs + tmpfs (or aufs)

root filesystem: boot time

Kernel image (before => after)

1.5GB

3.2MB

Demo: custom kernel+rootfs boot time

...[ 1.144025] Freeing init memory: 92K[ 1.228550] init start[ 1.249067] waiting eth0 to come up[ 1.345882] usb 1-1: new high-speed USB device number 2 using dwc_otg[ 1.346059] Indeed it is in host mode hprt0 = 00001101[ 1.556942] hub 1-1:1.0: USB hub found[ 1.557074] hub 1-1:1.0: 3 ports detected[ 1.836029] usb 1-1.1: new high-speed USB device number 3 using dwc_otg[ 1.940066] smsc95xx v1.0.4[ 1.951858] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24[ 2.086695] init done[ 3.406004] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1

- Shell after ~1.2 sec

- SSH after ~3.4 sec

References

https://github.com/arighi/rpi-mini-distro

LPC: Booting Linux in 5 Seconds - Arjan van de Ven's talk: http://lwn.net/Articles/299483/

elinux wiki - Tim's fastboot tools:
http://elinux.org/Tims_Fastboot_Tools

elinux wiki- RPi Kernel Compilation: http://elinux.org/RPi_Kernel_Compilation

Update on boot time reduction techniques - Michael Opdenacker

Linux: http://www.kernel.org

Busybox: http://busybox.net

Dropbear: https://matt.ucc.asn.au/dropbear/dropbear.html

Raspberry Pi: http://www.raspberrypi.org

http://it.emcelettronica.com/embedded-gnulinux-partendo-da-zero-test-sulla-raspberry-pi

Q/A

You're very welcome!

Twitter@arighi

#bem2013

Andrea Righi - andrea@betterlinux.com