Effective service and resource management with systemd
-
Upload
david-timothy-strauss -
Category
Technology
-
view
2.009 -
download
3
Transcript of Effective service and resource management with systemd
Effective Service + Resource Management with systemd
Adventures running millions of systemd services for
About Me and Pantheon
● Production users
of systemd since 2011
● Millions of units in
deployment across hundreds
of servers
● Committer since 2012
● Focus has been on journal
logging, control group
scalability, and general
systemd scalability
The Basic Steps
1 Define expected behavior and control
2 Plan for the unexpected
3 Tighten security
4 Manage, monitor, and automate
Service Types
1 Define expected behavior and control
Type=simple (the default)
systemctl start foo.service systemctl stop foo.service
ExecStart=/usr/bin/foo
/etc/systemd/system/foo.service
Considered started for dependencies
Considered stopped for dependencies
[Service]ExecStart=/usr/bin/foo
# systemctl daemon-reload
Type=oneshot
systemctl start foo.service systemctl stop foo.service
*Unless RemainAfterExit=true
*ExecStart=/usr/bin/foo
[Service]Type=oneshotExecStart=/usr/bin/fooRuntimeMaxSec=30
/etc/systemd/system/foo.service
RuntimeMaxSec=30
Type=forkingsystemctl start foo.service
systemctl stop foo.service
ExecStart...
PIDFile=/var/run/foo.pid
[Service]Type=forkingExecStart=/usr/bin/fooPIDFile=/var/run/foo.pidTimeoutStartSec=30
/etc/systemd/system/foo.service
TimeoutStartSec=30
Type=notify
systemctl start foo.service systemctl stop foo.service
ExecStart...
[Service]Type=notifyExecStart=/usr/bin/fooTimeoutStartSec=30NotifyAccess=all ⬅maybe
/etc/systemd/system/foo.service
Called from daemon:systemd-notify --ready
Best of All
Types
Service Shutdown and Reloading
1 Define expected behavior and control
KillMode=control-group (the default)
systemctl stop foo.service
[Service]ExecStart=/usr/bin/fooKillMode=control-groupTimeoutStopSec=30
/etc/systemd/system/foo.service
PID=100
101
102
103
…or “Oprah’s Favorite Signals”
SIGTERM
PID=100
101
102
103
SIGKILL
TimeoutStopSec=30
KillMode=none
systemctl stop foo.service
[Service]ExecStart=/usr/bin/fooKillMode=noneExecStop=/usr/bin/fooctl stop
/etc/systemd/system/foo.service
PID=100
101
102
103
PID=100
101
102
103
No CleanupExecStop=/usr/bin/fooctl stop
KillMode=process
systemctl stop foo.service
[Service]ExecStart=/usr/bin/fooKillMode=process
/etc/systemd/system/foo.service
PID=100
101
102
103
SIGTERM PID=100
101
102
103
No Cleanup
KillMode=mixed
systemctl stop foo.service
[Service]ExecStart=/usr/bin/fooKillMode=mixedTimeoutStopSec=30
/etc/systemd/system/foo.service
PID=100
101
102
103
SIGTERM PID=100
101
102
103
SIGKILL
TimeoutStopSec=30
Best for
Most
ExecReload=
systemctl reload foo.service
[Service]ExecStart=/usr/bin/fooExecReload=/bin/kill -HUP $MAINPID
/etc/systemd/system/foo.service
Use Me
ExecReload=/bin/kill -HUP $MAINPID
Dependencies and Transactions
1 Define expected behavior and control
WantedBy=
Implicit in late bootup:systemctl start multi-user.target
[Service]ExecStart=/usr/bin/foo
[Install]WantedBy=multi-user.target
/etc/systemd/system/foo.service
Use Me
# systemctl enable foo.service
Added to transaction by wants:systemctl start foo.service
multi-user.target completes startup
Operations in systemd happen in transactions, which are ordered sets of jobs.
…the successor to runlevels
Other DependenciesInclusion
These dependencies will add more units to a
transaction. There is no effect on ordering.
● Requires=bar.service
○ If foo.service is starting, starting bar.service
will also happen. A failure to start bar.service
will cause the entire transaction to fail.
○ Inverse of RequiredBy=
● Wants=bar.service
○ A weak form of Requires=. If bar.service fails
to start, the transaction will still succeed.
○ Inverse of WantedBy=
● Also=bar.service
○ When foo.service is enabled to start by
default, bar.service will also be enabled.
Ordering
These dependencies will order units in the
transaction. They will not add specified units if
they are not already in the transaction.
● Before=bar.service
○ If bar.service is in the same transaction, bar.
service will not begin starting until foo.
service is finished starting.
● After=bar.service
○ If bar.service is in the same transaction, foo.
service will not begin starting until bar.
service is finished starting.
[Unit]Requires=bar.serviceAfter=bar.service...
/etc/systemd/system/foo.service
Controlling Resources
1 Define expected behavior and control
Control Groups Options for ResourcesAbsolute Limits
● MemoryLimit=
○ Caution: Certain limits cause further
allocation for a group to use swap, impacting
system performance.
● TasksMax=
○ Maximum combined processes and threads,
including kernel threads.
● BlockIOReadBandwidth=
○ Limits reading block I/O to the specified
bytes
per second.
● BlockIOWriteBandwidth=
○ Limits writing block I/O to the specified
bytes
per second.
Relative Controls and More
● CPUShares=
○ When under contention, CPU is allocated by
the kernel proportionally using the number
for this service versus the combined shares of
all others.
● BlockIOWeight=
○ When under contention, block I/O is
allocated by the kernel proportionally using
the number for this service versus the
combined weights of all others.
● nftables for network traffic
○ Not configured in systemd, but nftables can
leverage systemd’s control groups for traffic
shaping and other rules.
Using Traditional ulimit/rlimit Options● CPU
○ LimitCPU=
○ LimitNPROC=
○ LimitRTPRIO=
○ LimitRTTIME=
○ LimitNICE=
● Disk
○ LimitCORE=
● Memory
○ LimitDATA=
○ LimitFSIZE=
○ LimitSTACK=
○ LimitMSGQUEUE=
○ LimitAS=
○ LimitRSS=
○ LimitMEMLOCK=
● Other
○ LimitSIGPENDING=
○ LimitNOFILE=
○ LimitLOCKS=
Handling Timeouts and Abnormal Exits
2 Plan for the unexpected
Directives for Detecting and Responding to FailureDetecting Failure
● SuccessExitStatus=
○ Whitelist of exit codes and signals to indicate a
normal exit. Defaults to zero and the usual process
signals for healthy processes.
● RestartPreventExitStatus=
○ Blacklist of exit codes and signals to not trigger
restarts. Useful to restart on most failures but not
unrecoverable ones like a bad configuration.
● RestartForceExitStatus=○ The opposite of the previous option.
● StartLimitInterval= and StartLimitBurst=
○ Thresholds at which attempted failure recovery
becomes a stickier failure.
Responding to Failure
● Restart=
○ Allows many options, but on-failure is
probably best for most cases.
● FailureAction=
○ Supports options like rebooting or shutting
down the system on service failure.
● StartLimitAction=
○ Same as FailureAction= but triggered when
StartLimit… thresholds get hit.
● systemctl reset-failed○ Resets status units marked as failed.
Built-In Service Monitoring with WatchdogServices
● WatchdogSec=
○ Configures the maximum interval for the
healthy service to ping systemd.
● $WATCHDOG_USEC and $WATCHDOG_PID
○ Environmental variables set for a service that
is expected to provide systemd with
watchdog pings.
● systemd-notify WATCHDOG=1
○ CLI; the most basic way for a service to send
systemd a watchdog ping.
● sd_notify(0, “WATCHDOG=1”);
○ A better way that requires linking to a
systemd library.
Overall System
● RuntimeWatchdogSec=
○ Configures the maximum interval for
systemd to ping the hardware watchdog
service (if it exists). If the hardware fails to
receive an expected ping, it will reboot the
system.
● ShutdownWatchdogSec=
○ Bounds the time the watchdog hardware is
willing to wait for a clean shutdown for the
triggered reboot.
Dropping Privileges and Access Early
3 Tighten security
Dropping Privileges and Access Early● Hardening options that mostly just work
○ User=<service-user>
○ PrivateTmp=true
○ PrivateDevices=true
○ ProtectSystem=full
○ ProtectHome=read-only
○ NoNewPrivileges=true
○ MountFlags=private
○ SystemCallArchitectures=native
○ SecureBits=noroot noroot-locked
● Restrict visible directories
○ ReadWriteDirectories=
○ ReadOnlyDirectories=
○ InaccessibleDirectories=
○ RootDirectory=
runs the service in chroot
● Whitelist capabilities and system calls
○ AmbientCapabilities=
○ CapabilityBoundingSet=
○ SystemCallFilter=
○ SystemCallErrorNumber=EPERM
tests filters in a non-enforcing mode
● Control sockets
○ RestrictAddressFamilies=
○ PrivateNetwork=true, which is best
combined with socket activation
● Bridge to mandatory access control (MAC)
○ SELinuxContext=
○ AppArmorProfile=
○ SmackProcessLabel=
Monitoring
4 Manage, monitor, and automate
Monitor at the Box LevelPlug a systemctl call into your monitoring tool:
# systemctl --state=failed --all
0 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
Automation
4 Manage, monitor, and automate
Pantheon is a Chef Shoptemplate '/etc/systemd/system/foo.service' do
mode '0644'
source 'foo.service.erb'
end
service 'foo.service' do
provider Chef::Provider::Service::Systemd
supports :status => true, :restart => true, :reload => true
action [ :enable, :start ]
end
Questions? Follow Ups?Reach out to me @DavidStrauss.
Want to get more hands-on? We’re hiring!
pantheon.io/careers