Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer...
Transcript of Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer...
![Page 1: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/1.jpg)
ElivepatchFlexible distributed Linux
Kernel live patchingAlice Ferrazzi
1
![Page 2: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/2.jpg)
Summary● Live patch explanation● Current live patch services
○ Motivation for elivepatch● Elivepatch solution
○ Implementation○ Challenge○ Status○ Future Work
● Conclusion
2
![Page 3: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/3.jpg)
kernel :~ $ whoami● Gentoo
○ Gentoo Kernel Project Leader○ Gentoo Kernel Security○ Gentoo board member○ Gentoo Google Summer of Code administrator and mentor for rust Gentoo
project● Cybertrust Japan
○ OSS Embedded Software Engineer● Researcher
○ ACM SIGOPS member○ Presented elivepatch as poster at SOSP 2017
3
![Page 4: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/4.jpg)
This project was part of Google Summer of Code 2017 for the Gentoo organization.
4
![Page 5: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/5.jpg)
Live patch explanation
5
![Page 6: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/6.jpg)
Live patch
Modify the kernel without the need to reboot.
6
![Page 7: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/7.jpg)
Why- Downtime is expensive (containers,
supercomputers)- Security (vulnerability time shorter)
7
![Page 8: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/8.jpg)
Where- Embedded- Mobile- Desktops- HPC (complex scientific computations)- Cloud- Any computer under heavy load
8
![Page 9: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/9.jpg)
What
9
![Page 10: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/10.jpg)
KgraftSuse Open Source live patching system that is routing the old function gradually.
10
![Page 11: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/11.jpg)
Kpatch
Red Hat Open Source live patching system and use ftrace and stop_machine() for route functions toward the new function version.
11
![Page 12: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/12.jpg)
LivepatchLivepatch is a hybrid of kpatch and kgraft.Livepatch has been merged into the kernel upstream.
Kpatch-build can work with both kpatch and livepatch for creating the live patch.
12
![Page 13: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/13.jpg)
Livepatch is just a module
13
![Page 14: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/14.jpg)
...
14
![Page 15: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/15.jpg)
A module that takes just about 1+ hour to compile in a modern server
Livepatch module problem
15
![Page 16: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/16.jpg)
At Gentoo, we know what means to compile something for more than 1 hour…
16
![Page 17: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/17.jpg)
17
![Page 18: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/18.jpg)
Gentoo solution to compile for 1+ hour compilation problem
● Gentoo “binary host”● Pre-compiled binary
18
![Page 19: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/19.jpg)
What options do we have for compiling livepatch modules?
19
![Page 20: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/20.jpg)
Current existing livepatch services
20
![Page 21: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/21.jpg)
Current vendor solutions● Oracle, Ksplice (support only Oracle Linux
kernels)● Suse Linux Enterprise Live Patching (support
only Suse Kernels for one year)● Canonical Live Patch (support only Ubuntu
16.04 LTS and Ubuntu 14.04 LTS)● Red Hat live patch (Support only Red Hat kernel)
21
![Page 22: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/22.jpg)
Motivation for elivepatch
22
![Page 23: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/23.jpg)
Problems of vendor solutions
● trusting on third-party vendors● Lacking support for custom kernel
configurations● Lacking support for request-driven
costumization● Lacking long term support● Closed source
23
![Page 24: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/24.jpg)
elivepatch solution
24
![Page 25: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/25.jpg)
A web service framework to deliver Linux kernel live patches● Supports custom kernel configurations● User participation via request-driven
customization● Open source
25
elivepatch
![Page 26: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/26.jpg)
Vendor solutions representation
26
![Page 27: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/27.jpg)
Elivepatch solution
27
![Page 28: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/28.jpg)
Implementation
28
![Page 29: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/29.jpg)
Elivepatch-server (Main language: Python)Flask + Flask-Restful + Werkzeug (not dependent)
Elivepatch-client (Main language: Python)Requests + GitPython
29
![Page 30: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/30.jpg)
Challenges
30
![Page 31: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/31.jpg)
Challenges with elivepatch● Some patches require manual modification to
converted to live patches● Reproducing the build environment can be
difficult:● Differences in compiler versions● Variations in the compiler and optimization
flags● Incompatible machine architectures (solaris, hpc) 31
![Page 32: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/32.jpg)
Incompatibility with GCCCCFLAGS and non vanilla gcc, can sometime broke elivepatch.
32
![Page 33: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/33.jpg)
Current status
33
![Page 34: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/34.jpg)
Elivepatch status● First open source release 0.1 on 2017/9/06● Packaged for Gentoo● Presented as poster at SOSP 2017● Close collaboration with kpatch mainteiners
34
![Page 35: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/35.jpg)
Future work
35
![Page 36: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/36.jpg)
Future work● Automate livepatch conversion● Increasing scalability using containers and virtual
machines● Livepatch signing● Kernel CI\CD check
36
![Page 37: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/37.jpg)
Automate livepatch conversion- Check patch for problems during conversion- Suggest changes to patch for conversion- Interest also for upstream to kpatch
https://github.com/aliceinwire/elivepatch_lintian
37
![Page 38: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/38.jpg)
Multi distributionSolve distributions compatibility issuesCurrent target:● Debian● Fedora● Gentoo● Android
38
![Page 39: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/39.jpg)
Elivepatch client on Debian
Work in progress…https://asciinema.org/a/187738
p.s. Gentoo kernel is still needed
39
![Page 40: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/40.jpg)
Livepatch signing● Implementing livepatch module signing in the
server ● Implementing signing verification for the client
40
![Page 41: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/41.jpg)
Kernel CI/CD checking● Implement a buildbot plugin for testing
elivepatch[You can test your livepatch with the same settings and hardware as where you want to deploy it]
41
![Page 42: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/42.jpg)
Conclusion
42
![Page 43: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/43.jpg)
● Live patch is a module that takes time compiling● Live patch vendor service solutions solving the
compilation problem● Elivepatch solution
43
Epilogue
![Page 44: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/44.jpg)
ConclusionWith the diffusion of embedded systems and
robotics,Livepatch services will become always more
important
44
![Page 45: Kernel live patching Flexible distributed Linux Elivepatch · OSS Embedded Software Engineer Researcher ACM SIGOPS member ... Lacking support for custom kernel configurations Lacking](https://reader033.fdocuments.us/reader033/viewer/2022042302/5ecd8812d525a4298018ecd8/html5/thumbnails/45.jpg)
If you are interested in contributing,Elivepatch is welcoming every form of
contributions
45