KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now...
Transcript of KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now...
![Page 1: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/1.jpg)
KernelCI: a new dawn
FOSDEM 2019Guillaume [email protected]
Copyright @ 2019 Collabora Limited. Some rights reserved.Open First
![Page 2: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/2.jpg)
2
Kernel testing landscape
● Developers: contributors to upstream, maintainers
→ Only run tests on their workstations / dev boards
![Page 3: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/3.jpg)
3
Kernel testing landscape
● Developers: contributors to upstream, maintainers
→ Only run tests on their workstations / dev boards
● Users: distros, OEMs, SoC/CPU vendors
→ Only run tests on their own hardware
→ Don’t necessarily send fixes upstream
![Page 4: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/4.jpg)
4
Kernel testing landscape
● Intel 0-Day and Linux Kernel Performance[1]
→ Only run tests on Intel x86
[1] https://01.org/lkp
![Page 5: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/5.jpg)
5
Kernel testing landscape
● Intel 0-Day and Linux Kernel Performance[1]
→ Only run tests on Intel x86
● Linaro Kernel Functional Tests[2]
→ Only run tests on Linaro member platforms
[1] https://01.org/lkp
[2] https://lkft.linaro.org/
![Page 6: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/6.jpg)
6
Total test coverageTotal test coverage
==
On the beaten tracksOn the beaten tracks
![Page 7: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/7.jpg)
7
KernelCI: off-road testing
● Run on all CPU architectures
● Test coverage aiming to grow over the whole kernel
● Run all tests on a wide range of hardware platforms
![Page 8: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/8.jpg)
8
KernelCI: a ghost project?● Initiated by Linaro in 2014
● Targeting ARM ecosystem
● Only did boot testing
● Limited community
● Needs a new home
![Page 9: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/9.jpg)
9
KernelCI: a new dawn
● KernelCI now becoming part of the Linux FoundationKernelCI now becoming part of the Linux Foundation
● Collabora to be a Premier memberCollabora to be a Premier member
● Filling gaps in kernel test coverageFilling gaps in kernel test coverage
● Big companies are investingBig companies are investing
– Google and Microsoft sponsored serversGoogle and Microsoft sponsored servers
● Funding, structure, maintenance, sustainability...Funding, structure, maintenance, sustainability...
![Page 10: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/10.jpg)
10
kernelci.org
Where are we now?
![Page 11: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/11.jpg)
11
The KernelCI loop
![Page 12: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/12.jpg)
12
KernelCI in numbersarm
arm64
x86
mips
arc
riscv
1 buildevery
24seconds
1 bootevery
40seconds
theobroma
mhart
baylibre
linarocollabora
76devicetypes
![Page 13: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/13.jpg)
13
Automated boot bisection● For all boot regressions found on:
– mainline
– stable branches
– linux-next
– some subsystems’ and maintainers’ trees
● A few noteworthy results:– QEMU x86 boot failure on linux-4.14.y
https://lists.linaro.org/pipermail/kernel-build-reports/2018-January/thread.html#27424
– Peach Pi Chromebook deadlock on v4.15-rc3https://lists.linaro.org/pipermail/kernel-build-reports/2017-December/thread.html#26688
– Big-endian arm64 preempt count bug on next-20181210https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=akpm&id=7faa313f05cad184e8b17750f0cbe5216ac6debb
![Page 14: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/14.jpg)
14
Functional tests
● Graphics: IGT (DRM/KMS)→ Subset run on a handful of devices, gradually expanding
● Media: v4l2-compliance→ Full test suite run on hardware and QEMU (vivid driver)
● Power: suspend / resume→ Run on many boards, finding issues regularly
● USB: smoke test→ Check that the USB subsystem is initialised
![Page 15: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/15.jpg)
15
kernelci.org
What’s next?
![Page 16: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/16.jpg)
16
Joining the Linux Foundation
● Membership scheme● Sustainable funding● KernelCI as a service● Premier members:
– Collabora– …waiting for official project launch
![Page 17: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/17.jpg)
17
More build power
● Microsoft Azure builders
● Google Compute Engine builders
● Linux Foundation membership can help too
● Adding more trees: subsystems, stable “autosel”
● Adding multiple compiler support (LLVM/Clang...)
![Page 18: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/18.jpg)
18
Horizontal / Vertical testing
KernelCI with boot testing
short tests on many configurations
long tests on few configurations
LKFT with LTP
depth
width
![Page 19: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/19.jpg)
19
Horizontal / Vertical testing
short tests on many configurations
long tests on few configurationsdepth
width
LKFT and KernelCI starting to join forces
![Page 20: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/20.jpg)
20
Mapping test areas of the kernel
● Drawing a map of all the kernel surface to test
● Gradually expanding coverage
– Adding source trees for each part of the kernel
– Adding test suites to target each specific part
– Adding hardware to test architectures and drivers
● Digging out issues found in “terra incognita”
![Page 21: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/21.jpg)
21
Case Study
The media subsystem
![Page 22: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/22.jpg)
22
Media subsystem as a pilot● Building media subsystem master branch
● Expanding v4l2 test plan
● Enabling QEMU with virtual video (vivid driver)
● Working with the developers to get feedback
● Improving test results
● Tracking regressions
● Improving email reports
![Page 23: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/23.jpg)
23
Issues found with UVC driver● The USB Video Class driver is for standard webcams
● Has been in mainline since v2.6.26
● Enabled testing on rk3399-gru-kevin Chromebook
● Found out that it fails v4l2-compliance:
Total: 48, Succeeded: 44, Failed: 4, Warnings: 9
● Full details:https://lava.collabora.co.uk/scheduler/job/1328514 https://lava.collabora.co.uk/results/1328514/0_v4l2?search=&length=50#table
![Page 24: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/24.jpg)
24
Showing failures
prompts developers
to fix them
![Page 25: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/25.jpg)
25
Thank you!
Any questions?
![Page 26: KernelCI: a new dawn - FOSDEM€¦ · Needs a new home. 9 KernelCI: a new dawn KernelCI now becoming part of the Linux Foundation Collabora to be a Premier member Filling gaps in](https://reader033.fdocuments.us/reader033/viewer/2022060221/5f073d6d7e708231d41c0160/html5/thumbnails/26.jpg)
26
Photo credits
● 1. dawn: https://www.nasa.gov/content/sunrise-from-the-international-space-station
● 2. landscape: https://www.flickr.com/photos/hemlit/8212362709/
● 6. sand: https://www.flickr.com/photos/156754622@N02/23962149187/
● 7. truck: https://www.flickr.com/photos/surfergirl30/44110720615/
● 8. haunted house: https://www.flickr.com/photos/adriensifre/7344166962/