ENGINEERS AND DEVICESWORKING TOGETHER

ENGINEERS AND DEVICES

WORKING TOGETHER

●○○○

●○○○○○

●○

●

ENGINEERS AND DEVICESWORKING TOGETHER

●○○○ printk()

●○○○

●○○○○

●○

ENGINEERS AND DEVICESWORKING TOGETHER

0x0 0x0000ffffffffffff 256TB

0xffff000000000000 0xffff000008000000 128MB

0xffff000008000000 0xffff00000807ffff 512KB

0xffff000008080000 10MB+

Kernel image specific 0xffff7dffbfff0000 ~126TB

0xffff7dfffe7fd000 0xffff7dfffec00000 4MB + 12KB

0xffff7dfffee00000 0xffff7dffffe00000 16MB

0xffff7e0000000000 0xffff800000000000 2048G

0xffff800000000000 0xffffffffffffffff 128TB

ENGINEERS AND DEVICESWORKING TOGETHER

ENGINEERS AND DEVICESWORKING TOGETHER

●○ ⇒○ ⇒○ ⇒○ ⇒

●○ printk○ git grep printk

●○

○

○

ENGINEERS AND DEVICESWORKING TOGETHER

●○ FRAME_WARN○ SCHED_STACK_END_CHECK

●○ slub_debug=

■ SLUB_DEBUG

○ DEBUG_PAGEALLOC○ PAGE_POISONING

●○ DEBUG_SPINLOCK○ DEBUG_MUTEXES○ DEBUG_ATOMIC_SLEEP○ DEBUG_LOCK_ALLOC○ LOCKUP_DETECTOR

●

ENGINEERS AND DEVICESWORKING TOGETHER

●○○○

●○○

●○○ PROVE_LOCKING○ KASAN

ENGINEERS AND DEVICESWORKING TOGETHER

PROVE_LOCKING

lock(subsys) lock(driver)lock(driver) lock(subsys)

DEBUG_SPINLOCKDEBUG_MUTEXES

lock(subsys)lock(driver)unlock(driver)unlock(subsys)lock(driver)lock(subsys)

PROVE_LOCKING

PROVE_LOCKING

ENGINEERS AND DEVICES

WORKING TOGETHER

●○○○

●○○○○○

●○

●

ENGINEERS AND DEVICESWORKING TOGETHER

●

●

●

●●

●

ENGINEERS AND DEVICESWORKING TOGETHER

make cscope git grep 'struct foo {'

printk() [ 0.001636] xyz: Found 10 widgets⇒ git grep "Found .* widgets"

I can’t reproduce butmy customer can

ENGINEERS AND DEVICESWORKING TOGETHER

●

○○○

●○○ ⇒○ quiet

LOG_BUF_SHIFT log_bug_len=● pr_debug()

○ <8> <7>○○ DYNAMIC_DEBUG

I can’t reproduce butmy customer can

ENGINEERS AND DEVICESWORKING TOGETHER

I can’t reproduce butmy customer can

SP

LR (r30)

FP (r29)

r19..r28

r18

IP0/IP1 (r16-r17)

r9..r15

r8

r0..r7

ENGINEERS AND DEVICESWORKING TOGETHER

git grep REGMAP_ALLOW_WRITE_DEBUGFS

I can’t reproduce butmy customer can

config DEBUG_FSbool "Debug Filesystem"select SRCUhelp debugfs is a virtual file system that kernel developers use to put debugging files into. Enable this option to be able to read and write to these files.

If unsure, say N.

ENGINEERS AND DEVICESWORKING TOGETHER

I can’t reproduce butmy customer can

<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30bl ffff000008092ea0 <_mcount>ldp x29, x30, [sp],#16ret

<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30nopldp x29, x30, [sp],#16ret

<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30bl ftrace_callerldp x29, x30, [sp],#16ret

ENGINEERS AND DEVICESWORKING TOGETHER

●●

○ ftrace=function ftrace_filter=mydrv_* tp_printk○ ftrace=function_graph ftrace_graph_notrace=rcu*,*lock,*spin*

●cd /sys/kernel/debug/tracingecho function > current_tracerecho 1 > tracing_on

●○○○ ftrace_dump_on_oops○

● trace_printk()

I can’t reproduce butmy customer can

ENGINEERS AND DEVICESWORKING TOGETHER

/proc/vmcore

console=ttyS0,1115200 … … crashkernel=128M

-p

./kexec -p vmlinux --dtb=xxx.dtb --append="root=/dev/mmcblk0p9 rw 1 maxcpus=1 reset_devices"

I can’t reproduce butmy customer can

ENGINEERS AND DEVICESWORKING TOGETHER

●

●

●

●

●

●

●printk()

ENGINEERS AND DEVICESWORKING TOGETHER

●

●●

●○○

○ echo 'sched:*' > /sys/kernel/debug/tracing/set_event

My widget missedits deadline

ENGINEERS AND DEVICESWORKING TOGETHER

●●●●●●

My widget missedits deadline

ENGINEERS AND DEVICESWORKING TOGETHER

●●●●

.

ENGINEERS AND DEVICESWORKING TOGETHER

●●

●

●

●

●

ENGINEERS AND DEVICESWORKING TOGETHER

●○○ initcall_debug○ initcall_debug clk_disable_unused() clk_ignore_unused○ DEBUG_SPINLOCK DEBUG_MUTEX LOCKUP_DETECTOR○

●○○

○

○○

My board juststopped dead

ENGINEERS AND DEVICESWORKING TOGETHER

●○

●○

○

○

●○○○

●○○

My board juststopped dead

ENGINEERS AND DEVICESWORKING TOGETHER

●

●●

○○

○●●

●

My board juststopped dead

ENGINEERS AND DEVICESWORKING TOGETHER

My board juststopped dead

[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78[ 277.600833] PCSR of cluster0 cpu2 is 0xffff00000809a934[ 277.609564] PCSR of cluster0 cpu3 is 0xffff00000809a934

ENGINEERS AND DEVICESWORKING TOGETHER

●

●

●

●

○○○

ENGINEERS AND DEVICESWORKING TOGETHER

git bisect start <bad_sha1> <good_sha1>git bisect run compile_kernel_and_run_test.sh

compile_kernel_and_test.sh

compile_kernel_and_test.sh

I’m sure thisused to work

ENGINEERS AND DEVICESWORKING TOGETHER

ktest.pl

compile_kernel_and_test.sh

Documentation/ tools/testing/ktest/sample.conftools/testing/ktest/examples

I’m sure thisused to work

ENGINEERS AND DEVICESWORKING TOGETHER

●●

●

●

●

●

ENGINEERS AND DEVICESWORKING TOGETHER

My board justrandomly failed

ENGINEERS AND DEVICES

WORKING TOGETHER

●○○○

●○○○○○

●○

●

ENGINEERS AND DEVICESWORKING TOGETHER

ENGINEERS AND DEVICES

WORKING TOGETHER

Graphic by Jost, CC0-PD

ENGINEERS AND DEVICES

WORKING TOGETHER

●○○○

●○○○○○

●○

●

ENGINEERS AND DEVICESWORKING TOGETHERom: http://linaro.co/bud17-kdb