bisecting cause commit starting from d678cbd2f867a564a3c5b276c454e873f43f02f8 building syzkaller on b270611864ec905fee493d0535175fc614201850 testing commit d678cbd2f867a564a3c5b276c454e873f43f02f8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: af72a3a48f04588498356cc0faad0259f06ad59226b4fa2839dcb997bf4b0535 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: failed to write control pipe: write |NUM: broken pipe run #1: crashed: possible deadlock in bpf_trace_printk run #2: crashed: possible deadlock in bpf_trace_printk run #3: crashed: possible deadlock in bpf_trace_printk run #4: crashed: possible deadlock in bpf_trace_printk run #5: crashed: possible deadlock in bpf_trace_printk run #6: crashed: possible deadlock in bpf_trace_printk run #7: crashed: possible deadlock in bpf_trace_printk run #8: crashed: possible deadlock in bpf_trace_printk run #9: crashed: possible deadlock in bpf_trace_printk run #10: crashed: possible deadlock in bpf_trace_printk run #11: crashed: possible deadlock in bpf_trace_printk run #12: crashed: possible deadlock in bpf_trace_printk run #13: crashed: possible deadlock in bpf_trace_printk run #14: crashed: possible deadlock in bpf_trace_printk run #15: crashed: possible deadlock in bpf_trace_printk run #16: crashed: possible deadlock in bpf_trace_printk run #17: crashed: possible deadlock in bpf_trace_printk run #18: crashed: possible deadlock in bpf_trace_printk run #19: crashed: possible deadlock in bpf_trace_printk testing release v5.18 testing commit 4b0986a3613c92f4ec1bdc7f60ec66fea135991f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: cbe6cbf89224449c66769241bc85172af4b4e7ec17cd7dcb0ad59fa4506817b9 all runs: OK # git bisect start d678cbd2f867a564a3c5b276c454e873f43f02f8 4b0986a3613c92f4ec1bdc7f60ec66fea135991f Bisecting: 5179 revisions left to test after this (roughly 13 steps) [2518f226c60d8e04d18ba4295500a5b0b8ac7659] Merge tag 'drm-next-2022-05-25' of git://anongit.freedesktop.org/drm/drm testing commit 2518f226c60d8e04d18ba4295500a5b0b8ac7659 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 02e1aee21c403832bb14b9f89884ecdb43871028565bd2dc73cf9ea080047e14 all runs: crashed: possible deadlock in bpf_trace_printk # git bisect bad 2518f226c60d8e04d18ba4295500a5b0b8ac7659 Bisecting: 3998 revisions left to test after this (roughly 12 steps) [5d1772b1739b085721431eef0c0400f3aff01abf] Merge branch 'for-5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq testing commit 5d1772b1739b085721431eef0c0400f3aff01abf compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 9c3aa9ef3a50b22d0d2e56f216d9f03a6714216bd6499c68a0fa3bd30e5937f8 all runs: crashed: possible deadlock in bpf_trace_printk # git bisect bad 5d1772b1739b085721431eef0c0400f3aff01abf Bisecting: 1498 revisions left to test after this (roughly 11 steps) [0350785b0a092c99c5ddd2ace0260dbe7b3f919f] Merge tag 'integrity-v5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/zohar/linux-integrity testing commit 0350785b0a092c99c5ddd2ace0260dbe7b3f919f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c43be8e632f4f3e3825d12f7c8707f6ddbfeaa70a2598026b9c436f042ff1dca run #0: crashed: possible deadlock in bpf_trace_printk run #1: crashed: possible deadlock in bpf_trace_printk run #2: crashed: possible deadlock in bpf_trace_printk run #3: crashed: possible deadlock in bpf_trace_printk run #4: crashed: possible deadlock in bpf_trace_printk run #5: crashed: possible deadlock in bpf_trace_printk run #6: crashed: possible deadlock in bpf_trace_printk run #7: crashed: possible deadlock in bpf_trace_printk run #8: crashed: WARNING in bpf_bprintf_prepare run #9: crashed: possible deadlock in bpf_trace_printk # git bisect bad 0350785b0a092c99c5ddd2ace0260dbe7b3f919f Bisecting: 720 revisions left to test after this (roughly 10 steps) [c5a3d3c01e90e74166f95eec9db6fcc3ba72a9d6] Merge tag 'x86_cpu_for_v5.19_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit c5a3d3c01e90e74166f95eec9db6fcc3ba72a9d6 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 77f0900ad2fd80ab3ae8e765796aa099fcf5c28e05ffd310cb0b82672e6d286b all runs: OK # git bisect good c5a3d3c01e90e74166f95eec9db6fcc3ba72a9d6 Bisecting: 286 revisions left to test after this (roughly 9 steps) [143a6252e1b8ab424b4b293512a97cca7295c182] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux testing commit 143a6252e1b8ab424b4b293512a97cca7295c182 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3833b293a892519b031aae7a8ba70eb759f14a78d9cb2cfba2c2ca4714f3d91f all runs: OK # git bisect good 143a6252e1b8ab424b4b293512a97cca7295c182 Bisecting: 137 revisions left to test after this (roughly 7 steps) [ac2ab99072cce553c78f326ea22d72856f570d88] Merge tag 'random-5.19-rc1-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/crng/random testing commit ac2ab99072cce553c78f326ea22d72856f570d88 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3c976d34f7bf98f111166d32a290f31454343d4c16027475cefb4df914b00c95 all runs: crashed: possible deadlock in bpf_trace_printk # git bisect bad ac2ab99072cce553c78f326ea22d72856f570d88 Bisecting: 72 revisions left to test after this (roughly 6 steps) [cfeb2522c3777923c2bd24772b914195dc93c86b] Merge tag 'perf-core-2022-05-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit cfeb2522c3777923c2bd24772b914195dc93c86b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0c7c88cdd51d6cc49e8971a30e50e251026e40c6abafcd45ce78c70e952c39d3 all runs: crashed: possible deadlock in bpf_trace_printk # git bisect bad cfeb2522c3777923c2bd24772b914195dc93c86b Bisecting: 33 revisions left to test after this (roughly 5 steps) [22922deae13fc8d3769790c2eb388e9afce9771d] Merge tag 'objtool-core-2022-05-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 22922deae13fc8d3769790c2eb388e9afce9771d compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b039ca3e8372fdd9953f0d9b3003a5a5420abf983c9c26448b5377c108988fc8 all runs: crashed: possible deadlock in bpf_trace_printk # git bisect bad 22922deae13fc8d3769790c2eb388e9afce9771d Bisecting: 20 revisions left to test after this (roughly 4 steps) [21e350233b07619dbfc3ce606ff1fc468fce2d82] scripts: Create objdump-func helper script testing commit 21e350233b07619dbfc3ce606ff1fc468fce2d82 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4bb2a8bc29fa9b1fff8ebc387d6785ec5fe9d1d4c22e3b749f9b1e402751f355 all runs: OK # git bisect good 21e350233b07619dbfc3ce606ff1fc468fce2d82 Bisecting: 10 revisions left to test after this (roughly 3 steps) [dc1f7893a70fe403983bd8492f177bf993940e2c] locking/mutex: Make contention tracepoints more consistent wrt adaptive spinning testing commit dc1f7893a70fe403983bd8492f177bf993940e2c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a618885e2cc3aca19559b609583b6c80acbbf5e30502813d8873671e7cc95740 all runs: crashed: possible deadlock in bpf_trace_printk # git bisect bad dc1f7893a70fe403983bd8492f177bf993940e2c Bisecting: 4 revisions left to test after this (roughly 2 steps) [f9e21aa9e6fb11355e54c8949a390d49ca21cde1] locking/rwsem: No need to check for handoff bit if wait queue empty testing commit f9e21aa9e6fb11355e54c8949a390d49ca21cde1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ddcf304a5bd48a76c933b80a4aa10761115acbbc4fc226a501a3cce134522927 all runs: OK # git bisect good f9e21aa9e6fb11355e54c8949a390d49ca21cde1 Bisecting: 2 revisions left to test after this (roughly 1 step) [1ee326196c66583006b0c95356a4b7dc51bf3531] locking/rwsem: Always try to wake waiters in out_nolock path testing commit 1ee326196c66583006b0c95356a4b7dc51bf3531 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: da87573bc79ff675c83727bccca8ea3ad1fe8176a274eba54652be5a9fc8a038 all runs: OK # git bisect good 1ee326196c66583006b0c95356a4b7dc51bf3531 Bisecting: 0 revisions left to test after this (roughly 1 step) [ee042be16cb455116d0fe99b77c6bc8baf87c8c6] locking: Apply contention tracepoints in the slow path testing commit ee042be16cb455116d0fe99b77c6bc8baf87c8c6 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bd7655d75ebd16dd12ecadf487f14bd2f6d3888b08e1dfc0bcda3d7f08cb096d all runs: OK # git bisect good ee042be16cb455116d0fe99b77c6bc8baf87c8c6 dc1f7893a70fe403983bd8492f177bf993940e2c is the first bad commit commit dc1f7893a70fe403983bd8492f177bf993940e2c Author: Peter Zijlstra Date: Wed Mar 30 13:06:54 2022 +0200 locking/mutex: Make contention tracepoints more consistent wrt adaptive spinning Have the trace_contention_*() tracepoints consistently include adaptive spinning. In order to differentiate between the spinning and non-spinning states add LCB_F_MUTEX and combine with LCB_F_SPIN. The consequence is that a mutex contention can now triggler multiple _begin() tracepoints before triggering an _end(). Additionally, this fixes one path where mutex would trigger _end() without ever seeing a _begin(). Signed-off-by: Peter Zijlstra (Intel) include/trace/events/lock.h | 4 +++- kernel/locking/mutex.c | 16 ++++++++++++---- 2 files changed, 15 insertions(+), 5 deletions(-) culprit signature: a618885e2cc3aca19559b609583b6c80acbbf5e30502813d8873671e7cc95740 parent signature: bd7655d75ebd16dd12ecadf487f14bd2f6d3888b08e1dfc0bcda3d7f08cb096d revisions tested: 15, total time: 3h18m9.927244906s (build: 1h35m15.929781183s, test: 1h41m28.707002168s) first bad commit: dc1f7893a70fe403983bd8492f177bf993940e2c locking/mutex: Make contention tracepoints more consistent wrt adaptive spinning recipients (to): ["linux-kernel@vger.kernel.org" "mingo@redhat.com" "peterz@infradead.org" "peterz@infradead.org" "rostedt@goodmis.org" "will@kernel.org"] recipients (cc): ["boqun.feng@gmail.com" "longman@redhat.com"] crash: possible deadlock in bpf_trace_printk ============================================ WARNING: possible recursive locking detected 5.18.0-rc1-syzkaller #0 Not tainted -------------------------------------------- syz-executor.0/4108 is trying to acquire lock: ffffffff8affbdb8 (trace_printk_lock){....}-{2:2}, at: ____bpf_trace_printk kernel/trace/bpf_trace.c:385 [inline] ffffffff8affbdb8 (trace_printk_lock){....}-{2:2}, at: bpf_trace_printk+0xa4/0x130 kernel/trace/bpf_trace.c:371 but task is already holding lock: ffffffff8affbdb8 (trace_printk_lock){....}-{2:2}, at: ____bpf_trace_printk kernel/trace/bpf_trace.c:385 [inline] ffffffff8affbdb8 (trace_printk_lock){....}-{2:2}, at: bpf_trace_printk+0xa4/0x130 kernel/trace/bpf_trace.c:371 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(trace_printk_lock); lock(trace_printk_lock); *** DEADLOCK *** May be due to missing lock nesting notation 7 locks held by syz-executor.0/4108: #0: ffffffff8afde808 (tracepoints_mutex){+.+.}-{3:3}, at: tracepoint_probe_unregister+0x25/0xbf0 kernel/tracepoint.c:548 #1: ffffffff8ae28f50 (cpu_hotplug_lock){++++}-{0:0}, at: static_key_disable+0x9/0x20 kernel/jump_label.c:214 #2: ffffffff8b052be8 (jump_label_mutex){+.+.}-{3:3}, at: jump_label_lock kernel/jump_label.c:27 [inline] #2: ffffffff8b052be8 (jump_label_mutex){+.+.}-{3:3}, at: static_key_disable_cpuslocked+0xd0/0x160 kernel/jump_label.c:205 #3: ffffffff8ae44528 (text_mutex){+.+.}-{3:3}, at: arch_jump_label_transform_queue+0x58/0x100 arch/x86/kernel/jump_label.c:136 #4: ffffffff8af79660 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run2+0x0/0x2b0 kernel/trace/./bpf_trace.h:11 #5: ffffffff8affbdb8 (trace_printk_lock){....}-{2:2}, at: ____bpf_trace_printk kernel/trace/bpf_trace.c:385 [inline] #5: ffffffff8affbdb8 (trace_printk_lock){....}-{2:2}, at: bpf_trace_printk+0xa4/0x130 kernel/trace/bpf_trace.c:371 #6: ffffffff8af79660 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run2+0x0/0x2b0 kernel/trace/./bpf_trace.h:11 stack backtrace: CPU: 1 PID: 4108 Comm: syz-executor.0 Not tainted 5.18.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 print_deadlock_bug kernel/locking/lockdep.c:2957 [inline] check_deadlock kernel/locking/lockdep.c:3000 [inline] validate_chain kernel/locking/lockdep.c:3785 [inline] __lock_acquire.cold+0x149/0x399 kernel/locking/lockdep.c:5022 lock_acquire kernel/locking/lockdep.c:5634 [inline] lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5599 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:162 ____bpf_trace_printk kernel/trace/bpf_trace.c:385 [inline] bpf_trace_printk+0xa4/0x130 kernel/trace/bpf_trace.c:371 bpf_prog_0605f9f479290f07+0x2f/0x33 bpf_dispatcher_nop_func include/linux/bpf.h:804 [inline] __bpf_prog_run include/linux/filter.h:628 [inline] bpf_prog_run include/linux/filter.h:635 [inline] __bpf_trace_run kernel/trace/bpf_trace.c:2022 [inline] bpf_trace_run2+0xe9/0x2b0 kernel/trace/bpf_trace.c:2059 __bpf_trace_contention_begin+0xb5/0xf0 include/trace/events/lock.h:95 trace_contention_begin.constprop.0+0xda/0x1b0 include/trace/events/lock.h:95 __pv_queued_spin_lock_slowpath+0x103/0xb50 kernel/locking/qspinlock.c:405 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:591 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline] queued_spin_lock include/asm-generic/qspinlock.h:85 [inline] do_raw_spin_lock+0x200/0x2a0 kernel/locking/spinlock_debug.c:115 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:111 [inline] _raw_spin_lock_irqsave+0x41/0x50 kernel/locking/spinlock.c:162 ____bpf_trace_printk kernel/trace/bpf_trace.c:385 [inline] bpf_trace_printk+0xa4/0x130 kernel/trace/bpf_trace.c:371 bpf_prog_0605f9f479290f07+0x2f/0x33 bpf_dispatcher_nop_func include/linux/bpf.h:804 [inline] __bpf_prog_run include/linux/filter.h:628 [inline] bpf_prog_run include/linux/filter.h:635 [inline] __bpf_trace_run kernel/trace/bpf_trace.c:2022 [inline] bpf_trace_run2+0xe9/0x2b0 kernel/trace/bpf_trace.c:2059 __bpf_trace_contention_begin+0xb5/0xf0 include/trace/events/lock.h:95 trace_contention_begin+0xc0/0x150 include/trace/events/lock.h:95 __mutex_lock_common kernel/locking/mutex.c:605 [inline] __mutex_lock+0x13c/0x1350 kernel/locking/mutex.c:747 arch_jump_label_transform_queue+0x58/0x100 arch/x86/kernel/jump_label.c:136 __jump_label_update+0xea/0x330 kernel/jump_label.c:451 static_key_disable_cpuslocked+0x100/0x160 kernel/jump_label.c:207 static_key_disable+0x11/0x20 kernel/jump_label.c:215 tracepoint_remove_func kernel/tracepoint.c:417 [inline] tracepoint_probe_unregister+0x49e/0xbf0 kernel/tracepoint.c:551 bpf_raw_tp_link_release+0x4c/0xa0 kernel/bpf/syscall.c:2891 bpf_link_free+0xc9/0x190 kernel/bpf/syscall.c:2469 bpf_link_release+0x2e/0x50 kernel/bpf/syscall.c:2504 __fput+0x1f5/0x8c0 fs/file_table.c:317 task_work_run+0xc0/0x160 kernel/task_work.c:164 resume_user_mode_work include/linux/resume_user_mode.h:49 [inline] exit_to_user_mode_loop kernel/entry/common.c:169 [inline] exit_to_user_mode_prepare+0x23c/0x250 kernel/entry/common.c:201 __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline] syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:294 do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f99db63bd4b Code: 0f 05 48 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 63 fc ff ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 a1 fc ff ff 8b 44 RSP: 002b:00007ffd483150c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000005 RCX: 00007f99db63bd4b RDX: 00007f99db7a10f8 RSI: ffffffffffffffff RDI: 0000000000000004 RBP: 00007f99db79d960 R08: 0000000000000000 R09: 00007f99db7a1100 R10: 00007ffd483151c0 R11: 0000000000000293 R12: 0000000000010d3f R13: 00007ffd483151c0 R14: 00007f99db79bf60 R15: 0000000000000226