bisecting cause commit starting from 9c54130cd25528028b2d38f6ada0c79e92210390 building syzkaller on 6a81331a1d4c744da9204d02ec88d558f7eea9c9 testing commit 9c54130cd25528028b2d38f6ada0c79e92210390 with gcc (GCC) 10.2.1 20210217 kernel signature: ca1683d14591197deabb3d2412597340e2c481fc53d8646af4c40150afed3b81 run #0: crashed: WARNING: refcount bug in sk_psock_get run #1: crashed: WARNING: refcount bug in sk_psock_get run #2: crashed: WARNING: refcount bug in sk_psock_get run #3: crashed: WARNING: refcount bug in sk_psock_get run #4: crashed: WARNING: refcount bug in sk_psock_get run #5: crashed: WARNING: refcount bug in sk_psock_get run #6: crashed: WARNING: refcount bug in sk_psock_get run #7: crashed: WARNING: refcount bug in sk_psock_get run #8: crashed: WARNING: refcount bug in sk_psock_get run #9: crashed: WARNING: refcount bug in sk_psock_get run #10: crashed: WARNING: refcount bug in sk_psock_get run #11: crashed: WARNING: refcount bug in sk_psock_get run #12: crashed: WARNING: refcount bug in sk_psock_get run #13: crashed: WARNING: refcount bug in sk_psock_get run #14: crashed: WARNING: refcount bug in sk_psock_get run #15: crashed: WARNING: refcount bug in sk_psock_get run #16: crashed: WARNING: refcount bug in bpf_exec_tx_verdict run #17: crashed: WARNING: refcount bug in sk_psock_get run #18: crashed: WARNING: refcount bug in sk_psock_get run #19: crashed: WARNING: refcount bug in sk_psock_get testing release v5.11 testing commit f40ddce88593482919761f74910f42f4b84c004b with gcc (GCC) 10.2.1 20210217 kernel signature: 293e6579f0d65cbda46b2d20e15f30555f6e4ea292502f9356c74a1affb84b2f all runs: OK # git bisect start 9c54130cd25528028b2d38f6ada0c79e92210390 f40ddce88593482919761f74910f42f4b84c004b Bisecting: 11797 revisions left to test after this (roughly 14 steps) [c5a58f877ca645a3303f7a57476f2de837fdb97a] Merge tag 'for-linus-5.12b-rc2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip testing commit c5a58f877ca645a3303f7a57476f2de837fdb97a with gcc (GCC) 10.2.1 20210217 kernel signature: de1a507df82f590ac1a94957dcc9351730494c7a6f157baff305207cc87106ab run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: OK run #6: crashed: WARNING in kvm_wait run #7: OK run #8: OK run #9: boot failed: WARNING in kvm_wait # git bisect bad c5a58f877ca645a3303f7a57476f2de837fdb97a Bisecting: 5910 revisions left to test after this (roughly 13 steps) [a864e8f159b13babf552aff14a5fbe11abc017e4] ALSA: hda: intel-nhlt: verify config type testing commit a864e8f159b13babf552aff14a5fbe11abc017e4 with gcc (GCC) 10.2.1 20210217 kernel signature: 9014d2641199ae492b0bb2f7c4f0ccfec9243fa42c9f4ba71108e242339af8d9 run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad a864e8f159b13babf552aff14a5fbe11abc017e4 Bisecting: 2929 revisions left to test after this (roughly 12 steps) [e4286926abbbaab9b047c8bc25cae78ec990928f] Merge tag 'tty-5.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty testing commit e4286926abbbaab9b047c8bc25cae78ec990928f with gcc (GCC) 10.2.1 20210217 kernel signature: 2e9570bafc934fe805de20b3e93c87ac6666701742006ed9940634afcacd209f all runs: OK # git bisect good e4286926abbbaab9b047c8bc25cae78ec990928f Bisecting: 1417 revisions left to test after this (roughly 11 steps) [d089f48fba28db14d0fe7753248f2575a9ddfc73] Merge tag 'core-rcu-2021-02-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit d089f48fba28db14d0fe7753248f2575a9ddfc73 with gcc (GCC) 10.2.1 20210217 kernel signature: 50220a141a16db910a95a2e8af528656ee4c342b57a63ea682676cb77c586f43 all runs: OK # git bisect good d089f48fba28db14d0fe7753248f2575a9ddfc73 Bisecting: 737 revisions left to test after this (roughly 10 steps) [66f73fb3facd42d0a7c899d7f4c712332b28499a] Merge tag 'for-linus-5.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rw/ubifs testing commit 66f73fb3facd42d0a7c899d7f4c712332b28499a with gcc (GCC) 10.2.1 20210217 kernel signature: 6d9d87680e368b536ad9f98528b7adb9972f18aaa902f201fa283d45ba80fc2d run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: crashed: WARNING in kvm_wait run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 66f73fb3facd42d0a7c899d7f4c712332b28499a Bisecting: 355 revisions left to test after this (roughly 8 steps) [08179b47e1fdf288e5d59f90e5ce31513bb019c3] Merge branch 'parisc-5.12-1' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux testing commit 08179b47e1fdf288e5d59f90e5ce31513bb019c3 with gcc (GCC) 10.2.1 20210217 kernel signature: 641ef7b0698bdda8a2632b93b8aeeb9cb2b806490f2b50b24c5056d0e8fbabc4 all runs: crashed: WARNING in kvm_wait # git bisect bad 08179b47e1fdf288e5d59f90e5ce31513bb019c3 Bisecting: 209 revisions left to test after this (roughly 7 steps) [4a037ad5d115b2cc79a5071a7854475f365476fa] Merge tag 'for-linus-5.12-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip testing commit 4a037ad5d115b2cc79a5071a7854475f365476fa with gcc (GCC) 10.2.1 20210217 kernel signature: c75a82230c27395fc919f84a529f6fd1bf97fdcdcf0d68945111763d06d4a76b run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: crashed: WARNING in kvm_wait run #6: OK run #7: crashed: WARNING in kvm_wait run #8: OK run #9: OK # git bisect bad 4a037ad5d115b2cc79a5071a7854475f365476fa Bisecting: 61 revisions left to test after this (roughly 6 steps) [c5e6fc08feb2b88dc5dac2f3c817e1c2a4cafda4] sched,x86: Allow !PREEMPT_DYNAMIC testing commit c5e6fc08feb2b88dc5dac2f3c817e1c2a4cafda4 with gcc (GCC) 10.2.1 20210217 kernel signature: 6cc79bb575fffab332d0a33bf223a83cc3955ca3258d107c1fa834aa3f504baf all runs: OK # git bisect good c5e6fc08feb2b88dc5dac2f3c817e1c2a4cafda4 Bisecting: 33 revisions left to test after this (roughly 5 steps) [9eef02334505411667a7b51a8f349f8c6c4f3b66] Merge tag 'locking-core-2021-02-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 9eef02334505411667a7b51a8f349f8c6c4f3b66 with gcc (GCC) 10.2.1 20210217 kernel signature: 19c1ba5885ef94ecdbb255d1c97c24e1d175ef45db97a04ab10a9b1cc55c158b run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: OK run #6: OK run #7: basic kernel testing failed: failed to copy test binary to VM: timedout ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "/tmp/syz-executor392730164" "root@10.128.1.2:./syz-executor392730164"] run #8: boot failed: WARNING in kvm_wait run #9: boot failed: WARNING in kvm_wait # git bisect bad 9eef02334505411667a7b51a8f349f8c6c4f3b66 Bisecting: 13 revisions left to test after this (roughly 4 steps) [442187f3c2de40bab13b8f9751b37925bede73b0] locking/rwsem: Remove empty rwsem.h testing commit 442187f3c2de40bab13b8f9751b37925bede73b0 with gcc (GCC) 10.2.1 20210217 kernel signature: a56c73b95c38fd7279077739ffd38c782c171c711f7e05b6eb18b30664a99083 run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: crashed: WARNING in kvm_wait run #6: crashed: WARNING in kvm_wait run #7: crashed: WARNING in kvm_wait run #8: crashed: WARNING in kvm_wait run #9: OK # git bisect bad 442187f3c2de40bab13b8f9751b37925bede73b0 Bisecting: 6 revisions left to test after this (roughly 3 steps) [7e923e6a3ceb877497dd9ee70d71fa33b94f332b] locking/selftests: Add local_lock inversion tests testing commit 7e923e6a3ceb877497dd9ee70d71fa33b94f332b with gcc (GCC) 10.2.1 20210217 kernel signature: 7c8257325884b4d0090d3718fcc5a705f2aa6ce136cb0bccd37304dc7b8f9247 all runs: OK # git bisect good 7e923e6a3ceb877497dd9ee70d71fa33b94f332b Bisecting: 3 revisions left to test after this (roughly 2 steps) [997acaf6b4b59c6a9c259740312a69ea549cc684] lockdep: report broken irq restoration testing commit 997acaf6b4b59c6a9c259740312a69ea549cc684 with gcc (GCC) 10.2.1 20210217 kernel signature: d97410920c25b0aa45d97b3ef5bf96ef6eb46b1a562b2dc990337fd4cada67ff run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: crashed: WARNING in kvm_wait run #6: crashed: WARNING in kvm_wait run #7: OK run #8: OK run #9: crashed: WARNING in kvm_wait # git bisect bad 997acaf6b4b59c6a9c259740312a69ea549cc684 Bisecting: 0 revisions left to test after this (roughly 1 step) [2f0df49c89acaa58571d509830bc481250699885] jump_label: Do not profile branch annotations testing commit 2f0df49c89acaa58571d509830bc481250699885 with gcc (GCC) 10.2.1 20210217 kernel signature: 7c8257325884b4d0090d3718fcc5a705f2aa6ce136cb0bccd37304dc7b8f9247 all runs: OK # git bisect good 2f0df49c89acaa58571d509830bc481250699885 997acaf6b4b59c6a9c259740312a69ea549cc684 is the first bad commit commit 997acaf6b4b59c6a9c259740312a69ea549cc684 Author: Mark Rutland Date: Mon Jan 11 15:37:07 2021 +0000 lockdep: report broken irq restoration We generally expect local_irq_save() and local_irq_restore() to be paired and sanely nested, and so local_irq_restore() expects to be called with irqs disabled. Thus, within local_irq_restore() we only trace irq flag changes when unmasking irqs. This means that a sequence such as: | local_irq_disable(); | local_irq_save(flags); | local_irq_enable(); | local_irq_restore(flags); ... is liable to break things, as the local_irq_restore() would mask irqs without tracing this change. Similar problems may exist for architectures whose arch_irq_restore() function depends on being called with irqs disabled. We don't consider such sequences to be a good idea, so let's define those as forbidden, and add tooling to detect such broken cases. This patch adds debug code to WARN() when raw_local_irq_restore() is called with irqs enabled. As raw_local_irq_restore() is expected to pair with raw_local_irq_save(), it should never be called with irqs enabled. To avoid the possibility of circular header dependencies between irqflags.h and bug.h, the warning is handled in a separate C file. The new code is all conditional on a new CONFIG_DEBUG_IRQFLAGS symbol which is independent of CONFIG_TRACE_IRQFLAGS. As noted above such cases will confuse lockdep, so CONFIG_DEBUG_LOCKDEP now selects CONFIG_DEBUG_IRQFLAGS. Signed-off-by: Mark Rutland Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210111153707.10071-1-mark.rutland@arm.com include/linux/irqflags.h | 12 ++++++++++++ kernel/locking/Makefile | 1 + kernel/locking/irqflag-debug.c | 11 +++++++++++ lib/Kconfig.debug | 8 ++++++++ 4 files changed, 32 insertions(+) create mode 100644 kernel/locking/irqflag-debug.c culprit signature: d97410920c25b0aa45d97b3ef5bf96ef6eb46b1a562b2dc990337fd4cada67ff parent signature: 7c8257325884b4d0090d3718fcc5a705f2aa6ce136cb0bccd37304dc7b8f9247 revisions tested: 15, total time: 3h52m11.87855939s (build: 1h38m35.148598075s, test: 2h11m36.282389147s) first bad commit: 997acaf6b4b59c6a9c259740312a69ea549cc684 lockdep: report broken irq restoration recipients (to): ["linux-kernel@vger.kernel.org" "mark.rutland@arm.com" "mingo@redhat.com" "peterz@infradead.org" "peterz@infradead.org" "will@kernel.org"] recipients (cc): ["akpm@linux-foundation.org" "masahiroy@kernel.org" "rafael.j.wysocki@intel.com" "rostedt@goodmis.org" "tglx@linutronix.de"] crash: WARNING in kvm_wait ------------[ cut here ]------------ raw_local_irq_restore() called with IRQs enabled WARNING: CPU: 0 PID: 8762 at kernel/locking/irqflag-debug.c:9 warn_bogus_irq_restore kernel/locking/irqflag-debug.c:9 [inline] WARNING: CPU: 0 PID: 8762 at kernel/locking/irqflag-debug.c:9 warn_bogus_irq_restore+0x1d/0x20 kernel/locking/irqflag-debug.c:7 Modules linked in: CPU: 0 PID: 8762 Comm: syz-execprog Not tainted 5.11.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:warn_bogus_irq_restore kernel/locking/irqflag-debug.c:9 [inline] RIP: 0010:warn_bogus_irq_restore+0x1d/0x20 kernel/locking/irqflag-debug.c:7 Code: 51 00 e9 3f fe ff ff cc cc cc cc cc cc 80 3d e0 b4 ce 0a 00 74 01 c3 48 c7 c7 60 f5 8a 88 c6 05 cf b4 ce 0a 01 e8 17 01 a4 06 <0f> 0b c3 48 c7 c0 a0 46 4d 8e 53 48 89 fb 48 ba 00 00 00 00 00 fc RSP: 0018:ffffc9000163fa18 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffffffff8a620fe0 RCX: 0000000000000000 RDX: 0000000000000002 RSI: ffffffff88ddbcc0 RDI: fffff520002c7f35 RBP: 0000000000000246 R08: 0000000000000001 R09: ffff8880b9e30827 R10: ffffed10173c6104 R11: 0000000000000001 R12: 0000000000000003 R13: fffffbfff14c41fc R14: 0000000000000001 R15: ffff8880b9e359c0 FS: 0000000000000000(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2804ae7718 CR3: 000000000a48e000 CR4: 00000000001506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: kvm_wait arch/x86/kernel/kvm.c:860 [inline] kvm_wait+0xc3/0xe0 arch/x86/kernel/kvm.c:837 pv_wait arch/x86/include/asm/paravirt.h:564 [inline] pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline] __pv_queued_spin_lock_slowpath+0x8b8/0xb40 kernel/locking/qspinlock.c:508 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:554 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline] queued_spin_lock include/asm-generic/qspinlock.h:85 [inline] do_raw_spin_lock+0x200/0x2b0 kernel/locking/spinlock_debug.c:113 spin_lock include/linux/spinlock.h:354 [inline] check_stack_usage kernel/exit.c:715 [inline] do_exit+0x1850/0x2570 kernel/exit.c:868 do_group_exit+0xe7/0x290 kernel/exit.c:922 get_signal+0x333/0x1bd0 kernel/signal.c:2770 arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811 handle_signal_work kernel/entry/common.c:147 [inline] exit_to_user_mode_loop kernel/entry/common.c:171 [inline] exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:201 __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline] syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:302 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x46ca23 Code: Unable to access opcode bytes at RIP 0x46c9f9. RSP: 002b:000000c00003dee0 EFLAGS: 00000286 ORIG_RAX: 00000000000000ca RAX: fffffffffffffe00 RBX: 000000c000275000 RCX: 000000000046ca23 RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000c000275148 RBP: 000000c00003df28 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000286 R12: 00000000000000f4 R13: 0000000000000000 R14: 000000000083f1a2 R15: 0000000000000000