bisecting cause commit starting from d09c548dbf3b31cb07bba562e0f452edfa01efe3 building syzkaller on 6972b10616d785401dea17cec890cca8916424a7 testing commit d09c548dbf3b31cb07bba562e0f452edfa01efe3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 6d6070afdadb23caa2cb93d37403a2aca0f90ece515c78fecc495e422d772a3d all runs: crashed: general protection fault in hwsim_new_edge_nl testing release v5.13 testing commit 62fb9874f5da54fdb243003b386128037319b219 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 909f5b513362d755e99068620c19525cb2afd91ca8725228219fd46574a48866 all runs: crashed: general protection fault in hwsim_new_edge_nl testing release v5.12 testing commit 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: bb4e26dd8f4a005b0fd76547575ef29621d4d502b5251ddaf0a5696974104629 all runs: crashed: general protection fault in hwsim_new_edge_nl testing release v5.11 testing commit f40ddce88593482919761f74910f42f4b84c004b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 3222e6ed80513dd888ba5df1ed0bb0fed90b38311471a07f8ebfef14f678fd42 all runs: OK # git bisect start 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 f40ddce88593482919761f74910f42f4b84c004b Bisecting: 6798 revisions left to test after this (roughly 13 steps) [d99676af540c2dc829999928fb81c58c80a1dce4] Merge tag 'drm-next-2021-02-19' of git://anongit.freedesktop.org/drm/drm testing commit d99676af540c2dc829999928fb81c58c80a1dce4 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: f0c248143fbb01c9220d7d749817b4bd905f8d78b2dcc1efd4218ef484b8cf57 run #0: basic kernel testing failed: WARNING in kvm_wait run #1: basic kernel testing failed: WARNING in kvm_wait run #2: basic kernel testing failed: WARNING in kvm_wait run #3: basic kernel testing failed: WARNING in kvm_wait run #4: basic kernel testing failed: WARNING in kvm_wait run #5: crashed: WARNING in kvm_wait run #6: crashed: WARNING in kvm_wait run #7: boot failed: WARNING in kvm_wait run #8: boot failed: WARNING in kvm_wait run #9: boot failed: WARNING in kvm_wait # git bisect bad d99676af540c2dc829999928fb81c58c80a1dce4 Bisecting: 3717 revisions left to test after this (roughly 12 steps) [f9d58de23152f2c16f326d7e014cfa2933b00304] Merge tag 'affs-for-5.12-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux testing commit f9d58de23152f2c16f326d7e014cfa2933b00304 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: a6431cb265fb2e1799b1881f6872b94bbbc521195447951498ba672474e46152 all runs: OK # git bisect good f9d58de23152f2c16f326d7e014cfa2933b00304 Bisecting: 1854 revisions left to test after this (roughly 11 steps) [de1617578849acab8e16c9ffdce39b91fb50639d] Merge tag 'media/v5.12-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media testing commit de1617578849acab8e16c9ffdce39b91fb50639d compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 0ad46f3eba83e6471ddd86d4ddf8dc64a0d769474ec1a2148fcc6e70ba5ea1ff run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: crashed: WARNING in kvm_wait run #6: boot failed: WARNING in kvm_wait run #7: boot failed: WARNING in kvm_wait run #8: boot failed: WARNING in kvm_wait run #9: boot failed: WARNING in kvm_wait # git bisect bad de1617578849acab8e16c9ffdce39b91fb50639d Bisecting: 929 revisions left to test after this (roughly 10 steps) [4a037ad5d115b2cc79a5071a7854475f365476fa] Merge tag 'for-linus-5.12-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip testing commit 4a037ad5d115b2cc79a5071a7854475f365476fa compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: aa8c9d8fa3261d366fd143ca070fd33032ef630e9463cf8c4cfb3607b47317d1 run #0: basic kernel testing failed: WARNING in kvm_wait run #1: basic kernel testing failed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: basic kernel testing failed: failed to copy test binary to VM: timedout ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "/syzkaller/jobs/linux/gopath/src/github.com/google/syzkaller/bin/linux_amd64/syz-execprog" "root@10.128.10.0:./syz-execprog"] Warning: Permanently added '10.128.10.0' (ECDSA) to the list of known hosts. run #5: crashed: WARNING in kvm_wait run #6: crashed: WARNING in kvm_wait run #7: boot failed: WARNING in kvm_wait run #8: boot failed: WARNING in kvm_wait run #9: boot failed: WARNING in kvm_wait # git bisect bad 4a037ad5d115b2cc79a5071a7854475f365476fa Bisecting: 451 revisions left to test after this (roughly 9 steps) [582cd91f69de8e44857cb610ebca661dac8656b7] Merge tag 'for-5.12/block-2021-02-17' of git://git.kernel.dk/linux-block testing commit 582cd91f69de8e44857cb610ebca661dac8656b7 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: d09c1a632a47f797f7f0e07e6a91fa440e604b9d1f750b792f8ceec7583a5d4d all runs: OK # git bisect good 582cd91f69de8e44857cb610ebca661dac8656b7 Bisecting: 247 revisions left to test after this (roughly 8 steps) [3f6ec19f2d05d800bbc42d95dece433da7697864] Merge tag 'timers-core-2021-02-15' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 3f6ec19f2d05d800bbc42d95dece433da7697864 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 40f134f4973ed2405f14f8d163a96b11a4f9817003c1bc1280ca38414a45ce90 all runs: OK # git bisect good 3f6ec19f2d05d800bbc42d95dece433da7697864 Bisecting: 125 revisions left to test after this (roughly 7 steps) [85e853c5ec8486117182baab10c98b321daa6d47] Merge branch 'for-mingo-rcu' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu testing commit 85e853c5ec8486117182baab10c98b321daa6d47 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 2f9c105bf2f762d79083df9bea690f09508e72a816ef08d9985d50664b18e1c1 all runs: OK # git bisect good 85e853c5ec8486117182baab10c98b321daa6d47 Bisecting: 73 revisions left to test after this (roughly 6 steps) [c5e6fc08feb2b88dc5dac2f3c817e1c2a4cafda4] sched,x86: Allow !PREEMPT_DYNAMIC testing commit c5e6fc08feb2b88dc5dac2f3c817e1c2a4cafda4 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 12482fdae8b81bb545eb8b34903680d860b7f7aface473468f603c56a877c702 all runs: OK # git bisect good c5e6fc08feb2b88dc5dac2f3c817e1c2a4cafda4 Bisecting: 33 revisions left to test after this (roughly 5 steps) [9eef02334505411667a7b51a8f349f8c6c4f3b66] Merge tag 'locking-core-2021-02-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 9eef02334505411667a7b51a8f349f8c6c4f3b66 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: ea61eb9ac361737a8f433b9996d0921a13bb47bb482e4c06822fc32af694c193 run #0: basic kernel testing failed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: crashed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: crashed: WARNING in kvm_wait run #6: boot failed: WARNING in kvm_wait run #7: boot failed: WARNING in kvm_wait run #8: boot failed: WARNING in kvm_wait run #9: boot failed: WARNING in kvm_wait # git bisect bad 9eef02334505411667a7b51a8f349f8c6c4f3b66 Bisecting: 19 revisions left to test after this (roughly 4 steps) [62137364e3e8afcc745846c5c67cacf943149073] Merge branch 'linus' into locking/core, to pick up upstream fixes testing commit 62137364e3e8afcc745846c5c67cacf943149073 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 2417d89124420523a73004e4c2cff21eb410846167d1cb6183242602fc9871e8 run #0: basic kernel testing failed: WARNING in kvm_wait run #1: basic kernel testing failed: WARNING in kvm_wait run #2: basic kernel testing failed: WARNING in kvm_wait run #3: basic kernel testing failed: WARNING in kvm_wait run #4: crashed: WARNING in kvm_wait run #5: boot failed: WARNING in kvm_wait run #6: boot failed: WARNING in kvm_wait run #7: boot failed: WARNING in kvm_wait run #8: boot failed: WARNING in kvm_wait run #9: boot failed: WARNING in kvm_wait # git bisect bad 62137364e3e8afcc745846c5c67cacf943149073 Bisecting: 9 revisions left to test after this (roughly 3 steps) [997acaf6b4b59c6a9c259740312a69ea549cc684] lockdep: report broken irq restoration testing commit 997acaf6b4b59c6a9c259740312a69ea549cc684 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 81423d473fe48a229b2dd408865bd71fdf67cd81f07c6cf901cfda35974ed86b run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: crashed: WARNING in kvm_wait run #3: boot failed: WARNING in kvm_wait run #4: boot failed: WARNING in kvm_wait run #5: boot failed: WARNING in kvm_wait run #6: boot failed: WARNING in kvm_wait run #7: boot failed: WARNING in kvm_wait run #8: boot failed: WARNING in kvm_wait run #9: boot failed: WARNING in kvm_wait # git bisect bad 997acaf6b4b59c6a9c259740312a69ea549cc684 Bisecting: 4 revisions left to test after this (roughly 2 steps) [175b1a60e8805617d74aefe17ce0d3a32eceb55c] locking/lockdep: Clean up check_redundant() a bit testing commit 175b1a60e8805617d74aefe17ce0d3a32eceb55c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 8a3aec7ca66dce10e60efcc6d46dae50902286e89bcb17af8d4f53a8959ed7db all runs: OK # git bisect good 175b1a60e8805617d74aefe17ce0d3a32eceb55c Bisecting: 2 revisions left to test after this (roughly 1 step) [7e923e6a3ceb877497dd9ee70d71fa33b94f332b] locking/selftests: Add local_lock inversion tests testing commit 7e923e6a3ceb877497dd9ee70d71fa33b94f332b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 643ee273daae5d8a3cf870c0fb5ff5a76380a1070d7d479a8e594d33b7a89dac all runs: OK # git bisect good 7e923e6a3ceb877497dd9ee70d71fa33b94f332b Bisecting: 0 revisions left to test after this (roughly 1 step) [2f0df49c89acaa58571d509830bc481250699885] jump_label: Do not profile branch annotations testing commit 2f0df49c89acaa58571d509830bc481250699885 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 643ee273daae5d8a3cf870c0fb5ff5a76380a1070d7d479a8e594d33b7a89dac all runs: OK # git bisect good 2f0df49c89acaa58571d509830bc481250699885 997acaf6b4b59c6a9c259740312a69ea549cc684 is the first bad commit commit 997acaf6b4b59c6a9c259740312a69ea549cc684 Author: Mark Rutland Date: Mon Jan 11 15:37:07 2021 +0000 lockdep: report broken irq restoration We generally expect local_irq_save() and local_irq_restore() to be paired and sanely nested, and so local_irq_restore() expects to be called with irqs disabled. Thus, within local_irq_restore() we only trace irq flag changes when unmasking irqs. This means that a sequence such as: | local_irq_disable(); | local_irq_save(flags); | local_irq_enable(); | local_irq_restore(flags); ... is liable to break things, as the local_irq_restore() would mask irqs without tracing this change. Similar problems may exist for architectures whose arch_irq_restore() function depends on being called with irqs disabled. We don't consider such sequences to be a good idea, so let's define those as forbidden, and add tooling to detect such broken cases. This patch adds debug code to WARN() when raw_local_irq_restore() is called with irqs enabled. As raw_local_irq_restore() is expected to pair with raw_local_irq_save(), it should never be called with irqs enabled. To avoid the possibility of circular header dependencies between irqflags.h and bug.h, the warning is handled in a separate C file. The new code is all conditional on a new CONFIG_DEBUG_IRQFLAGS symbol which is independent of CONFIG_TRACE_IRQFLAGS. As noted above such cases will confuse lockdep, so CONFIG_DEBUG_LOCKDEP now selects CONFIG_DEBUG_IRQFLAGS. Signed-off-by: Mark Rutland Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210111153707.10071-1-mark.rutland@arm.com include/linux/irqflags.h | 12 ++++++++++++ kernel/locking/Makefile | 1 + kernel/locking/irqflag-debug.c | 11 +++++++++++ lib/Kconfig.debug | 8 ++++++++ 4 files changed, 32 insertions(+) create mode 100644 kernel/locking/irqflag-debug.c culprit signature: 81423d473fe48a229b2dd408865bd71fdf67cd81f07c6cf901cfda35974ed86b parent signature: 643ee273daae5d8a3cf870c0fb5ff5a76380a1070d7d479a8e594d33b7a89dac revisions tested: 18, total time: 4h39m58.860216765s (build: 2h2m35.674568251s, test: 2h35m18.704581569s) first bad commit: 997acaf6b4b59c6a9c259740312a69ea549cc684 lockdep: report broken irq restoration recipients (to): ["linux-kernel@vger.kernel.org" "mark.rutland@arm.com" "mingo@redhat.com" "peterz@infradead.org" "peterz@infradead.org" "will@kernel.org"] recipients (cc): ["akpm@linux-foundation.org" "frederic@kernel.org" "masahiroy@kernel.org" "npiggin@gmail.com" "rafael.j.wysocki@intel.com" "rostedt@goodmis.org" "sedat.dilek@gmail.com" "tglx@linutronix.de" "vitor@massaru.org"] crash: WARNING in kvm_wait ------------[ cut here ]------------ raw_local_irq_restore() called with IRQs enabled WARNING: CPU: 0 PID: 8750 at kernel/locking/irqflag-debug.c:9 warn_bogus_irq_restore kernel/locking/irqflag-debug.c:9 [inline] WARNING: CPU: 0 PID: 8750 at kernel/locking/irqflag-debug.c:9 warn_bogus_irq_restore+0x1d/0x20 kernel/locking/irqflag-debug.c:7 Modules linked in: CPU: 0 PID: 8750 Comm: syz-executor.2 Not tainted 5.11.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:warn_bogus_irq_restore kernel/locking/irqflag-debug.c:9 [inline] RIP: 0010:warn_bogus_irq_restore+0x1d/0x20 kernel/locking/irqflag-debug.c:7 Code: 51 00 e9 3f fe ff ff cc cc cc cc cc cc 80 3d c0 03 cf 0a 00 74 01 c3 48 c7 c7 60 f5 8a 88 c6 05 af 03 cf 0a 01 e8 17 71 a5 06 <0f> 0b c3 48 c7 c0 a0 96 4f 8e 53 48 89 fb 48 ba 00 00 00 00 00 fc RSP: 0018:ffffc9000175f9f8 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ffff88814a40e040 RCX: 0000000000000000 RDX: 0000000000000002 RSI: ffffffff88ddc880 RDI: fffff520002ebf31 RBP: 0000000000000246 R08: 0000000000000001 R09: ffff8880b9e4c827 R10: ffffed10173c9904 R11: 0000000000000001 R12: 0000000000000003 R13: ffffed1029481c08 R14: 0000000000000001 R15: ffff8880b9e519c0 FS: 0000000002caf400(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000002cb8848 CR3: 000000001f2a2000 CR4: 00000000001506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: kvm_wait arch/x86/kernel/kvm.c:860 [inline] kvm_wait+0xc3/0xe0 arch/x86/kernel/kvm.c:837 pv_wait arch/x86/include/asm/paravirt.h:564 [inline] pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline] __pv_queued_spin_lock_slowpath+0x8b8/0xb40 kernel/locking/qspinlock.c:508 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:554 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline] queued_spin_lock include/asm-generic/qspinlock.h:85 [inline] do_raw_spin_lock+0x200/0x2b0 kernel/locking/spinlock_debug.c:113 spin_lock include/linux/spinlock.h:354 [inline] ext4_lock_group fs/ext4/ext4.h:3379 [inline] __ext4_new_inode+0x2da2/0x44d0 fs/ext4/ialloc.c:1187 ext4_mkdir+0x298/0x910 fs/ext4/namei.c:2793 vfs_mkdir+0x413/0x660 fs/namei.c:3652 do_mkdirat+0x1eb/0x250 fs/namei.c:3675 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x4656f7 Code: 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 53 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffc952eca48 EFLAGS: 00000206 ORIG_RAX: 0000000000000053 RAX: ffffffffffffffda RBX: 00007ffc952ecae0 RCX: 00000000004656f7 RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007ffc952ecae0 RBP: 00007ffc952ecabc R08: 0000000000000000 R09: 0000000000000006 R10: 00007ffc952ec7e4 R11: 0000000000000206 R12: 0000000000000032 R13: 0000000000028ed0 R14: 0000000000000002 R15: 00007ffc952ecb20