bisecting cause commit starting from df04fbe8680bfe07f3d7487eccff9f768bb02533 building syzkaller on 38a885d1c94053040c0881240224dd60328a4836 testing commit df04fbe8680bfe07f3d7487eccff9f768bb02533 with gcc (GCC) 10.2.1 20210217 kernel signature: 2b19318c65346dab044cb147edaeb18f6b64fbed40f83b6c412aeb3b9c0a62d5 run #0: basic kernel testing failed: possible deadlock in fs_reclaim_acquire run #1: basic kernel testing failed: possible deadlock in fs_reclaim_acquire run #2: crashed: BUG: spinlock bad magic in synchronize_srcu run #3: crashed: general protection fault in rcu_segcblist_enqueue run #4: crashed: general protection fault in rcu_segcblist_enqueue run #5: crashed: general protection fault in rcu_segcblist_enqueue run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK reproducer seems to be flaky testing release v5.13 testing commit 62fb9874f5da54fdb243003b386128037319b219 with gcc (GCC) 10.2.1 20210217 kernel signature: 88bd5fb2e08990cddd91902619765862bb163346f2acc8c02b62983e11d863c6 run #0: crashed: BUG: spinlock bad magic in synchronize_srcu run #1: crashed: BUG: spinlock bad magic in synchronize_srcu run #2: crashed: BUG: spinlock bad magic in synchronize_srcu run #3: crashed: BUG: spinlock bad magic in synchronize_srcu run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK testing release v5.12 testing commit 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 with gcc (GCC) 10.2.1 20210217 kernel signature: dcc28281bf34c11f1b173d9393d066a98426ea9e53b3a60aa232fc5e21961164 run #0: crashed: BUG: spinlock bad magic in synchronize_srcu run #1: crashed: BUG: spinlock bad magic in synchronize_srcu run #2: crashed: BUG: spinlock bad magic in synchronize_srcu run #3: crashed: BUG: spinlock bad magic in synchronize_srcu run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK testing release v5.11 testing commit f40ddce88593482919761f74910f42f4b84c004b with gcc (GCC) 10.2.1 20210217 kernel signature: ab08d6a200d9c497e38b623298a2aaf5238d520eb2870b77952104eaebb9c2d5 all runs: OK # git bisect start 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 f40ddce88593482919761f74910f42f4b84c004b Bisecting: 6798 revisions left to test after this (roughly 13 steps) [d99676af540c2dc829999928fb81c58c80a1dce4] Merge tag 'drm-next-2021-02-19' of git://anongit.freedesktop.org/drm/drm testing commit d99676af540c2dc829999928fb81c58c80a1dce4 with gcc (GCC) 10.2.1 20210217 kernel signature: 21cfea88744581c12db37e78e8c0e8337748ec58b5de81848349b2d0c51d5183 run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad d99676af540c2dc829999928fb81c58c80a1dce4 Bisecting: 3717 revisions left to test after this (roughly 12 steps) [f9d58de23152f2c16f326d7e014cfa2933b00304] Merge tag 'affs-for-5.12-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux testing commit f9d58de23152f2c16f326d7e014cfa2933b00304 with gcc (GCC) 10.2.1 20210217 kernel signature: 0067b0b3e16efdb632fb5694ff888b29586b2033eb8e3a4e6069caaf64fa3b56 all runs: OK # git bisect good f9d58de23152f2c16f326d7e014cfa2933b00304 Bisecting: 1854 revisions left to test after this (roughly 11 steps) [de1617578849acab8e16c9ffdce39b91fb50639d] Merge tag 'media/v5.12-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media testing commit de1617578849acab8e16c9ffdce39b91fb50639d with gcc (GCC) 10.2.1 20210217 kernel signature: ae4bdd60cade51b9fe41317a1f8cbe4ec7287f69a73c813d4a5add22a80f3ef2 run #0: crashed: WARNING in kvm_wait run #1: crashed: BUG: spinlock bad magic in synchronize_srcu run #2: crashed: BUG: spinlock bad magic in synchronize_srcu run #3: crashed: BUG: spinlock bad magic in synchronize_srcu run #4: crashed: BUG: spinlock bad magic in synchronize_srcu run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: boot failed: WARNING in kvm_wait # git bisect bad de1617578849acab8e16c9ffdce39b91fb50639d Bisecting: 929 revisions left to test after this (roughly 10 steps) [4a037ad5d115b2cc79a5071a7854475f365476fa] Merge tag 'for-linus-5.12-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip testing commit 4a037ad5d115b2cc79a5071a7854475f365476fa with gcc (GCC) 10.2.1 20210217 kernel signature: 21d55dd39058eb0b563b7373dd5ad20a2f3a431571a966aa06f410b31dd4e4cf run #0: crashed: WARNING in kvm_wait run #1: crashed: WARNING in kvm_wait run #2: OK run #3: crashed: WARNING in kvm_wait run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad 4a037ad5d115b2cc79a5071a7854475f365476fa Bisecting: 451 revisions left to test after this (roughly 9 steps) [582cd91f69de8e44857cb610ebca661dac8656b7] Merge tag 'for-5.12/block-2021-02-17' of git://git.kernel.dk/linux-block testing commit 582cd91f69de8e44857cb610ebca661dac8656b7 with gcc (GCC) 10.2.1 20210217 kernel signature: faa3407a426fa9bc343293ba9ae36e7444e1ba24fd6fdf2fa4abfd8c1e52e4dd run #0: crashed: BUG: spinlock bad magic in synchronize_srcu run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad 582cd91f69de8e44857cb610ebca661dac8656b7 Bisecting: 233 revisions left to test after this (roughly 8 steps) [99f1a5872b706094ece117368170a92c66b2e242] Merge tag 'nfsd-5.12' of git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux testing commit 99f1a5872b706094ece117368170a92c66b2e242 with gcc (GCC) 10.2.1 20210217 kernel signature: c307d65aa571cab17b6d3486a6580a90d6a4cd3c76ade158ba440e9fa40a8d74 all runs: OK # git bisect good 99f1a5872b706094ece117368170a92c66b2e242 Bisecting: 107 revisions left to test after this (roughly 7 steps) [24880bef417f6e9069158c750969d18793427a10] Merge tag 'oprofile-removal-5.12' of git://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux testing commit 24880bef417f6e9069158c750969d18793427a10 with gcc (GCC) 10.2.1 20210217 kernel signature: 4bb96619203d3ee876ad345e6b1fd7441fbf61b2e986529a68799e9540b8ac49 all runs: OK # git bisect good 24880bef417f6e9069158c750969d18793427a10 Bisecting: 53 revisions left to test after this (roughly 6 steps) [482e302a61f1fc62b0e13be20bc7a11a91b5832d] blk: wbt: remove unused parameter from wbt_should_throttle testing commit 482e302a61f1fc62b0e13be20bc7a11a91b5832d with gcc (GCC) 10.2.1 20210217 kernel signature: 0b72f5bb9b2ea7de6d818625d35c2bd55a7b54f3e17060ae644fe59bdbabeefc run #0: crashed: BUG: spinlock bad magic in synchronize_srcu run #1: crashed: BUG: spinlock bad magic in synchronize_srcu run #2: crashed: BUG: spinlock bad magic in synchronize_srcu run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad 482e302a61f1fc62b0e13be20bc7a11a91b5832d Bisecting: 26 revisions left to test after this (roughly 5 steps) [c495a17679523c95f77f13697a71921dd5c224cd] block: don't pass BIOSET_NEED_BVECS for q->bio_split testing commit c495a17679523c95f77f13697a71921dd5c224cd with gcc (GCC) 10.2.1 20210217 kernel signature: 787a89a0c5ae3f194d53ee6b27bfbea0f102c7434081b16008f6d39bad1d01f4 all runs: OK # git bisect good c495a17679523c95f77f13697a71921dd5c224cd Bisecting: 13 revisions left to test after this (roughly 4 steps) [3e1a88ec96259282b9a8b45c3f1fda7a3ff4f6ea] bio: add a helper calculating nr segments to alloc testing commit 3e1a88ec96259282b9a8b45c3f1fda7a3ff4f6ea with gcc (GCC) 10.2.1 20210217 kernel signature: 730c763b269d33f352397086346db18cb083d03647574f3e3c6043fdd85098c3 all runs: OK # git bisect good 3e1a88ec96259282b9a8b45c3f1fda7a3ff4f6ea Bisecting: 6 revisions left to test after this (roughly 3 steps) [5a5436b98d5cd2714feaaa579cec49dd7f7057bb] block, bfq: save also injection state on queue merging testing commit 5a5436b98d5cd2714feaaa579cec49dd7f7057bb with gcc (GCC) 10.2.1 20210217 kernel signature: 5361a79aa795d033537d99ec6bed71c5b92c0fe0865c197bb3e0f80ebd902463 all runs: OK # git bisect good 5a5436b98d5cd2714feaaa579cec49dd7f7057bb Bisecting: 3 revisions left to test after this (roughly 2 steps) [a7c7f7b2b641bef52212fbe8be4a66ede043d3c7] nvme: use bio_set_dev to assign ->bi_bdev testing commit a7c7f7b2b641bef52212fbe8be4a66ede043d3c7 with gcc (GCC) 10.2.1 20210217 kernel signature: d14d9aa3085b005ae767e4c9d5667464e9cb778b0a5365050a77b5eb85f2a11c run #0: crashed: BUG: spinlock bad magic in synchronize_srcu run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad a7c7f7b2b641bef52212fbe8be4a66ede043d3c7 Bisecting: 0 revisions left to test after this (roughly 1 step) [a5bf0a92e1b8282c93018383b2526ca59602dd08] bfq: bfq_check_waker() should be static testing commit a5bf0a92e1b8282c93018383b2526ca59602dd08 with gcc (GCC) 10.2.1 20210217 kernel signature: 5b37d4b5d8f263d415e5d7c38f16d02ae1ba4019c73bdd718c4ffe0a9cc9176c run #0: crashed: BUG: spinlock bad magic in synchronize_srcu run #1: crashed: BUG: spinlock bad magic in synchronize_srcu run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad a5bf0a92e1b8282c93018383b2526ca59602dd08 Bisecting: 0 revisions left to test after this (roughly 0 steps) [71217df39dc67a0aeed83352b0d712b7892036a2] block, bfq: make waker-queue detection more robust testing commit 71217df39dc67a0aeed83352b0d712b7892036a2 with gcc (GCC) 10.2.1 20210217 kernel signature: e469935cf3d183a40bb653597c6dce709dbb4cc4dd121e5982828ee3db0850f2 run #0: crashed: BUG: spinlock bad magic in synchronize_srcu run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad 71217df39dc67a0aeed83352b0d712b7892036a2 71217df39dc67a0aeed83352b0d712b7892036a2 is the first bad commit commit 71217df39dc67a0aeed83352b0d712b7892036a2 Author: Paolo Valente Date: Mon Jan 25 20:02:48 2021 +0100 block, bfq: make waker-queue detection more robust In the presence of many parallel I/O flows, the detection of waker bfq_queues suffers from false positives. This commits addresses this issue by making the filtering of actual wakers more selective. In more detail, a candidate waker must be found to meet waker requirements three times before being promoted to actual waker. Tested-by: Jan Kara Signed-off-by: Paolo Valente Signed-off-by: Jens Axboe block/bfq-iosched.c | 211 +++++++++++++++++++++++++--------------------------- block/bfq-iosched.h | 7 +- 2 files changed, 108 insertions(+), 110 deletions(-) culprit signature: e469935cf3d183a40bb653597c6dce709dbb4cc4dd121e5982828ee3db0850f2 parent signature: 5361a79aa795d033537d99ec6bed71c5b92c0fe0865c197bb3e0f80ebd902463 Reproducer flagged being flaky revisions tested: 18, total time: 4h54m29.486111837s (build: 2h4m11.564533507s, test: 2h48m24.866080634s) first bad commit: 71217df39dc67a0aeed83352b0d712b7892036a2 block, bfq: make waker-queue detection more robust recipients (to): ["axboe@kernel.dk" "jack@suse.cz" "paolo.valente@linaro.org"] recipients (cc): [] crash: BUG: spinlock bad magic in synchronize_srcu BUG: spinlock bad magic on CPU#0, syz-executor.3/751 lock: 0xffff8880b9e00040, .magic: 00000000, .owner: /-1, .owner_cpu: 0 CPU: 0 PID: 751 Comm: syz-executor.3 Not tainted 5.11.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0x9a/0xcc lib/dump_stack.c:120 debug_spin_lock_before kernel/locking/spinlock_debug.c:83 [inline] do_raw_spin_lock+0x216/0x2b0 kernel/locking/spinlock_debug.c:112 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:117 [inline] _raw_spin_lock_irqsave+0x41/0x50 kernel/locking/spinlock.c:159 srcu_might_be_idle kernel/rcu/srcutree.c:772 [inline] synchronize_srcu+0x4f/0x1c0 kernel/rcu/srcutree.c:999 kvm_mmu_uninit_vm+0x10/0x20 arch/x86/kvm/mmu/mmu.c:5483 kvm_arch_destroy_vm+0x42c/0x5f0 arch/x86/kvm/x86.c:10552 kvm_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:820 [inline] kvm_dev_ioctl_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:4037 [inline] kvm_dev_ioctl+0xc64/0x1100 arch/x86/kvm/../../../virt/kvm/kvm_main.c:4089 vfs_ioctl fs/ioctl.c:48 [inline] __do_sys_ioctl fs/ioctl.c:753 [inline] __se_sys_ioctl fs/ioctl.c:739 [inline] __x64_sys_ioctl+0x11f/0x190 fs/ioctl.c:739 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x4665d9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fc537799188 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 000000000056bf80 RCX: 00000000004665d9 RDX: 0000000000000000 RSI: 000000000000ae01 RDI: 0000000000000003 RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf80 R13: 00007ffc26e3e8ff R14: 00007fc537799300 R15: 0000000000022000 general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007] CPU: 0 PID: 751 Comm: syz-executor.3 Not tainted 5.11.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:rcu_segcblist_enqueue+0x90/0xf0 kernel/rcu/rcu_segcblist.c:250 Code: 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 75 4e 48 b8 00 00 00 00 00 fc ff df 48 8b 6b 20 48 89 ea 48 c1 ea 03 <80> 3c 02 00 75 21 48 89 75 00 48 89 73 20 48 83 c4 08 5b 5d c3 48 RSP: 0018:ffffc90005d4fc10 EFLAGS: 00010046 RAX: dffffc0000000000 RBX: ffff8880b9e00080 RCX: ffffffff8151d5f0 RDX: 0000000000000000 RSI: ffffc90005d4fcf8 RDI: ffff8880b9e000a0 RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000003 R10: fffff52000ba9f74 R11: 6637303030302052 R12: ffffc90005d4fcf8 R13: 0000000000000000 R14: ffff8880b9e00080 R15: ffff8880b9e00040 FS: 00007fc537799700(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000511ef0 CR3: 0000000010345000 CR4: 00000000001526f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __call_srcu+0x193/0xc50 kernel/rcu/srcutree.c:859 __synchronize_srcu+0x128/0x220 kernel/rcu/srcutree.c:923 kvm_mmu_uninit_vm+0x10/0x20 arch/x86/kvm/mmu/mmu.c:5483 kvm_arch_destroy_vm+0x42c/0x5f0 arch/x86/kvm/x86.c:10552 kvm_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:820 [inline] kvm_dev_ioctl_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:4037 [inline] kvm_dev_ioctl+0xc64/0x1100 arch/x86/kvm/../../../virt/kvm/kvm_main.c:4089 vfs_ioctl fs/ioctl.c:48 [inline] __do_sys_ioctl fs/ioctl.c:753 [inline] __se_sys_ioctl fs/ioctl.c:739 [inline] __x64_sys_ioctl+0x11f/0x190 fs/ioctl.c:739 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x4665d9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fc537799188 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 000000000056bf80 RCX: 00000000004665d9 RDX: 0000000000000000 RSI: 000000000000ae01 RDI: 0000000000000003 RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf80 R13: 00007ffc26e3e8ff R14: 00007fc537799300 R15: 0000000000022000 Modules linked in: ---[ end trace 5d0f43e3124d8bde ]--- RIP: 0010:rcu_segcblist_enqueue+0x90/0xf0 kernel/rcu/rcu_segcblist.c:250 Code: 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 75 4e 48 b8 00 00 00 00 00 fc ff df 48 8b 6b 20 48 89 ea 48 c1 ea 03 <80> 3c 02 00 75 21 48 89 75 00 48 89 73 20 48 83 c4 08 5b 5d c3 48 RSP: 0018:ffffc90005d4fc10 EFLAGS: 00010046 RAX: dffffc0000000000 RBX: ffff8880b9e00080 RCX: ffffffff8151d5f0 RDX: 0000000000000000 RSI: ffffc90005d4fcf8 RDI: ffff8880b9e000a0 RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000003 R10: fffff52000ba9f74 R11: 6637303030302052 R12: ffffc90005d4fcf8 R13: 0000000000000000 R14: ffff8880b9e00080 R15: ffff8880b9e00040 FS: 00007fc537799700(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000511ef0 CR3: 0000000010345000 CR4: 00000000001526f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400