bisecting cause commit starting from 3dbdb38e286903ec220aaf1fb29a8d94297da246 building syzkaller on 4846d5c1dcbf362c2e9949b24feca9670ca9b4b9 testing commit 3dbdb38e286903ec220aaf1fb29a8d94297da246 with gcc (GCC) 10.2.1 20210217 kernel signature: be86f3f3b2899214e9271a92aa000cb0f01513f966a0a7a9a6242b89884a3943 run #0: basic kernel testing failed: BUG: sleeping function called from invalid context in stack_depot_save run #1: crashed: general protection fault in blk_mq_run_hw_queues run #2: crashed: general protection fault in blk_mq_run_hw_queues run #3: crashed: general protection fault in blk_mq_run_hw_queues run #4: crashed: general protection fault in blk_mq_run_hw_queues run #5: crashed: general protection fault in blk_mq_run_hw_queues run #6: crashed: general protection fault in blk_mq_run_hw_queues run #7: crashed: general protection fault in blk_mq_run_hw_queues run #8: crashed: possible deadlock in del_gendisk run #9: crashed: general protection fault in blk_mq_run_hw_queues run #10: crashed: possible deadlock in del_gendisk run #11: crashed: general protection fault in blk_mq_run_hw_queues run #12: crashed: possible deadlock in del_gendisk run #13: crashed: general protection fault in blk_mq_run_hw_queues run #14: crashed: general protection fault in blk_mq_run_hw_queues run #15: crashed: possible deadlock in del_gendisk run #16: crashed: INFO: task hung in genl_rcv_msg run #17: crashed: INFO: task hung in genl_rcv_msg run #18: crashed: INFO: task hung in genl_rcv_msg run #19: crashed: INFO: task hung in genl_rcv_msg testing release v5.13 testing commit 62fb9874f5da54fdb243003b386128037319b219 with gcc (GCC) 10.2.1 20210217 kernel signature: e9ace9c5ee2829ecfd31f47be41f589c91c15eff6b62baab7051a38e73106e48 all runs: crashed: possible deadlock in del_gendisk testing release v5.12 testing commit 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 with gcc (GCC) 10.2.1 20210217 kernel signature: e8c8edc56ed7eecefb295a9ce860712552bfe61648872ba3f622452206b435fc all runs: OK # git bisect start 62fb9874f5da54fdb243003b386128037319b219 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 Bisecting: 8739 revisions left to test after this (roughly 13 steps) [85f3f17b5db2dd9f8a094a0ddc665555135afd22] Merge branch 'md-fixes' of https://git.kernel.org/pub/scm/linux/kernel/git/song/md into block-5.13 testing commit 85f3f17b5db2dd9f8a094a0ddc665555135afd22 with gcc (GCC) 10.2.1 20210217 kernel signature: fdecd512f2d5271a13216aa39f3c075e543fd6e694cb35e0039ecafd92b435a1 all runs: crashed: possible deadlock in del_gendisk # git bisect bad 85f3f17b5db2dd9f8a094a0ddc665555135afd22 Bisecting: 4263 revisions left to test after this (roughly 12 steps) [ca62e9090d229926f43f20291bb44d67897baab7] Merge tag 'regulator-v5.13' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator testing commit ca62e9090d229926f43f20291bb44d67897baab7 with gcc (GCC) 10.2.1 20210217 kernel signature: b4c52b570b7116522769dab4259b630004173e8bd950c00488e249fda5d46d14 all runs: OK # git bisect good ca62e9090d229926f43f20291bb44d67897baab7 Bisecting: 1907 revisions left to test after this (roughly 11 steps) [68a32ba14177d4a21c4a9a941cf1d7aea86d436f] Merge tag 'drm-next-2021-04-28' of git://anongit.freedesktop.org/drm/drm testing commit 68a32ba14177d4a21c4a9a941cf1d7aea86d436f with gcc (GCC) 10.2.1 20210217 kernel signature: 189525b1edb608ab9f27f7babb04c99c09561f3af95626951c66362a394d8856 all runs: OK # git bisect good 68a32ba14177d4a21c4a9a941cf1d7aea86d436f Bisecting: 968 revisions left to test after this (roughly 10 steps) [be18cd1fcae2ed7db58d92d20733dfa8aa0a5173] Merge tag 'mmc-v5.13' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc testing commit be18cd1fcae2ed7db58d92d20733dfa8aa0a5173 with gcc (GCC) 10.2.1 20210217 kernel signature: 5d6a7dc23ab178708a0d72405e35fcf374dd3e0949c01219ac797a3ada670341 all runs: crashed: possible deadlock in del_gendisk # git bisect bad be18cd1fcae2ed7db58d92d20733dfa8aa0a5173 Bisecting: 473 revisions left to test after this (roughly 9 steps) [fc0586062816559defb14c947319ef8c4c326fb3] Merge tag 'for-5.13/drivers-2021-04-27' of git://git.kernel.dk/linux-block testing commit fc0586062816559defb14c947319ef8c4c326fb3 with gcc (GCC) 10.2.1 20210217 kernel signature: 70e0954121da7e454edb3ba2293d9d8e151742a1be501a70978d2b91f489a1a1 all runs: crashed: possible deadlock in del_gendisk # git bisect bad fc0586062816559defb14c947319ef8c4c326fb3 Bisecting: 256 revisions left to test after this (roughly 8 steps) [42dec9a936e7696bea1f27d3c5a0068cd9aa95fd] Merge tag 'perf-core-2021-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 42dec9a936e7696bea1f27d3c5a0068cd9aa95fd with gcc (GCC) 10.2.1 20210217 kernel signature: 3f9c01ce85ab9e4bcd2bd06de040c5ecec54c4d89ec77897ea2b123565b1e288 all runs: OK # git bisect good 42dec9a936e7696bea1f27d3c5a0068cd9aa95fd Bisecting: 128 revisions left to test after this (roughly 7 steps) [2958a995edc94654df690318df7b9b49e5a3ef88] block/rnbd-clt: Support polling mode for IO latency optimization testing commit 2958a995edc94654df690318df7b9b49e5a3ef88 with gcc (GCC) 10.2.1 20210217 kernel signature: 0d0e7e29dc427196899691a7c5624dc4ba5b8699697466c021346b2956c11fc2 run #0: boot failed: WARNING in kvm_wait run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 2958a995edc94654df690318df7b9b49e5a3ef88 Bisecting: 65 revisions left to test after this (roughly 6 steps) [16b3d0cf5bad844daaf436ad2e9061de0fe36e5c] Merge tag 'sched-core-2021-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 16b3d0cf5bad844daaf436ad2e9061de0fe36e5c with gcc (GCC) 10.2.1 20210217 kernel signature: d98ecf2d9e8d41a49843a0f662018010213aeffb7a854f9ba91dff4f943d7b36 all runs: OK # git bisect good 16b3d0cf5bad844daaf436ad2e9061de0fe36e5c Bisecting: 32 revisions left to test after this (roughly 5 steps) [cbb749cf377aa8aa32a036ebe9dd9f2d89037bf0] block: remove an incorrect check from blk_rq_append_bio testing commit cbb749cf377aa8aa32a036ebe9dd9f2d89037bf0 with gcc (GCC) 10.2.1 20210217 kernel signature: ef2a295c3a9745eef407f9c145c0ca0967877aa76c3a75df6fd1a24848ab9bed all runs: crashed: possible deadlock in del_gendisk # git bisect bad cbb749cf377aa8aa32a036ebe9dd9f2d89037bf0 Bisecting: 16 revisions left to test after this (roughly 4 steps) [9bb33f24abbd0fa2fadad01ec75438d7cc239189] block: refactor the bounce buffering code testing commit 9bb33f24abbd0fa2fadad01ec75438d7cc239189 with gcc (GCC) 10.2.1 20210217 kernel signature: 5c034bc43c4b7428bc6d6d6c3fc55cd33be2dcc502f642ef508d81c18a24de1f all runs: OK # git bisect good 9bb33f24abbd0fa2fadad01ec75438d7cc239189 Bisecting: 8 revisions left to test after this (roughly 3 steps) [c76f48eb5c084b1e15c931ae8cc1826cd771d70d] block: take bd_mutex around delete_partitions in del_gendisk testing commit c76f48eb5c084b1e15c931ae8cc1826cd771d70d with gcc (GCC) 10.2.1 20210217 kernel signature: a208d02b6687b580ba64369e54fa64a3accfe0a5fa9b9ea5a8b7a5d8c6b3fad2 all runs: crashed: possible deadlock in del_gendisk # git bisect bad c76f48eb5c084b1e15c931ae8cc1826cd771d70d Bisecting: 3 revisions left to test after this (roughly 2 steps) [b896fa85e0ee4f09ba4be48a3f405fc82c38afb4] dasd: use bdev_disk_changed instead of blk_drop_partitions testing commit b896fa85e0ee4f09ba4be48a3f405fc82c38afb4 with gcc (GCC) 10.2.1 20210217 kernel signature: 878b5b155cb64cffc3465d52dd5ef272b03d774b3511a7993201f89593792d5c all runs: OK # git bisect good b896fa85e0ee4f09ba4be48a3f405fc82c38afb4 Bisecting: 1 revision left to test after this (roughly 1 step) [473338be3aaea117a7133720305f240eb7f68951] block: move more syncing and invalidation to delete_partition testing commit 473338be3aaea117a7133720305f240eb7f68951 with gcc (GCC) 10.2.1 20210217 kernel signature: 903aed7ba114305bf6d7c3d8ec1954a096a4d6c852e53d96dc164d6da37b0c9b all runs: OK # git bisect good 473338be3aaea117a7133720305f240eb7f68951 Bisecting: 0 revisions left to test after this (roughly 0 steps) [d3c4a43d9291279c28b26757351a6ab72c110753] block: refactor blk_drop_partitions testing commit d3c4a43d9291279c28b26757351a6ab72c110753 with gcc (GCC) 10.2.1 20210217 kernel signature: 51009f8f1675f211509add1b5b7d963050cc41adbbcba5a919e70dee4c62be9d all runs: OK # git bisect good d3c4a43d9291279c28b26757351a6ab72c110753 c76f48eb5c084b1e15c931ae8cc1826cd771d70d is the first bad commit commit c76f48eb5c084b1e15c931ae8cc1826cd771d70d Author: Christoph Hellwig Date: Tue Apr 6 08:22:56 2021 +0200 block: take bd_mutex around delete_partitions in del_gendisk There is nothing preventing an ioctl from trying do delete partition concurrenly with del_gendisk, so take open_mutex to serialize against that. Signed-off-by: Christoph Hellwig Link: https://lore.kernel.org/r/20210406062303.811835-6-hch@lst.de Signed-off-by: Jens Axboe block/genhd.c | 4 ++++ block/partitions/core.c | 2 ++ 2 files changed, 6 insertions(+) culprit signature: a208d02b6687b580ba64369e54fa64a3accfe0a5fa9b9ea5a8b7a5d8c6b3fad2 parent signature: 51009f8f1675f211509add1b5b7d963050cc41adbbcba5a919e70dee4c62be9d revisions tested: 17, total time: 4h25m15.340713187s (build: 2h1m26.318393844s, test: 2h21m20.603448692s) first bad commit: c76f48eb5c084b1e15c931ae8cc1826cd771d70d block: take bd_mutex around delete_partitions in del_gendisk recipients (to): ["axboe@kernel.dk" "axboe@kernel.dk" "hch@lst.de" "linux-block@vger.kernel.org"] recipients (cc): ["hare@suse.de" "jack@suse.cz" "linux-kernel@vger.kernel.org"] crash: possible deadlock in del_gendisk ====================================================== WARNING: possible circular locking dependency detected 5.12.0-rc4-syzkaller #0 Not tainted ------------------------------------------------------ syz-executor.2/10107 is trying to acquire lock: ffff88801767dca0 (&bdev->bd_mutex){+.+.}-{3:3}, at: del_gendisk+0x225/0x960 block/genhd.c:684 but task is already holding lock: ffffffff8af82810 (bdev_lookup_sem){++++}-{3:3}, at: del_gendisk+0x1f7/0x960 block/genhd.c:682 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #2 (bdev_lookup_sem){++++}-{3:3}: down_write+0x92/0x150 kernel/locking/rwsem.c:1406 del_gendisk+0x1f7/0x960 block/genhd.c:682 nbd_dev_remove drivers/block/nbd.c:226 [inline] nbd_put.part.0+0xa4/0x1b0 drivers/block/nbd.c:250 nbd_put drivers/block/nbd.c:1928 [inline] nbd_genl_connect+0xeee/0x1290 drivers/block/nbd.c:1968 genl_family_rcv_msg_doit+0x1e4/0x2f0 net/netlink/genetlink.c:739 genl_family_rcv_msg net/netlink/genetlink.c:783 [inline] genl_rcv_msg+0x27a/0x4a0 net/netlink/genetlink.c:800 netlink_rcv_skb+0x118/0x370 net/netlink/af_netlink.c:2502 genl_rcv+0x1f/0x30 net/netlink/genetlink.c:811 netlink_unicast_kernel net/netlink/af_netlink.c:1312 [inline] netlink_unicast+0x42e/0x700 net/netlink/af_netlink.c:1338 netlink_sendmsg+0x70e/0xbe0 net/netlink/af_netlink.c:1927 sock_sendmsg_nosec net/socket.c:654 [inline] sock_sendmsg+0xab/0xe0 net/socket.c:674 ____sys_sendmsg+0x5bf/0x7a0 net/socket.c:2350 ___sys_sendmsg+0xd3/0x150 net/socket.c:2404 __sys_sendmsg+0xb2/0x140 net/socket.c:2433 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae -> #1 (nbd_index_mutex){+.+.}-{3:3}: __mutex_lock_common kernel/locking/mutex.c:949 [inline] __mutex_lock+0x139/0x1120 kernel/locking/mutex.c:1096 nbd_open+0x6f/0x630 drivers/block/nbd.c:1460 __blkdev_get+0x10a/0x8d0 fs/block_dev.c:1304 blkdev_get_by_dev fs/block_dev.c:1456 [inline] blkdev_get_by_dev+0x1fb/0x510 fs/block_dev.c:1424 blkdev_open+0xf6/0x220 fs/block_dev.c:1553 do_dentry_open+0x42a/0xfb0 fs/open.c:826 do_open fs/namei.c:3365 [inline] path_openat+0x9ee/0x22b0 fs/namei.c:3498 do_filp_open+0x16d/0x390 fs/namei.c:3525 do_sys_openat2+0x11e/0x360 fs/open.c:1187 do_sys_open fs/open.c:1203 [inline] __do_sys_open fs/open.c:1211 [inline] __se_sys_open fs/open.c:1207 [inline] __x64_sys_open+0xfd/0x1a0 fs/open.c:1207 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae -> #0 (&bdev->bd_mutex){+.+.}-{3:3}: check_prev_add kernel/locking/lockdep.c:2936 [inline] check_prevs_add kernel/locking/lockdep.c:3059 [inline] validate_chain kernel/locking/lockdep.c:3674 [inline] __lock_acquire+0x2b37/0x57d0 kernel/locking/lockdep.c:4900 lock_acquire kernel/locking/lockdep.c:5510 [inline] lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5475 __mutex_lock_common kernel/locking/mutex.c:949 [inline] __mutex_lock+0x139/0x1120 kernel/locking/mutex.c:1096 del_gendisk+0x225/0x960 block/genhd.c:684 nbd_dev_remove drivers/block/nbd.c:226 [inline] nbd_put.part.0+0xa4/0x1b0 drivers/block/nbd.c:250 nbd_put drivers/block/nbd.c:1928 [inline] nbd_genl_connect+0xeee/0x1290 drivers/block/nbd.c:1968 genl_family_rcv_msg_doit+0x1e4/0x2f0 net/netlink/genetlink.c:739 genl_family_rcv_msg net/netlink/genetlink.c:783 [inline] genl_rcv_msg+0x27a/0x4a0 net/netlink/genetlink.c:800 netlink_rcv_skb+0x118/0x370 net/netlink/af_netlink.c:2502 genl_rcv+0x1f/0x30 net/netlink/genetlink.c:811 netlink_unicast_kernel net/netlink/af_netlink.c:1312 [inline] netlink_unicast+0x42e/0x700 net/netlink/af_netlink.c:1338 netlink_sendmsg+0x70e/0xbe0 net/netlink/af_netlink.c:1927 sock_sendmsg_nosec net/socket.c:654 [inline] sock_sendmsg+0xab/0xe0 net/socket.c:674 ____sys_sendmsg+0x5bf/0x7a0 net/socket.c:2350 ___sys_sendmsg+0xd3/0x150 net/socket.c:2404 __sys_sendmsg+0xb2/0x140 net/socket.c:2433 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae other info that might help us debug this: Chain exists of: &bdev->bd_mutex --> nbd_index_mutex --> bdev_lookup_sem Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(bdev_lookup_sem); lock(nbd_index_mutex); lock(bdev_lookup_sem); lock(&bdev->bd_mutex); *** DEADLOCK *** 4 locks held by syz-executor.2/10107: #0: ffffffff8be306b0 (cb_lock){++++}-{3:3}, at: genl_rcv+0x10/0x30 net/netlink/genetlink.c:810 #1: ffffffff8be30768 (genl_mutex){+.+.}-{3:3}, at: genl_lock net/netlink/genetlink.c:33 [inline] #1: ffffffff8be30768 (genl_mutex){+.+.}-{3:3}, at: genl_rcv_msg+0x315/0x4a0 net/netlink/genetlink.c:798 #2: ffffffff8b1f7b68 (nbd_index_mutex){+.+.}-{3:3}, at: refcount_dec_and_mutex_lock lib/refcount.c:118 [inline] #2: ffffffff8b1f7b68 (nbd_index_mutex){+.+.}-{3:3}, at: refcount_dec_and_mutex_lock+0x2b/0xd0 lib/refcount.c:113 #3: ffffffff8af82810 (bdev_lookup_sem){++++}-{3:3}, at: del_gendisk+0x1f7/0x960 block/genhd.c:682 stack backtrace: CPU: 0 PID: 10107 Comm: syz-executor.2 Not tainted 5.12.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0xa5/0xe6 lib/dump_stack.c:120 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2127 check_prev_add kernel/locking/lockdep.c:2936 [inline] check_prevs_add kernel/locking/lockdep.c:3059 [inline] validate_chain kernel/locking/lockdep.c:3674 [inline] __lock_acquire+0x2b37/0x57d0 kernel/locking/lockdep.c:4900 lock_acquire kernel/locking/lockdep.c:5510 [inline] lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5475 __mutex_lock_common kernel/locking/mutex.c:949 [inline] __mutex_lock+0x139/0x1120 kernel/locking/mutex.c:1096 del_gendisk+0x225/0x960 block/genhd.c:684 nbd_dev_remove drivers/block/nbd.c:226 [inline] nbd_put.part.0+0xa4/0x1b0 drivers/block/nbd.c:250 nbd_put drivers/block/nbd.c:1928 [inline] nbd_genl_connect+0xeee/0x1290 drivers/block/nbd.c:1968 genl_family_rcv_msg_doit+0x1e4/0x2f0 net/netlink/genetlink.c:739 genl_family_rcv_msg net/netlink/genetlink.c:783 [inline] genl_rcv_msg+0x27a/0x4a0 net/netlink/genetlink.c:800 netlink_rcv_skb+0x118/0x370 net/netlink/af_netlink.c:2502 genl_rcv+0x1f/0x30 net/netlink/genetlink.c:811 netlink_unicast_kernel net/netlink/af_netlink.c:1312 [inline] netlink_unicast+0x42e/0x700 net/netlink/af_netlink.c:1338 netlink_sendmsg+0x70e/0xbe0 net/netlink/af_netlink.c:1927 sock_sendmsg_nosec net/socket.c:654 [inline] sock_sendmsg+0xab/0xe0 net/socket.c:674 ____sys_sendmsg+0x5bf/0x7a0 net/socket.c:2350 ___sys_sendmsg+0xd3/0x150 net/socket.c:2404 __sys_sendmsg+0xb2/0x140 net/socket.c:2433 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x4665d9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f8cc0dbb188 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 000000000056bf80 RCX: 00000000004665d9 RDX: 0000000000000800 RSI: 0000000020000340 RDI: 0000000000000005 RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf80 R13: 00007ffe4a5f58ff R14: 00007f8cc0dbb300 R15: 0000000000022000