block nbd1: Receive control failed (result -1) ====================================================== WARNING: possible circular locking dependency detected syzkaller #0 Tainted: G L ------------------------------------------------------ kworker/u35:5/18136 is trying to acquire lock: ffff888042e4aae0 (sk_lock-AF_INET6-NBD){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1713 [inline] ffff888042e4aae0 (sk_lock-AF_INET6-NBD){+.+.}-{0:0}, at: inet_shutdown+0x67/0x410 net/ipv4/af_inet.c:913 but task is already holding lock: ffff888054355868 (&nsock->tx_lock){+.+.}-{4:4}, at: recv_work+0x5e7/0x8c0 drivers/block/nbd.c:1020 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #6 (&nsock->tx_lock){+.+.}-{4:4}: lock_acquire kernel/locking/lockdep.c:5868 [inline] lock_acquire+0x1b9/0x370 kernel/locking/lockdep.c:5825 __mutex_lock_common kernel/locking/mutex.c:646 [inline] __mutex_lock+0x1a4/0x1bd0 kernel/locking/mutex.c:821 nbd_handle_cmd drivers/block/nbd.c:1143 [inline] nbd_queue_rq+0x428/0x1080 drivers/block/nbd.c:1207 blk_mq_dispatch_rq_list+0x422/0x1e70 block/blk-mq.c:2117 __blk_mq_do_dispatch_sched block/blk-mq-sched.c:168 [inline] blk_mq_do_dispatch_sched block/blk-mq-sched.c:182 [inline] __blk_mq_sched_dispatch_requests+0xcea/0x1620 block/blk-mq-sched.c:307 blk_mq_sched_dispatch_requests+0xd7/0x1c0 block/blk-mq-sched.c:329 blk_mq_run_work_fn+0x135/0x380 block/blk-mq.c:2531 process_one_work+0xa23/0x1940 kernel/workqueue.c:3322 process_scheduled_works kernel/workqueue.c:3405 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3486 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x72b/0xd50 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 -> #5 (&cmd->lock){+.+.}-{4:4}: lock_acquire kernel/locking/lockdep.c:5868 [inline] lock_acquire+0x1b9/0x370 kernel/locking/lockdep.c:5825 __mutex_lock_common kernel/locking/mutex.c:646 [inline] __mutex_lock+0x1a4/0x1bd0 kernel/locking/mutex.c:821 nbd_queue_rq+0xba/0x1080 drivers/block/nbd.c:1199 blk_mq_dispatch_rq_list+0x422/0x1e70 block/blk-mq.c:2117 __blk_mq_do_dispatch_sched block/blk-mq-sched.c:168 [inline] blk_mq_do_dispatch_sched block/blk-mq-sched.c:182 [inline] __blk_mq_sched_dispatch_requests+0xcea/0x1620 block/blk-mq-sched.c:307 blk_mq_sched_dispatch_requests+0xd7/0x1c0 block/blk-mq-sched.c:329 blk_mq_run_work_fn+0x135/0x380 block/blk-mq.c:2531 process_one_work+0xa23/0x1940 kernel/workqueue.c:3322 process_scheduled_works kernel/workqueue.c:3405 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3486 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x72b/0xd50 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 -> #4 (set->srcu){.+.+}-{0:0}: lock_sync kernel/locking/lockdep.c:5916 [inline] lock_sync+0x9d/0x110 kernel/locking/lockdep.c:5904 srcu_lock_sync include/linux/srcu.h:199 [inline] __synchronize_srcu+0xa2/0x300 kernel/rcu/srcutree.c:1481 blk_mq_wait_quiesce_done block/blk-mq.c:284 [inline] blk_mq_wait_quiesce_done block/blk-mq.c:281 [inline] blk_mq_quiesce_queue block/blk-mq.c:304 [inline] blk_mq_quiesce_queue+0x149/0x1c0 block/blk-mq.c:299 elevator_switch+0x17b/0x7e0 block/elevator.c:576 elevator_change+0x352/0x530 block/elevator.c:681 elevator_set_default+0x29e/0x360 block/elevator.c:754 blk_register_queue+0x48e/0x630 block/blk-sysfs.c:992 __add_disk+0x73f/0xe40 block/genhd.c:528 add_disk_fwnode+0x118/0x5c0 block/genhd.c:597 add_disk include/linux/blkdev.h:800 [inline] nbd_dev_add+0x77a/0xb10 drivers/block/nbd.c:2021 nbd_init+0x291/0x2b0 drivers/block/nbd.c:2729 do_one_initcall+0x11d/0x700 init/main.c:1347 do_initcall_level init/main.c:1409 [inline] do_initcalls init/main.c:1425 [inline] do_basic_setup init/main.c:1445 [inline] kernel_init_freeable+0x6ea/0x7b0 init/main.c:1658 kernel_init+0x1f/0x1e0 init/main.c:1548 ret_from_fork+0x72b/0xd50 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 -> #3 (&q->elevator_lock){+.+.}-{4:4}: lock_acquire kernel/locking/lockdep.c:5868 [inline] lock_acquire+0x1b9/0x370 kernel/locking/lockdep.c:5825 __mutex_lock_common kernel/locking/mutex.c:646 [inline] __mutex_lock+0x1a4/0x1bd0 kernel/locking/mutex.c:821 elevator_change+0x1bc/0x530 block/elevator.c:679 elevator_set_none+0x92/0xf0 block/elevator.c:769 blk_mq_elv_switch_none block/blk-mq.c:5101 [inline] __blk_mq_update_nr_hw_queues block/blk-mq.c:5146 [inline] blk_mq_update_nr_hw_queues+0x4c1/0x15f0 block/blk-mq.c:5211 nbd_start_device+0x1a6/0xbd0 drivers/block/nbd.c:1526 nbd_genl_connect+0xff2/0x1a40 drivers/block/nbd.c:2276 genl_family_rcv_msg_doit+0x214/0x300 net/netlink/genetlink.c:1114 genl_family_rcv_msg net/netlink/genetlink.c:1194 [inline] genl_rcv_msg+0x560/0x800 net/netlink/genetlink.c:1209 netlink_rcv_skb+0x159/0x420 net/netlink/af_netlink.c:2556 genl_rcv+0x28/0x40 net/netlink/genetlink.c:1218 netlink_unicast_kernel net/netlink/af_netlink.c:1319 [inline] netlink_unicast+0x585/0x850 net/netlink/af_netlink.c:1345 netlink_sendmsg+0x8b0/0xda0 net/netlink/af_netlink.c:1900 sock_sendmsg_nosec net/socket.c:775 [inline] __sock_sendmsg net/socket.c:790 [inline] ____sys_sendmsg+0xa4d/0xbe0 net/socket.c:2684 ___sys_sendmsg+0x190/0x1e0 net/socket.c:2738 __sys_sendmsg+0x160/0x210 net/socket.c:2770 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0x115/0x870 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f -> #2 (&q->q_usage_counter(io)#50){++++}-{0:0}: lock_acquire kernel/locking/lockdep.c:5868 [inline] lock_acquire+0x1b9/0x370 kernel/locking/lockdep.c:5825 blk_alloc_queue+0x610/0x790 block/blk-core.c:504 blk_mq_alloc_queue+0x174/0x290 block/blk-mq.c:4420 __blk_mq_alloc_disk+0x29/0x120 block/blk-mq.c:4467 nbd_dev_add+0x492/0xb10 drivers/block/nbd.c:1991 nbd_init+0x291/0x2b0 drivers/block/nbd.c:2729 do_one_initcall+0x11d/0x700 init/main.c:1347 do_initcall_level init/main.c:1409 [inline] do_initcalls init/main.c:1425 [inline] do_basic_setup init/main.c:1445 [inline] kernel_init_freeable+0x6ea/0x7b0 init/main.c:1658 kernel_init+0x1f/0x1e0 init/main.c:1548 ret_from_fork+0x72b/0xd50 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 -> #1 (fs_reclaim){+.+.}-{0:0}: lock_acquire kernel/locking/lockdep.c:5868 [inline] lock_acquire+0x1b9/0x370 kernel/locking/lockdep.c:5825 __fs_reclaim_acquire mm/page_alloc.c:4329 [inline] fs_reclaim_acquire+0xc4/0x100 mm/page_alloc.c:4343 might_alloc include/linux/sched/mm.h:317 [inline] slab_pre_alloc_hook mm/slub.c:4565 [inline] slab_alloc_node mm/slub.c:4925 [inline] __do_kmalloc_node mm/slub.c:5361 [inline] __kmalloc_noprof+0xce/0x820 mm/slub.c:5387 _kmalloc_noprof include/linux/slab.h:973 [inline] _kzalloc_noprof include/linux/slab.h:1290 [inline] tls_get_rec net/tls/tls_sw.c:366 [inline] tls_sw_sendmsg_locked net/tls/tls_sw.c:811 [inline] tls_sw_sendmsg+0x10f4/0x2b10 net/tls/tls_sw.c:1008 inet6_sendmsg+0x11c/0x140 net/ipv6/af_inet6.c:641 sock_sendmsg_nosec net/socket.c:775 [inline] __sock_sendmsg net/socket.c:790 [inline] sock_sendmsg+0x28c/0x410 net/socket.c:813 splice_to_socket+0xb3c/0x11a0 fs/splice.c:884 do_splice_from fs/splice.c:936 [inline] direct_splice_actor+0x192/0x690 fs/splice.c:1159 splice_direct_to_actor+0x345/0xa30 fs/splice.c:1103 do_splice_direct_actor fs/splice.c:1202 [inline] do_splice_direct+0x174/0x240 fs/splice.c:1228 do_sendfile+0xadc/0xe20 fs/read_write.c:1371 __do_sys_sendfile64 fs/read_write.c:1432 [inline] __se_sys_sendfile64 fs/read_write.c:1418 [inline] __x64_sys_sendfile64+0x1d8/0x220 fs/read_write.c:1418 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0x115/0x870 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f -> #0 (sk_lock-AF_INET6-NBD){+.+.}-{0:0}: check_prev_add+0xeb/0xe60 kernel/locking/lockdep.c:3165 check_prevs_add kernel/locking/lockdep.c:3284 [inline] validate_chain kernel/locking/lockdep.c:3908 [inline] __lock_acquire+0x136c/0x1a40 kernel/locking/lockdep.c:5237 lock_acquire kernel/locking/lockdep.c:5868 [inline] lock_acquire+0x1b9/0x370 kernel/locking/lockdep.c:5825 lock_sock_nested+0x41/0xf0 net/core/sock.c:3825 lock_sock include/net/sock.h:1713 [inline] inet_shutdown+0x67/0x410 net/ipv4/af_inet.c:913 nbd_mark_nsock_dead+0xae/0x5c0 drivers/block/nbd.c:318 recv_work+0x5fb/0x8c0 drivers/block/nbd.c:1021 process_one_work+0xa23/0x1940 kernel/workqueue.c:3322 process_scheduled_works kernel/workqueue.c:3405 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3486 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x72b/0xd50 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 other info that might help us debug this: Chain exists of: sk_lock-AF_INET6-NBD --> &cmd->lock --> &nsock->tx_lock Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&nsock->tx_lock); lock(&cmd->lock); lock(&nsock->tx_lock); lock(sk_lock-AF_INET6-NBD); *** DEADLOCK *** 3 locks held by kworker/u35:5/18136: #0: ffff888028547940 ((wq_completion)nbd1-recv){+.+.}-{0:0}, at: process_one_work+0x12b1/0x1940 kernel/workqueue.c:3297 #1: ffffc900077dfd08 ((work_completion)(&args->work)){+.+.}-{0:0}, at: process_one_work+0x988/0x1940 kernel/workqueue.c:3298 #2: ffff888054355868 (&nsock->tx_lock){+.+.}-{4:4}, at: recv_work+0x5e7/0x8c0 drivers/block/nbd.c:1020 stack backtrace: CPU: 3 UID: 0 PID: 18136 Comm: kworker/u35:5 Tainted: G L syzkaller #0 PREEMPT(full) Tainted: [L]=SOFTLOCKUP Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 Workqueue: nbd1-recv recv_work Call Trace: __dump_stack lib/dump_stack.c:94 [inline] dump_stack_lvl+0x100/0x190 lib/dump_stack.c:120 print_circular_bug.cold+0x178/0x1c7 kernel/locking/lockdep.c:2043 check_noncircular+0x146/0x160 kernel/locking/lockdep.c:2175 check_prev_add+0xeb/0xe60 kernel/locking/lockdep.c:3165 check_prevs_add kernel/locking/lockdep.c:3284 [inline] validate_chain kernel/locking/lockdep.c:3908 [inline] __lock_acquire+0x136c/0x1a40 kernel/locking/lockdep.c:5237 lock_acquire kernel/locking/lockdep.c:5868 [inline] lock_acquire+0x1b9/0x370 kernel/locking/lockdep.c:5825 lock_sock_nested+0x41/0xf0 net/core/sock.c:3825 lock_sock include/net/sock.h:1713 [inline] inet_shutdown+0x67/0x410 net/ipv4/af_inet.c:913 nbd_mark_nsock_dead+0xae/0x5c0 drivers/block/nbd.c:318 recv_work+0x5fb/0x8c0 drivers/block/nbd.c:1021 process_one_work+0xa23/0x1940 kernel/workqueue.c:3322 process_scheduled_works kernel/workqueue.c:3405 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3486 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x72b/0xd50 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245