block nbd5: Receive control failed (result -22) block nbd5: shutting down sockets kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env ============================================ WARNING: possible recursive locking detected 4.19.79 #0 Not tainted -------------------------------------------- kworker/u5:1/9149 is trying to acquire lock: kobject: 'loop3' (000000009af191bc): fill_kobj_path: path = '/devices/virtual/block/loop3' 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: flush_workqueue+0xf7/0x14b0 kernel/workqueue.c:2652 but task is already holding lock: kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path = '/devices/virtual/block/loop4' 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: __write_once_size include/linux/compiler.h:220 [inline] 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline] 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:59 [inline] 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: set_work_data kernel/workqueue.c:617 [inline] 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: process_one_work+0x87e/0x1750 kernel/workqueue.c:2124 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock((wq_completion)"knbd%d-recv"nbd->index); lock((wq_completion)"knbd%d-recv"nbd->index); *** DEADLOCK *** May be due to missing lock nesting notation 3 locks held by kworker/u5:1/9149: #0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: __write_once_size include/linux/compiler.h:220 [inline] #0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline] #0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:59 [inline] #0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: set_work_data kernel/workqueue.c:617 [inline] #0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at: process_one_work+0x87e/0x1750 kernel/workqueue.c:2124 EXT4-fs (loop4): VFS: Can't find ext4 filesystem #1: 000000007f0b712b ((work_completion)(&args->work)){+.+.}, at: process_one_work+0x8b4/0x1750 kernel/workqueue.c:2128 #2: 0000000025c76702 (&nbd->config_lock){+.+.}, at: refcount_dec_and_mutex_lock lib/refcount.c:311 [inline] #2: 0000000025c76702 (&nbd->config_lock){+.+.}, at: refcount_dec_and_mutex_lock+0x56/0x90 lib/refcount.c:306 stack backtrace: CPU: 1 PID: 9149 Comm: kworker/u5:1 Not tainted 4.19.79 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: knbd5-recv recv_work Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x172/0x1f0 lib/dump_stack.c:113 print_deadlock_bug kernel/locking/lockdep.c:1759 [inline] check_deadlock kernel/locking/lockdep.c:1803 [inline] validate_chain kernel/locking/lockdep.c:2399 [inline] __lock_acquire.cold+0x20f/0x4a7 kernel/locking/lockdep.c:3411 lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:3903 flush_workqueue+0x126/0x14b0 kernel/workqueue.c:2655 drain_workqueue+0x1b4/0x470 kernel/workqueue.c:2820 destroy_workqueue+0x21/0x6b0 kernel/workqueue.c:4158 nbd_config_put+0x3cf/0x860 drivers/block/nbd.c:1132 recv_work+0x19b/0x200 drivers/block/nbd.c:740 process_one_work+0x989/0x1750 kernel/workqueue.c:2153 worker_thread+0x98/0xe40 kernel/workqueue.c:2296 kthread+0x354/0x420 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415 kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop0' (00000000bdb0fb12): kobject_uevent_env kobject: 'loop0' (00000000bdb0fb12): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop2' (000000000c77fc03): kobject_uevent_env kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env kobject: 'loop2' (000000000c77fc03): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path = '/devices/virtual/block/loop4'