block nbd4: Receive control failed (result -22) block nbd4: Receive control failed (result -22) block nbd0: Receive control failed (result -22) block nbd0: Receive control failed (result -22) ====================================================== WARNING: possible circular locking dependency detected block nbd5: Device being setup by another task 4.14.151 #0 Not tainted ------------------------------------------------------ kworker/u5:4/8679 is trying to acquire lock: (&nbd->config_lock){+.+.}, at: [] refcount_dec_and_mutex_lock lib/refcount.c:312 [inline] (&nbd->config_lock){+.+.}, at: [] refcount_dec_and_mutex_lock+0x49/0x6c lib/refcount.c:307 but task is already holding lock: block nbd5: Device being setup by another task ((&args->work)){+.+.}, at: [] process_one_work+0x7ab/0x1600 kernel/workqueue.c:2089 which lock already depends on the new lock. block nbd5: Device being setup by another task the existing dependency chain (in reverse order) is: -> #2 ((&args->work)){+.+.}: lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994 process_one_work+0x803/0x1600 kernel/workqueue.c:2090 block nbd5: Device being setup by another task worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248 kthread+0x319/0x430 kernel/kthread.c:232 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404 -> #1 block nbd4: shutting down sockets ("knbd%d-recv"nbd->index){+.+.}: lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994 flush_workqueue+0x109/0x1400 kernel/workqueue.c:2616 drain_workqueue+0x177/0x3e0 kernel/workqueue.c:2781 destroy_workqueue+0x21/0x620 kernel/workqueue.c:4088 nbd_config_put+0x43c/0x7a0 drivers/block/nbd.c:1124 kobject: 'loop4' (ffff8880a4a9a0e0): kobject_uevent_env nbd_release+0xdb/0x120 drivers/block/nbd.c:1393 kobject: 'loop4' (ffff8880a4a9a0e0): fill_kobj_path: path = '/devices/virtual/block/loop4' __blkdev_put+0x434/0x7f0 fs/block_dev.c:1791 blkdev_put+0x88/0x510 fs/block_dev.c:1856 block nbd4: Receive control failed (result -22) blkdev_close+0x8b/0xb0 fs/block_dev.c:1863 block nbd4: Receive control failed (result -22) __fput+0x275/0x7a0 fs/file_table.c:210 ____fput+0x16/0x20 fs/file_table.c:244 kobject: 'nbd4' (ffff8880a4137560): kobject_uevent_env task_work_run+0x114/0x190 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:191 [inline] exit_to_usermode_loop+0x1da/0x220 arch/x86/entry/common.c:164 prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline] syscall_return_slowpath arch/x86/entry/common.c:270 [inline] do_syscall_64+0x4bc/0x640 arch/x86/entry/common.c:297 kobject: 'nbd4' (ffff8880a4137560): fill_kobj_path: path = '/devices/virtual/block/nbd4' entry_SYSCALL_64_after_hwframe+0x42/0xb7 -> #0 (&nbd->config_lock){+.+.}: check_prev_add kernel/locking/lockdep.c:1901 [inline] check_prevs_add kernel/locking/lockdep.c:2018 [inline] validate_chain kernel/locking/lockdep.c:2460 [inline] __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487 lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893 block nbd1: Receive control failed (result -22) mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 refcount_dec_and_mutex_lock lib/refcount.c:312 [inline] refcount_dec_and_mutex_lock+0x49/0x6c lib/refcount.c:307 nbd_config_put+0x31/0x7a0 drivers/block/nbd.c:1102 recv_work+0x18d/0x1f0 drivers/block/nbd.c:724 process_one_work+0x863/0x1600 kernel/workqueue.c:2114 block nbd1: Receive control failed (result -22) worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248 kthread+0x319/0x430 kernel/kthread.c:232 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404 other info that might help us debug this: kobject: 'nbd1' (ffff8880a4002b60): kobject_uevent_env Chain exists of: &nbd->config_lock --> "knbd%d-recv"nbd->index --> (&args->work) Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock((&args->work)); lock( kobject: 'nbd1' (ffff8880a4002b60): fill_kobj_path: path = '/devices/virtual/block/nbd1' "knbd%d-recv"nbd->index); lock((&args->work)); lock(&nbd->config_lock); *** DEADLOCK *** 2 locks held by kworker/u5:4/8679: #0: ("knbd%d-recv"nbd->index){+.+.}, at: [] work_static include/linux/workqueue.h:199 [inline] #0: ("knbd%d-recv"nbd->index){+.+.}, at: [] set_work_data kernel/workqueue.c:619 [inline] #0: ("knbd%d-recv"nbd->index){+.+.}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:646 [inline] #0: ("knbd%d-recv"nbd->index){+.+.}, at: [] process_one_work+0x76e/0x1600 kernel/workqueue.c:2085 #1: ((&args->work)){+.+.}, at: [] process_one_work+0x7ab/0x1600 kernel/workqueue.c:2089 stack backtrace: CPU: 0 PID: 8679 Comm: kworker/u5:4 Not tainted 4.14.151 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: knbd5-recv recv_work Call Trace: __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x138/0x197 lib/dump_stack.c:53 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1258 check_prev_add kernel/locking/lockdep.c:1901 [inline] check_prevs_add kernel/locking/lockdep.c:2018 [inline] validate_chain kernel/locking/lockdep.c:2460 [inline] __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487 lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 refcount_dec_and_mutex_lock lib/refcount.c:312 [inline] refcount_dec_and_mutex_lock+0x49/0x6c lib/refcount.c:307 nbd_config_put+0x31/0x7a0 drivers/block/nbd.c:1102 recv_work+0x18d/0x1f0 drivers/block/nbd.c:724 process_one_work+0x863/0x1600 kernel/workqueue.c:2114 worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248 kthread+0x319/0x430 kernel/kthread.c:232 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404 kobject: 'loop5' (ffff8880a4aee9e0): kobject_uevent_env kobject: 'loop5' (ffff8880a4aee9e0): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: '1' (ffff8880a9d8a728): kobject_cleanup, parent (null) block nbd0: shutting down sockets block nbd4: shutting down sockets block nbd5: shutting down sockets block nbd1: shutting down sockets kobject: '1' (ffff8880a9d8a728): calling ktype release kobject: '1': free name kobject: '0' (ffff8880a4107068): kobject_add_internal: parent: 'mq', set: '' kobject: '0' (ffff8880a408afe8): kobject_add_internal: parent: 'mq', set: '' kobject: 'cpu0' (ffffe8ffffc31298): kobject_add_internal: parent: '0', set: '' kobject: 'cpu0' (ffffe8ffffc30b98): kobject_add_internal: parent: '0', set: '' kobject: '1' (ffff8880a53aa868): kobject_add_internal: parent: 'mq', set: '' kobject: '0' (ffff8880a408afe8): kobject_add_internal: parent: 'mq', set: '' kobject: 'cpu1' (ffffe8ffffd31298): kobject_add_internal: parent: '1', set: '' kobject: 'cpu0' (ffffe8ffffc30b98): kobject_add_internal: parent: '0', set: '' kobject: 'nbd3' (ffff8880a40ef4e0): kobject_uevent_env block nbd3: Receive control failed (result -22) block nbd3: Receive control failed (result -22) kobject: 'nbd3' (ffff8880a40ef4e0): fill_kobj_path: path = '/devices/virtual/block/nbd3' kobject: 'cpu1' (ffffe8ffffd30b98): kobject_add_internal: parent: '0', set: '' block nbd2: Receive control failed (result -22) kobject: 'nbd2' (ffff8880a405d460): kobject_uevent_env kobject: 'nbd2' (ffff8880a405d460): fill_kobj_path: path = '/devices/virtual/block/nbd2' block nbd2: shutting down sockets kobject: 'loop2' (ffff8880a4a177a0): kobject_uevent_env kobject: 'loop2' (ffff8880a4a177a0): fill_kobj_path: path = '/devices/virtual/block/loop2'