======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
kworker/0:1H/264 is trying to acquire lock:
ffff88814753c938 ((wq_completion)nvmet-wq){+.+.}-{0:0}, at: flush_workqueue+0x14e/0x1390 kernel/workqueue.c:2830

but task is already holding lock:
ffffc9000253fd00 ((work_completion)(&queue->io_work)){+.+.}-{0:0}, at: process_one_work+0x79e/0xff0 kernel/workqueue.c:2285

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&queue->io_work)){+.+.}-{0:0}:
       __flush_work+0x116/0x210 kernel/workqueue.c:3090
       nvmet_tcp_release_queue_work+0x32f/0xdb0 drivers/nvme/target/tcp.c:1511
       process_one_work+0x867/0xff0 kernel/workqueue.c:2310
       worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
       kthread+0x42e/0x520 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #1 ((work_completion)(&queue->release_work)){+.+.}-{0:0}:
       process_one_work+0x7ba/0xff0 kernel/workqueue.c:2286
       worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
       kthread+0x42e/0x520 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #0 ((wq_completion)nvmet-wq){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain kernel/locking/lockdep.c:3788 [inline]
       __lock_acquire+0x2c66/0x7b50 kernel/locking/lockdep.c:5012
       lock_acquire+0x19e/0x400 kernel/locking/lockdep.c:5623
       flush_workqueue+0x16a/0x1390 kernel/workqueue.c:2830
       nvmet_tcp_install_queue+0x7e/0x380 drivers/nvme/target/tcp.c:1865
       nvmet_install_queue+0x335/0x760 drivers/nvme/target/fabrics-cmd.c:154
       nvmet_execute_admin_connect+0x332/0x750 drivers/nvme/target/fabrics-cmd.c:217
       nvmet_tcp_execute_request drivers/nvme/target/tcp.c:588 [inline]
       nvmet_tcp_try_recv_data drivers/nvme/target/tcp.c:1211 [inline]
       nvmet_tcp_try_recv_one drivers/nvme/target/tcp.c:1273 [inline]
       nvmet_tcp_try_recv drivers/nvme/target/tcp.c:1299 [inline]
       nvmet_tcp_io_work+0x19c4/0x8510 drivers/nvme/target/tcp.c:1349
       process_one_work+0x867/0xff0 kernel/workqueue.c:2310
       worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
       kthread+0x42e/0x520 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

other info that might help us debug this:

Chain exists of:
  (wq_completion)nvmet-wq --> (work_completion)(&queue->release_work) --> (work_completion)(&queue->io_work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&queue->io_work));
                               lock((work_completion)(&queue->release_work));
                               lock((work_completion)(&queue->io_work));
  lock((wq_completion)nvmet-wq);

 *** DEADLOCK ***

2 locks held by kworker/0:1H/264:
 #0: ffff88814753d538 ((wq_completion)nvmet_tcp_wq){+.+.}-{0:0}, at: process_one_work+0x75c/0xff0 kernel/workqueue.c:-1
 #1: ffffc9000253fd00 ((work_completion)(&queue->io_work)){+.+.}-{0:0}, at: process_one_work+0x79e/0xff0 kernel/workqueue.c:2285

stack backtrace:
CPU: 0 PID: 264 Comm: kworker/0:1H Not tainted syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/24/2026
Workqueue: nvmet_tcp_wq nvmet_tcp_io_work
Call Trace:
 <TASK>
 dump_stack_lvl+0x188/0x250 lib/dump_stack.c:106
 check_noncircular+0x296/0x330 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain kernel/locking/lockdep.c:3788 [inline]
 __lock_acquire+0x2c66/0x7b50 kernel/locking/lockdep.c:5012
 lock_acquire+0x19e/0x400 kernel/locking/lockdep.c:5623
 flush_workqueue+0x16a/0x1390 kernel/workqueue.c:2830
 nvmet_tcp_install_queue+0x7e/0x380 drivers/nvme/target/tcp.c:1865
 nvmet_install_queue+0x335/0x760 drivers/nvme/target/fabrics-cmd.c:154
 nvmet_execute_admin_connect+0x332/0x750 drivers/nvme/target/fabrics-cmd.c:217
 nvmet_tcp_execute_request drivers/nvme/target/tcp.c:588 [inline]
 nvmet_tcp_try_recv_data drivers/nvme/target/tcp.c:1211 [inline]
 nvmet_tcp_try_recv_one drivers/nvme/target/tcp.c:1273 [inline]
 nvmet_tcp_try_recv drivers/nvme/target/tcp.c:1299 [inline]
 nvmet_tcp_io_work+0x19c4/0x8510 drivers/nvme/target/tcp.c:1349
 process_one_work+0x867/0xff0 kernel/workqueue.c:2310
 worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
 kthread+0x42e/0x520 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>
nvmet: creating controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:047137bc-7dc0-410b-ba75-2d0d9998a3b2.