======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:6/4466 is trying to acquire lock:
ffff0000cf8189c8 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}, at: __flush_work+0xe4/0x1ec kernel/workqueue.c:3069

but task is already holding lock:
ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_io_queues drivers/nvme/host/core.c:5465 [inline]
ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_queues+0x2c/0xf4 drivers/nvme/host/core.c:5474

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&ctrl->namespaces_rwsem){++++}-{3:3}:
       down_read+0x64/0x300 kernel/locking/rwsem.c:1520
       nvme_start_freeze+0x2c/0xc0 drivers/nvme/host/core.c:5416
       nvme_dev_disable+0x1b8/0x5ec drivers/nvme/host/pci.c:2747
       nvme_timeout+0x628/0xa88 drivers/nvme/host/pci.c:-1
       blk_mq_rq_timed_out block/blk-mq.c:1566 [inline]
       blk_mq_handle_expired+0x148/0x270 block/blk-mq.c:1634
       bt_iter+0x2e4/0x3bc block/blk-mq-tag.c:295
       __sbitmap_for_each_set include/linux/sbitmap.h:286 [inline]
       sbitmap_for_each_set+0x2b8/0x558 include/linux/sbitmap.h:307
       bt_for_each block/blk-mq-tag.c:327 [inline]
       blk_mq_queue_tag_busy_iter+0x440/0x730 block/blk-mq-tag.c:536
       blk_mq_timeout_work+0x16c/0x300 block/blk-mq.c:1676
       process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
       worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
       kthread+0x254/0x2e0 kernel/kthread.c:376
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

-> #1 (&dev->shutdown_lock){+.+.}-{3:3}:
       __mutex_lock_common+0x1a0/0x1f3c kernel/locking/mutex.c:603
       __mutex_lock kernel/locking/mutex.c:747 [inline]
       mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
       nvme_dev_disable+0x5c/0x5ec drivers/nvme/host/pci.c:2735
       nvme_timeout+0x628/0xa88 drivers/nvme/host/pci.c:-1
       blk_mq_rq_timed_out block/blk-mq.c:1566 [inline]
       blk_mq_handle_expired+0x148/0x270 block/blk-mq.c:1634
       bt_iter+0x2e4/0x3bc block/blk-mq-tag.c:295
       __sbitmap_for_each_set include/linux/sbitmap.h:286 [inline]
       sbitmap_for_each_set+0x2b8/0x558 include/linux/sbitmap.h:307
       bt_for_each block/blk-mq-tag.c:327 [inline]
       blk_mq_queue_tag_busy_iter+0x440/0x730 block/blk-mq-tag.c:536
       blk_mq_timeout_work+0x16c/0x300 block/blk-mq.c:1676
       process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
       worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
       kthread+0x254/0x2e0 kernel/kthread.c:376
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

-> #0 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3095 [inline]
       check_prevs_add kernel/locking/lockdep.c:3214 [inline]
       validate_chain kernel/locking/lockdep.c:3830 [inline]
       __lock_acquire+0x2808/0x6634 kernel/locking/lockdep.c:5063
       lock_acquire+0x20c/0x638 kernel/locking/lockdep.c:5676
       __flush_work+0x10c/0x1ec kernel/workqueue.c:3072
       __cancel_work_timer+0x2f8/0x450 kernel/workqueue.c:3163
       cancel_work_sync+0x24/0x38 kernel/workqueue.c:3199
       blk_sync_queue+0x28/0x38 block/blk-core.c:232
       nvme_sync_io_queues drivers/nvme/host/core.c:5467 [inline]
       nvme_sync_queues+0x70/0xf4 drivers/nvme/host/core.c:5474
       nvme_reset_work+0x13c/0x477c drivers/nvme/host/pci.c:2890
       process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
       worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
       kthread+0x254/0x2e0 kernel/kthread.c:376
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

other info that might help us debug this:

Chain exists of:
  (work_completion)(&q->timeout_work) --> &dev->shutdown_lock --> &ctrl->namespaces_rwsem

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&ctrl->namespaces_rwsem);
                               lock(&dev->shutdown_lock);
                               lock(&ctrl->namespaces_rwsem);
  lock((work_completion)(&q->timeout_work));

 *** DEADLOCK ***

3 locks held by kworker/u4:6/4466:
 #0: ffff0000cfa0c938 ((wq_completion)nvme-reset-wq){+.+.}-{0:0}, at: process_one_work+0x69c/0x13bc kernel/workqueue.c:-1
 #1: ffff800020fe7c40 ((work_completion)(&dev->ctrl.reset_work)){+.+.}-{0:0}, at: process_one_work+0x6e8/0x13bc kernel/workqueue.c:2267
 #2: ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_io_queues drivers/nvme/host/core.c:5465 [inline]
 #2: ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_queues+0x2c/0xf4 drivers/nvme/host/core.c:5474

stack backtrace:
CPU: 1 PID: 4466 Comm: kworker/u4:6 Not tainted syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/07/2026
Workqueue: nvme-reset-wq nvme_reset_work
Call trace:
 dump_backtrace+0x1c4/0x1f0 arch/arm64/kernel/stacktrace.c:158
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
 __dump_stack+0x30/0x40 lib/dump_stack.c:88
 dump_stack_lvl+0xf4/0x15c lib/dump_stack.c:106
 dump_stack+0x1c/0x5c lib/dump_stack.c:113
 print_circular_bug+0x148/0x1b0 kernel/locking/lockdep.c:2048
 check_noncircular+0x264/0x2f8 kernel/locking/lockdep.c:2170
 check_prev_add kernel/locking/lockdep.c:3095 [inline]
 check_prevs_add kernel/locking/lockdep.c:3214 [inline]
 validate_chain kernel/locking/lockdep.c:3830 [inline]
 __lock_acquire+0x2808/0x6634 kernel/locking/lockdep.c:5063
 lock_acquire+0x20c/0x638 kernel/locking/lockdep.c:5676
 __flush_work+0x10c/0x1ec kernel/workqueue.c:3072
 __cancel_work_timer+0x2f8/0x450 kernel/workqueue.c:3163
 cancel_work_sync+0x24/0x38 kernel/workqueue.c:3199
 blk_sync_queue+0x28/0x38 block/blk-core.c:232
 nvme_sync_io_queues drivers/nvme/host/core.c:5467 [inline]
 nvme_sync_queues+0x70/0xf4 drivers/nvme/host/core.c:5474
 nvme_reset_work+0x13c/0x477c drivers/nvme/host/pci.c:2890
 process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
 worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
 kthread+0x254/0x2e0 kernel/kthread.c:376
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850
nvme nvme0: 2/0/0 default/read/poll queues