nvme nvme0: I/O 290 QID 2 timeout, disable controller
======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
kworker/1:1H/52 is trying to acquire lock:
ffff0000ce2e4340 (&dev->shutdown_lock){+.+.}-{3:3}, at: nvme_dev_disable+0x5c/0x5ec drivers/nvme/host/pci.c:2735

but task is already holding lock:
ffff80001e0d7c40 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}, at: process_one_work+0x6e8/0x13bc kernel/workqueue.c:2267

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}:
       __flush_work+0x10c/0x1ec kernel/workqueue.c:3072
       __cancel_work_timer+0x2f8/0x450 kernel/workqueue.c:3163
       cancel_work_sync+0x24/0x38 kernel/workqueue.c:3199
       blk_sync_queue+0x28/0x38 block/blk-core.c:232
       nvme_sync_io_queues drivers/nvme/host/core.c:5467 [inline]
       nvme_sync_queues+0x70/0xf4 drivers/nvme/host/core.c:5474
       nvme_reset_work+0x13c/0x477c drivers/nvme/host/pci.c:2890
       process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
       worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
       kthread+0x254/0x2e0 kernel/kthread.c:376
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

-> #1 (&ctrl->namespaces_rwsem){++++}-{3:3}:
       down_read+0x64/0x300 kernel/locking/rwsem.c:1520
       nvme_start_freeze+0x2c/0xc0 drivers/nvme/host/core.c:5416
       nvme_dev_disable+0x1b8/0x5ec drivers/nvme/host/pci.c:2747
       nvme_reset_work+0x128/0x477c drivers/nvme/host/pci.c:2889
       process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
       worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
       kthread+0x254/0x2e0 kernel/kthread.c:376
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

-> #0 (&dev->shutdown_lock){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3095 [inline]
       check_prevs_add kernel/locking/lockdep.c:3214 [inline]
       validate_chain kernel/locking/lockdep.c:3830 [inline]
       __lock_acquire+0x2808/0x6634 kernel/locking/lockdep.c:5063
       lock_acquire+0x20c/0x638 kernel/locking/lockdep.c:5678
       __mutex_lock_common+0x1a0/0x1f3c kernel/locking/mutex.c:603
       __mutex_lock kernel/locking/mutex.c:747 [inline]
       mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
       nvme_dev_disable+0x5c/0x5ec drivers/nvme/host/pci.c:2735
       nvme_timeout+0x400/0xa88 drivers/nvme/host/pci.c:1424
       blk_mq_rq_timed_out block/blk-mq.c:1566 [inline]
       blk_mq_handle_expired+0x148/0x270 block/blk-mq.c:1634
       bt_iter+0x2e4/0x3bc block/blk-mq-tag.c:295
       __sbitmap_for_each_set include/linux/sbitmap.h:286 [inline]
       sbitmap_for_each_set+0x2b8/0x558 include/linux/sbitmap.h:307
       bt_for_each block/blk-mq-tag.c:327 [inline]
       blk_mq_queue_tag_busy_iter+0x440/0x730 block/blk-mq-tag.c:536
       blk_mq_timeout_work+0x16c/0x300 block/blk-mq.c:1676
       process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
       worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
       kthread+0x254/0x2e0 kernel/kthread.c:376
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

other info that might help us debug this:

Chain exists of:
  &dev->shutdown_lock --> &ctrl->namespaces_rwsem --> (work_completion)(&q->timeout_work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&q->timeout_work));
                               lock(&ctrl->namespaces_rwsem);
                               lock((work_completion)(&q->timeout_work));
  lock(&dev->shutdown_lock);

 *** DEADLOCK ***

2 locks held by kworker/1:1H/52:
 #0: ffff0000c43f8938 ((wq_completion)kblockd){+.+.}-{0:0}, at: process_one_work+0x69c/0x13bc kernel/workqueue.c:-1
 #1: ffff80001e0d7c40 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}, at: process_one_work+0x6e8/0x13bc kernel/workqueue.c:2267

stack backtrace:
CPU: 1 PID: 52 Comm: kworker/1:1H Not tainted syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/14/2026
Workqueue: kblockd blk_mq_timeout_work
Call trace:
 dump_backtrace+0x1c4/0x1f0 arch/arm64/kernel/stacktrace.c:158
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
 __dump_stack+0x30/0x40 lib/dump_stack.c:88
 dump_stack_lvl+0xf4/0x15c lib/dump_stack.c:106
 dump_stack+0x1c/0x5c lib/dump_stack.c:113
 print_circular_bug+0x148/0x1b0 kernel/locking/lockdep.c:2048
 check_noncircular+0x264/0x2f8 kernel/locking/lockdep.c:2170
 check_prev_add kernel/locking/lockdep.c:3095 [inline]
 check_prevs_add kernel/locking/lockdep.c:3214 [inline]
 validate_chain kernel/locking/lockdep.c:3830 [inline]
 __lock_acquire+0x2808/0x6634 kernel/locking/lockdep.c:5063
 lock_acquire+0x20c/0x638 kernel/locking/lockdep.c:5678
 __mutex_lock_common+0x1a0/0x1f3c kernel/locking/mutex.c:603
 __mutex_lock kernel/locking/mutex.c:747 [inline]
 mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
 nvme_dev_disable+0x5c/0x5ec drivers/nvme/host/pci.c:2735
 nvme_timeout+0x400/0xa88 drivers/nvme/host/pci.c:1424
 blk_mq_rq_timed_out block/blk-mq.c:1566 [inline]
 blk_mq_handle_expired+0x148/0x270 block/blk-mq.c:1634
 bt_iter+0x2e4/0x3bc block/blk-mq-tag.c:295
 __sbitmap_for_each_set include/linux/sbitmap.h:286 [inline]
 sbitmap_for_each_set+0x2b8/0x558 include/linux/sbitmap.h:307
 bt_for_each block/blk-mq-tag.c:327 [inline]
 blk_mq_queue_tag_busy_iter+0x440/0x730 block/blk-mq-tag.c:536
 blk_mq_timeout_work+0x16c/0x300 block/blk-mq.c:1676
 process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
 worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
 kthread+0x254/0x2e0 kernel/kthread.c:376
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850
I/O error, dev nvme0n1, sector 1375808 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
EXT4-fs warning (device nvme0n1p2): ext4_end_bio:347: I/O error 10 writing to inode 1868 starting block 171976)
EXT4-fs warning (device nvme0n1p2): ext4_end_bio:347: I/O error 10 writing to inode 1868 starting block 174024)