syzbot
possible deadlock in blk_revalidate_disk_zones
Status:
upstream: reported C repro on 2026/09/28 21:54
Subsystems:
block
Labels:
prio:high
[Documentation on labels]
Reported-by: syzbot+b0910be96b7c31314822@syzkaller.appspotmail.com
Fix commit:
nvme-multipath: revalidate head zones after unfreezing the head queue
Patched on: [ci-upstream-linux-next-kasan-gce-root ci-upstream-rust-kasan-gce], missing on: [ci-qemu-gce-upstream-auto ci-qemu-native-arm64-kvm ci-qemu-upstream ci-qemu-upstream-386 ci-qemu2-arm32 ci-qemu2-arm64 ci-qemu2-arm64-compat ci-qemu2-arm64-mte ci-qemu2-riscv64 ci-snapshot-upstream-root ci-upstream-bpf-kasan-gce ci-upstream-bpf-next-kasan-gce ci-upstream-gce-arm64 ci-upstream-gce-leak ci-upstream-kasan-badwrites-root ci-upstream-kasan-gce ci-upstream-kasan-gce-386 ci-upstream-kasan-gce-root ci-upstream-kasan-gce-selinux-root ci-upstream-kasan-gce-smack-root ci-upstream-kmsan-gce-386-root ci-upstream-kmsan-gce-root ci-upstream-net-kasan-gce ci-upstream-net-this-kasan-gce ci2-upstream-fs ci2-upstream-kcsan-gce ci2-upstream-usb]
First crash: 13d, last: 11d
Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
kworker/u8:10/3352 is trying to acquire lock:
ffff88801fae6450 (&disk->zone_revalidate_mutex){+.+.}-{4:4}, at: blk_revalidate_disk_zones+0x1c5/0x1650 block/blk-zoned.c:2560
but task is already holding lock:
ffff88802ac3a5a0 (&q->q_usage_counter(io)#75){++++}-{0:0}, at: blk_mq_freeze_queue include/linux/blk-mq.h:956 [inline]
ffff88802ac3a5a0 (&q->q_usage_counter(io)#75){++++}-{0:0}, at: nvme_update_ns_info+0x3ac/0x1200 drivers/nvme/host/core.c:2583
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #3 (&q->q_usage_counter(io)#75){++++}-{0:0}:
blk_alloc_queue+0x544/0x690 block/blk-core.c:504
__blk_alloc_disk+0xe3/0x1d0 block/genhd.c:1527
nvme_mpath_alloc_disk+0x5c2/0x8b0 drivers/nvme/host/multipath.c:769
nvme_alloc_ns_head drivers/nvme/host/core.c:4049 [inline]
nvme_init_ns_head drivers/nvme/host/core.c:4153 [inline]
nvme_alloc_ns drivers/nvme/host/core.c:4267 [inline]
nvme_scan_ns+0x2e22/0x46c0 drivers/nvme/host/core.c:4483
async_run_entry_fn+0x9d/0x430 kernel/async.c:129
process_one_work kernel/workqueue.c:3495 [inline]
process_scheduled_works+0xc3d/0x1630 kernel/workqueue.c:3578
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3659
kthread+0x38b/0x480 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
-> #2 (fs_reclaim){+.+.}-{0:0}:
__fs_reclaim_acquire mm/page_alloc.c:4365 [inline]
fs_reclaim_acquire+0x71/0x100 mm/page_alloc.c:4379
might_alloc include/linux/sched/mm.h:316 [inline]
slab_pre_alloc_hook mm/slub.c:4673 [inline]
slab_alloc_node mm/slub.c:5011 [inline]
__kmalloc_cache_node_noprof+0x66/0x620 mm/slub.c:5695
_kmalloc_node_noprof include/linux/slab.h:1196 [inline]
alloc_worker kernel/workqueue.c:2853 [inline]
init_rescuer+0x167/0x530 kernel/workqueue.c:6002
workqueue_init+0x214/0x680 kernel/workqueue.c:8517
kernel_init_freeable+0x26b/0x3e0 init/main.c:1658
kernel_init+0x22/0x1d0 init/main.c:1563
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
-> #1 (wq_pool_mutex){+.+.}-{4:4}:
__mutex_lock_common kernel/locking/mutex.c:646 [inline]
__mutex_lock+0x197/0x15a0 kernel/locking/mutex.c:821
__alloc_workqueue+0xe8d/0x21d0 kernel/workqueue.c:6201
alloc_workqueue_va kernel/workqueue.c:6253 [inline]
alloc_workqueue_noprof+0xe3/0x210 kernel/workqueue.c:6269
disk_alloc_zone_resources block/blk-zoned.c:2101 [inline]
blk_revalidate_disk_zones+0x8d1/0x1650 block/blk-zoned.c:2575
null_add_dev+0x12b8/0x1d70 drivers/block/null_blk/main.c:2056
nullb_device_power_store+0x240/0x380 drivers/block/null_blk/main.c:511
flush_write_buffer fs/configfs/file.c:207 [inline]
configfs_write_iter+0x337/0x430 fs/configfs/file.c:229
new_sync_write fs/read_write.c:595 [inline]
vfs_write+0x612/0xba0 fs/read_write.c:687
ksys_write+0x150/0x270 fs/read_write.c:739
do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84
entry_SYSCALL_64_after_hwframe+0x77/0x7f
-> #0 (&disk->zone_revalidate_mutex){+.+.}-{4:4}:
check_prev_add kernel/locking/lockdep.c:3209 [inline]
check_prevs_add kernel/locking/lockdep.c:3328 [inline]
validate_chain kernel/locking/lockdep.c:3952 [inline]
__lock_acquire+0x164c/0x2de0 kernel/locking/lockdep.c:5288
lock_acquire+0x115/0x350 kernel/locking/lockdep.c:5942
__mutex_lock_common kernel/locking/mutex.c:646 [inline]
__mutex_lock+0x197/0x15a0 kernel/locking/mutex.c:821
blk_revalidate_disk_zones+0x1c5/0x1650 block/blk-zoned.c:2560
nvme_mpath_revalidate_zones+0x106/0x1c0 drivers/nvme/host/multipath.c:301
nvme_update_ns_info+0x984/0x1200 drivers/nvme/host/core.c:2620
nvme_alloc_ns drivers/nvme/host/core.c:4293 [inline]
nvme_scan_ns+0x34c1/0x46c0 drivers/nvme/host/core.c:4483
async_run_entry_fn+0x9d/0x430 kernel/async.c:129
process_one_work kernel/workqueue.c:3495 [inline]
process_scheduled_works+0xc3d/0x1630 kernel/workqueue.c:3578
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3659
kthread+0x38b/0x480 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
other info that might help us debug this:
Chain exists of:
&disk->zone_revalidate_mutex --> fs_reclaim --> &q->q_usage_counter(io)#75
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&q->q_usage_counter(io)#75);
lock(fs_reclaim);
lock(&q->q_usage_counter(io)#75);
lock(&disk->zone_revalidate_mutex);
*** DEADLOCK ***
locks held by kworker/u8:10/3352: 4, last CPU#0:
#0: ffff88801c29b948 ((wq_completion)async){+.+.}-{0:0}, at: rcu_lock_acquire include/linux/rcupdate.h:309 [inline]
#0: ffff88801c29b948 ((wq_completion)async){+.+.}-{0:0}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
#0: ffff88801c29b948 ((wq_completion)async){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3460 [inline]
#0: ffff88801c29b948 ((wq_completion)async){+.+.}-{0:0}, at: process_scheduled_works+0x97a/0x1630 kernel/workqueue.c:3578
#1: ffffc9000f277c40 ((work_completion)(&entry->work)){+.+.}-{0:0}, at: rcu_lock_acquire include/linux/rcupdate.h:309 [inline]
#1: ffffc9000f277c40 ((work_completion)(&entry->work)){+.+.}-{0:0}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
#1: ffffc9000f277c40 ((work_completion)(&entry->work)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3460 [inline]
#1: ffffc9000f277c40 ((work_completion)(&entry->work)){+.+.}-{0:0}, at: process_scheduled_works+0x97a/0x1630 kernel/workqueue.c:3578
#2: ffff88802ac3a5a0 (&q->q_usage_counter(io)#75){++++}-{0:0}, at: blk_mq_freeze_queue include/linux/blk-mq.h:956 [inline]
#2: ffff88802ac3a5a0 (&q->q_usage_counter(io)#75){++++}-{0:0}, at: nvme_update_ns_info+0x3ac/0x1200 drivers/nvme/host/core.c:2583
#3: ffff88802ac3a5d8 (&q->q_usage_counter(queue)#58){+.+.}-{0:0}, at: blk_mq_freeze_queue include/linux/blk-mq.h:956 [inline]
#3: ffff88802ac3a5d8 (&q->q_usage_counter(queue)#58){+.+.}-{0:0}, at: nvme_update_ns_info+0x3ac/0x1200 drivers/nvme/host/core.c:2583
stack backtrace:
CPU: 0 UID: 0 PID: 3352 Comm: kworker/u8:10 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/24/2026
Workqueue: async async_run_entry_fn
Call Trace:
<TASK>
dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
print_circular_bug+0x2e2/0x300 kernel/locking/lockdep.c:2087
check_noncircular+0x12f/0x150 kernel/locking/lockdep.c:2219
check_prev_add kernel/locking/lockdep.c:3209 [inline]
check_prevs_add kernel/locking/lockdep.c:3328 [inline]
validate_chain kernel/locking/lockdep.c:3952 [inline]
__lock_acquire+0x164c/0x2de0 kernel/locking/lockdep.c:5288
lock_acquire+0x115/0x350 kernel/locking/lockdep.c:5942
__mutex_lock_common kernel/locking/mutex.c:646 [inline]
__mutex_lock+0x197/0x15a0 kernel/locking/mutex.c:821
blk_revalidate_disk_zones+0x1c5/0x1650 block/blk-zoned.c:2560
nvme_mpath_revalidate_zones+0x106/0x1c0 drivers/nvme/host/multipath.c:301
nvme_update_ns_info+0x984/0x1200 drivers/nvme/host/core.c:2620
nvme_alloc_ns drivers/nvme/host/core.c:4293 [inline]
nvme_scan_ns+0x34c1/0x46c0 drivers/nvme/host/core.c:4483
async_run_entry_fn+0x9d/0x430 kernel/async.c:129
process_one_work kernel/workqueue.c:3495 [inline]
process_scheduled_works+0xc3d/0x1630 kernel/workqueue.c:3578
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3659
kthread+0x38b/0x480 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
nvmet: Created nvm controller 9 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
nvmet: Created nvm controller 14 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
nvmet: Created nvm controller 33 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
nvmet: Created nvm controller 35 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
nvmet: Created nvm controller 36 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
nvmet: Created nvm controller 38 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
nvmet: Created nvm controller 48 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
nvmet: Created nvm controller 68 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:5dcefdd2-eda3-4d66-a2a4-c36f6ffd95da.
* Struck through repros no longer work on HEAD.