syzbot
possible deadlock in throtl_pending_timer_fn (3)
Status:
upstream: reported on 2026/09/17 06:14
Reported-by: syzbot+57d4f59c9de7e9f54572@syzkaller.appspotmail.com
First crash: 12d, last: 2d05h
Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
kworker/0:11/4889 is trying to acquire lock:
ffff8881443c00d0 (
&q->queue_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:376 [inline]
&q->queue_lock){..-.}-{2:2}, at: throtl_pending_timer_fn+0x10e/0x1dd0 block/blk-throttle.c:1200
but task is already holding lock:
ffffc90000007bc0 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xcf/0x670 kernel/time/timer.c:1698
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 ((&sq->pending_timer)){+.-.}-{0:0}:
__timer_delete_sync+0xe1/0x2e0 kernel/time/timer.c:1556
del_timer_sync include/linux/timer.h:200 [inline]
throtl_pd_free+0x15/0x40 block/blk-throttle.c:493
blkcg_deactivate_policy+0x35c/0x5c0 block/blk-cgroup.c:1533
blk_throtl_exit+0xa6/0x140 block/blk-throttle.c:2415
blkcg_init_disk+0x37b/0x3f0 block/blk-cgroup.c:1310
__alloc_disk_node+0x2b3/0x570 block/genhd.c:1432
__blk_alloc_disk+0x2c/0x80 block/genhd.c:1471
alloc_dev drivers/md/dm.c:2055 [inline]
dm_create+0x46b/0xf90 drivers/md/dm.c:2259
dev_create+0x12e/0x2c0 drivers/md/dm-ioctl.c:875
ctl_ioctl drivers/md/dm-ioctl.c:2055 [inline]
dm_ctl_ioctl+0xa9c/0xec0 drivers/md/dm-ioctl.c:2077
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:870 [inline]
__se_sys_ioctl+0xfa/0x170 fs/ioctl.c:856
do_syscall_x64 arch/x86/entry/common.c:46 [inline]
do_syscall_64+0x4c/0xa0 arch/x86/entry/common.c:76
entry_SYSCALL_64_after_hwframe+0x68/0xd2
-> #1 (&blkcg->lock){....}-{2:2}
:
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
blkg_create+0xbe9/0x13c0 block/blk-cgroup.c:322
blkcg_init_disk+0x28c/0x3f0 block/blk-cgroup.c:1286
__alloc_disk_node+0x2b3/0x570 block/genhd.c:1432
__blk_alloc_disk+0x2c/0x80 block/genhd.c:1471
brd_alloc+0x377/0x760 drivers/block/brd.c:424
brd_init+0x125/0x1a5 drivers/block/brd.c:529
do_one_initcall+0x257/0x800 init/main.c:1270
do_initcall_level+0x13d/0x1ed init/main.c:1343
do_initcalls+0x4b/0x8a init/main.c:1359
kernel_init_freeable+0x401/0x5ab init/main.c:1598
kernel_init+0x19/0x1b0 init/main.c:1486
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
-> #0
(&q->queue_lock
){..-.}-{2:2}
:
check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3830 [inline]
__lock_acquire+0x2d18/0x7c40 kernel/locking/lockdep.c:5063
lock_acquire+0x1bb/0x4a0 kernel/locking/lockdep.c:5678
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
_raw_spin_lock_irq+0xab/0xf0 kernel/locking/spinlock.c:170
spin_lock_irq include/linux/spinlock.h:376 [inline]
throtl_pending_timer_fn+0x10e/0x1dd0 block/blk-throttle.c:1200
call_timer_fn+0x1ac/0x670 kernel/time/timer.c:1701
expire_timers kernel/time/timer.c:1752 [inline]
__run_timers+0x56c/0x810 kernel/time/timer.c:2023
run_timer_softirq+0x63/0xf0 kernel/time/timer.c:2036
handle_softirqs+0x291/0x910 kernel/softirq.c:596
__do_softirq kernel/softirq.c:630 [inline]
invoke_softirq kernel/softirq.c:470 [inline]
__irq_exit_rcu+0x13b/0x230 kernel/softirq.c:679
irq_exit_rcu+0x5/0x20 kernel/softirq.c:691
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1118 [inline]
sysvec_apic_timer_interrupt+0xa0/0xc0 arch/x86/kernel/apic/apic.c:1118
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:691
rdtsc_ordered arch/x86/include/asm/msr.h:215 [inline]
delay_tsc+0x6d/0xc0 arch/x86/lib/delay.c:72
try_check_zero+0x3bf/0x3e0 kernel/rcu/srcutree.c:975
srcu_advance_state kernel/rcu/srcutree.c:1549 [inline]
process_srcu+0x1a7/0x11a0 kernel/rcu/srcutree.c:1651
process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
kthread+0x29d/0x330 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
other info that might help us debug this:
Chain exists of:
&q->queue_lock --> &blkcg->lock --> (&sq->pending_timer)
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock((&sq->pending_timer));
lock(&blkcg->lock);
lock((&sq->pending_timer));
lock(&q->queue_lock);
*** DEADLOCK ***
4 locks held by kworker/0:11/4889:
#0: ffff888017472138 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267
#1: ffffc9000d227d00 ((work_completion)(&(&ssp->work)->work)){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267
#2: ffffffff96ce3558 (&ssp->srcu_gp_mutex){+.+.}-{3:3}
, at: srcu_advance_state kernel/rcu/srcutree.c:1498 [inline]
, at: process_srcu+0x2f/0x11a0 kernel/rcu/srcutree.c:1651
#3: ffffc90000007bc0 ((&sq->pending_timer)
){+.-.}-{0:0}, at: call_timer_fn+0xcf/0x670 kernel/time/timer.c:1698
stack backtrace:
CPU: 0 PID: 4889 Comm: kworker/0:11 Not tainted syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/24/2026
Workqueue: rcu_gp process_srcu
Call Trace:
<IRQ>
dump_stack_lvl+0x188/0x24e lib/dump_stack.c:106
check_noncircular+0x296/0x330 kernel/locking/lockdep.c:2170
check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3830 [inline]
__lock_acquire+0x2d18/0x7c40 kernel/locking/lockdep.c:5063
lock_acquire+0x1bb/0x4a0 kernel/locking/lockdep.c:5678
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
_raw_spin_lock_irq+0xab/0xf0 kernel/locking/spinlock.c:170
spin_lock_irq include/linux/spinlock.h:376 [inline]
throtl_pending_timer_fn+0x10e/0x1dd0 block/blk-throttle.c:1200
call_timer_fn+0x1ac/0x670 kernel/time/timer.c:1701
expire_timers kernel/time/timer.c:1752 [inline]
__run_timers+0x56c/0x810 kernel/time/timer.c:2023
run_timer_softirq+0x63/0xf0 kernel/time/timer.c:2036
handle_softirqs+0x291/0x910 kernel/softirq.c:596
__do_softirq kernel/softirq.c:630 [inline]
invoke_softirq kernel/softirq.c:470 [inline]
__irq_exit_rcu+0x13b/0x230 kernel/softirq.c:679
irq_exit_rcu+0x5/0x20 kernel/softirq.c:691
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1118 [inline]
sysvec_apic_timer_interrupt+0xa0/0xc0 arch/x86/kernel/apic/apic.c:1118
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:691
RIP: 0010:rdtsc_ordered arch/x86/include/asm/msr.h:215 [inline]
RIP: 0010:delay_tsc+0x6d/0xc0 arch/x86/lib/delay.c:72
Code: 00 00 e8 76 b2 32 f7 65 8b 05 47 62 dc 75 85 c0 74 1c f3 90 bf 01 00 00 00 e8 cf b0 32 f7 e8 ea 8f 0c 00 39 c5 75 0e 0f 01 f9 <66> 90 eb bd e8 3a 76 da f6 eb dd 4c 29 fb e8 d0 8f 0c 00 89 c5 0f
RSP: 0018:ffffc9000d227ac0 EFLAGS: 00000246
RAX: 0000000054f79868 RBX: 0000000000002af9 RCX: 0000000000000000
RDX: 000000000000003f RSI: ffffffff8adf6880 RDI: ffffffff8adf6840
RBP: 0000000000000000 R08: ffffffff8e20b46f R09: 1ffffffff1c4168d
R10: dffffc0000000000 R11: fffffbfff1c4168e R12: ffff8880b8f44208
R13: dffffc0000000000 R14: 0000003f54f77127 R15: 0000000000002672
try_check_zero+0x3bf/0x3e0 kernel/rcu/srcutree.c:975
srcu_advance_state kernel/rcu/srcutree.c:1549 [inline]
process_srcu+0x1a7/0x11a0 kernel/rcu/srcutree.c:1651
process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
kthread+0x29d/0x330 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
</TASK>
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
nf_conntrack: new packet for ffff88807741bb00
nf_conntrack: Confirming conntrack ffff88807741bb00
geneve1: no route to fc00:0000:0000:0000:0000:0000:0000:0001
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
net_ratelimit: 48 callbacks suppressed
wireguard: wg0: Sending keepalive packet to peer 14 ([::1]:20003/0%0)
No more buffers in queue
Receiving skb proto 0x0800 len 52 type 0
Receiving skb proto 0x0800 len 52 type 0
Receiving skb proto 0x0800 len 52 type 0
Receiving skb proto 0x0800 len 52 type 0
No more buffers in queue
tcp_conntracks:
syn=0 ack=1 fin=1 rst=0 old=3 new=4
tcp_conntracks:
syn=0 ack=1 fin=1 rst=0 old=4 new=6
tcp_conntracks:
syn=0 ack=1 fin=1 rst=0 old=6 new=6
tcp_conntracks:
syn=0 ack=1 fin=1 rst=0 old=6 new=6
wireguard: wg0: Sending keepalive packet to peer 20 ([::1]:20003/0%0)
nf_nat: Already setup manip DST for ct ffff88807741bb00 (status bits 0x188)
nf_nat: Already setup manip SRC for ct ffff88807741bb00 (status bits 0x188)
No more buffers in queue
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
Receiving skb proto 0x0800 len 40 type 0
No more buffers in queue
tcp_conntracks:
syn=0 ack=0 fin=0 rst=1 old=6 new=8
No more buffers in queue
No more buffers in queue
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
wireguard: wg2: Packet has unallowed src IP (fe80::42) from peer 23 ([::1]:20001/0%0)
wireguard: wg2: Receiving keepalive packet from peer 23 ([::1]:20001/0%0)
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
No more buffers in queue
Receiving skb proto 0x0806 len 28 type 1
Receiving skb proto 0x0800 len 40 type 0
No more buffers in queue
No more buffers in queue
eth0: xmit ffff88801e66b640 42:01:0a:80:00:01
Added buffer head 1 to ffff888024e10400
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
tcp_conntracks:
syn=0 ack=0 fin=0 rst=1 old=8 new=8
Sent skb ffff88801e66b640
No more buffers in queue
No more buffers in queue
No more buffers in queue
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
wireguard: wg0: Sending keepalive packet to peer 2 ([::1]:20003/0%0)
wireguard: wg0: Sending keepalive packet to peer 26 ([::1]:20003/0%0)
wireguard: wg0: Sending keepalive packet to peer 8 ([::1]:20003/0%0)
No more buffers in queue
Receiving skb proto 0x0800 len 40 type 0
No more buffers in queue
tcp_conntracks:
syn=0 ack=0 fin=0 rst=1 old=8 new=8
geneve1: no route to fc00:0000:0000:0000:0000:0000:0000:0001
geneve1: no route to fc00:0000:0000:0000:0000:0000:0000:0001
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
Sent skb ffff88805380b640
No more buffers in queue
Receiving skb proto 0x0800 len 40 type 0
No more buffers in queue
tcp_conntracks:
syn=0 ack=0 fin=0 rst=1 old=8 new=8
No more buffers in queue
wireguard: wg2: Receiving keepalive packet from peer 29 ([::1]:20001/0%0)
nf_conntrack: new packet for ffff88807741b680
nf_nat: Already setup manip DST for ct ffff88807741b680 (status bits 0x188)
nf_nat: Already setup manip SRC for ct ffff88807741b680 (status bits 0x188)
nf_nat: Already setup manip DST for ct ffff88807741b680 (status bits 0x188)
nf_nat: Already setup manip SRC for ct ffff88807741b680 (status bits 0x188)
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
percpu_ref_switch_to_atomic_rcu: global 9223372036854775809 percpu 0
percpu_ref_switch_to_atomic_rcu: global 9223372036854775809 percpu 1
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
Sent skb ffff8880687ee640
No more buffers in queue
Receiving skb proto 0x0800 len 40 type 0
No more buffers in queue
tcp_conntracks:
syn=0 ack=0 fin=0 rst=1 old=8 new=8
No more buffers in queue
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
Sent skb ffff888077a79c70
No more buffers in queue
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
Sent skb ffff888077a79270
No more buffers in queue
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
Sent skb ffff888077a794f0
No more buffers in queue
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
Sent skb ffff88802ea77b40
No more buffers in queue
net_ratelimit: 21 callbacks suppressed
wireguard: wg1: Receiving keepalive packet from peer 21 ([::1]:20001/0%0)
wireguard: wg1: Receiving keepalive packet from peer 21 ([::1]:20001/0%0)
wireguard: wg0: Sending keepalive packet to peer 2 ([::1]:20003/0%0)
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
wireguard: wg1: Receiving keepalive packet from peer 3 ([::1]:20001/0%0)
wireguard: wg0: Sending keepalive packet to peer 8 ([::1]:20003/0%0)
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
virtqueue callback for ffff888024e10200 (ffffffff858a0df0)
nf_conntrack: new packet for ffff88807741b800
No more buffers in queue
Receiving skb proto 0x0806 len 28 type 1
No more buffers in queue
No more buffers in queue
eth0: xmit ffff88807d72fa00 42:01:0a:80:00:01
Added buffer head 1 to ffff888024e10400
virtqueue callback for ffff888024e10400 (ffffffff858a0ee0)
Sent skb ffff88807d72fa00
No more buffers in queue
virtqueue callback for ffff888018660400 (ffffffff8475da80)
virtqueue interrupt with no work for ffff888018660200
virtqueue interrupt with no work for ffff888018660000
----------------
Code disassembly (best guess):
0: 00 00 add %al,(%rax)
2: e8 76 b2 32 f7 call 0xf732b27d
7: 65 8b 05 47 62 dc 75 mov %gs:0x75dc6247(%rip),%eax # 0x75dc6255
e: 85 c0 test %eax,%eax
10: 74 1c je 0x2e
12: f3 90 pause
14: bf 01 00 00 00 mov $0x1,%edi
19: e8 cf b0 32 f7 call 0xf732b0ed
1e: e8 ea 8f 0c 00 call 0xc900d
23: 39 c5 cmp %eax,%ebp
25: 75 0e jne 0x35
27: 0f 01 f9 rdtscp
* 2a: 66 90 xchg %ax,%ax <-- trapping instruction
2c: eb bd jmp 0xffffffeb
2e: e8 3a 76 da f6 call 0xf6da766d
33: eb dd jmp 0x12
35: 4c 29 fb sub %r15,%rbx
38: e8 d0 8f 0c 00 call 0xc900d
3d: 89 c5 mov %eax,%ebp
3f: 0f .byte 0xf
Struck through repros no longer work on HEAD.