syzbot


possible deadlock in throtl_pending_timer_fn (2)

Status: upstream: reported on 2025/01/04 14:43
Reported-by: syzbot+2dbae66a2b0fa90d2995@syzkaller.appspotmail.com
First crash: 31d, last: 31d
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in throtl_pending_timer_fn block cgroups C done 12 855d 1043d 0/28 auto-obsoleted due to no activity on 2023/04/20 01:03
linux-6.1 possible deadlock in throtl_pending_timer_fn C done 266 696d 697d 3/3 fixed on 2023/04/11 15:30

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.1.123-syzkaller #0 Not tainted
------------------------------------------------------
syz.0.189/5122 is trying to acquire lock:
ffff0000cf5d9040 (&q->queue_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:376 [inline]
ffff0000cf5d9040 (&q->queue_lock){..-.}-{2:2}, at: throtl_pending_timer_fn+0x104/0xdcc block/blk-throttle.c:1200

but task is already holding lock:
ffff800008017c60 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:41 [inline]
ffff800008017c60 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd0/0xa1c kernel/time/timer.c:1494

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((&sq->pending_timer)){+.-.}-{0:0}:
       timer_delete_sync+0x9c/0x210 kernel/time/timer.c:1448
       del_timer_sync include/linux/timer.h:198 [inline]
       throtl_pd_free+0x20/0x48 block/blk-throttle.c:493
       blkcg_deactivate_policy+0x2cc/0x4a8 block/blk-cgroup.c:1497
       blk_throtl_exit+0x9c/0x13c block/blk-throttle.c:2415
       blkcg_init_disk+0x2a4/0x318 block/blk-cgroup.c:1283
       __alloc_disk_node+0x26c/0x484 block/genhd.c:1412
       __blk_alloc_disk+0x40/0xbc block/genhd.c:1451
       md_alloc+0x4e0/0xae8 drivers/md/md.c:5743
       md_alloc_and_put drivers/md/md.c:5799 [inline]
       md_probe+0x74/0xb0 drivers/md/md.c:5812
       blk_request_module+0x184/0x1a8 block/genhd.c:752
       blkdev_get_no_open+0x48/0xdc block/bdev.c:745
       blkdev_get_by_dev+0x8c/0x8ec block/bdev.c:802
       swsusp_check+0xf8/0x454 kernel/power/swap.c:1526
       software_resume+0x130/0x680 kernel/power/hibernate.c:995
       resume_store+0x108/0x1b0 kernel/power/hibernate.c:1200
       kobj_attr_store+0x6c/0x90 lib/kobject.c:832
       sysfs_kf_write+0x200/0x280 fs/sysfs/file.c:136
       kernfs_fop_write_iter+0x334/0x48c fs/kernfs/file.c:334
       call_write_iter include/linux/fs.h:2265 [inline]
       new_sync_write fs/read_write.c:491 [inline]
       vfs_write+0x610/0x91c fs/read_write.c:584
       ksys_write+0x15c/0x26c fs/read_write.c:637
       __do_sys_write fs/read_write.c:649 [inline]
       __se_sys_write fs/read_write.c:646 [inline]
       __arm64_sys_write+0x7c/0x90 fs/read_write.c:646
       __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
       invoke_syscall+0x98/0x2bc arch/arm64/kernel/syscall.c:52
       el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:140
       do_el0_svc+0x58/0x13c arch/arm64/kernel/syscall.c:204
       el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
       el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
       el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:585

-> #1 (&blkcg->lock){....}-{2:2}:
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x54/0x6c kernel/locking/spinlock.c:154
       spin_lock include/linux/spinlock.h:351 [inline]
       blkg_create+0x9f4/0x1158 block/blk-cgroup.c:320
       blkcg_init_disk+0xd0/0x318 block/blk-cgroup.c:1259
       __alloc_disk_node+0x26c/0x484 block/genhd.c:1412
       __blk_alloc_disk+0x40/0xbc block/genhd.c:1451
       brd_alloc+0x324/0x610 drivers/block/brd.c:424
       brd_init+0x134/0x1a8 drivers/block/brd.c:529
       do_one_initcall+0x260/0xacc init/main.c:1298
       do_initcall_level+0x154/0x214 init/main.c:1371
       do_initcalls+0x58/0xac init/main.c:1387
       do_basic_setup+0x8c/0xa0 init/main.c:1406
       kernel_init_freeable+0x3a4/0x528 init/main.c:1626
       kernel_init+0x24/0x29c init/main.c:1514
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:864

-> #0 (&q->queue_lock){..-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3090 [inline]
       check_prevs_add kernel/locking/lockdep.c:3209 [inline]
       validate_chain kernel/locking/lockdep.c:3825 [inline]
       __lock_acquire+0x3338/0x7680 kernel/locking/lockdep.c:5049
       lock_acquire+0x26c/0x7cc kernel/locking/lockdep.c:5662
       __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
       _raw_spin_lock_irq+0x70/0x9c kernel/locking/spinlock.c:170
       spin_lock_irq include/linux/spinlock.h:376 [inline]
       throtl_pending_timer_fn+0x104/0xdcc block/blk-throttle.c:1200
       call_timer_fn+0x1c0/0xa1c kernel/time/timer.c:1504
       expire_timers kernel/time/timer.c:1549 [inline]
       __run_timers+0x554/0x718 kernel/time/timer.c:1820
       run_timer_softirq+0x7c/0x114 kernel/time/timer.c:1833
       handle_softirqs+0x318/0xd58 kernel/softirq.c:571
       __do_softirq+0x14/0x20 kernel/softirq.c:605
       ____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:80
       call_on_irq_stack+0x24/0x4c arch/arm64/kernel/entry.S:893
       do_softirq_own_stack+0x20/0x2c arch/arm64/kernel/irq.c:85
       invoke_softirq kernel/softirq.c:452 [inline]
       __irq_exit_rcu+0x264/0x4d4 kernel/softirq.c:654
       irq_exit_rcu+0x14/0x84 kernel/softirq.c:666
       el0_interrupt+0x80/0x260 arch/arm64/kernel/entry-common.c:717
       __el0_irq_handler_common+0x18/0x24 arch/arm64/kernel/entry-common.c:724
       el0t_64_irq_handler+0x10/0x1c arch/arm64/kernel/entry-common.c:729
       el0t_64_irq+0x18c/0x190 arch/arm64/kernel/entry.S:586

other info that might help us debug this:

Chain exists of:
  &q->queue_lock --> &blkcg->lock --> (&sq->pending_timer)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((&sq->pending_timer));
                               lock(&blkcg->lock);
                               lock((&sq->pending_timer));
  lock(&q->queue_lock);

 *** DEADLOCK ***

1 lock held by syz.0.189/5122:
 #0: ffff800008017c60 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:41 [inline]
 #0: ffff800008017c60 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd0/0xa1c kernel/time/timer.c:1494

stack backtrace:
CPU: 1 PID: 5122 Comm: syz.0.189 Not tainted 6.1.123-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call trace:
 dump_backtrace+0x1c8/0x1f4 arch/arm64/kernel/stacktrace.c:158
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
 dump_stack+0x1c/0x58 lib/dump_stack.c:113
 print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2048
 check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2170
 check_prev_add kernel/locking/lockdep.c:3090 [inline]
 check_prevs_add kernel/locking/lockdep.c:3209 [inline]
 validate_chain kernel/locking/lockdep.c:3825 [inline]
 __lock_acquire+0x3338/0x7680 kernel/locking/lockdep.c:5049
 lock_acquire+0x26c/0x7cc kernel/locking/lockdep.c:5662
 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
 _raw_spin_lock_irq+0x70/0x9c kernel/locking/spinlock.c:170
 spin_lock_irq include/linux/spinlock.h:376 [inline]
 throtl_pending_timer_fn+0x104/0xdcc block/blk-throttle.c:1200
 call_timer_fn+0x1c0/0xa1c kernel/time/timer.c:1504
 expire_timers kernel/time/timer.c:1549 [inline]
 __run_timers+0x554/0x718 kernel/time/timer.c:1820
 run_timer_softirq+0x7c/0x114 kernel/time/timer.c:1833
 handle_softirqs+0x318/0xd58 kernel/softirq.c:571
 __do_softirq+0x14/0x20 kernel/softirq.c:605
 ____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:80
 call_on_irq_stack+0x24/0x4c arch/arm64/kernel/entry.S:893
 do_softirq_own_stack+0x20/0x2c arch/arm64/kernel/irq.c:85
 invoke_softirq kernel/softirq.c:452 [inline]
 __irq_exit_rcu+0x264/0x4d4 kernel/softirq.c:654
 irq_exit_rcu+0x14/0x84 kernel/softirq.c:666
 el0_interrupt+0x80/0x260 arch/arm64/kernel/entry-common.c:717
 __el0_irq_handler_common+0x18/0x24 arch/arm64/kernel/entry-common.c:724
 el0t_64_irq_handler+0x10/0x1c arch/arm64/kernel/entry-common.c:729
 el0t_64_irq+0x18c/0x190 arch/arm64/kernel/entry.S:586
vkms_vblank_simulate: vblank timer overrun

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/01/04 14:42 linux-6.1.y 7dc732d24ff7 f3558dbf .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in throtl_pending_timer_fn
* Struck through repros no longer work on HEAD.