syzbot


possible deadlock in throtl_pending_timer_fn

Status: fixed on 2023/04/11 15:30
Reported-by: syzbot+bd917fbd065e28eccc86@syzkaller.appspotmail.com
Fix commit: b5dae1cd0d83 Revert "blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()"
First crash: 622d, last: 621d
Fix bisection: fixed by (bisect log) :
commit b5dae1cd0d8368b4338430ff93403df67f0b8bcc
Author: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date: Sat Mar 11 09:34:32 2023 +0000

  Revert "blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()"

  
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in throtl_pending_timer_fn block cgroups C done 12 779d 967d 0/28 auto-obsoleted due to no activity on 2023/04/20 01:03

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.1.16-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor155/4313 is trying to acquire lock:
ffff0000cfd20cc8 (&q->queue_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:375 [inline]
ffff0000cfd20cc8 (&q->queue_lock){..-.}-{2:2}, at: throtl_pending_timer_fn+0x104/0xdcc block/blk-throttle.c:1193

but task is already holding lock:
ffff800008017c80 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:41 [inline]
ffff800008017c80 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd0/0xcf4 kernel/time/timer.c:1464

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((&sq->pending_timer)){+.-.}-{0:0}:
       del_timer_sync+0x9c/0x210 kernel/time/timer.c:1417
       throtl_pd_free+0x20/0x48 block/blk-throttle.c:493
       blkcg_deactivate_policy+0x2d8/0x4bc block/blk-cgroup.c:1499
       blk_throtl_exit+0x9c/0x13c block/blk-throttle.c:2406
       blkcg_exit_disk+0x4c/0x5c block/blk-cgroup.c:1301
       disk_release+0x170/0x2d8 block/genhd.c:1171
       device_release+0x8c/0x1ac
       kobject_cleanup lib/kobject.c:681 [inline]
       kobject_release lib/kobject.c:712 [inline]
       kref_put include/linux/kref.h:65 [inline]
       kobject_put+0x2a8/0x41c lib/kobject.c:729
       put_device+0x28/0x40 drivers/base/core.c:3772
       put_disk+0x4c/0x64 block/genhd.c:1450
       nbd_dev_remove drivers/block/nbd.c:253 [inline]
       nbd_dev_remove_work+0x50/0xe8 drivers/block/nbd.c:269
       process_one_work+0x868/0x16f4 kernel/workqueue.c:2289
       worker_thread+0x8e4/0xfec kernel/workqueue.c:2436
       kthread+0x24c/0x2d4 kernel/kthread.c:376
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860

-> #1 (&blkcg->lock){....}-{2:2}:
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x54/0x6c kernel/locking/spinlock.c:154
       spin_lock include/linux/spinlock.h:350 [inline]
       blkg_create+0x9f4/0x1158 block/blk-cgroup.c:336
       blkcg_init_disk+0xe4/0x32c block/blk-cgroup.c:1260
       __alloc_disk_node+0x26c/0x484 block/genhd.c:1385
       __blk_alloc_disk+0x40/0xbc block/genhd.c:1424
       brd_alloc+0x2ac/0x5c8 drivers/block/brd.c:397
       brd_init+0x108/0x1c4 drivers/block/brd.c:484
       do_one_initcall+0x310/0xda4 init/main.c:1303
       do_initcall_level+0x154/0x214 init/main.c:1376
       do_initcalls+0x58/0xac init/main.c:1392
       do_basic_setup+0x8c/0xa0 init/main.c:1411
       kernel_init_freeable+0x3a4/0x528 init/main.c:1631
       kernel_init+0x24/0x29c init/main.c:1519
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860

-> #0 (&q->queue_lock){..-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3098 [inline]
       check_prevs_add kernel/locking/lockdep.c:3217 [inline]
       validate_chain kernel/locking/lockdep.c:3832 [inline]
       __lock_acquire+0x3338/0x764c kernel/locking/lockdep.c:5056
       lock_acquire+0x300/0x8e4 kernel/locking/lockdep.c:5669
       __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
       _raw_spin_lock_irq+0x70/0x9c kernel/locking/spinlock.c:170
       spin_lock_irq include/linux/spinlock.h:375 [inline]
       throtl_pending_timer_fn+0x104/0xdcc block/blk-throttle.c:1193
       call_timer_fn+0x270/0xcf4 kernel/time/timer.c:1474
       expire_timers kernel/time/timer.c:1519 [inline]
       __run_timers+0x554/0x718 kernel/time/timer.c:1790
       run_timer_softirq+0x7c/0x114 kernel/time/timer.c:1803
       __do_softirq+0x37c/0xff4 kernel/softirq.c:571
       ____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:79
       call_on_irq_stack+0x2c/0x54 arch/arm64/kernel/entry.S:889
       do_softirq_own_stack+0x20/0x2c arch/arm64/kernel/irq.c:84
       invoke_softirq kernel/softirq.c:452 [inline]
       __irq_exit_rcu+0x28c/0x534 kernel/softirq.c:650
       irq_exit_rcu+0x14/0x84 kernel/softirq.c:662
       __el1_irq arch/arm64/kernel/entry-common.c:472 [inline]
       el1_interrupt+0x38/0x68 arch/arm64/kernel/entry-common.c:486
       el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491
       el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:577
       arch_local_irq_restore arch/arm64/include/asm/irqflags.h:122 [inline]
       __free_object+0x1c0/0x868 lib/debugobjects.c:358
       free_object lib/debugobjects.c:427 [inline]
       debug_object_free+0x27c/0x438 lib/debugobjects.c:857
       destroy_hrtimer_on_stack kernel/time/hrtimer.c:452 [inline]
       hrtimer_nanosleep+0x2d4/0x384 kernel/time/hrtimer.c:2112
       common_nsleep+0xa8/0xc0 kernel/time/posix-timers.c:1236
       __do_sys_clock_nanosleep kernel/time/posix-timers.c:1277 [inline]
       __se_sys_clock_nanosleep kernel/time/posix-timers.c:1254 [inline]
       __arm64_sys_clock_nanosleep+0x350/0x38c kernel/time/posix-timers.c:1254
       __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
       invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
       el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
       do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
       el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
       el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
       el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581

other info that might help us debug this:

Chain exists of:
  &q->queue_lock --> &blkcg->lock --> (&sq->pending_timer)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((&sq->pending_timer));
                               lock(&blkcg->lock);
                               lock((&sq->pending_timer));
  lock(&q->queue_lock);

 *** DEADLOCK ***

1 lock held by syz-executor155/4313:
 #0: ffff800008017c80 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:41 [inline]
 #0: ffff800008017c80 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd0/0xcf4 kernel/time/timer.c:1464

stack backtrace:
CPU: 1 PID: 4313 Comm: syz-executor155 Not tainted 6.1.16-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
Call trace:
 dump_backtrace+0x1c8/0x1f4 arch/arm64/kernel/stacktrace.c:158
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
 dump_stack+0x1c/0x5c lib/dump_stack.c:113
 print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2056
 check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2178
 check_prev_add kernel/locking/lockdep.c:3098 [inline]
 check_prevs_add kernel/locking/lockdep.c:3217 [inline]
 validate_chain kernel/locking/lockdep.c:3832 [inline]
 __lock_acquire+0x3338/0x764c kernel/locking/lockdep.c:5056
 lock_acquire+0x300/0x8e4 kernel/locking/lockdep.c:5669
 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
 _raw_spin_lock_irq+0x70/0x9c kernel/locking/spinlock.c:170
 spin_lock_irq include/linux/spinlock.h:375 [inline]
 throtl_pending_timer_fn+0x104/0xdcc block/blk-throttle.c:1193
 call_timer_fn+0x270/0xcf4 kernel/time/timer.c:1474
 expire_timers kernel/time/timer.c:1519 [inline]
 __run_timers+0x554/0x718 kernel/time/timer.c:1790
 run_timer_softirq+0x7c/0x114 kernel/time/timer.c:1803
 __do_softirq+0x37c/0xff4 kernel/softirq.c:571
 ____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:79
 call_on_irq_stack+0x2c/0x54 arch/arm64/kernel/entry.S:889
 do_softirq_own_stack+0x20/0x2c arch/arm64/kernel/irq.c:84
 invoke_softirq kernel/softirq.c:452 [inline]
 __irq_exit_rcu+0x28c/0x534 kernel/softirq.c:650
 irq_exit_rcu+0x14/0x84 kernel/softirq.c:662
 __el1_irq arch/arm64/kernel/entry-common.c:472 [inline]
 el1_interrupt+0x38/0x68 arch/arm64/kernel/entry-common.c:486
 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491
 el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:577
 arch_local_irq_restore arch/arm64/include/asm/irqflags.h:122 [inline]
 __free_object+0x1c0/0x868 lib/debugobjects.c:358
 free_object lib/debugobjects.c:427 [inline]
 debug_object_free+0x27c/0x438 lib/debugobjects.c:857
 destroy_hrtimer_on_stack kernel/time/hrtimer.c:452 [inline]
 hrtimer_nanosleep+0x2d4/0x384 kernel/time/hrtimer.c:2112
 common_nsleep+0xa8/0xc0 kernel/time/posix-timers.c:1236
 __do_sys_clock_nanosleep kernel/time/posix-timers.c:1277 [inline]
 __se_sys_clock_nanosleep kernel/time/posix-timers.c:1254 [inline]
 __arm64_sys_clock_nanosleep+0x350/0x38c kernel/time/posix-timers.c:1254
 __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
 invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
 el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
 do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
 el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
 el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
 el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581

Crashes (266):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/03/10 17:59 linux-6.1.y 8a923980a190 5205ef30 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in throtl_pending_timer_fn
2023/03/10 18:23 linux-6.1.y 8a923980a190 5205ef30 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in throtl_pending_timer_fn
2023/03/11 09:47 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/11 08:32 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/11 06:38 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/11 04:56 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/11 04:50 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/11 03:22 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/11 01:22 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 23:25 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 20:55 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 20:21 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 20:10 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 19:08 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 18:07 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 17:23 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 16:39 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 15:56 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 15:14 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 14:17 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 14:15 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 13:15 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 12:20 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 12:11 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 12:10 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 12:00 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:58 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:52 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:49 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:47 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:42 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:39 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:37 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:34 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:33 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:30 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:28 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:20 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:17 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:14 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:12 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:06 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 11:00 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 10:59 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 10:58 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 10:54 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 10:52 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
2023/03/10 10:08 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in throtl_pending_timer_fn
* Struck through repros no longer work on HEAD.