syzbot


possible deadlock in blkcg_deactivate_policy

Status: fixed on 2023/05/30 15:47
Reported-by: syzbot+a61a7dbe3e052af19de8@syzkaller.appspotmail.com
Fix commit: b5dae1cd0d83 Revert "blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()"
First crash: 406d, last: 360d
Fix bisection: fixed by (bisect log) :
commit b5dae1cd0d8368b4338430ff93403df67f0b8bcc
Author: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date: Sat Mar 11 09:34:32 2023 +0000

  Revert "blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()"

  
Bug presence (1)
Date Name Commit Repro Result
2023/05/17 upstream (ToT) 4d6d4c7f541d C Didn't crash
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in blkcg_deactivate_policy (2) origin:lts-only C inconclusive 3 95d 137d 0/3 upstream: reported C repro on 2023/12/04 11:30
upstream possible deadlock in blkcg_deactivate_policy block cgroups C done error 93 483d 734d 0/26 auto-obsoleted due to no activity on 2023/05/12 00:27

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.1.16-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor181/4306 is trying to acquire lock:
ffff0000ccd080a8 ((&sq->pending_timer)){+.-.}-{0:0}, at: del_timer_sync+0x74/0x210 kernel/time/timer.c:1417

but task is already holding lock:
ffff800019965b30 (&blkcg->lock){....}-{2:2}, at: spin_lock include/linux/spinlock.h:350 [inline]
ffff800019965b30 (&blkcg->lock){....}-{2:2}, at: blkcg_deactivate_policy+0x1b8/0x4bc block/blk-cgroup.c:1495

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&blkcg->lock){....}-{2:2}:
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x54/0x6c kernel/locking/spinlock.c:154
       spin_lock include/linux/spinlock.h:350 [inline]
       blkg_create+0x9f4/0x1158 block/blk-cgroup.c:336
       blkcg_init_disk+0xe4/0x32c block/blk-cgroup.c:1260
       __alloc_disk_node+0x26c/0x484 block/genhd.c:1385
       __blk_alloc_disk+0x40/0xbc block/genhd.c:1424
       brd_alloc+0x2ac/0x5c8 drivers/block/brd.c:397
       brd_init+0x108/0x1c4 drivers/block/brd.c:484
       do_one_initcall+0x310/0xda4 init/main.c:1303
       do_initcall_level+0x154/0x214 init/main.c:1376
       do_initcalls+0x58/0xac init/main.c:1392
       do_basic_setup+0x8c/0xa0 init/main.c:1411
       kernel_init_freeable+0x3a4/0x528 init/main.c:1631
       kernel_init+0x24/0x29c init/main.c:1519
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860

-> #1 (&q->queue_lock){..-.}-{2:2}:
       __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
       _raw_spin_lock_irq+0x70/0x9c kernel/locking/spinlock.c:170
       spin_lock_irq include/linux/spinlock.h:375 [inline]
       throtl_pending_timer_fn+0x104/0xdcc block/blk-throttle.c:1193
       call_timer_fn+0x270/0xcf4 kernel/time/timer.c:1474
       expire_timers kernel/time/timer.c:1519 [inline]
       __run_timers+0x554/0x718 kernel/time/timer.c:1790
       run_timer_softirq+0x7c/0x114 kernel/time/timer.c:1803
       __do_softirq+0x37c/0xff4 kernel/softirq.c:571
       ____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:79
       call_on_irq_stack+0x2c/0x54 arch/arm64/kernel/entry.S:889
       do_softirq_own_stack+0x20/0x2c arch/arm64/kernel/irq.c:84
       invoke_softirq kernel/softirq.c:452 [inline]
       __irq_exit_rcu+0x28c/0x534 kernel/softirq.c:650
       irq_exit_rcu+0x14/0x84 kernel/softirq.c:662
       __el1_irq arch/arm64/kernel/entry-common.c:472 [inline]
       el1_interrupt+0x38/0x68 arch/arm64/kernel/entry-common.c:486
       el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491
       el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:577
       arch_local_irq_enable+0xc/0x18 arch/arm64/include/asm/irqflags.h:35
       default_idle_call+0x68/0xdc kernel/sched/idle.c:109
       cpuidle_idle_call kernel/sched/idle.c:191 [inline]
       do_idle+0x1e0/0x514 kernel/sched/idle.c:303
       cpu_startup_entry+0x24/0x28 kernel/sched/idle.c:400
       secondary_start_kernel+0x19c/0x1c4 arch/arm64/kernel/smp.c:265
       __secondary_switched+0xb0/0xb4 arch/arm64/kernel/head.S:618

-> #0 ((&sq->pending_timer)){+.-.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3098 [inline]
       check_prevs_add kernel/locking/lockdep.c:3217 [inline]
       validate_chain kernel/locking/lockdep.c:3832 [inline]
       __lock_acquire+0x3338/0x764c kernel/locking/lockdep.c:5056
       lock_acquire+0x300/0x8e4 kernel/locking/lockdep.c:5669
       del_timer_sync+0x9c/0x210 kernel/time/timer.c:1417
       throtl_pd_free+0x20/0x48 block/blk-throttle.c:493
       blkcg_deactivate_policy+0x2d8/0x4bc block/blk-cgroup.c:1499
       blk_throtl_exit+0x9c/0x13c block/blk-throttle.c:2406
       blkcg_exit_disk+0x4c/0x5c block/blk-cgroup.c:1301
       disk_release+0x170/0x2d8 block/genhd.c:1171
       device_release+0x8c/0x1ac
       kobject_cleanup lib/kobject.c:681 [inline]
       kobject_release lib/kobject.c:712 [inline]
       kref_put include/linux/kref.h:65 [inline]
       kobject_put+0x2a8/0x41c lib/kobject.c:729
       put_device+0x28/0x40 drivers/base/core.c:3772
       put_disk+0x4c/0x64 block/genhd.c:1450
       loop_remove drivers/block/loop.c:2080 [inline]
       loop_control_remove drivers/block/loop.c:2128 [inline]
       loop_control_ioctl+0x534/0x650 drivers/block/loop.c:2166
       vfs_ioctl fs/ioctl.c:51 [inline]
       __do_sys_ioctl fs/ioctl.c:870 [inline]
       __se_sys_ioctl fs/ioctl.c:856 [inline]
       __arm64_sys_ioctl+0x14c/0x1c8 fs/ioctl.c:856
       __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
       invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
       el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
       do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
       el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
       el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
       el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581

other info that might help us debug this:

Chain exists of:
  (&sq->pending_timer) --> &q->queue_lock --> &blkcg->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&blkcg->lock);
                               lock(&q->queue_lock);
                               lock(&blkcg->lock);
  lock((&sq->pending_timer));

 *** DEADLOCK ***

3 locks held by syz-executor181/4306:
 #0: ffff0000cccf0b28 (&q->blkcg_mutex){+.+.}-{3:3}, at: blkcg_deactivate_policy+0xfc/0x4bc block/blk-cgroup.c:1487
 #1: ffff0000cccf0918 (&q->queue_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:375 [inline]
 #1: ffff0000cccf0918 (&q->queue_lock){..-.}-{2:2}, at: blkcg_deactivate_policy+0x108/0x4bc block/blk-cgroup.c:1488
 #2: ffff800019965b30 (&blkcg->lock){....}-{2:2}, at: spin_lock include/linux/spinlock.h:350 [inline]
 #2: ffff800019965b30 (&blkcg->lock){....}-{2:2}, at: blkcg_deactivate_policy+0x1b8/0x4bc block/blk-cgroup.c:1495

stack backtrace:
CPU: 1 PID: 4306 Comm: syz-executor181 Not tainted 6.1.16-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
Call trace:
 dump_backtrace+0x1c8/0x1f4 arch/arm64/kernel/stacktrace.c:158
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
 dump_stack+0x1c/0x5c lib/dump_stack.c:113
 print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2056
 check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2178
 check_prev_add kernel/locking/lockdep.c:3098 [inline]
 check_prevs_add kernel/locking/lockdep.c:3217 [inline]
 validate_chain kernel/locking/lockdep.c:3832 [inline]
 __lock_acquire+0x3338/0x764c kernel/locking/lockdep.c:5056
 lock_acquire+0x300/0x8e4 kernel/locking/lockdep.c:5669
 del_timer_sync+0x9c/0x210 kernel/time/timer.c:1417
 throtl_pd_free+0x20/0x48 block/blk-throttle.c:493
 blkcg_deactivate_policy+0x2d8/0x4bc block/blk-cgroup.c:1499
 blk_throtl_exit+0x9c/0x13c block/blk-throttle.c:2406
 blkcg_exit_disk+0x4c/0x5c block/blk-cgroup.c:1301
 disk_release+0x170/0x2d8 block/genhd.c:1171
 device_release+0x8c/0x1ac
 kobject_cleanup lib/kobject.c:681 [inline]
 kobject_release lib/kobject.c:712 [inline]
 kref_put include/linux/kref.h:65 [inline]
 kobject_put+0x2a8/0x41c lib/kobject.c:729
 put_device+0x28/0x40 drivers/base/core.c:3772
 put_disk+0x4c/0x64 block/genhd.c:1450
 loop_remove drivers/block/loop.c:2080 [inline]
 loop_control_remove drivers/block/loop.c:2128 [inline]
 loop_control_ioctl+0x534/0x650 drivers/block/loop.c:2166
 vfs_ioctl fs/ioctl.c:51 [inline]
 __do_sys_ioctl fs/ioctl.c:870 [inline]
 __se_sys_ioctl fs/ioctl.c:856 [inline]
 __arm64_sys_ioctl+0x14c/0x1c8 fs/ioctl.c:856
 __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
 invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
 el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
 do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
 el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
 el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
 el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581

Crashes (168):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/03/10 18:44 linux-6.1.y 8a923980a190 5205ef30 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/04/25 11:08 linux-6.1.y f17b0ab65d17 65320f8e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/04/08 06:57 linux-6.1.y 543aff194ab6 71147e29 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/21 12:13 linux-6.1.y 7eaef76fbc46 7939252e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 08:21 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 07:45 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 06:45 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 06:43 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 05:43 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 05:43 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 03:44 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 03:07 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 02:06 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 02:06 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/11 00:53 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 22:13 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 21:10 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 19:52 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:39 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:39 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:38 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:36 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:31 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:31 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:28 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:28 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:21 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:14 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:11 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:07 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:07 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:04 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:03 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:02 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 12:00 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:57 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:53 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:52 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:49 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:46 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:45 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:45 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:40 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:29 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:15 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 11:02 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
2023/03/10 09:38 linux-6.1.y 8a923980a190 5205ef30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in blkcg_deactivate_policy
* Struck through repros no longer work on HEAD.