syzbot


possible deadlock in obj_cgroup_charge

Status: upstream: reported on 2024/09/07 13:03
Subsystems: mm cgroups
[Documentation on labels]
Reported-by: syzbot+57765728d598e67e505a@syzkaller.appspotmail.com
First crash: 78d, last: 69d
Discussions (2)
Title Replies (including bot) Last reply
[syzbot] Monthly cgroups report (Sep 2024) 0 (1) 2024/09/20 22:45
[syzbot] [cgroups?] [mm?] possible deadlock in obj_cgroup_charge 0 (1) 2024/09/07 13:03

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.11.0-rc7-syzkaller-00095-gb8e7cd09ae54 #0 Not tainted
------------------------------------------------------
kworker/0:7/5835 is trying to acquire lock:
ffff8880b883dd18 (&pool->lock){-.-.}-{2:2}, at: __queue_work+0x26b/0x1070 kernel/workqueue.c:2301

but task is already holding lock:
ffff8880b88397e0 (stock_lock){..-.}-{2:2}, at: local_lock_acquire+0x10/0x150 include/linux/local_lock_internal.h:29

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 (stock_lock){..-.}-{2:2}:
       local_lock_acquire+0x23/0x150 include/linux/local_lock_internal.h:29
       consume_obj_stock mm/memcontrol.c:2789 [inline]
       obj_cgroup_charge+0x7e/0x4d0 mm/memcontrol.c:2919
       __memcg_slab_post_alloc_hook+0x1b6/0x9c0 mm/memcontrol.c:3012
       memcg_slab_post_alloc_hook mm/slub.c:2164 [inline]
       slab_post_alloc_hook mm/slub.c:4002 [inline]
       slab_alloc_node mm/slub.c:4041 [inline]
       __do_kmalloc_node mm/slub.c:4161 [inline]
       __kmalloc_node_noprof+0x3b0/0x430 mm/slub.c:4168
       kmalloc_node_noprof include/linux/slab.h:708 [inline]
       bpf_map_kmalloc_node+0x98/0x4a0 kernel/bpf/syscall.c:422
       lpm_trie_node_alloc kernel/bpf/lpm_trie.c:299 [inline]
       trie_update_elem+0x1ef/0xe00 kernel/bpf/lpm_trie.c:342
       bpf_map_update_value+0x2c4/0x6c0 kernel/bpf/syscall.c:203
       map_update_elem+0x623/0x910 kernel/bpf/syscall.c:1654
       __sys_bpf+0x90c/0x4a00 kernel/bpf/syscall.c:5698
       __do_sys_bpf kernel/bpf/syscall.c:5817 [inline]
       __se_sys_bpf kernel/bpf/syscall.c:5815 [inline]
       __x64_sys_bpf+0x78/0xc0 kernel/bpf/syscall.c:5815
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xcd/0x250 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #3 (&trie->lock){-.-.}-{2:2}:
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x3a/0x60 kernel/locking/spinlock.c:162
       trie_delete_elem+0xb0/0x820 kernel/bpf/lpm_trie.c:462
       0xffffffffa000085b
       bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
       __bpf_prog_run include/linux/filter.h:691 [inline]
       bpf_prog_run include/linux/filter.h:698 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2406 [inline]
       bpf_trace_run2+0x234/0x590 kernel/trace/bpf_trace.c:2447
       __bpf_trace_tlb_flush+0xd2/0x110 include/trace/events/tlb.h:38
       trace_tlb_flush+0xf3/0x170 include/trace/events/tlb.h:38
       switch_mm_irqs_off+0x697/0xbb0 arch/x86/mm/tlb.c:642
       context_switch kernel/sched/core.c:5172 [inline]
       __schedule+0xc4d/0x5490 kernel/sched/core.c:6529
       __schedule_loop kernel/sched/core.c:6606 [inline]
       schedule+0xe7/0x350 kernel/sched/core.c:6621
       futex_wait_queue+0xfc/0x1f0 kernel/futex/waitwake.c:370
       __futex_wait+0x291/0x3c0 kernel/futex/waitwake.c:669
       futex_wait+0xe9/0x380 kernel/futex/waitwake.c:697
       do_futex+0x22b/0x350 kernel/futex/syscalls.c:102
       __do_sys_futex kernel/futex/syscalls.c:179 [inline]
       __se_sys_futex kernel/futex/syscalls.c:160 [inline]
       __x64_sys_futex+0x1e1/0x4c0 kernel/futex/syscalls.c:160
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xcd/0x250 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #2 (&rq->__lock){-.-.}-{2:2}:
       _raw_spin_lock_nested+0x31/0x40 kernel/locking/spinlock.c:378
       raw_spin_rq_lock_nested+0x29/0x130 kernel/sched/core.c:560
       raw_spin_rq_lock kernel/sched/sched.h:1415 [inline]
       rq_lock kernel/sched/sched.h:1714 [inline]
       task_fork_fair+0x73/0x250 kernel/sched/fair.c:12710
       sched_cgroup_fork+0x3d2/0x510 kernel/sched/core.c:4633
       copy_process+0x43a1/0x8de0 kernel/fork.c:2483
       kernel_clone+0xfd/0x960 kernel/fork.c:2781
       user_mode_thread+0xb4/0xf0 kernel/fork.c:2859
       rest_init+0x23/0x2b0 init/main.c:712
       start_kernel+0x3df/0x4c0 init/main.c:1103
       x86_64_start_reservations+0x18/0x30 arch/x86/kernel/head64.c:507
       x86_64_start_kernel+0xb2/0xc0 arch/x86/kernel/head64.c:488
       common_startup_64+0x13e/0x148

-> #1 (&p->pi_lock){-.-.}-{2:2}:
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x3a/0x60 kernel/locking/spinlock.c:162
       class_raw_spinlock_irqsave_constructor include/linux/spinlock.h:551 [inline]
       try_to_wake_up+0x9a/0x13e0 kernel/sched/core.c:4051
       create_worker+0x333/0x800 kernel/workqueue.c:2827
       workqueue_init+0x4b4/0xb70 kernel/workqueue.c:7845
       kernel_init_freeable+0x32f/0xca0 init/main.c:1562
       kernel_init+0x1c/0x2b0 init/main.c:1467
       ret_from_fork+0x48/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

-> #0 (&pool->lock){-.-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3133 [inline]
       check_prevs_add kernel/locking/lockdep.c:3252 [inline]
       validate_chain kernel/locking/lockdep.c:3868 [inline]
       __lock_acquire+0x24ed/0x3cb0 kernel/locking/lockdep.c:5142
       lock_acquire kernel/locking/lockdep.c:5759 [inline]
       lock_acquire+0x1b1/0x560 kernel/locking/lockdep.c:5724
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       __queue_work+0x26b/0x1070 kernel/workqueue.c:2301
       queue_work_on+0x11a/0x140 kernel/workqueue.c:2392
       percpu_ref_put_many include/linux/percpu-refcount.h:335 [inline]
       percpu_ref_put include/linux/percpu-refcount.h:351 [inline]
       css_put include/linux/cgroup_refcnt.h:79 [inline]
       css_put include/linux/cgroup_refcnt.h:76 [inline]
       drain_stock+0x287/0x2e0 mm/memcontrol.c:1774
       __refill_stock+0x80/0x270 mm/memcontrol.c:1811
       drain_obj_stock+0x2ab/0x3b0 mm/memcontrol.c:2820
       drain_local_stock+0xb0/0x330 mm/memcontrol.c:1792
       process_one_work+0x9c8/0x1b40 kernel/workqueue.c:3231
       process_scheduled_works kernel/workqueue.c:3312 [inline]
       worker_thread+0x6c8/0xf00 kernel/workqueue.c:3393
       kthread+0x2c4/0x3a0 kernel/kthread.c:389
       ret_from_fork+0x48/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

other info that might help us debug this:

Chain exists of:
  &pool->lock --> &trie->lock --> stock_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(stock_lock);
                               lock(&trie->lock);
                               lock(stock_lock);
  lock(&pool->lock);

 *** DEADLOCK ***

5 locks held by kworker/0:7/5835:
 #0: ffff88801ac78948 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x1277/0x1b40 kernel/workqueue.c:3206
 #1: ffffc90002dbfd80 ((work_completion)(&({ do { const void *__vpp_verify = (typeof((&memcg_stock) + 0))((void *)0); (void)__vpp_verify; } while (0); ({ unsigned long __ptr; __asm__ ("" : "=r"(__ptr) : "0"((typeof(*((&memcg_stock))) *)((&memcg_stock)))); (typeof((typeof(*((&memcg_stock))) *)((&memcg_stock)))) (__ptr + (((__per_cpu_offset[(cpu)])))); }); })->work)){+.+.}-{0:0}, at: process_one_work+0x921/0x1b40 kernel/workqueue.c:3207
 #2: ffff8880b88397e0 (stock_lock){..-.}-{2:2}, at: local_lock_acquire+0x10/0x150 include/linux/local_lock_internal.h:29
 #3: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:326 [inline]
 #3: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:838 [inline]
 #3: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: percpu_ref_put_many include/linux/percpu-refcount.h:330 [inline]
 #3: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: percpu_ref_put include/linux/percpu-refcount.h:351 [inline]
 #3: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: css_put include/linux/cgroup_refcnt.h:79 [inline]
 #3: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: css_put include/linux/cgroup_refcnt.h:76 [inline]
 #3: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: drain_stock+0x13c/0x2e0 mm/memcontrol.c:1774
 #4: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:326 [inline]
 #4: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:838 [inline]
 #4: ffffffff8ddb9fe0 (rcu_read_lock){....}-{1:2}, at: __queue_work+0xf2/0x1070 kernel/workqueue.c:2261

stack backtrace:
CPU: 0 UID: 0 PID: 5835 Comm: kworker/0:7 Not tainted 6.11.0-rc7-syzkaller-00095-gb8e7cd09ae54 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Workqueue: events drain_local_stock
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:93 [inline]
 dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:119
 check_noncircular+0x31a/0x400 kernel/locking/lockdep.c:2186
 check_prev_add kernel/locking/lockdep.c:3133 [inline]
 check_prevs_add kernel/locking/lockdep.c:3252 [inline]
 validate_chain kernel/locking/lockdep.c:3868 [inline]
 __lock_acquire+0x24ed/0x3cb0 kernel/locking/lockdep.c:5142
 lock_acquire kernel/locking/lockdep.c:5759 [inline]
 lock_acquire+0x1b1/0x560 kernel/locking/lockdep.c:5724
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 __queue_work+0x26b/0x1070 kernel/workqueue.c:2301
 queue_work_on+0x11a/0x140 kernel/workqueue.c:2392
 percpu_ref_put_many include/linux/percpu-refcount.h:335 [inline]
 percpu_ref_put include/linux/percpu-refcount.h:351 [inline]
 css_put include/linux/cgroup_refcnt.h:79 [inline]
 css_put include/linux/cgroup_refcnt.h:76 [inline]
 drain_stock+0x287/0x2e0 mm/memcontrol.c:1774
 __refill_stock+0x80/0x270 mm/memcontrol.c:1811
 drain_obj_stock+0x2ab/0x3b0 mm/memcontrol.c:2820
 drain_local_stock+0xb0/0x330 mm/memcontrol.c:1792
 process_one_work+0x9c8/0x1b40 kernel/workqueue.c:3231
 process_scheduled_works kernel/workqueue.c:3312 [inline]
 worker_thread+0x6c8/0xf00 kernel/workqueue.c:3393
 kthread+0x2c4/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x48/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/09/13 06:44 upstream b8e7cd09ae54 73e8a465 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-badwrites-root possible deadlock in obj_cgroup_charge
2024/09/13 06:44 upstream b8e7cd09ae54 73e8a465 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-badwrites-root possible deadlock in obj_cgroup_charge
2024/09/05 18:54 upstream c763c4339688 36ae3a26 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in obj_cgroup_charge
2024/09/03 12:57 upstream 67784a74e258 8045124c .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in obj_cgroup_charge
* Struck through repros no longer work on HEAD.