syzbot


possible deadlock in kvfree_call_rcu

Status: upstream: reported C repro on 2024/03/21 20:02
Reported-by: syzbot+e5952d44ee3e1993cddf@syzkaller.appspotmail.com
First crash: 42d, last: 21m
Bug presence (1)
Date Name Commit Repro Result
2024/04/16 upstream (ToT) 96fca68c4fbf C Failed due to an error; will retry later
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in kvfree_call_rcu bpf C 1280 now 37d 4/26 upstream: reported C repro on 2024/03/26 19:00
linux-5.15 possible deadlock in kvfree_call_rcu origin:upstream C 117 2h53m 38d 0/3 upstream: reported C repro on 2024/03/25 23:20

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.1.83-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:1/11 is trying to acquire lock:
ffff8880b9928118 (krc.lock){....}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2990 [inline]
ffff8880b9928118 (krc.lock){....}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3310 [inline]
ffff8880b9928118 (krc.lock){....}-{2:2}, at: kvfree_call_rcu+0x1b2/0x8c0 kernel/rcu/tree.c:3401

but task is already holding lock:
ffff888077510a38 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:451

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){....}-{2:2}:
       lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:451
       bpf_prog_2c29ac5cdc6b1842+0x3a/0x3e
       bpf_dispatcher_nop_func include/linux/bpf.h:989 [inline]
       __bpf_prog_run include/linux/filter.h:603 [inline]
       bpf_prog_run include/linux/filter.h:610 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2273 [inline]
       bpf_trace_run3+0x231/0x440 kernel/trace/bpf_trace.c:2313
       trace_timer_start include/trace/events/timer.h:53 [inline]
       enqueue_timer+0x440/0x600 kernel/time/timer.c:609
       __mod_timer+0x92b/0xee0
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1704
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       toggle_allocation_gate+0x427/0x480 mm/kfence/core.c:820
       process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
       worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
       kthread+0x28d/0x320 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307

-> #1 (&base->lock){-.-.}-{2:2}:
       lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       lock_timer_base+0x120/0x260 kernel/time/timer.c:999
       __mod_timer+0x1cb/0xee0 kernel/time/timer.c:1072
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1704
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       schedule_delayed_monitor_work kernel/rcu/tree.c:3171 [inline]
       kvfree_call_rcu+0x520/0x8c0 kernel/rcu/tree.c:3427
       rtnl_register_internal+0x489/0x580 net/core/rtnetlink.c:260
       rtnl_register+0x32/0x70 net/core/rtnetlink.c:310
       ip_rt_init+0x335/0x3c7 net/ipv4/route.c:3768
       ip_init+0xa/0x14 net/ipv4/ip_output.c:1767
       inet_init+0x2ae/0x3c0 net/ipv4/af_inet.c:2031
       do_one_initcall+0x265/0x8f0 init/main.c:1297
       do_initcall_level+0x157/0x207 init/main.c:1370
       do_initcalls+0x49/0x86 init/main.c:1386
       kernel_init_freeable+0x45c/0x60f init/main.c:1625
       kernel_init+0x19/0x290 init/main.c:1513
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307

-> #0 (krc.lock){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3090 [inline]
       check_prevs_add kernel/locking/lockdep.c:3209 [inline]
       validate_chain+0x1661/0x5950 kernel/locking/lockdep.c:3825
       __lock_acquire+0x125b/0x1f80 kernel/locking/lockdep.c:5049
       lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
       krc_this_cpu_lock kernel/rcu/tree.c:2990 [inline]
       add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3310 [inline]
       kvfree_call_rcu+0x1b2/0x8c0 kernel/rcu/tree.c:3401
       trie_delete_elem+0x520/0x690
       bpf_prog_2c29ac5cdc6b1842+0x3a/0x3e
       bpf_dispatcher_nop_func include/linux/bpf.h:989 [inline]
       __bpf_prog_run include/linux/filter.h:603 [inline]
       bpf_prog_run include/linux/filter.h:610 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2273 [inline]
       bpf_trace_run3+0x231/0x440 kernel/trace/bpf_trace.c:2313
       trace_timer_start include/trace/events/timer.h:53 [inline]
       enqueue_timer+0x440/0x600 kernel/time/timer.c:609
       __mod_timer+0x92b/0xee0
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1704
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       toggle_allocation_gate+0x427/0x480 mm/kfence/core.c:820
       process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
       worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
       kthread+0x28d/0x320 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307

other info that might help us debug this:

Chain exists of:
  krc.lock --> &base->lock --> &trie->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&trie->lock);
                               lock(&base->lock);
                               lock(&trie->lock);
  lock(krc.lock);

 *** DEADLOCK ***

5 locks held by kworker/u4:1/11:
 #0: ffff888012479138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #1: ffffc90000107d20 ((work_completion)(&(&kfence_timer)->work)){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #2: ffff8880b9928358 (&base->lock){-.-.}-{2:2}, at: lock_timer_base+0x120/0x260 kernel/time/timer.c:999
 #3: ffffffff8d12a940 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #3: ffffffff8d12a940 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #3: ffffffff8d12a940 (rcu_read_lock){....}-{1:2}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2272 [inline]
 #3: ffffffff8d12a940 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run3+0x146/0x440 kernel/trace/bpf_trace.c:2313
 #4: ffff888077510a38 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:451

stack backtrace:
CPU: 1 PID: 11 Comm: kworker/u4:1 Not tainted 6.1.83-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Workqueue: events_unbound toggle_allocation_gate
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 check_noncircular+0x2fa/0x3b0 kernel/locking/lockdep.c:2170
 check_prev_add kernel/locking/lockdep.c:3090 [inline]
 check_prevs_add kernel/locking/lockdep.c:3209 [inline]
 validate_chain+0x1661/0x5950 kernel/locking/lockdep.c:3825
 __lock_acquire+0x125b/0x1f80 kernel/locking/lockdep.c:5049
 lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 krc_this_cpu_lock kernel/rcu/tree.c:2990 [inline]
 add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3310 [inline]
 kvfree_call_rcu+0x1b2/0x8c0 kernel/rcu/tree.c:3401
 trie_delete_elem+0x520/0x690
 bpf_prog_2c29ac5cdc6b1842+0x3a/0x3e
 bpf_dispatcher_nop_func include/linux/bpf.h:989 [inline]
 __bpf_prog_run include/linux/filter.h:603 [inline]
 bpf_prog_run include/linux/filter.h:610 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:2273 [inline]
 bpf_trace_run3+0x231/0x440 kernel/trace/bpf_trace.c:2313
 trace_timer_start include/trace/events/timer.h:53 [inline]
 enqueue_timer+0x440/0x600 kernel/time/timer.c:609
 __mod_timer+0x92b/0xee0
 queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1704
 queue_delayed_work include/linux/workqueue.h:527 [inline]
 toggle_allocation_gate+0x427/0x480 mm/kfence/core.c:820
 process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
 worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>

Crashes (215):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/04/02 07:03 linux-6.1.y e5cd595e23c1 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 05:53 linux-6.1.y e5cd595e23c1 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 03:44 linux-6.1.y e5cd595e23c1 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/01 21:44 linux-6.1.y e5cd595e23c1 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/05/03 16:16 linux-6.1.y 909ba1f1b414 dd26401e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/05/03 04:50 linux-6.1.y 909ba1f1b414 ddfc15a1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/03 00:16 linux-6.1.y 909ba1f1b414 ddfc15a1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/05/02 22:30 linux-6.1.y 909ba1f1b414 ddfc15a1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/05/02 20:02 linux-6.1.y 909ba1f1b414 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/05/02 03:49 linux-6.1.y dcbc050cb0d3 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/05/01 13:35 linux-6.1.y dcbc050cb0d3 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/01 11:58 linux-6.1.y dcbc050cb0d3 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/01 05:34 linux-6.1.y dcbc050cb0d3 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/30 23:14 linux-6.1.y dcbc050cb0d3 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/30 23:00 linux-6.1.y dcbc050cb0d3 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/30 17:06 linux-6.1.y dcbc050cb0d3 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/30 12:31 linux-6.1.y dcbc050cb0d3 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/30 00:37 linux-6.1.y dcbc050cb0d3 f10afd69 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/29 23:15 linux-6.1.y dcbc050cb0d3 f10afd69 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/29 16:16 linux-6.1.y dcbc050cb0d3 27e33c58 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/29 11:40 linux-6.1.y dcbc050cb0d3 27e33c58 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/29 09:01 linux-6.1.y f2295faba5e8 27e33c58 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/28 15:26 linux-6.1.y f2295faba5e8 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 23:41 linux-6.1.y f2295faba5e8 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 21:30 linux-6.1.y f2295faba5e8 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 20:00 linux-6.1.y f2295faba5e8 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 19:14 linux-6.1.y f2295faba5e8 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 17:58 linux-6.1.y f2295faba5e8 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/27 07:50 linux-6.1.y 6741e066ec76 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 05:13 linux-6.1.y 6741e066ec76 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 03:44 linux-6.1.y 6741e066ec76 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/27 01:58 linux-6.1.y 6741e066ec76 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/27 00:00 linux-6.1.y 6741e066ec76 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/26 19:57 linux-6.1.y 6741e066ec76 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/26 18:31 linux-6.1.y 6741e066ec76 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/26 15:25 linux-6.1.y 6741e066ec76 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/26 13:14 linux-6.1.y 6741e066ec76 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/26 02:06 linux-6.1.y 6741e066ec76 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/25 23:13 linux-6.1.y 6741e066ec76 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/25 17:36 linux-6.1.y 6741e066ec76 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/24 22:13 linux-6.1.y 6741e066ec76 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 20:53 linux-6.1.y 6741e066ec76 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/24 18:41 linux-6.1.y 6741e066ec76 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 15:40 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/24 12:19 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/24 10:45 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 06:29 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 02:38 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/23 23:41 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/23 10:50 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/23 08:15 linux-6.1.y 6741e066ec76 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/23 07:00 linux-6.1.y 6741e066ec76 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/23 02:08 linux-6.1.y 6741e066ec76 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/22 22:30 linux-6.1.y 6741e066ec76 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/04/22 18:30 linux-6.1.y 6741e066ec76 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/22 17:24 linux-6.1.y 6741e066ec76 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
2024/03/21 20:02 linux-6.1.y d7543167affd 7a239ce7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in kvfree_call_rcu
* Struck through repros no longer work on HEAD.