syzbot


possible deadlock in kvfree_call_rcu

Status: upstream: reported C repro on 2024/03/25 23:20
Bug presence: origin:upstream
Labels: missing-backport
[Documentation on labels]
Reported-by: syzbot+cb88665052d93c3f0be3@syzkaller.appspotmail.com
First crash: 240d, last: 1h59m
Bug presence (3)
Date Name Commit Repro Result
2024/06/07 linux-5.15.y (ToT) c61bd26ae81a C [report] possible deadlock in trie_delete_elem
2024/04/23 upstream (ToT) 71b1543c83d6 C [report] possible deadlock in trie_delete_elem
2024/06/07 upstream (ToT) 8a92980606e3 C Didn't crash
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in kvfree_call_rcu origin:upstream C 609 2d07h 244d 0/3 upstream: reported C repro on 2024/03/21 20:02
upstream possible deadlock in kvfree_call_rcu (2) net C unreliable 1096 3d09h 163d 0/28 upstream: reported C repro on 2024/06/10 22:50
upstream possible deadlock in kvfree_call_rcu bpf C 1431 170d 239d 25/28 fixed on 2024/06/05 13:52
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/06/07 17:58 4h58m fix candidate upstream OK (1) job log
marked invalid by nogikh@google.com

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.167-syzkaller #0 Not tainted
------------------------------------------------------
rcu_preempt/15 is trying to acquire lock:
ffff8880b9027e78 (krc.lock){....}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
ffff8880b9027e78 (krc.lock){....}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
ffff8880b9027e78 (krc.lock){....}-{2:2}, at: kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597

but task is already holding lock:
ffff88802368d1b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){....}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450
       bpf_prog_4febba1e7eaf042b+0x42/0x1f0
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1913
       rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
       rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #1 (&base->lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       lock_timer_base+0x120/0x260 kernel/time/timer.c:946
       __mod_timer+0x1d6/0xeb0 kernel/time/timer.c:1019
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       schedule_delayed_work include/linux/workqueue.h:631 [inline]
       kvfree_call_rcu+0x50e/0x8a0 kernel/rcu/tree.c:3625
       rtnl_register_internal+0x443/0x530 net/core/rtnetlink.c:223
       rtnl_register+0x32/0x70 net/core/rtnetlink.c:273
       ip_rt_init+0x2e6/0x390 net/ipv4/route.c:3750
       ip_init+0xa/0x20 net/ipv4/ip_output.c:1749
       inet_init+0x27c/0x390 net/ipv4/af_inet.c:2009
       do_one_initcall+0x22b/0x7a0 init/main.c:1302
       do_initcall_level+0x157/0x210 init/main.c:1375
       do_initcalls+0x49/0x90 init/main.c:1391
       kernel_init_freeable+0x425/0x5c0 init/main.c:1615
       kernel_init+0x19/0x290 init/main.c:1506
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #0 (krc.lock){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
       krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
       add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
       kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
       trie_delete_elem+0x520/0x690
       bpf_prog_4febba1e7eaf042b+0x42/0x1f0
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1913
       rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
       rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

other info that might help us debug this:

Chain exists of:
  krc.lock --> &base->lock --> &trie->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&trie->lock);
                               lock(&base->lock);
                               lock(&trie->lock);
  lock(krc.lock);

 *** DEADLOCK ***

3 locks held by rcu_preempt/15:
 #0: ffff8880b9028098 (&base->lock){-.-.}-{2:2}, at: lock_timer_base+0x120/0x260 kernel/time/timer.c:946
 #1: ffffffff8c91fc60 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #2: ffff88802368d1b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

stack backtrace:
CPU: 0 PID: 15 Comm: rcu_preempt Not tainted 5.15.167-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
 add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
 kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
 trie_delete_elem+0x520/0x690
 bpf_prog_4febba1e7eaf042b+0x42/0x1f0
 bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
 trace_timer_start include/trace/events/timer.h:52 [inline]
 enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
 internal_add_timer kernel/time/timer.c:611 [inline]
 __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
 schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1913
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>

Crashes (596):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/09/22 16:31 linux-5.15.y 3a5928702e71 6f888b75 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/19 05:54 linux-5.15.y 3a5928702e71 c673ca06 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/21 19:40 linux-5.15.y 4878aadf2d15 edc5149a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/21 18:59 linux-5.15.y 4878aadf2d15 edc5149a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 03:24 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 01:49 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/01 23:03 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/27 21:11 linux-5.15.y 9465fef4ae35 120789fd .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/25 23:20 linux-5.15.y b95c01af2113 0ea90952 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/21 12:03 linux-5.15.y 0a51d2d4527b 4b25d554 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/21 09:26 linux-5.15.y 0a51d2d4527b 4b25d554 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/20 21:55 linux-5.15.y 0a51d2d4527b 4fca1650 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/20 16:40 linux-5.15.y 0a51d2d4527b 4fca1650 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/20 09:34 linux-5.15.y 0a51d2d4527b 7d02db5a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/20 04:35 linux-5.15.y 0a51d2d4527b 7d02db5a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/19 10:25 linux-5.15.y 0a51d2d4527b 571351cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/19 06:53 linux-5.15.y 0a51d2d4527b 571351cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/19 05:18 linux-5.15.y 0a51d2d4527b 571351cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/19 02:42 linux-5.15.y 0a51d2d4527b 571351cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/19 02:42 linux-5.15.y 0a51d2d4527b 571351cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/18 22:41 linux-5.15.y 0a51d2d4527b e7bb5d6e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/18 09:45 linux-5.15.y 0a51d2d4527b cfe3a04a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/15 15:41 linux-5.15.y d98fd109f827 f6ede3a3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/10 06:33 linux-5.15.y 3c17fc483905 6b856513 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/10 01:23 linux-5.15.y 3c17fc483905 6b856513 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/08 13:21 linux-5.15.y 72244eab0dad 179b040e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/08 12:17 linux-5.15.y 72244eab0dad 179b040e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/07 22:24 linux-5.15.y 72244eab0dad 867e44df .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/06 13:21 linux-5.15.y 72244eab0dad 3a465482 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/06 11:37 linux-5.15.y 72244eab0dad 3a465482 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/04 15:22 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/04 15:19 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/04 14:00 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/04 09:44 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/04 08:02 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/11/03 13:25 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/03 12:24 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/03 03:17 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/11/01 10:49 linux-5.15.y 72244eab0dad 96eb609f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/30 22:41 linux-5.15.y 74cdd62cb470 fb888278 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/27 18:13 linux-5.15.y 74cdd62cb470 65e8686b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/23 11:12 linux-5.15.y 74cdd62cb470 15fa2979 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/23 04:04 linux-5.15.y 74cdd62cb470 15fa2979 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/23 03:59 linux-5.15.y 74cdd62cb470 15fa2979 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/22 08:05 linux-5.15.y 584a40a22cb9 a93682b3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/21 16:52 linux-5.15.y 584a40a22cb9 f1e4447c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/20 09:03 linux-5.15.y 584a40a22cb9 cd6fc0a3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/18 18:18 linux-5.15.y 584a40a22cb9 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/18 05:54 linux-5.15.y 584a40a22cb9 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/17 08:13 linux-5.15.y 3a5928702e71 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/17 06:30 linux-5.15.y 3a5928702e71 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/17 04:15 linux-5.15.y 3a5928702e71 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/17 00:26 linux-5.15.y 3a5928702e71 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/16 22:43 linux-5.15.y 3a5928702e71 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/16 03:47 linux-5.15.y 3a5928702e71 bde2d81c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/14 18:53 linux-5.15.y 3a5928702e71 084d8178 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
* Struck through repros no longer work on HEAD.