syzbot


possible deadlock in kvfree_call_rcu

Status: upstream: reported C repro on 2024/03/25 23:20
Bug presence: origin:upstream
Labels: missing-backport
[Documentation on labels]
Reported-by: syzbot+cb88665052d93c3f0be3@syzkaller.appspotmail.com
First crash: 193d, last: 2h23m
Bug presence (3)
Date Name Commit Repro Result
2024/06/07 linux-5.15.y (ToT) c61bd26ae81a C [report] possible deadlock in trie_delete_elem
2024/04/23 upstream (ToT) 71b1543c83d6 C [report] possible deadlock in trie_delete_elem
2024/06/07 upstream (ToT) 8a92980606e3 C Didn't crash
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in kvfree_call_rcu origin:upstream C 556 18h52m 197d 0/3 upstream: reported C repro on 2024/03/21 20:02
upstream possible deadlock in kvfree_call_rcu (2) net C unreliable 787 8h49m 116d 0/28 upstream: reported C repro on 2024/06/10 22:50
upstream possible deadlock in kvfree_call_rcu bpf C 1431 123d 192d 25/28 fixed on 2024/06/05 13:52
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/06/07 17:58 4h58m fix candidate upstream OK (1) job log
marked invalid by nogikh@google.com

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.167-syzkaller #0 Not tainted
------------------------------------------------------
rcu_preempt/15 is trying to acquire lock:
ffff8880b9027e78 (krc.lock){....}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
ffff8880b9027e78 (krc.lock){....}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
ffff8880b9027e78 (krc.lock){....}-{2:2}, at: kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597

but task is already holding lock:
ffff88802368d1b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){....}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450
       bpf_prog_4febba1e7eaf042b+0x42/0x1f0
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1913
       rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
       rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #1 (&base->lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       lock_timer_base+0x120/0x260 kernel/time/timer.c:946
       __mod_timer+0x1d6/0xeb0 kernel/time/timer.c:1019
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       schedule_delayed_work include/linux/workqueue.h:631 [inline]
       kvfree_call_rcu+0x50e/0x8a0 kernel/rcu/tree.c:3625
       rtnl_register_internal+0x443/0x530 net/core/rtnetlink.c:223
       rtnl_register+0x32/0x70 net/core/rtnetlink.c:273
       ip_rt_init+0x2e6/0x390 net/ipv4/route.c:3750
       ip_init+0xa/0x20 net/ipv4/ip_output.c:1749
       inet_init+0x27c/0x390 net/ipv4/af_inet.c:2009
       do_one_initcall+0x22b/0x7a0 init/main.c:1302
       do_initcall_level+0x157/0x210 init/main.c:1375
       do_initcalls+0x49/0x90 init/main.c:1391
       kernel_init_freeable+0x425/0x5c0 init/main.c:1615
       kernel_init+0x19/0x290 init/main.c:1506
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #0 (krc.lock){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
       krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
       add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
       kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
       trie_delete_elem+0x520/0x690
       bpf_prog_4febba1e7eaf042b+0x42/0x1f0
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1913
       rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
       rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

other info that might help us debug this:

Chain exists of:
  krc.lock --> &base->lock --> &trie->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&trie->lock);
                               lock(&base->lock);
                               lock(&trie->lock);
  lock(krc.lock);

 *** DEADLOCK ***

3 locks held by rcu_preempt/15:
 #0: ffff8880b9028098 (&base->lock){-.-.}-{2:2}, at: lock_timer_base+0x120/0x260 kernel/time/timer.c:946
 #1: ffffffff8c91fc60 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #2: ffff88802368d1b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

stack backtrace:
CPU: 0 PID: 15 Comm: rcu_preempt Not tainted 5.15.167-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
 add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
 kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
 trie_delete_elem+0x520/0x690
 bpf_prog_4febba1e7eaf042b+0x42/0x1f0
 bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
 trace_timer_start include/trace/events/timer.h:52 [inline]
 enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
 internal_add_timer kernel/time/timer.c:611 [inline]
 __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
 schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1913
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>

Crashes (504):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/09/22 16:31 linux-5.15.y 3a5928702e71 6f888b75 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/19 05:54 linux-5.15.y 3a5928702e71 c673ca06 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/21 19:40 linux-5.15.y 4878aadf2d15 edc5149a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/21 18:59 linux-5.15.y 4878aadf2d15 edc5149a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 03:24 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 01:49 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/01 23:03 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/27 21:11 linux-5.15.y 9465fef4ae35 120789fd .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/25 23:20 linux-5.15.y b95c01af2113 0ea90952 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/05 01:19 linux-5.15.y 3a5928702e71 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/04 23:27 linux-5.15.y 3a5928702e71 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/04 11:26 linux-5.15.y 3a5928702e71 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/10/04 08:46 linux-5.15.y 3a5928702e71 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/10/01 16:13 linux-5.15.y 3a5928702e71 ea2b66a6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/29 22:54 linux-5.15.y 3a5928702e71 ba29ff75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/29 21:45 linux-5.15.y 3a5928702e71 ba29ff75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/29 18:59 linux-5.15.y 3a5928702e71 ba29ff75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/27 09:23 linux-5.15.y 3a5928702e71 9314348a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/27 06:37 linux-5.15.y 3a5928702e71 9314348a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/27 03:48 linux-5.15.y 3a5928702e71 9314348a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/26 20:35 linux-5.15.y 3a5928702e71 9314348a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/26 03:45 linux-5.15.y 3a5928702e71 0d19f247 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/24 18:46 linux-5.15.y 3a5928702e71 5643e0e9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/22 22:33 linux-5.15.y 3a5928702e71 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/22 19:19 linux-5.15.y 3a5928702e71 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/22 19:15 linux-5.15.y 3a5928702e71 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/22 10:03 linux-5.15.y 3a5928702e71 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/21 22:48 linux-5.15.y 3a5928702e71 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/21 09:19 linux-5.15.y 3a5928702e71 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/21 04:52 linux-5.15.y 3a5928702e71 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/19 13:10 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/18 20:46 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/18 19:19 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/18 12:09 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/18 03:59 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/18 01:49 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/18 00:46 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/18 00:31 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/17 19:10 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/15 07:09 linux-5.15.y 3a5928702e71 08d8a733 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/15 00:46 linux-5.15.y 3a5928702e71 08d8a733 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/14 12:28 linux-5.15.y 3a5928702e71 c7e35043 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/13 05:50 linux-5.15.y 3a5928702e71 73e8a465 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/12 21:09 linux-5.15.y 3a5928702e71 60e1a995 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/12 08:57 linux-5.15.y 14e468424d3e d94c83d8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/12 07:04 linux-5.15.y 14e468424d3e d94c83d8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/12 05:46 linux-5.15.y 14e468424d3e d94c83d8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/12 03:19 linux-5.15.y 14e468424d3e d94c83d8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/12 01:37 linux-5.15.y 14e468424d3e d94c83d8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/11 23:51 linux-5.15.y 14e468424d3e d94c83d8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/10 22:46 linux-5.15.y 14e468424d3e 86aa7bd7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/09/10 11:32 linux-5.15.y 14e468424d3e 784df80e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/10 08:40 linux-5.15.y 14e468424d3e 784df80e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/09/10 07:30 linux-5.15.y 14e468424d3e 784df80e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
* Struck through repros no longer work on HEAD.