syzbot


possible deadlock in kvfree_call_rcu

Status: upstream: reported C repro on 2024/03/25 23:20
Bug presence: origin:upstream
[Documentation on labels]
Reported-by: syzbot+cb88665052d93c3f0be3@syzkaller.appspotmail.com
First crash: 42d, last: 10h20m
Bug presence (1)
Date Name Commit Repro Result
2024/04/23 upstream (ToT) 71b1543c83d6 C [report] possible deadlock in trie_delete_elem
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in kvfree_call_rcu origin:upstream C 225 3h11m 46d 0/3 upstream: reported C repro on 2024/03/21 20:02
upstream possible deadlock in kvfree_call_rcu bpf C 1310 4h06m 41d 4/26 upstream: reported C repro on 2024/03/26 19:00

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.153-syzkaller #0 Not tainted
------------------------------------------------------
rcu_preempt/15 is trying to acquire lock:
ffff8880b9b27e78 (krc.lock){....}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
ffff8880b9b27e78 (krc.lock){....}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
ffff8880b9b27e78 (krc.lock){....}-{2:2}, at: kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597

but task is already holding lock:
ffff88807a3f65b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){....}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450
       bpf_prog_2c29ac5cdc6b1842+0x3a/0x2e4
       bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1883
       rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
       rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
       kthread+0x3f6/0x4f0 kernel/kthread.c:319
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298

-> #1 (&base->lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       lock_timer_base+0x120/0x260 kernel/time/timer.c:946
       __mod_timer+0x1d6/0xeb0 kernel/time/timer.c:1019
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       schedule_delayed_work include/linux/workqueue.h:631 [inline]
       kvfree_call_rcu+0x50e/0x8a0 kernel/rcu/tree.c:3625
       rtnl_register_internal+0x443/0x530 net/core/rtnetlink.c:223
       rtnl_register+0x32/0x70 net/core/rtnetlink.c:273
       ip_rt_init+0x2e6/0x390 net/ipv4/route.c:3755
       ip_init+0xa/0x14 net/ipv4/ip_output.c:1749
       inet_init+0x27c/0x38e net/ipv4/af_inet.c:2005
       do_one_initcall+0x22b/0x7a0 init/main.c:1300
       do_initcall_level+0x157/0x207 init/main.c:1373
       do_initcalls+0x49/0x86 init/main.c:1389
       kernel_init_freeable+0x425/0x5b5 init/main.c:1613
       kernel_init+0x19/0x290 init/main.c:1504
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298

-> #0 (krc.lock){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
       krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
       add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
       kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
       trie_delete_elem+0x520/0x690
       bpf_prog_2c29ac5cdc6b1842+0x3a/0x2e4
       bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1883
       rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
       rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
       kthread+0x3f6/0x4f0 kernel/kthread.c:319
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298

other info that might help us debug this:

Chain exists of:
  krc.lock --> &base->lock --> &trie->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&trie->lock);
                               lock(&base->lock);
                               lock(&trie->lock);
  lock(krc.lock);

 *** DEADLOCK ***

3 locks held by rcu_preempt/15:
 #0: ffff8880b9b28098 (&base->lock){-.-.}-{2:2}, at: lock_timer_base+0x120/0x260 kernel/time/timer.c:946
 #1: ffffffff8c91f720 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #2: ffff88807a3f65b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

stack backtrace:
CPU: 1 PID: 15 Comm: rcu_preempt Not tainted 5.15.153-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
 add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
 kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
 trie_delete_elem+0x520/0x690
 bpf_prog_2c29ac5cdc6b1842+0x3a/0x2e4
 bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
 trace_timer_start include/trace/events/timer.h:52 [inline]
 enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
 internal_add_timer kernel/time/timer.c:611 [inline]
 __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
 schedule_timeout+0x1b4/0x300 kernel/time/timer.c:1883
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:319
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>

Crashes (124):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/04/02 03:24 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 01:49 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/01 23:03 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/27 21:11 linux-5.15.y 9465fef4ae35 120789fd .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/25 23:20 linux-5.15.y b95c01af2113 0ea90952 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/06 17:27 linux-5.15.y 284087d4f7d5 d884b519 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/06 16:16 linux-5.15.y 284087d4f7d5 d884b519 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/05 02:44 linux-5.15.y 284087d4f7d5 610f2a54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/04 16:08 linux-5.15.y 284087d4f7d5 610f2a54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/04 10:13 linux-5.15.y 284087d4f7d5 610f2a54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/03 17:02 linux-5.15.y 284087d4f7d5 dd26401e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/03 13:45 linux-5.15.y 284087d4f7d5 dd26401e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/03 05:45 linux-5.15.y 284087d4f7d5 ddfc15a1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/03 02:50 linux-5.15.y 284087d4f7d5 ddfc15a1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/02 01:57 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/01 12:34 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/01 01:04 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/30 22:58 linux-5.15.y b925f60c6ee7 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/30 13:14 linux-5.15.y b925f60c6ee7 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/29 22:32 linux-5.15.y b925f60c6ee7 f10afd69 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/29 17:54 linux-5.15.y b925f60c6ee7 27e33c58 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/29 16:53 linux-5.15.y b925f60c6ee7 27e33c58 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/29 12:54 linux-5.15.y b925f60c6ee7 27e33c58 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/29 01:48 linux-5.15.y b925f60c6ee7 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/28 15:19 linux-5.15.y b925f60c6ee7 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/27 23:55 linux-5.15.y b925f60c6ee7 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/27 21:59 linux-5.15.y b925f60c6ee7 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/27 18:06 linux-5.15.y b925f60c6ee7 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/27 10:58 linux-5.15.y c52b9710c83d 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/27 08:11 linux-5.15.y c52b9710c83d 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/27 04:21 linux-5.15.y c52b9710c83d 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/27 00:28 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/26 20:54 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/26 18:17 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/26 02:02 linux-5.15.y c52b9710c83d 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/24 16:36 linux-5.15.y c52b9710c83d 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 15:21 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 13:55 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 12:07 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 06:19 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 04:02 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/24 00:36 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/23 19:42 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/23 13:02 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/23 08:30 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/23 01:50 linux-5.15.y c52b9710c83d 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/22 16:37 linux-5.15.y c52b9710c83d 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/21 21:02 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/21 09:17 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/04/21 06:13 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/20 04:01 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
* Struck through repros no longer work on HEAD.