syzbot


possible deadlock in kvfree_call_rcu

Status: upstream: reported C repro on 2024/03/25 23:20
Bug presence: origin:upstream
Labels: missing-backport
[Documentation on labels]
Reported-by: syzbot+cb88665052d93c3f0be3@syzkaller.appspotmail.com
First crash: 90d, last: 3h41m
Fix commit to backport (bisect log) :
tree: upstream
commit f7b68543642136164ce7348945d3ada707c4e635
Author: Rameez Rehman <rameezrehman408@hotmail.com>
Date: Sun Mar 31 20:03:44 2024 +0000

  bpftool: Use simpler indentation in source rST for documentation

  
Bug presence (3)
Date Name Commit Repro Result
2024/06/07 linux-5.15.y (ToT) c61bd26ae81a C [report] possible deadlock in trie_delete_elem
2024/04/23 upstream (ToT) 71b1543c83d6 C [report] possible deadlock in trie_delete_elem
2024/06/07 upstream (ToT) 8a92980606e3 C Didn't crash
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in kvfree_call_rcu origin:upstream C 355 11h07m 94d 0/3 upstream: reported C repro on 2024/03/21 20:02
upstream possible deadlock in kvfree_call_rcu (2) net C unreliable 155 1d01h 13d 0/27 upstream: reported C repro on 2024/06/10 22:50
upstream possible deadlock in kvfree_call_rcu bpf C 1431 20d 89d 26/27 fixed on 2024/06/05 13:52
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/06/07 17:58 4h58m fix candidate upstream job log (1)

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.161-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:3/381 is trying to acquire lock:
ffff8880b9a27e78 (krc.lock){....}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
ffff8880b9a27e78 (krc.lock){....}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
ffff8880b9a27e78 (krc.lock){....}-{2:2}, at: kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597

but task is already holding lock:
ffff888019fab5b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){....}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450
       bpf_prog_2c29ac5cdc6b1842+0x3a/0xda0
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       wb_wakeup_delayed+0xbc/0x100 mm/backing-dev.c:263
       wb_workfn+0x1009/0x1130 fs/fs-writeback.c:2305
       process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
       worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300

-> #1 (&base->lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       lock_timer_base+0x120/0x260 kernel/time/timer.c:946
       __mod_timer+0x1d6/0xeb0 kernel/time/timer.c:1019
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       schedule_delayed_work include/linux/workqueue.h:631 [inline]
       kvfree_call_rcu+0x50e/0x8a0 kernel/rcu/tree.c:3625
       rtnl_register_internal+0x443/0x530 net/core/rtnetlink.c:223
       rtnl_register+0x32/0x70 net/core/rtnetlink.c:273
       ip_rt_init+0x2e6/0x390 net/ipv4/route.c:3750
       ip_init+0xa/0x20 net/ipv4/ip_output.c:1749
       inet_init+0x27c/0x390 net/ipv4/af_inet.c:2005
       do_one_initcall+0x22b/0x7a0 init/main.c:1302
       do_initcall_level+0x157/0x210 init/main.c:1375
       do_initcalls+0x49/0x90 init/main.c:1391
       kernel_init_freeable+0x425/0x5c0 init/main.c:1615
       kernel_init+0x19/0x290 init/main.c:1506
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300

-> #0 (krc.lock){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
       krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
       add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
       kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
       trie_delete_elem+0x520/0x690
       bpf_prog_2c29ac5cdc6b1842+0x3a/0xda0
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       wb_wakeup_delayed+0xbc/0x100 mm/backing-dev.c:263
       wb_workfn+0x1009/0x1130 fs/fs-writeback.c:2305
       process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
       worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300

other info that might help us debug this:

Chain exists of:
  krc.lock --> &base->lock --> &trie->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&trie->lock);
                               lock(&base->lock);
                               lock(&trie->lock);
  lock(krc.lock);

 *** DEADLOCK ***

6 locks held by kworker/u4:3/381:
 #0: ffff88814478d138 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x78a/0x10c0 kernel/workqueue.c:2283
 #1: ffffc90003017d20 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x7d0/0x10c0 kernel/workqueue.c:2285
 #2: ffff88801e4b8388 (&wb->work_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:388 [inline]
 #2: ffff88801e4b8388 (&wb->work_lock){..-.}-{2:2}, at: wb_wakeup_delayed+0x56/0x100 mm/backing-dev.c:261
 #3: ffff8880b9a28098 (&base->lock){-.-.}-{2:2}, at: lock_timer_base+0x120/0x260 kernel/time/timer.c:946
 #4: ffffffff8c91fae0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #5: ffff888019fab5b8 (&trie->lock){....}-{2:2}, at: trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450

stack backtrace:
CPU: 0 PID: 381 Comm: kworker/u4:3 Not tainted 5.15.161-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024
Workqueue: writeback wb_workfn (flush-8:0)
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 krc_this_cpu_lock kernel/rcu/tree.c:3199 [inline]
 add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3506 [inline]
 kvfree_call_rcu+0x1b5/0x8a0 kernel/rcu/tree.c:3597
 trie_delete_elem+0x520/0x690
 bpf_prog_2c29ac5cdc6b1842+0x3a/0xda0
 bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1918
 trace_timer_start include/trace/events/timer.h:52 [inline]
 enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
 internal_add_timer kernel/time/timer.c:611 [inline]
 __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
 queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
 queue_delayed_work include/linux/workqueue.h:527 [inline]
 wb_wakeup_delayed+0xbc/0x100 mm/backing-dev.c:263
 wb_workfn+0x1009/0x1130 fs/fs-writeback.c:2305
 process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
 worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
 </TASK>

Crashes (196):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/06/21 19:40 linux-5.15.y 4878aadf2d15 edc5149a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/21 18:59 linux-5.15.y 4878aadf2d15 edc5149a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 03:24 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/02 01:49 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/04/01 23:03 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/27 21:11 linux-5.15.y 9465fef4ae35 120789fd .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/03/25 23:20 linux-5.15.y b95c01af2113 0ea90952 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/24 06:27 linux-5.15.y 4878aadf2d15 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/23 00:52 linux-5.15.y 4878aadf2d15 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/22 11:03 linux-5.15.y 4878aadf2d15 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/21 12:03 linux-5.15.y 4878aadf2d15 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/21 07:09 linux-5.15.y 4878aadf2d15 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/19 21:10 linux-5.15.y 4878aadf2d15 41b7e219 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/19 00:12 linux-5.15.y 4878aadf2d15 639d6cdf .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/18 19:35 linux-5.15.y 4878aadf2d15 639d6cdf .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/18 17:03 linux-5.15.y 4878aadf2d15 639d6cdf .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/13 23:30 linux-5.15.y c61bd26ae81a a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/13 20:41 linux-5.15.y c61bd26ae81a a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/13 15:51 linux-5.15.y c61bd26ae81a a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/13 09:31 linux-5.15.y c61bd26ae81a 2aa5052f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/13 08:30 linux-5.15.y c61bd26ae81a 2aa5052f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/11 20:09 linux-5.15.y c61bd26ae81a 4d75f4f7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/11 16:36 linux-5.15.y c61bd26ae81a b7d9eb04 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/09 07:50 linux-5.15.y c61bd26ae81a 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/08 01:01 linux-5.15.y c61bd26ae81a 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/07 01:39 linux-5.15.y c61bd26ae81a 121701b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/06 20:56 linux-5.15.y c61bd26ae81a 121701b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/06 03:10 linux-5.15.y c61bd26ae81a 121701b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/05 22:32 linux-5.15.y c61bd26ae81a 5aa1a7c9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/05 21:14 linux-5.15.y c61bd26ae81a 5aa1a7c9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/05 06:43 linux-5.15.y c61bd26ae81a e1e2c66e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/04 02:33 linux-5.15.y c61bd26ae81a a1feae05 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/03 23:16 linux-5.15.y c61bd26ae81a a1feae05 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/03 11:38 linux-5.15.y c61bd26ae81a 0aba2352 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/02 05:18 linux-5.15.y c61bd26ae81a 3113787f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/06/02 00:39 linux-5.15.y c61bd26ae81a 3113787f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/06/01 23:35 linux-5.15.y c61bd26ae81a 3113787f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/30 01:19 linux-5.15.y c61bd26ae81a 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/29 11:41 linux-5.15.y c61bd26ae81a 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/29 07:01 linux-5.15.y c61bd26ae81a 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/27 05:37 linux-5.15.y c61bd26ae81a a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/26 19:29 linux-5.15.y c61bd26ae81a a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/26 15:43 linux-5.15.y c61bd26ae81a a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/26 03:46 linux-5.15.y c61bd26ae81a a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/26 01:48 linux-5.15.y c61bd26ae81a a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/24 17:22 linux-5.15.y 83655231580b 8f98448e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/23 05:09 linux-5.15.y 83655231580b 4d098039 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in kvfree_call_rcu
2024/05/21 03:19 linux-5.15.y 83655231580b c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/20 04:04 linux-5.15.y 83655231580b c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/20 02:06 linux-5.15.y 83655231580b c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
2024/05/19 21:59 linux-5.15.y 83655231580b c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in kvfree_call_rcu
* Struck through repros no longer work on HEAD.