syzbot


possible deadlock in trie_update_elem

Status: upstream: reported C repro on 2024/03/19 12:15
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+02f80c2ffc3d585f9512@syzkaller.appspotmail.com
First crash: 49d, last: 12h07m
Bug presence (2)
Date Name Commit Repro Result
2024/04/29 linux-5.15.y (ToT) b925f60c6ee7 C [report] possible deadlock in trie_delete_elem
2024/04/29 upstream (ToT) e67572cd2204 C Didn't crash
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in trie_update_elem bpf 34 10h08m 40d 0/26 upstream: reported on 2024/03/27 23:12
linux-6.1 possible deadlock in trie_update_elem 3 17d 33d 0/3 upstream: reported on 2024/04/04 11:38

Sample crash report:
ODEBUG: Out of memory. ODEBUG disabled
======================================================
WARNING: possible circular locking dependency detected
5.15.156-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor304/3537 is trying to acquire lock:
ffffffff8c7fc358 ((console_sem).lock){....}-{2:2}, at: down_trylock+0x1c/0xa0 kernel/locking/semaphore.c:138

but task is already holding lock:
ffff888022db39b8 (&trie->lock){-...}-{2:2}, at: trie_update_elem+0xc5/0xc00 kernel/bpf/lpm_trie.c:323

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (&trie->lock){-...}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450
       bpf_prog_2c29ac5cdc6b1842+0x3a/0x6f0
       bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
       bpf_trace_run2+0x19e/0x340 kernel/trace/bpf_trace.c:1917
       trace_tlb_flush+0xed/0x110 include/trace/events/tlb.h:38
       switch_mm_irqs_off+0x748/0xa30
       context_switch kernel/sched/core.c:5016 [inline]
       __schedule+0x1167/0x45b0 kernel/sched/core.c:6376
       schedule+0x11b/0x1f0 kernel/sched/core.c:6459
       freezable_schedule include/linux/freezer.h:172 [inline]
       futex_wait_queue_me+0x25b/0x480 kernel/futex/core.c:2863
       futex_wait+0x2f8/0x740 kernel/futex/core.c:2964
       do_futex+0x1414/0x1810 kernel/futex/core.c:3982
       __do_sys_futex kernel/futex/core.c:4059 [inline]
       __se_sys_futex+0x407/0x490 kernel/futex/core.c:4040
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x66/0xd0

-> #2 (&rq->__lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       _raw_spin_lock_nested+0x2d/0x40 kernel/locking/spinlock.c:368
       raw_spin_rq_lock_nested+0x26/0x140 kernel/sched/core.c:475
       raw_spin_rq_lock kernel/sched/sched.h:1326 [inline]
       rq_lock kernel/sched/sched.h:1621 [inline]
       task_fork_fair+0x5d/0x350 kernel/sched/fair.c:11480
       sched_cgroup_fork+0x2d3/0x330 kernel/sched/core.c:4466
       copy_process+0x224a/0x3ef0 kernel/fork.c:2320
       kernel_clone+0x210/0x960 kernel/fork.c:2604
       kernel_thread+0x168/0x1e0 kernel/fork.c:2656
       rest_init+0x21/0x330 init/main.c:704
       start_kernel+0x48c/0x540 init/main.c:1138
       secondary_startup_64_no_verify+0xb1/0xbb

-> #1 (&p->pi_lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       try_to_wake_up+0xae/0x1300 kernel/sched/core.c:4030
       up+0x6e/0x90 kernel/locking/semaphore.c:190
       __up_console_sem+0x11a/0x1e0 kernel/printk/printk.c:256
       console_unlock+0x1145/0x12b0 kernel/printk/printk.c:2760
       vprintk_emit+0xbf/0x150 kernel/printk/printk.c:2274
       dev_vprintk_emit+0x2aa/0x330 drivers/base/core.c:4615
       dev_printk_emit+0xd9/0x120 drivers/base/core.c:4626
       _dev_warn+0x11e/0x170 drivers/base/core.c:4682
       firmware_fallback_sysfs+0x681/0xc90 drivers/base/firmware_loader/fallback.c:654
       _request_firmware+0xbb1/0x1200 drivers/base/firmware_loader/main.c:848
       request_firmware_work_func+0x126/0x270 drivers/base/firmware_loader/main.c:1097
       process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
       worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300

-> #0 ((console_sem).lock){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       down_trylock+0x1c/0xa0 kernel/locking/semaphore.c:138
       __down_trylock_console_sem+0x105/0x250 kernel/printk/printk.c:239
       console_trylock kernel/printk/printk.c:2575 [inline]
       console_trylock_spinning+0x8a/0x3f0 kernel/printk/printk.c:1867
       vprintk_emit+0xa6/0x150 kernel/printk/printk.c:2273
       _printk+0xd1/0x120 kernel/printk/printk.c:2299
       debug_objects_oom+0xb4/0x370 lib/debugobjects.c:472
       debug_object_activate+0x42d/0x4e0 lib/debugobjects.c:698
       debug_rcu_head_queue kernel/rcu/rcu.h:176 [inline]
       kvfree_call_rcu+0xb6/0x8a0 kernel/rcu/tree.c:3587
       trie_update_elem+0x808/0xc00 kernel/bpf/lpm_trie.c:384
       bpf_map_update_value+0x5d7/0x6c0 kernel/bpf/syscall.c:221
       generic_map_update_batch+0x54d/0x8b0 kernel/bpf/syscall.c:1421
       bpf_map_do_batch+0x4d0/0x620
       __sys_bpf+0x55c/0x670
       __do_sys_bpf kernel/bpf/syscall.c:4755 [inline]
       __se_sys_bpf kernel/bpf/syscall.c:4753 [inline]
       __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:4753
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x66/0xd0

other info that might help us debug this:

Chain exists of:
  (console_sem).lock --> &rq->__lock --> &trie->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&trie->lock);
                               lock(&rq->__lock);
                               lock(&trie->lock);
  lock((console_sem).lock);

 *** DEADLOCK ***

2 locks held by syz-executor304/3537:
 #0: ffffffff8c91fb20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #1: ffff888022db39b8 (&trie->lock){-...}-{2:2}, at: trie_update_elem+0xc5/0xc00 kernel/bpf/lpm_trie.c:323

stack backtrace:
CPU: 1 PID: 3537 Comm: syz-executor304 Not tainted 5.15.156-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
 down_trylock+0x1c/0xa0 kernel/locking/semaphore.c:138
 __down_trylock_console_sem+0x105/0x250 kernel/printk/printk.c:239
 console_trylock kernel/printk/printk.c:2575 [inline]
 console_trylock_spinning+0x8a/0x3f0 kernel/printk/printk.c:1867
 vprintk_emit+0xa6/0x150 kernel/printk/printk.c:2273
 _printk+0xd1/0x120 kernel/printk/printk.c:2299
 debug_objects_oom+0xb4/0x370 lib/debugobjects.c:472
 debug_object_activate+0x42d/0x4e0 lib/debugobjects.c:698
 debug_rcu_head_queue kernel/rcu/rcu.h:176 [inline]
 kvfree_call_rcu+0xb6/0x8a0 kernel/rcu/tree.c:3587
 trie_update_elem+0x808/0xc00 kernel/bpf/lpm_trie.c:384
 bpf_map_update_value+0x5d7/0x6c0 kernel/bpf/syscall.c:221
 generic_map_update_batch+0x54d/0x8b0 kernel/bpf/syscall.c:1421
 bpf_map_do_batch+0x4d0/0x620
 __sys_bpf+0x55c/0x670
 __do_sys_bpf kernel/bpf/syscall.c:4755 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:4753 [inline]
 __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:4753
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7fd774224919
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 01 1a 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fd7741e2168 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007fd7742ae348 RCX: 00007fd774224919
RDX: 0000000000000038 RSI: 0000000020000000 RDI: 000000000000001a
RBP: 00007fd7742ae340 R08: 00007fd7741e26c0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fd7742ae34c
R13: 0000000000000000 R14: 00007fff5c8308a0 R15: 00007fff5c830988
 </TASK>

Crashes (126):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/04/18 17:58 linux-5.15.y c52b9710c83d af24b050 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/03/31 09:58 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/05/07 02:30 linux-5.15.y 284087d4f7d5 fa7a5cf0 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/05/06 03:20 linux-5.15.y 284087d4f7d5 610f2a54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/05/03 17:11 linux-5.15.y 284087d4f7d5 dd26401e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/05/03 13:32 linux-5.15.y 284087d4f7d5 dd26401e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/05/02 20:11 linux-5.15.y 284087d4f7d5 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/05/01 03:19 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/05/01 01:00 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/30 20:51 linux-5.15.y b925f60c6ee7 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/30 13:04 linux-5.15.y b925f60c6ee7 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/30 00:49 linux-5.15.y b925f60c6ee7 f10afd69 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/29 13:29 linux-5.15.y b925f60c6ee7 27e33c58 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/29 02:31 linux-5.15.y b925f60c6ee7 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/28 06:35 linux-5.15.y b925f60c6ee7 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/27 14:57 linux-5.15.y c52b9710c83d 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/27 06:25 linux-5.15.y c52b9710c83d 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/27 03:16 linux-5.15.y c52b9710c83d 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/26 21:35 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/26 17:24 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/26 13:10 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/25 16:36 linux-5.15.y c52b9710c83d 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/24 13:35 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/24 06:43 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/24 04:38 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/23 22:00 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/23 12:58 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/23 09:15 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/22 22:30 linux-5.15.y c52b9710c83d 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/22 20:52 linux-5.15.y c52b9710c83d 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/22 13:31 linux-5.15.y c52b9710c83d 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/22 11:49 linux-5.15.y c52b9710c83d 36c961ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/22 03:31 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/20 19:09 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/18 23:58 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/18 11:08 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/18 09:02 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/17 23:09 linux-5.15.y c52b9710c83d acc528cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/17 20:23 linux-5.15.y c52b9710c83d acc528cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/17 16:13 linux-5.15.y c52b9710c83d acc528cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/17 11:45 linux-5.15.y c52b9710c83d 18f6e127 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/17 02:28 linux-5.15.y fa3df276cd36 18f6e127 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/16 13:29 linux-5.15.y fa3df276cd36 0d592ce4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/16 12:21 linux-5.15.y fa3df276cd36 0d592ce4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/16 08:34 linux-5.15.y fa3df276cd36 0d592ce4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/16 07:24 linux-5.15.y fa3df276cd36 0d592ce4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/16 03:39 linux-5.15.y fa3df276cd36 0d592ce4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/04/15 18:25 linux-5.15.y fa3df276cd36 b9af7e61 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/15 16:59 linux-5.15.y fa3df276cd36 b9af7e61 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/03/19 12:15 linux-5.15.y b95c01af2113 e104824c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
* Struck through repros no longer work on HEAD.