syzbot


possible deadlock in trie_update_elem

Status: upstream: reported C repro on 2024/03/19 12:15
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+02f80c2ffc3d585f9512@syzkaller.appspotmail.com
First crash: 278d, last: 6h29m
Fix commit to backport (bisect log) :
tree: upstream
commit 01b44456a7aa7c3b24fa9db7d1714b208b8ef3d8
Author: Mel Gorman <mgorman@techsingularity.net>
Date: Fri Jun 24 12:54:23 2022 +0000

  mm/page_alloc: replace local_lock with normal spinlock

  
Bug presence (2)
Date Name Commit Repro Result
2024/04/29 linux-5.15.y (ToT) b925f60c6ee7 C [report] possible deadlock in trie_delete_elem
2024/04/29 upstream (ToT) e67572cd2204 C Didn't crash
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in trie_update_elem bpf 137 47d 269d 0/28 upstream: reported on 2024/03/27 23:12
linux-6.1 possible deadlock in trie_update_elem 5 128d 262d 0/3 auto-obsoleted due to no activity on 2024/11/24 16:05
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/06/04 14:41 5h46m fix candidate upstream OK (1) job log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.173-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor184/4198 is trying to acquire lock:
ffff8880b9135c68 (lock){..-.}-{2:2}, at: local_lock_acquire+0x7/0x130 include/linux/local_lock_internal.h:28

but task is already holding lock:
ffff8880717bc1b8 (&trie->lock){....}-{2:2}, at: trie_update_elem+0xc5/0xc00 kernel/bpf/lpm_trie.c:323

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&trie->lock){....}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       trie_delete_elem+0x90/0x690 kernel/bpf/lpm_trie.c:450
       0xffffffffa0018385
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1878 [inline]
       bpf_trace_run1+0x168/0x2f0 kernel/trace/bpf_trace.c:1914
       __bpf_trace_mm_page_free_batched+0x41/0x60 include/trace/events/kmem.h:182
       trace_mm_page_free_batched include/trace/events/kmem.h:182 [inline]
       free_unref_page_list+0x79b/0x8e0 mm/page_alloc.c:3465
       release_pages+0x1bb9/0x1f40 mm/swap.c:963
       tlb_batch_pages_flush mm/mmu_gather.c:49 [inline]
       tlb_flush_mmu_free mm/mmu_gather.c:240 [inline]
       tlb_flush_mmu mm/mmu_gather.c:247 [inline]
       tlb_finish_mmu+0x177/0x320 mm/mmu_gather.c:338
       exit_mmap+0x3cd/0x670 mm/mmap.c:3188
       __mmput+0x112/0x3b0 kernel/fork.c:1127
       exit_mm+0x688/0x7f0 kernel/exit.c:550
       do_exit+0x626/0x2480 kernel/exit.c:861
       do_group_exit+0x144/0x310 kernel/exit.c:996
       __do_sys_exit_group kernel/exit.c:1007 [inline]
       __se_sys_exit_group kernel/exit.c:1005 [inline]
       __x64_sys_exit_group+0x3b/0x40 kernel/exit.c:1005
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x66/0xd0

-> #0 (lock){..-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       local_lock_acquire+0x23/0x130 include/linux/local_lock_internal.h:29
       rmqueue_pcplist mm/page_alloc.c:3699 [inline]
       rmqueue mm/page_alloc.c:3742 [inline]
       get_page_from_freelist+0x3227/0x3d40 mm/page_alloc.c:4189
       __alloc_pages+0x272/0x700 mm/page_alloc.c:5464
       alloc_slab_page mm/slub.c:1775 [inline]
       allocate_slab mm/slub.c:1912 [inline]
       new_slab+0xbb/0x4b0 mm/slub.c:1975
       ___slab_alloc+0x6f6/0xe10 mm/slub.c:3008
       __slab_alloc mm/slub.c:3095 [inline]
       slab_alloc_node mm/slub.c:3186 [inline]
       __kmalloc_node+0x1fa/0x390 mm/slub.c:4451
       kmalloc_node include/linux/slab.h:614 [inline]
       bpf_map_kmalloc_node+0xdb/0x160 kernel/bpf/syscall.c:430
       lpm_trie_node_alloc kernel/bpf/lpm_trie.c:290 [inline]
       trie_update_elem+0x1cb/0xc00 kernel/bpf/lpm_trie.c:332
       bpf_map_update_value+0x5d7/0x6c0 kernel/bpf/syscall.c:221
       generic_map_update_batch+0x54d/0x8b0 kernel/bpf/syscall.c:1421
       bpf_map_do_batch+0x4d0/0x620
       __sys_bpf+0x55c/0x670
       __do_sys_bpf kernel/bpf/syscall.c:4755 [inline]
       __se_sys_bpf kernel/bpf/syscall.c:4753 [inline]
       __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:4753
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x66/0xd0

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&trie->lock);
                               lock(lock);
                               lock(&trie->lock);
  lock(lock);

 *** DEADLOCK ***

2 locks held by syz-executor184/4198:
 #0: ffffffff8c91fc60 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #1: ffff8880717bc1b8 (&trie->lock){....}-{2:2}, at: trie_update_elem+0xc5/0xc00 kernel/bpf/lpm_trie.c:323

stack backtrace:
CPU: 1 PID: 4198 Comm: syz-executor184 Not tainted 5.15.173-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/30/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 local_lock_acquire+0x23/0x130 include/linux/local_lock_internal.h:29
 rmqueue_pcplist mm/page_alloc.c:3699 [inline]
 rmqueue mm/page_alloc.c:3742 [inline]
 get_page_from_freelist+0x3227/0x3d40 mm/page_alloc.c:4189
 __alloc_pages+0x272/0x700 mm/page_alloc.c:5464
 alloc_slab_page mm/slub.c:1775 [inline]
 allocate_slab mm/slub.c:1912 [inline]
 new_slab+0xbb/0x4b0 mm/slub.c:1975
 ___slab_alloc+0x6f6/0xe10 mm/slub.c:3008
 __slab_alloc mm/slub.c:3095 [inline]
 slab_alloc_node mm/slub.c:3186 [inline]
 __kmalloc_node+0x1fa/0x390 mm/slub.c:4451
 kmalloc_node include/linux/slab.h:614 [inline]
 bpf_map_kmalloc_node+0xdb/0x160 kernel/bpf/syscall.c:430
 lpm_trie_node_alloc kernel/bpf/lpm_trie.c:290 [inline]
 trie_update_elem+0x1cb/0xc00 kernel/bpf/lpm_trie.c:332
 bpf_map_update_value+0x5d7/0x6c0 kernel/bpf/syscall.c:221
 generic_map_update_batch+0x54d/0x8b0 kernel/bpf/syscall.c:1421
 bpf_map_do_batch+0x4d0/0x620
 __sys_bpf+0x55c/0x670
 __do_sys_bpf kernel/bpf/syscall.c:4755 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:4753 [inline]
 __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:4753
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7f3fca205df9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 c1 17 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fff75b92868 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3fca205df9
RDX: 0000000000000038 RSI: 0000000020000340 RDI: 000000000000001a
RBP: 0000000000000000 R08: 0000000000000006 R09: 0000000000000006
R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
 </TASK>

Crashes (243):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/11/18 06:19 linux-5.15.y 0a51d2d4527b cfe3a04a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/04/18 17:58 linux-5.15.y c52b9710c83d af24b050 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/03/31 09:58 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/12/22 14:16 linux-5.15.y 91786f140358 b4fbdbd4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/12/22 07:28 linux-5.15.y 91786f140358 d7f584ee .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/19 22:19 linux-5.15.y 91786f140358 5905cb39 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/19 04:50 linux-5.15.y 963e654022cc 1432fc84 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/16 17:13 linux-5.15.y 963e654022cc eec85da6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/13 20:49 linux-5.15.y 0a51d2d4527b 7cbfbb3a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/12/11 19:00 linux-5.15.y 0a51d2d4527b ff949d25 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/10 09:26 linux-5.15.y 0a51d2d4527b cfc402b4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/08 05:28 linux-5.15.y 0a51d2d4527b 9ac0fdc6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/12/07 21:49 linux-5.15.y 0a51d2d4527b 9ac0fdc6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/06 09:22 linux-5.15.y 0a51d2d4527b 946d28f0 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/12/03 02:41 linux-5.15.y 0a51d2d4527b 578925bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/12/02 22:46 linux-5.15.y 0a51d2d4527b b499ea68 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/11/28 20:13 linux-5.15.y 0a51d2d4527b 5df23865 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/11/19 17:19 linux-5.15.y 0a51d2d4527b 7d02db5a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/11/19 08:17 linux-5.15.y 0a51d2d4527b 571351cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/11/18 00:53 linux-5.15.y 0a51d2d4527b cfe3a04a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/11/08 13:27 linux-5.15.y 72244eab0dad 179b040e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/11/05 20:57 linux-5.15.y 72244eab0dad da38b4c9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/11/02 23:40 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/10/27 00:47 linux-5.15.y 74cdd62cb470 65e8686b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/10/24 16:43 linux-5.15.y 74cdd62cb470 0d144d1a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/10/22 09:02 linux-5.15.y 584a40a22cb9 a93682b3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/10/20 08:44 linux-5.15.y 584a40a22cb9 cd6fc0a3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/10/16 10:45 linux-5.15.y 3a5928702e71 bde2d81c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/10/05 13:44 linux-5.15.y 3a5928702e71 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/10/04 20:36 linux-5.15.y 3a5928702e71 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/09/29 01:06 linux-5.15.y 3a5928702e71 ba29ff75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/09/28 09:26 linux-5.15.y 3a5928702e71 440b26ec .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/09/25 11:53 linux-5.15.y 3a5928702e71 349a68c4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/09/19 06:46 linux-5.15.y 3a5928702e71 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/09/13 13:28 linux-5.15.y 3a5928702e71 73e8a465 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/09/13 08:40 linux-5.15.y 3a5928702e71 73e8a465 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/09/12 11:16 linux-5.15.y 3a5928702e71 60e1a995 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/09/11 17:27 linux-5.15.y 14e468424d3e 8ab55d0e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/09/09 22:57 linux-5.15.y 14e468424d3e 073f8be2 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/08/30 01:06 linux-5.15.y fa93fa65db6e 9bd464fc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/08/25 05:07 linux-5.15.y fa93fa65db6e d7d32352 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in trie_update_elem
2024/08/23 23:10 linux-5.15.y fa93fa65db6e d7d32352 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/08/19 00:28 linux-5.15.y 7e89efd3ae1c dbc93b08 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/08/14 07:04 linux-5.15.y 7e89efd3ae1c bde81f6f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/08/12 02:50 linux-5.15.y 7e89efd3ae1c 6f4edef4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/08/10 13:23 linux-5.15.y 7e89efd3ae1c 6f4edef4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/08/05 23:20 linux-5.15.y 7e89efd3ae1c e35c337f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
2024/03/19 12:15 linux-5.15.y b95c01af2113 e104824c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in trie_update_elem
* Struck through repros no longer work on HEAD.