syzbot


BUG: soft lockup in rcu_core_si

Status: upstream: reported C repro on 2024/04/13 10:25
Reported-by: syzbot+d9f240112f807cc7e46c@syzkaller.appspotmail.com
First crash: 228d, last: 67d
Fix bisection: fixed by (bisect log) :
commit 11e8ecc5b86037fec43d07b1c162e233e131b1d9
Author: Jakub Sitnicki <jakub@cloudflare.com>
Date: Mon May 27 11:20:07 2024 +0000

  bpf: Allow delete from sockmap/sockhash only if update is allowed

  
Bug presence (2)
Date Name Commit Repro Result
2024/11/02 lts (merge base) 3a5928702e71 C Didn't crash
2024/11/02 upstream (ToT) 11066801dd4b C Didn't crash

Sample crash report:
watchdog: BUG: soft lockup - CPU#1 stuck for 143s! [ksoftirqd/1:25]
Modules linked in:
CPU: 1 PID: 25 Comm: ksoftirqd/1 Not tainted 5.15.156-syzkaller-00821-g29d153aabd54 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
RIP: 0010:native_safe_halt arch/x86/include/asm/irqflags.h:51 [inline]
RIP: 0010:arch_safe_halt arch/x86/include/asm/irqflags.h:89 [inline]
RIP: 0010:kvm_wait+0x147/0x180 arch/x86/kernel/kvm.c:918
Code: 4c 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 44 8b 74 24 1c 75 34 41 0f b6 45 00 44 38 f0 75 10 66 90 0f 00 2d 6b ef 12 04 fb f4 <e9> 24 ff ff ff fb e9 1e ff ff ff 44 89 e9 80 e1 07 38 c1 7c a3 4c
RSP: 0018:ffffc900001a72a0 EFLAGS: 00000246
RAX: 0000000000000003 RBX: 1ffff92000034e58 RCX: ffffffff81552d3f
RDX: dffffc0000000000 RSI: 0000000000000003 RDI: ffff88810a26e348
RBP: ffffc900001a7350 R08: dffffc0000000000 R09: ffffed102144dc6a
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff88810a26e348 R14: 0000000000000003 R15: 1ffff92000034e5c
FS:  0000000000000000(0000) GS:ffff8881f7100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b3265ffff CR3: 000000012a68c000 CR4: 00000000003506a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 pv_wait arch/x86/include/asm/paravirt.h:597 [inline]
 pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline]
 __pv_queued_spin_lock_slowpath+0x6bc/0xc40 kernel/locking/qspinlock.c:508
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:585 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:85 [inline]
 do_raw_spin_lock include/linux/spinlock.h:187 [inline]
 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:136 [inline]
 _raw_spin_lock_bh+0x139/0x1b0 kernel/locking/spinlock.c:178
 sock_hash_delete_elem+0x177/0x400 net/core/sock_map.c:943
 bpf_prog_2c29ac5cdc6b1842+0x3a/0xc98
 bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run2+0xec/0x210 kernel/trace/bpf_trace.c:1917
 __bpf_trace_kfree+0x6f/0x90 include/trace/events/kmem.h:118
 __traceiter_kfree+0x2a/0x40 include/trace/events/kmem.h:118
 trace_kfree include/trace/events/kmem.h:118 [inline]
 kfree+0x1f3/0x220 mm/slub.c:4569
 sk_psock_free_link include/linux/skmsg.h:420 [inline]
 sock_map_del_link net/core/sock_map.c:160 [inline]
 sock_map_unref+0x352/0x4d0 net/core/sock_map.c:182
 sock_hash_delete_elem+0x388/0x400 net/core/sock_map.c:947
 bpf_prog_2c29ac5cdc6b1842+0x3a/0xc98
 bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run2+0xec/0x210 kernel/trace/bpf_trace.c:1917
 __bpf_trace_kfree+0x6f/0x90 include/trace/events/kmem.h:118
 __traceiter_kfree+0x2a/0x40 include/trace/events/kmem.h:118
 trace_kfree include/trace/events/kmem.h:118 [inline]
 kfree+0x1f3/0x220 mm/slub.c:4569
 kvfree+0x35/0x40 mm/util.c:664
 __bpf_prog_put_rcu+0x3b/0xc0 kernel/bpf/syscall.c:1765
 rcu_do_batch+0x57a/0xc10 kernel/rcu/tree.c:2523
 rcu_core+0x517/0x1020 kernel/rcu/tree.c:2763
 rcu_core_si+0x9/0x10 kernel/rcu/tree.c:2776
 __do_softirq+0x26d/0x5bf kernel/softirq.c:565
 run_ksoftirqd+0x23/0x30 kernel/softirq.c:937
 smpboot_thread_fn+0x466/0x8d0 kernel/smpboot.c:164
 kthread+0x421/0x510 kernel/kthread.c:337
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 100 Comm: udevd Not tainted 5.15.156-syzkaller-00821-g29d153aabd54 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
RIP: 0010:native_safe_halt arch/x86/include/asm/irqflags.h:51 [inline]
RIP: 0010:arch_safe_halt arch/x86/include/asm/irqflags.h:89 [inline]
RIP: 0010:kvm_wait+0x147/0x180 arch/x86/kernel/kvm.c:918
Code: 4c 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 44 8b 74 24 1c 75 34 41 0f b6 45 00 44 38 f0 75 10 66 90 0f 00 2d 6b ef 12 04 fb f4 <e9> 24 ff ff ff fb e9 1e ff ff ff 44 89 e9 80 e1 07 38 c1 7c a3 4c
RSP: 0018:ffffc900009d72c0 EFLAGS: 00000246
RAX: 0000000000000001 RBX: 1ffff9200013ae5c RCX: 1ffffffff0d5ab14
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8881f7038ad4
RBP: ffffc900009d7370 R08: dffffc0000000000 R09: ffffed103ee0715b
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff8881f7038ad4 R14: 0000000000000001 R15: 1ffff9200013ae60
FS:  00007f1146f31c80(0000) GS:ffff8881f7000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b32a5ffff CR3: 000000010c52b000 CR4: 00000000003506b0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 pv_wait arch/x86/include/asm/paravirt.h:597 [inline]
 pv_wait_node kernel/locking/qspinlock_paravirt.h:325 [inline]
 __pv_queued_spin_lock_slowpath+0x41b/0xc40 kernel/locking/qspinlock.c:473
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:585 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:85 [inline]
 do_raw_spin_lock include/linux/spinlock.h:187 [inline]
 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:136 [inline]
 _raw_spin_lock_bh+0x139/0x1b0 kernel/locking/spinlock.c:178
 sock_hash_delete_elem+0x177/0x400 net/core/sock_map.c:943
 bpf_prog_2c29ac5cdc6b1842+0x3a/0xc98
 bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run2+0xec/0x210 kernel/trace/bpf_trace.c:1917
 __bpf_trace_kfree+0x6f/0x90 include/trace/events/kmem.h:118
 __traceiter_kfree+0x2a/0x40 include/trace/events/kmem.h:118
 trace_kfree include/trace/events/kmem.h:118 [inline]
 kfree+0x1f3/0x220 mm/slub.c:4569
 kernfs_get_open_node fs/kernfs/file.c:547 [inline]
 kernfs_fop_open+0x73c/0xab0 fs/kernfs/file.c:698
 do_dentry_open+0x81c/0xfd0 fs/open.c:828
 vfs_open+0x73/0x80 fs/open.c:958
 do_open fs/namei.c:3608 [inline]
 path_openat+0x26f0/0x2f40 fs/namei.c:3742
 do_filp_open+0x21c/0x460 fs/namei.c:3769
 do_sys_openat2+0x13f/0x830 fs/open.c:1234
 do_sys_open fs/open.c:1250 [inline]
 __do_sys_openat fs/open.c:1266 [inline]
 __se_sys_openat fs/open.c:1261 [inline]
 __x64_sys_openat+0x243/0x290 fs/open.c:1261
 x64_sys_call+0x6bf/0x9a0 arch/x86/include/generated/asm/syscalls_64.h:258
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7f114705c9a4
Code: 24 20 48 8d 44 24 30 48 89 44 24 28 64 8b 04 25 18 00 00 00 85 c0 75 2c 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 76 60 48 8b 15 55 a4 0d 00 f7 d8 64 89 02 48 83
RSP: 002b:00007fff29a31f90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f114705c9a4
RDX: 0000000000080000 RSI: 00007fff29a320c8 RDI: 00000000ffffff9c
RBP: 00007fff29a320c8 R08: 0000000000000008 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000080000
R13: 0000558edab4bb42 R14: 0000000000000001 R15: 0000000000000000
 </TASK>

Crashes (46):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/09/01 09:16 android13-5.15-lts 29d153aabd54 1eda0d14 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/08/09 23:12 android13-5.15-lts 70e1a731d986 a83d9288 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/07/30 18:24 android13-5.15-lts d0a6506ecafd 6fde257d .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/07/25 06:10 android13-5.15-lts 5356d2f17edf d1a1b0ca .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/07/17 07:15 android13-5.15-lts db06c48ab67e 215bec2d .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/06/24 17:52 android13-5.15-lts 9044d25b8ff5 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/06/24 17:49 android13-5.15-lts 9044d25b8ff5 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/06/08 13:49 android13-5.15-lts 79bd336c7a94 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/06/04 15:42 android13-5.15-lts 79bd336c7a94 11f2afa5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/06/02 21:51 android13-5.15-lts 424f92bcbe8f 3113787f .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/05/30 23:48 android13-5.15-lts 424f92bcbe8f 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/29 20:32 android13-5.15-lts 424f92bcbe8f 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/28 12:45 android13-5.15-lts 424f92bcbe8f f550015e .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/22 21:47 android13-5.15-lts 5d96939590c0 4d098039 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/22 07:47 android13-5.15-lts 5d96939590c0 1014eca7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/05/19 12:06 android13-5.15-lts 5d96939590c0 c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/19 08:27 android13-5.15-lts 5d96939590c0 c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/05/17 15:35 android13-5.15-lts 5d96939590c0 a12e99e7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/05/16 01:27 android13-5.15-lts 5d96939590c0 ef5d53ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/05/09 06:22 android13-5.15-lts 5d96939590c0 20bf80e1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/05 15:19 android13-5.15-lts 5d96939590c0 610f2a54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/05 03:28 android13-5.15-lts 5d96939590c0 610f2a54 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/04 23:00 android13-5.15-lts 5d96939590c0 610f2a54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/05/03 17:14 android13-5.15-lts 5d96939590c0 dd26401e .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/05/02 14:54 android13-5.15-lts 5d96939590c0 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/05/01 02:35 android13-5.15-lts 5d96939590c0 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/28 14:16 android13-5.15-lts 5d96939590c0 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/27 14:50 android13-5.15-lts 5d96939590c0 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/27 09:37 android13-5.15-lts 5d96939590c0 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/27 05:19 android13-5.15-lts 5d96939590c0 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/25 16:25 android13-5.15-lts ad06eaf051cd 8bdc0f22 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/04/20 09:01 android13-5.15-lts ad06eaf051cd af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/20 05:06 android13-5.15-lts ad06eaf051cd af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/19 17:29 android13-5.15-lts ad06eaf051cd af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/18 17:20 android13-5.15-lts ad06eaf051cd af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/18 13:19 android13-5.15-lts ad06eaf051cd af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in rcu_core_si
2024/04/17 22:16 android13-5.15-lts ad06eaf051cd acc528cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/17 11:09 android13-5.15-lts ad06eaf051cd 18f6e127 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/17 06:17 android13-5.15-lts ad06eaf051cd 18f6e127 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/15 06:27 android13-5.15-lts 993bed180178 c8349e48 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/15 05:18 android13-5.15-lts 993bed180178 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/14 21:59 android13-5.15-lts 993bed180178 c8349e48 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/13 10:22 android13-5.15-lts 993bed180178 c8349e48 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
2024/04/07 05:33 android13-5.15-lts 993bed180178 ca620dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15 BUG: soft lockup in rcu_core_si
* Struck through repros no longer work on HEAD.