syzbot


INFO: rcu detected stall in NF_HOOK (2)

Status: upstream: reported C repro on 2024/10/10 12:45
Subsystems: netfilter
[Documentation on labels]
Reported-by: syzbot+34c2df040c6cfa15fdfe@syzkaller.appspotmail.com
First crash: 95d, last: 5h32m
Cause bisection: the cause commit could be any of (bisect log):
  781773e3b680 sched/fair: Implement ENQUEUE_DELAYED
  a1c446611e31 sched,freezer: Mark TASK_FROZEN special
  e1459a50ba31 sched: Teach dequeue_task() about special task states
  f12e148892ed sched/fair: Prepare pick_next_task() for delayed dequeue
  152e11f6df29 sched/fair: Implement delayed dequeue
  2e0199df252a sched/fair: Prepare exit/cleanup paths for delayed_dequeue
  54a58a787791 sched/fair: Implement DELAY_ZERO
  
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [netfilter?] INFO: rcu detected stall in NF_HOOK (2) 0 (1) 2024/10/10 12:45
Similar bugs (5)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in NF_HOOK netfilter usb C error 13 102d 117d 0/28 closed as dup on 2024/06/20 17:29
linux-5.15 INFO: rcu detected stall in NF_HOOK 1 222d 222d 0/3 auto-obsoleted due to no activity on 2024/06/15 14:02
linux-6.1 INFO: rcu detected stall in NF_HOOK 14 2d03h 133d 0/3 upstream: reported on 2024/06/04 14:31
linux-5.15 INFO: rcu detected stall in NF_HOOK (2) 1 104d 104d 0/3 auto-obsoleted due to no activity on 2024/10/11 06:09
android-54 BUG: soft lockup in NF_HOOK 1 125d 125d 0/2 auto-obsoleted due to no activity on 2024/09/10 19:37

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	(detected by 0, t=10502 jiffies, g=6461, q=2714666 ncpus=2)
rcu: All QSes seen, last rcu_preempt kthread activity 10503 (4294972492-4294961989), jiffies_till_next_fqs=1, root ->qsmask 0x0
rcu: rcu_preempt kthread starved for 10504 jiffies! g6461 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:26144 pid:17    tgid:17    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5315 [inline]
 __schedule+0x1895/0x4b30 kernel/sched/core.c:6675
 __schedule_loop kernel/sched/core.c:6752 [inline]
 schedule+0x14b/0x320 kernel/sched/core.c:6767
 schedule_timeout+0x1be/0x310 kernel/time/timer.c:2615
 rcu_gp_fqs_loop+0x2df/0x1330 kernel/rcu/tree.c:2045
 rcu_gp_kthread+0xa7/0x3b0 kernel/rcu/tree.c:2247
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 24 Comm: ksoftirqd/1 Not tainted 6.12.0-rc1-syzkaller-00229-gd521db38f339 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:hrtimer_interrupt+0x533/0x990 kernel/time/hrtimer.c:1827
Code: 00 48 8b 5c 24 70 74 08 48 89 df e8 47 65 7c 00 4c 89 23 48 8b 44 24 08 42 0f b6 04 30 84 c0 0f 85 c0 01 00 00 41 80 65 4c fd <4c> 89 ef 48 8b 74 24 20 e8 20 ea 4c 0a 4c 89 e7 31 f6 e8 36 ce 03
RSP: 0018:ffffc90000a18eb8 EFLAGS: 00000006
RAX: 0000000000000000 RBX: ffff8880b872c8e0 RCX: ffff88801d2f0000
RDX: 0000000000010100 RSI: 7fffffffffffffff RDI: 000000527594bc00
RBP: ffff8880b872ca68 R08: ffffffff81822830 R09: 0000000000000000
R10: ffff8880b872d1a0 R11: ffffed10170e5a36 R12: 000000527594bc00
R13: ffff8880b872c880 R14: dffffc0000000000 R15: ffff8880b872cc68
FS:  0000000000000000(0000) GS:ffff8880b8700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005578d972b578 CR3: 000000000e734000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1026 [inline]
 __sysvec_apic_timer_interrupt+0x110/0x3f0 arch/x86/kernel/apic/apic.c:1043
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1037 [inline]
 sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1037
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:stack_trace_consume_entry+0x59/0x280 kernel/stacktrace.c:89
Code: 84 c0 0f 85 06 01 00 00 44 8b 43 10 48 8d 6b 08 49 89 ec 49 c1 ec 03 41 0f b6 04 14 84 c0 0f 85 1a 01 00 00 44 3b 45 00 73 3c <48> 8d 7b 0c 49 89 fd 49 c1 ed 03 41 0f b6 44 15 00 84 c0 0f 85 36
RSP: 0018:ffffc900001e6e30 EFLAGS: 00000287
RAX: 0000000000000000 RBX: ffffc900001e6f60 RCX: ffffffff917b9000
RDX: dffffc0000000000 RSI: ffffffff81808e88 RDI: ffffc900001e6f60
RBP: ffffc900001e6f68 R08: 0000000000000000 R09: ffffc900001e6f70
R10: ffffc900001e6ed0 R11: ffffffff81808f50 R12: 1ffff9200003cded
R13: ffffffff81808f50 R14: ffffc900001e6f60 R15: 1ffff9200003cdee
 arch_stack_walk+0x10e/0x150 arch/x86/kernel/stacktrace.c:27
 stack_trace_save+0x118/0x1d0 kernel/stacktrace.c:122
 kasan_save_stack mm/kasan/common.c:47 [inline]
 kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
 poison_kmalloc_redzone mm/kasan/common.c:377 [inline]
 __kasan_kmalloc+0x98/0xb0 mm/kasan/common.c:394
 kasan_kmalloc include/linux/kasan.h:257 [inline]
 __kmalloc_cache_noprof+0x19c/0x2c0 mm/slub.c:4296
 kmalloc_noprof include/linux/slab.h:878 [inline]
 slab_free_hook mm/slub.c:2295 [inline]
 slab_free mm/slub.c:4580 [inline]
 kmem_cache_free+0x177/0x420 mm/slub.c:4682
 nf_conntrack_free+0x2fd/0x390 net/netfilter/nf_conntrack_core.c:1735
 nf_ct_put include/net/netfilter/nf_conntrack.h:186 [inline]
 nf_conntrack_in+0xb51/0x1890 net/netfilter/nf_conntrack_core.c:2061
 nf_hook_entry_hookfn include/linux/netfilter.h:154 [inline]
 nf_hook_slow+0xc3/0x220 net/netfilter/core.c:626
 nf_hook include/linux/netfilter.h:269 [inline]
 NF_HOOK+0x29e/0x450 include/linux/netfilter.h:312
 __netif_receive_skb_one_core net/core/dev.c:5666 [inline]
 __netif_receive_skb+0x2bf/0x650 net/core/dev.c:5779
 process_backlog+0x662/0x15b0 net/core/dev.c:6111
 __napi_poll+0xcb/0x490 net/core/dev.c:6775
 napi_poll net/core/dev.c:6844 [inline]
 net_rx_action+0x89b/0x1240 net/core/dev.c:6966
 handle_softirqs+0x2c5/0x980 kernel/softirq.c:554
 run_ksoftirqd+0xca/0x130 kernel/softirq.c:927
 smpboot_thread_fn+0x544/0xa30 kernel/smpboot.c:164
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.341 msecs

Crashes (33):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/10/06 12:32 net-next d521db38f339 d7906eff .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/10/15 18:50 upstream eca631b8fe80 14943bb8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in NF_HOOK
2024/10/08 11:47 upstream 87d6aab2389e 402f1df0 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/10/01 20:25 upstream e32cde8d2bd7 ea2b66a6 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/29 00:26 upstream 3efc57369a0c ba29ff75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/27 18:42 upstream e477dba5442c 2b1784d6 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/26 17:25 upstream 11a299a7933e 0d19f247 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in NF_HOOK
2024/09/24 14:27 upstream abf2050f51fd 5643e0e9 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/23 16:35 upstream de5cb0dcb74c 89298aad .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/23 12:30 upstream de5cb0dcb74c 89298aad .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/23 11:29 upstream de5cb0dcb74c 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/23 11:23 upstream de5cb0dcb74c 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/20 09:40 upstream 2004cef11ea0 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/08/11 05:35 upstream 5189dafa4cf9 6f4edef4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in NF_HOOK
2024/08/08 05:49 upstream 6a0e38264012 de12cf65 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in NF_HOOK
2024/08/05 08:14 upstream de9c2c66ad8e 1786a2a8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/07/15 08:54 upstream 0c3836482481 c605e6a2 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/10/12 14:51 upstream 9e4c6c1ad9a1 084d8178 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in NF_HOOK
2024/09/25 11:09 upstream 684a64bf32b6 349a68c4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in NF_HOOK
2024/09/22 09:52 upstream 88264981f208 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in NF_HOOK
2024/08/08 17:15 upstream 6a0e38264012 de12cf65 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in NF_HOOK
2024/08/02 03:37 upstream c0ecd6388360 1e9c4cf3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in NF_HOOK
2024/07/25 07:22 upstream c33ffdb70cc6 b24754ac .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in NF_HOOK
2024/07/12 10:43 upstream 43db1e03c086 eaeb5c15 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in NF_HOOK
2024/10/10 15:38 net a354733c738d 8fbfc0c8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/10/12 04:14 net-next d677aebd663d 084d8178 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/10/08 18:01 net-next 489cee4caeba 402f1df0 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/10/02 21:02 net-next 44badc908f2c a4c7fd36 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/07/22 09:41 net-next d7e78951a8b8 b88348e9 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in NF_HOOK
2024/09/30 15:15 linux-next cea5425829f7 bbd4e0a4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in NF_HOOK
2024/09/27 15:34 linux-next 40e0c9d414f5 2b1784d6 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in NF_HOOK
2024/10/11 02:48 bpf 60f802e2d6e1 8fbfc0c8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-bpf-kasan-gce BUG: soft lockup in NF_HOOK
2024/10/09 06:55 bpf b24d7f0da6ef 402f1df0 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-bpf-kasan-gce BUG: soft lockup in NF_HOOK
* Struck through repros no longer work on HEAD.